Predict Telco Customer Churn

Build a churn model and rank telco customers for focused retention.

Introduction

30 Second Summary

Retention teams cannot contact every customer who might leave. Without a reliable ranking, limited outreach can miss the customers most likely to need attention.

In this project, you will build a churn prediction workflow in a JupyterHub notebook. The final ranking helps a retention team focus limited outreach on customers with the highest predicted risk.

What You'll Build

Picture running one notebook that finishes with a ranked customer list ready for a focused retention campaign.

By the end of this project, you'll have:

A visible data audit that confirms the exact 7,043-row, 21-column schema before showing the churn split in a two-bar chart.
A defensible model choice where validation ROC AUC compares logistic regression with random forest before one held-out evaluation.
A ranked retention list that places the highest-risk customers first and gives each one a churn probability.
Secret Mission: A 100-contact campaign test that measures how many actual churners the shortlist captures.

Are there any prerequisites?

You need intermediate Python and machine-learning experience.

The BlastChar Kaggle Telco Customer Churn CSV must already be loaded into a pandas DataFrame named df in your existing JupyterHub notebook.

Before We Start

Before you begin modeling, commit to the decision this notebook needs to support. You are predicting customer churn so a retention team can focus its outreach on the customers with the highest ranked churn scores.

Validate the Dataset

Your existing notebook already holds the df DataFrame. A plausible-looking table can still be the wrong input.

A trustworthy churn model starts with proof that this object matches the intended dataset. This step also preserves a clean source copy before any values change.

In this step, get ready to:

Audit the installed library versions against verified references.
Confirm the exact 7,043-row schema and visualize the target labels.
Create a clean modeling copy with separate identifiers, features, and targets.

Audit the library versions

A version audit makes the notebook environment reproducible. It records each installed package beside the verified reference version.

Return to the JupyterHub notebook from earlier.
Add a new code cell directly below the existing cell that loads df.
Paste the import group into the new cell by using the code below:

# Cell 1: Imports and reference-version audit
from importlib.metadata import version

import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
from sklearn.compose import ColumnTransformer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
    roc_curve,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

What do these imports prepare?

Pandas handles the tabular dataset.
Matplotlib creates the chart figures.
Seaborn provides the statistical chart functions.
Scikit-learn supplies the preprocessing, model, split, and evaluation tools used throughout the notebook.

Save telco_customer_churn_prediction.ipynb.
Run the import cell.

The cell should finish with an execution number and no exception. That clears the environment check because every required import is available.

Did an import stop the cell?

Confirm that the notebook is using the existing Python environment where the project libraries are available.
Restart the notebook environment if a recently installed library is still unavailable.

Help me diagnose an import failure in my churn-prediction notebook.

Append the version audit beneath the imports in the same cell by pasting the code below:

reference_versions = {
    "pandas": "3.0.5",
    "scikit-learn": "1.9.0",
    "matplotlib": "3.11.1",
    "seaborn": "0.13.2",
}

version_audit = pd.DataFrame(
    [
        {
            "package": package,
            "installed": version(package),
            "reference": reference,
        }
        for package, reference in reference_versions.items()
    ]
)
version_audit

What does the version audit show?

The reference dictionary stores the verified package versions used for this project.
The audit reads the corresponding version from your notebook environment.
The version_audit table places the installed and reference values side by side.

Save telco_customer_churn_prediction.ipynb.
Run the completed version-audit cell.

You should see a four-row table with package, installed, and reference columns. Good work. Your notebook now carries an environment record alongside its analysis.

Missing the version table?

Confirm that version_audit remains the final line of the cell.
Check that each package name matches the version reference exactly.

Help me fix the missing version-audit table.

Validate the exact schema

Dataset validation checks the shape, column order, and target labels before modeling begins. These checks prevent a different table from quietly producing misleading results.

Add a new code cell below the version-audit cell.
Define the required column order in that cell by pasting the code below:

# Cell 2: Validate and clean the exact dataset
expected_columns = [
    "customerID",
    "gender",
    "SeniorCitizen",
    "Partner",
    "Dependents",
    "tenure",
    "PhoneService",
    "MultipleLines",
    "InternetService",
    "OnlineSecurity",
    "OnlineBackup",
    "DeviceProtection",
    "TechSupport",
    "StreamingTV",
    "StreamingMovies",
    "Contract",
    "PaperlessBilling",
    "PaymentMethod",
    "MonthlyCharges",
    "TotalCharges",
    "Churn",
]

Why preserve the column order?

The expected_columns list acts as an ordered contract for the dataset.
The later assertion detects missing, extra, renamed, or rearranged columns.

Save telco_customer_churn_prediction.ipynb.
Run the schema-definition cell.

The cell should finish with an execution number. The ordered schema is now available for the validation checks.

Did the schema cell stop?

Compare the quotation marks around every column name.
Check for a missing comma between adjacent column names.

Help me find the syntax problem in my expected column list.

Append the validation checks and churn chart beneath expected_columns by pasting the code below:

assert df.shape == (7043, 21), f"Expected (7043, 21), found {df.shape}"
assert list(df.columns) == expected_columns, "The loaded columns do not match the expected Kaggle schema."
assert set(df["Churn"]) == {"No", "Yes"}, "Expected Churn labels to be No and Yes."

work_df = df.copy()
churn_counts = work_df["Churn"].value_counts()

plt.figure(figsize=(6, 4))
sns.barplot(x=churn_counts.index, y=churn_counts.values, errorbar=None, color="steelblue")
plt.title("Telco Customer Churn Distribution")
plt.xlabel("Churn")
plt.ylabel("Customers")
plt.tight_layout()
plt.show()

What do these checks prove?

The first assertion confirms the required 7,043 rows and 21 columns.
The second assertion compares the actual column order with expected_columns.
The third assertion confirms that the target contains the expected No and Yes labels.
The work_df copy protects the original df from later transformations.
The bar chart makes the target imbalance visible before modeling.

Before you run the expanded cell, do you expect all three assertions to pass with your current df?

Save telco_customer_churn_prediction.ipynb.
Run the expanded dataset-validation cell.

Your output should show a two-bar chart for No and Yes. The No bar should be taller.

The uninterrupted run confirms the exact shape, ordered schema, and target labels. You have now proved that the notebook is using the intended Telco dataset.

Did an assertion stop the cell?

Return to the earlier loading cell to confirm that df still refers to the intended Telco CSV.
Compare the displayed shape or column names with the assertion that stopped.
Keep the validation values unchanged because they define the required dataset.

Help me diagnose which Telco dataset assertion failed.

Prepare the modeling copy

The model needs numeric values for charges and the churn target. A separate customer_ids series keeps each future score connected to the correct customer.

Add a new code cell below the dataset-validation cell.
Paste the model-ready transformations into the new cell by using the code below:

work_df["TotalCharges"] = pd.to_numeric(work_df["TotalCharges"], errors="coerce")
work_df["Churn"] = [1 if value == "Yes" else 0 for value in work_df["Churn"]]

customer_ids = work_df["customerID"]
X = work_df.drop(columns=["customerID", "Churn"])
y = work_df["Churn"]

numeric_features = ["SeniorCitizen", "tenure", "MonthlyCharges", "TotalCharges"]
categorical_features = [column for column in X.columns if column not in numeric_features]

print(f"Rows and columns: {work_df.shape}")
print(f"Missing TotalCharges after conversion: {work_df['TotalCharges'].isna().sum()}")
print(f"Numeric features: {numeric_features}")
print(f"Categorical feature count: {len(categorical_features)}")

How does this prepare the data?

Numeric conversion changes invalid TotalCharges values into missing values that the later preprocessing pipeline can handle.
The target conversion maps No to 0.
The same conversion maps Yes to 1.
The customer_ids series preserves identifiers outside the model features.
The X matrix contains the customer attributes.
The y series contains the encoded churn target.
The feature lists separate numeric columns from categorical columns for later preprocessing.

Before you run this cell, which printed line should confirm that the clean copy still has the full dataset shape?

Save telco_customer_churn_prediction.ipynb.
Run the model-ready transformation cell.

You should see Rows and columns: (7043, 21). The conversion also prints a numeric count for missing TotalCharges values.

You should see four numeric feature names. The final line reports a categorical feature count of 15.

Seeing a column-related failure?

Confirm that the dataset-validation cell ran successfully before this cell.
Check that TotalCharges, Churn, and customerID match the validated column names exactly.
Rerun the earlier cells in notebook order after restarting the notebook environment.

Help me troubleshoot the model-ready transformation cell.

✔️ Awesome, I've got everything!

Your notebook now contains the complete version audit, dataset validation, churn chart, and model-ready data definitions.

ⓧ I'd like to double check the full code

# Cell 1: Imports and reference-version audit
from importlib.metadata import version

import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
from sklearn.compose import ColumnTransformer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
    roc_curve,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

reference_versions = {
    "pandas": "3.0.5",
    "scikit-learn": "1.9.0",
    "matplotlib": "3.11.1",
    "seaborn": "0.13.2",
}

version_audit = pd.DataFrame(
    [
        {
            "package": package,
            "installed": version(package),
            "reference": reference,
        }
        for package, reference in reference_versions.items()
    ]
)
version_audit

# Cell 2: Validate and clean the exact dataset
expected_columns = [
    "customerID",
    "gender",
    "SeniorCitizen",
    "Partner",
    "Dependents",
    "tenure",
    "PhoneService",
    "MultipleLines",
    "InternetService",
    "OnlineSecurity",
    "OnlineBackup",
    "DeviceProtection",
    "TechSupport",
    "StreamingTV",
    "StreamingMovies",
    "Contract",
    "PaperlessBilling",
    "PaymentMethod",
    "MonthlyCharges",
    "TotalCharges",
    "Churn",
]

assert df.shape == (7043, 21), f"Expected (7043, 21), found {df.shape}"
assert list(df.columns) == expected_columns, "The loaded columns do not match the expected Kaggle schema."
assert set(df["Churn"]) == {"No", "Yes"}, "Expected Churn labels to be No and Yes."

work_df = df.copy()
churn_counts = work_df["Churn"].value_counts()

plt.figure(figsize=(6, 4))
sns.barplot(x=churn_counts.index, y=churn_counts.values, errorbar=None, color="steelblue")
plt.title("Telco Customer Churn Distribution")
plt.xlabel("Churn")
plt.ylabel("Customers")
plt.tight_layout()
plt.show()

work_df["TotalCharges"] = pd.to_numeric(work_df["TotalCharges"], errors="coerce")
work_df["Churn"] = [1 if value == "Yes" else 0 for value in work_df["Churn"]]

customer_ids = work_df["customerID"]
X = work_df.drop(columns=["customerID", "Churn"])
y = work_df["Churn"]

numeric_features = ["SeniorCitizen", "tenure", "MonthlyCharges", "TotalCharges"]
categorical_features = [column for column in X.columns if column not in numeric_features]

print(f"Rows and columns: {work_df.shape}")
print(f"Missing TotalCharges after conversion: {work_df['TotalCharges'].isna().sum()}")
print(f"Numeric features: {numeric_features}")
print(f"Categorical feature count: {len(categorical_features)}")

Your source data is validated without changing the original df. Next, you can test whether a majority-only prediction offers any real value for retention.

Expose the Accuracy Trap

Your notebook now proves that the source data has the expected schema. The next goal is to create a fair test of how useful a churn model really is.

A majority-class model can score well on overall accuracy while missing the customers a retention team needs to find. A deliberately simple baseline makes the gap visible through churn recall.

In this step, get ready to:

Create stratified training, validation, and untouched test partitions.
Run a majority-class baseline against the validation data.
Use churn recall to judge whether the baseline supports focused retention.

Create stratified data partitions

Stratification keeps the churn ratio similar across each partition. The test partition stays untouched until the final evaluation.

Add a code cell beneath Cell 2 in the notebook from earlier.
Create the three partitions by running this code in the new cell:

# Cell 3: Create train, validation, and test partitions
X_train_full, X_test, y_train_full, y_test, _, id_test = train_test_split(
    X,
    y,
    customer_ids,
    test_size=0.20,
    random_state=42,
    stratify=y,
)

X_train, X_val, y_train, y_val = train_test_split(
    X_train_full,
    y_train_full,
    test_size=0.25,
    random_state=42,
    stratify=y_train_full,
)

print(f"Training rows: {len(X_train)}")
print(f"Validation rows: {len(X_val)}")
print(f"Test rows: {len(X_test)}")

What does this code do?

The first train_test_split() call reserves 20 percent of the rows for the final test.
The second train_test_split() call divides the remaining rows into training and validation partitions.
Splitting customer_ids beside the features keeps id_test aligned with X_test.
Using random_state=42 makes the row assignments reproducible.

Save telco_customer_churn_prediction.ipynb.
Confirm that the output lists 4225 training rows.
Confirm that the output lists 1409 validation rows.
Confirm that the output lists 1409 test rows.

Partition cell not running?

Run Cell 2 again if X, y, or customer_ids is unavailable.
Check that all three objects contain the same number of rows before running the split.

Help me diagnose my train, validation, and test split.

Run the majority-class baseline

A DummyClassifier from scikit-learn provides a deliberately simple benchmark. Its most-frequent strategy always predicts the most common class observed during training.

Before the run, one question matters: can always choosing the most common label still produce convincing accuracy.

Add a code cell beneath the partition output.
Test the majority-class strategy by running this code in the new cell:

# Cell 4: Expose the majority-class baseline
baseline = DummyClassifier(strategy="most_frequent")
baseline.fit(X_train, y_train)
baseline_predictions = baseline.predict(X_val)

print("Majority-class baseline on validation data")
print(classification_report(y_val, baseline_predictions, zero_division=0))

What does this code do?

The baseline learns which target class appears most often in y_train.
The baseline_predictions test that strategy against the separate validation rows.
The classification report compares precision, recall, and F1 for both target classes.
The zero_division=0 setting reports zero for metrics that the baseline cannot calculate from its predictions.

You will see accuracy around 0.73. The churn class has recall of 0.00.

You found the accuracy trap: the baseline can look acceptable on accuracy while identifying no churners. That shortfall is the intended result of this test.

Save telco_customer_churn_prediction.ipynb.
Confirm that the classification report shows churn-class recall of 0.00.

Classification report not showing?

Run the partition cell again if X_train, y_train, X_val, or y_val is unavailable.
Check that baseline_predictions uses X_val.
Check that the baseline fits only on X_train and y_train.

Help me fix my majority-class classification report.

✔️ Awesome, I've got everything!

Great. Double check that you saved the notebook with both new cells.

ⓧ I'd like to double check the full code

# Cell 1: Imports and reference-version audit
from importlib.metadata import version

import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
from sklearn.compose import ColumnTransformer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
    roc_curve,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

reference_versions = {
    "pandas": "3.0.5",
    "scikit-learn": "1.9.0",
    "matplotlib": "3.11.1",
    "seaborn": "0.13.2",
}

version_audit = pd.DataFrame(
    [
        {
            "package": package,
            "installed": version(package),
            "reference": reference,
        }
        for package, reference in reference_versions.items()
    ]
)
version_audit

# Cell 2: Validate and clean the exact dataset
expected_columns = [
    "customerID",
    "gender",
    "SeniorCitizen",
    "Partner",
    "Dependents",
    "tenure",
    "PhoneService",
    "MultipleLines",
    "InternetService",
    "OnlineSecurity",
    "OnlineBackup",
    "DeviceProtection",
    "TechSupport",
    "StreamingTV",
    "StreamingMovies",
    "Contract",
    "PaperlessBilling",
    "PaymentMethod",
    "MonthlyCharges",
    "TotalCharges",
    "Churn",
]

assert df.shape == (7043, 21), f"Expected (7043, 21), found {df.shape}"
assert list(df.columns) == expected_columns, "The loaded columns do not match the expected Kaggle schema."
assert set(df["Churn"]) == {"No", "Yes"}, "Expected Churn labels to be No and Yes."

work_df = df.copy()
churn_counts = work_df["Churn"].value_counts()

plt.figure(figsize=(6, 4))
sns.barplot(x=churn_counts.index, y=churn_counts.values, errorbar=None, color="steelblue")
plt.title("Telco Customer Churn Distribution")
plt.xlabel("Churn")
plt.ylabel("Customers")
plt.tight_layout()
plt.show()

work_df["TotalCharges"] = pd.to_numeric(work_df["TotalCharges"], errors="coerce")
work_df["Churn"] = [1 if value == "Yes" else 0 for value in work_df["Churn"]]

customer_ids = work_df["customerID"]
X = work_df.drop(columns=["customerID", "Churn"])
y = work_df["Churn"]

numeric_features = ["SeniorCitizen", "tenure", "MonthlyCharges", "TotalCharges"]
categorical_features = [column for column in X.columns if column not in numeric_features]

print(f"Rows and columns: {work_df.shape}")
print(f"Missing TotalCharges after conversion: {work_df['TotalCharges'].isna().sum()}")
print(f"Numeric features: {numeric_features}")
print(f"Categorical feature count: {len(categorical_features)}")

# Cell 3: Create train, validation, and test partitions
X_train_full, X_test, y_train_full, y_test, _, id_test = train_test_split(
    X,
    y,
    customer_ids,
    test_size=0.20,
    random_state=42,
    stratify=y,
)

X_train, X_val, y_train, y_val = train_test_split(
    X_train_full,
    y_train_full,
    test_size=0.25,
    random_state=42,
    stratify=y_train_full,
)

print(f"Training rows: {len(X_train)}")
print(f"Validation rows: {len(X_val)}")
print(f"Test rows: {len(X_test)}")

# Cell 4: Expose the majority-class baseline
baseline = DummyClassifier(strategy="most_frequent")
baseline.fit(X_train, y_train)
baseline_predictions = baseline.predict(X_val)

print("Majority-class baseline on validation data")
print(classification_report(y_val, baseline_predictions, zero_division=0))

Record why recall matters

Accuracy measures the share of all predictions that are correct. Churn recall measures the share of actual churners that the model finds.

A retention team cannot contact a churner that the model never identifies. The baseline therefore provides no useful shortlist despite its apparent accuracy.

Capture the visible classification report in the checkpoint below.
Record why churn recall must accompany accuracy in this retention workflow.

You caught the accuracy trap by proving that the majority baseline misses every churner. Next, you will replace that weak benchmark with models that learn from customer attributes.

Build Leakage-Safe Pipelines

The majority baseline made the problem visible. Its apparent accuracy came with zero useful churn recall.

Customer attributes now need to drive each prediction. A ColumnTransformer gives numeric columns and categorical columns their own preparation paths.

A Pipeline keeps those transformations inside model training. Each pipeline learns preprocessing rules from the training rows only.

In this step, get ready to:

Build separate preprocessing branches for numeric features and categorical features.
Create logistic-regression and random-forest pipelines.
Compare both models using validation ROC AUC.

Build the preprocessing branches

The dataset mixes numeric measurements with categorical labels. Each data type needs transformations suited to its values.

Add one code cell directly below Cell 4 in the notebook from earlier.
Copy the preprocessing function below into Cell 5:

# Cell 5: Build leakage-safe preprocessing and model pipelines
def make_preprocessor():
    numeric_pipeline = Pipeline(
        steps=[
            ("imputer", SimpleImputer(strategy="median")),
            ("scaler", StandardScaler()),
        ]
    )
    categorical_pipeline = Pipeline(
        steps=[
            ("imputer", SimpleImputer(strategy="most_frequent")),
            ("encoder", OneHotEncoder(handle_unknown="ignore")),
        ]
    )
    return ColumnTransformer(
        transformers=[
            ("numeric", numeric_pipeline, numeric_features),
            ("categorical", categorical_pipeline, categorical_features),
        ]
    )

What does this code do?

The numeric branch replaces missing values with each feature's median.
The numeric branch standardizes feature scales before logistic regression uses them.
The categorical branch replaces missing values with each feature's most frequent value.
The categorical branch converts category labels into one-hot encoded columns.
The make_preprocessor() function returns both branches inside one transformer.

Save telco_customer_churn_prediction.ipynb.
Execute Cell 5 using the notebook toolbar.

You should see Cell 5 receive an execution count without a traceback. This confirms that the preprocessing constructor is ready.

Preprocessing cell showing a traceback?

Check that numeric_features and categorical_features still exist from the dataset-validation cell.

Confirm that every closing parenthesis matches the preprocessing code above.

Still stuck? Help me debug my preprocessing function.

Create the model pipelines

Logistic regression provides an interpretable linear baseline. A random forest adds a nonlinear comparison based on many decision trees.

Each model receives a fresh preprocessor. This keeps imputation and encoding fitted to X_train during the next training loop.

Move to the end of Cell 5.
Add the model dictionary by copying this code below the preprocessing function:

models = {
    "Logistic Regression": Pipeline(
        steps=[
            ("preprocessor", make_preprocessor()),
            ("classifier", LogisticRegression(solver="lbfgs", max_iter=1000)),
        ]
    ),
    "Random Forest": Pipeline(
        steps=[
            ("preprocessor", make_preprocessor()),
            (
                "classifier",
                RandomForestClassifier(
                    n_estimators=300,
                    min_samples_leaf=5,
                    n_jobs=-1,
                    random_state=42,
                ),
            ),
        ]
    ),
}

How do these pipelines stay leakage-safe?

Each model stores preprocessing as its first pipeline step. Calling fit() later learns imputation values and category mappings from the supplied training rows.

The validation rows pass through those learned transformations during scoring. They never influence how the transformations are fitted.

Save telco_customer_churn_prediction.ipynb.
Execute Cell 5 using the notebook toolbar.

Cell 5 should finish with a new execution count and no traceback. The models dictionary is now available to the notebook.

Model dictionary failing to execute?

Confirm that make_preprocessor() executed before the models dictionary is created.

Check the indentation around the random-forest classifier tuple. Its closing parentheses must match the reference code.

Need another pair of eyes? Help me find the pipeline construction error.

Train and compare the models

ROC AUC measures how well each model ranks churners above non-churners across decision thresholds. Validation data provides this comparison while the test partition stays untouched.

The positive-class helper finds class 1 explicitly. This keeps the churn-probability column aligned with the encoded target.

Move to the end of Cell 5.
Add the training and comparison logic by copying this code below the model dictionary:

def positive_probabilities(model, X_data):
    classifier = model.named_steps["classifier"]
    positive_class_index = list(classifier.classes_).index(1)
    return model.predict_proba(X_data)[:, positive_class_index]


validation_probabilities = {}
validation_rows = []

for model_name, model in models.items():
    model.fit(X_train, y_train)
    probabilities = positive_probabilities(model, X_val)
    validation_probabilities[model_name] = probabilities
    validation_rows.append(
        {
            "model": model_name,
            "validation_roc_auc": roc_auc_score(y_val, probabilities),
        }
    )

model_comparison = pd.DataFrame(validation_rows).sort_values(
    "validation_roc_auc",
    ascending=False,
    ignore_index=True,
)
model_comparison

What does the comparison code do?

The positive_probabilities() helper returns the predicted probability for churn class 1.
The training loop fits each complete pipeline using X_train and y_train.
Each fitted pipeline scores X_val without refitting its preprocessing steps.
The final table sorts both validation ROC AUC scores from highest to lowest.

Save telco_customer_churn_prediction.ipynb.
Make one mental prediction about which model will rank first on validation ROC AUC.
Execute the completed Cell 5 using the notebook toolbar.

You should see a two-row table with Logistic Regression and Random Forest sorted by validation_roc_auc from highest to lowest.

That is the baseline limitation resolved. Both models now use customer attributes through preprocessing fitted only on the training partition.

Comparison table missing or incomplete?

Run the earlier split cells before Cell 5 so X_train and X_val are available.

Confirm that the final line of Cell 5 is model_comparison so JupyterHub displays the table.

If training stops with another issue, help me debug the validation comparison.

✔️ Awesome, I've got everything!

Great work. Save the notebook with the two fitted pipelines and the sorted validation comparison in place.

ⓧ I'd like to double check the full code

Compare your notebook with this cumulative reference. Cells 1 through 5 should appear in this order.

# Cell 1: Imports and reference-version audit
from importlib.metadata import version

import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
from sklearn.compose import ColumnTransformer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
    roc_curve,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

reference_versions = {
    "pandas": "3.0.5",
    "scikit-learn": "1.9.0",
    "matplotlib": "3.11.1",
    "seaborn": "0.13.2",
}

version_audit = pd.DataFrame(
    [
        {
            "package": package,
            "installed": version(package),
            "reference": reference,
        }
        for package, reference in reference_versions.items()
    ]
)
version_audit

# Cell 2: Validate and clean the exact dataset
expected_columns = [
    "customerID",
    "gender",
    "SeniorCitizen",
    "Partner",
    "Dependents",
    "tenure",
    "PhoneService",
    "MultipleLines",
    "InternetService",
    "OnlineSecurity",
    "OnlineBackup",
    "DeviceProtection",
    "TechSupport",
    "StreamingTV",
    "StreamingMovies",
    "Contract",
    "PaperlessBilling",
    "PaymentMethod",
    "MonthlyCharges",
    "TotalCharges",
    "Churn",
]

assert df.shape == (7043, 21), f"Expected (7043, 21), found {df.shape}"
assert list(df.columns) == expected_columns, "The loaded columns do not match the expected Kaggle schema."
assert set(df["Churn"]) == {"No", "Yes"}, "Expected Churn labels to be No and Yes."

work_df = df.copy()
churn_counts = work_df["Churn"].value_counts()

plt.figure(figsize=(6, 4))
sns.barplot(x=churn_counts.index, y=churn_counts.values, errorbar=None, color="steelblue")
plt.title("Telco Customer Churn Distribution")
plt.xlabel("Churn")
plt.ylabel("Customers")
plt.tight_layout()
plt.show()

work_df["TotalCharges"] = pd.to_numeric(work_df["TotalCharges"], errors="coerce")
work_df["Churn"] = [1 if value == "Yes" else 0 for value in work_df["Churn"]]

customer_ids = work_df["customerID"]
X = work_df.drop(columns=["customerID", "Churn"])
y = work_df["Churn"]

numeric_features = ["SeniorCitizen", "tenure", "MonthlyCharges", "TotalCharges"]
categorical_features = [column for column in X.columns if column not in numeric_features]

print(f"Rows and columns: {work_df.shape}")
print(f"Missing TotalCharges after conversion: {work_df['TotalCharges'].isna().sum()}")
print(f"Numeric features: {numeric_features}")
print(f"Categorical feature count: {len(categorical_features)}")

# Cell 3: Create train, validation, and test partitions
X_train_full, X_test, y_train_full, y_test, _, id_test = train_test_split(
    X,
    y,
    customer_ids,
    test_size=0.20,
    random_state=42,
    stratify=y,
)

X_train, X_val, y_train, y_val = train_test_split(
    X_train_full,
    y_train_full,
    test_size=0.25,
    random_state=42,
    stratify=y_train_full,
)

print(f"Training rows: {len(X_train)}")
print(f"Validation rows: {len(X_val)}")
print(f"Test rows: {len(X_test)}")

# Cell 4: Expose the majority-class baseline
baseline = DummyClassifier(strategy="most_frequent")
baseline.fit(X_train, y_train)
baseline_predictions = baseline.predict(X_val)

print("Majority-class baseline on validation data")
print(classification_report(y_val, baseline_predictions, zero_division=0))

# Cell 5: Build leakage-safe preprocessing and model pipelines
def make_preprocessor():
    numeric_pipeline = Pipeline(
        steps=[
            ("imputer", SimpleImputer(strategy="median")),
            ("scaler", StandardScaler()),
        ]
    )
    categorical_pipeline = Pipeline(
        steps=[
            ("imputer", SimpleImputer(strategy="most_frequent")),
            ("encoder", OneHotEncoder(handle_unknown="ignore")),
        ]
    )
    return ColumnTransformer(
        transformers=[
            ("numeric", numeric_pipeline, numeric_features),
            ("categorical", categorical_pipeline, categorical_features),
        ]
    )


models = {
    "Logistic Regression": Pipeline(
        steps=[
            ("preprocessor", make_preprocessor()),
            ("classifier", LogisticRegression(solver="lbfgs", max_iter=1000)),
        ]
    ),
    "Random Forest": Pipeline(
        steps=[
            ("preprocessor", make_preprocessor()),
            (
                "classifier",
                RandomForestClassifier(
                    n_estimators=300,
                    min_samples_leaf=5,
                    n_jobs=-1,
                    random_state=42,
                ),
            ),
        ]
    ),
}


def positive_probabilities(model, X_data):
    classifier = model.named_steps["classifier"]
    positive_class_index = list(classifier.classes_).index(1)
    return model.predict_proba(X_data)[:, positive_class_index]


validation_probabilities = {}
validation_rows = []

for model_name, model in models.items():
    model.fit(X_train, y_train)
    probabilities = positive_probabilities(model, X_val)
    validation_probabilities[model_name] = probabilities
    validation_rows.append(
        {
            "model": model_name,
            "validation_roc_auc": roc_auc_score(y_val, probabilities),
        }
    )

model_comparison = pd.DataFrame(validation_rows).sort_values(
    "validation_roc_auc",
    ascending=False,
    ignore_index=True,
)
model_comparison

How to use this reference

Check names and indentation exactly. The final expression must remain model_comparison so the sorted table appears after execution.

Your training workflow can now compare real models without leaking validation information into preprocessing. Next, you will turn the stronger model's scores into a retention-focused decision threshold.

Tune the Retention Threshold

Your fitted Logistic Regression and Random Forest pipelines now have validation ROC AUC scores. The comparison identifies which model ranks churn risk most effectively on validation data.

A probability score becomes an outreach decision only after you choose a cutoff. This step tunes that cutoff on validation data so the held-out test set remains an honest final check.

In this step, get ready to:

Select the fitted model with the highest validation ROC AUC.
Compare candidate thresholds using validation precision, recall, and F1.
Choose the top validation threshold using F1 with recall as the tie-breaker.

Choose the validation winner

Threshold tuning follows model selection because each model produces a different score distribution. The first row of model_comparison identifies the fitted pipeline with the highest validation ROC AUC.

Read the first row of the displayed model_comparison table.
Confirm that validation_roc_auc runs from highest to lowest.

The top row is the only pipeline this step carries into threshold tuning.

Tune and verify the threshold

A decision threshold converts each churn probability into a class prediction. Testing several cutoffs reveals which balance of precision and recall produces the strongest validation F1 score.

Add a new code cell below the cell that displays model_comparison in the JupyterHub notebook from earlier.
Build the validation-only threshold table by pasting this code into the new cell:

# Cell 6: Select the model and tune its threshold on validation data
selected_model_name = model_comparison.loc[0, "model"]
selected_model = models[selected_model_name]
selected_validation_probabilities = validation_probabilities[selected_model_name]

threshold_rows = []
for threshold in [0.20, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50, 0.55, 0.60, 0.65, 0.70]:
    predictions = [
        int(probability >= threshold)
        for probability in selected_validation_probabilities
    ]
    threshold_rows.append(
        {
            "threshold": threshold,
            "precision": precision_score(y_val, predictions, zero_division=0),
            "recall": recall_score(y_val, predictions, zero_division=0),
            "f1": f1_score(y_val, predictions, zero_division=0),
        }
    )

threshold_table = pd.DataFrame(threshold_rows).sort_values(
    ["f1", "recall"],
    ascending=[False, False],
    ignore_index=True,
)
chosen_threshold = float(threshold_table.loc[0, "threshold"])

print(f"Selected model: {selected_model_name}")
print(f"Chosen validation threshold: {chosen_threshold:.2f}")
threshold_table.round(3)

How does validation-only tuning work?

The first row of model_comparison supplies the model name. The matching fitted pipeline comes from models.
The loop tests 11 thresholds from 0.20 through 0.70 against the stored validation probabilities.
Each threshold receives precision, recall, and F1 measurements calculated against y_val.
The sort places the highest F1 result first. Recall resolves any F1 tie.
The code never references X_test, y_test, or id_test. This preserves the test partition for final evaluation.

Before you run the cell, do you expect the threshold with the best F1 score to stay at 0.50?

Run the cell by pressing Shift+Enter.

You'll see the selected model name with the chosen validation threshold above the table. The first row of the table has the highest F1 score with recall used as the tie-breaker.

Threshold table missing?

If the cell cannot find model_comparison or validation_probabilities, rerun the model-training cell from the previous step.

If the best row does not appear first, compare the column names and descending sort settings with the reference code.

Help me diagnose my threshold-tuning cell.

✔️ Awesome, I've got everything!

Great. Your notebook now stores a validation-selected model. It also stores a validation-selected threshold while preserving the test partition.

ⓧ I'd like to double check the full code

# Cell 1: Imports and reference-version audit
from importlib.metadata import version

import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
from sklearn.compose import ColumnTransformer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
    roc_curve,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

reference_versions = {
    "pandas": "3.0.5",
    "scikit-learn": "1.9.0",
    "matplotlib": "3.11.1",
    "seaborn": "0.13.2",
}

version_audit = pd.DataFrame(
    [
        {
            "package": package,
            "installed": version(package),
            "reference": reference,
        }
        for package, reference in reference_versions.items()
    ]
)
version_audit

# Cell 2: Validate and clean the exact dataset
expected_columns = [
    "customerID",
    "gender",
    "SeniorCitizen",
    "Partner",
    "Dependents",
    "tenure",
    "PhoneService",
    "MultipleLines",
    "InternetService",
    "OnlineSecurity",
    "OnlineBackup",
    "DeviceProtection",
    "TechSupport",
    "StreamingTV",
    "StreamingMovies",
    "Contract",
    "PaperlessBilling",
    "PaymentMethod",
    "MonthlyCharges",
    "TotalCharges",
    "Churn",
]

assert df.shape == (7043, 21), f"Expected (7043, 21), found {df.shape}"
assert list(df.columns) == expected_columns, "The loaded columns do not match the expected Kaggle schema."
assert set(df["Churn"]) == {"No", "Yes"}, "Expected Churn labels to be No and Yes."

work_df = df.copy()
churn_counts = work_df["Churn"].value_counts()

plt.figure(figsize=(6, 4))
sns.barplot(x=churn_counts.index, y=churn_counts.values, errorbar=None, color="steelblue")
plt.title("Telco Customer Churn Distribution")
plt.xlabel("Churn")
plt.ylabel("Customers")
plt.tight_layout()
plt.show()

work_df["TotalCharges"] = pd.to_numeric(work_df["TotalCharges"], errors="coerce")
work_df["Churn"] = [1 if value == "Yes" else 0 for value in work_df["Churn"]]

customer_ids = work_df["customerID"]
X = work_df.drop(columns=["customerID", "Churn"])
y = work_df["Churn"]

numeric_features = ["SeniorCitizen", "tenure", "MonthlyCharges", "TotalCharges"]
categorical_features = [column for column in X.columns if column not in numeric_features]

print(f"Rows and columns: {work_df.shape}")
print(f"Missing TotalCharges after conversion: {work_df['TotalCharges'].isna().sum()}")
print(f"Numeric features: {numeric_features}")
print(f"Categorical feature count: {len(categorical_features)}")

# Cell 3: Create train, validation, and test partitions
X_train_full, X_test, y_train_full, y_test, _, id_test = train_test_split(
    X,
    y,
    customer_ids,
    test_size=0.20,
    random_state=42,
    stratify=y,
)

X_train, X_val, y_train, y_val = train_test_split(
    X_train_full,
    y_train_full,
    test_size=0.25,
    random_state=42,
    stratify=y_train_full,
)

print(f"Training rows: {len(X_train)}")
print(f"Validation rows: {len(X_val)}")
print(f"Test rows: {len(X_test)}")

# Cell 4: Expose the majority-class baseline
baseline = DummyClassifier(strategy="most_frequent")
baseline.fit(X_train, y_train)
baseline_predictions = baseline.predict(X_val)

print("Majority-class baseline on validation data")
print(classification_report(y_val, baseline_predictions, zero_division=0))

# Cell 5: Build leakage-safe preprocessing and model pipelines
def make_preprocessor():
    numeric_pipeline = Pipeline(
        steps=[
            ("imputer", SimpleImputer(strategy="median")),
            ("scaler", StandardScaler()),
        ]
    )
    categorical_pipeline = Pipeline(
        steps=[
            ("imputer", SimpleImputer(strategy="most_frequent")),
            ("encoder", OneHotEncoder(handle_unknown="ignore")),
        ]
    )
    return ColumnTransformer(
        transformers=[
            ("numeric", numeric_pipeline, numeric_features),
            ("categorical", categorical_pipeline, categorical_features),
        ]
    )


models = {
    "Logistic Regression": Pipeline(
        steps=[
            ("preprocessor", make_preprocessor()),
            ("classifier", LogisticRegression(solver="lbfgs", max_iter=1000)),
        ]
    ),
    "Random Forest": Pipeline(
        steps=[
            ("preprocessor", make_preprocessor()),
            (
                "classifier",
                RandomForestClassifier(
                    n_estimators=300,
                    min_samples_leaf=5,
                    n_jobs=-1,
                    random_state=42,
                ),
            ),
        ]
    ),
}


def positive_probabilities(model, X_data):
    classifier = model.named_steps["classifier"]
    positive_class_index = list(classifier.classes_).index(1)
    return model.predict_proba(X_data)[:, positive_class_index]


validation_probabilities = {}
validation_rows = []

for model_name, model in models.items():
    model.fit(X_train, y_train)
    probabilities = positive_probabilities(model, X_val)
    validation_probabilities[model_name] = probabilities
    validation_rows.append(
        {
            "model": model_name,
            "validation_roc_auc": roc_auc_score(y_val, probabilities),
        }
    )

model_comparison = pd.DataFrame(validation_rows).sort_values(
    "validation_roc_auc",
    ascending=False,
    ignore_index=True,
)
model_comparison

# Cell 6: Select the model and tune its threshold on validation data
selected_model_name = model_comparison.loc[0, "model"]
selected_model = models[selected_model_name]
selected_validation_probabilities = validation_probabilities[selected_model_name]

threshold_rows = []
for threshold in [0.20, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50, 0.55, 0.60, 0.65, 0.70]:
    predictions = [
        int(probability >= threshold)
        for probability in selected_validation_probabilities
    ]
    threshold_rows.append(
        {
            "threshold": threshold,
            "precision": precision_score(y_val, predictions, zero_division=0),
            "recall": recall_score(y_val, predictions, zero_division=0),
            "f1": f1_score(y_val, predictions, zero_division=0),
        }
    )

threshold_table = pd.DataFrame(threshold_rows).sort_values(
    ["f1", "recall"],
    ascending=[False, False],
    ignore_index=True,
)
chosen_threshold = float(threshold_table.loc[0, "threshold"])

print(f"Selected model: {selected_model_name}")
print(f"Chosen validation threshold: {chosen_threshold:.2f}")
threshold_table.round(3)

Your validation-only decision rule is ready. Next, you'll apply it once to the untouched test data before building the ranked retention list.

Evaluate and Prioritize Customers

Your validation workflow has chosen a model plus a threshold without consulting the held-out test rows. That separation gives you one clean chance to measure performance on unseen customers.

The project now needs an honest final evaluation on the untouched test set. It also needs a customer-level list that makes the model actionable for retention.

In this step, get ready to:

Evaluate the selected model once on held-out customers with metrics plus diagnostic plots.
Interpret the strongest positive logistic-regression churn signals.
Rank test customers by churn probability for focused retention outreach.

Evaluate the held-out test set

The test partition has stayed untouched throughout model selection plus threshold tuning. This one-time evaluation estimates how your selected pipeline handles unseen customers.

Use the add-cell control in the notebook toolbar to insert a new code cell directly below Cell 6.
Calculate the held-out metrics by pasting this code into the new cell:

# Cell 7: Evaluate once on untouched test data
selected_test_probabilities = positive_probabilities(selected_model, X_test)
selected_test_predictions = [
    int(probability >= chosen_threshold)
    for probability in selected_test_probabilities
]
test_roc_auc = roc_auc_score(y_test, selected_test_probabilities)

print(f"Held-out ROC AUC: {test_roc_auc:.3f}")
print(classification_report(y_test, selected_test_predictions, zero_division=0))

What does this code do?

The selected model assigns a churn probability to every customer in the untouched test partition.
The validation-selected threshold converts each probability into a churn prediction.
The classification report measures precision plus recall plus F1 for both classes.
The held-out ROC AUC measures how well the model ranks churners above non-churners across possible thresholds.

Save telco_customer_churn_prediction.ipynb.
Run Cell 7 to evaluate the selected model.

You should see a held-out ROC AUC value followed by a classification report for classes 0 plus 1.

That is the honest evaluation locked in. Your untouched customers have now measured the model exactly once.

Missing the held-out metrics?

Confirm that Cells 1 through 6 ran in order within the current notebook session.
Rerun the earlier cells if the notebook kernel was restarted.
Check that the metric code remains inside Cell 7.

Need another pair of eyes? Help me diagnose why Cell 7 cannot evaluate my selected churn model.

A single score cannot show which prediction errors the model makes. A confusion matrix counts those errors while a ROC curve shows ranking performance across possible cutoffs.

Append the diagnostic plotting code below the classification report code in Cell 7:

matrix = confusion_matrix(y_test, selected_test_predictions, labels=[0, 1])
fpr, tpr, _ = roc_curve(y_test, selected_test_probabilities)

figure, axes = plt.subplots(1, 2, figsize=(12, 4))
sns.heatmap(
    matrix,
    annot=True,
    fmt="d",
    cmap="Blues",
    cbar=False,
    xticklabels=["Predicted No", "Predicted Yes"],
    yticklabels=["Actual No", "Actual Yes"],
    ax=axes[0],
)
axes[0].set_title("Held-out Confusion Matrix")
axes[0].set_xlabel("Prediction")
axes[0].set_ylabel("Actual")

axes[1].plot(fpr, tpr, label=f"{selected_model_name} AUC = {test_roc_auc:.3f}")
axes[1].plot([0, 1], [0, 1], "--", color="gray")
axes[1].set_title("Held-out ROC Curve")
axes[1].set_xlabel("False Positive Rate")
axes[1].set_ylabel("True Positive Rate")
axes[1].legend()

plt.tight_layout()
plt.show()

What do these plots reveal?

The heatmap separates correct predictions from false positives plus false negatives.
The curve plots the true positive rate against the false positive rate across possible thresholds.
The diagonal line represents ranking with no useful separation between the two classes.
The selected model name plus held-out AUC appear in the curve legend.

Save telco_customer_churn_prediction.ipynb.
Run the completed Cell 7 to display both diagnostic plots.

You should see a blue held-out confusion matrix beside a ROC curve. The curve legend should show the selected model name plus its held-out AUC.

Plots not appearing?

Confirm that the plotting snippet sits below the metric code inside Cell 7.
Run the entire cell so the matrix plus ROC values exist before plotting.
Rerun the import cell if the notebook session no longer recognizes the plotting libraries.

Still blocked? Help me find why the held-out confusion matrix or ROC curve is missing.

Interpret the logistic-regression baseline

Logistic-regression coefficients provide a consistent baseline view of which transformed features push predictions toward churn. Treat these coefficients as predictive associations only.

The fitted logistic pipeline already contains its preprocessing feature names plus classifier coefficients. Matching those arrays produces a readable table of positive churn signals.

Use the add-cell control in the notebook toolbar to insert a new code cell directly below Cell 7.
Extract the strongest positive logistic-regression signals by pasting this code into the new cell:

# Cell 8: Interpret the logistic-regression baseline
logistic_pipeline = models["Logistic Regression"]
feature_names = logistic_pipeline.named_steps["preprocessor"].get_feature_names_out()
coefficients = logistic_pipeline.named_steps["classifier"].coef_[0]

coefficient_table = pd.DataFrame(
    {
        "feature": feature_names,
        "coefficient": coefficients,
    }
)
top_positive_drivers = coefficient_table.sort_values(
    "coefficient",
    ascending=False,
).head(10)

plt.figure(figsize=(9, 5))
sns.barplot(
    data=top_positive_drivers,
    x="coefficient",
    y="feature",
    errorbar=None,
    color="tomato",
)
plt.title("Strongest Positive Logistic-Regression Churn Signals")
plt.xlabel("Coefficient")
plt.ylabel("Transformed Feature")
plt.tight_layout()
plt.show()

What does this code do?

The fitted preprocessing branch supplies the transformed feature names used by logistic regression.
The classifier supplies one coefficient for each transformed feature.
Sorting in descending order surfaces the ten strongest positive associations with predicted churn.
The horizontal chart keeps long one-hot-encoded feature names readable.

Save telco_customer_churn_prediction.ipynb.
Run Cell 8 to display the coefficient chart.

You should see a tomato-colored horizontal chart containing ten transformed features. Features with larger positive coefficients have stronger positive associations with predicted churn.

Coefficient chart missing?

Confirm that the logistic-regression pipeline finished fitting during the model-comparison step.
Check that Cell 8 uses the existing Logistic Regression model key exactly.
Rerun the pipeline-training cell before retrying Cell 8 if the kernel was restarted.

Need help tracing the fitted pipeline? Help me diagnose why Cell 8 cannot extract or plot logistic-regression coefficients.

Build the retention priority list

A retention team needs customer-level priorities instead of an abstract model score. The test customer IDs remain aligned with their outcomes plus probabilities from the original split.

Sorting those rows by churn probability puts the highest-risk customers first. A numeric priority rank turns that ordering into an outreach queue.

Use the add-cell control in the notebook toolbar to insert a new code cell directly below Cell 8.
Build the ranked retention list by pasting this code into the new cell:

# Cell 9: Build the focused retention priority list
retention_list = pd.DataFrame(
    {
        "customerID": list(id_test),
        "actual_churn": list(y_test),
        "churn_probability": selected_test_probabilities,
        "predicted_churn": selected_test_predictions,
    }
).sort_values(
    "churn_probability",
    ascending=False,
    ignore_index=True,
)
retention_list["priority_rank"] = range(1, len(retention_list) + 1)
retention_list = retention_list[
    [
        "priority_rank",
        "customerID",
        "churn_probability",
        "predicted_churn",
        "actual_churn",
    ]
]
retention_list.head(20)

How does the ranking work?

Each row combines a held-out customer ID with its true outcome plus model outputs.
Descending probability sorting places the customers with the highest predicted churn risk first.
The priority rank starts at 1 for the highest-probability customer.
The final line displays the first twenty retention candidates.

Save telco_customer_churn_prediction.ipynb.

Before you run this cell, which test customers do you expect to receive the smallest priority numbers?

Run Cell 9 to generate the retention priority list.

You should see twenty rows sorted from the highest churn probability downward. The first row should have a priority rank of 1.

That completes the core workflow. Your notebook now measures model performance plus explains the logistic baseline plus identifies the customers a retention team should review first.

Ranking looks misaligned?

Confirm that Cell 9 uses id_test plus y_test from the same split.
Check that the probability plus prediction arrays both come from X_test.
Rerun Cell 7 before Cell 9 if the test probabilities are unavailable.

Want help checking row alignment? Help me verify that my retention list matches each customer with the correct test outcome and churn score.

The tabs below provide a final comparison of Cells 1 through 9.

✔️ Awesome, I've got everything!

Your notebook now contains the complete validation plus modeling plus prioritization workflow.

Save telco_customer_churn_prediction.ipynb before moving to the checkpoint.

ⓧ I'd like to double check the full code

# Cell 1: Imports and reference-version audit
from importlib.metadata import version

import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
from sklearn.compose import ColumnTransformer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
    roc_curve,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

reference_versions = {
    "pandas": "3.0.5",
    "scikit-learn": "1.9.0",
    "matplotlib": "3.11.1",
    "seaborn": "0.13.2",
}

version_audit = pd.DataFrame(
    [
        {
            "package": package,
            "installed": version(package),
            "reference": reference,
        }
        for package, reference in reference_versions.items()
    ]
)
version_audit

# Cell 2: Validate and clean the exact dataset
expected_columns = [
    "customerID",
    "gender",
    "SeniorCitizen",
    "Partner",
    "Dependents",
    "tenure",
    "PhoneService",
    "MultipleLines",
    "InternetService",
    "OnlineSecurity",
    "OnlineBackup",
    "DeviceProtection",
    "TechSupport",
    "StreamingTV",
    "StreamingMovies",
    "Contract",
    "PaperlessBilling",
    "PaymentMethod",
    "MonthlyCharges",
    "TotalCharges",
    "Churn",
]

assert df.shape == (7043, 21), f"Expected (7043, 21), found {df.shape}"
assert list(df.columns) == expected_columns, "The loaded columns do not match the expected Kaggle schema."
assert set(df["Churn"]) == {"No", "Yes"}, "Expected Churn labels to be No and Yes."

work_df = df.copy()
churn_counts = work_df["Churn"].value_counts()

plt.figure(figsize=(6, 4))
sns.barplot(x=churn_counts.index, y=churn_counts.values, errorbar=None, color="steelblue")
plt.title("Telco Customer Churn Distribution")
plt.xlabel("Churn")
plt.ylabel("Customers")
plt.tight_layout()
plt.show()

work_df["TotalCharges"] = pd.to_numeric(work_df["TotalCharges"], errors="coerce")
work_df["Churn"] = [1 if value == "Yes" else 0 for value in work_df["Churn"]]

customer_ids = work_df["customerID"]
X = work_df.drop(columns=["customerID", "Churn"])
y = work_df["Churn"]

numeric_features = ["SeniorCitizen", "tenure", "MonthlyCharges", "TotalCharges"]
categorical_features = [column for column in X.columns if column not in numeric_features]

print(f"Rows and columns: {work_df.shape}")
print(f"Missing TotalCharges after conversion: {work_df['TotalCharges'].isna().sum()}")
print(f"Numeric features: {numeric_features}")
print(f"Categorical feature count: {len(categorical_features)}")

# Cell 3: Create train, validation, and test partitions
X_train_full, X_test, y_train_full, y_test, _, id_test = train_test_split(
    X,
    y,
    customer_ids,
    test_size=0.20,
    random_state=42,
    stratify=y,
)

X_train, X_val, y_train, y_val = train_test_split(
    X_train_full,
    y_train_full,
    test_size=0.25,
    random_state=42,
    stratify=y_train_full,
)

print(f"Training rows: {len(X_train)}")
print(f"Validation rows: {len(X_val)}")
print(f"Test rows: {len(X_test)}")

# Cell 4: Expose the majority-class baseline
baseline = DummyClassifier(strategy="most_frequent")
baseline.fit(X_train, y_train)
baseline_predictions = baseline.predict(X_val)

print("Majority-class baseline on validation data")
print(classification_report(y_val, baseline_predictions, zero_division=0))

# Cell 5: Build leakage-safe preprocessing and model pipelines
def make_preprocessor():
    numeric_pipeline = Pipeline(
        steps=[
            ("imputer", SimpleImputer(strategy="median")),
            ("scaler", StandardScaler()),
        ]
    )
    categorical_pipeline = Pipeline(
        steps=[
            ("imputer", SimpleImputer(strategy="most_frequent")),
            ("encoder", OneHotEncoder(handle_unknown="ignore")),
        ]
    )
    return ColumnTransformer(
        transformers=[
            ("numeric", numeric_pipeline, numeric_features),
            ("categorical", categorical_pipeline, categorical_features),
        ]
    )


models = {
    "Logistic Regression": Pipeline(
        steps=[
            ("preprocessor", make_preprocessor()),
            ("classifier", LogisticRegression(solver="lbfgs", max_iter=1000)),
        ]
    ),
    "Random Forest": Pipeline(
        steps=[
            ("preprocessor", make_preprocessor()),
            (
                "classifier",
                RandomForestClassifier(
                    n_estimators=300,
                    min_samples_leaf=5,
                    n_jobs=-1,
                    random_state=42,
                ),
            ),
        ]
    ),
}


def positive_probabilities(model, X_data):
    classifier = model.named_steps["classifier"]
    positive_class_index = list(classifier.classes_).index(1)
    return model.predict_proba(X_data)[:, positive_class_index]


validation_probabilities = {}
validation_rows = []

for model_name, model in models.items():
    model.fit(X_train, y_train)
    probabilities = positive_probabilities(model, X_val)
    validation_probabilities[model_name] = probabilities
    validation_rows.append(
        {
            "model": model_name,
            "validation_roc_auc": roc_auc_score(y_val, probabilities),
        }
    )

model_comparison = pd.DataFrame(validation_rows).sort_values(
    "validation_roc_auc",
    ascending=False,
    ignore_index=True,
)
model_comparison

# Cell 6: Select the model and tune its threshold on validation data
selected_model_name = model_comparison.loc[0, "model"]
selected_model = models[selected_model_name]
selected_validation_probabilities = validation_probabilities[selected_model_name]

threshold_rows = []
for threshold in [0.20, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50, 0.55, 0.60, 0.65, 0.70]:
    predictions = [
        int(probability >= threshold)
        for probability in selected_validation_probabilities
    ]
    threshold_rows.append(
        {
            "threshold": threshold,
            "precision": precision_score(y_val, predictions, zero_division=0),
            "recall": recall_score(y_val, predictions, zero_division=0),
            "f1": f1_score(y_val, predictions, zero_division=0),
        }
    )

threshold_table = pd.DataFrame(threshold_rows).sort_values(
    ["f1", "recall"],
    ascending=[False, False],
    ignore_index=True,
)
chosen_threshold = float(threshold_table.loc[0, "threshold"])

print(f"Selected model: {selected_model_name}")
print(f"Chosen validation threshold: {chosen_threshold:.2f}")
threshold_table.round(3)

# Cell 7: Evaluate once on untouched test data
selected_test_probabilities = positive_probabilities(selected_model, X_test)
selected_test_predictions = [
    int(probability >= chosen_threshold)
    for probability in selected_test_probabilities
]
test_roc_auc = roc_auc_score(y_test, selected_test_probabilities)

print(f"Held-out ROC AUC: {test_roc_auc:.3f}")
print(classification_report(y_test, selected_test_predictions, zero_division=0))

matrix = confusion_matrix(y_test, selected_test_predictions, labels=[0, 1])
fpr, tpr, _ = roc_curve(y_test, selected_test_probabilities)

figure, axes = plt.subplots(1, 2, figsize=(12, 4))
sns.heatmap(
    matrix,
    annot=True,
    fmt="d",
    cmap="Blues",
    cbar=False,
    xticklabels=["Predicted No", "Predicted Yes"],
    yticklabels=["Actual No", "Actual Yes"],
    ax=axes[0],
)
axes[0].set_title("Held-out Confusion Matrix")
axes[0].set_xlabel("Prediction")
axes[0].set_ylabel("Actual")

axes[1].plot(fpr, tpr, label=f"{selected_model_name} AUC = {test_roc_auc:.3f}")
axes[1].plot([0, 1], [0, 1], "--", color="gray")
axes[1].set_title("Held-out ROC Curve")
axes[1].set_xlabel("False Positive Rate")
axes[1].set_ylabel("True Positive Rate")
axes[1].legend()

plt.tight_layout()
plt.show()

# Cell 8: Interpret the logistic-regression baseline
logistic_pipeline = models["Logistic Regression"]
feature_names = logistic_pipeline.named_steps["preprocessor"].get_feature_names_out()
coefficients = logistic_pipeline.named_steps["classifier"].coef_[0]

coefficient_table = pd.DataFrame(
    {
        "feature": feature_names,
        "coefficient": coefficients,
    }
)
top_positive_drivers = coefficient_table.sort_values(
    "coefficient",
    ascending=False,
).head(10)

plt.figure(figsize=(9, 5))
sns.barplot(
    data=top_positive_drivers,
    x="coefficient",
    y="feature",
    errorbar=None,
    color="tomato",
)
plt.title("Strongest Positive Logistic-Regression Churn Signals")
plt.xlabel("Coefficient")
plt.ylabel("Transformed Feature")
plt.tight_layout()
plt.show()

# Cell 9: Build the focused retention priority list
retention_list = pd.DataFrame(
    {
        "customerID": list(id_test),
        "actual_churn": list(y_test),
        "churn_probability": selected_test_probabilities,
        "predicted_churn": selected_test_predictions,
    }
).sort_values(
    "churn_probability",
    ascending=False,
    ignore_index=True,
)
retention_list["priority_rank"] = range(1, len(retention_list) + 1)
retention_list = retention_list[
    [
        "priority_rank",
        "customerID",
        "churn_probability",
        "predicted_churn",
        "actual_churn",
    ]
]
retention_list.head(20)

How to use this reference

Compare the reference cell by cell against your notebook. Keep the existing CSV-loading cell above Cell 1 because it creates df for this workflow.

Secret mission

Measure Retention Campaign Capacity

Your ranked retention list orders customers by churn probability. Limit outreach to the top 100 customers to measure how much held-out churn risk a fixed-capacity campaign can capture.

Clean Up Your Resources

Your completed churn analysis stays inside telco_customer_churn_prediction.ipynb on JupyterHub. Leaving it there adds no project-specific ongoing cost.

Resources you used:

The completed telco_customer_churn_prediction.ipynb file.
The saved tables and charts stored inside the notebook.
The active Python kernel that holds work_df, the fitted models, retention_list, and campaign_summary in memory.

Keep everything running

No action needed. Choose this if you want to revisit the model or test another retention capacity.

Keep telco_customer_churn_prediction.ipynb in your JupyterHub workspace.
Preserve the saved charts and evaluation tables for future comparison.
Return to contact_capacity when you want to test a different outreach limit.

Pause - I'll come back to this later

A paused project keeps the notebook file while releasing the active kernel. Your saved cells and outputs remain available for the next session.

Save telco_customer_churn_prediction.ipynb through the notebook editor's save control.
End the active kernel through the session controls in JupyterHub.
Leave telco_customer_churn_prediction.ipynb in the JupyterHub file browser for your next session.

Delete - I don't want to use this again

Deleting the notebook removes the project code plus its saved outputs. This choice is permanent, so use it only when you no longer need the analysis.

End the active kernel through the session controls in JupyterHub.
Select telco_customer_churn_prediction.ipynb in the JupyterHub file browser.
Use the file browser's delete action to remove the selected notebook.
Confirm that telco_customer_churn_prediction.ipynb no longer appears in the file browser.

Nice Work!

You did it! Your notebook now turns the Telco Customer Churn data into a ranked retention plan backed by held-out evaluation.

What you learned:

Validated the exact 21-column schema in the original df before creating a clean modeling copy.
Exposed the majority-class accuracy trap by showing that apparent accuracy can hide zero churn recall.
Built leakage-safe model pipelines that convert held-out churn scores into a ranked customer list using a validation-selected threshold.
Secret Mission: Measured campaign capacity by counting actual churners captured within the top 100 rows of retention_list.

Ready to quiz yourself?