Benchmark Tabular ML Models

Compare TabPFN, XGBoost, and Random Forest on automotive defect data.

Introduction

30 Second Summary

Every car rolling off an assembly line contains thousands of parts, each assembled by machines that generate sensor data. Catching a defective part before it reaches the customer depends on analyzing that data with the right model.

In this project, you will build a benchmarking pipeline that compares a pretrained tabular foundation model against traditional machine learning approaches on automotive manufacturing defect prediction data.

What You'll Build

A complete ML benchmarking system that processes production-line sensor data with PySpark, trains three different modeling approaches, and produces a comprehensive evaluation report with recommendations.

By the end of this project, you'll have:

  • A PySpark feature engineering pipeline that ingests raw sensor data, engineers interaction features, and outputs a clean train/test split.
  • A benchmarking framework comparing TabPFN (zero-shot, no tuning), XGBoost (tuned gradient boosting), and a Random Forest baseline with AUROC curves and calibration plots.
  • A performance-vs-data-size analysis showing when pretrained tabular models outperform traditional approaches.
  • Secret Mission: Build a data drift detector that uses the Kolmogorov-Smirnov test to flag when incoming production data has shifted from the training distribution.

Are there any prerequisites?

This project assumes intermediate Python experience and a basic understanding of classification concepts (what a model predicts, what training and test sets are).

Familiarity with the scikit-learn API pattern (fit, predict, predict_proba) will help you move faster through the modeling steps.

Before We Start

Before you can benchmark machine learning models, you need a working Python environment with all the right libraries installed.

In this step, you will install Python and VS Code, set up a project directory with a virtual environment, and create the Jupyter notebook where all your benchmarking code will live.

In this step, get ready to:
  • Install Python 3.11+ and VS Code.
  • Install all project dependencies (TabPFN, XGBoost, PySpark, scikit-learn, and more).
  • Create a Jupyter notebook and verify all imports work.
Install Python and VS Code

This project requires Python 3.11 or higher because one of the key dependencies (XGBoost) requires Python 3.10+, and the project pins to 3.11 for full compatibility across all libraries.

  • Check your Python version by running this command in your terminal:
python --version

✔️ I see version 3.11 or higher

Your Python version is compatible. Move on to the VS Code check below.

ⓧ I see an older version

Your system has Python installed, but the version is too old for this project. On macOS, the system Python is often 3.9 or lower.

  • Download the latest Python 3.11+ installer from python.org/downloads.
  • Run the installer and follow the prompts.
  • On macOS, the new install may not override the system Python. After installing, try running:
python3.11 --version

If that works, use python3.11 in place of python for the rest of this project.

ⓧ Command not found

Python is not installed on your system yet.

  • Go to python.org/downloads and download the latest Python 3.11+ installer for your operating system.
  • Run the installer and follow the prompts. On Windows, check the box that says Add Python to PATH before clicking Install.
  • Close and reopen your terminal, then run the version check again.

Now open VS Code.

  • If you do not have VS Code installed, download it from code.visualstudio.com.
  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.
Install project dependencies

You need a dedicated project folder and a virtual environment to keep this project's libraries isolated from other Python work on your machine.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir benchmark-tabular-models
cd benchmark-tabular-models
  • Create a virtual environment by running:
python -m venv venv
  • Activate the virtual environment by running this command:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the start of your terminal prompt. This confirms the virtual environment is active.

  • Install all the project dependencies by running:
pip install tabpfn-client xgboost scikit-learn pyspark pandas numpy matplotlib seaborn jupyter

What are all these packages?

  • tabpfn-client gives you access to the TabPFN foundation model via a cloud API (no GPU needed).
  • xgboost is a gradient boosted decision tree library, widely used in production ML.
  • scikit-learn provides the Random Forest baseline, evaluation metrics, and preprocessing tools.
  • pyspark lets you build scalable feature engineering pipelines using Apache Spark.
  • jupyter runs your interactive notebook environment.

The install may take a few minutes as it downloads all packages and their sub-dependencies.

Seeing an error during install?

  • If you see a permissions error, make sure your virtual environment is activated (look for (venv) in your prompt).
  • If pyspark fails, it may need Java installed. PySpark requires Java 17 or later. Download it from adoptium.net and set the JAVA_HOME environment variable.
  • If pip is not recognized, try python -m pip install instead.

Help me troubleshoot my dependency installation errors.

Create your Jupyter notebook and verify imports

With all dependencies installed, you can now create the notebook where you will write all your benchmarking code.

  • Launch Jupyter Notebook by running:
jupyter notebook

Your default web browser opens to the Jupyter dashboard, showing the files in your project folder.

  • Click the New button in the top right corner of the dashboard.
  • Select Python 3 from the dropdown menu.
  • Click the notebook title at the top (it will say Untitled) and rename it to benchmark_tabular_models.

Now verify that all the libraries you installed are importable.

  • Paste the following code into the first cell of your notebook, then press Shift+Enter to run it:
# Cell 1: Imports and Setup
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    roc_auc_score, roc_curve, precision_recall_curve,
    f1_score, log_loss, classification_report
)
from sklearn.calibration import calibration_curve
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from xgboost import XGBClassifier
from tabpfn_client import TabPFNClassifier
from pyspark.sql import SparkSession
from pyspark.ml import Pipeline as SparkPipeline
from pyspark.ml.feature import (
    VectorAssembler, StandardScaler as SparkStandardScaler,
    Imputer as SparkImputer
)
import time

print("All imports successful!")

What does this code do?

  • The sklearn imports bring in model classes, evaluation metrics, and preprocessing tools from scikit-learn.
  • XGBClassifier is the XGBoost gradient boosting classifier you will tune and benchmark.
  • TabPFNClassifier connects to the TabPFN cloud API for zero-shot tabular predictions.
  • The PySpark imports set up the feature engineering pipeline components you will use in the next step.

You should see All imports successful! printed below the cell with no error messages.

Seeing a ModuleNotFoundError?

  • Make sure Jupyter is using your virtual environment's Python. Close the notebook, run which python (macOS) or where python (Windows) in your terminal to confirm it points inside your venv folder.
  • If the path does not point to your venv folder, deactivate, reactivate the virtual environment, and relaunch jupyter notebook.
  • If pyspark specifically fails to import, you likely need Java installed. Check that running java -version in your terminal shows Java 17 or later.

Help me fix my import errors in the Jupyter notebook.

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook (Cmd+S on macOS, Ctrl+S on Windows).

ⓧ I'd like to double check the full code

Here is the complete code for Cell 1 of your notebook:

# Cell 1: Imports and Setup
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    roc_auc_score, roc_curve, precision_recall_curve,
    f1_score, log_loss, classification_report
)
from sklearn.calibration import calibration_curve
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from xgboost import XGBClassifier
from tabpfn_client import TabPFNClassifier
from pyspark.sql import SparkSession
from pyspark.ml import Pipeline as SparkPipeline
from pyspark.ml.feature import (
    VectorAssembler, StandardScaler as SparkStandardScaler,
    Imputer as SparkImputer
)
import time

print("All imports successful!")

Your environment is ready to go. Next up, you will load and explore automotive sensor data using PySpark.

Load and Explore Automotive Sensor Data with PySpark

With your environment ready, it's time to get some data flowing through the pipeline. In manufacturing, production lines generate thousands of sensor readings per minute.

In this step, you'll create a synthetic automotive sensor dataset that mimics real assembly station measurements (torques, angles, vibrations, and currents). Then you'll load it into PySpark for exploration at scale.

In this step, get ready to:
  • Generate a synthetic automotive sensor dataset with realistic class imbalance and missing values.
  • Initialize a local PySpark session and load the dataset into a Spark DataFrame.
  • Explore class distribution, missing values, and summary statistics.
Generate the synthetic automotive sensor dataset

Real automotive datasets from companies like FORVIA often contain sensitive production data. For this project, you'll generate synthetic data with realistic characteristics: 20 sensor features, a 5% defect rate (typical in manufacturing), and 3% missing values (simulating sensor dropouts).

  • In your benchmark_tabular_models.ipynb notebook, create a new cell below Cell 1 and add this code:
# Cell 2: Generate Synthetic Automotive Sensor Data
# Simulating production-line sensor measurements (torques, angles, vibrations)
np.random.seed(42)

n_samples = 5000
n_features = 20

# Create imbalanced classification (5% defect rate, typical in manufacturing)
X, y = make_classification(
    n_samples=n_samples,
    n_features=n_features,
    n_informative=12,
    n_redundant=4,
    n_clusters_per_class=2,
    weights=[0.95, 0.05],
    flip_y=0.02,
    random_state=42
)

# Name features like production-line sensors
sensor_names = [
    "OP070_torque_value", "OP070_angle_value", "OP080_torque_value",
    "OP080_angle_value", "OP090_current_value", "OP090_voltage_value",
    "OP100_pressure_value", "OP100_temperature_value", "OP110_vibration_x",
    "OP110_vibration_y", "OP110_vibration_z", "OP120_rodage_current",
    "OP120_rodage_voltage", "OP120_speed_value", "OP070_force_value",
    "OP080_force_value", "OP090_resistance_value", "OP100_flow_rate",
    "OP110_frequency_value", "OP120_duration_value"
]

df_sensors = pd.DataFrame(X, columns=sensor_names)
df_sensors["defect"] = y

# Add some missing values (realistic for sensor data)
mask = np.random.random(df_sensors[sensor_names].shape) < 0.03
df_sensors[sensor_names] = df_sensors[sensor_names].where(~mask)

print(f"Dataset shape: {df_sensors.shape}")
print(f"Class distribution:\n{df_sensors['defect'].value_counts()}")
print(f"Defect rate: {df_sensors['defect'].mean():.2%}")
print(f"Missing values: {df_sensors.isnull().sum().sum()}")

What does this code do?

This cell builds a synthetic dataset that mimics real automotive production-line data.

  • make_classification generates 5000 samples with 20 features. The weights=[0.95, 0.05] parameter creates a 95/5 class split, which is realistic for defect detection where most parts pass inspection.
  • The sensor_names list names each feature after real assembly operations (OP070, OP080, etc.) with measurement types like torque, angle, and vibration.
  • The mask randomly sets ~3% of sensor values to NaN, simulating real sensor dropouts and communication failures on the production line.
  • Run Cell 2 by pressing Shift+Enter.
  • You should see output showing a dataset shape of (5000, 21), a defect rate around 5%, and a count of missing values.

Seeing a NameError?

If you see an error like NameError: name 'np' is not defined, make sure you ran Cell 1 (your imports cell) first. Jupyter cells must be executed in order for variables to be available.

Help me fix cell execution order issues in Jupyter

Why synthetic data instead of a real dataset?

Real automotive datasets (from Valeo, Bosch, or FORVIA) are proprietary or require competition registration. Synthetic data with the same statistical properties (class imbalance, missing values, correlated sensor features) teaches the same skills without access barriers.

Initialize PySpark and explore the data

Now you'll load this pandas DataFrame into a Spark session to practice the same workflow used at scale in production environments. In a real FORVIA pipeline, your data would arrive as distributed Spark DataFrames from a data lake.

  • Create a new cell below Cell 2 and add this code to initialize a Spark session and explore the data:
# Cell 3: PySpark Feature Engineering Pipeline
spark = SparkSession.builder \
    .master("local[*]") \
    .appName("AutomotiveDefectPrediction") \
    .getOrCreate()

# Convert pandas DataFrame to Spark DataFrame
spark_df = spark.createDataFrame(df_sensors)
spark_df.printSchema()
spark_df.describe().show()

# Check class distribution in Spark
spark_df.groupBy("defect").count().show()

What does this code do?

  • SparkSession.builder.master("local[*]") creates a local Spark session that uses all available CPU cores for parallel processing.
  • createDataFrame(df_sensors) converts your pandas DataFrame into a distributed Spark DataFrame.
  • printSchema() shows the column types Spark inferred from your data.
  • describe().show() computes summary statistics (count, mean, stddev, min, max) for every sensor column.
  • groupBy("defect").count().show() displays the class distribution, confirming the 95/5 imbalance in your dataset.
  • Run Cell 3 by pressing Shift+Enter.
  • You should see the schema listing all 21 columns (20 sensors as double type plus defect as long), followed by a statistics table and a two-row count showing the class split.

Spark taking a long time to start?

The first time PySpark initializes on your machine, it may take 10-20 seconds to start the JVM. This is normal. If you see a Java-related error, make sure you have Java 8, 11, or 17 installed on your system.

Help me install the correct Java version for PySpark

Prepare for downstream ML modeling

You now have the same data accessible in two forms: the original df_sensors pandas DataFrame and the spark_df Spark DataFrame.

For the ML modeling steps ahead, you'll work with the pandas version directly. With 5000 samples and 20 features, this dataset fits comfortably in memory. This is exactly the small-data regime where TabPFN excels.

Why use PySpark if the data fits in pandas?

In production at automotive companies, sensor data arrives as millions of rows across distributed storage. PySpark handles that scale.

Here, you're learning the PySpark pipeline pattern (which you'll expand in Step 3) alongside local modeling. The same pipeline code works whether you have 5,000 rows or 5 million.

✔️ Awesome, I've got everything!

Great. Make sure you've saved your notebook before moving on.

ⓧ I'd like to double check the full code

Here is the complete code for your notebook so far, including all three cells:

# Cell 1: Imports and Setup
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    roc_auc_score, roc_curve, precision_recall_curve,
    f1_score, log_loss, classification_report
)
from sklearn.calibration import calibration_curve
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from xgboost import XGBClassifier
from tabpfn_client import TabPFNClassifier
from pyspark.sql import SparkSession
from pyspark.ml import Pipeline as SparkPipeline
from pyspark.ml.feature import (
    VectorAssembler, StandardScaler as SparkStandardScaler,
    Imputer as SparkImputer
)
import time

print("All imports successful!")

# Cell 2: Generate Synthetic Automotive Sensor Data
np.random.seed(42)

n_samples = 5000
n_features = 20

X, y = make_classification(
    n_samples=n_samples,
    n_features=n_features,
    n_informative=12,
    n_redundant=4,
    n_clusters_per_class=2,
    weights=[0.95, 0.05],
    flip_y=0.02,
    random_state=42
)

sensor_names = [
    "OP070_torque_value", "OP070_angle_value", "OP080_torque_value",
    "OP080_angle_value", "OP090_current_value", "OP090_voltage_value",
    "OP100_pressure_value", "OP100_temperature_value", "OP110_vibration_x",
    "OP110_vibration_y", "OP110_vibration_z", "OP120_rodage_current",
    "OP120_rodage_voltage", "OP120_speed_value", "OP070_force_value",
    "OP080_force_value", "OP090_resistance_value", "OP100_flow_rate",
    "OP110_frequency_value", "OP120_duration_value"
]

df_sensors = pd.DataFrame(X, columns=sensor_names)
df_sensors["defect"] = y

mask = np.random.random(df_sensors[sensor_names].shape) < 0.03
df_sensors[sensor_names] = df_sensors[sensor_names].where(~mask)

print(f"Dataset shape: {df_sensors.shape}")
print(f"Class distribution:\n{df_sensors['defect'].value_counts()}")
print(f"Defect rate: {df_sensors['defect'].mean():.2%}")
print(f"Missing values: {df_sensors.isnull().sum().sum()}")

# Cell 3: PySpark Feature Engineering Pipeline
spark = SparkSession.builder \
    .master("local[*]") \
    .appName("AutomotiveDefectPrediction") \
    .getOrCreate()

spark_df = spark.createDataFrame(df_sensors)
spark_df.printSchema()
spark_df.describe().show()

spark_df.groupBy("defect").count().show()

Your dataset is loaded and explored. Next up, you'll build a full PySpark ML Pipeline with feature engineering to transform these raw sensor readings into ML-ready features.

Build the PySpark Feature Engineering Pipeline

In the previous step, you loaded the automotive sensor data into PySpark and confirmed the class imbalance. Now it is time to build a preprocessing pipeline that transforms raw sensor readings into clean, scaled features ready for ML models.

Raw sensor data from production lines contains missing values and features measured on different scales. A pipeline chains these preprocessing steps together so you can apply the same transformations consistently to new data.

In this step, get ready to:
  • Build a PySpark ML Pipeline with imputation, vector assembly, and scaling.
  • Engineer interaction features that combine key sensor measurements.
  • Perform a time-aware train/test split for realistic evaluation.
Build a PySpark ML Pipeline with imputation and scaling

PySpark ML Pipelines chain multiple preprocessing stages into a single reusable object. Your pipeline needs three stages: impute missing sensor values, assemble features into a vector, and scale to zero mean and unit variance.

  • Create a new cell in your notebook (Cell 4) and add the following code to build and fit the PySpark pipeline:
# Cell 4: Build PySpark ML Pipeline
# Impute missing values
imputer = SparkImputer(
    inputCols=sensor_names,
    outputCols=[f"{col}_imputed" for col in sensor_names],
    strategy="median"
)

imputed_cols = [f"{col}_imputed" for col in sensor_names]

# Assemble features into a single vector
assembler = VectorAssembler(
    inputCols=imputed_cols,
    outputCol="features_raw"
)

# Scale features
scaler = SparkStandardScaler(
    inputCol="features_raw",
    outputCol="features_scaled",
    withStd=True,
    withMean=True
)

# Build and fit the pipeline
spark_pipeline = SparkPipeline(stages=[imputer, assembler, scaler])
pipeline_model = spark_pipeline.fit(spark_df)
transformed_df = pipeline_model.transform(spark_df)

print("PySpark pipeline fitted and transformed successfully!")
transformed_df.select("features_scaled", "defect").show(5, truncate=True)

What does this pipeline do?

The pipeline chains three stages that execute in order:

  • `SparkImputer` replaces missing sensor values with the median of each column. The median is robust to outliers, which is common in noisy sensor data.
  • `VectorAssembler` combines all 20 imputed columns into a single vector column called `features_raw`. PySpark ML algorithms require features in this vector format.
  • `SparkStandardScaler` normalizes the vector so each feature has zero mean and unit variance. This ensures no single sensor dominates due to its measurement scale.

The imputer must run before the assembler because `VectorAssembler` will throw an error if it encounters null values in input columns.

  • Run the cell by pressing **Shift+Enter**.
  • You should see the message `PySpark pipeline fitted and transformed successfully!` followed by a table showing the `features_scaled` vector column and the `defect` label for 5 rows.

Seeing a Java or Py4J error?

PySpark requires Java to be installed. If you see a Java-related error, make sure Java 8, 11, or 17 is installed and the `JAVA_HOME` environment variable is set.

If you see an error mentioning null values in VectorAssembler, double-check that `SparkImputer` appears before `VectorAssembler` in the `stages` list.

Help me debug this PySpark pipeline error

Engineer interaction features in pandas

The PySpark pipeline demonstrated the scalable approach. For the ML modeling, you will switch to pandas because the dataset (5,000 samples) fits comfortably in memory, which is the regime where TabPFN excels.

  • Create a new cell in your notebook (Cell 5) and add the following code to engineer interaction features:
# Cell 5: Feature Engineering - Interaction Terms
df_processed = df_sensors.copy()

# Impute missing values
imputer_sk = SimpleImputer(strategy="median")
df_processed[sensor_names] = imputer_sk.fit_transform(df_processed[sensor_names])

# Engineer interaction features
df_processed["torque_angle_interaction"] = (
    df_processed["OP070_torque_value"] * df_processed["OP070_angle_value"]
)
df_processed["current_voltage_product"] = (
    df_processed["OP090_current_value"] * df_processed["OP090_voltage_value"]
)
df_processed["vibration_magnitude"] = np.sqrt(
    df_processed["OP110_vibration_x"]**2 +
    df_processed["OP110_vibration_y"]**2 +
    df_processed["OP110_vibration_z"]**2
)

Why interaction features?

Individual sensor readings tell you what happened at one measurement point. Interaction features capture relationships between sensors that reveal underlying physical phenomena.

  • `torque_angle_interaction` approximates the work done during a tightening operation. A high torque at a low angle could signal a cross-threaded bolt.
  • `current_voltage_product` represents power consumption, which spikes during abnormal motor loads.
  • `vibration_magnitude` collapses three vibration axes into one scalar, capturing total vibration energy regardless of direction.

Do not run this cell yet. You will add more code to it in the next substep.

Perform a time-aware train/test split

In manufacturing, models must predict defects on future parts using data from past parts. A random shuffle would let the model peek at future patterns during training, so you will use a time-aware split that preserves chronological order.

  • Still in Cell 5, add the following code directly below the interaction features you just wrote:
feature_cols = sensor_names + [
    "torque_angle_interaction",
    "current_voltage_product",
    "vibration_magnitude"
]

X_all = df_processed[feature_cols].values
y_all = df_processed["defect"].values

# Time-aware split (simulating production order)
X_train, X_test, y_train, y_test = train_test_split(
    X_all, y_all, test_size=0.2, random_state=42, shuffle=False
)

# Scale features
scaler_sk = StandardScaler()
X_train_scaled = scaler_sk.fit_transform(X_train)
X_test_scaled = scaler_sk.transform(X_test)

print(f"Training set: {X_train_scaled.shape[0]} samples")
print(f"Test set: {X_test_scaled.shape[0]} samples")
print(f"Training defect rate: {y_train.mean():.2%}")
print(f"Test defect rate: {y_test.mean():.2%}")

What does shuffle=False do?

Setting `shuffle=False` in `train_test_split` takes the first 80% of rows as training and the last 20% as test. Since the data is ordered by production sequence, this ensures the model never trains on parts produced after the test parts.

The `StandardScaler` is fitted only on the training set, then applied to the test set. This prevents data leakage from the test distribution into the scaling parameters.

  • Run Cell 5 by pressing **Shift+Enter**.
  • You should see output showing `Training set: 4000 samples`, `Test set: 1000 samples`, and the defect rates for each split.

Seeing a NameError for sensor_names or df_sensors?

Make sure you have run all previous cells (Cells 1 through 4) in order. Jupyter notebooks require cells to be executed sequentially for variables to persist in memory.

If your kernel was restarted, select **Cell** then **Run All Above** to re-execute all previous cells before running Cell 5.

Help me fix this NameError in my feature engineering cell

✔️ Awesome, I've got everything!

Great. Double check that Cell 5 is saved and you can see the training/test split output.

ⓧ I'd like to double check the full code

Here is the complete code for Cells 4 and 5. Make sure your notebook matches this exactly:

# Cell 4: Build PySpark ML Pipeline
# Impute missing values
imputer = SparkImputer(
    inputCols=sensor_names,
    outputCols=[f"{col}_imputed" for col in sensor_names],
    strategy="median"
)

imputed_cols = [f"{col}_imputed" for col in sensor_names]

# Assemble features into a single vector
assembler = VectorAssembler(
    inputCols=imputed_cols,
    outputCol="features_raw"
)

# Scale features
scaler = SparkStandardScaler(
    inputCol="features_raw",
    outputCol="features_scaled",
    withStd=True,
    withMean=True
)

# Build and fit the pipeline
spark_pipeline = SparkPipeline(stages=[imputer, assembler, scaler])
pipeline_model = spark_pipeline.fit(spark_df)
transformed_df = pipeline_model.transform(spark_df)

print("PySpark pipeline fitted and transformed successfully!")
transformed_df.select("features_scaled", "defect").show(5, truncate=True)
# Cell 5: Feature Engineering - Interaction Terms
df_processed = df_sensors.copy()

# Impute missing values
imputer_sk = SimpleImputer(strategy="median")
df_processed[sensor_names] = imputer_sk.fit_transform(df_processed[sensor_names])

# Engineer interaction features
df_processed["torque_angle_interaction"] = (
    df_processed["OP070_torque_value"] * df_processed["OP070_angle_value"]
)
df_processed["current_voltage_product"] = (
    df_processed["OP090_current_value"] * df_processed["OP090_voltage_value"]
)
df_processed["vibration_magnitude"] = np.sqrt(
    df_processed["OP110_vibration_x"]**2 +
    df_processed["OP110_vibration_y"]**2 +
    df_processed["OP110_vibration_z"]**2
)

feature_cols = sensor_names + [
    "torque_angle_interaction",
    "current_voltage_product",
    "vibration_magnitude"
]

X_all = df_processed[feature_cols].values
y_all = df_processed["defect"].values

# Time-aware split (simulating production order)
X_train, X_test, y_train, y_test = train_test_split(
    X_all, y_all, test_size=0.2, random_state=42, shuffle=False
)

# Scale features
scaler_sk = StandardScaler()
X_train_scaled = scaler_sk.fit_transform(X_train)
X_test_scaled = scaler_sk.transform(X_test)

print(f"Training set: {X_train_scaled.shape[0]} samples")
print(f"Test set: {X_test_scaled.shape[0]} samples")
print(f"Training defect rate: {y_train.mean():.2%}")
print(f"Test defect rate: {y_test.mean():.2%}")

Your feature engineering pipeline is complete. You have 23 features (20 original sensors plus 3 interaction terms), cleanly split into 4,000 training samples and 1,000 test samples. Next up, you will feed these features into TabPFN and see how a pretrained foundation model handles tabular classification with zero tuning.

Train and Evaluate TabPFN

Your feature-engineered data is split and scaled, ready for model training. Now it's time to put the first contender through its paces.

In this step, you will train TabPFN, a pretrained tabular foundation model that works like GPT but for structured data. Instead of training from scratch on your dataset, TabPFN has already learned patterns from millions of synthetic classification tasks. You will feed it your automotive sensor data and measure how well it predicts defects with zero hyperparameter tuning.

In this step, get ready to:
  • Train the TabPFN foundation model on your manufacturing sensor data and generate calibrated predictions.
  • Compute AUROC, F1, and log-loss evaluation metrics with timing measurements.
  • Run a performance-vs-data-size experiment to reveal TabPFN's small-data advantage.
Initialize and train the TabPFN classifier

Traditional ML models require you to choose hyperparameters, tune them with cross-validation, and iterate. TabPFN skips all of that. It uses a pretrained neural network (served via the cloud API) that predicts on new tabular tasks in a single forward pass.

  • Create a new cell in your notebook (Cell 6) and add the following code to train TabPFN and compute evaluation metrics:
# Cell 6: Train and Evaluate TabPFN
print("=" * 60)
print("MODEL 1: TabPFN (Pretrained Tabular Foundation Model)")
print("=" * 60)

# TabPFN via cloud API (no GPU required)
tabpfn_clf = TabPFNClassifier()

start_time = time.time()
tabpfn_clf.fit(X_train_scaled, y_train)
tabpfn_fit_time = time.time() - start_time

start_time = time.time()
tabpfn_probs = tabpfn_clf.predict_proba(X_test_scaled)
tabpfn_preds = tabpfn_clf.predict(X_test_scaled)
tabpfn_inference_time = time.time() - start_time

tabpfn_auroc = roc_auc_score(y_test, tabpfn_probs[:, 1])
tabpfn_f1 = f1_score(y_test, tabpfn_preds)
tabpfn_logloss = log_loss(y_test, tabpfn_probs)

print(f"AUROC: {tabpfn_auroc:.4f}")
print(f"F1 Score: {tabpfn_f1:.4f}")
print(f"Log-Loss: {tabpfn_logloss:.4f}")
print(f"Fit time: {tabpfn_fit_time:.2f}s")
print(f"Inference time: {tabpfn_inference_time:.2f}s")

What does this code do?

TabPFNClassifier() creates a classifier that uses the TabPFN cloud API for inference. No GPU is needed on your machine because the heavy computation happens on PriorLabs' servers.

The fit call uploads your training data. Then predict_proba returns calibrated class probabilities (not just hard labels), which is critical for manufacturing quality decisions where you need to know how confident the model is.

The three metrics capture different aspects of model quality: AUROC measures ranking ability across all thresholds, F1 balances precision and recall at the default threshold, and log-loss penalizes overconfident wrong predictions.

What happens on first run?

The first time you run this cell, TabPFN will automatically open a browser window for you to log in via PriorLabs and accept the license terms. Create a free account, complete the login process, then return to your notebook. Your authentication token is cached locally so you only need to do this once.

  • Run the cell.
  • If a browser window opens for authentication, complete the login/registration process and return to your notebook.
  • You should see AUROC, F1, log-loss, fit time, and inference time printed in the output.

Cell hanging after authentication?

If you are using Safari, the authentication callback may not work. Try using Chrome or Firefox as your default browser instead.

If the browser did not open at all, make sure you have an active internet connection. The TabPFN client requires network access to send data to the cloud API.

Help me debug my TabPFN authentication issue

Test performance across different data sizes

One of TabPFN's key advantages is its performance on small datasets. In manufacturing, you often have limited defective samples because defects are rare.

This experiment trains TabPFN on progressively larger subsets of your data to show how its AUROC changes with training size.

  • Create a new cell (Cell 7) and add the following code to run the data-size experiment:
# Cell 7: TabPFN Performance vs Data Size
print("\nTabPFN Performance vs Training Data Size:")
print("-" * 45)

subset_sizes = [100, 250, 500, 1000, 2000, len(X_train_scaled)]
tabpfn_aurocs_by_size = []

for size in subset_sizes:
    X_sub = X_train_scaled[:size]
    y_sub = y_train[:size]

    clf_sub = TabPFNClassifier()
    clf_sub.fit(X_sub, y_sub)
    probs_sub = clf_sub.predict_proba(X_test_scaled)
    auroc_sub = roc_auc_score(y_test, probs_sub[:, 1])
    tabpfn_aurocs_by_size.append(auroc_sub)
    print(f"  n={size:>5}: AUROC = {auroc_sub:.4f}")

What does this experiment reveal?

The loop trains a fresh TabPFN on subsets of 100, 250, 500, 1000, 2000, and all 4000 training samples. Each model is evaluated on the same held-out test set.

You should see that TabPFN achieves surprisingly strong AUROC even with just a few hundred samples. This is because it has already learned general patterns about tabular classification from its pretraining on millions of synthetic tasks. Later, you will compare these results against XGBoost to see where each model wins.

  • Run the cell.
  • You should see a table of AUROC scores, one per subset size, showing how TabPFN's accuracy scales with training data.

Seeing rate limit errors?

This cell makes multiple API calls (one per subset size). If you hit a rate limit, wait a few seconds and re-run the cell. The free tier has a generous daily credit limit that resets at 00:00 UTC.

Help me fix TabPFN rate limit or API errors

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook.

ⓧ I'd like to double check the full code

Here is the complete notebook up to this point:

# Cell 1: Imports and Setup
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    roc_auc_score, roc_curve, precision_recall_curve,
    f1_score, log_loss, classification_report
)
from sklearn.calibration import calibration_curve
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from xgboost import XGBClassifier
from tabpfn_client import TabPFNClassifier
from pyspark.sql import SparkSession
from pyspark.ml import Pipeline as SparkPipeline
from pyspark.ml.feature import (
    VectorAssembler, StandardScaler as SparkStandardScaler,
    Imputer as SparkImputer
)
import time

print("All imports successful!")

# Cell 2: Generate Synthetic Automotive Sensor Data
np.random.seed(42)

n_samples = 5000
n_features = 20

X, y = make_classification(
    n_samples=n_samples,
    n_features=n_features,
    n_informative=12,
    n_redundant=4,
    n_clusters_per_class=2,
    weights=[0.95, 0.05],
    flip_y=0.02,
    random_state=42
)

sensor_names = [
    "OP070_torque_value", "OP070_angle_value", "OP080_torque_value",
    "OP080_angle_value", "OP090_current_value", "OP090_voltage_value",
    "OP100_pressure_value", "OP100_temperature_value", "OP110_vibration_x",
    "OP110_vibration_y", "OP110_vibration_z", "OP120_rodage_current",
    "OP120_rodage_voltage", "OP120_speed_value", "OP070_force_value",
    "OP080_force_value", "OP090_resistance_value", "OP100_flow_rate",
    "OP110_frequency_value", "OP120_duration_value"
]

df_sensors = pd.DataFrame(X, columns=sensor_names)
df_sensors["defect"] = y

mask = np.random.random(df_sensors[sensor_names].shape) < 0.03
df_sensors[sensor_names] = df_sensors[sensor_names].where(~mask)

print(f"Dataset shape: {df_sensors.shape}")
print(f"Class distribution:\n{df_sensors['defect'].value_counts()}")
print(f"Defect rate: {df_sensors['defect'].mean():.2%}")
print(f"Missing values: {df_sensors.isnull().sum().sum()}")

# Cell 3: PySpark Feature Engineering Pipeline
spark = SparkSession.builder \
    .master("local[*]") \
    .appName("AutomotiveDefectPrediction") \
    .getOrCreate()

spark_df = spark.createDataFrame(df_sensors)
spark_df.printSchema()
spark_df.describe().show()

spark_df.groupBy("defect").count().show()

# Cell 4: Build PySpark ML Pipeline
imputer = SparkImputer(
    inputCols=sensor_names,
    outputCols=[f"{col}_imputed" for col in sensor_names],
    strategy="median"
)

imputed_cols = [f"{col}_imputed" for col in sensor_names]

assembler = VectorAssembler(
    inputCols=imputed_cols,
    outputCol="features_raw"
)

scaler = SparkStandardScaler(
    inputCol="features_raw",
    outputCol="features_scaled",
    withStd=True,
    withMean=True
)

spark_pipeline = SparkPipeline(stages=[imputer, assembler, scaler])
pipeline_model = spark_pipeline.fit(spark_df)
transformed_df = pipeline_model.transform(spark_df)

print("PySpark pipeline fitted and transformed successfully!")
transformed_df.select("features_scaled", "defect").show(5, truncate=True)

# Cell 5: Feature Engineering - Interaction Terms
df_processed = df_sensors.copy()

imputer_sk = SimpleImputer(strategy="median")
df_processed[sensor_names] = imputer_sk.fit_transform(df_processed[sensor_names])

df_processed["torque_angle_interaction"] = (
    df_processed["OP070_torque_value"] * df_processed["OP070_angle_value"]
)
df_processed["current_voltage_product"] = (
    df_processed["OP090_current_value"] * df_processed["OP090_voltage_value"]
)
df_processed["vibration_magnitude"] = np.sqrt(
    df_processed["OP110_vibration_x"]**2 +
    df_processed["OP110_vibration_y"]**2 +
    df_processed["OP110_vibration_z"]**2
)

feature_cols = sensor_names + [
    "torque_angle_interaction",
    "current_voltage_product",
    "vibration_magnitude"
]

X_all = df_processed[feature_cols].values
y_all = df_processed["defect"].values

X_train, X_test, y_train, y_test = train_test_split(
    X_all, y_all, test_size=0.2, random_state=42, shuffle=False
)

scaler_sk = StandardScaler()
X_train_scaled = scaler_sk.fit_transform(X_train)
X_test_scaled = scaler_sk.transform(X_test)

print(f"Training set: {X_train_scaled.shape[0]} samples")
print(f"Test set: {X_test_scaled.shape[0]} samples")
print(f"Training defect rate: {y_train.mean():.2%}")
print(f"Test defect rate: {y_test.mean():.2%}")

# Cell 6: Train and Evaluate TabPFN
print("=" * 60)
print("MODEL 1: TabPFN (Pretrained Tabular Foundation Model)")
print("=" * 60)

tabpfn_clf = TabPFNClassifier()

start_time = time.time()
tabpfn_clf.fit(X_train_scaled, y_train)
tabpfn_fit_time = time.time() - start_time

start_time = time.time()
tabpfn_probs = tabpfn_clf.predict_proba(X_test_scaled)
tabpfn_preds = tabpfn_clf.predict(X_test_scaled)
tabpfn_inference_time = time.time() - start_time

tabpfn_auroc = roc_auc_score(y_test, tabpfn_probs[:, 1])
tabpfn_f1 = f1_score(y_test, tabpfn_preds)
tabpfn_logloss = log_loss(y_test, tabpfn_probs)

print(f"AUROC: {tabpfn_auroc:.4f}")
print(f"F1 Score: {tabpfn_f1:.4f}")
print(f"Log-Loss: {tabpfn_logloss:.4f}")
print(f"Fit time: {tabpfn_fit_time:.2f}s")
print(f"Inference time: {tabpfn_inference_time:.2f}s")

# Cell 7: TabPFN Performance vs Data Size
print("\nTabPFN Performance vs Training Data Size:")
print("-" * 45)

subset_sizes = [100, 250, 500, 1000, 2000, len(X_train_scaled)]
tabpfn_aurocs_by_size = []

for size in subset_sizes:
    X_sub = X_train_scaled[:size]
    y_sub = y_train[:size]

    clf_sub = TabPFNClassifier()
    clf_sub.fit(X_sub, y_sub)
    probs_sub = clf_sub.predict_proba(X_test_scaled)
    auroc_sub = roc_auc_score(y_test, probs_sub[:, 1])
    tabpfn_aurocs_by_size.append(auroc_sub)
    print(f"  n={size:>5}: AUROC = {auroc_sub:.4f}")

You now have a complete evaluation of TabPFN on your manufacturing data. Next up, you will train XGBoost with hyperparameter tuning and see how a traditional gradient boosting approach compares.

Train and Evaluate XGBoost

TabPFN showed what a pretrained foundation model can achieve with zero hyperparameter tuning. Now it's time to bring in the production workhorse.

XGBoost is one of the most widely deployed gradient boosting libraries in industry. In this step, you'll configure it to handle the severe class imbalance in your defect data, tune its hyperparameters with cross-validation, and extract feature importances that reveal which sensors matter most.

In this step, get ready to:
  • Configure XGBClassifier with class imbalance handling and run a 5-fold grid search.
  • Evaluate XGBoost on the test set with AUROC, F1, and log-loss metrics.
  • Extract and visualize feature importances to identify the most predictive sensors.
Configure XGBoost with class imbalance handling

Your dataset has a 95/5 class split. If XGBoost treats both classes equally, it will learn to predict "OK" every time and still achieve 95% accuracy. The scale_pos_weight parameter tells XGBoost to penalize misclassifying the rare defect class more heavily.

  • Create a new cell in your notebook and add the following code to configure XGBoost and run a grid search:
# Cell 8: Train and Evaluate XGBoost
print("\n" + "=" * 60)
print("MODEL 2: XGBoost (Gradient Boosted Decision Trees)")
print("=" * 60)

# Calculate scale_pos_weight for class imbalance
n_negative = (y_train == 0).sum()
n_positive = (y_train == 1).sum()
scale_pos_weight = n_negative / n_positive

# Hyperparameter grid search
param_grid = {
    "learning_rate": [0.05, 0.1, 0.2],
    "max_depth": [3, 5, 7],
    "n_estimators": [100, 200]
}

xgb_clf = XGBClassifier(
    objective="binary:logistic",
    scale_pos_weight=scale_pos_weight,
    eval_metric="logloss",
    random_state=42
)

start_time = time.time()
grid_search = GridSearchCV(
    xgb_clf, param_grid,
    scoring="roc_auc", cv=5, n_jobs=-1, verbose=0
)
grid_search.fit(X_train_scaled, y_train)
xgb_fit_time = time.time() - start_time

best_xgb = grid_search.best_estimator_
print(f"Best params: {grid_search.best_params_}")
print(f"Best CV AUROC: {grid_search.best_score_:.4f}")

start_time = time.time()
xgb_probs = best_xgb.predict_proba(X_test_scaled)
xgb_preds = best_xgb.predict(X_test_scaled)
xgb_inference_time = time.time() - start_time

xgb_auroc = roc_auc_score(y_test, xgb_probs[:, 1])
xgb_f1 = f1_score(y_test, xgb_preds)
xgb_logloss = log_loss(y_test, xgb_probs)

print(f"\nTest AUROC: {xgb_auroc:.4f}")
print(f"F1 Score: {xgb_f1:.4f}")
print(f"Log-Loss: {xgb_logloss:.4f}")
print(f"Training time (with CV): {xgb_fit_time:.2f}s")
print(f"Inference time: {xgb_inference_time:.4f}s")

What does this code do?

This cell configures XGBoost for imbalanced classification, finds the best hyperparameters, and evaluates the tuned model.

  • scale_pos_weight is the ratio of negative to positive samples. This tells XGBoost to weight defect samples higher during training so it doesn't ignore the minority class.
  • GridSearchCV tries every combination of learning_rate, max_depth, and n_estimators (18 total combinations x 5 folds = 90 model fits) and picks the one with the best AUROC.
  • best_xgb stores the best-performing model from the grid search. Predictions and metrics are then computed on the held-out test set.
  • Run the cell. This may take 30-60 seconds as the grid search trains 90 models.
  • You should see the best hyperparameters printed, followed by the test AUROC, F1 score, log-loss, and timing information.

Why use AUROC instead of accuracy?

With a 95/5 class split, a model that predicts "OK" for every part would score 95% accuracy. AUROC measures how well the model separates defective parts from OK parts regardless of the classification threshold. It's the standard metric for imbalanced industrial quality datasets.

Grid search taking too long?

If the grid search runs for more than 2 minutes, check that n_jobs=-1 is set. This uses all available CPU cores. On older hardware, you can reduce the grid by removing one value from max_depth (e.g., keep only [3, 5]).

Help me debug why my XGBoost grid search is running slowly

Extract and visualize feature importances

One of XGBoost's key advantages over black-box models is interpretability. Feature importances reveal which production-line sensors contribute most to defect prediction.

  • Create a new cell below and add this code to visualize the top 15 feature importances:
# Cell 9: XGBoost Feature Importance
importances = best_xgb.feature_importances_
importance_df = pd.DataFrame({
    "feature": feature_cols,
    "importance": importances
}).sort_values("importance", ascending=False)

plt.figure(figsize=(10, 8))
sns.barplot(data=importance_df.head(15), x="importance", y="feature")
plt.title("Top 15 Feature Importances (XGBoost)")
plt.xlabel("Feature Importance (Gain)")
plt.ylabel("Sensor Feature")
plt.tight_layout()
plt.savefig("xgboost_feature_importance.png", dpi=150)
plt.show()

print("\nTop 5 most important features:")
for _, row in importance_df.head(5).iterrows():
    print(f"  {row['feature']}: {row['importance']:.4f}")

What does this code do?

  • feature_importances_ extracts how much each feature contributed to reducing the loss function across all trees. Higher values mean the feature was more useful for splitting decisions.
  • The bar chart shows the top 15 features ranked by importance (gain). In a real manufacturing setting, this tells engineers which sensors to monitor most closely.
  • plt.savefig("xgboost_feature_importance.png", dpi=150) saves the chart to disk so you can include it in reports or presentations.
  • Run the cell.
  • You should see a horizontal bar chart showing the top 15 sensor features ranked by importance, plus a printed list of the top 5 features with their importance scores.

Why feature importances matter in manufacturing

In automotive production, knowing that OP070_torque_value is a top predictor means engineers can prioritize calibrating that specific torque sensor. If the engineered features (like vibration_magnitude) rank high, it validates that your feature engineering added real predictive power beyond raw readings.

Bar chart not displaying?

If you see a blank output or the chart doesn't render inline, make sure you're running in a Jupyter notebook (not a plain Python script). In Jupyter, plt.show() renders the figure below the cell. Check that matplotlib and seaborn are both imported in Cell 1.

Help me troubleshoot why my matplotlib chart isn't showing in Jupyter

✔️ Awesome, I've got everything!

Great. Double check you've saved your notebook before moving on.

ⓧ I'd like to double check the full code

Here is all the code in your notebook up to this point. Compare your cells against this:

# Cell 1: Imports and Setup
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    roc_auc_score, roc_curve, precision_recall_curve,
    f1_score, log_loss, classification_report
)
from sklearn.calibration import calibration_curve
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from xgboost import XGBClassifier
from tabpfn_client import TabPFNClassifier
from pyspark.sql import SparkSession
from pyspark.ml import Pipeline as SparkPipeline
from pyspark.ml.feature import (
    VectorAssembler, StandardScaler as SparkStandardScaler,
    Imputer as SparkImputer
)
import time

print("All imports successful!")

# Cell 2: Generate Synthetic Automotive Sensor Data
np.random.seed(42)

n_samples = 5000
n_features = 20

X, y = make_classification(
    n_samples=n_samples,
    n_features=n_features,
    n_informative=12,
    n_redundant=4,
    n_clusters_per_class=2,
    weights=[0.95, 0.05],
    flip_y=0.02,
    random_state=42
)

sensor_names = [
    "OP070_torque_value", "OP070_angle_value", "OP080_torque_value",
    "OP080_angle_value", "OP090_current_value", "OP090_voltage_value",
    "OP100_pressure_value", "OP100_temperature_value", "OP110_vibration_x",
    "OP110_vibration_y", "OP110_vibration_z", "OP120_rodage_current",
    "OP120_rodage_voltage", "OP120_speed_value", "OP070_force_value",
    "OP080_force_value", "OP090_resistance_value", "OP100_flow_rate",
    "OP110_frequency_value", "OP120_duration_value"
]

df_sensors = pd.DataFrame(X, columns=sensor_names)
df_sensors["defect"] = y

mask = np.random.random(df_sensors[sensor_names].shape) < 0.03
df_sensors[sensor_names] = df_sensors[sensor_names].where(~mask)

print(f"Dataset shape: {df_sensors.shape}")
print(f"Class distribution:\n{df_sensors['defect'].value_counts()}")
print(f"Defect rate: {df_sensors['defect'].mean():.2%}")
print(f"Missing values: {df_sensors.isnull().sum().sum()}")

# Cell 3: PySpark Feature Engineering Pipeline
spark = SparkSession.builder \
    .master("local[*]") \
    .appName("AutomotiveDefectPrediction") \
    .getOrCreate()

spark_df = spark.createDataFrame(df_sensors)
spark_df.printSchema()
spark_df.describe().show()

spark_df.groupBy("defect").count().show()

# Cell 4: Build PySpark ML Pipeline
imputer = SparkImputer(
    inputCols=sensor_names,
    outputCols=[f"{col}_imputed" for col in sensor_names],
    strategy="median"
)

imputed_cols = [f"{col}_imputed" for col in sensor_names]

assembler = VectorAssembler(
    inputCols=imputed_cols,
    outputCol="features_raw"
)

scaler = SparkStandardScaler(
    inputCol="features_raw",
    outputCol="features_scaled",
    withStd=True,
    withMean=True
)

spark_pipeline = SparkPipeline(stages=[imputer, assembler, scaler])
pipeline_model = spark_pipeline.fit(spark_df)
transformed_df = pipeline_model.transform(spark_df)

print("PySpark pipeline fitted and transformed successfully!")
transformed_df.select("features_scaled", "defect").show(5, truncate=True)

# Cell 5: Feature Engineering - Interaction Terms
df_processed = df_sensors.copy()

imputer_sk = SimpleImputer(strategy="median")
df_processed[sensor_names] = imputer_sk.fit_transform(df_processed[sensor_names])

df_processed["torque_angle_interaction"] = (
    df_processed["OP070_torque_value"] * df_processed["OP070_angle_value"]
)
df_processed["current_voltage_product"] = (
    df_processed["OP090_current_value"] * df_processed["OP090_voltage_value"]
)
df_processed["vibration_magnitude"] = np.sqrt(
    df_processed["OP110_vibration_x"]**2 +
    df_processed["OP110_vibration_y"]**2 +
    df_processed["OP110_vibration_z"]**2
)

feature_cols = sensor_names + [
    "torque_angle_interaction",
    "current_voltage_product",
    "vibration_magnitude"
]

X_all = df_processed[feature_cols].values
y_all = df_processed["defect"].values

X_train, X_test, y_train, y_test = train_test_split(
    X_all, y_all, test_size=0.2, random_state=42, shuffle=False
)

scaler_sk = StandardScaler()
X_train_scaled = scaler_sk.fit_transform(X_train)
X_test_scaled = scaler_sk.transform(X_test)

print(f"Training set: {X_train_scaled.shape[0]} samples")
print(f"Test set: {X_test_scaled.shape[0]} samples")
print(f"Training defect rate: {y_train.mean():.2%}")
print(f"Test defect rate: {y_test.mean():.2%}")

# Cell 6: Train and Evaluate TabPFN
print("=" * 60)
print("MODEL 1: TabPFN (Pretrained Tabular Foundation Model)")
print("=" * 60)

tabpfn_clf = TabPFNClassifier()

start_time = time.time()
tabpfn_clf.fit(X_train_scaled, y_train)
tabpfn_fit_time = time.time() - start_time

start_time = time.time()
tabpfn_probs = tabpfn_clf.predict_proba(X_test_scaled)
tabpfn_preds = tabpfn_clf.predict(X_test_scaled)
tabpfn_inference_time = time.time() - start_time

tabpfn_auroc = roc_auc_score(y_test, tabpfn_probs[:, 1])
tabpfn_f1 = f1_score(y_test, tabpfn_preds)
tabpfn_logloss = log_loss(y_test, tabpfn_probs)

print(f"AUROC: {tabpfn_auroc:.4f}")
print(f"F1 Score: {tabpfn_f1:.4f}")
print(f"Log-Loss: {tabpfn_logloss:.4f}")
print(f"Fit time: {tabpfn_fit_time:.2f}s")
print(f"Inference time: {tabpfn_inference_time:.2f}s")

# Cell 7: TabPFN Performance vs Data Size
print("\nTabPFN Performance vs Training Data Size:")
print("-" * 45)

subset_sizes = [100, 250, 500, 1000, 2000, len(X_train_scaled)]
tabpfn_aurocs_by_size = []

for size in subset_sizes:
    X_sub = X_train_scaled[:size]
    y_sub = y_train[:size]

    clf_sub = TabPFNClassifier()
    clf_sub.fit(X_sub, y_sub)
    probs_sub = clf_sub.predict_proba(X_test_scaled)
    auroc_sub = roc_auc_score(y_test, probs_sub[:, 1])
    tabpfn_aurocs_by_size.append(auroc_sub)
    print(f"  n={size:>5}: AUROC = {auroc_sub:.4f}")

# Cell 8: Train and Evaluate XGBoost
print("\n" + "=" * 60)
print("MODEL 2: XGBoost (Gradient Boosted Decision Trees)")
print("=" * 60)

# Calculate scale_pos_weight for class imbalance
n_negative = (y_train == 0).sum()
n_positive = (y_train == 1).sum()
scale_pos_weight = n_negative / n_positive

# Hyperparameter grid search
param_grid = {
    "learning_rate": [0.05, 0.1, 0.2],
    "max_depth": [3, 5, 7],
    "n_estimators": [100, 200]
}

xgb_clf = XGBClassifier(
    objective="binary:logistic",
    scale_pos_weight=scale_pos_weight,
    eval_metric="logloss",
    random_state=42
)

start_time = time.time()
grid_search = GridSearchCV(
    xgb_clf, param_grid,
    scoring="roc_auc", cv=5, n_jobs=-1, verbose=0
)
grid_search.fit(X_train_scaled, y_train)
xgb_fit_time = time.time() - start_time

best_xgb = grid_search.best_estimator_
print(f"Best params: {grid_search.best_params_}")
print(f"Best CV AUROC: {grid_search.best_score_:.4f}")

start_time = time.time()
xgb_probs = best_xgb.predict_proba(X_test_scaled)
xgb_preds = best_xgb.predict(X_test_scaled)
xgb_inference_time = time.time() - start_time

xgb_auroc = roc_auc_score(y_test, xgb_probs[:, 1])
xgb_f1 = f1_score(y_test, xgb_preds)
xgb_logloss = log_loss(y_test, xgb_probs)

print(f"\nTest AUROC: {xgb_auroc:.4f}")
print(f"F1 Score: {xgb_f1:.4f}")
print(f"Log-Loss: {xgb_logloss:.4f}")
print(f"Training time (with CV): {xgb_fit_time:.2f}s")
print(f"Inference time: {xgb_inference_time:.4f}s")

# Cell 9: XGBoost Feature Importance
importances = best_xgb.feature_importances_
importance_df = pd.DataFrame({
    "feature": feature_cols,
    "importance": importances
}).sort_values("importance", ascending=False)

plt.figure(figsize=(10, 8))
sns.barplot(data=importance_df.head(15), x="importance", y="feature")
plt.title("Top 15 Feature Importances (XGBoost)")
plt.xlabel("Feature Importance (Gain)")
plt.ylabel("Sensor Feature")
plt.tight_layout()
plt.savefig("xgboost_feature_importance.png", dpi=150)
plt.show()

print("\nTop 5 most important features:")
for _, row in importance_df.head(5).iterrows():
    print(f"  {row['feature']}: {row['importance']:.4f}")

You now have a fully tuned XGBoost model with interpretable feature importances. Next up, you'll train a Random Forest baseline and create a head-to-head comparison of all three models.

Build the Baseline Model and Comparative Analysis

You now have two powerful models evaluated: TabPFN (zero-shot foundation model) and XGBoost (tuned gradient boosting). But without a simpler baseline to compare against, it is hard to know whether the complexity of those models is paying off.

In this step, you will train a Random Forest baseline, then bring all three models together into a comprehensive comparison. The final output is a benchmarking report with overlaid ROC curves, calibration plots, and a written recommendation on when each approach is the right fit.

In this step, get ready to:
  • Train a Random Forest classifier as the baseline model.
  • Plot comparative ROC and calibration curves for all three models.
  • Write a data-driven recommendation on when to use each modeling approach.
Train the Random Forest baseline

A Random Forest is an ensemble of decision trees that votes on the final prediction. It requires almost no tuning and handles class imbalance natively through its class_weight parameter.

  • Create a new cell in your notebook (Cell 10) and train the Random Forest baseline by pasting this code:
# Cell 10: Random Forest Baseline
print("\n" + "=" * 60)
print("MODEL 3: Random Forest (Baseline)")
print("=" * 60)

rf_clf = RandomForestClassifier(
    n_estimators=200,
    class_weight="balanced",
    max_depth=10,
    random_state=42,
    n_jobs=-1
)

start_time = time.time()
rf_clf.fit(X_train_scaled, y_train)
rf_fit_time = time.time() - start_time

start_time = time.time()
rf_probs = rf_clf.predict_proba(X_test_scaled)
rf_preds = rf_clf.predict(X_test_scaled)
rf_inference_time = time.time() - start_time

rf_auroc = roc_auc_score(y_test, rf_probs[:, 1])
rf_f1 = f1_score(y_test, rf_preds)
rf_logloss = log_loss(y_test, rf_probs)

print(f"AUROC: {rf_auroc:.4f}")
print(f"F1 Score: {rf_f1:.4f}")
print(f"Log-Loss: {rf_logloss:.4f}")
print(f"Training time: {rf_fit_time:.2f}s")
print(f"Inference time: {rf_inference_time:.4f}s")

What does this code do?

This cell trains a Random Forest with 200 trees and a maximum depth of 10. The key parameter is class_weight="balanced", which automatically adjusts sample weights inversely proportional to class frequencies. This is scikit-learn's built-in way to handle the 95/5 class imbalance without manually computing weights.

The n_jobs=-1 parameter uses all available CPU cores for parallel tree construction, speeding up training.

  • Run the cell.
  • You should see the AUROC, F1, Log-Loss, training time, and inference time printed below the cell.

Seeing a MemoryError?

If your machine runs out of memory, reduce n_estimators from 200 to 100. The results will be slightly different but the comparison will still be valid.

Help me debug a MemoryError when training my Random Forest classifier.

Create the comparative visualization

Individual metrics are useful, but overlaying all three models on the same plot reveals the story at a glance. An AUROC curve shows each model's trade-off between true positives and false positives. A calibration curve shows whether predicted probabilities match observed frequencies.

  • Create a new cell (Cell 11) and generate the comparative plots by pasting this code:
# Cell 11: Comparative ROC Curves
fig, axes = plt.subplots(1, 2, figsize=(14, 5))

ax = axes[0]
for name, probs, color in [
    ("TabPFN", tabpfn_probs[:, 1], "blue"),
    ("XGBoost", xgb_probs[:, 1], "red"),
    ("Random Forest", rf_probs[:, 1], "green")
]:
    fpr, tpr, _ = roc_curve(y_test, probs)
    auroc = roc_auc_score(y_test, probs)
    ax.plot(fpr, tpr, color=color, label=f"{name} (AUROC={auroc:.3f})")

ax.plot([0, 1], [0, 1], "k--", alpha=0.5)
ax.set_xlabel("False Positive Rate")
ax.set_ylabel("True Positive Rate")
ax.set_title("ROC Curve Comparison")
ax.legend(loc="lower right")
ax.grid(True, alpha=0.3)

ax = axes[1]
for name, probs, color in [
    ("TabPFN", tabpfn_probs[:, 1], "blue"),
    ("XGBoost", xgb_probs[:, 1], "red"),
    ("Random Forest", rf_probs[:, 1], "green")
]:
    fraction_pos, mean_predicted = calibration_curve(y_test, probs, n_bins=10)
    ax.plot(mean_predicted, fraction_pos, "o-", color=color, label=name)

ax.plot([0, 1], [0, 1], "k--", alpha=0.5, label="Perfectly calibrated")
ax.set_xlabel("Mean Predicted Probability")
ax.set_ylabel("Fraction of Positives")
ax.set_title("Calibration Curves")
ax.legend(loc="lower right")
ax.grid(True, alpha=0.3)

plt.tight_layout()
plt.savefig("model_comparison.png", dpi=150)
plt.show()

What does this code do?

The left plot overlays three ROC curves. Each curve shows how well a model separates defective parts from OK parts at every possible threshold. A curve closer to the top-left corner means better discrimination.

The right plot checks probability calibration. A perfectly calibrated model's curve hugs the diagonal. If a model says "30% chance of defect" for a batch of parts, roughly 30% should actually be defective. Poor calibration means predicted probabilities are misleading.

  • Run the cell.
  • You should see a side-by-side figure with ROC curves (left) and calibration curves (right) for all three models, color-coded with AUROC values in the legend.
  • The file model_comparison.png is also saved to your project folder.

Plot not displaying inline?

If the plot does not render inside your notebook, make sure you are running in a Jupyter environment (not a plain Python script). The model_comparison.png file is still saved regardless. You can open it directly from your file browser.

Help me get matplotlib plots to display inline in my Jupyter notebook.

Write the benchmarking summary and recommendation

Numbers and plots tell the story, but a written recommendation turns benchmarking into a decision. This final cell consolidates all metrics into a comparison table and documents when each model is the right choice.

  • Create a new cell (Cell 12) and build the summary report by pasting this code:
# Cell 12: Summary Comparison Table
print("\n" + "=" * 60)
print("BENCHMARKING SUMMARY")
print("=" * 60)

results = pd.DataFrame({
    "Model": ["TabPFN", "XGBoost (tuned)", "Random Forest"],
    "AUROC": [tabpfn_auroc, xgb_auroc, rf_auroc],
    "F1": [tabpfn_f1, xgb_f1, rf_f1],
    "Log-Loss": [tabpfn_logloss, xgb_logloss, rf_logloss],
    "Train Time (s)": [tabpfn_fit_time, xgb_fit_time, rf_fit_time],
    "Inference Time (s)": [tabpfn_inference_time, xgb_inference_time, rf_inference_time]
})

print(results.to_string(index=False))

print("\n" + "-" * 60)
print("RECOMMENDATIONS:")
print("-" * 60)
print("""
1. TabPFN excels in the small-data regime (< 10,000 samples):
   - Zero hyperparameter tuning required
   - Calibrated probabilities out of the box
   - Ideal for rapid prototyping and initial feasibility studies

2. XGBoost is the production workhorse for larger datasets:
   - Best performance with tuning on datasets > 10K samples
   - Feature importance for interpretability
   - Faster inference for real-time deployment

3. Random Forest provides a reliable baseline:
   - Good default performance with class_weight='balanced'
   - Useful as a sanity check before trying more complex models
""")

spark.stop()

What does this code do?

The results DataFrame gathers every metric (AUROC, F1, Log-Loss) and timing measurement (training time, inference time) for all three models into one table. This makes it easy to compare trade-offs at a glance.

The recommendations section captures the key insight: TabPFN shines on small datasets with zero tuning. XGBoost wins on larger datasets where you can invest in hyperparameter search. Random Forest serves as a quick sanity check.

Finally, spark.stop() shuts down the PySpark session to free up system resources now that all processing is complete.

  • Run the cell.
  • You should see a formatted comparison table with all five metrics for each model, followed by the three-part recommendation text.

How do I read the results?

Higher AUROC and F1 are better. Lower Log-Loss is better (it penalizes confident wrong predictions). Training time matters when you retrain frequently. Inference time matters for real-time production deployment on the manufacturing line.

Seeing an error about Spark session?

If you see a warning that the Spark session is already stopped, that is fine. It means the session was cleaned up earlier. The rest of the cell output (the comparison table and recommendations) should still print correctly.

Help me understand why spark.stop() is giving me an error.

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook.

ⓧ I'd like to double check the full code

Here is the complete notebook code from Cell 1 through Cell 12. Compare yours against this:

# Cell 1: Imports and Setup
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    roc_auc_score, roc_curve, precision_recall_curve,
    f1_score, log_loss, classification_report
)
from sklearn.calibration import calibration_curve
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from xgboost import XGBClassifier
from tabpfn_client import TabPFNClassifier
from pyspark.sql import SparkSession
from pyspark.ml import Pipeline as SparkPipeline
from pyspark.ml.feature import (
    VectorAssembler, StandardScaler as SparkStandardScaler,
    Imputer as SparkImputer
)
import time

print("All imports successful!")

# Cell 2: Generate Synthetic Automotive Sensor Data
np.random.seed(42)

n_samples = 5000
n_features = 20

X, y = make_classification(
    n_samples=n_samples,
    n_features=n_features,
    n_informative=12,
    n_redundant=4,
    n_clusters_per_class=2,
    weights=[0.95, 0.05],
    flip_y=0.02,
    random_state=42
)

sensor_names = [
    "OP070_torque_value", "OP070_angle_value", "OP080_torque_value",
    "OP080_angle_value", "OP090_current_value", "OP090_voltage_value",
    "OP100_pressure_value", "OP100_temperature_value", "OP110_vibration_x",
    "OP110_vibration_y", "OP110_vibration_z", "OP120_rodage_current",
    "OP120_rodage_voltage", "OP120_speed_value", "OP070_force_value",
    "OP080_force_value", "OP090_resistance_value", "OP100_flow_rate",
    "OP110_frequency_value", "OP120_duration_value"
]

df_sensors = pd.DataFrame(X, columns=sensor_names)
df_sensors["defect"] = y

mask = np.random.random(df_sensors[sensor_names].shape) < 0.03
df_sensors[sensor_names] = df_sensors[sensor_names].where(~mask)

print(f"Dataset shape: {df_sensors.shape}")
print(f"Class distribution:\n{df_sensors['defect'].value_counts()}")
print(f"Defect rate: {df_sensors['defect'].mean():.2%}")
print(f"Missing values: {df_sensors.isnull().sum().sum()}")

# Cell 3: PySpark Feature Engineering Pipeline
spark = SparkSession.builder \
    .master("local[*]") \
    .appName("AutomotiveDefectPrediction") \
    .getOrCreate()

spark_df = spark.createDataFrame(df_sensors)
spark_df.printSchema()
spark_df.describe().show()

spark_df.groupBy("defect").count().show()

# Cell 4: Build PySpark ML Pipeline
imputer = SparkImputer(
    inputCols=sensor_names,
    outputCols=[f"{col}_imputed" for col in sensor_names],
    strategy="median"
)

imputed_cols = [f"{col}_imputed" for col in sensor_names]

assembler = VectorAssembler(
    inputCols=imputed_cols,
    outputCol="features_raw"
)

scaler = SparkStandardScaler(
    inputCol="features_raw",
    outputCol="features_scaled",
    withStd=True,
    withMean=True
)

spark_pipeline = SparkPipeline(stages=[imputer, assembler, scaler])
pipeline_model = spark_pipeline.fit(spark_df)
transformed_df = pipeline_model.transform(spark_df)

print("PySpark pipeline fitted and transformed successfully!")
transformed_df.select("features_scaled", "defect").show(5, truncate=True)

# Cell 5: Feature Engineering - Interaction Terms
df_processed = df_sensors.copy()

imputer_sk = SimpleImputer(strategy="median")
df_processed[sensor_names] = imputer_sk.fit_transform(df_processed[sensor_names])

df_processed["torque_angle_interaction"] = (
    df_processed["OP070_torque_value"] * df_processed["OP070_angle_value"]
)
df_processed["current_voltage_product"] = (
    df_processed["OP090_current_value"] * df_processed["OP090_voltage_value"]
)
df_processed["vibration_magnitude"] = np.sqrt(
    df_processed["OP110_vibration_x"]**2 +
    df_processed["OP110_vibration_y"]**2 +
    df_processed["OP110_vibration_z"]**2
)

feature_cols = sensor_names + [
    "torque_angle_interaction",
    "current_voltage_product",
    "vibration_magnitude"
]

X_all = df_processed[feature_cols].values
y_all = df_processed["defect"].values

X_train, X_test, y_train, y_test = train_test_split(
    X_all, y_all, test_size=0.2, random_state=42, shuffle=False
)

scaler_sk = StandardScaler()
X_train_scaled = scaler_sk.fit_transform(X_train)
X_test_scaled = scaler_sk.transform(X_test)

print(f"Training set: {X_train_scaled.shape[0]} samples")
print(f"Test set: {X_test_scaled.shape[0]} samples")
print(f"Training defect rate: {y_train.mean():.2%}")
print(f"Test defect rate: {y_test.mean():.2%}")

# Cell 6: Train and Evaluate TabPFN
print("=" * 60)
print("MODEL 1: TabPFN (Pretrained Tabular Foundation Model)")
print("=" * 60)

tabpfn_clf = TabPFNClassifier()

start_time = time.time()
tabpfn_clf.fit(X_train_scaled, y_train)
tabpfn_fit_time = time.time() - start_time

start_time = time.time()
tabpfn_probs = tabpfn_clf.predict_proba(X_test_scaled)
tabpfn_preds = tabpfn_clf.predict(X_test_scaled)
tabpfn_inference_time = time.time() - start_time

tabpfn_auroc = roc_auc_score(y_test, tabpfn_probs[:, 1])
tabpfn_f1 = f1_score(y_test, tabpfn_preds)
tabpfn_logloss = log_loss(y_test, tabpfn_probs)

print(f"AUROC: {tabpfn_auroc:.4f}")
print(f"F1 Score: {tabpfn_f1:.4f}")
print(f"Log-Loss: {tabpfn_logloss:.4f}")
print(f"Fit time: {tabpfn_fit_time:.2f}s")
print(f"Inference time: {tabpfn_inference_time:.2f}s")

# Cell 7: TabPFN Performance vs Data Size
print("\nTabPFN Performance vs Training Data Size:")
print("-" * 45)

subset_sizes = [100, 250, 500, 1000, 2000, len(X_train_scaled)]
tabpfn_aurocs_by_size = []

for size in subset_sizes:
    X_sub = X_train_scaled[:size]
    y_sub = y_train[:size]

    clf_sub = TabPFNClassifier()
    clf_sub.fit(X_sub, y_sub)
    probs_sub = clf_sub.predict_proba(X_test_scaled)
    auroc_sub = roc_auc_score(y_test, probs_sub[:, 1])
    tabpfn_aurocs_by_size.append(auroc_sub)
    print(f"  n={size:>5}: AUROC = {auroc_sub:.4f}")

# Cell 8: Train and Evaluate XGBoost
print("\n" + "=" * 60)
print("MODEL 2: XGBoost (Gradient Boosted Decision Trees)")
print("=" * 60)

n_negative = (y_train == 0).sum()
n_positive = (y_train == 1).sum()
scale_pos_weight = n_negative / n_positive

param_grid = {
    "learning_rate": [0.05, 0.1, 0.2],
    "max_depth": [3, 5, 7],
    "n_estimators": [100, 200]
}

xgb_clf = XGBClassifier(
    objective="binary:logistic",
    scale_pos_weight=scale_pos_weight,
    eval_metric="logloss",
    random_state=42
)

start_time = time.time()
grid_search = GridSearchCV(
    xgb_clf, param_grid,
    scoring="roc_auc", cv=5, n_jobs=-1, verbose=0
)
grid_search.fit(X_train_scaled, y_train)
xgb_fit_time = time.time() - start_time

best_xgb = grid_search.best_estimator_
print(f"Best params: {grid_search.best_params_}")
print(f"Best CV AUROC: {grid_search.best_score_:.4f}")

start_time = time.time()
xgb_probs = best_xgb.predict_proba(X_test_scaled)
xgb_preds = best_xgb.predict(X_test_scaled)
xgb_inference_time = time.time() - start_time

xgb_auroc = roc_auc_score(y_test, xgb_probs[:, 1])
xgb_f1 = f1_score(y_test, xgb_preds)
xgb_logloss = log_loss(y_test, xgb_probs)

print(f"\nTest AUROC: {xgb_auroc:.4f}")
print(f"F1 Score: {xgb_f1:.4f}")
print(f"Log-Loss: {xgb_logloss:.4f}")
print(f"Training time (with CV): {xgb_fit_time:.2f}s")
print(f"Inference time: {xgb_inference_time:.4f}s")

# Cell 9: XGBoost Feature Importance
importances = best_xgb.feature_importances_
importance_df = pd.DataFrame({
    "feature": feature_cols,
    "importance": importances
}).sort_values("importance", ascending=False)

plt.figure(figsize=(10, 8))
sns.barplot(data=importance_df.head(15), x="importance", y="feature")
plt.title("Top 15 Feature Importances (XGBoost)")
plt.xlabel("Feature Importance (Gain)")
plt.ylabel("Sensor Feature")
plt.tight_layout()
plt.savefig("xgboost_feature_importance.png", dpi=150)
plt.show()

print("\nTop 5 most important features:")
for _, row in importance_df.head(5).iterrows():
    print(f"  {row['feature']}: {row['importance']:.4f}")

# Cell 10: Random Forest Baseline
print("\n" + "=" * 60)
print("MODEL 3: Random Forest (Baseline)")
print("=" * 60)

rf_clf = RandomForestClassifier(
    n_estimators=200,
    class_weight="balanced",
    max_depth=10,
    random_state=42,
    n_jobs=-1
)

start_time = time.time()
rf_clf.fit(X_train_scaled, y_train)
rf_fit_time = time.time() - start_time

start_time = time.time()
rf_probs = rf_clf.predict_proba(X_test_scaled)
rf_preds = rf_clf.predict(X_test_scaled)
rf_inference_time = time.time() - start_time

rf_auroc = roc_auc_score(y_test, rf_probs[:, 1])
rf_f1 = f1_score(y_test, rf_preds)
rf_logloss = log_loss(y_test, rf_probs)

print(f"AUROC: {rf_auroc:.4f}")
print(f"F1 Score: {rf_f1:.4f}")
print(f"Log-Loss: {rf_logloss:.4f}")
print(f"Training time: {rf_fit_time:.2f}s")
print(f"Inference time: {rf_inference_time:.4f}s")

# Cell 11: Comparative ROC Curves
fig, axes = plt.subplots(1, 2, figsize=(14, 5))

ax = axes[0]
for name, probs, color in [
    ("TabPFN", tabpfn_probs[:, 1], "blue"),
    ("XGBoost", xgb_probs[:, 1], "red"),
    ("Random Forest", rf_probs[:, 1], "green")
]:
    fpr, tpr, _ = roc_curve(y_test, probs)
    auroc = roc_auc_score(y_test, probs)
    ax.plot(fpr, tpr, color=color, label=f"{name} (AUROC={auroc:.3f})")

ax.plot([0, 1], [0, 1], "k--", alpha=0.5)
ax.set_xlabel("False Positive Rate")
ax.set_ylabel("True Positive Rate")
ax.set_title("ROC Curve Comparison")
ax.legend(loc="lower right")
ax.grid(True, alpha=0.3)

ax = axes[1]
for name, probs, color in [
    ("TabPFN", tabpfn_probs[:, 1], "blue"),
    ("XGBoost", xgb_probs[:, 1], "red"),
    ("Random Forest", rf_probs[:, 1], "green")
]:
    fraction_pos, mean_predicted = calibration_curve(y_test, probs, n_bins=10)
    ax.plot(mean_predicted, fraction_pos, "o-", color=color, label=name)

ax.plot([0, 1], [0, 1], "k--", alpha=0.5, label="Perfectly calibrated")
ax.set_xlabel("Mean Predicted Probability")
ax.set_ylabel("Fraction of Positives")
ax.set_title("Calibration Curves")
ax.legend(loc="lower right")
ax.grid(True, alpha=0.3)

plt.tight_layout()
plt.savefig("model_comparison.png", dpi=150)
plt.show()

# Cell 12: Summary Comparison Table
print("\n" + "=" * 60)
print("BENCHMARKING SUMMARY")
print("=" * 60)

results = pd.DataFrame({
    "Model": ["TabPFN", "XGBoost (tuned)", "Random Forest"],
    "AUROC": [tabpfn_auroc, xgb_auroc, rf_auroc],
    "F1": [tabpfn_f1, xgb_f1, rf_f1],
    "Log-Loss": [tabpfn_logloss, xgb_logloss, rf_logloss],
    "Train Time (s)": [tabpfn_fit_time, xgb_fit_time, rf_fit_time],
    "Inference Time (s)": [tabpfn_inference_time, xgb_inference_time, rf_inference_time]
})

print(results.to_string(index=False))

print("\n" + "-" * 60)
print("RECOMMENDATIONS:")
print("-" * 60)
print("""
1. TabPFN excels in the small-data regime (< 10,000 samples):
   - Zero hyperparameter tuning required
   - Calibrated probabilities out of the box
   - Ideal for rapid prototyping and initial feasibility studies

2. XGBoost is the production workhorse for larger datasets:
   - Best performance with tuning on datasets > 10K samples
   - Feature importance for interpretability
   - Faster inference for real-time deployment

3. Random Forest provides a reliable baseline:
   - Good default performance with class_weight='balanced'
   - Useful as a sanity check before trying more complex models
""")

spark.stop()

Secret mission

Build a Data Drift Detector

Your benchmarking pipeline tells you which model is best today. But what happens when the production line changes? Sensors degrade, calibration drifts, and new materials get introduced. Build a statistical drift detector that flags when incoming data no longer matches what your model was trained on.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally with free open-source tools and a free TabPFN cloud API, so there are no ongoing costs.

Resources you used:

  • Project folder on your Desktop (contains your virtual environment, Jupyter notebook, and generated images).
  • Python packages installed inside the virtual environment (tabpfn-client, xgboost, scikit-learn, pyspark, pandas, numpy, matplotlib, seaborn, jupyter).
  • Generated output files (model_comparison.png, xgboost_feature_importance.png, drift_detection.png).

Keep everything running

No action needed. Choose this if you plan to continue experimenting with the notebook, trying different hyperparameters, or extending the benchmarking pipeline.

  • Your project folder, virtual environment, and all generated files stay on your Desktop.
  • Your TabPFN cloud account remains active for future API calls.
  • To resume work later, activate your virtual environment and run jupyter notebook from your project folder.

Pause - I'll come back to this later

Shut down running processes to free up memory. Your files stay intact for next time.

  • In the terminal where Jupyter is running, press Ctrl+C to stop the Jupyter server.
  • Deactivate your virtual environment by running:
deactivate
  • Your notebook, images, and virtual environment remain saved in your project folder on the Desktop.
  • To resume later, navigate to your project folder, reactivate the virtual environment, and launch Jupyter.

Delete - I don't want to use this again

Remove all project files, the virtual environment, and generated outputs from your machine.

  • In the terminal where Jupyter is running, press Ctrl+C to stop the Jupyter server.
  • Deactivate your virtual environment if it is still active:
deactivate
  • Navigate to your Desktop and delete the entire project folder by running:

macOS

cd ~/Desktop
rm -rf benchmark-tabular-ml

Windows

cd ~\Desktop
rmdir /s /q benchmark-tabular-ml
  • Confirm the folder is gone by running:
ls
  • You should no longer see your project folder in the output.
  • This removes your notebook, virtual environment, all installed packages inside it, and the generated image files.
  • Python, VS Code, and Java (if installed for PySpark) remain on your system as general-purpose tools.

Nice Work!

Nice Work!

You just built a complete tabular ML benchmarking system for automotive defect prediction.

You've learned how to:

  • Built a PySpark feature engineering pipeline that processes raw sensor data from production lines into ML-ready features with imputation, scaling, and interaction terms.
  • Trained and evaluated TabPFN, a pretrained tabular foundation model, demonstrating its zero-shot capabilities on small manufacturing datasets against tuned XGBoost and Random Forest baselines.
  • Created a comprehensive model comparison framework with AUROC analysis, calibration curves, and performance-vs-data-size profiling to produce data-driven recommendations on when to use each approach.
  • Built a statistical data drift detector using Kolmogorov-Smirnov tests for production monitoring.

Ready to quiz yourself?