Predict Band Gaps with Machine Learning

Build an ML pipeline to predict semiconductor band gaps from chemical formulas.

Introduction

30 Second Summary

Every LED, solar panel, and glowing material depends on one electronic property: the band gap. It determines whether a material absorbs light, emits light, or stays completely transparent.

In this project, you will build a machine learning pipeline that predicts a material's band gap from its chemical formula alone. You'll train models on thousands of experimentally measured semiconductors and discover which elemental properties drive the predictions.

What You'll Build

A complete materials informatics pipeline in a Jupyter Notebook that converts chemical formulas like "GaAs" or "ZnO" into predicted band gap values using composition-based features and scikit-learn models.

By the end of this project, you'll have:

A trained Random Forest model that predicts experimental band gaps from chemical composition, compared against a Linear Regression baseline.
Scatter plots and performance metrics (R-squared, RMSE) showing how well each model predicts band gaps.
A feature importance chart revealing which elemental properties like electronegativity and atomic radius are most predictive of band gap.
A reusable prediction function that estimates the band gap for any chemical formula you provide. Plus a Secret Mission to test your model on known phosphorescent host materials and analyze what makes them special.

Are there any prerequisites?

You should be comfortable with basic Python syntax (variables, loops, functions) and have a basic familiarity with pandas DataFrames. No prior materials science knowledge is required.

Before We Start

Before We Start...

Set Up Your Environment

Machine learning for materials science requires specialized libraries beyond the standard data science stack. The most important one for this project is matminer, which converts chemical formulas into numerical feature vectors that ML models can process.

In this step, you'll create an isolated Python environment and install all the packages you need. Then you'll verify everything loads correctly inside a Jupyter Notebook.

In this step, get ready to:

Create a conda environment with all required Python packages.
Launch Jupyter Notebook and create your project notebook.
Verify all library imports load without errors.

Create a conda environment and install dependencies

A conda environment keeps this project's packages separate from your other Python work. This prevents version conflicts between different projects.

Press the Windows key to open your search bar.
Type Ubuntu and press Enter to open your WSL2 terminal.
Create a new conda environment with Python 3.11 by running this command:

conda create -n bandgap-ml python=3.11 -y

Activate your new environment by running:

conda activate bandgap-ml

Your terminal prompt now shows (bandgap-ml) at the beginning, confirming you're inside the new environment.

Install all required packages by running this command:

conda install -c conda-forge matminer scikit-learn matplotlib jupyter -y

What does this install?

matminer is a materials informatics library that converts chemical formulas into numerical features. It automatically installs pymatgen and pandas as dependencies.
scikit-learn provides the machine learning models you'll train and evaluate.
matplotlib handles data visualization and plotting.
jupyter lets you run interactive notebooks in your browser.
-c conda-forge pulls packages from the conda-forge channel, which hosts the latest versions of scientific Python packages.

Pro tip: this takes a few minutes

This installation downloads matminer and all its dependencies, which can take 3-5 minutes depending on your internet speed. You'll see conda resolving dependencies and downloading packages throughout.

The installation is complete when you see a list of installed packages and your terminal prompt returns.

Seeing 'conda: command not found'?

Make sure you're running this inside your WSL2 terminal, not PowerShell or Command Prompt.
Confirm Miniconda is installed by checking if the ~/miniconda3 folder exists. Run ls ~/miniconda3 to check.
If conda still isn't recognized, your shell might need to be initialized. Run ~/miniconda3/bin/conda init bash, then close and reopen your terminal.
Help me troubleshoot conda not being found in my WSL2 terminal.

Launch Jupyter Notebook and create a new notebook

Jupyter Notebook runs as a local web server inside your terminal. When you launch it from WSL2, it opens in your Windows browser automatically.

Launch Jupyter Notebook by running this command:

jupyter notebook

A browser window opens showing the Jupyter file browser. If it doesn't open automatically, look for a URL in your terminal output starting with http://localhost:8888 and paste it into your browser.

Click the New button in the top right corner of the Jupyter interface.
Select Python 3 (ipykernel) from the dropdown menu.
Click Untitled at the top of the new notebook to rename it.
Type bandgap_prediction as the new name and click Rename.

You now have a fresh notebook called bandgap_prediction.ipynb ready for your ML pipeline.

Jupyter not opening in your browser?

Copy the full URL from your terminal output (it looks like http://localhost:8888/?token=...) and paste it into your Windows browser.
Make sure no other Jupyter instance is already running on port 8888.
Help me get Jupyter Notebook to open from WSL2.

Verify your imports

Before writing any pipeline code, confirm that all the libraries you installed are accessible from your notebook's kernel.

In the first cell of your notebook, paste the following code:

# Cell 1: Imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from matminer.datasets import load_dataset
from matminer.featurizers.conversions import StrToComposition
from matminer.featurizers.composition.composite import ElementProperty
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score
print("All imports successful!")

What are these libraries?

pandas and numpy handle data manipulation and numerical operations.
matplotlib creates plots and visualizations.
matminer provides the dataset loader and featurizers that convert chemical formulas into numerical features.
sklearn (scikit-learn) provides ML models, train/test splitting, and evaluation metrics.

Run the cell by pressing Shift+Enter.

You should see All imports successful! printed below the cell with no error messages.

Seeing a ModuleNotFoundError?

Your notebook might be using the wrong Python kernel. Click Kernel in the menu bar, then Change Kernel, and select the kernel from your bandgap-ml environment.
If the bandgap-ml kernel doesn't appear in the list, go back to your terminal, activate the environment with conda activate bandgap-ml, and run python -m ipykernel install --user --name bandgap-ml. Then restart Jupyter.
Help me fix ModuleNotFoundError for matminer in Jupyter.

Awesome, I've got everything!

Great. Double check you've saved your notebook by pressing Ctrl+S.

I'd like to double check the full code

Here's the complete code your notebook should contain so far:

# Cell 1: Imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from matminer.datasets import load_dataset
from matminer.featurizers.conversions import StrToComposition
from matminer.featurizers.composition.composite import ElementProperty
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score
print("All imports successful!")

Your environment is ready and all libraries are loaded. Next up, you'll load a real dataset of 4,604 experimentally measured band gaps and explore what it contains.

Load and Explore the Band Gap Dataset

Your environment is set up and all your libraries are imported. Now it is time to get your hands on the actual data.

The dataset you will work with contains 4,604 experimentally measured band gaps of inorganic materials. The band gap is the most fundamental electronic property of a material. It determines whether a material conducts electricity, absorbs light, or emits it. In this step, you will load this dataset and explore what it looks like before feeding it to a machine learning model.

In this step, get ready to:

Load the matbench_expt_gap dataset from matminer.
Visualize the distribution of band gap values with a histogram.
Examine summary statistics to build intuition about the data.

Load the matbench_expt_gap dataset

The MatBench benchmark suite includes several curated datasets for materials science. The one you need is matbench_expt_gap, which contains 4,604 experimentally measured band gaps from the Zhuo et al. 2018 study.

In your Jupyter Notebook, create a new cell below your imports and load the dataset by running this code:

# Load the experimental band gap dataset from MatBench
df = load_dataset("matbench_expt_gap")

# Check the size and column names
print(f"Dataset shape: {df.shape}")
print(f"Columns: {df.columns.tolist()}")

# Preview the first 10 rows
df.head(10)

What does this code do?

load_dataset("matbench_expt_gap") downloads and caches the dataset locally the first time you run it. No API key is needed.
The dataset has two columns: composition (chemical formula strings like "GaAs" or "ZnO") and gap expt (the experimental band gap measured in electron volts).
df.head(10) displays the first 10 rows so you can see what the raw data looks like.

Run the cell by pressing Shift+Enter.

You should see Dataset shape: (4604, 2) printed, followed by a table showing chemical formulas alongside their band gap values.

Dataset not loading?

The first time you call load_dataset, matminer downloads the data from the internet. Make sure your machine has an active internet connection.
If you see an ImportError, go back to your terminal and confirm you activated the bandgap-ml conda environment before launching Jupyter.
If you see a NameError for load_dataset, make sure you ran the imports cell (Cell 1) first.

Help me debug my matminer dataset loading error.

Visualize the distribution of band gaps

Before training any model, you need to understand your data. A histogram reveals how band gap values are spread across the dataset and whether the distribution is balanced or skewed.

In a new cell, plot the distribution of band gaps by running this code:

# Plot a histogram of all band gap values in the dataset
plt.figure(figsize=(8, 5))
plt.hist(df["gap expt"], bins=50, edgecolor="black", alpha=0.7)
plt.xlabel("Experimental Band Gap (eV)")
plt.ylabel("Count")
plt.title("Distribution of Experimental Band Gaps")

# Add reference lines for well-known semiconductors
plt.axvline(x=1.1, color="red", linestyle="--", label="Si (~1.1 eV)")
plt.axvline(x=3.4, color="blue", linestyle="--", label="GaN (~3.4 eV)")
plt.legend()
plt.tight_layout()
plt.show()

What does this plot tell you?

Most materials cluster at lower band gap values, with a long tail toward wide-gap materials. This means the dataset is skewed.
The red dashed line marks silicon (Si) at ~1.1 eV. Silicon is the most widely used semiconductor and sits in the low-gap region.
The blue dashed line marks gallium nitride (GaN) at ~3.4 eV. GaN is a wide-gap material used in blue LEDs.
Materials with small gaps absorb visible or infrared light. Wide-gap materials (above ~3 eV) are transparent to visible light, which is why they make excellent hosts for luminescent dopants in phosphors.

Run the cell by pressing Shift+Enter.

You should see a histogram with most bars clustered on the left side (low band gaps) and a long tail stretching to the right. Two colored dashed lines mark Si and GaN.

Plot not showing up?

Make sure you ran the cell that loads the dataset (Cell 2) before this one. The variable df must exist.
If you see a blank output, check that plt.show() is at the end of the cell. Some Jupyter configurations require it to render inline plots.

Help me debug why my matplotlib plot is not showing in Jupyter.

Examine summary statistics

The histogram gave you a visual feel for the data. Now pull up the numerical summary to see the exact mean, spread, and range of band gap values.

In a new cell, generate summary statistics by running this code:

# Show statistical summary of the band gap values
df.describe()

What should I look for?

The mean tells you the average band gap across all 4,604 materials.
The min and max show the full range. Materials near 0 eV are metals or narrow-gap semiconductors. Materials near the max are wide-gap insulators.
The 50% row is the median. If it is lower than the mean, the distribution is right-skewed, confirming what you saw in the histogram.

Run the cell by pressing Shift+Enter.

You should see a table with count, mean, std, min, 25%, 50%, 75%, and max values for the gap expt column.

Awesome, I've got everything!

Great. Double check you have saved your notebook before moving on.

I'd like to double check the full code

Here is all the code from Cells 1 through 4. Each block below corresponds to a separate Jupyter cell.

# Cell 1: Imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from matminer.datasets import load_dataset
from matminer.featurizers.conversions import StrToComposition
from matminer.featurizers.composition.composite import ElementProperty
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score
print("All imports successful!")

# Cell 2: Load dataset
df = load_dataset("matbench_expt_gap")
print(f"Dataset shape: {df.shape}")
print(f"Columns: {df.columns.tolist()}")
df.head(10)

# Cell 3: Visualize band gap distribution
plt.figure(figsize=(8, 5))
plt.hist(df["gap expt"], bins=50, edgecolor="black", alpha=0.7)
plt.xlabel("Experimental Band Gap (eV)")
plt.ylabel("Count")
plt.title("Distribution of Experimental Band Gaps")
plt.axvline(x=1.1, color="red", linestyle="--", label="Si (~1.1 eV)")
plt.axvline(x=3.4, color="blue", linestyle="--", label="GaN (~3.4 eV)")
plt.legend()
plt.tight_layout()
plt.show()

# Cell 4: Examine statistics
df.describe()

You now have a solid understanding of what the dataset contains and how band gap values are distributed. Next up, you will convert these chemical formula strings into numerical feature vectors that a machine learning model can actually learn from.

Featurize Compositions into Numerical Vectors

You've explored the band gap dataset and seen how 4,604 experimental measurements are distributed across thousands of semiconductors. Now the question is: how do you feed a chemical formula like "GaAs" into a machine learning model?

ML models only understand numbers. You need to convert each composition into a numerical vector that captures the chemistry of its elements, a process called featurization.

In this step, get ready to:

Convert chemical formula strings into structured Composition objects.
Generate 132 numerical features from elemental properties using the Magpie data source.
Clean the data and prepare feature and target arrays for model training.

Convert composition strings to Composition objects

The dataset's composition column contains plain text strings like "GaAs" and "ZnO". Before matminer's featurizers can extract numerical properties, these strings need to be parsed into structured pymatgen Composition objects.

Create a new cell in your notebook and paste the following code:

# Convert strings to Composition objects
stc = StrToComposition()
df = stc.featurize_dataframe(df, "composition")
df.head()

What does this code do?

StrToComposition() creates a featurizer that knows how to parse chemical formula strings.
featurize_dataframe(df, "composition") reads every string in the composition column and adds a new column called composition_obj containing pymatgen Composition objects.
The original composition column (plain strings) is preserved. The next featurizer will use the new composition_obj column.

Run the cell by pressing Shift+Enter.
You should see a table preview with a new composition_obj column on the right side of the DataFrame. The values will look similar to the composition column, but they are now structured pymatgen objects.

Seeing a NameError?

If you see NameError: name 'StrToComposition' is not defined, make sure you ran the imports cell (Cell 1) first. Jupyter cells must be run in order.
If you see NameError: name 'df' is not defined, go back and run the cell that loads the dataset with load_dataset("matbench_expt_gap").

Still stuck? Ask for help.

Generate elemental property features with the Magpie preset

This is where the magic happens. The ElementProperty featurizer takes each Composition object and computes statistics (mean, min, max, range, and more) of elemental properties like electronegativity, atomic radius, and melting point across all the elements in the formula.

Create a new cell and paste the following code to generate all 132 features:

# Generate elemental property features
ep = ElementProperty.from_preset("magpie")
df = ep.featurize_dataframe(df, col_id="composition_obj", ignore_errors=True)
print(f"Number of features generated: {len(ep.feature_labels())}")
print(f"DataFrame shape after featurization: {df.shape}")
df.head()

What does this code do?

ElementProperty.from_preset("magpie") creates a featurizer using the Magpie dataset, which contains 22 elemental properties (like Mendeleev number, melting temperature, electronegativity, valence electron counts).
For each of those 22 properties, the featurizer computes 6 statistics across the elements in the composition: mean, mean absolute deviation, minimum, maximum, range, and average deviation. That gives you 22 x 6 = 132 features.
col_id="composition_obj" tells the featurizer to read from the pymatgen objects column you created in the previous step, not the original string column.
ignore_errors=True lets the featurizer skip any exotic compositions that are missing elemental data instead of crashing.

Why does this representation work?

A formula like "GaAs" becomes a row of 132 numbers capturing the "average chemistry" of gallium and arsenic. The mean electronegativity tells you how strongly the atoms attract electrons. The range of atomic radius tells you how different the atoms are in size.

These properties directly relate to how tightly electrons are bound in a material, which is exactly what determines the band gap.

Run the cell by pressing Shift+Enter. This may take 30-60 seconds as the featurizer processes all 4,604 compositions.
You should see output showing Number of features generated: 132 and a DataFrame shape with 134+ columns. The table preview will now have many new columns with names like MagpieData mean MeltingT and MagpieData range Electronegativity.

Featurization seems stuck or slow?

Processing 4,604 compositions takes time. If the cell shows [*] next to it, the cell is still running. Wait up to 2 minutes before assuming something is wrong.
If you see a KeyError about composition_obj, make sure you ran the StrToComposition cell first. The featurizer needs the pymatgen objects from that step.

Still stuck? Ask for help.

Clean the featurized data and prepare arrays for training

Some exotic compositions may be missing elemental data, leaving NaN values in the feature columns. You also need to separate the features (the 132 numerical columns) from the target (the band gap values) before feeding them into a model.

Create a new cell and paste the following code to clean and prepare your data:

# Clean and prepare data
df = df.dropna()
print(f"Shape after dropping NaN: {df.shape}")

excluded = ["composition", "gap expt", "composition_obj"]
feature_cols = [c for c in df.columns if c not in excluded]
X = df[feature_cols]
y = df["gap expt"]
print(f"Feature matrix shape: {X.shape}")
print(f"Target vector shape: {y.shape}")

What does this code do?

df.dropna() removes any rows that have missing values. This ensures the ML model receives complete data for every sample.
The excluded list names the three columns that are NOT features: the original text formula, the target band gap value, and the pymatgen object. Everything else is a numerical feature.
X is your feature matrix (rows of compositions, columns of elemental property statistics). y is your target vector (the experimental band gap in eV for each composition).

Run the cell by pressing Shift+Enter.
You should see the shape after dropping NaN values (close to the original 4,604 rows), a feature matrix shape like (~4600, 132), and a target vector shape matching the number of rows.

Feature matrix has 0 columns or wrong shape?

If X.shape shows 0 columns, make sure you ran the ElementProperty featurization cell before this one. The 132 feature columns must already exist in the DataFrame.
If you see 0 rows after dropna(), something went wrong during featurization. Go back to the ElementProperty cell and re-run it. Check that the DataFrame has numerical values in the new columns, not all NaN.

Still stuck? Ask for help.

Awesome, I've got everything!

Great. Double check you've saved your notebook before moving on.

I'd like to double check the full code

Here is all the code from Cell 1 through Cell 7. Each cell should be in its own notebook cell.

# Cell 1: Imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from matminer.datasets import load_dataset
from matminer.featurizers.conversions import StrToComposition
from matminer.featurizers.composition.composite import ElementProperty
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score
print("All imports successful!")

# Cell 2: Load dataset
df = load_dataset("matbench_expt_gap")
print(f"Dataset shape: {df.shape}")
print(f"Columns: {df.columns.tolist()}")
df.head(10)

# Cell 3: Visualize band gap distribution
plt.figure(figsize=(8, 5))
plt.hist(df["gap expt"], bins=50, edgecolor="black", alpha=0.7)
plt.xlabel("Experimental Band Gap (eV)")
plt.ylabel("Count")
plt.title("Distribution of Experimental Band Gaps")
plt.axvline(x=1.1, color="red", linestyle="--", label="Si (~1.1 eV)")
plt.axvline(x=3.4, color="blue", linestyle="--", label="GaN (~3.4 eV)")
plt.legend()
plt.tight_layout()
plt.show()

# Cell 4: Examine statistics
df.describe()

# Cell 5: Convert strings to Composition objects
stc = StrToComposition()
df = stc.featurize_dataframe(df, "composition")
df.head()

# Cell 6: Generate elemental property features
ep = ElementProperty.from_preset("magpie")
df = ep.featurize_dataframe(df, col_id="composition_obj", ignore_errors=True)
print(f"Number of features generated: {len(ep.feature_labels())}")
print(f"DataFrame shape after featurization: {df.shape}")
df.head()

# Cell 7: Clean and prepare data
df = df.dropna()
print(f"Shape after dropping NaN: {df.shape}")

excluded = ["composition", "gap expt", "composition_obj"]
feature_cols = [c for c in df.columns if c not in excluded]
X = df[feature_cols]
y = df["gap expt"]
print(f"Feature matrix shape: {X.shape}")
print(f"Target vector shape: {y.shape}")

Your compositions are now 132-dimensional numerical vectors. Next up, you'll feed these vectors into two machine learning models and see which one predicts band gaps more accurately.

Train and Compare ML Models

Your dataset is now featurized. Each material's chemical formula has been transformed into 132 numerical features capturing the statistics of elemental properties like electronegativity and atomic radius.

The big question is whether these features contain enough information to predict band gaps. You will train two machine learning models to find out and compare their predictions against experimental measurements.

In this step, get ready to:

Split your data into training and test sets.
Train a Linear Regression baseline model.
Train a Random Forest model and compare performance.

Split data into training and test sets

Before training any model, you need to set aside a portion of your data for testing. This ensures the model is evaluated on data it has never seen during training.

Create an 80/20 train/test split by adding this code to a new cell in your notebook:

# Cell 8: Train/test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")

What does this code do?

train_test_split randomly shuffles your data and divides it into a training set (80%) and a test set (20%).
random_state=42 fixes the random shuffle so you get the exact same split every time you run the cell. This makes your results reproducible.
The model learns patterns from the training set. The test set acts as unseen data to measure how well those patterns generalize.

Run the cell by pressing Shift+Enter.
You should see output showing roughly 3,600 training samples and 900 test samples.

Seeing a NameError for X or y?

These variables are created in the previous step's cleaning cell (Cell 7). Make sure you ran all cells from earlier steps before running this one.

In the menu bar, click Cell, then Run All Above to re-run every cell before this one.

Help me fix a NameError in my Jupyter Notebook

Train a Linear Regression baseline

Linear Regression assumes the band gap is a weighted sum of all 132 input features. It is the simplest model you can train, which makes it a perfect baseline to measure more complex models against.

Train a Linear Regression model and evaluate it by adding this code to a new cell:

# Cell 9: Train Linear Regression
lr = LinearRegression()
lr.fit(X_train, y_train)
y_pred_lr = lr.predict(X_test)

# Evaluate using R-squared and RMSE
r2_lr = r2_score(y_test, y_pred_lr)
rmse_lr = np.sqrt(mean_squared_error(y_test, y_pred_lr))
print(f"Linear Regression - R\u00b2: {r2_lr:.3f}, RMSE: {rmse_lr:.3f} eV")

What does this code do?

lr.fit(X_train, y_train) trains the model by finding the best weight for each of the 132 features to predict band gap.
r2_score calculates R-squared, which measures how well the model explains the variance in band gaps. A value of 1.0 means perfect predictions. A value of 0.0 means the model is no better than always guessing the average band gap.
RMSE (root mean squared error) tells you the average prediction error in electron volts. Lower is better.

Run the cell by pressing Shift+Enter.
You should see the R-squared value printed, likely somewhere in the 0.6 to 0.7 range, along with the RMSE in eV.

Seeing a very low or negative R-squared?

A negative R-squared means the model performs worse than always predicting the mean. Double check that your X_train and y_train variables are correctly defined from Cell 7 and Cell 8.

Also check that feature_cols excludes the composition, gap expt, and composition_obj columns. Including the target column in the features would give misleadingly perfect results.

Help me debug my Linear Regression results

Train a Random Forest model and compare

Random Forest is an ensemble method that builds many individual decision trees and averages their predictions. Unlike Linear Regression, it can capture nonlinear relationships between features and band gap.

Train a Random Forest model and compare it to your baseline by adding this code to a new cell:

# Cell 10: Train Random Forest
rf = RandomForestRegressor(n_estimators=100, random_state=42, n_jobs=-1)
rf.fit(X_train, y_train)
y_pred_rf = rf.predict(X_test)

# Evaluate and compare to Linear Regression
r2_rf = r2_score(y_test, y_pred_rf)
rmse_rf = np.sqrt(mean_squared_error(y_test, y_pred_rf))
print(f"Random Forest    - R\u00b2: {r2_rf:.3f}, RMSE: {rmse_rf:.3f} eV")
print(f"\nImprovement in R\u00b2: {r2_rf - r2_lr:.3f}")

What does this code do?

n_estimators=100 means the forest contains 100 individual decision trees. Each tree sees a random subset of the data and features, then the final prediction is the average of all 100 trees.
n_jobs=-1 uses all available CPU cores to train trees in parallel. This speeds up training significantly.
The improvement line at the end directly shows how much the Random Forest outperforms the linear baseline. A positive improvement confirms that band gap has nonlinear relationships with composition features.

Run the cell by pressing Shift+Enter.
You should see the Random Forest R-squared is noticeably higher than Linear Regression, likely in the 0.7 to 0.85 range. The RMSE should be lower, meaning smaller prediction errors.

Training taking a long time?

Random Forest with 100 trees on roughly 3,600 samples typically finishes in under 30 seconds. If it takes much longer, check that n_jobs=-1 is set correctly. This enables parallel training across all CPU cores.

My Random Forest model is taking too long to train

Why does the Random Forest win?

Band gap depends on composition in complex, nonlinear ways. For example, adding a small amount of nitrogen to GaAs dramatically changes its band gap. Linear Regression can only model straight-line relationships and misses these interactions.

Random Forest handles this by splitting the feature space into regions with different prediction rules. Each tree captures a different slice of these nonlinear patterns.

Awesome, I've got everything!

Great. Double check you have saved your notebook before moving on.

I'd like to double check the full code

Here is all the code in your notebook up to and including this step. Each comment marks the start of a new cell.

# Cell 1: Imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from matminer.datasets import load_dataset
from matminer.featurizers.conversions import StrToComposition
from matminer.featurizers.composition.composite import ElementProperty
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score
print("All imports successful!")

# Cell 2: Load dataset
df = load_dataset("matbench_expt_gap")
print(f"Dataset shape: {df.shape}")
print(f"Columns: {df.columns.tolist()}")
df.head(10)

# Cell 3: Visualize band gap distribution
plt.figure(figsize=(8, 5))
plt.hist(df["gap expt"], bins=50, edgecolor="black", alpha=0.7)
plt.xlabel("Experimental Band Gap (eV)")
plt.ylabel("Count")
plt.title("Distribution of Experimental Band Gaps")
plt.axvline(x=1.1, color="red", linestyle="--", label="Si (~1.1 eV)")
plt.axvline(x=3.4, color="blue", linestyle="--", label="GaN (~3.4 eV)")
plt.legend()
plt.tight_layout()
plt.show()

# Cell 4: Examine statistics
df.describe()

# Cell 5: Convert strings to Composition objects
stc = StrToComposition()
df = stc.featurize_dataframe(df, "composition")
df.head()

# Cell 6: Generate elemental property features
ep = ElementProperty.from_preset("magpie")
df = ep.featurize_dataframe(df, col_id="composition_obj", ignore_errors=True)
print(f"Number of features generated: {len(ep.feature_labels())}")
print(f"DataFrame shape after featurization: {df.shape}")
df.head()

# Cell 7: Clean and prepare data
df = df.dropna()
print(f"Shape after dropping NaN: {df.shape}")

excluded = ["composition", "gap expt", "composition_obj"]
feature_cols = [c for c in df.columns if c not in excluded]
X = df[feature_cols]
y = df["gap expt"]
print(f"Feature matrix shape: {X.shape}")
print(f"Target vector shape: {y.shape}")

# Cell 8: Train/test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")

# Cell 9: Train Linear Regression
lr = LinearRegression()
lr.fit(X_train, y_train)
y_pred_lr = lr.predict(X_test)

r2_lr = r2_score(y_test, y_pred_lr)
rmse_lr = np.sqrt(mean_squared_error(y_test, y_pred_lr))
print(f"Linear Regression - R\u00b2: {r2_lr:.3f}, RMSE: {rmse_lr:.3f} eV")

# Cell 10: Train Random Forest
rf = RandomForestRegressor(n_estimators=100, random_state=42, n_jobs=-1)
rf.fit(X_train, y_train)
y_pred_rf = rf.predict(X_test)

r2_rf = r2_score(y_test, y_pred_rf)
rmse_rf = np.sqrt(mean_squared_error(y_test, y_pred_rf))
print(f"Random Forest    - R\u00b2: {r2_rf:.3f}, RMSE: {rmse_rf:.3f} eV")
print(f"\nImprovement in R\u00b2: {r2_rf - r2_lr:.3f}")

Both models are trained and you have concrete performance numbers to compare. Next up, you will visualize these predictions with scatter plots and discover which elemental properties matter most for predicting band gaps.

Visualize Results and Interpret Feature Importance

Your models are trained and you have R-squared and RMSE numbers for each one. Numbers are useful, but they don't show you where predictions break down or which compositions are hardest to get right.

In this step, you will create visualizations that reveal the full picture. Scatter plots will show where each model struggles, and a feature importance chart will tell you which elemental properties drive the Random Forest's predictions.

In this step, get ready to:

Create predicted vs. actual scatter plots for both models.
Plot the top 15 most important features from the Random Forest.
Build a function to predict band gaps for any new chemical formula.

Create predicted vs. actual scatter plots

A predicted vs. actual scatter plot is the standard way to evaluate regression models visually. If the model were perfect, every point would land exactly on the diagonal line.

Create a new cell in your notebook and generate side-by-side scatter plots for both models by pasting this code:

# Create a figure with two side-by-side subplots
fig, axes = plt.subplots(1, 2, figsize=(14, 6))

# Left plot: Linear Regression predictions vs actual values
axes[0].scatter(y_test, y_pred_lr, alpha=0.3, s=10)
axes[0].plot([0, 10], [0, 10], "r--", linewidth=1)  # Perfect prediction line
axes[0].set_xlabel("Actual Band Gap (eV)")
axes[0].set_ylabel("Predicted Band Gap (eV)")
axes[0].set_title(f"Linear Regression (R²={r2_lr:.3f})")
axes[0].set_xlim(0, 10)
axes[0].set_ylim(0, 10)

# Right plot: Random Forest predictions vs actual values
axes[1].scatter(y_test, y_pred_rf, alpha=0.3, s=10)
axes[1].plot([0, 10], [0, 10], "r--", linewidth=1)  # Perfect prediction line
axes[1].set_xlabel("Actual Band Gap (eV)")
axes[1].set_ylabel("Predicted Band Gap (eV)")
axes[1].set_title(f"Random Forest (R²={r2_rf:.3f})")
axes[1].set_xlim(0, 10)
axes[1].set_ylim(0, 10)

plt.tight_layout()
plt.show()

What does this code do?

plt.subplots(1, 2) creates two plots side by side so you can visually compare the models.
Each scatter point represents one material from the test set. Its x-position is the real band gap and its y-position is what the model predicted.
The red dashed line marks perfect prediction. Points close to this line are accurate predictions. Points far from it are errors.
alpha=0.3 makes points semi-transparent so you can see where predictions cluster.

Run the cell.
You should see two scatter plots appear below the cell. The Linear Regression plot on the left will show points scattered more widely away from the diagonal. The Random Forest plot on the right will show points clustering much more tightly around the red dashed line.

How do I read these plots?

Look at how the cloud of points spreads around the diagonal. For the Random Forest, you will likely see tighter clustering at low band gaps (0-3 eV) and more scatter at higher values. This tells you the model is best at predicting common semiconductors and less confident about rare wide-gap materials.

Not seeing any plots?

Make sure you ran all cells in order from the top of the notebook. The variables y_test, y_pred_lr, y_pred_rf, r2_lr, and r2_rf must exist from the previous step's cells.
If you see a NameError, go back to the cells in Step 4 and re-run them before running this cell.
Ask for help: My scatter plots are not rendering in Jupyter Notebook. I'm using matplotlib and plt.show() but nothing appears.

Plot the top 15 most important features

The Random Forest doesn't just make predictions. It also tracks how much each feature contributed to those predictions. This is called feature importance.

Create a new cell and generate a feature importance bar chart by pasting this code:

# Extract feature importances from the trained Random Forest
importances = rf.feature_importances_
feat_imp = pd.Series(importances, index=feature_cols).sort_values(ascending=True)
top_15 = feat_imp.tail(15)  # Select the 15 most important features

# Plot as a horizontal bar chart
plt.figure(figsize=(10, 7))
top_15.plot(kind="barh")
plt.xlabel("Feature Importance")
plt.title("Top 15 Features for Band Gap Prediction")
plt.tight_layout()
plt.show()

What does this code do?

rf.feature_importances_ returns an array where each value represents how much a feature reduced prediction error across all 100 trees.
The code wraps these values in a pandas Series with the feature names as labels, sorts them, and takes the top 15.
A horizontal bar chart makes long feature names easy to read.

Run the cell.
You should see a horizontal bar chart with 15 labeled bars. Features related to electronegativity, atomic number, and atomic radius will likely appear near the top.

What do these features mean chemically?

The Magpie feature names follow a pattern: a statistic (mean, range, maximum) applied to an elemental property (Electronegativity, Number, MeltingT). Here is what the top features typically tell you.

Electronegativity features capture how strongly atoms attract electrons. Materials with highly electronegative elements (like O or F) bonded to electropositive metals (like Sr or Ba) tend to have wide band gaps.
Atomic number features reflect the size and complexity of the atoms. The range of atomic numbers in a composition indicates how chemically diverse it is.
Atomic radius features relate to bond lengths. Shorter bonds generally mean stronger interactions and wider band gaps.

These connections make physical sense. The band gap is determined by how tightly electrons are bound in the material, and electronegativity and atomic size directly govern that.

Feature names look garbled or cut off?

The plt.tight_layout() call should prevent labels from being clipped. If they are still cut off, try increasing the figure width in figsize=(10, 7) to something like figsize=(12, 8).
Ask for help: My feature importance bar chart has labels that are cut off or overlapping.

Make predictions on new compositions

Your model currently only works on the test set data that was already prepared and featurized. To predict the band gap of any new material, you need a function that takes a raw chemical formula, featurizes it, and runs it through the trained model.

Create a new cell and build a prediction function by pasting this code:

from pymatgen.core import Composition

# Define a function that takes a formula string and returns a predicted band gap
def predict_band_gap(formula):
    comp = Composition(formula)  # Parse the formula string
    features = ep.featurize(comp)  # Generate Magpie features for this composition
    feature_df = pd.DataFrame([features], columns=ep.feature_labels())
    prediction = rf.predict(feature_df)[0]  # Predict using the trained Random Forest
    return prediction

# Test on well-known semiconductors
test_materials = ["Si", "GaAs", "ZnO", "TiO2", "GaN", "CdS"]
for mat in test_materials:
    pred = predict_band_gap(mat)
    print(f"{mat}: Predicted band gap = {pred:.2f} eV")

What does this code do?

Composition(formula) from pymatgen parses a string like "GaAs" into a structured composition object with element ratios.
ep.featurize(comp) reuses the same Magpie featurizer from Step 3 to generate the 132 numerical features for this single composition.
The features are wrapped in a DataFrame so rf.predict() receives the same format it was trained on.
The test list includes materials with band gaps spanning from ~1.1 eV (Si) to ~3.4 eV (GaN), giving you a range to evaluate.

Run the cell.
You should see a printed prediction for each material. Compare these to known experimental values: Si (~1.1 eV), GaAs (~1.4 eV), ZnO (~3.3 eV), TiO2 (~3.2 eV), GaN (~3.4 eV), CdS (~2.4 eV).

How close should the predictions be?

With an RMSE of roughly 0.5-0.7 eV, your predictions should be in the right ballpark but not exact. If Si predicts around 1.0-1.5 eV and GaN predicts around 2.8-3.8 eV, your model is working as expected. Perfect accuracy is not the goal here. The model captures the overall trend from composition alone, which is remarkable given that it never sees crystal structure data.

Getting a ValueError or KeyError?

Make sure the ep variable still exists. It was created in Step 3. If your kernel restarted, re-run all cells from the top.
Check that your formula strings are valid chemical formulas. pymatgen expects standard notation like "ZnO" or "TiO2".
Ask for help: My predict_band_gap function throws an error when I try to predict a new material.

Try changing the formulas in the test_materials list to materials you are curious about. Any valid chemical formula will work.

Awesome, I've got everything!

Great. Double check you have saved your notebook.

I'd like to double check the full code

Here is the complete notebook code from Cell 1 through Cell 13. Compare each cell in your notebook against this reference.

# Cell 1: Imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from matminer.datasets import load_dataset
from matminer.featurizers.conversions import StrToComposition
from matminer.featurizers.composition.composite import ElementProperty
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score
print("All imports successful!")

# Cell 2: Load dataset
df = load_dataset("matbench_expt_gap")
print(f"Dataset shape: {df.shape}")
print(f"Columns: {df.columns.tolist()}")
df.head(10)

# Cell 3: Visualize band gap distribution
plt.figure(figsize=(8, 5))
plt.hist(df["gap expt"], bins=50, edgecolor="black", alpha=0.7)
plt.xlabel("Experimental Band Gap (eV)")
plt.ylabel("Count")
plt.title("Distribution of Experimental Band Gaps")
plt.axvline(x=1.1, color="red", linestyle="--", label="Si (~1.1 eV)")
plt.axvline(x=3.4, color="blue", linestyle="--", label="GaN (~3.4 eV)")
plt.legend()
plt.tight_layout()
plt.show()

# Cell 4: Examine statistics
df.describe()

# Cell 5: Convert strings to Composition objects
stc = StrToComposition()
df = stc.featurize_dataframe(df, "composition")
df.head()

# Cell 6: Generate elemental property features
ep = ElementProperty.from_preset("magpie")
df = ep.featurize_dataframe(df, col_id="composition_obj", ignore_errors=True)
print(f"Number of features generated: {len(ep.feature_labels())}")
print(f"DataFrame shape after featurization: {df.shape}")
df.head()

# Cell 7: Clean and prepare data
df = df.dropna()
print(f"Shape after dropping NaN: {df.shape}")

excluded = ["composition", "gap expt", "composition_obj"]
feature_cols = [c for c in df.columns if c not in excluded]
X = df[feature_cols]
y = df["gap expt"]
print(f"Feature matrix shape: {X.shape}")
print(f"Target vector shape: {y.shape}")

# Cell 8: Train/test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")

# Cell 9: Train Linear Regression
lr = LinearRegression()
lr.fit(X_train, y_train)
y_pred_lr = lr.predict(X_test)

r2_lr = r2_score(y_test, y_pred_lr)
rmse_lr = np.sqrt(mean_squared_error(y_test, y_pred_lr))
print(f"Linear Regression - R²: {r2_lr:.3f}, RMSE: {rmse_lr:.3f} eV")

# Cell 10: Train Random Forest
rf = RandomForestRegressor(n_estimators=100, random_state=42, n_jobs=-1)
rf.fit(X_train, y_train)
y_pred_rf = rf.predict(X_test)

r2_rf = r2_score(y_test, y_pred_rf)
rmse_rf = np.sqrt(mean_squared_error(y_test, y_pred_rf))
print(f"Random Forest    - R²: {r2_rf:.3f}, RMSE: {rmse_rf:.3f} eV")
print(f"\nImprovement in R²: {r2_rf - r2_lr:.3f}")

# Cell 11: Predicted vs actual scatter plots
fig, axes = plt.subplots(1, 2, figsize=(14, 6))

axes[0].scatter(y_test, y_pred_lr, alpha=0.3, s=10)
axes[0].plot([0, 10], [0, 10], "r--", linewidth=1)
axes[0].set_xlabel("Actual Band Gap (eV)")
axes[0].set_ylabel("Predicted Band Gap (eV)")
axes[0].set_title(f"Linear Regression (R²={r2_lr:.3f})")
axes[0].set_xlim(0, 10)
axes[0].set_ylim(0, 10)

axes[1].scatter(y_test, y_pred_rf, alpha=0.3, s=10)
axes[1].plot([0, 10], [0, 10], "r--", linewidth=1)
axes[1].set_xlabel("Actual Band Gap (eV)")
axes[1].set_ylabel("Predicted Band Gap (eV)")
axes[1].set_title(f"Random Forest (R²={r2_rf:.3f})")
axes[1].set_xlim(0, 10)
axes[1].set_ylim(0, 10)

plt.tight_layout()
plt.show()

# Cell 12: Feature importance
importances = rf.feature_importances_
feat_imp = pd.Series(importances, index=feature_cols).sort_values(ascending=True)
top_15 = feat_imp.tail(15)

plt.figure(figsize=(10, 7))
top_15.plot(kind="barh")
plt.xlabel("Feature Importance")
plt.title("Top 15 Features for Band Gap Prediction")
plt.tight_layout()
plt.show()

# Cell 13: Predict new materials
from pymatgen.core import Composition

def predict_band_gap(formula):
    comp = Composition(formula)
    features = ep.featurize(comp)
    feature_df = pd.DataFrame([features], columns=ep.feature_labels())
    prediction = rf.predict(feature_df)[0]
    return prediction

test_materials = ["Si", "GaAs", "ZnO", "TiO2", "GaN", "CdS"]
for mat in test_materials:
    pred = predict_band_gap(mat)
    print(f"{mat}: Predicted band gap = {pred:.2f} eV")

Secret mission

Investigate Phosphorescent Host Materials

Your model can predict band gaps for any chemical formula. But how does it perform on materials specifically designed to glow in the dark? Test your trained Random Forest on real phosphorescent host materials and discover why their chemistry clusters in a surprising region of band gap space.

Before you go

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs.

Resources you used:

Conda environment bandgap-ml with Python 3.11, matminer, scikit-learn, matplotlib, and Jupyter.
Jupyter Notebook file bandgap_prediction.ipynb.
Cached matbench dataset downloaded by matminer on first load.

Keep everything running

No action needed. Choose this if you're still actively building or plan to extend your notebook with the Secret Mission.

Your bandgap-ml conda environment and all installed packages will remain available.
Your notebook and cached dataset will stay on disk.
To resume, open your terminal, run conda activate bandgap-ml, and launch jupyter notebook.

Pause - I'll come back to this later

Shut down the Jupyter Notebook server to free up memory. Your files and conda environment will be preserved for when you return.

In the terminal where Jupyter is running, press Ctrl + C twice to stop the server.
Your notebook file and the cached dataset will remain on disk.
To resume later, activate the environment and relaunch Jupyter.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes the conda environment, your notebook, and the cached dataset.

Stop the Jupyter server by pressing Ctrl + C twice in the terminal.
Deactivate the conda environment.

conda deactivate

Remove the conda environment entirely.

conda remove -n bandgap-ml --all -y

Delete your notebook file.

rm bandgap_prediction.ipynb

Optionally, clear matminer's cached datasets to reclaim disk space.

rm -rf ~/.matminer

Mission accomplished

That's a Wrap!

Nice work! You just built a complete materials informatics pipeline that predicts band gaps from nothing more than a chemical formula.

You learned how to:

Convert chemical formulas into numerical feature vectors using matminer's Magpie elemental property descriptors, turning compositions like "GaAs" into 132 measurable quantities.
Train and compare Linear Regression and Random Forest models on 4,604 experimentally measured band gaps from the MatBench benchmark suite.
Interpret feature importance to understand which elemental properties (electronegativity, atomic radius, atomic number) are most predictive of a material's band gap.
Investigated why phosphorescent host materials cluster in the wide-gap region and connected your model's feature importances to the chemistry of luminescence.

Ready to quiz yourself?