Energy Demand Prediction in Databricks

Build a Random Forest model to predict city power consumption in Databricks.

Introduction

30 Second Summary

Every time you flip a light switch or crank up the air conditioning, your local utility company is balancing the power grid in real time. Predicting when thousands of people will demand more electricity is one of the biggest challenges in the energy industry.

In this project, you will build a machine learning model in Databricks that predicts power consumption for a city's electrical distribution network using weather and time-based features. You will explore the data with SQL, train a Random Forest model in Python, and translate your findings into business recommendations for demand response planning.

What You'll Build

A complete data science pipeline in Databricks that predicts power consumption for a city's electrical distribution network using Random Forest regression, SQL exploration, and Python visualization.

By the end of this project, you'll have:

  • A Databricks notebook combining SQL queries and Python ML code that explores 52,417 energy consumption records.
  • A trained Random Forest model with R-squared evaluation and a feature importance visualization showing which factors drive energy demand.
  • A business recommendation report identifying optimal demand response triggers backed by data.
  • Secret Mission: Build a peak demand classifier that labels hours as "peak" or "off-peak" and evaluate it with precision and recall metrics.

Are there any prerequisites?

You should have basic familiarity with Python and SQL syntax. No prior machine learning experience is required. Everything runs in Databricks Free Edition (no credit card needed), so you only need a web browser to get started.

Before We Start

Before We Start...

Set Up Databricks and Load the Dataset

Before you can build a prediction model, you need a workspace and data to work with. Databricks Free Edition gives you a cloud-based notebook environment where you can write both SQL and Python in the same project.

In this step, you will create your Databricks account, download a real-world energy consumption dataset, and upload it so it is ready for analysis.

In this step, get ready to:
  • Sign up for Databricks Free Edition.
  • Download the Power Consumption of Tetouan City dataset from the UCI Machine Learning Repository.
  • Upload the dataset and create a table in Databricks.
Sign up for Databricks Free Edition

Databricks Free Edition is a perpetually free, no-credit-card-required workspace designed for learning and experimentation. It includes serverless compute and default storage so you can start immediately.

  • Open your browser and navigate to the Databricks Free Edition signup page:
  • https://www.databricks.com/try-databricks/free
  • Choose your preferred signup method: Sign in with Google, Sign in with Microsoft, or email OTP (one-time password).
  • Select Free Edition if prompted to choose between Free Edition and a 14-day trial.
  • Wait for Databricks to create your workspace. This takes about 30 seconds.

Once your workspace loads, you should see the Databricks home page with a sidebar menu on the left.

Why Databricks Free Edition?

The Free Edition is designed for students, educators, and hobbyists who want to learn data science and AI. It is separate from the 14-day trial, which is meant for business evaluation.

Free Edition does not expire and does not require a credit card. It has daily usage quotas, but they are more than sufficient for this project.

Not seeing the Free Edition option?

If the signup page only shows a 14-day trial, make sure you are on the correct signup URL above. You can also try signing up with a personal email address (Gmail, Outlook) rather than a work email, which defaults to the business trial.

help me find the Databricks Free Edition signup page

Download the energy consumption dataset

The dataset you will use is called "Power Consumption of Tetouan City" from the UCI Machine Learning Repository. It contains 52,417 records of power consumption measurements taken every 10 minutes across three electrical distribution zones in Tetouan, Morocco.

  • Open the dataset page in a new browser tab:
  • https://archive.ics.uci.edu/dataset/849/power+consumption+of+tetouan+city
  • Click the Download button on the dataset page. This downloads a zip file (about 1.3 MB).
  • Extract the zip file on your computer.
  • Locate the extracted file called Tetuan City power consumption.csv. This is the 4 MB CSV file you will upload to Databricks.

What is in this dataset?

Each row represents a 10-minute snapshot of weather and power consumption data. The columns include DateTime, Temperature, Humidity, Wind Speed, general diffuse flows, diffuse flows, and power consumption for three distribution zones.

This is real utility data with no missing values, making it ideal for building a prediction model without complex data cleaning.

Having trouble downloading?

Make sure you are downloading from the UCI repository link above (dataset ID 849). If your browser blocks the download, try right-clicking the Download button and selecting Save link as.

help me download the Tetouan City power consumption dataset from UCI

Upload the CSV and create a table in Databricks

Now you will load the CSV into Databricks so it becomes a queryable table. This mirrors how production data is ingested into a lakehouse platform.

  • In your Databricks workspace, click New in the left sidebar.
  • Select Add data from the dropdown menu.
  • Drag and drop the Tetuan City power consumption.csv file into the upload area, or click to browse and select it from your computer.
  • When prompted for a table name, enter power_consumption in the table name field.
  • Leave the catalog as default and the schema as default.
  • Review the auto-detected column types. You should see columns like DateTime, Temperature, Humidity, Wind Speed, general diffuse flows, diffuse flows, Zone 1 Power Consumption, Zone 2 Power Consumption, and Zone 3 Power Consumption.
  • Click Create table to finish the upload.

After a few seconds, Databricks confirms the table was created successfully. You now have a table called power_consumption containing all 52,417 records.

Why upload instead of using an API?

Databricks Free Edition restricts outbound internet access to a limited set of trusted domains. This means you cannot use pip install to fetch external packages or call external APIs from within a notebook.

Uploading data through the UI is also how production teams load data into a lakehouse. Data lives inside the platform, not pulled from external sources at runtime.

Table creation failed?

If you see an error during upload, check that the CSV file name matches Tetuan City power consumption.csv. Some zip extractors create a subfolder. Make sure you are uploading the CSV itself, not the zip or a folder.

Also verify the table name is power_consumption (all lowercase with an underscore, no spaces).

help me fix my Databricks table upload error

Finally, create a new notebook where you will write all your SQL queries and Python code for the rest of the project.

  • Click New in the left sidebar again.
  • Select Notebook.
  • Databricks creates a blank notebook with Python as the default language.
  • Rename the notebook by clicking the title at the top and typing Energy Demand Prediction.

You should see an empty notebook with one blank cell ready for input. The notebook language in the top right should say Python.

Serverless compute starts automatically

In Databricks Free Edition, serverless compute attaches to your notebook automatically when you run your first cell. You do not need to configure or start a cluster manually.

Your workspace is ready and your data is loaded. Next up, you will write SQL queries to explore consumption patterns and uncover the relationship between weather and energy demand.

Explore the Data with SQL

Your dataset is loaded into Databricks, but before building a machine learning model, you need to understand what the data actually looks like. This is the exploratory data analysis (EDA) phase that every data scientist completes before modeling.

In this step, you will use SQL directly inside your Databricks notebook to query the power_consumption table. You will uncover how consumption varies by time of day, temperature, and zone. These patterns will inform the features you engineer for your prediction model in the next step.

In this step, get ready to:
  • Query the table structure, row count, and date range.
  • Identify peak consumption hours and top demand records.
  • Analyze the relationship between temperature ranges and power consumption.
Inspect the table structure and basic statistics

Databricks notebooks let you mix Python and SQL cells in a single notebook. To write SQL, you type %sql at the top of a cell. This tells Databricks to interpret everything below as a SQL query instead of Python.

  • In your notebook, click the + Code button below the existing empty cell to add a new cell.
  • Type the following SQL query into the cell to check the dataset size and date range:
%sql
SELECT COUNT(*) as total_records,
       MIN(DateTime) as start_date,
       MAX(DateTime) as end_date
FROM default.power_consumption;

What does this query do?

  • COUNT(*) counts every row in the table, telling you how many 10-minute intervals were recorded.
  • MIN(DateTime) and MAX(DateTime) reveal the time range covered by the dataset.
  • Click the Run cell button (the play icon at the top-right of the cell) or press Shift+Enter to execute the query.
  • You should see a results table showing 52,417 total records with a date range spanning 2017.

Seeing 'Table or view not found'?

  • Make sure the table name is exactly default.power_consumption. If you used a different name during upload, replace it in the query.
  • Check that you typed %sql on the very first line of the cell. Without this, Databricks tries to run the SQL as Python and throws a syntax error.
  • Help me fix my SQL query in Databricks

Now you know the dataset is complete. Next, pull summary statistics for the key weather and consumption columns.

  • Add a new cell below by clicking + Code.
  • Type this query to calculate averages and ranges for temperature and consumption across all three zones:
%sql
SELECT
  ROUND(AVG(Temperature), 2) as avg_temp,
  ROUND(MIN(Temperature), 2) as min_temp,
  ROUND(MAX(Temperature), 2) as max_temp,
  ROUND(AVG(`Zone 1 Power Consumption`), 2) as avg_zone1,
  ROUND(AVG(`Zone 2 Power Consumption`), 2) as avg_zone2,
  ROUND(AVG(`Zone 3 Power Consumption`), 2) as avg_zone3
FROM default.power_consumption;

What does this query do?

  • AVG, MIN, and MAX on Temperature show the weather range covered in the data.
  • Comparing avg_zone1, avg_zone2, and avg_zone3 reveals which distribution zone draws the most power on average.
  • Backticks around column names with spaces (like `Zone 1 Power Consumption`) are required in Databricks SQL.
  • Run the cell with Shift+Enter.
  • You should see a single row of results with temperature ranging roughly from 3 to 40 degrees, and three average consumption values for each zone.

Getting an 'AnalysisException' error?

  • Check that column names with spaces are wrapped in backticks, not single quotes. For example: `Zone 1 Power Consumption` not 'Zone 1 Power Consumption'.
  • Help me fix my column name references in Databricks SQL
Query consumption patterns by time

Utility companies plan demand response events around peak hours. You need to find which hours of the day see the highest consumption so your model can leverage this pattern.

  • Add a new cell and type this query to calculate average consumption for each hour of the day:
%sql
SELECT
  HOUR(DateTime) as hour_of_day,
  ROUND(AVG(`Zone 1 Power Consumption`), 2) as avg_consumption
FROM default.power_consumption
GROUP BY HOUR(DateTime)
ORDER BY hour_of_day;

What does this query do?

  • HOUR(DateTime) extracts just the hour (0-23) from each timestamp.
  • GROUP BY collapses all records for the same hour into one row, and AVG computes the mean consumption for that hour.
  • The result is a 24-row table showing the typical consumption profile across the day.
  • Run the cell with Shift+Enter.
  • You should see 24 rows (one per hour). Notice that consumption tends to be lower in early morning hours and higher during midday and evening.

Now find the individual records with the highest consumption to see what conditions produce extreme demand.

  • Add a new cell and type this query to find the top 10 peak consumption moments:
%sql
SELECT DateTime, Temperature, Humidity, `Wind Speed`,
       `Zone 1 Power Consumption` as zone1_consumption
FROM default.power_consumption
ORDER BY `Zone 1 Power Consumption` DESC
LIMIT 10;

What does this query do?

  • ORDER BY ... DESC sorts all 52,417 records from highest to lowest consumption.
  • LIMIT 10 returns only the top 10. Looking at the temperature and time of these extreme records reveals what conditions trigger peak demand.
  • Run the cell with Shift+Enter.
  • You should see 10 rows showing the highest consumption moments. Notice the temperature values for these peak records.

Results look unexpected or empty?

  • If you see 0 rows, double-check that the table name matches exactly: default.power_consumption.
  • If the consumption values look odd (e.g., negative or unreasonably large), verify you uploaded the correct CSV file from UCI dataset 849.
  • Help me troubleshoot my SQL query results in Databricks
Analyze the temperature-demand relationship

Utilities use temperature thresholds to trigger demand response events. This final query groups consumption into temperature buckets so you can see exactly how much power demand changes when it gets hot.

  • Add a new cell and type this query to compare consumption across cold, mild, and hot temperature ranges:
%sql
SELECT
  CASE
    WHEN Temperature < 15 THEN 'Cold (< 15)'
    WHEN Temperature BETWEEN 15 AND 25 THEN 'Mild (15-25)'
    ELSE 'Hot (> 25)'
  END as temp_range,
  ROUND(AVG(`Zone 1 Power Consumption`), 2) as avg_consumption,
  COUNT(*) as num_records
FROM default.power_consumption
GROUP BY
  CASE
    WHEN Temperature < 15 THEN 'Cold (< 15)'
    WHEN Temperature BETWEEN 15 AND 25 THEN 'Mild (15-25)'
    ELSE 'Hot (> 25)'
  END
ORDER BY avg_consumption DESC;

What does this query do?

  • CASE WHEN creates three temperature buckets: Cold (below 15 degrees), Mild (15-25), and Hot (above 25).
  • For each bucket, you get the average power consumption and the number of data points. This reveals whether hot weather drives higher demand.
  • num_records confirms you have enough data in each bucket for the averages to be meaningful.
  • Run the cell with Shift+Enter.
  • You should see 3 rows ordered by consumption. The "Hot" range should have notably higher average consumption than "Cold" or "Mild". This confirms that temperature is a strong predictor of energy demand.

Seeing a syntax error on the CASE statement?

  • Make sure the CASE WHEN block in the GROUP BY matches the one in the SELECT exactly. Both must have the same three conditions.
  • Check that END appears after the last ELSE clause in both locations.
  • Help me fix my CASE WHEN SQL syntax in Databricks

Why does this matter for utilities?

A utility like Xcel Energy uses exactly this type of temperature-demand analysis to decide when to trigger demand response programs. If consumption jumps 30% above a certain temperature, that temperature becomes a trigger threshold for asking customers to reduce usage.

✔️ Awesome, I've got everything!

Great. Make sure all 5 SQL cells have been run successfully and you can see results below each one.

ⓧ I'd like to double check the full code

Here are all 5 SQL cells you should have in your notebook at this point. Each one should be in its own cell with %sql on the first line.

-- CELL 1: Inspect the table structure and basic stats
%sql
SELECT COUNT(*) as total_records,
       MIN(DateTime) as start_date,
       MAX(DateTime) as end_date
FROM default.power_consumption;
-- CELL 2: Basic statistics for key columns
%sql
SELECT
  ROUND(AVG(Temperature), 2) as avg_temp,
  ROUND(MIN(Temperature), 2) as min_temp,
  ROUND(MAX(Temperature), 2) as max_temp,
  ROUND(AVG(`Zone 1 Power Consumption`), 2) as avg_zone1,
  ROUND(AVG(`Zone 2 Power Consumption`), 2) as avg_zone2,
  ROUND(AVG(`Zone 3 Power Consumption`), 2) as avg_zone3
FROM default.power_consumption;
-- CELL 3: Average consumption by hour of day
%sql
SELECT
  HOUR(DateTime) as hour_of_day,
  ROUND(AVG(`Zone 1 Power Consumption`), 2) as avg_consumption
FROM default.power_consumption
GROUP BY HOUR(DateTime)
ORDER BY hour_of_day;
-- CELL 4: Top 10 peak consumption records
%sql
SELECT DateTime, Temperature, Humidity, `Wind Speed`,
       `Zone 1 Power Consumption` as zone1_consumption
FROM default.power_consumption
ORDER BY `Zone 1 Power Consumption` DESC
LIMIT 10;
-- CELL 5: Consumption by temperature range
%sql
SELECT
  CASE
    WHEN Temperature < 15 THEN 'Cold (< 15)'
    WHEN Temperature BETWEEN 15 AND 25 THEN 'Mild (15-25)'
    ELSE 'Hot (> 25)'
  END as temp_range,
  ROUND(AVG(`Zone 1 Power Consumption`), 2) as avg_consumption,
  COUNT(*) as num_records
FROM default.power_consumption
GROUP BY
  CASE
    WHEN Temperature < 15 THEN 'Cold (< 15)'
    WHEN Temperature BETWEEN 15 AND 25 THEN 'Mild (15-25)'
    ELSE 'Hot (> 25)'
  END
ORDER BY avg_consumption DESC;

You now have a clear picture of the data. Consumption peaks at specific hours, spikes with high temperatures, and varies across zones. Next up, you will use Python to engineer these patterns into features that a machine learning model can learn from.

Engineer Features and Prepare the Model Data

Your SQL queries uncovered clear patterns in the data. Consumption varies dramatically by hour of day, temperature range, and zone.

Raw data alone is not enough for a machine learning model. You need to extract meaningful signals from the timestamp column so the model can learn when demand spikes, not just what the weather looked like. This process is called feature engineering.

In this step, get ready to:
  • Load the power consumption table into a pandas DataFrame for Python-based analysis.
  • Engineer temporal features from the DateTime column to capture time-based demand patterns.
  • Split the data into training and test sets for model evaluation.
Load the data and engineer temporal features

Your SQL cells queried the table directly in the Databricks lakehouse. Now you need to bring that same data into pandas so you can manipulate it with Python and feed it to a machine learning model.

  • Click the + Code button below your last SQL cell to create a new Python cell.
  • Paste the following code into the new cell:
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import matplotlib.pyplot as plt
import seaborn as sns

# Load table into pandas
df = spark.table("default.power_consumption").toPandas()
print(f"Dataset shape: {df.shape}")

# Parse DateTime and extract temporal features
df['DateTime'] = pd.to_datetime(df['DateTime'])
df['hour'] = df['DateTime'].dt.hour
df['day_of_week'] = df['DateTime'].dt.dayofweek
df['month'] = df['DateTime'].dt.month
df['is_weekend'] = (df['day_of_week'] >= 5).astype(int)

print(f"\nTemporal features added. Columns: {df.columns.tolist()}")
print(f"\nDate range: {df['DateTime'].min()} to {df['DateTime'].max()}")
df[['Temperature', 'Humidity', 'Wind Speed', 'Zone 1 Power Consumption']].describe()

What does this code do?

  • spark.table("default.power_consumption").toPandas() loads the table you uploaded earlier into a pandas DataFrame. This is the standard way to pull lakehouse data into Python in Databricks.
  • The .dt accessor extracts components from the DateTime column. Each new column (hour, day_of_week, month, is_weekend) becomes a feature the model can learn from.
  • is_weekend is a binary flag. It equals 1 if the day is Saturday or Sunday, 0 otherwise. This lets the model distinguish weekday from weekend consumption patterns.
  • Run the cell by pressing Shift+Enter or clicking the play button at the top left of the cell.
  • You should see Dataset shape: (52417, 9) printed first, followed by a list of columns that now includes hour, day_of_week, month, and is_weekend.
  • At the bottom of the output, a statistics table shows the min, max, mean, and standard deviation of the four key columns.

Seeing an error when you run the cell?

  • If you see AnalysisException: Table or view not found, double-check that your table is named power_consumption in the default schema. Open the Catalog panel in the left sidebar to verify.
  • If you see ModuleNotFoundError, make sure you selected Python as the cell language. All the libraries used here are pre-installed in the Databricks runtime.

Still stuck? Ask for help.

Why engineer features from a timestamp?

A raw datetime string like 2017-01-01 00:00:00 means nothing to a machine learning model. It is just a sequence of characters.

By extracting hour, day_of_week, and month as separate numeric columns, the model can learn patterns like "demand peaks at hour 19" or "weekends have lower usage." These are the exact patterns your SQL queries already revealed.

Define the feature matrix and split the data

With temporal features engineered, you now need to tell the model which columns to use as inputs and which column to predict. Then you split the data so you can train on one portion and test on another.

  • Create another new Python cell below Cell 6 by clicking + Code.
  • Paste the following code into the new cell:
feature_cols = [
    'Temperature', 'Humidity', 'Wind Speed',
    'general diffuse flows', 'diffuse flows',
    'hour', 'day_of_week', 'month', 'is_weekend'
]
target_col = 'Zone 1 Power Consumption'

X = df[feature_cols]
y = df[target_col]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")
print(f"\nFeature matrix columns: {feature_cols}")

What does this code do?

  • feature_cols lists the 9 input columns the model will learn from. These include 5 weather measurements from the original dataset plus the 4 temporal features you just created.
  • target_col is what the model will try to predict. Zone 1 Power Consumption represents one distribution zone's demand.
  • train_test_split from scikit-learn randomly divides the data into 80% for training and 20% for testing. The random_state=42 makes the split reproducible so you get consistent results every run.
  • Run the cell by pressing Shift+Enter.
  • You should see output showing approximately 41,933 training samples and 10,484 test samples, followed by the list of 9 feature column names.

Seeing a KeyError or NameError?

  • A KeyError means a column name in feature_cols does not match the actual column names in your table. Check for extra spaces or capitalization differences in column names like general diffuse flows.
  • A NameError: name 'df' is not defined means Cell 6 was not run successfully before this cell. Go back and run Cell 6 first.

Need help debugging?

Why 80/20 for the split?

With 52,417 rows, an 80/20 split gives the model over 41,000 training examples to learn patterns from. The remaining 10,000+ test samples are enough to reliably measure prediction accuracy.

The test set simulates "unseen" future data. If the model performs well on test data it never trained on, you can be confident it will generalize to real forecasting scenarios.

✔️ Awesome, I've got everything!

Great. Double check you have both Cell 6 and Cell 7 producing output with no errors before moving on.

ⓧ I'd like to double check the full code

Here is the complete code for both Python cells you added in this step. Compare yours against this reference.

# CELL 6: Load data and engineer features
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import matplotlib.pyplot as plt
import seaborn as sns

# Load table into pandas
df = spark.table("default.power_consumption").toPandas()
print(f"Dataset shape: {df.shape}")

# Parse DateTime and extract temporal features
df['DateTime'] = pd.to_datetime(df['DateTime'])
df['hour'] = df['DateTime'].dt.hour
df['day_of_week'] = df['DateTime'].dt.dayofweek
df['month'] = df['DateTime'].dt.month
df['is_weekend'] = (df['day_of_week'] >= 5).astype(int)

print(f"\nTemporal features added. Columns: {df.columns.tolist()}")
print(f"\nDate range: {df['DateTime'].min()} to {df['DateTime'].max()}")
df[['Temperature', 'Humidity', 'Wind Speed', 'Zone 1 Power Consumption']].describe()
# CELL 7: Prepare features and train/test split
feature_cols = [
    'Temperature', 'Humidity', 'Wind Speed',
    'general diffuse flows', 'diffuse flows',
    'hour', 'day_of_week', 'month', 'is_weekend'
]
target_col = 'Zone 1 Power Consumption'

X = df[feature_cols]
y = df[target_col]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")
print(f"\nFeature matrix columns: {feature_cols}")

Your data is prepped and split. Next up, you will train a Random Forest model on this data and see how accurately it predicts energy demand.

Train and Evaluate the Random Forest Model

Your features are defined and your data is split into training and test sets. Now comes the core of any data science project: training a model and measuring how well it performs.

You'll train a Random Forest regression model on your 9 weather and temporal features. Then you'll evaluate its accuracy using three standard metrics that utility data science teams report on: MAE, RMSE, and R-squared.

In this step, get ready to:
  • Train a Random Forest regression model on 41,000+ energy consumption records.
  • Evaluate model accuracy with MAE, RMSE, and R-squared metrics.
  • Create an Actual vs Predicted scatter plot to visually assess performance.
Train the model and calculate performance metrics

Random Forest works by building many decision trees on random subsets of your data, then averaging their predictions. This makes it robust to overfitting and effective at capturing non-linear relationships between weather conditions and power consumption.

  • Create a new Python cell in your notebook (click the + button below your last cell).
  • Paste the following code to train the model and evaluate it:
rf_model = RandomForestRegressor(
    n_estimators=100,
    max_depth=15,
    random_state=42,
    n_jobs=-1
)
rf_model.fit(X_train, y_train)

y_pred = rf_model.predict(X_test)

mae = mean_absolute_error(y_test, y_pred)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
r2 = r2_score(y_test, y_pred)

print("Model Performance:")
print(f"  MAE:  {mae:.2f}")
print(f"  RMSE: {rmse:.2f}")
print(f"  R2:   {r2:.4f}")

What does this code do?

This cell trains a Random Forest model and measures how well it predicts energy demand.

  • n_estimators=100 builds 100 decision trees that vote together on predictions.
  • max_depth=15 limits tree depth to prevent overfitting on training data.
  • n_jobs=-1 uses all available CPU cores to speed up training.
  • MAE (Mean Absolute Error) is the average prediction error in the same units as power consumption.
  • R-squared tells you what percentage of consumption variance the model explains. Closer to 1.0 is better.
  • Run the cell by pressing Shift+Enter.
  • You should see a Model Performance: printout with MAE, RMSE, and R2 values. An R2 value above 0.85 indicates a strong model.

Seeing a NameError?

If you see NameError: name 'X_train' is not defined, your previous cells have not been executed in this session. Go back to Cell 6 and Cell 7 and run them again (Shift+Enter on each).

If the cell takes more than 30 seconds, your serverless compute may be starting up. Databricks Free Edition sometimes has a cold-start delay.

Still stuck? Help me debug my Random Forest training cell in Databricks.

Why Random Forest over other algorithms?

Random Forest is interpretable, robust to overfitting with default parameters, and handles non-linear relationships well. It also provides a built-in feature importance ranking, which you'll use in the next step to generate business recommendations.

For an associate-level data science role, Random Forest demonstrates competence with ML fundamentals without overcomplicating the analysis. Production teams often start with Random Forest as a baseline before exploring more complex models.

Visualize predictions with a scatter plot

Numbers alone can be misleading. A scatter plot comparing actual vs predicted values lets you visually confirm that the model performs well across the full range of consumption values, not just on average.

  • Create another new Python cell below Cell 8.
  • Paste the following code to generate the scatter plot:
plt.figure(figsize=(8, 6))
plt.scatter(y_test, y_pred, alpha=0.3, s=10, color='steelblue')
plt.plot(
    [y_test.min(), y_test.max()],
    [y_test.min(), y_test.max()],
    'r--', lw=2, label='Perfect Prediction'
)
plt.xlabel('Actual Power Consumption')
plt.ylabel('Predicted Power Consumption')
plt.title('Actual vs Predicted: Zone 1 Power Consumption')
plt.legend()
plt.tight_layout()
plt.show()

What does this chart show?

Each blue dot is one test sample. The x-axis is the actual power consumption and the y-axis is what the model predicted.

The red dashed line represents perfect predictions (where actual equals predicted). The closer your dots cluster around that diagonal line, the more accurate your model is. Dots far from the line are predictions the model got wrong.

  • Run the cell by pressing Shift+Enter.
  • You should see a scatter plot with blue dots clustered tightly around the red diagonal line. Most points should fall close to the line, with some spread at higher consumption values.

Plot not displaying?

Databricks renders matplotlib plots inline automatically. If you see no output, check that you included plt.show() at the end of your cell.

If you see NameError: name 'y_pred' is not defined, run Cell 8 first. This cell depends on the predictions generated there.

Need more help? Help me troubleshoot my matplotlib scatter plot in Databricks.

✔️ Awesome, I've got everything!

Great. Double check you've run both Cell 8 and Cell 9 and can see both the metrics output and the scatter plot.

ⓧ I'd like to double check the full code

Here is the complete code for both cells added in this step. Compare against what you have in your notebook.

# =============================================================
# CELL 8 (Python): Train Random Forest model
# =============================================================
rf_model = RandomForestRegressor(
    n_estimators=100,
    max_depth=15,
    random_state=42,
    n_jobs=-1
)
rf_model.fit(X_train, y_train)

y_pred = rf_model.predict(X_test)

mae = mean_absolute_error(y_test, y_pred)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
r2 = r2_score(y_test, y_pred)

print("Model Performance:")
print(f"  MAE:  {mae:.2f}")
print(f"  RMSE: {rmse:.2f}")
print(f"  R2:   {r2:.4f}")

# =============================================================
# CELL 9 (Python): Actual vs Predicted scatter plot
# =============================================================
plt.figure(figsize=(8, 6))
plt.scatter(y_test, y_pred, alpha=0.3, s=10, color='steelblue')
plt.plot(
    [y_test.min(), y_test.max()],
    [y_test.min(), y_test.max()],
    'r--', lw=2, label='Perfect Prediction'
)
plt.xlabel('Actual Power Consumption')
plt.ylabel('Predicted Power Consumption')
plt.title('Actual vs Predicted: Zone 1 Power Consumption')
plt.legend()
plt.tight_layout()
plt.show()

Your model is trained and validated. Next up, you'll extract the insights hidden inside it. You'll visualize which features drive demand the most and translate those patterns into actionable business recommendations for a utility's demand response program.

Visualize Insights and Deliver Business Recommendations

Your model is trained and the numbers look promising. But raw metrics like MAE and R-squared only tell part of the story.

A utility program manager does not read Python output. They need clear visualizations showing which factors matter most and concrete recommendations for when to trigger demand response events. In this step, you will create those visualizations and translate your model's findings into actionable business guidance.

In this step, get ready to:
  • Visualize feature importance to identify the strongest demand drivers.
  • Plot hourly consumption patterns to reveal peak demand windows.
  • Generate programmatic business recommendations for demand response scheduling.
Create the feature importance chart

Your Random Forest model internally tracks how much each feature contributes to its predictions. Extracting this into a bar chart makes it immediately clear which weather and temporal factors drive energy demand the most.

  • Create a new cell in your notebook below Cell 9 and paste the following code:
# Extract feature importances from the trained model
importances = rf_model.feature_importances_
feature_importance_df = pd.DataFrame({
    'feature': feature_cols,
    'importance': importances
}).sort_values('importance', ascending=False)

# Create horizontal bar chart showing each feature's contribution
plt.figure(figsize=(10, 6))
sns.barplot(
    data=feature_importance_df,
    x='importance', y='feature',
    palette='viridis'
)
plt.title('Feature Importance for Power Consumption Prediction')
plt.xlabel('Importance Score')
plt.ylabel('Feature')
plt.tight_layout()
plt.show()

# Print the top 3 features numerically
print("\nTop 3 most important features:")
for i, row in feature_importance_df.head(3).iterrows():
    print(f"  {row['feature']}: {row['importance']:.4f}")

What does this code do?

The feature_importances_ attribute returns an array of scores (one per feature) that sum to 1.0. A higher score means that feature contributed more to the model's predictions.

The code sorts these into a DataFrame and plots them as a horizontal bar chart using seaborn. This makes it visually obvious which factors (temperature, hour of day, humidity) matter most for predicting demand.

  • Run Cell 10 by clicking the play button or pressing Shift+Enter.
  • You should see a horizontal bar chart with features ranked by importance, plus printed text showing the top 3 features and their scores.

Not seeing the bar chart?

Make sure Cell 8 (model training) ran successfully first. The rf_model variable must exist in memory before you can access its feature_importances_ attribute.

If you see a NameError for feature_cols, run Cell 7 again to re-define the feature column list.

Help me debug my feature importance chart

Plot the hourly consumption pattern

Knowing which hours see the highest demand is critical for scheduling demand response events. A line chart of average consumption by hour reveals the daily demand curve at a glance.

  • Create a new cell below Cell 10 and paste this code:
# Calculate average consumption for each hour across all days
hourly_avg = df.groupby('hour')[target_col].mean()

# Plot the daily demand curve
plt.figure(figsize=(10, 5))
plt.plot(hourly_avg.index, hourly_avg.values, marker='o', linewidth=2, color='darkorange')
plt.xlabel('Hour of Day')
plt.ylabel('Average Power Consumption')
plt.title('Average Hourly Power Consumption Pattern (Zone 1)')
plt.xticks(range(0, 24))
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

What does this code do?

The groupby('hour') aggregates all records by their hour value (0 through 23) and computes the mean consumption for each. This smooths out day-to-day noise and reveals the underlying daily demand shape.

The resulting line chart shows exactly when demand peaks and dips. Utilities use this pattern to decide when to pre-cool buildings, shift industrial loads, or activate battery storage.

  • Run Cell 11.
  • You should see a line chart with orange dots at each hour. The curve will show clear peaks during daytime hours and dips overnight.

Chart looks flat or has a single point?

Check that Cell 6 ran successfully. The hour column is created by the feature engineering cell. Without it, the groupby will fail or produce unexpected results.

Also confirm that target_col is still defined from Cell 7. If your kernel restarted, re-run Cells 6 and 7 first.

Help me fix my hourly consumption chart

Generate business recommendations from your model

Data science work is only valuable if it drives decisions. This final cell takes your model's outputs and the hourly consumption data, then programmatically generates four concrete recommendations a utility program manager can act on.

  • Create a new cell below Cell 11 and paste this code:
# Generate actionable business recommendations from model findings
print("=" * 60)
print("BUSINESS RECOMMENDATIONS FOR DEMAND RESPONSE PROGRAM")
print("=" * 60)

# 1. Identify the 5 highest-demand hours
peak_hours = hourly_avg.nlargest(5).index.tolist()
print(f"\n1. PEAK DEMAND HOURS: {sorted(peak_hours)}")
print(f"   Schedule demand response events during hours")
print(f"   {sorted(peak_hours)} when consumption is highest.")

# 2. Find the temperature threshold where consumption spikes
temp_threshold = df['Temperature'].quantile(0.9)
high_temp_mask = df['Temperature'] > temp_threshold
avg_high_temp = df.loc[high_temp_mask, target_col].mean()
avg_normal = df.loc[~high_temp_mask, target_col].mean()
pct_increase = ((avg_high_temp - avg_normal) / avg_normal) * 100

print(f"\n2. TEMPERATURE TRIGGER: {temp_threshold:.1f} degrees")
print(f"   When temperature exceeds {temp_threshold:.1f}, consumption")
print(f"   increases by {pct_increase:.1f}% on average.")
print(f"   Pre-trigger demand response when forecasted temperature")
print(f"   exceeds this threshold.")

# 3. Compare weekday vs weekend consumption
weekend_avg = df[df['is_weekend'] == 1][target_col].mean()
weekday_avg = df[df['is_weekend'] == 0][target_col].mean()
higher_period = 'weekdays' if weekday_avg > weekend_avg else 'weekends'

print(f"\n3. WEEKDAY vs WEEKEND CONSUMPTION:")
print(f"   Weekday average: {weekday_avg:,.0f}")
print(f"   Weekend average: {weekend_avg:,.0f}")
print(f"   Focus demand response resources on {higher_period}.")

# 4. Report model confidence and primary drivers
print(f"\n4. MODEL CONFIDENCE: R2 = {r2:.4f}")
print(f"   The model explains {r2*100:.1f}% of consumption variance.")
print(f"   Primary drivers: {feature_importance_df.iloc[0]['feature']}")
print(f"   and {feature_importance_df.iloc[1]['feature']}.")
print(f"\n" + "=" * 60)

What does this code do?

This cell generates four distinct recommendations by querying your data and model programmatically:

  • Peak demand hours: uses nlargest(5) on the hourly averages to find when demand response events should be scheduled.
  • Temperature trigger: calculates the 90th percentile temperature and measures how much consumption increases above that threshold.
  • Weekday vs weekend: compares average consumption to determine where to focus resources.
  • Model confidence: reports R-squared and names the two strongest predictors from your feature importance analysis.
  • Run Cell 12.
  • You should see a formatted recommendations block with four numbered sections. Each section contains specific numbers derived from your data (hours, temperature threshold, percentages).

Why generate recommendations programmatically?

Hardcoding recommendation text is fragile. If the dataset changes or you retrain the model, hardcoded insights become stale. By computing thresholds and comparisons from the data directly, the recommendations update automatically whenever you re-run the notebook with new data.

This is exactly how utility data science teams work. They build repeatable pipelines that stakeholders can refresh monthly or quarterly without touching the code.

Seeing a NameError for hourly_avg or r2?

Cell 12 depends on variables created in earlier cells. The hourly_avg variable comes from Cell 11, r2 comes from Cell 8, and feature_importance_df comes from Cell 10. If any of those cells were not run in this session, re-run them first.

Help me fix my business recommendations cell

✔️ Awesome, I've got everything!

Great. Double check you have saved your notebook with all 12 cells.

ⓧ I'd like to double check the full code

Here is the complete notebook code from Cell 1 through Cell 12. Each section between the divider comments corresponds to one cell in your Databricks notebook.

# Databricks Notebook: Energy Demand Prediction Model
# This notebook alternates between SQL and Python cells.
# Lines starting with "# %sql" indicate SQL cells in Databricks.
# Copy each section into its own cell in your Databricks notebook.

# =============================================================
# CELL 1 (SQL): Inspect the table structure and basic stats
# =============================================================
# %sql
# SELECT COUNT(*) as total_records,
#        MIN(DateTime) as start_date,
#        MAX(DateTime) as end_date
# FROM default.power_consumption;

# =============================================================
# CELL 2 (SQL): Basic statistics for key columns
# =============================================================
# %sql
# SELECT
#   ROUND(AVG(Temperature), 2) as avg_temp,
#   ROUND(MIN(Temperature), 2) as min_temp,
#   ROUND(MAX(Temperature), 2) as max_temp,
#   ROUND(AVG(`Zone 1 Power Consumption`), 2) as avg_zone1,
#   ROUND(AVG(`Zone 2 Power Consumption`), 2) as avg_zone2,
#   ROUND(AVG(`Zone 3 Power Consumption`), 2) as avg_zone3
# FROM default.power_consumption;

# =============================================================
# CELL 3 (SQL): Average consumption by hour of day
# =============================================================
# %sql
# SELECT
#   HOUR(DateTime) as hour_of_day,
#   ROUND(AVG(`Zone 1 Power Consumption`), 2) as avg_consumption
# FROM default.power_consumption
# GROUP BY HOUR(DateTime)
# ORDER BY hour_of_day;

# =============================================================
# CELL 4 (SQL): Top 10 peak consumption records
# =============================================================
# %sql
# SELECT DateTime, Temperature, Humidity, `Wind Speed`,
#        `Zone 1 Power Consumption` as zone1_consumption
# FROM default.power_consumption
# ORDER BY `Zone 1 Power Consumption` DESC
# LIMIT 10;

# =============================================================
# CELL 5 (SQL): Consumption by temperature range
# =============================================================
# %sql
# SELECT
#   CASE
#     WHEN Temperature < 15 THEN 'Cold (< 15)'
#     WHEN Temperature BETWEEN 15 AND 25 THEN 'Mild (15-25)'
#     ELSE 'Hot (> 25)'
#   END as temp_range,
#   ROUND(AVG(`Zone 1 Power Consumption`), 2) as avg_consumption,
#   COUNT(*) as num_records
# FROM default.power_consumption
# GROUP BY
#   CASE
#     WHEN Temperature < 15 THEN 'Cold (< 15)'
#     WHEN Temperature BETWEEN 15 AND 25 THEN 'Mild (15-25)'
#     ELSE 'Hot (> 25)'
#   END
# ORDER BY avg_consumption DESC;

# =============================================================
# CELL 6 (Python): Load data and engineer features
# =============================================================
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import matplotlib.pyplot as plt
import seaborn as sns

df = spark.table("default.power_consumption").toPandas()
print(f"Dataset shape: {df.shape}")

df['DateTime'] = pd.to_datetime(df['DateTime'])
df['hour'] = df['DateTime'].dt.hour
df['day_of_week'] = df['DateTime'].dt.dayofweek
df['month'] = df['DateTime'].dt.month
df['is_weekend'] = (df['day_of_week'] >= 5).astype(int)

print(f"\nTemporal features added. Columns: {df.columns.tolist()}")
print(f"\nDate range: {df['DateTime'].min()} to {df['DateTime'].max()}")
df[['Temperature', 'Humidity', 'Wind Speed', 'Zone 1 Power Consumption']].describe()

# =============================================================
# CELL 7 (Python): Prepare features and train/test split
# =============================================================
feature_cols = [
    'Temperature', 'Humidity', 'Wind Speed',
    'general diffuse flows', 'diffuse flows',
    'hour', 'day_of_week', 'month', 'is_weekend'
]
target_col = 'Zone 1 Power Consumption'

X = df[feature_cols]
y = df[target_col]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")
print(f"\nFeature matrix columns: {feature_cols}")

# =============================================================
# CELL 8 (Python): Train Random Forest model
# =============================================================
rf_model = RandomForestRegressor(
    n_estimators=100,
    max_depth=15,
    random_state=42,
    n_jobs=-1
)
rf_model.fit(X_train, y_train)

y_pred = rf_model.predict(X_test)

mae = mean_absolute_error(y_test, y_pred)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
r2 = r2_score(y_test, y_pred)

print("Model Performance:")
print(f"  MAE:  {mae:.2f}")
print(f"  RMSE: {rmse:.2f}")
print(f"  R2:   {r2:.4f}")

# =============================================================
# CELL 9 (Python): Actual vs Predicted scatter plot
# =============================================================
plt.figure(figsize=(8, 6))
plt.scatter(y_test, y_pred, alpha=0.3, s=10, color='steelblue')
plt.plot(
    [y_test.min(), y_test.max()],
    [y_test.min(), y_test.max()],
    'r--', lw=2, label='Perfect Prediction'
)
plt.xlabel('Actual Power Consumption')
plt.ylabel('Predicted Power Consumption')
plt.title('Actual vs Predicted: Zone 1 Power Consumption')
plt.legend()
plt.tight_layout()
plt.show()

# =============================================================
# CELL 10 (Python): Feature importance visualization
# =============================================================
importances = rf_model.feature_importances_
feature_importance_df = pd.DataFrame({
    'feature': feature_cols,
    'importance': importances
}).sort_values('importance', ascending=False)

plt.figure(figsize=(10, 6))
sns.barplot(
    data=feature_importance_df,
    x='importance', y='feature',
    palette='viridis'
)
plt.title('Feature Importance for Power Consumption Prediction')
plt.xlabel('Importance Score')
plt.ylabel('Feature')
plt.tight_layout()
plt.show()

print("\nTop 3 most important features:")
for i, row in feature_importance_df.head(3).iterrows():
    print(f"  {row['feature']}: {row['importance']:.4f}")

# =============================================================
# CELL 11 (Python): Hourly consumption pattern
# =============================================================
hourly_avg = df.groupby('hour')[target_col].mean()

plt.figure(figsize=(10, 5))
plt.plot(hourly_avg.index, hourly_avg.values, marker='o', linewidth=2, color='darkorange')
plt.xlabel('Hour of Day')
plt.ylabel('Average Power Consumption')
plt.title('Average Hourly Power Consumption Pattern (Zone 1)')
plt.xticks(range(0, 24))
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

# =============================================================
# CELL 12 (Python): Business recommendations
# =============================================================
print("=" * 60)
print("BUSINESS RECOMMENDATIONS FOR DEMAND RESPONSE PROGRAM")
print("=" * 60)

peak_hours = hourly_avg.nlargest(5).index.tolist()
print(f"\n1. PEAK DEMAND HOURS: {sorted(peak_hours)}")
print(f"   Schedule demand response events during hours")
print(f"   {sorted(peak_hours)} when consumption is highest.")

temp_threshold = df['Temperature'].quantile(0.9)
high_temp_mask = df['Temperature'] > temp_threshold
avg_high_temp = df.loc[high_temp_mask, target_col].mean()
avg_normal = df.loc[~high_temp_mask, target_col].mean()
pct_increase = ((avg_high_temp - avg_normal) / avg_normal) * 100

print(f"\n2. TEMPERATURE TRIGGER: {temp_threshold:.1f} degrees")
print(f"   When temperature exceeds {temp_threshold:.1f}, consumption")
print(f"   increases by {pct_increase:.1f}% on average.")
print(f"   Pre-trigger demand response when forecasted temperature")
print(f"   exceeds this threshold.")

weekend_avg = df[df['is_weekend'] == 1][target_col].mean()
weekday_avg = df[df['is_weekend'] == 0][target_col].mean()
higher_period = 'weekdays' if weekday_avg > weekend_avg else 'weekends'

print(f"\n3. WEEKDAY vs WEEKEND CONSUMPTION:")
print(f"   Weekday average: {weekday_avg:,.0f}")
print(f"   Weekend average: {weekend_avg:,.0f}")
print(f"   Focus demand response resources on {higher_period}.")

print(f"\n4. MODEL CONFIDENCE: R2 = {r2:.4f}")
print(f"   The model explains {r2*100:.1f}% of consumption variance.")
print(f"   Primary drivers: {feature_importance_df.iloc[0]['feature']}")
print(f"   and {feature_importance_df.iloc[1]['feature']}.")
print(f"\n" + "=" * 60)

Your notebook is complete. You have gone from raw data exploration to a trained model to actionable business recommendations, all within a single Databricks notebook. This is exactly the end-to-end workflow utility data science teams deliver.

Secret mission

Build a Peak Demand Classifier

Utilities don't just want to predict how much energy will be consumed. They need to know whether the next hour will be a peak event so they can trigger demand response before the grid strains. Can you build a classifier that makes that call?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on Databricks Free Edition, so there are no ongoing costs. Decide whether to keep your resources in place, pause by stopping any active compute, or delete them entirely.

Resources you used:

  • Databricks Free Edition workspace and account.
  • Databricks notebook (energy demand prediction model).
  • The power_consumption table in the default catalog.
  • The downloaded CSV file on your local machine.

Keep everything running

No action needed. Choose this if you plan to extend your notebook, try new models, or revisit the data later.

  • Your Databricks Free Edition workspace stays active indefinitely at no cost.
  • Your notebook and table remain in the workspace ready for future use.
  • Keep the local CSV file if you want a backup copy of the dataset.

Pause - I'll come back to this later

Databricks Free Edition uses serverless compute, so there are no clusters to manually stop. Your workspace stays idle at no cost when you are not running cells.

  • Close any open notebook tabs in Databricks to keep your workspace tidy.
  • Delete the local CSV file from your Downloads folder if you want to free up disk space.
  • Your table and notebook will be waiting when you come back.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow these steps to clean up completely.

  • In Databricks, open your notebook, click the three-dot menu in the top right, and select Move to Trash.
  • Navigate to Catalog in the left sidebar, find the power_consumption table under the default schema, click the three-dot menu, and select Delete.
  • Delete the local CSV file from your computer.
  • If you want to remove your Databricks account entirely, go to your account settings and follow the account deletion process.

That's a Wrap!

That's a Wrap!

Nice work! You just built a complete energy demand prediction pipeline in Databricks, from SQL exploration to ML modeling to business recommendations.

You've learned how to:

  • Explored 52,417 energy consumption records using SQL in a Databricks lakehouse environment and engineered temporal features for machine learning.
  • Trained a Random Forest regression model that predicts power consumption from weather and time features, evaluated with MAE, RMSE, and R-squared metrics.
  • Translated model insights into actionable business recommendations identifying peak demand hours, temperature triggers, and weekday vs weekend patterns for demand response scheduling.
  • Built a peak demand classifier with precision/recall evaluation for triggering demand response events in advance.

Ready to quiz yourself?