Master Polars: Think in Expressions

Build a Polars expression mental model by writing progressive Python scripts.

Introduction

30 Second Summary

Every time you filter a spreadsheet, sort a list, or calculate an average, you're doing data processing. Polars is a Python library that lets you describe those same transformations as composable building blocks called expressions.

In this project, you will build a collection of runnable Python scripts that teach you to think in Polars expressions. You'll progress from basic concepts to advanced patterns, proving your understanding through independent coding challenges.

What You'll Build

A progressive series of self-contained Python scripts that demonstrate every core Polars pattern, from DataFrame creation to lazy evaluation and expression expansion.

By the end of this project, you'll have:

  • A mental model of Polars' expression-based philosophy that lets you read any Polars code and understand what it does.
  • A collection of runnable Python scripts demonstrating every core pattern from simple to advanced.
  • Completed challenge exercises that prove you can write Polars code independently without looking at solutions.
  • Secret Mission: Build a zero-variable data pipeline using only lazy evaluation and expression chaining.

Are there any prerequisites?

You only need basic Python knowledge (variables, functions, lists, dictionaries). No prior data processing experience is needed.

Create Your First Script: Expressions as Lazy Building Blocks

Your environment is ready. Now it's time to understand the single most important idea in Polars: expressions are lazy building blocks that describe a computation without executing it.

In most data tools, when you write a formula, it runs immediately. Polars works differently. You write an expression (a recipe), and then you place it inside a context (like df.select()) to produce actual results. Once you internalize this, every piece of Polars code becomes readable.

In this step, get ready to:
  • Write a worked example showing that expressions are descriptions, not results.
  • Complete a guided exercise creating product value expressions.
  • Tackle a hard challenge computing student score statistics independently.
Write the worked example: expressions as descriptions

The key insight you're about to see: a Polars expression is just a description of a transformation. It doesn't compute anything until you place it inside a context.

  • Open 01_basics.py in your text editor and add the following code to create a DataFrame and define your first expression:
import polars as pl

# === WORKED EXAMPLE: Expressions Are Lazy Building Blocks ===

# Create a DataFrame
df = pl.DataFrame(
    {
        "name": ["Alice", "Ben", "Chloe", "Daniel"],
        "weight": [57.9, 72.5, 53.6, 83.1],  # kg
        "height": [1.56, 1.77, 1.65, 1.75],  # m
    }
)
print("Our DataFrame:")
print(df)
print()

# Define an expression - this is just a DESCRIPTION, not a result
bmi_expr = pl.col("weight") / (pl.col("height") ** 2)
print("The expression object (not computed yet):")
print(bmi_expr)
print()

What does this code do?

pl.DataFrame() creates a table with three columns: name, weight, and height. This is the data you'll transform with expressions.

pl.col("weight") / (pl.col("height") ** 2) defines a BMI formula. The critical thing: this line does NOT calculate any BMI values. It creates an expression object that describes the calculation. When you print it, you'll see a formula-like description referencing column names instead of actual numbers.

  • Save 01_basics.py.
  • Run the script to see the expression object by running:
python3 01_basics.py
  • You should see the DataFrame printed as a table, followed by the expression object printed as a formula description (not calculated values).

Seeing a ModuleNotFoundError?

If you see a message about polars not being found, make sure you installed Polars in the same Python environment you're running the script with. Try running pip install polars==1.41.0 again.

help me debug why Python can't find the polars module

  • Add this code to the bottom of 01_basics.py to execute the expression in a select context:
# Execute the expression in a context to get actual values
result = df.select(bmi=bmi_expr)
print("Expression executed in a 'select' context:")
print(result)
print()

What does this code do?

df.select(bmi=bmi_expr) is where the magic happens. The select method is a context. It takes your expression recipe and runs it against the DataFrame's actual data.

The bmi= part names the output column. Without a context, your expression is just a description. Inside a context, it produces real values.

  • Save and run 01_basics.py again.
  • You should now see calculated BMI values in a new table at the bottom of the output.
  • Add this to the bottom of 01_basics.py to demonstrate that expressions are reusable and composable:
# Expressions are reusable building blocks
result2 = df.select(
    bmi=bmi_expr,
    avg_bmi=bmi_expr.mean(),
    ideal_max_bmi=25,
)
print("Multiple expressions in one context:")
print(result2)
print()

# Key insight: expressions compose - build complex from simple
deviation_expr = (bmi_expr - bmi_expr.mean()) / bmi_expr.std()
result3 = df.select(bmi_zscore=deviation_expr)
print("Composed expression (z-score of BMI):")
print(result3)

What does this code do?

The first block shows reuse: bmi_expr appears twice in the same context. You can call .mean() on it to get the average, and Polars broadcasts the scalar result to fill the column.

The second block shows composition: deviation_expr builds a z-score formula entirely out of bmi_expr. You can nest expressions inside expressions to build complex transformations from simple pieces.

  • Save and run 01_basics.py one more time.
  • You should see three new sections of output: the raw BMI values, a table with BMI plus its average plus the literal 25, and then the z-score values.

Seeing a ColumnNotFoundError?

Check that the column names in your pl.col() calls match the dictionary keys in your DataFrame exactly. Column names are case-sensitive.

help me fix a ColumnNotFoundError in my Polars script

✔️ Awesome, I've got everything!

Great. Double check you've saved 01_basics.py and that your output shows the expression object, calculated BMI values, reused expressions, and composed z-scores.

ⓧ I'd like to double check the full code

Here's the complete 01_basics.py file for comparison:

import polars as pl

# === WORKED EXAMPLE: Expressions Are Lazy Building Blocks ===

# Create a DataFrame
df = pl.DataFrame(
    {
        "name": ["Alice", "Ben", "Chloe", "Daniel"],
        "weight": [57.9, 72.5, 53.6, 83.1],  # kg
        "height": [1.56, 1.77, 1.65, 1.75],  # m
    }
)
print("Our DataFrame:")
print(df)
print()

# Define an expression - this is just a DESCRIPTION, not a result
bmi_expr = pl.col("weight") / (pl.col("height") ** 2)
print("The expression object (not computed yet):")
print(bmi_expr)
print()

# Execute the expression in a context to get actual values
result = df.select(bmi=bmi_expr)
print("Expression executed in a 'select' context:")
print(result)
print()

# Expressions are reusable building blocks
result2 = df.select(
    bmi=bmi_expr,
    avg_bmi=bmi_expr.mean(),
    ideal_max_bmi=25,
)
print("Multiple expressions in one context:")
print(result2)
print()

# Key insight: expressions compose - build complex from simple
deviation_expr = (bmi_expr - bmi_expr.mean()) / bmi_expr.std()
result3 = df.select(bmi_zscore=deviation_expr)
print("Composed expression (z-score of BMI):")
print(result3)
Guided exercise: product value expressions

Time to practice. You'll create expressions for a products DataFrame without a worked solution in front of you. Hints are available if you need them.

  • Create a new file for this exercise by running:
touch exercises/basics_guided.py
  • Confirm the file exists by running:
ls exercises/
  • You should see basics_guided.py in the output.
  • Open exercises/basics_guided.py in your text editor and add this starter DataFrame:
import polars as pl

# Products DataFrame
df = pl.DataFrame(
    {
        "product": ["Laptop", "Mouse", "Keyboard", "Monitor", "Webcam"],
        "price": [999.99, 29.99, 79.99, 349.99, 69.99],
        "quantity": [5, 50, 30, 12, 25],
    }
)
print("Products:")
print(df)
print()

# YOUR TASK:
# 1. Create an expression for total_value (price * quantity)
# 2. Create an expression for discounted_price (price * 0.85)
# 3. Apply both in a single df.select() context
#    Include the product name column too!

Hints for this exercise

  • pl.col() references a column by name. Use it to build arithmetic expressions.
  • Expressions are just building blocks until you put them in a context. Try storing them in variables first, then apply them.
  • To keep the product name in your output, include pl.col("product") in your select context.
  • Write your solution, save the file, and run it:
python3 exercises/basics_guided.py
  • You should see a table with product names, total values, and discounted prices.

Give it your best shot before checking the solution.

✔️ I'll try it first

Go for it. Remember: define the expression, then place it in a context.

ⓧ Let me check my work

Here's one way to solve it:

import polars as pl

# Products DataFrame
df = pl.DataFrame(
    {
        "product": ["Laptop", "Mouse", "Keyboard", "Monitor", "Webcam"],
        "price": [999.99, 29.99, 79.99, 349.99, 69.99],
        "quantity": [5, 50, 30, 12, 25],
    }
)
print("Products:")
print(df)
print()

# Define expressions (just descriptions, not results yet)
total_value_expr = pl.col("price") * pl.col("quantity")
discounted_price_expr = pl.col("price") * 0.85

# Apply both in a single select context
result = df.select(
    product=pl.col("product"),
    total_value=total_value_expr,
    discounted_price=discounted_price_expr,
)
print("Result:")
print(result)
Hard challenge: student score statistics

This one has no hints. You have all the knowledge you need from the worked example.

  • Create a new file for the challenge by running:
touch exercises/basics_challenge.py
  • Confirm the file exists by running:
ls exercises/
  • You should see both basics_guided.py and basics_challenge.py in the output.
  • Open exercises/basics_challenge.py and add this starter DataFrame:
import polars as pl

# Students DataFrame
df = pl.DataFrame(
    {
        "student": ["Alice", "Ben", "Chloe", "Daniel", "Eve"],
        "math_score": [92, 78, 85, 95, 88],
        "science_score": [88, 82, 90, 79, 94],
        "english_score": [76, 91, 88, 84, 80],
    }
)
print("Students:")
print(df)
print()

# YOUR CHALLENGE:
# Create expressions that compute:
# 1. The average across all three subjects for each student
# 2. The highest score across all three subjects for each student
# 3. How far each student's average is from the class mean average
# Apply all three in a single df.select() context.
# Include the student name column too!
  • Write your solution, save, and run it:
python3 exercises/basics_challenge.py
  • You should see a table with each student's average, highest score, and deviation from the class mean.

This challenge requires you to combine multiple columns horizontally and compose expressions. Try it before looking at the solution.

✔️ I'll give it a go!

Think about what operations you need. You can add columns together, divide by a literal, and use methods like .mean() to compose bigger expressions.

ⓧ Let me check my work

Here's one possible solution:

import polars as pl

# Students DataFrame
df = pl.DataFrame(
    {
        "student": ["Alice", "Ben", "Chloe", "Daniel", "Eve"],
        "math_score": [92, 78, 85, 95, 88],
        "science_score": [88, 82, 90, 79, 94],
        "english_score": [76, 91, 88, 84, 80],
    }
)
print("Students:")
print(df)
print()

# Expression for the average across all three subjects
avg_expr = (
    (pl.col("math_score") + pl.col("science_score") + pl.col("english_score")) / 3
)

# Expression for the highest score (max of three columns)
highest_expr = pl.max_horizontal(
    "math_score", "science_score", "english_score"
)

# Expression for deviation from the class mean
deviation_expr = avg_expr - avg_expr.mean()

# Apply all three in a single context
result = df.select(
    student=pl.col("student"),
    average=avg_expr,
    highest=highest_expr,
    deviation_from_mean=deviation_expr,
)
print("Result:")
print(result)

You've just internalized the most important concept in Polars: expressions are recipes, contexts are kitchens. Next, you'll learn the four contexts that give expressions their power.

Master the Four Contexts

In the last step, you discovered that expressions are lazy building blocks. They describe what to compute, but don't produce results on their own.

The missing piece is where an expression runs. Polars has four contexts that give expressions meaning: select, with_columns, filter, and group_by. The same expression behaves differently depending on which context you place it in.

In this step, get ready to:
  • Write a worked example demonstrating all four contexts side by side.
  • Complete a guided exercise combining filter, with_columns, and group_by on employee data.
  • Take on a hard challenge aggregating sales data with no hints.
Write the worked example

You will build 02_contexts.py piece by piece, running the script after each section to observe how contexts shape expression output.

  • Create a new file called 02_contexts.py in your polars-lab folder by running:
touch 02_contexts.py
  • Confirm the file exists by running:
ls
  • You should see 01_basics.py, 02_contexts.py, and the exercises/ folder in the output.
  • Open 02_contexts.py in your text editor and add the imports and sample DataFrame:
import polars as pl
from datetime import date

# === WORKED EXAMPLE: The Four Contexts ===

df = pl.DataFrame(
    {
        "name": ["Alice Archer", "Ben Brown", "Chloe Cooper", "Daniel Donovan"],
        "birthdate": [
            date(1997, 1, 10),
            date(1985, 2, 15),
            date(1983, 3, 22),
            date(1981, 4, 30),
        ],
        "weight": [57.9, 72.5, 53.6, 83.1],
        "height": [1.56, 1.77, 1.65, 1.75],
    }
)
print("Original DataFrame:")
print(df)
print()

What does this code do?

This creates a four-person DataFrame with name, birthdate, weight, and height columns. You will apply all four contexts to this single dataset so you can compare their behavior directly.

  • Save 02_contexts.py and run it to confirm the DataFrame prints correctly:
python3 02_contexts.py
  • You should see a table with 4 rows and 4 columns (name, birthdate, weight, height).

Seeing a ModuleNotFoundError?

Make sure you are in the polars-lab directory and that you installed Polars with pip install polars==1.41.0 in the same Python environment you are using to run the script.

Help me fix this ModuleNotFoundError

  • Now add the first two contexts below your existing code. These two are paired because they both produce columns:
# --- CONTEXT 1: select ---
bmi_expr = pl.col("weight") / (pl.col("height") ** 2)
result_select = df.select(
    name=pl.col("name"),
    bmi=bmi_expr,
)
print("select - only the columns you specify exist in the result:")
print(result_select)
print()

# --- CONTEXT 2: with_columns ---
result_with = df.with_columns(
    bmi=bmi_expr,
)
print("with_columns - adds bmi while keeping all original columns:")
print(result_with)
print()

What is the difference between select and with_columns?

select creates a brand new DataFrame containing only the columns you specify. Everything else disappears.

with_columns adds new columns to the existing DataFrame while keeping all the originals intact. Use it when you want to enrich your data without losing anything.

  • Save and run 02_contexts.py again.
  • The select output should show only 2 columns (name, bmi). The with_columns output should show all 5 columns (the original 4 plus bmi).

Both outputs look the same?

Double-check that your select call uses name=pl.col("name") and bmi=bmi_expr as keyword arguments. If you accidentally pass the raw columns without naming them, the output may look different than expected.

Help me debug my select and with_columns outputs

  • Next, add the filter context below your existing code:
# --- CONTEXT 3: filter ---
result_filter = df.filter(
    pl.col("birthdate").is_between(date(1982, 12, 31), date(1996, 1, 1)),
    pl.col("height") > 1.7,
)
print("filter - only rows matching ALL conditions:")
print(result_filter)
print()

How does filter differ from select?

select and with_columns work on columns. filter works on rows. It keeps only the rows where every boolean expression evaluates to True. Passing multiple expressions to filter is an AND condition.

  • Save and run the script. The filter output should show only 1 row (Ben Brown), because he is the only person born between 1983 and 1996 who is also taller than 1.7m.
  • Now add the group_by context:
# --- CONTEXT 4: group_by + agg ---
result_group = df.group_by(
    (pl.col("birthdate").dt.year() // 10 * 10).alias("decade"),
).agg(
    pl.len().alias("count"),
    pl.col("name"),
    pl.col("weight").mean().alias("avg_weight"),
)
print("group_by - rows grouped by decade, aggregated:")
print(result_group)
print()

What does group_by do?

group_by groups rows by the unique values of an expression, then agg applies aggregating expressions to each group. The result has one row per group. Notice that pl.col("name") without an aggregation function collects all names in a group into a list.

The grouping expression (pl.col("birthdate").dt.year() // 10 * 10) computes the decade each person was born in (1980, 1990, etc.).

  • Save and run the script. You should see 2 groups in the output (one for the 1980s decade, one for the 1990s decade), each with a count, a list of names, and an average weight.

Seeing different group order each time?

Polars does not guarantee group order by default. Your groups are correct even if the 1990s row appears before the 1980s row. The data inside each group is still accurate.

Why is my group order different?

  • Finally, add the key insight section that proves the same expression behaves differently in different contexts:
# --- KEY INSIGHT: Same expression, different context, different result ---
weight_mean_expr = pl.col("weight").mean()

in_select = df.select(avg_weight=weight_mean_expr)
print("weight.mean() in select (scalar broadcast to all rows):")
print(in_select)
print()

in_group = df.group_by(
    (pl.col("height") < 1.7).alias("short"),
).agg(avg_weight=weight_mean_expr)
print("weight.mean() in group_by (one value per group):")
print(in_group)

Why is this the most important part?

The expression pl.col("weight").mean() is identical in both cases. In select, it computes a single scalar (the global mean) and broadcasts it to fill 4 rows. In group_by, it computes one mean per group.

The expression is the recipe. The context decides how to interpret it. This is the core of Polars thinking.

  • Save 02_contexts.py and run the final version.
  • The select output shows 4 identical rows (the same mean repeated). The group_by output shows 2 rows (one mean for people shorter than 1.7m, one for taller).

✔️ Awesome, I've got everything!

Great. Double check you have saved 02_contexts.py and that all five sections (select, with_columns, filter, group_by, key insight) appear in your terminal output.

ⓧ I'd like to double check the full code

import polars as pl
from datetime import date

# === WORKED EXAMPLE: The Four Contexts ===

df = pl.DataFrame(
    {
        "name": ["Alice Archer", "Ben Brown", "Chloe Cooper", "Daniel Donovan"],
        "birthdate": [
            date(1997, 1, 10),
            date(1985, 2, 15),
            date(1983, 3, 22),
            date(1981, 4, 30),
        ],
        "weight": [57.9, 72.5, 53.6, 83.1],
        "height": [1.56, 1.77, 1.65, 1.75],
    }
)
print("Original DataFrame:")
print(df)
print()

# --- CONTEXT 1: select ---
bmi_expr = pl.col("weight") / (pl.col("height") ** 2)
result_select = df.select(
    name=pl.col("name"),
    bmi=bmi_expr,
)
print("select - only the columns you specify exist in the result:")
print(result_select)
print()

# --- CONTEXT 2: with_columns ---
result_with = df.with_columns(
    bmi=bmi_expr,
)
print("with_columns - adds bmi while keeping all original columns:")
print(result_with)
print()

# --- CONTEXT 3: filter ---
result_filter = df.filter(
    pl.col("birthdate").is_between(date(1982, 12, 31), date(1996, 1, 1)),
    pl.col("height") > 1.7,
)
print("filter - only rows matching ALL conditions:")
print(result_filter)
print()

# --- CONTEXT 4: group_by + agg ---
result_group = df.group_by(
    (pl.col("birthdate").dt.year() // 10 * 10).alias("decade"),
).agg(
    pl.len().alias("count"),
    pl.col("name"),
    pl.col("weight").mean().alias("avg_weight"),
)
print("group_by - rows grouped by decade, aggregated:")
print(result_group)
print()

# --- KEY INSIGHT: Same expression, different context, different result ---
weight_mean_expr = pl.col("weight").mean()

in_select = df.select(avg_weight=weight_mean_expr)
print("weight.mean() in select (scalar broadcast to all rows):")
print(in_select)
print()

in_group = df.group_by(
    (pl.col("height") < 1.7).alias("short"),
).agg(avg_weight=weight_mean_expr)
print("weight.mean() in group_by (one value per group):")
print(in_group)
Complete the guided exercise

Time to practice without copying code. You will combine three contexts to analyze employee data.

Create a new file called exercise_contexts.py inside your exercises/ folder. In it, create an employee DataFrame with these columns: name, department, salary, and years_employed. Invent 6-8 rows of sample data.

  • Use filter to find employees whose salary is above the mean salary.
  • Use with_columns to add a seniority column: if years_employed is greater than 5, the value is "Senior". Otherwise it is "Junior".
  • Use group_by on department to compute average salary and count of employees per department.

✔️ I'll give it a go!

Hints to get started

  • For conditional logic, use pl.when(...).then(...).otherwise(...).
  • For counting rows in a group, use pl.len().
  • Remember that pl.col("salary").mean() inside filter compares each row's salary against the global mean.

ⓧ Let me check my work

Here is one way to solve the exercise. Your column names and data values may differ.

import polars as pl

# Sample employee data
df = pl.DataFrame(
    {
        "name": ["Alice", "Ben", "Chloe", "Dan", "Eve", "Frank"],
        "department": ["Eng", "Eng", "Sales", "Sales", "Eng", "Sales"],
        "salary": [95000, 78000, 65000, 72000, 110000, 58000],
        "years_employed": [8, 3, 6, 2, 12, 4],
    }
)

# Filter: salary above the mean
high_earners = df.filter(pl.col("salary") > pl.col("salary").mean())
print("Employees above mean salary:")
print(high_earners)
print()

# with_columns: add seniority
with_seniority = df.with_columns(
    seniority=pl.when(pl.col("years_employed") > 5)
    .then(pl.lit("Senior"))
    .otherwise(pl.lit("Junior")),
)
print("With seniority column:")
print(with_seniority)
print()

# group_by: stats per department
by_dept = df.group_by("department").agg(
    pl.col("salary").mean().alias("avg_salary"),
    pl.len().alias("employee_count"),
)
print("Department stats:")
print(by_dept)
Take on the hard challenge

No hints this time. This challenge tests whether you can combine grouping with multiple aggregation expressions in a single chain.

Create a file called challenge_contexts.py in your exercises/ folder. Build a sales DataFrame with columns date, product, region, amount, and quantity. Create at least 10 rows with 3+ regions and 3+ products.

  • In a single chain, group by region and compute: total revenue (sum of amount), average order size (mean of amount), the best-selling product (product with highest total quantity), and the number of unique products sold.

✔️ I'll try it first

Give it your best shot. Once you have a working script, run it and confirm you get one row per region with all four computed columns.

ⓧ Let me check my work

Here is one approach. Your data values will differ but the structure should be similar.

import polars as pl

df = pl.DataFrame(
    {
        "date": ["2025-01-01"] * 4 + ["2025-01-02"] * 4 + ["2025-01-03"] * 4,
        "product": ["Widget", "Gadget", "Widget", "Doohickey"] * 3,
        "region": ["North", "North", "South", "South"] * 3,
        "amount": [100, 200, 150, 80, 120, 180, 90, 70, 110, 210, 140, 60],
        "quantity": [5, 3, 7, 2, 6, 4, 8, 1, 5, 3, 9, 2],
    }
)

result = df.group_by("region").agg(
    pl.col("amount").sum().alias("total_revenue"),
    pl.col("amount").mean().alias("avg_order_size"),
    pl.col("product").sort_by("quantity", descending=True).first().alias("best_seller"),
    pl.col("product").n_unique().alias("unique_products"),
)
print(result)

You now have a solid grasp of where expressions run and how context shapes their meaning. Next up, you will discover when expressions run by exploring lazy evaluation and query optimization.

Unlock Lazy Evaluation and Query Optimization

You now know how to write expressions and place them in contexts. But so far, every operation has executed immediately. What if Polars could look at your entire pipeline first and find shortcuts before doing any work?

That is exactly what lazy evaluation does. Instead of executing each step the moment you write it, Polars builds a query plan and optimizes it as a whole. In this step, you will see how the optimizer eliminates unnecessary work automatically.

In this step, get ready to:
  • Write a script that compares eager and lazy execution side by side.
  • Read an optimized query plan and identify predicate pushdown and projection pushdown.
  • Complete exercises that deepen your understanding of lazy pipelines.
Write the lazy evaluation script

This worked example demonstrates the difference between eager execution (everything runs immediately) and lazy execution (Polars builds a plan, optimizes it, then runs it once).

  • Create a new file called 03_lazy.py in your polars-lab folder by running:
touch 03_lazy.py
  • Confirm the file exists by running:
ls

You should see 03_lazy.py listed alongside 01_basics.py and 02_contexts.py.

  • Open 03_lazy.py in your text editor and add the imports and sample data setup:
import polars as pl
import os

# === WORKED EXAMPLE: Lazy Evaluation ===

# First, create a sample CSV to work with
sample_data = pl.DataFrame(
    {
        "sepal_length": [5.1, 4.9, 7.0, 6.4, 5.9, 5.0, 4.8, 6.7, 5.5, 6.3],
        "sepal_width": [3.5, 3.0, 3.2, 3.2, 3.0, 3.6, 3.4, 3.1, 2.3, 3.3],
        "petal_length": [1.4, 1.4, 4.7, 4.5, 4.2, 1.4, 1.6, 4.4, 4.0, 6.0],
        "petal_width": [0.2, 0.2, 1.4, 1.5, 1.5, 0.2, 0.2, 1.4, 1.3, 2.5],
        "species": [
            "setosa", "setosa", "versicolor", "versicolor", "versicolor",
            "setosa", "setosa", "versicolor", "versicolor", "virginica",
        ],
    }
)
sample_data.write_csv("iris_sample.csv")

What does this code do?

This creates a small dataset with 5 columns (sepal_length, sepal_width, petal_length, petal_width, species) and writes it to a CSV file. Having a real file on disk lets us demonstrate how Polars optimizes file reading.

The os import is for cleaning up the CSV at the end of the script.

  • Below the CSV creation, add the eager approach section:
# --- EAGER approach ---
print("=== EAGER (every step executes immediately) ===")
df = pl.read_csv("iris_sample.csv")
df_filtered = df.filter(pl.col("sepal_length") > 5)
df_result = df_filtered.group_by("species").agg(pl.col("sepal_width").mean())
print(df_result)
print()

What does the eager approach do?

Each line executes the moment Python hits it. pl.read_csv() reads ALL 5 columns and ALL 10 rows into memory immediately. Then .filter() creates a second copy with only matching rows. Then .group_by().agg() creates yet another result.

With a 10-row dataset this is fine. With millions of rows and dozens of columns, reading everything upfront wastes time and memory.

  • Now add the lazy approach section below the eager code:
# --- LAZY approach ---
print("=== LAZY (build plan, optimize, execute once) ===")
q = (
    pl.scan_csv("iris_sample.csv")
    .filter(pl.col("sepal_length") > 5)
    .group_by("species")
    .agg(pl.col("sepal_width").mean())
)

print("Query plan (notice the optimizations):")
print(q.explain())
print()

result = q.collect()
print("Result (same as eager, but Polars read fewer columns and rows):")
print(result)
print()

What does the lazy approach do?

pl.scan_csv() does NOT read the file yet. It creates a LazyFrame. Each method after it (.filter(), .group_by(), .agg()) adds a step to the query plan without executing anything.

q.explain() prints the optimized plan so you can see what Polars will actually do. Look for SELECTION (predicate pushdown) and PROJECT (projection pushdown) in the output.

q.collect() finally executes the entire optimized plan in one pass. Polars reads only the columns it needs and filters rows during reading.

  • Finally, add the section that reveals eager uses lazy under the hood, plus cleanup:
# --- The eager API secretly uses lazy under the hood ---
print("=== Even eager uses lazy internally ===")
eager_result = df.filter(pl.col("sepal_length") > 5).group_by("species").agg(
    pl.col("sepal_width").mean()
)
print(eager_result)

# Cleanup
os.remove("iris_sample.csv")

Why show that eager uses lazy internally?

The eager API actually calls the lazy API under the hood and immediately collects the result. The difference is that pl.scan_csv() gives the optimizer maximum freedom because it can push filters and projections all the way down into the file reading step.

With pl.read_csv(), the file is already fully loaded before any optimization can happen.

  • Save 03_lazy.py and run it:
python3 03_lazy.py

You should see three sections of output. The most important is the query plan printed between the eager and lazy results.

What should I see in the query plan?

The query plan output will contain lines like PROJECT (showing how many columns Polars actually reads) and SELECTION (showing the filter applied during reading). These are the two key optimizations:

  • Predicate pushdown: the filter condition is pushed down to the file-reading step, so rows that don't match are never loaded into memory.
  • Projection pushdown: only the columns needed for the final result are read from the file.

Seeing a FileNotFoundError?

Make sure you are running the script from inside your polars-lab folder. The script creates iris_sample.csv in the current directory, so it needs to also read from the current directory.

If you ran the script before and it failed partway through, the CSV may have already been deleted by os.remove(). Run the full script again from the top.

help me debug a FileNotFoundError when running my Polars lazy evaluation script

✔️ Awesome, I've got everything!

Great. Double check you have saved 03_lazy.py and observed the query plan output.

ⓧ I'd like to double check the full code

import polars as pl
import os

# === WORKED EXAMPLE: Lazy Evaluation ===

# First, create a sample CSV to work with
sample_data = pl.DataFrame(
    {
        "sepal_length": [5.1, 4.9, 7.0, 6.4, 5.9, 5.0, 4.8, 6.7, 5.5, 6.3],
        "sepal_width": [3.5, 3.0, 3.2, 3.2, 3.0, 3.6, 3.4, 3.1, 2.3, 3.3],
        "petal_length": [1.4, 1.4, 4.7, 4.5, 4.2, 1.4, 1.6, 4.4, 4.0, 6.0],
        "petal_width": [0.2, 0.2, 1.4, 1.5, 1.5, 0.2, 0.2, 1.4, 1.3, 2.5],
        "species": [
            "setosa", "setosa", "versicolor", "versicolor", "versicolor",
            "setosa", "setosa", "versicolor", "versicolor", "virginica",
        ],
    }
)
sample_data.write_csv("iris_sample.csv")

# --- EAGER approach ---
print("=== EAGER (every step executes immediately) ===")
df = pl.read_csv("iris_sample.csv")
df_filtered = df.filter(pl.col("sepal_length") > 5)
df_result = df_filtered.group_by("species").agg(pl.col("sepal_width").mean())
print(df_result)
print()

# --- LAZY approach ---
print("=== LAZY (build plan, optimize, execute once) ===")
q = (
    pl.scan_csv("iris_sample.csv")
    .filter(pl.col("sepal_length") > 5)
    .group_by("species")
    .agg(pl.col("sepal_width").mean())
)

print("Query plan (notice the optimizations):")
print(q.explain())
print()

result = q.collect()
print("Result (same as eager, but Polars read fewer columns and rows):")
print(result)
print()

# --- The eager API secretly uses lazy under the hood ---
print("=== Even eager uses lazy internally ===")
eager_result = df.filter(pl.col("sepal_length") > 5).group_by("species").agg(
    pl.col("sepal_width").mean()
)
print(eager_result)

# Cleanup
os.remove("iris_sample.csv")
Guided exercise: compare query plans with streaming

Now it is your turn to build a lazy pipeline and observe the optimizer in action. You will create a CSV with 6 columns, then write a lazy query that only uses 2 of them.

  • Create a new file for your exercise by running:
touch exercises/lazy_exercise.py

Your task is to write a script that does the following:

  • Create a DataFrame with 6 columns (for example: id, name, category, price, quantity, rating) and write it to a CSV.
  • Use pl.scan_csv() to read it lazily.
  • Filter rows based on one column (e.g. price > 20).
  • Aggregate using only one other column (e.g. average rating by category).
  • Print the query plan with .explain() and observe how many columns Polars actually reads.
  • Collect the result with .collect(engine="streaming") and confirm it produces the same output.

✔️ I'll give it a go!

Write your solution in exercises/lazy_exercise.py. Run it with python3 exercises/lazy_exercise.py and check the query plan output.

ⓧ Maybe a little help...

Here are some hints to get you started:

  • pl.scan_csv() starts the lazy pipeline. Every method call after it (.filter(), .group_by(), .agg()) builds the plan without executing.
  • Only .collect() triggers execution. Everything before it is just planning.
  • In the query plan output, look for PROJECT followed by a fraction like 3/6 COLUMNS. That tells you Polars skipped reading 3 columns entirely.
  • To use streaming, replace .collect() with .collect(engine="streaming"). The result is identical, but this mode can handle datasets larger than your available RAM.

Your script structure should look like: create DataFrame, write CSV, scan CSV lazily, filter, group_by, agg, explain, collect.

Hard challenge: eager vs lazy pipeline with a join

This challenge has no hints. You will write both an eager and a lazy version of the same multi-step pipeline to see how the optimizer handles a more complex query.

  • Create a new file for your challenge by running:
touch exercises/lazy_challenge.py

Your challenge:

  • Create two DataFrames (for example: orders and customers). Write both to CSV files.
  • Write an eager version: read both CSVs with pl.read_csv(), filter one, join them together, then aggregate.
  • Write a lazy version: scan both CSVs with pl.scan_csv(), build the same pipeline (filter, join, aggregate), print the plan with .explain(), then .collect().
  • Study the query plan output. Identify which optimizations Polars applied (predicate pushdown, projection pushdown, or both).
  • Verify both versions produce the same result.
  • Run your challenge script with:
python3 exercises/lazy_challenge.py

What to look for in the join query plan

When a lazy pipeline includes a join, the optimizer can push filters down to run before the join happens. This means fewer rows enter the join, which is a major performance win on large datasets.

Look for the filter appearing below the join in the plan output. That confirms predicate pushdown worked across the join boundary.

You have seen how Polars defers execution to optimize your entire pipeline as a unit. Next up, you will explore expression expansion and window functions to write even more powerful transformations with less code.

Apply Expression Expansion and Window Functions

In Step 4, you learned how lazy evaluation lets Polars optimize entire pipelines before executing them. Now you have one more power tool to unlock: shortcuts that let you apply the same transformation across many columns at once, and compute per-group values without collapsing your DataFrame.

These two features solve a common pain point. Without expression expansion, you would write the same expression once per column. Without window functions, you would have to group, aggregate, and then join results back manually. Polars gives you one-line solutions for both.

In this step, get ready to:
  • Run a worked example demonstrating expression expansion and window functions.
  • Complete a guided exercise using expansion and .over() to normalize columns and compute rolling differences.
  • Tackle a hard challenge computing z-scores, anomaly flags, and running counts in one chain.
Run the expression expansion and window function examples

Expression expansion is shorthand for applying the same transformation to multiple columns in one expression. Window functions (using .over()) compute per-group aggregations while keeping all original rows intact.

  • Create a new file called 04_advanced.py in your polars-lab folder by running:
touch 04_advanced.py
  • Confirm the file exists by running:
ls

You should see 04_advanced.py alongside your other scripts.

  • Open 04_advanced.py in your text editor and paste the following code to set up the DataFrame and demonstrate expression expansion:
import polars as pl

# === WORKED EXAMPLE: Expression Expansion and Window Functions ===

df = pl.DataFrame(
    {
        "name": ["Alice", "Ben", "Chloe", "Daniel", "Eve", "Frank"],
        "group": ["A", "A", "B", "B", "A", "B"],
        "weight": [57.9, 72.5, 53.6, 83.1, 61.0, 75.2],
        "height": [1.56, 1.77, 1.65, 1.75, 1.62, 1.80],
        "age": [27, 39, 41, 43, 30, 35],
    }
)
print("Original DataFrame:")
print(df)
print()

# --- EXPRESSION EXPANSION ---
# Apply the same operation to multiple columns at once

# Select multiple columns by name
print("=== Expression Expansion ===")
result = df.select(pl.col("weight", "height").mean())
print("Mean of weight AND height (one expression, two results):")
print(result)
print()

# Select columns by data type
result2 = df.select((pl.col(pl.Float64) * 1.1).name.suffix("_inflated"))
print("All Float64 columns multiplied by 1.1:")
print(result2)
print()

# Rename expanded results with .name methods
result3 = df.select(
    pl.col("weight", "height", "age").mean().name.prefix("avg_")
)
print("Prefix-renamed averages:")
print(result3)
print()

What does this code do?

This first section demonstrates three forms of expression expansion:

  • pl.col("weight", "height").mean() applies .mean() to both columns with one expression. Polars expands it internally into two separate expressions.
  • pl.col(pl.Float64) selects all columns matching the Float64 data type dynamically. You do not need to know column names in advance.
  • .name.suffix() and .name.prefix() rename the results of expanded expressions so they do not overwrite the originals.
  • Below the expression expansion code, add the window functions section:
# --- WINDOW FUNCTIONS with .over() ---
# group_by collapses rows. .over() keeps all rows but computes per-group.

print("=== Window Functions (.over) ===")
result_window = df.with_columns(
    group_avg_weight=pl.col("weight").mean().over("group"),
    weight_pct_of_group=pl.col("weight") / pl.col("weight").sum().over("group"),
    rank_in_group=pl.col("weight").rank().over("group"),
)
print("Window functions - per-group stats WITHOUT collapsing rows:")
print(result_window)
print()

# Key insight: .over("group") is like an invisible group_by that returns
# values back to each original row. The DataFrame shape doesn't change.

What is .over() doing here?

The .over("group") method is like group_by that does not collapse your rows. It computes the aggregation per group, then maps each result back to the original rows.

  • pl.col("weight").mean().over("group") computes the average weight within each group and broadcasts it back to every row in that group.
  • pl.col("weight").rank().over("group") ranks each person's weight within their group only.
  • The DataFrame keeps all 6 rows. No information is lost.
  • Finally, add the section that combines expression expansion with window functions and conditional logic:
# --- COMBINING EVERYTHING ---
print("=== Combining Patterns ===")
result_combined = df.with_columns(
    # Expression expansion + window function
    pl.col("weight", "height").mean().over("group").name.suffix("_group_avg"),
    # Conditional logic
    pl.when(pl.col("age") > 35)
    .then(pl.lit("Senior"))
    .otherwise(pl.lit("Junior"))
    .alias("seniority"),
    # Computed column using other columns
    bmi=(pl.col("weight") / (pl.col("height") ** 2)),
)
print("Multiple advanced patterns in one with_columns call:")
print(result_combined)

What does this code do?

This final section shows that all the patterns you have learned compose together inside a single with_columns call:

  • Expression expansion + window function: pl.col("weight", "height").mean().over("group") computes the group average for both columns in one expression.
  • Conditional logic: pl.when(...).then(...).otherwise(...) creates a new column based on a condition.
  • Computed column: the BMI formula from earlier steps works here too.

Polars evaluates all separate expressions within a context in parallel automatically.

  • Save 04_advanced.py.
  • Run the script by executing:
python3 04_advanced.py

You should see four output sections: the original DataFrame, expression expansion results, window function results (all 6 rows with new columns), and the combined patterns output.

Seeing a column not found error?

  • Check that your DataFrame includes all five columns: name, group, weight, height, and age.
  • Make sure .over("group") references the column name exactly as defined in your DataFrame (lowercase group).
  • Verify there are no typos in .name.suffix("_group_avg") or .name.prefix("avg_").

Help me debug my 04_advanced.py expression expansion or window function error.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

Here is the complete 04_advanced.py file for comparison:

import polars as pl

# === WORKED EXAMPLE: Expression Expansion and Window Functions ===

df = pl.DataFrame(
    {
        "name": ["Alice", "Ben", "Chloe", "Daniel", "Eve", "Frank"],
        "group": ["A", "A", "B", "B", "A", "B"],
        "weight": [57.9, 72.5, 53.6, 83.1, 61.0, 75.2],
        "height": [1.56, 1.77, 1.65, 1.75, 1.62, 1.80],
        "age": [27, 39, 41, 43, 30, 35],
    }
)
print("Original DataFrame:")
print(df)
print()

# --- EXPRESSION EXPANSION ---
print("=== Expression Expansion ===")
result = df.select(pl.col("weight", "height").mean())
print("Mean of weight AND height (one expression, two results):")
print(result)
print()

result2 = df.select((pl.col(pl.Float64) * 1.1).name.suffix("_inflated"))
print("All Float64 columns multiplied by 1.1:")
print(result2)
print()

result3 = df.select(
    pl.col("weight", "height", "age").mean().name.prefix("avg_")
)
print("Prefix-renamed averages:")
print(result3)
print()

# --- WINDOW FUNCTIONS with .over() ---
print("=== Window Functions (.over) ===")
result_window = df.with_columns(
    group_avg_weight=pl.col("weight").mean().over("group"),
    weight_pct_of_group=pl.col("weight") / pl.col("weight").sum().over("group"),
    rank_in_group=pl.col("weight").rank().over("group"),
)
print("Window functions - per-group stats WITHOUT collapsing rows:")
print(result_window)
print()

# --- COMBINING EVERYTHING ---
print("=== Combining Patterns ===")
result_combined = df.with_columns(
    pl.col("weight", "height").mean().over("group").name.suffix("_group_avg"),
    pl.when(pl.col("age") > 35)
    .then(pl.lit("Senior"))
    .otherwise(pl.lit("Junior"))
    .alias("seniority"),
    bmi=(pl.col("weight") / (pl.col("height") ** 2)),
)
print("Multiple advanced patterns in one with_columns call:")
print(result_combined)
Complete the guided exercise

Now it is your turn to apply expression expansion and window functions to a new problem. You will normalize numeric columns and compute per-category differences.

  • Create a new file for your exercise solution:
touch exercises/expansion_exercise.py

Here is the challenge:

  • Create a DataFrame with numeric columns col_a, col_b, col_c and a category column.
  • Normalize all numeric columns to a 0-1 range using expression expansion (subtract the min, then divide by max minus min).
  • Compute a rolling difference per category using .over().

✔️ I'll give it a go!

Here are some hints to get you started:

  • pl.col(pl.Float64) selects all float columns by their data type.
  • The normalization formula is: (value - min) / (max - min).
  • .over("category") makes any aggregation return one value per row, partitioned by category.
  • For the rolling difference, look at .diff() combined with .over().

Save your solution in exercises/expansion_exercise.py and run it to verify the output.

ⓧ Maybe a little help...

Here is a worked solution you can study:

import polars as pl

# Create the DataFrame
df = pl.DataFrame(
    {
        "category": ["X", "X", "X", "Y", "Y", "Y"],
        "col_a": [10.0, 20.0, 30.0, 5.0, 15.0, 25.0],
        "col_b": [100.0, 200.0, 300.0, 50.0, 150.0, 250.0],
        "col_c": [1.0, 2.0, 3.0, 0.5, 1.5, 2.5],
    }
)
print("Original:")
print(df)
print()

# Normalize all float columns to 0-1 range using expression expansion
normalized = df.with_columns(
    ((pl.col(pl.Float64) - pl.col(pl.Float64).min())
     / (pl.col(pl.Float64).max() - pl.col(pl.Float64).min())
    ).name.suffix("_norm")
)
print("Normalized (0-1 range):")
print(normalized)
print()

# Compute rolling difference per category using .over()
with_diff = df.with_columns(
    pl.col("col_a").diff().over("category").alias("col_a_diff_in_category")
)
print("Rolling difference per category:")
print(with_diff)

What does this solution do?

  • pl.col(pl.Float64) expands to all three numeric columns at once.
  • The normalization formula (col - min) / (max - min) is applied to each float column identically through expansion.
  • .diff().over("category") computes the difference between consecutive values, but only within each category group.
Take on the hard challenge

This challenge tests whether you can compose expressions, window functions, and expansion into a single chain with no Python loops and no intermediate variables.

  • Create a new file for your challenge solution:
touch exercises/advanced_challenge.py

Here is the challenge:

  • Create a time-series DataFrame with columns timestamp, sensor_id, reading, and location.
  • Compute the z-score of each reading within its sensor group.
  • Add a flag for anomalies (z-score greater than 2).
  • Add a running count of anomalies per sensor.
  • Output everything in a single chained operation. No Python loops. No intermediate variables.

What you are proving

This challenge demonstrates that you can think entirely in Polars' expression-based paradigm. Every computation is expressed as a combination of expressions, contexts, and .over() partitioning.

There are no hints for this one. Use what you have learned across all four scripts to compose the solution.

  • Save your solution in exercises/advanced_challenge.py and run it to verify the output makes sense.

Secret mission

Zero-Variable Pipeline Challenge

You've written expressions, used all four contexts, and built lazy pipelines. Now prove you can think entirely in Polars' native paradigm. Build a complete data analysis pipeline as a single chained Python expression with zero intermediate variables.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • The polars-lab/ project folder on your Desktop (contains all Python scripts and the exercises/ subfolder).
  • The Polars Python package (installed via pip).

Keep everything running

No action needed. Choose this if you're still actively experimenting with your scripts or plan to add more exercises.

  • Your polars-lab/ folder stays on your Desktop, ready for you to revisit anytime.
  • Polars remains installed and available for any future Python scripts you write.
  • Python stays installed on your system (useful for many other projects).

Pause - I'll come back to this later

Since this project is entirely local with no running processes, there is nothing to pause. Your files sit idle on disk and consume no resources beyond storage.

  • Your scripts are ready to run again whenever you return. Just open a terminal, navigate to polars-lab/ on your Desktop, and run them.

Delete - I don't want to use this again

Remove all project files and uninstall the Polars package to start fresh.

  • Delete the project folder by running these commands in your terminal:

macOS

cd ~/Desktop
rm -rf polars-lab

Windows

cd ~\Desktop
rmdir /s /q polars-lab
  • Uninstall the Polars package by running:
pip uninstall polars
  • Confirm the removal by typing y when prompted.

That's a Wrap!

That's a Wrap!

Nice work! You've built a complete mental model of Polars' expression-based philosophy and a library of runnable scripts to prove it.

You've learned how to:

  • Think in expressions as lazy building blocks and apply them in the four contexts (select, with_columns, filter, group_by) to produce different results from the same expression.
  • Use lazy evaluation with pl.scan_csv() and .collect() to let Polars optimize your pipelines with predicate pushdown and projection pushdown.
  • Apply expression expansion across multiple columns and use window functions with .over() to compute group-level statistics without collapsing rows.
  • Built a zero-variable pipeline that chains scan, filter, transform, aggregate, and sort into a single Python expression, proving you can think entirely in Polars' native paradigm.

Ready to quiz yourself?