Build a Web Scraping ETL Pipeline

Scrape, clean, and export book data using Python, BeautifulSoup, and pandas.

Introduction

30 Second Summary

Every day, useful data sits on websites in messy, unstructured formats. Getting that data into a clean spreadsheet you can actually analyze requires a pipeline that fetches it, cleans it, and exports it automatically.

In this project, you will build a Python data pipeline that scrapes book listings from a live website, transforms the raw data into properly typed columns, and saves a polished CSV file ready for analysis.

What You'll Build

A complete ETL data pipeline in Python that extracts book listings from a web scraping sandbox site, transforms messy strings into properly typed data, and loads the results into a clean CSV file.

By the end of this project, you'll have:

  • A working Python script that scrapes 100 books from a live website using requests and BeautifulSoup.
  • A clean CSV file with properly typed columns including floats for prices, integers for ratings, and booleans for availability.
  • A reusable, modular pipeline with separate extract, transform, and load functions that you can adapt to other scraping targets.
  • Secret Mission: Scale your pipeline to scrape all 1000 books with polite rate limiting, error handling, and a price-by-rating analysis report.

Are there any prerequisites?

You should be comfortable with Python fundamentals including functions, loops, dictionaries, and lists. No prior experience with web scraping or pandas is needed.

Set Up Your Project

Every data pipeline starts with a clean workspace and the right tools installed. Before you write any scraping logic, you need a dedicated project folder, a Python file, and the three libraries that power each stage of the pipeline.

In this step, you'll set up your environment and install requests for fetching web pages, beautifulsoup4 for parsing HTML, and pandas for data manipulation.

In this step, get ready to:
  • Create a project folder and open it in VS Code.
  • Install the required Python packages.
  • Verify everything works by running your first imports.
Create your project folder and files

You need a dedicated folder for your pipeline code and the CSV file it will produce.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.
  • Open the integrated terminal by pressing Ctrl+` (the backtick key, above Tab on most keyboards).

Where is the backtick key?

On most keyboards, the backtick (`) is directly above the Tab key and to the left of the number 1. This shortcut works on both macOS and Windows.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir book-scraper
cd book-scraper
  • Now open this folder in VS Code by clicking File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the book-scraper folder, and click Open.

VS Code reloads with book-scraper as the workspace. The terminal also resets to this folder.

  • Create the pipeline file by running this command in the terminal:
touch pipeline.py
  • Confirm the file exists by running:
ls
  • You should see pipeline.py in the output.
  • Click on pipeline.py in the Explorer sidebar on the left to open it in the editor.

Don't see pipeline.py in the sidebar?

Make sure you opened the book-scraper folder (not a parent folder) via File > Open Folder. The Explorer sidebar shows files relative to the opened folder.

On Windows, if touch is not recognized, open the terminal and run New-Item pipeline.py instead.

Help me create pipeline.py in my VS Code project

Install the required packages

Your pipeline depends on three libraries that don't come with Python by default. You'll install them all at once using pip, Python's package manager.

  • Install all three packages by running this command in the terminal:
pip install requests==2.34.2 beautifulsoup4==4.15.0 pandas==2.3.3

What are these packages?

requests sends HTTP requests to websites and returns the HTML content. It handles all the networking for you.

beautifulsoup4 parses that HTML into a navigable tree structure, so you can extract specific elements like titles and prices.

pandas provides DataFrames for organizing, cleaning, and exporting tabular data. It's the standard tool for data manipulation in Python.

  • Wait for the installation to finish. You should see Successfully installed followed by a list of packages in the terminal output.

Seeing a 'pip not found' error?

Try running pip3 instead of pip. On some systems, Python 3's package manager is only available as pip3.

If that doesn't work either, try python -m pip install requests==2.34.2 beautifulsoup4==4.15.0 pandas==2.3.3 to invoke pip through Python directly.

If you see errors about version compatibility or Python version, your Python installation may be older than 3.10. Run python --version to check, and upgrade Python if needed.

Help me fix my pip installation error

Verify the installation

The quickest way to confirm everything installed correctly is to import the packages in your Python file and run it. If Python can find all three libraries, the script runs silently with no errors.

  • In pipeline.py, add the following import statements:
import requests
from bs4 import BeautifulSoup
import pandas as pd

What does this code do?

import requests loads the HTTP library you'll use to fetch web pages.

from bs4 import BeautifulSoup loads the HTML parser from the beautifulsoup4 package. The package is named beautifulsoup4 but you import from bs4.

import pandas as pd loads the data manipulation library. The alias pd is the universal convention in data work.

  • Save pipeline.py.
  • Run the file to verify the imports work by running this command:
python pipeline.py
  • If the command completes with no output and no errors, all three packages are installed correctly.

Seeing a ModuleNotFoundError?

This means Python can't find one of the packages you installed. The error message tells you which one is missing.

Try running python3 pipeline.py instead. Sometimes python points to a different Python installation than the one pip installed packages into.

If the error persists, reinstall the missing package using the same Python: python3 -m pip install requests beautifulsoup4 pandas.

Help me fix my ModuleNotFoundError

✔️ Awesome, I've got everything!

Great. Double check you've saved pipeline.py and you're ready to start building the extract stage.

ⓧ I'd like to double check the full code

Here's what your pipeline.py should look like at this point:

import requests
from bs4 import BeautifulSoup
import pandas as pd

Your workspace is ready and all three libraries are installed. Next up, you'll build the extraction function that reaches out to the web and pulls raw book data from HTML pages.

Extract - Scrape Book Data

Your packages are installed and your project file is ready. Now it's time to build the first stage of your pipeline: extraction.

The extract stage pulls raw data from an external source. In this step, you'll scrape book listings from books.toscrape.com, a sandbox website designed specifically for practicing web scraping. The site has 1000 books across 50 pages with predictable HTML structure, making it perfect for learning.

In this step, get ready to:
  • Write a function to scrape a single page of book listings.
  • Write a function to loop through multiple pages.
  • Test the extraction and confirm the raw data structure.
Write the scrape_page() function

Each page on books.toscrape.com contains 20 book listings. Your first function needs to fetch a single page, parse its HTML, and pull out the data for every book on that page.

  • In pipeline.py, add the following function below your import statements:
def scrape_page(url):
    """Scrape all books from a single page."""
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) BookScraper/1.0"
    }
    response = requests.get(url, headers=headers)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")
    articles = soup.find_all("article", class_="product_pod")

    books = []
    for article in articles:
        title = article.find("h3").find("a")["title"]
        price = article.find("p", class_="price_color").text
        rating = article.find("p", class_="star-rating")["class"][1]
        availability = article.find("p", class_="instock availability").text.strip()

        books.append({
            "title": title,
            "price": price,
            "rating": rating,
            "availability": availability,
        })

    return books

What does this code do?

The headers dictionary sets a User-Agent string so the website knows your scraper is a bot, not a browser pretending to be human.

requests.get() fetches the page's HTML. raise_for_status() throws an error if the request fails (like a 404 or 500 response), so your pipeline doesn't silently process garbage.

BeautifulSoup parses the raw HTML into a navigable tree. find_all("article", class_="product_pod") grabs every book container on the page. Then the loop extracts four fields from each container: title, price, rating, and availability.

  • Save pipeline.py.

You won't see any output yet. The next function will call scrape_page() to put it to work.

Seeing a syntax error when you save?

  • Check that the function is indented consistently. Every line inside the function should use 4 spaces (no tabs mixed in).
  • Make sure you have a blank line between your import statements and the def scrape_page(url): line.
  • Verify that the books.append({...}) block has the closing } and ) on separate lines inside the loop.

Why is my scrape_page() function showing a syntax error in VS Code?

Write the scrape_books() function

A single page only gives you 20 books. You need a second function that loops through multiple pages and collects all the results into one big list.

  • Below your scrape_page() function in pipeline.py, add the following function:
def scrape_books(num_pages=5):
    """Extract book data from multiple pages."""
    all_books = []

    for page_num in range(1, num_pages + 1):
        url = f"https://books.toscrape.com/catalogue/page-{page_num}.html"
        print(f"Scraping page {page_num}...")
        page_books = scrape_page(url)
        all_books.extend(page_books)

    print(f"Extraction complete: {len(all_books)} books scraped.")
    return all_books

What does this code do?

scrape_books() defaults to scraping 5 pages (100 books total). It constructs each page's URL using the pattern https://books.toscrape.com/catalogue/page-{page_num}.html and calls your scrape_page() function for each one.

all_books.extend(page_books) adds every book from the current page into the master list. The print() calls give you progress updates so you know scraping is working while it runs.

  • Save pipeline.py.

You won't see output yet. Next, you'll add test code to run the pipeline and see real data.

VS Code showing an indentation warning?

  • Make sure there is one blank line between the end of scrape_page() and the def scrape_books(num_pages=5): line.
  • The f before each string (like f"Scraping page {page_num}...") is required for the curly-brace substitution to work.

Why is my scrape_books() function showing an error?

Test the extract stage

Both functions are written. Time to run them and confirm you're pulling real book data from the website.

  • At the bottom of pipeline.py, add the following test code:
# Test
raw = scrape_books()
print(raw[:3])

What does this test do?

This calls scrape_books() which triggers the full extraction across 5 pages. Then print(raw[:3]) shows the first 3 raw book dictionaries so you can verify the data structure before moving on.

  • Save pipeline.py.
  • Run the script in your terminal by running:
python pipeline.py

You should see progress messages for each page, followed by 3 dictionaries containing raw book data with keys title, price, rating, and availability.

Not seeing any output or getting an error?

  • If you see ModuleNotFoundError, double check that you installed the packages in Step 1. Run pip install requests beautifulsoup4 pandas again.
  • If you see ConnectionError, check your internet connection. The script needs to reach books.toscrape.com to fetch data.
  • If you see IndentationError, make sure every line inside each function uses exactly 4 spaces. Mixing tabs and spaces causes this.

Help me fix an error when running my web scraping pipeline.

✔️ Awesome, I've got everything!

Great. Double check you've saved pipeline.py and your terminal shows 3 raw dictionaries.

ⓧ I'd like to double check the full code

Here is the full pipeline.py up to this point:

import requests
from bs4 import BeautifulSoup
import pandas as pd


def scrape_page(url):
    """Scrape all books from a single page."""
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) BookScraper/1.0"
    }
    response = requests.get(url, headers=headers)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")
    articles = soup.find_all("article", class_="product_pod")

    books = []
    for article in articles:
        title = article.find("h3").find("a")["title"]
        price = article.find("p", class_="price_color").text
        rating = article.find("p", class_="star-rating")["class"][1]
        availability = article.find("p", class_="instock availability").text.strip()

        books.append({
            "title": title,
            "price": price,
            "rating": rating,
            "availability": availability,
        })

    return books


def scrape_books(num_pages=5):
    """Extract book data from multiple pages."""
    all_books = []

    for page_num in range(1, num_pages + 1):
        url = f"https://books.toscrape.com/catalogue/page-{page_num}.html"
        print(f"Scraping page {page_num}...")
        page_books = scrape_page(url)
        all_books.extend(page_books)

    print(f"Extraction complete: {len(all_books)} books scraped.")
    return all_books


# Test
raw = scrape_books()
print(raw[:3])

Your extraction stage is working. You have raw book data flowing in from the web. Next up, you'll build the transform stage to clean this messy data into properly typed columns.

Transform - Clean the Data

Your extraction stage is working. You have 100 raw book dictionaries with messy string data sitting in memory.

Raw scraped data is never ready for analysis. Prices have currency symbols baked in, ratings are words instead of numbers, and availability is a freeform text string. The Transform stage of an ETL pipeline takes this raw material and reshapes it into clean, typed columns that tools like pandas can actually compute on.

In this step, get ready to:
  • Write a transformation function that cleans prices, ratings, and availability.
  • Remove duplicate rows and reset the DataFrame index.
  • Verify the cleaned data has correct types and structure.
Write the price and rating transformations

The transform function takes in a raw list of dictionaries and outputs a clean DataFrame with properly typed columns.

  • In pipeline.py, find the test code at the bottom of the file (the lines starting with # Test).
  • Add the following function above the test code, directly below the scrape_books() function:
def transform_books(raw_books):
    """Transform raw book data into a clean DataFrame."""
    df = pd.DataFrame(raw_books)

    # Clean price: remove currency symbol and convert to float
    df["price"] = df["price"].str.replace("£", "", regex=False).astype(float)

    # Clean rating: map word to integer
    rating_map = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}
    df["rating"] = df["rating"].map(rating_map)

What does this code do?

pd.DataFrame(raw_books) converts the list of dictionaries into a table where each key becomes a column.

str.replace("£", "", regex=False).astype(float) strips the pound sign from every price string, then converts the remaining number text to a decimal (float). The regex=False flag tells pandas to treat the £ as a literal character, not a pattern.

rating_map is a dictionary that translates word ratings into numbers. The .map() method looks up each value in the dictionary and swaps it for the matching integer.

You won't see output yet. The next chunk finishes the function, and then you'll run it to verify.

Clean availability and remove duplicates

Two more cleaning operations remain. Availability needs to become a simple True/False value, and any duplicate book titles should be dropped so your dataset stays accurate.

  • Still in pipeline.py, add the following code directly below the df["rating"] = df["rating"].map(rating_map) line you just wrote:
    # Clean availability: convert to boolean
    df["availability"] = df["availability"].str.contains("In stock")

    # Remove duplicates
    df = df.drop_duplicates(subset=["title"]).reset_index(drop=True)

    print(f"Transform complete: {len(df)} clean records.")
    return df

What does this code do?

str.contains("In stock") checks whether each availability string contains the phrase "In stock". It returns True if it does, False otherwise. This converts messy text into a clean boolean column.

drop_duplicates(subset=["title"]) removes any rows that share the same book title. reset_index(drop=True) renumbers the rows from 0 so there are no gaps in the index after duplicates are removed.

You still won't see output. The next substep updates the test code so you can run the full transform and verify it.

Test the transform stage

Time to wire up the transform function and check that every column has the correct data type.

  • In pipeline.py, scroll down to the test code at the bottom of the file.
  • Find this block:
# Test
raw = scrape_books()
print(raw[:3])
  • Replace it with the following test code:
# Test
raw = scrape_books()
clean_df = transform_books(raw)
print(clean_df.info())
print(clean_df.head())

What does this test code do?

clean_df.info() prints a summary of the DataFrame showing each column's name, non-null count, and data type. This is how you verify the types are correct (float for price, int for rating, bool for availability).

clean_df.head() prints the first 5 rows so you can visually inspect that the values look right.

  • Save pipeline.py.
  • Run the script by entering this command in your terminal:
python pipeline.py
  • In the output, look for the column types listed by .info(). You should see price as float64, rating as int64, and availability as bool.
  • The .head() output should show prices as plain numbers (no £ symbol), ratings as integers 1-5, and availability as True or False.

Seeing a KeyError or unexpected NaN values?

If you see KeyError: 'price', make sure the dictionary keys in scrape_page() match exactly what transform_books() expects. Column names are case-sensitive.

If ratings show NaN, check that your rating_map dictionary keys start with a capital letter ("One", not "one"). The scraped values from the HTML class names are capitalized.

If the script errors on the str.replace line, confirm you included regex=False as a parameter.

Help me debug my transform function.

✔️ Awesome, I've got everything!

Great. Double check you've saved pipeline.py before moving on.

ⓧ I'd like to double check the full code

import requests
from bs4 import BeautifulSoup
import pandas as pd


def scrape_page(url):
    """Scrape all books from a single page."""
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) BookScraper/1.0"
    }
    response = requests.get(url, headers=headers)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")
    articles = soup.find_all("article", class_="product_pod")

    books = []
    for article in articles:
        title = article.find("h3").find("a")["title"]
        price = article.find("p", class_="price_color").text
        rating = article.find("p", class_="star-rating")["class"][1]
        availability = article.find("p", class_="instock availability").text.strip()

        books.append({
            "title": title,
            "price": price,
            "rating": rating,
            "availability": availability,
        })

    return books


def scrape_books(num_pages=5):
    """Extract book data from multiple pages."""
    all_books = []

    for page_num in range(1, num_pages + 1):
        url = f"https://books.toscrape.com/catalogue/page-{page_num}.html"
        print(f"Scraping page {page_num}...")
        page_books = scrape_page(url)
        all_books.extend(page_books)

    print(f"Extraction complete: {len(all_books)} books scraped.")
    return all_books


def transform_books(raw_books):
    """Transform raw book data into a clean DataFrame."""
    df = pd.DataFrame(raw_books)

    # Clean price: remove currency symbol and convert to float
    df["price"] = df["price"].str.replace("£", "", regex=False).astype(float)

    # Clean rating: map word to integer
    rating_map = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}
    df["rating"] = df["rating"].map(rating_map)

    # Clean availability: convert to boolean
    df["availability"] = df["availability"].str.contains("In stock")

    # Remove duplicates
    df = df.drop_duplicates(subset=["title"]).reset_index(drop=True)

    print(f"Transform complete: {len(df)} clean records.")
    return df


# Test
raw = scrape_books()
clean_df = transform_books(raw)
print(clean_df.info())
print(clean_df.head())

Your data is clean and properly typed. Next up, you'll save this DataFrame to a CSV file and build the final orchestration that ties extract, transform, and load together.

Load - Save and Validate

Your extraction pulls raw data from the web and your transformation cleans it into a structured DataFrame. The final piece of any ETL pipeline is the load stage, where clean data lands in a permanent, shareable format.

In this step, you will write the load function, build a main() orchestrator that ties all three stages together, and run the full pipeline to produce a clean CSV file.

In this step, get ready to:
  • Save the cleaned DataFrame to a CSV file.
  • Build a main function that orchestrates extract, transform, and load.
  • Run the complete pipeline and verify the output.
Write the load function

The load function takes your clean DataFrame and writes it to a CSV file that you can open in Excel, import into a database, or share with a teammate.

  • In pipeline.py, scroll to the bottom of the file and find the test code:
# Test
raw = scrape_books()
clean_df = transform_books(raw)
print(clean_df.info())
print(clean_df.head())
  • Delete all of these test lines. Replace them with the following load_books() function:
def load_books(df, filename="books_clean.csv"):
    """Load the cleaned DataFrame to a CSV file."""
    df.to_csv(filename, index=False)
    print(f"Load complete: saved to {filename}")

What does this code do?

  • df.to_csv(filename, index=False) writes the DataFrame to a CSV file. The index=False argument prevents pandas from adding an extra row-number column.
  • The default filename is books_clean.csv, but you could pass a different name if you wanted to save multiple versions.
  • Save pipeline.py.
  • You won't see a change yet. The next chunk adds the main() function that actually calls load_books().
Add the main function to orchestrate the pipeline

A main() function is the single entry point that calls each pipeline stage in order. This pattern keeps your script organized so anyone reading the code can immediately see the overall flow.

  • Add the following main() function and entry-point guard below load_books():
def main():
    """Run the full ETL pipeline."""
    print("=" * 50)
    print("Book Scraping Pipeline")
    print("=" * 50)

    # Extract
    raw_books = scrape_books(num_pages=5)

    # Transform
    clean_df = transform_books(raw_books)

    # Load
    load_books(clean_df)

    # Summary
    print("\n--- Pipeline Summary ---")
    print(f"Total books: {len(clean_df)}")
    print(f"Average price: £{clean_df['price'].mean():.2f}")
    print(f"Price range: £{clean_df['price'].min():.2f} - £{clean_df['price'].max():.2f}")
    print(f"\nRating distribution:")
    print(clean_df["rating"].value_counts().sort_index().to_string())


if __name__ == "__main__":
    main()

What does this code do?

  • main() calls the three pipeline stages in order: extract, transform, load. Each stage flows into the next.
  • The summary section prints statistics about the scraped data so you can quickly verify the pipeline worked.
  • if __name__ == "__main__": ensures main() only runs when you execute the script directly, not when you import it as a module.
  • Save pipeline.py.

Seeing a syntax error on save?

  • Check that the if __name__ line uses double underscores on both sides of name and main.
  • Make sure you have two equal signs (==) in the comparison, not one.
  • Verify the f-strings use curly braces {} around expressions, not parentheses.

Why is my main function not working?

Run the complete pipeline

Your pipeline is fully assembled. Time to run it end-to-end and watch it scrape, clean, and save 100 books.

  • Run the pipeline by executing this command in your terminal:
python pipeline.py
  • You should see output showing each page being scraped, followed by the extraction and transform confirmations, and finally a pipeline summary.

What should I see?

Your terminal will show progress for each page ("Scraping page 1..." through "Scraping page 5..."), then "Extraction complete: 100 books scraped.", "Transform complete" with the record count, "Load complete: saved to books_clean.csv", and a summary with the total books, average price, price range, and rating distribution.

  • Confirm the CSV file was created by running:
ls
  • You should see books_clean.csv alongside pipeline.py in the output.
  • Open books_clean.csv in VS Code by clicking on it in the Explorer sidebar.
  • Verify the file has a header row (title, price, rating, availability) followed by 100 rows of clean data.

Don't see books_clean.csv?

  • Make sure you deleted the old test code at the bottom of the file. If both the test code and main() exist, the script may run the test code first and skip main().
  • Check that your terminal is in the same folder as pipeline.py. The CSV file is created in whatever directory you run the command from.
  • If you see a connection error, check your internet connection. The script needs to reach books.toscrape.com.

Why isn't my pipeline producing a CSV file?

✔️ Awesome, I've got everything!

Great. Double check you have saved your file.

ⓧ I'd like to double check the full code

import requests
from bs4 import BeautifulSoup
import pandas as pd


def scrape_page(url):
    """Scrape all books from a single page."""
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) BookScraper/1.0"
    }
    response = requests.get(url, headers=headers)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")
    articles = soup.find_all("article", class_="product_pod")

    books = []
    for article in articles:
        title = article.find("h3").find("a")["title"]
        price = article.find("p", class_="price_color").text
        rating = article.find("p", class_="star-rating")["class"][1]
        availability = article.find("p", class_="instock availability").text.strip()

        books.append({
            "title": title,
            "price": price,
            "rating": rating,
            "availability": availability,
        })

    return books


def scrape_books(num_pages=5):
    """Extract book data from multiple pages."""
    all_books = []

    for page_num in range(1, num_pages + 1):
        url = f"https://books.toscrape.com/catalogue/page-{page_num}.html"
        print(f"Scraping page {page_num}...")
        page_books = scrape_page(url)
        all_books.extend(page_books)

    print(f"Extraction complete: {len(all_books)} books scraped.")
    return all_books


def transform_books(raw_books):
    """Transform raw book data into a clean DataFrame."""
    df = pd.DataFrame(raw_books)

    # Clean price: remove currency symbol and convert to float
    df["price"] = df["price"].str.replace("£", "", regex=False).astype(float)

    # Clean rating: map word to integer
    rating_map = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}
    df["rating"] = df["rating"].map(rating_map)

    # Clean availability: convert to boolean
    df["availability"] = df["availability"].str.contains("In stock")

    # Remove duplicates
    df = df.drop_duplicates(subset=["title"]).reset_index(drop=True)

    print(f"Transform complete: {len(df)} clean records.")
    return df


def load_books(df, filename="books_clean.csv"):
    """Load the cleaned DataFrame to a CSV file."""
    df.to_csv(filename, index=False)
    print(f"Load complete: saved to {filename}")


def main():
    """Run the full ETL pipeline."""
    print("=" * 50)
    print("Book Scraping Pipeline")
    print("=" * 50)

    # Extract
    raw_books = scrape_books(num_pages=5)

    # Transform
    clean_df = transform_books(raw_books)

    # Load
    load_books(clean_df)

    # Summary
    print("\n--- Pipeline Summary ---")
    print(f"Total books: {len(clean_df)}")
    print(f"Average price: £{clean_df['price'].mean():.2f}")
    print(f"Price range: £{clean_df['price'].min():.2f} - £{clean_df['price'].max():.2f}")
    print(f"\nRating distribution:")
    print(clean_df["rating"].value_counts().sort_index().to_string())


if __name__ == "__main__":
    main()

Secret mission

Scale to All 1000 Books with Rate Limiting

Your pipeline scrapes 100 books from 5 pages. But books.toscrape.com has 1000 books across 50 pages. Can you scale your pipeline to handle all of them while being a polite scraper, gracefully handling errors, and producing a data analysis report?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your files and packages, or delete them entirely.

Resources you used:

  • Project folder containing pipeline.py and books_clean.csv.
  • Python packages installed via pip: requests, beautifulsoup4, pandas.

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your pipeline or want to reuse the code for other scraping targets.

  • Your project folder stays on your Desktop with pipeline.py and books_clean.csv intact.
  • The requests, beautifulsoup4, and pandas packages remain installed in your Python environment.

Pause - I'll come back to this later

There are no running processes to stop. Your files and packages will be waiting for you when you come back.

  • Everything stays as-is. Since the pipeline only runs when you execute it manually, nothing consumes resources while idle.

Delete - I don't want to use this again

Remove all project files and uninstall the packages you added.

  • Delete your project folder by running the appropriate command for your operating system.
  • On macOS/Linux, open your terminal and run:
cd ~/Desktop
rm -rf book-scraper
  • On Windows, open PowerShell and run:
cd ~\Desktop
Remove-Item -Recurse -Force book-scraper
  • If your project folder has a different name, replace book-scraper with the folder name you chose in Step 1.
  • Uninstall the Python packages by running:
pip uninstall requests beautifulsoup4 pandas -y
  • Confirm each removal when prompted, or use the -y flag as shown above to skip confirmations.

That's a Wrap!

That's a Wrap!

Nice work! You just built a complete data pipeline that scrapes a live website, cleans messy data, and produces an analysis-ready CSV file.

You've learned how to:

  • Built a complete ETL pipeline that extracts raw book data from a live website using requests and BeautifulSoup.
  • Transformed messy string data into properly typed columns (floats for prices, integers for ratings, booleans for availability).
  • Loaded clean data into a CSV file ready for analysis, using the modular extract/transform/load pattern used in professional data engineering.
  • Secret Mission: Scaled the pipeline to handle 50 pages with polite rate limiting, error resilience, and produced a data analysis report.

Ready to quiz yourself?