Automate the Web with Playwright

Scrape a bookstore, record clicks with codegen, and apply stealth techniques.

Introduction

30 Second Summary

Every day, people spend hours clicking through websites to collect information, fill out forms, or check prices. What if your computer could do all of that for you, exactly the way you would, but in seconds?

In this project, you will build a browser automation toolkit using Playwright and Python that navigates a real website, extracts structured data, and applies stealth techniques to avoid bot detection.

What You'll Build

You will have a stealth-aware scraper that you can point at an online bookstore, watch it flip through pages collecting titles and prices, and see the results saved to a neat JSON file on your machine.

By the end of this project, you'll have:

  • A CLI code generator workflow where you click around a website and Playwright writes working Python code for you automatically.
  • A multi-page scraper that navigates an online bookstore, extracts titles, prices, and ratings across pages, and saves structured results to JSON.
  • A clear picture of your detection fingerprint showing exactly which signals expose your script as a bot, then stealth techniques that make those signals vanish.
  • Secret Mission: Intercept network requests to block tracking scripts and handle popup overlays programmatically.

Are there any prerequisites?

You should have basic Python familiarity (variables, functions, loops, if statements). You also need an existing Python venv with Playwright already installed and its browser binaries downloaded.

If you do not have Playwright installed yet, run pip install playwright in a virtual environment, then run python -m playwright install to download the browser binaries before starting this project.

Before We Start

Before we dive into automating browsers, let's lock in what you're about to build and why it matters to you.

Set Up and Launch Your First Browser

Your Playwright installation is ready and your venv is active. Before you start automating websites, you need one additional package and a quick end-to-end test to prove the whole toolchain works.

In this step, you'll install a stealth library (used in a later step to disguise your browser as a real user) and write a minimal script that opens a live webpage, captures its title, and saves a screenshot to disk.

In this step, get ready to:
  • Install the tf-playwright-stealth package for later use.
  • Launch a browser, navigate to a live website, and save a screenshot.
Install tf-playwright-stealth

Later in this project you'll discover that websites can detect your automated browser within milliseconds. The tf-playwright-stealth package patches those detection signals. You'll install it now so it's ready when you need it.

  • Install the stealth library by running this command in your terminal:
pip install tf-playwright-stealth
  • Confirm the installation succeeded by running:
pip show tf-playwright-stealth

You should see output including Name: tf-playwright-stealth and Version: 1.2.0.

Install failed or version mismatch?

Make sure your venv is active. You should see your venv name in parentheses at the start of your terminal prompt. If not, reactivate it before retrying the install.

If you see a dependency conflict mentioning Playwright versions, confirm you have playwright version 1.61.0 installed by running pip show playwright.

Still stuck? Help me fix my tf-playwright-stealth installation

Create and run your first automation script

Time to prove the full automation chain works. You'll write a short script that launches a real Chromium window, navigates to a bookstore website, prints the page title, and saves a screenshot.

  • Create a new file called hello_browser.py in your project directory.
  • Paste the following code into hello_browser.py:
from playwright.sync_api import sync_playwright


with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()
    page.goto("https://books.toscrape.com/")
    print(f"Page title: {page.title()}")
    page.screenshot(path="hello_screenshot.png")
    print("Screenshot saved to hello_screenshot.png")
    browser.close()

What does this code do?

  • sync_playwright() starts the Playwright engine and gives you access to browser launchers.
  • p.chromium.launch(headless=False) opens a visible Chromium browser window so you can watch the automation happen.
  • page.goto() navigates to the URL and waits for the page to load.
  • page.screenshot() captures the current viewport and saves it as a PNG file in your project directory.
  • browser.close() shuts down the browser after the work is done, freeing system resources.

Before you run this, predict what you'll see. Will a browser window open? How long will it stay visible?

  • Run your script:
python hello_browser.py

A Chromium window flashes open, loads the Books to Scrape homepage, then closes. In your terminal you should see:

Page title: All products | Books to Scrape - Sandbox
Screenshot saved to hello_screenshot.png
  • Confirm the screenshot file was created by running:
ls hello_screenshot.png

You should see hello_screenshot.png listed in the output.

Browser didn't open or script errored?

If you see ModuleNotFoundError, your venv is not active. Reactivate it and try again.

If the browser opens but the script hangs, the site might be slow to respond. Wait 10 seconds. If it still hangs, press Ctrl+C to kill the script and try running it again.

If you see an error about missing browser binaries, run python -m playwright install chromium to re-download them.

Something else going wrong? Help me debug my hello_browser.py script

✔️ Awesome, I've got everything!

Great. Double check you've saved hello_browser.py and that hello_screenshot.png exists in your project directory.

ⓧ I'd like to double check the full code

Here is the complete hello_browser.py file. Make sure your code matches exactly:

from playwright.sync_api import sync_playwright


with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()
    page.goto("https://books.toscrape.com/")
    print(f"Page title: {page.title()}")
    page.screenshot(path="hello_screenshot.png")
    print("Screenshot saved to hello_screenshot.png")
    browser.close()

Your automation toolchain is working end to end. Next up, you'll let Playwright write code for you by recording your clicks with its built-in codegen tool.

Record a Workflow with Codegen

Your browser launched and took a screenshot in the previous step. Now you need a way to navigate websites, click buttons, and interact with pages.

Writing selectors and navigation code by hand is slow and error-prone. Playwright's codegen tool solves this by letting you click around a real website while it generates working Python code automatically. You click, it codes.

In this step, get ready to:
  • Launch codegen and record browser interactions into a Python script.
  • Run the generated script and watch the browser replay your exact clicks.
  • Explore codegen flags for device emulation and session persistence.
Launch codegen against the practice site

Codegen is a CLI tool built into Playwright that opens a browser and an inspector window side by side. As you click around in the browser, the inspector writes Python code in real time.

  • Launch codegen against the practice bookstore by running this command:
python -m playwright codegen --target=python -o recorded_script.py https://books.toscrape.com

What happens when you run this?

Two windows open simultaneously. The first is a Chromium browser navigated to the bookstore. The second is the Playwright Inspector, which shows the Python code being generated as you interact with the browser.

The --target=python flag tells codegen to output synchronous Python code. The -o recorded_script.py flag saves the generated code to a file when you close the inspector.

Now perform three interactions in the codegen browser window:

  • Click on any book title on the homepage.
  • Watch the Inspector update with a new line of code, then click the browser's back button to return to the homepage.
  • Click the next pagination link at the bottom of the page.
  • Close the Playwright Inspector window (not the browser) to stop recording and save the generated code to recorded_script.py.
  • Confirm the file was saved by running:
ls recorded_script.py

You should see recorded_script.py listed in the output.

Don't see the file?

Make sure you closed the Inspector window (the panel showing generated code), not the browser window. Closing the browser first discards the recording.

If that happened, re-run the codegen command and try again. Help me troubleshoot codegen not saving my file.

Run the recorded script

The generated script is a fully working Python file. Before running it, open it in your editor to see what codegen produced.

  • Open recorded_script.py in VS Code or Neovim.
  • Look at the selectors codegen chose for each interaction.

What should I see in the generated code?

Codegen uses resilient selector strategies like page.get_by_role() and page.locator() instead of fragile CSS selectors or XPaths.

Role-based selectors target elements by their accessibility role (link, button, heading). This makes your scripts more resistant to design changes. If a site swaps its CSS classes, role-based selectors still find the right element.

Before you run the script, what do you think will happen? Will a browser open? Will it be fast or slow?

  • Run the recorded script by executing:
python recorded_script.py

A Chromium window opens and replays your exact interactions. You see it navigate to the bookstore, click the book title you chose, go back, and click the next page link. The browser then closes automatically.

Script errors or browser doesn't open?

If you see an import error, confirm your venv is still activated. The prompt should show your venv name in parentheses.

If the script runs but nothing visible happens, the generated code might use headless=True. Open the file and change it to headless=False to see the browser window. Help me debug my recorded Playwright script.

Explore codegen flags

Codegen can simulate different devices and save authentication state. These flags let you record scripts that behave like a mobile phone or stay logged in across runs.

  • Re-run codegen with device emulation to see how it changes the generated code:
python -m playwright codegen --target=python --device="iPhone 13" https://books.toscrape.com

Notice the browser opens with a narrow mobile viewport. The generated code in the Inspector now includes a viewport size and user agent string matching the iPhone 13.

  • Click on a couple of elements so you can see the generated code, then close the Inspector window.

What about authenticated workflows?

The --save-storage=auth.json flag tells codegen to save cookies, localStorage, and IndexedDB data to a JSON file at the end of your session.

This means you can log into a site once during a codegen session, save that state to auth.json, and then load it in future scripts to skip the login step entirely. You will not need this for the practice bookstore, but it is essential for automating sites that require authentication.

Codegen as a starting point

Codegen is best used as a bootstrapping tool. Record a rough workflow, then clean up and extend the generated code by hand. In the next step, you will write a scraper from scratch that does more than codegen can record.

You now have a working script that was written by clicking, not typing. Next up, you will go beyond what codegen can record and build a multi-page scraper that extracts structured data and saves it to a file.

Build a Multi-Page Scraper

In the last step, codegen recorded your browser interactions and turned them into working Python code. That is powerful for bootstrapping, but real automation needs more.

Your recorded script replays clicks, but it does not extract structured data or save results to a file. In this step, you will write a scraper that navigates multiple pages of the bookstore, pulls out titles, prices, and ratings from each book, and saves everything to JSON.

In this step, get ready to:
  • Extract structured book data from page 1 using Playwright locators.
  • Navigate into a detail page and then to page 2 for more books.
  • Save all collected data to a JSON file.
Navigate and extract data from page 1

Playwright's locator API lets you target elements on a page using CSS selectors, then pull out text, attributes, or counts. You will use locators to select every book card on the page and extract its title, price, and star rating into a Python dictionary.

  • Create a new file called scrape_books.py in your project directory and paste the following code:
import json
from playwright.sync_api import sync_playwright


def main():
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False)
        page = browser.new_page()

        # Navigate to the bookstore
        page.goto("https://books.toscrape.com/")
        page.screenshot(path="homepage.png")

        browser.close()


if __name__ == "__main__":
    main()

What does this code do?

  • The script wraps everything in a main() function so you can run it cleanly from the command line.
  • page.goto() navigates to the bookstore and waits for the page to load.
  • page.screenshot() captures a PNG of the current page state for verification.
  • Save scrape_books.py.
  • Run the script to confirm it opens the browser and saves a screenshot:
python scrape_books.py

You should see the browser open the bookstore briefly, then close.

  • Confirm that homepage.png appeared in your project directory by running:
ls homepage.png

Browser didn't open or no screenshot?

  • Make sure you saved scrape_books.py before running it.
  • Check that your venv is activated. If you see ModuleNotFoundError: No module named 'playwright', activate it again.

Why is my script failing to launch the browser?

Now that the skeleton works, add the extraction logic. This is the core of the scraper.

  • In scrape_books.py, find the line browser.close() and add this code directly above it:
        # Extract book data from page 1
        books = []
        book_elements = page.locator("article.product_pod")

        for i in range(book_elements.count()):
            book = book_elements.nth(i)
            title = book.locator("h3 a").get_attribute("title")
            price = book.locator(".price_color").text_content()
            rating_class = book.locator("p.star-rating").get_attribute("class")
            rating = rating_class.replace("star-rating ", "")
            books.append({"title": title, "price": price, "rating": rating})

        print(f"Found {len(books)} books on page 1")

What does this code do?

  • page.locator("article.product_pod") selects every book card on the page. Each card is an <article> element with the class product_pod.
  • book_elements.count() tells you how many matches exist (20 per page on this site).
  • For each book, you pull the title from the title attribute of the link, the price text, and the star rating from the CSS class name (e.g. star-rating Three becomes Three).
  • Save scrape_books.py.
  • Run the script to see it extract data from page 1:
python scrape_books.py

You should see the browser open the bookstore, then your terminal prints:

Found 20 books on page 1

Not seeing 20 books?

  • Make sure you saved scrape_books.py before running it.
  • If the browser opens but nothing prints, confirm the URL is exactly https://books.toscrape.com/ with the trailing slash.
  • Check that the extraction code is placed above browser.close(), not below it. Once the browser closes, you cannot access page data.

Why is my script not finding any books on the page?

Navigate to a detail page and page 2

A real scraper does more than read one page. You will click into the first book to grab its description, then navigate back and move to page 2 to double your data.

  • In scrape_books.py, find the line browser.close() and add this code directly above it:
        # Click into the first book for details
        page.locator("article.product_pod h3 a").first.click()
        page.screenshot(path="book_detail.png")

        # Extract detail page info
        detail_title = page.locator("h1").text_content()
        detail_description = page.locator("#product_description ~ p").text_content()
        print(f"Book detail: {detail_title}")
        print(f"Description preview: {detail_description[:100]}...")

What does this code do?

  • .first.click() clicks the first matching link among all book cards. This takes you to that book's detail page.
  • page.locator("#product_description ~ p") uses a CSS sibling selector to grab the paragraph that follows the product description heading.
  • detail_description[:100] slices the first 100 characters so the terminal output stays readable.
  • Save scrape_books.py and run it:
python scrape_books.py

Watch the browser navigate into the first book's detail page. Your terminal now prints the book title and a description preview alongside the page 1 count.

Seeing a timeout error on the detail page?

  • Check that the locator is page.locator("article.product_pod h3 a").first.click(). Missing .first causes ambiguity errors because multiple links match.
  • If the description selector fails, confirm the page loaded fully. The site is sometimes slow on first load.

Help me debug why my detail page extraction is timing out.

Now add the pagination logic to navigate back and scrape page 2.

  • In scrape_books.py, add this code below the description preview print and above browser.close():
        # Go back and navigate to page 2
        page.go_back()
        page.locator("li.next a").click()

        # Extract books from page 2
        book_elements = page.locator("article.product_pod")
        for i in range(book_elements.count()):
            book = book_elements.nth(i)
            title = book.locator("h3 a").get_attribute("title")
            price = book.locator(".price_color").text_content()
            rating_class = book.locator("p.star-rating").get_attribute("class")
            rating = rating_class.replace("star-rating ", "")
            books.append({"title": title, "price": price, "rating": rating})

What does this code do?

  • page.go_back() works like clicking the browser's back button. It returns you to page 1 from the detail page.
  • page.locator("li.next a").click() finds the "next" pagination link and clicks it to load page 2.
  • The extraction loop is identical to page 1. You reuse the same locator pattern and append to the same books list so all results accumulate.
  • Save scrape_books.py and run it again:
python scrape_books.py

Watch the browser navigate into the first book, return, then advance to page 2. The terminal prints the page 1 count and book detail info from earlier.

Script crashes when clicking next?

  • Make sure page.go_back() appears before the next-page click. If you skip it, you are still on the detail page where no "next" link exists.
  • Confirm the locator is li.next a (lowercase). A typo like li.Next will not match.

Why does my script fail when trying to navigate to page 2?

Save results to JSON

Your script collects books from two pages but the data disappears when the script ends. Saving to JSON gives you a permanent, structured file you can open and inspect.

  • In scrape_books.py, find the line browser.close() and add this code directly above it:
        print(f"Total books collected: {len(books)}")

        # Save results
        with open("books_data.json", "w") as f:
            json.dump(books, f, indent=2)

        print("Data saved to books_data.json")

What does this code do?

  • json.dump(books, f, indent=2) writes the entire list of book dictionaries to a file with readable indentation.
  • The import json at the top of your file (already there from the first chunk) provides this function.
  • The total count print confirms both pages were scraped before the file is written.
  • Save scrape_books.py.

Before you run this, how many total books do you expect to see collected from two pages of 20 books each?

  • Run the final version of your scraper:
python scrape_books.py

Your terminal prints all of this in sequence:

Found 20 books on page 1
Book detail: A Light in the Attic
Description preview: It's hard to imagine a world without A Light in the Attic. This now-classic lect...
Total books collected: 40
Data saved to books_data.json
  • Confirm the JSON file was created by running:
ls books_data.json
  • Open books_data.json in your editor to see structured data with title, price, and rating fields for approximately 40 books.

JSON file is empty or missing?

  • If the file exists but is empty, check that json.dump(books, f, indent=2) uses books (the list variable) not book (the loop variable).
  • If the script crashes before saving, scroll up in your terminal to find the error. A common cause is a typo in the page 2 extraction loop.
  • Make sure your save code is placed above browser.close(), not below it. Once the browser closes, you cannot access page data.

My books_data.json file is missing or empty after running the script.

✔️ Awesome, I've got everything!

Great. Double check you have saved scrape_books.py and that books_data.json contains about 40 book entries.

ⓧ I'd like to double check the full code

Your scrape_books.py should contain these sections in order. Scroll up and confirm each one is present:

  • Imports (json and sync_playwright).
  • main() function definition with browser launch and navigation to the bookstore.
  • Homepage screenshot saved to homepage.png.
  • Page 1 extraction loop using article.product_pod locator, appending title, price, and rating.
  • Print statement: Found {len(books)} books on page 1.
  • Click into first book, screenshot to book_detail.png, extract detail title and description, print both.
  • page.go_back() then li.next a click to reach page 2.
  • Page 2 extraction loop (same pattern as page 1).
  • Total count print, JSON save to books_data.json, and data-saved print.
  • browser.close() and the if __name__ == "__main__" guard.

You now have a working multi-page scraper that extracts structured data and saves it to disk. Next up, you will discover exactly how websites can tell your script is a bot.

Discover Your Detection Fingerprint

Your scraper collected 40 books across two pages without a single hiccup. That worked because books.toscrape.com is a practice site with zero bot detection.

Most real websites are not that generous. They check values like navigator.webdriver within milliseconds of page load. If that value says "automated", you are blocked before you can read a single element. In this step, you will probe your own browser to see exactly which signals give you away.

In this step, get ready to:
  • Create a detection probe script that checks your browser's automation signals.
  • Run the script in both headed and headless modes and see that both are trivially detectable.
Create a detection probe script

Every browser exposes JavaScript properties that reveal whether it is controlled by automation. A real user's browser sets navigator.webdriver to undefined or false. Playwright sets it to True.

  • Create a new file called detect_check.py in your project directory and add this code:
from playwright.sync_api import sync_playwright


def main():
    print("=" * 50)
    print("AUTOMATION DETECTION CHECK")
    print("=" * 50)

    with sync_playwright() as p:
        # Test in headed mode
        browser = p.chromium.launch(headless=False)
        page = browser.new_page()
        page.goto("https://example.com")

        webdriver = page.evaluate("navigator.webdriver")
        user_agent = page.evaluate("navigator.userAgent")
        languages = page.evaluate("JSON.stringify(navigator.languages)")
        plugins_count = page.evaluate("navigator.plugins.length")

        print(f"\n--- Headed Mode (headless=False) ---")
        print(f"navigator.webdriver:    {webdriver}")
        print(f"User-Agent:             {user_agent}")
        print(f"Languages:              {languages}")
        print(f"Plugins count:          {plugins_count}")

        browser.close()


if __name__ == "__main__":
    main()

What does this code do?

Each page.evaluate() call runs JavaScript inside the browser and returns the result to Python. You are reading the same values that any website's anti-bot script would check.

  • navigator.webdriver is the single biggest giveaway. Automation frameworks set it to True.
  • navigator.plugins.length reveals whether browser plugins (like PDF viewers) are loaded. Real browsers always have a few.
  • navigator.userAgent can contain the string "HeadlessChrome" when running headless.
  • navigator.languages should return a realistic array like ["en-US","en"]. An empty array is suspicious.

Before you run this, take a guess. Your browser opens with a visible window (headed mode). Do you think it will pass or fail these detection checks?

  • Save detect_check.py and run it:
python detect_check.py

A browser window flashes open, visits https://example.com, and closes. Your terminal prints the raw detection values. You should see navigator.webdriver: True and Plugins count: 0.

Don't see any output?

  • Make sure you saved the file before running. Check for a typo in if __name__ == "__main__": at the bottom.
  • If you see a ModuleNotFoundError, confirm your venv is activated.
  • help me debug my detect_check.py script
Add verdict logic and headless comparison

The raw values confirm you are detectable, but a structured verdict makes it unmistakable. You will also add a headless mode test to see whether hiding the browser window helps at all.

  • In detect_check.py, find the line that says browser.close().
  • Add this verdict logic directly above browser.close():
        print(f"\n--- Verdict ---")
        if webdriver:
            print("[DETECTED] navigator.webdriver is True")
        else:
            print("[CLEAN] navigator.webdriver is not set")

        if "HeadlessChrome" in str(user_agent):
            print("[DETECTED] User-Agent contains 'HeadlessChrome'")
        else:
            print("[CLEAN] User-Agent looks normal")

        if plugins_count == 0:
            print("[DETECTED] No browser plugins (real browsers have plugins)")
        else:
            print(f"[CLEAN] {plugins_count} plugins detected")

What does this code do?

Each if block checks one detection signal and prints a clear [DETECTED] or [CLEAN] label. This mirrors what a real anti-bot script does internally. If any signal reads as automated, the site blocks the request.

  • Now add the headless mode comparison. Find the line if __name__ == "__main__": at the bottom of your file.
  • Add this code directly above that line (still inside the with sync_playwright() block, at the same indentation level as the first browser.close()):
        # Test in headless mode
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto("https://example.com")

        webdriver_headless = page.evaluate("navigator.webdriver")
        ua_headless = page.evaluate("navigator.userAgent")
        plugins_headless = page.evaluate("navigator.plugins.length")

        print(f"\n--- Headless Mode (headless=True) ---")
        print(f"navigator.webdriver:    {webdriver_headless}")
        print(f"User-Agent:             {ua_headless}")
        print(f"Plugins count:          {plugins_headless}")

        print(f"\n--- Conclusion ---")
        print("Both headed and headless modes expose automation signals.")
        print("Any site checking these values will block you instantly.")

        browser.close()

What does this code do?

This launches a second browser instance with headless=True (no visible window) and checks the same signals. The conclusion prints after both tests finish.

Many people assume that running headless is what makes a browser detectable. This test proves that assumption wrong.

Before you run the updated script, make a prediction. Do you think headless mode will be more detectable, less detectable, or exactly the same as headed mode?

  • Save detect_check.py and run it again:
python detect_check.py

Your terminal prints verdicts for both modes. The key result: navigator.webdriver is True in BOTH headed and headless. Plugins count is 0 in BOTH modes.

Even a fully visible browser window is trivially detectable as automated. Hiding the window changes nothing about your fingerprint.

Why did your scraper work on books.toscrape.com then?

That site is purpose-built for scraping practice. It deliberately ignores these signals. Any site with even basic bot detection (a simple if (navigator.webdriver) check in JavaScript) would have blocked you on the first page load.

Seeing an error in headless mode?

  • Check that the headless section is inside the with sync_playwright() as p: block. It should be indented to the same level as the first browser = p.chromium.launch() line.
  • Make sure browser.close() appears after the headed section AND after the headless section (two separate close calls).
  • help me fix my detection check script

✔️ Awesome, I've got everything!

Great. Double check you have saved detect_check.py and that running it prints both headed and headless detection results.

ⓧ I'd like to double check the full code

Here is the complete detect_check.py file. Compare it with yours:

Your file has three main sections built across the chunks above:

  • The headed mode probe (import, main function, banner, launch, evaluate 4 values, print them).
  • The verdict logic (three if/else blocks printing DETECTED or CLEAN, then browser.close()).
  • The headless mode section (new launch with headless=True, evaluate 3 values, print them, print conclusion, browser.close()).
  • The if __name__ == "__main__": main() block at the very bottom.

Scroll up through each chunk and confirm they are present in order. Every section should be inside the with sync_playwright() as p: block except the if __name__ guard.

Your browser is flagged as automated regardless of whether you can see the window. In the next step, you will apply stealth techniques that mask these signals and watch them disappear.

Apply Stealth and Reduce Your Fingerprint

In Step 4, you ran a detection probe and confirmed the bad news. Your automated browser broadcasts navigator.webdriver: True and zero plugins in every mode.

Now you will mask those signals at the JavaScript level using tf-playwright-stealth. This defeats basic bot detection checks that most simpler sites rely on. It does not guarantee bypassing advanced protections like Cloudflare Turnstile (which checks TLS fingerprints and network-level signals), but it covers the first line of defence.

In this step, get ready to:
  • Apply stealth patches to your browser context and re-run the detection checks.
  • Compare stealth output to Step 4 and scrape with stealth active.
  • Document the detection layers that JavaScript-level stealth cannot address.
Apply stealth to your browser

The stealth_sync function patches the page's JavaScript environment before you navigate anywhere. It overrides properties like navigator.webdriver so they return values a real browser would.

  • Create a new file called stealth_scraper.py and add the following imports, browser setup, and detection evaluations:
import json
from playwright.sync_api import sync_playwright
from playwright_stealth import stealth_sync


def main():
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False)
        context = browser.new_context(
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
            viewport={"width": 1920, "height": 1080},
            locale="en-US",
            timezone_id="America/New_York",
        )
        page = context.new_page()
        stealth_sync(page)

        # Re-run detection checks
        page.goto("https://example.com")

        webdriver = page.evaluate("navigator.webdriver")
        user_agent = page.evaluate("navigator.userAgent")
        plugins_count = page.evaluate("navigator.plugins.length")

What does this code do?

Instead of using Playwright's default context, you create a custom one with a realistic user_agent string, a full HD viewport, and a matching locale/timezone_id pair. These make the browser context look like a real person's Chrome session on Windows.

The critical line is stealth_sync(page). It patches the page BEFORE any navigation so that detection scripts run against the patched environment, not the raw Playwright defaults.

  • Now add the print and verdict section below the evaluates, plus the close logic and main guard:
        print("=" * 50)
        print("STEALTH DETECTION CHECK")
        print("=" * 50)
        print(f"navigator.webdriver:    {webdriver}")
        print(f"User-Agent:             {user_agent}")
        print(f"Plugins count:          {plugins_count}")

        print(f"\n--- Verdict ---")
        if not webdriver:
            print("[SUCCESS] navigator.webdriver is hidden!")
        else:
            print("[FAILED] navigator.webdriver still True")

        if "HeadlessChrome" not in str(user_agent):
            print("[SUCCESS] User-Agent looks like a real browser!")
        else:
            print("[FAILED] User-Agent still contains HeadlessChrome")

        if plugins_count > 0:
            print(f"[SUCCESS] {plugins_count} plugins present (looks real)")
        else:
            print("[FAILED] Still no plugins")

        context.close()
        browser.close()


if __name__ == "__main__":
    main()

What does this verdict logic do?

Each conditional checks the same signals you probed in detect_check.py. If stealth worked, navigator.webdriver is no longer True, the User-Agent no longer contains HeadlessChrome, and plugins are present.

Before you run this, think back to Step 4. Do you expect navigator.webdriver to still be True, or will stealth have hidden it?

  • Save stealth_scraper.py and run it:
python stealth_scraper.py

You should see all three checks print [SUCCESS]. The signals that were red flags in Step 4 are now masked.

Seeing [FAILED] on navigator.webdriver?

Make sure stealth_sync(page) is called BEFORE page.goto(). If you navigate first, the detection scripts have already run against the unpatched environment.

Also confirm you imported from playwright_stealth (underscore), not playwright-stealth (hyphen). The import module name uses an underscore.

help me debug why stealth_sync isn't hiding navigator.webdriver in my Playwright script

Compare before and after

Your detection output flipped from Step 4's wall of [DETECTED] to [SUCCESS] across all three signals. The stealth patches are working at the JavaScript level.

Now prove that stealth does not break normal scraping. You will navigate to the bookstore with stealth active and extract data.

  • In stealth_scraper.py, find the lines:
        context.close()
        browser.close()
  • Add the following scraping section ABOVE those two close lines:
        # Now scrape with stealth applied
        print(f"\n--- Stealth Scraping ---")
        page.goto("https://books.toscrape.com/")
        page.screenshot(path="stealth_homepage.png")

        books = []
        book_elements = page.locator("article.product_pod")

        for i in range(book_elements.count()):
            book = book_elements.nth(i)
            title = book.locator("h3 a").get_attribute("title")
            price = book.locator(".price_color").text_content()
            books.append({"title": title, "price": price})

        print(f"Stealth scrape successful: {len(books)} books collected")

        with open("stealth_books.json", "w") as f:
            json.dump(books, f, indent=2)

        print("Data saved to stealth_books.json")

What does this code do?

This reuses the same stealth-patched page to navigate to the bookstore. It extracts titles and prices from all 20 books on page 1 and saves them to stealth_books.json. The screenshot proves the page loaded correctly under stealth.

  • Save stealth_scraper.py and run it again:
python stealth_scraper.py

You should see the three [SUCCESS] lines from the detection check, followed by Stealth scrape successful: 20 books collected and a confirmation that data was saved.

  • Confirm the output files exist by running:
ls stealth_homepage.png stealth_books.json

Both files should appear in the listing.

Scrape returns 0 books?

Check that you placed the scraping code ABOVE context.close(). If the context is closed before the scrape runs, the page object is no longer usable.

Also confirm the locator string is exactly article.product_pod (no space, no extra dot).

help me figure out why my stealth scraper collects 0 books

Understand the limits

Stealth patches defeat basic JavaScript-level checks. But the anti-bot landscape has multiple layers, and most of them operate before your JavaScript ever runs.

  • In stealth_scraper.py, find the line print("Data saved to stealth_books.json") and add this comment block directly below it:
        # Understanding the limits
        # --------------------------------------------------------
        # What stealth_sync CANNOT protect against:
        #
        # 1. TLS Fingerprinting (JA3/JA4): The network handshake
        #    identifies automation BEFORE any JavaScript runs.
        #    Cloudflare correlates your TLS signature with your
        #    User-Agent and IP. No JS patch can fix this.
        #
        # 2. IP Reputation: Datacenter IPs are flagged regardless
        #    of browser fingerprint. Residential IPs score higher.
        #
        # 3. Behavioral Analysis: Mouse movement patterns, typing
        #    speed, scroll behavior. ML models detect robotic input.
        #
        # 4. Browser Binary Detection: Advanced systems (Cloudflare
        #    Turnstile) detect that stealth patches THEMSELVES are
        #    present by checking property descriptors and stack traces.
        #
        # For sites with these protections, you need either:
        # - A modified browser binary (Camoufox, Patchright)
        # - A remote browser service with real hardware fingerprints
        # - Official APIs or partnerships with the target site
        # --------------------------------------------------------

Why document these limits in the code?

This comment block is a reference for your future self. When you hit a site that blocks you despite stealth, you will know that the problem is not your code. It is a layer (TLS, IP, or behavioral) that no JavaScript patch can fix.

For those sites, practitioners use modified browser binaries (Camoufox, Patchright) or remote browser services with real hardware fingerprints. These are infrastructure-level solutions, not library-level ones.

  • Save stealth_scraper.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved stealth_scraper.py with all four sections: imports/setup, detection verdict, stealth scraping, and the limits comment block.

ⓧ I'd like to double check the full code

Scroll up and confirm your file contains these sections in order:

  • Chunk 1: Imports, main() function, browser launch, context creation with user_agent/viewport/locale/timezone_id, stealth_sync(page), navigation to example.com, and three page.evaluate() calls.
  • Chunk 2: Six print lines for the detection header and values, then three if/else verdict blocks checking webdriver, User-Agent, and plugins_count.
  • Chunk 3: The # Now scrape with stealth applied section navigating to books.toscrape.com, extracting titles/prices, saving to stealth_books.json.
  • Chunk 4: The # Understanding the limits comment block covering TLS, IP, behavioral, and binary detection.
  • Finally: context.close(), browser.close(), and the if __name__ == "__main__": guard.

Your browser now passes basic bot detection checks. In the Secret Mission, you will combine this stealth foundation with network interception and overlay handling to build a production-ready automation toolkit.

Secret mission

Intercept Requests and Handle Overlays

Your scraper works, but real websites throw two extra obstacles at automated browsers. Tracking scripts phone home to analytics services, and popup overlays block the elements you need to click. Can you intercept network requests and handle overlays programmatically?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. The only resources on disk are Python scripts, output files, and Playwright's downloaded browser binaries (which take roughly 500MB of space).

Decide whether to keep your resources as-is, free up disk space by removing browser binaries, or delete everything entirely.

Resources you used:

  • Python scripts in your project directory: hello_browser.py, scrape_books.py, detect_check.py, stealth_scraper.py, advanced_scraper.py, recorded_script.py.
  • Output files: books_data.json, stealth_books.json, advanced_books.json, and screenshot .png files.
  • Playwright browser binaries stored locally (~500MB disk space).
  • Python packages: playwright and tf-playwright-stealth.

Keep everything running

No action needed. Choose this if you plan to continue experimenting with browser automation or want to reference these scripts later.

  • Your scripts, output files, and browser binaries stay on disk.
  • There are no running processes or cloud services to worry about.
  • The browser binaries take ~500MB but cost nothing to keep.

Pause - I'll come back to this later

Free up disk space by removing the browser binaries while keeping your scripts intact. You can re-download the binaries later with a single command.

  • Remove the downloaded browser binaries by running:
python -m playwright uninstall
  • This frees ~500MB of disk space.
  • Your Python scripts and output files remain untouched.
  • When you come back, re-download the browsers by running python -m playwright install.

Delete - I don't want to use this again

Remove all project resources, browser binaries, and Python packages to start fresh.

  • Remove the Playwright browser binaries by running:
python -m playwright uninstall
  • Uninstall the Python packages by running:
pip uninstall playwright tf-playwright-stealth
  • Confirm removal by typing y when prompted for each package.
  • Delete the project scripts and output files by running:
rm hello_browser.py scrape_books.py detect_check.py stealth_scraper.py advanced_scraper.py recorded_script.py
rm books_data.json stealth_books.json advanced_books.json
rm hello_screenshot.png homepage.png book_detail.png stealth_homepage.png
  • On Windows, use del instead of rm for each file.
  • If you also want to remove the auth.json file created during codegen exploration, delete that too.

That's a Wrap!

That's a Wrap!

Nice work! You just built a complete browser automation toolkit that records interactions, scrapes structured data across multiple pages, and reduces its detection fingerprint.

You've learned how to:

  • Use Playwright's CLI codegen to record browser interactions and auto-generate working Python scripts.
  • Build a multi-page scraper that navigates a website, extracts structured data (titles, prices, ratings), takes screenshots, and saves results to JSON.
  • Discover exactly how websites detect automation by probing your browser's fingerprint signals like navigator.webdriver and plugin counts.
  • Apply stealth techniques and watch detection signals vanish, while understanding the limits of JS-level evasion against advanced protections like TLS fingerprinting, IP reputation, and behavioral analysis.
  • Secret Mission: Intercepted network requests to block trackers using page.route() and handled popup overlays programmatically with page.add_locator_handler().

Ready to quiz yourself?