Link Discovery Crawler with Crawlee

Build a multi-page crawler that discovers and extracts product data.

Introduction

30 Second Summary

Every online store has hundreds of product pages, each holding data you might need for price monitoring, competitor research, or market analysis. Visiting them one by one is not an option.

In this project, you will build a web crawler that automatically navigates a multi-page e-commerce site, decides which links are worth following, and extracts structured product data from every product page it finds.

What You'll Build

Picture running a single command and watching your crawler visit category pages, discover product links, skip irrelevant pages like login and cart, then save clean structured data for every product it reaches.

By the end of this project, you'll have:

  • A running PlaywrightCrawler that automatically discovers and follows links across a multi-page site using Crawlee's enqueueLinks with globs, exclude, selector, and label filtering.
  • Structured JSON output saved to Crawlee's Dataset containing product titles, prices, SKUs, and stock availability extracted with Playwright locators.
  • A cleanly routed crawler architecture using createPlaywrightRouter where each page type has its own dedicated handler.
  • Secret Mission: Add transformRequestFunction to enrich requests with custom metadata and use regexps for advanced URL pattern matching.

Are there any prerequisites?

You should have basic TypeScript knowledge (variables, functions, async/await) and Node.js 16 or later installed on your machine. Step 1 walks you through verifying your setup.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up a Crawlee + Playwright TypeScript Project

To build a crawler that discovers links and extracts data, you need three things working together: a crawling framework, a browser engine, and a way to run TypeScript directly.

In this step, you will install Crawlee (the crawling framework), Playwright (the browser engine it controls), and tsx (a zero-config TypeScript runner).

In this step, get ready to:
  • Create the project folder and install Crawlee, Playwright, and tsx.
  • Configure TypeScript for ES module support.
  • Run a minimal crawler to verify everything works.
Create the project and install dependencies
  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • Open the integrated terminal by pressing Ctrl+` (both macOS and Windows).
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir link-crawler
cd link-crawler
  • Initialize your project and install the dependencies by running these commands:
npm init -y
npm install crawlee@3.17.0 playwright@1.61.1
npm install -D tsx@4.23.1

The installs will take a minute. Playwright downloads a Chromium browser binary alongside the npm package.

What do these packages do?

  • crawlee is the crawling framework. It manages request queues, retries, concurrency, and data storage.
  • playwright is the browser engine. Crawlee drives it to load pages and interact with them.
  • tsx runs TypeScript files directly without a separate compile step.

Install failed or timed out?

  • If you see a network error, check your internet connection and try running the install commands again.
  • If you see a permissions error on macOS, prefix the command with sudo and enter your password when prompted.
  • help me debug my npm install error
Configure TypeScript for ES modules

Crawlee uses top-level await, which requires your project to be configured as an ES module with ES2022 or above.

  • In the VS Code explorer sidebar, right-click inside the link-crawler folder and select New File.
  • Name the file tsconfig.json and press Enter.
  • Paste the following configuration into tsconfig.json:
{
  "compilerOptions": {
    "module": "ES2022",
    "target": "ES2022",
    "moduleResolution": "Node16",
    "outDir": "./dist",
    "esModuleInterop": true,
    "strict": true,
    "skipLibCheck": true
  },
  "include": ["src"]
}

What does this configuration do?

  • module and target set to ES2022 enable top-level await, which Crawlee requires.
  • moduleResolution set to Node16 tells TypeScript how to find imported modules in an ESM project.
  • include restricts compilation to files inside the src folder.
  • Save tsconfig.json with Cmd+S (macOS) or Ctrl+S (Windows).

Now you need to tell Node.js this is an ES module project and add a convenience start script.

  • Open package.json in VS Code by clicking it in the explorer sidebar.
  • Replace the entire contents of package.json with this:
{
  "name": "link-crawler",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "start": "npx tsx src/main.ts"
  },
  "dependencies": {
    "crawlee": "3.17.0",
    "playwright": "1.61.1"
  },
  "devDependencies": {
    "tsx": "4.23.1"
  }
}

What changed in package.json?

  • "type": "module" tells Node.js to treat .js and .ts files as ES modules instead of CommonJS.
  • The start script gives you a shortcut to run the crawler with npm start.
  • Extra fields from npm init (like main, keywords, author) are removed because this project does not need them.
  • Save package.json.
Create a minimal crawler to verify setup

Your environment is configured. Time to write a tiny crawler that proves everything is wired up correctly.

  • Create a src folder inside link-crawler by running this in the terminal:
mkdir src
  • Confirm the folder exists by running:
ls

You should see src listed alongside node_modules, package.json, and tsconfig.json.

  • In the VS Code explorer sidebar, right-click the src folder and select New File.
  • Name the file main.ts and press Enter.
  • Paste the following code into src/main.ts:
import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
    async requestHandler({ request, log }) {
        log.info(`Visited: ${request.url}`);
    },
    maxRequestsPerCrawl: 1,
});

await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);

console.log('Crawler finished.');

What does this code do?

  • PlaywrightCrawler launches a real browser and visits each URL in its queue.
  • requestHandler runs once for every page the crawler visits. Right now it just logs the URL.
  • maxRequestsPerCrawl: 1 limits the crawler to a single page so this test finishes quickly.
  • await crawler.run([...]) starts the crawl with a seed URL. This is a top-level await, which is why you configured ES2022 modules.
  • Save src/main.ts.

Before you run this, what do you think you will see in the terminal? Will it print anything before the final message?

  • Run your crawler by executing this command:
npx tsx src/main.ts

You will see Crawlee log output including a line with Visited: https://warehouse-theme-metal.myshopify.com/collections followed by Crawler finished. at the end.

Seeing an error instead?

  • If you see a module-related error, confirm package.json has "type": "module" and that you saved the file.
  • If you see a file-not-found error, make sure main.ts is inside the src folder, not in the project root.
  • If the crawler hangs with no output, check your internet connection. Playwright needs to reach the target site.
  • help me debug my Crawlee setup error

✔️ Awesome, I've got everything!

Great. Double check you have saved all three files: tsconfig.json, package.json, and src/main.ts.

ⓧ I'd like to double check the full code

Here is each file you should have. Compare yours against these.

package.json:

{
  "name": "link-crawler",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "start": "npx tsx src/main.ts"
  },
  "dependencies": {
    "crawlee": "3.17.0",
    "playwright": "1.61.1"
  },
  "devDependencies": {
    "tsx": "4.23.1"
  }
}

tsconfig.json:

{
  "compilerOptions": {
    "module": "ES2022",
    "target": "ES2022",
    "moduleResolution": "Node16",
    "outDir": "./dist",
    "esModuleInterop": true,
    "strict": true,
    "skipLibCheck": true
  },
  "include": ["src"]
}

src/main.ts:

import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
    async requestHandler({ request, log }) {
        log.info(`Visited: ${request.url}`);
    },
    maxRequestsPerCrawl: 1,
});

await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);

console.log('Crawler finished.');

Your environment is locked in and your first crawler ran successfully. Next up, you will teach it to discover links across multiple pages and pull HTML content from them.

Crawl Pages and Extract HTML with enqueueLinks

Your crawler can visit a single page and report back. That proves the setup works, but a real crawler needs to follow links across an entire site.

Before you can filter links intelligently, you need to see what happens when you follow every link with no filter at all. Crawlee's enqueueLinks() function discovers all links on a page and adds them to the crawl queue in a single call.

In this step, get ready to:
  • Add enqueueLinks() to automatically discover and follow links on each page.
  • Extract page titles, link counts, and HTML content from visited pages.
  • Observe the crawler visiting irrelevant pages you never asked for.
Add enqueueLinks and HTML extraction

Right now your requestHandler only logs the URL it visits. You need it to do three things: grab the page title, count how many links exist on the page, and capture the raw HTML. Then enqueueLinks() adds every same-hostname link it finds to the crawl queue automatically.

  • In VS Code, open src/main.ts and replace the entire file contents with this code:
import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
    async requestHandler({ request, page, enqueueLinks, log }) {
        // Extract page metadata
        const title = await page.title();
        const linkCount = await page.locator('a').count();
        const html = await page.content();

        log.info(`Title: "${title}" | Links found: ${linkCount} | HTML length: ${html.length}`);

        // Add all same-hostname links to the crawl queue
        await enqueueLinks();
    },
    maxRequestsPerCrawl: 10,
});

await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);

console.log('Crawler finished.');

What does this code do?

  • page.title() grabs the <title> tag content from the rendered page.
  • page.locator('a').count() uses a Playwright locator to count every anchor element on the page.
  • page.content() returns the full rendered HTML as a string, including any content loaded by JavaScript.
  • enqueueLinks() with no arguments uses the default same-hostname strategy. It finds every link on the page that points to the same host and adds it to the request queue.
  • maxRequestsPerCrawl: 10 caps the total pages visited so the crawl finishes quickly for testing.
  • Save src/main.ts.

✔️ Awesome, I've got everything!

Great. Double check you have saved src/main.ts before running the crawler.

ⓧ I'd like to double check the full code

Here is the complete src/main.ts at this point:

import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
    async requestHandler({ request, page, enqueueLinks, log }) {
        const title = await page.title();
        const linkCount = await page.locator('a').count();
        const html = await page.content();

        log.info(`Title: "${title}" | Links found: ${linkCount} | HTML length: ${html.length}`);

        await enqueueLinks();
    },
    maxRequestsPerCrawl: 10,
});

await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);

console.log('Crawler finished.');
Run and observe the problem

Your crawler is now set to follow every same-hostname link it finds. Before you run it, think about what pages you expect it to visit. Will it only visit product and collection pages, or will it wander elsewhere?

  • Run the crawler by executing this command in your terminal:
npx tsx src/main.ts

Watch the log output scroll by. You will see titles and link counts for each page the crawler visits.

Notice the URLs it follows. Alongside collection and product pages, the crawler visits pages like /policies, /account, and /cart. These are completely irrelevant to your goal of extracting product data.

Why does it visit irrelevant pages?

When you call enqueueLinks() with no options, it uses the default same-hostname strategy. Every link pointing to the same host gets added to the queue with no filtering.

The site's navigation bar, footer, and sidebar all contain links to account pages, policy pages, and more. The crawler has no way to know which links matter to you and which are noise.

Not seeing any log output?

  • Make sure you saved src/main.ts after replacing its contents.
  • Check that the destructured parameters in requestHandler include page, enqueueLinks, and log. Missing any of these causes a runtime error.
  • Confirm you are running the command from inside the link-crawler folder.

Help me debug why my crawler isn't producing log output.

This is exactly the problem you will solve next. In Step 3, you will use globs and exclude to teach your crawler which links are worth following and which to ignore completely.

Filter Links with Globs and Exclude

In the previous step, you saw the problem first-hand. The default enqueueLinks() follows every link on the same hostname, pulling in account pages, policy pages, and anything else it finds.

Crawlee's enqueueLinks accepts two filtering options that solve this. globs whitelists URL patterns you want to follow. exclude blacklists patterns you never want.

In this step, get ready to:
  • Filter discovered links to only collection and product page URLs.
  • Block account, cart, and policy URLs from entering the queue.
  • Verify the crawler now visits only relevant pages.
Add globs to only follow collection and product URLs

Globs are URL patterns that use wildcards for flexible matching. The * character matches any characters within a path segment. When you pass a globs array to enqueueLinks, only URLs matching at least one pattern get added to the queue.

  • In src/main.ts, find this line:
        await enqueueLinks();
  • Replace it with the following code that adds a globs array matching only collection and product page URLs:
        await enqueueLinks({
            globs: [
                'https://warehouse-theme-metal.myshopify.com/collections/*',
                'https://warehouse-theme-metal.myshopify.com/products/*',
            ],
        });

What does this code do?

  • The first glob matches any URL under /collections/, like /collections/speakers or /collections/headphones.
  • The second glob matches any URL under /products/, like /products/sony-speaker.
  • Any link on the page that does not match one of these patterns is silently dropped from the queue.
  • Save src/main.ts.
Add exclude to block unwanted URL patterns

Globs whitelist what you want. The exclude option is the opposite. It blacklists URL patterns that should never be enqueued, even if they match a glob.

  • In src/main.ts, find the enqueueLinks call you just updated. Add an exclude array after the globs array so the full call looks like this:
        await enqueueLinks({
            globs: [
                'https://warehouse-theme-metal.myshopify.com/collections/*',
                'https://warehouse-theme-metal.myshopify.com/products/*',
            ],
            exclude: [
                '**/account/**',
                '**/cart/**',
                '**/policies/**',
            ],
        });

What does exclude do?

The ** wildcard matches any number of path segments. So **/account/** blocks any URL containing /account/ anywhere in its path.

Exclude patterns act as a final safety net. Even if a URL somehow matches your globs, it will still be blocked if it also matches an exclude pattern.

  • Save src/main.ts.
Run and compare the filtered results

Before you run this, think about what you expect to see. Will the crawler visit the same pages as before, or fewer?

  • Delete the ./storage folder so the crawler starts fresh, by running:
rm -rf ./storage
  • Run the crawler by running:
npx tsx src/main.ts

The logs now show only URLs matching your collection and product patterns. No more /account, /policies, or /cart pages appearing in the output.

Still seeing irrelevant URLs in the logs?

  • Make sure you deleted the ./storage folder before re-running. Crawlee caches previously queued URLs and will replay them if the storage directory persists.
  • Check that your globs array contains the full URL including https://. A pattern like /collections/* without the domain prefix will not match anything.
  • Confirm you saved src/main.ts after adding both globs and exclude.

Why is my crawler still visiting irrelevant pages after adding globs?

✔️ Awesome, I've got everything!

Great. Double check you have saved your file.

ⓧ I'd like to double check the full code

import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
    async requestHandler({ request, page, enqueueLinks, log }) {
        const title = await page.title();
        const linkCount = await page.locator('a').count();
        const html = await page.content();

        log.info(`Title: "${title}" | Links found: ${linkCount} | HTML length: ${html.length}`);

        await enqueueLinks({
            globs: [
                'https://warehouse-theme-metal.myshopify.com/collections/*',
                'https://warehouse-theme-metal.myshopify.com/products/*',
            ],
            exclude: [
                '**/account/**',
                '**/cart/**',
                '**/policies/**',
            ],
        });
    },
    maxRequestsPerCrawl: 10,
});

await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);

console.log('Crawler finished.');

Your crawler now makes smart decisions about which links to follow. Next up, you will extract structured product data from the pages it discovers.

Extract Structured Data with Playwright Locators

Your crawler now discovers and follows only the pages you care about. The enqueueLinks config filters URLs so the crawler visits only collection and product pages, skipping everything irrelevant.

Discovering the right pages is only half the job. Now you need to reach into each product page's HTML and pull out structured data like titles, prices, and stock status. In this step, you'll use Playwright locators to query specific elements and save the results as JSON.

In this step, get ready to:
  • Detect product pages by checking the URL pattern.
  • Extract structured fields (title, price, SKU, stock status) using Playwright locators.
  • Save extracted product data to Crawlee's Dataset as JSON files.
Detect product pages and extract data with locators

Your crawler visits both collection pages and product pages. You only want to extract data from product pages, so you need a way to tell them apart.

  • In src/main.ts, find the closing }); of the enqueueLinks() call.
  • Add this code below it, still inside the requestHandler function:
        if (request.url.includes('/products/')) {
            // Extract the product title from the h1 element
            const title = await page.locator('.product-meta h1').textContent();
            // Extract the SKU number
            const sku = await page.locator('span.product-meta__sku-number').textContent();
            // Find the price element that contains a dollar sign
            const priceElement = page.locator('span.price').filter({ hasText: '$' }).first();
            const currentPriceString = await priceElement.textContent();
            // Parse the numeric value from the price string
            const rawPrice = currentPriceString?.split('$')[1] ?? '0';
            const price = Number(rawPrice.replaceAll(',', ''));
            // Check whether the product is in stock
            const inStockElement = page.locator('span.product-form__inventory').filter({ hasText: 'In stock' }).first();
            const inStock = (await inStockElement.count()) > 0;

            log.info(`Product: ${title} | $${price} | SKU: ${sku} | In stock: ${inStock}`);
        }

What does this code do?

request.url.includes('/products/') checks if the current page is a product detail page. Only product pages get the extraction treatment.

Each page.locator() call targets a specific CSS selector on the page. The .filter({ hasText: '$' }) narrows results to elements containing a dollar sign, and .first() grabs the first match.

The price parsing splits on $, strips commas, and converts to a number. For stock status, checking .count() > 0 returns true if the "In stock" element exists on the page.

  • Save src/main.ts.
  • Run the crawler to see the extraction in action:
npx tsx src/main.ts

You should see log lines like Product: "Product Name" | $299 | SKU: ABC123 | In stock: true for each product page visited.

Not seeing any product log lines?

  • Make sure the if block is inside the requestHandler function, after the enqueueLinks() call.
  • Check that your start URL includes /collections so the crawler can discover product links from there.
  • help me debug why my crawler isn't logging product data
Save extracted data to Dataset

Logging data to the console is useful for debugging, but you need a way to persist it. Crawlee's Dataset saves each record as an individual JSON file on disk.

  • In src/main.ts, find the import line at the top:
import { PlaywrightCrawler } from 'crawlee';
  • Replace it with this updated import that adds Dataset:
import { PlaywrightCrawler, Dataset } from 'crawlee';
  • Now find the log.info line inside your if (request.url.includes('/products/')) block.
  • Add this code directly below it, still inside the if block:
            // Save the structured product data to Crawlee's Dataset
            await Dataset.pushData({
                url: request.url,
                title,
                sku,
                currentPrice: price,
                availableInStock: inStock,
            });

What does Dataset.pushData do?

Each call to Dataset.pushData() creates a new JSON file in the ./storage/datasets/default/ folder. Crawlee manages this folder automatically.

The object you pass becomes the file's contents. After the crawl finishes, you have one JSON file per product with all the fields you extracted.

  • Save src/main.ts.
Run and inspect the output

Time to see your structured data appear on disk. Before you run the crawler, think about what you expect to find in the storage folder.

  • If the ./storage folder exists from a previous run, delete it so you start fresh:
rm -rf ./storage

How many JSON files do you expect to appear in the dataset folder after the crawl?

  • Run the crawler:
npx tsx src/main.ts
  • After the crawl finishes, list the dataset output files:
ls ./storage/datasets/default/

You should see numbered JSON files like 000000001.json, 000000002.json, and so on. Each file is one product's structured data.

  • Open any one of the JSON files to inspect its contents:
cat ./storage/datasets/default/000000001.json

You should see a JSON object with url, title, sku, currentPrice, and availableInStock fields containing real product data from the store.

No files in the storage folder?

  • Confirm you imported Dataset from 'crawlee' at the top of src/main.ts.
  • Make sure Dataset.pushData() is inside the if (request.url.includes('/products/')) block, not outside it.
  • Check the terminal output for product log lines. If no products are logged, the crawler may not be reaching product URLs.
  • help me debug why my Dataset has no output files

✔️ Awesome, I've got everything!

Great. Double check that you've saved src/main.ts and that your ./storage/datasets/default/ folder contains JSON files with product data.

ⓧ I'd like to double check the full code

Scroll up and confirm each of these pieces is present in your src/main.ts, in order:

  • The import line includes both PlaywrightCrawler and Dataset from 'crawlee'.
  • The requestHandler extracts page.title(), page.locator('a').count(), and page.content() with a log line.
  • The enqueueLinks() call has globs for collections and products, plus exclude for account, cart, and policies.
  • The if (request.url.includes('/products/')) block uses locators for title, sku, price, and stock status.
  • Inside the if block, Dataset.pushData() saves an object with url, title, sku, currentPrice, and availableInStock.
  • maxRequestsPerCrawl is set to 10.

Your crawler now does two jobs: it discovers the right pages AND extracts clean, structured data from each one. Next up, you'll split this growing request handler into clean, labeled route handlers using Crawlee's router pattern.

Route with createPlaywrightRouter and Labels

Your crawler discovers the right pages and extracts product data. But look at the requestHandler in src/main.ts. It has a growing if/else block checking the URL to decide what to do with each page.

Every new page type you add means another branch in one function. Crawlee's Router pattern fixes this by giving each page type its own clean, isolated handler. You'll split the monolithic function into three route handlers using createPlaywrightRouter.

In this step, get ready to:
  • Create src/routes.ts with a default handler and a CATEGORY handler.
  • Add a DETAIL handler that extracts product data.
  • Update src/main.ts to use the router.
Create the router file with the default and CATEGORY handlers

The router separates your crawling logic by page type. Each page type gets a label, and the router dispatches requests to the matching handler automatically.

  • Create a new file called src/routes.ts and add the following code:
import { createPlaywrightRouter, Dataset } from 'crawlee';

export const router = createPlaywrightRouter();

// Default handler: processes the start URL (no label)
router.addDefaultHandler(async ({ request, page, enqueueLinks, log }) => {
    log.debug(`Enqueueing categories from page: ${request.url}`);
    await page.waitForSelector('.collection-block-item');
    await enqueueLinks({
        selector: '.collection-block-item',
        label: 'CATEGORY',
        globs: ['https://warehouse-theme-metal.myshopify.com/collections/*'],
    });
});

// CATEGORY handler: finds product links on category pages
router.addHandler('CATEGORY', async ({ request, page, enqueueLinks, log }) => {
    log.debug(`Enqueueing products from: ${request.url}`);
    await page.waitForSelector('.product-item > a');
    await enqueueLinks({
        selector: '.product-item > a',
        label: 'DETAIL',
        globs: ['https://warehouse-theme-metal.myshopify.com/products/*'],
        exclude: ['**/account/**', '**/cart/**', '**/policies/**'],
    });
});

What does this code do?

createPlaywrightRouter() creates a router that dispatches requests by label. You register handlers for specific labels, and unlabeled requests go to the default handler.

  • addDefaultHandler handles the start URL (which has no label). It waits for category link elements to render, then enqueues them with label: 'CATEGORY'.
  • addHandler('CATEGORY', ...) fires when a request arrives with the CATEGORY label. It enqueues product links with label: 'DETAIL'.
  • The selector option tells enqueueLinks to only look at links inside matching CSS elements, not every <a> tag on the page.

Why selector instead of bare globs?

In previous steps, enqueueLinks examined every link on the page. The selector option narrows the search to links inside specific CSS elements. Combined with label, this means each discovered link arrives at the right handler without any if/else logic.

  • Save src/routes.ts.
  • Confirm the file exists by running:
ls src/

You should see both main.ts and routes.ts listed.

Don't see routes.ts?

Make sure you created the file inside the src/ folder, not in the project root. The tsconfig.json only includes the src directory.

help me figure out why my routes.ts file isn't in the right place

Add the DETAIL handler for data extraction

The DETAIL handler runs when a product page request arrives. It uses the same Playwright locators you built earlier to pull structured data from each product page.

  • In src/routes.ts, add the following code below the CATEGORY handler:
router.addHandler('DETAIL', async ({ request, page, log }) => {
    log.debug(`Extracting data from: ${request.url}`);

    // Extract product fields using Playwright locators
    const title = await page.locator('.product-meta h1').textContent();
    const sku = await page.locator('span.product-meta__sku-number').textContent();
    const priceElement = page.locator('span.price').filter({ hasText: '$' }).first();
    const currentPriceString = await priceElement.textContent();
    const rawPrice = currentPriceString?.split('$')[1] ?? '0';
    const price = Number(rawPrice.replaceAll(',', ''));
    const inStockElement = page
        .locator('span.product-form__inventory')
        .filter({ hasText: 'In stock' })
        .first();
    const inStock = (await inStockElement.count()) > 0;

    // Build the results object and save to Dataset
    const results = {
        url: request.url,
        title,
        sku,
        currentPrice: price,
        availableInStock: inStock,
    };

    log.debug(`Saving data: ${request.url}`);
    await Dataset.pushData(results);
});

What does this handler do?

This is the same extraction logic from your previous if (request.url.includes('/products/')) block, now living in its own isolated handler. It only runs when a request has the DETAIL label.

The Dataset import at the top of the file handles saving. Each product's data lands as a JSON file in ./storage/datasets/default/.

  • Save src/routes.ts.
Update src/main.ts to use the router

With the router ready, src/main.ts becomes minimal. It only needs to create the crawler and point it at the router.

  • Replace the entire contents of src/main.ts with the following:
import { PlaywrightCrawler, log } from 'crawlee';
import { router } from './routes.js';

// Enable debug-level logging to see labeled handler messages
log.setLevel(log.LEVELS.DEBUG);

const crawler = new PlaywrightCrawler({
    requestHandler: router,
    maxRequestsPerCrawl: 20,
});

await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);

console.log('Crawler finished.');

What changed?

  • The inline requestHandler function is gone. The router import takes its place.
  • log.setLevel(log.LEVELS.DEBUG) turns on debug messages so you can see which handler fires for each request.
  • The import path uses ./routes.js (with a .js extension) because ESM TypeScript requires the compiled extension in import paths.
  • maxRequestsPerCrawl is increased to 20 so the crawler processes enough pages to demonstrate all three handlers.
  • Save src/main.ts.

Before you run this, what do you think the log output will look like compared to the previous version? Think about which messages you expect from each handler.

  • Run the crawler:
npx tsx src/main.ts

You'll see labeled debug log messages from each handler. Look for lines starting with "Enqueueing categories from page" (default handler), "Enqueueing products from" (CATEGORY handler), and "Extracting data from" (DETAIL handler).

  • Check the ./storage/datasets/default/ folder. You should see JSON files containing product objects with title, currentPrice, sku, and availableInStock fields.

Seeing an import error or "Cannot find module"?

  • Make sure the import in main.ts uses './routes.js' (not './routes.ts' or './routes'). ESM requires the .js extension.
  • Confirm that routes.ts exports the router with export const router (not a default export).
  • Check that package.json still has "type": "module".

help me fix my import error when running the Crawlee router

✔️ Awesome, I've got everything!

Great. Double check you have saved both src/routes.ts and src/main.ts.

ⓧ I'd like to double check the full code

Here is what src/main.ts should look like:

import { PlaywrightCrawler, log } from 'crawlee';
import { router } from './routes.js';

log.setLevel(log.LEVELS.DEBUG);

const crawler = new PlaywrightCrawler({
    requestHandler: router,
    maxRequestsPerCrawl: 20,
});

await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);

console.log('Crawler finished.');

For src/routes.ts, confirm you have these three sections in order:

  • The import line and export const router = createPlaywrightRouter() at the top.
  • The default handler (addDefaultHandler) that enqueues categories.
  • The CATEGORY handler (addHandler('CATEGORY', ...)) that enqueues product links.
  • The DETAIL handler (addHandler('DETAIL', ...)) that extracts and saves product data.

Your crawler architecture is now clean and scalable. Each page type has its own handler, and adding a new page type means registering one more handler instead of nesting another if/else branch. Next up, the Secret Mission challenges you to enrich requests on the fly and use regex patterns for advanced URL filtering.

Secret mission

Advanced Link Filtering with transformRequestFunction and Regexps

Production crawlers need more than simple globs. In this challenge, you'll enrich each request with category metadata using transformRequestFunction and replace glob patterns with regular expressions for surgical URL matching.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your project files for reference, pause by clearing cached data, or delete everything entirely.

Resources you used:

  • link-crawler project folder on your Desktop (source code, node_modules, and configuration files).
  • ./storage directory inside link-crawler (crawled datasets and request queue state).
  • Playwright browser binaries (cached globally by Playwright for browser automation).

Keep everything running

No action needed. Choose this if you want to keep your crawler code and crawled data for reference or further development.

  • Your source code stays in ~/Desktop/link-crawler.
  • Your crawled product data stays in ./storage/datasets/default/ as JSON files.
  • Playwright's browser binaries remain cached on your machine for future projects that use Playwright.

Pause - I'll come back to this later

Free up disk space by removing cached crawl data while keeping your source code intact. Crawlee automatically recreates the ./storage directory on the next run.

  • Delete the ./storage folder by running this from inside your link-crawler directory:
rm -rf ./storage
  • Optionally delete node_modules to free more space. You can reinstall later with npm install:
rm -rf node_modules

Delete - I don't want to use this again

Remove all project files, dependencies, and cached browser binaries completely.

  • Delete the entire project folder by running:
cd ~/Desktop
rm -rf link-crawler
  • Remove Playwright's cached browser binaries by running:
npx playwright uninstall --all

What does this command do?

Playwright downloads browser binaries (Chromium, Firefox, WebKit) to a global cache folder when installed. The npx playwright uninstall --all command removes all cached browsers from your system, freeing several hundred megabytes of disk space.

That's a Wrap!

That's a Wrap!

Nice work! You just built a production-style web crawler that intelligently discovers links across a multi-page e-commerce site and extracts structured product data from every product page it finds.

Here's what you learned:

  • Built a PlaywrightCrawler with intelligent link discovery using enqueueLinks options like globs, exclude, selector, and label to control exactly which URLs your crawler follows.
  • Extracted structured product data from HTML using Playwright locators and saved it to Crawlee's Dataset as clean JSON files ready for downstream processing.
  • Refactored from a monolithic requestHandler to clean, modular route handlers with createPlaywrightRouter and labeled requests that classify each discovered page by type.
  • Secret Mission: Added transformRequestFunction for request enrichment and regexps for advanced URL pattern matching.

Ready to quiz yourself?