Build a Bounded Web Crawler

Crawl a local site safely and search its pages with Meilisearch.

Introduction

30 Second Summary

Finding one useful page becomes frustrating when you remember the topic but not the link. A site full of cross-links can also send an automated visitor around the same loop forever.

In this project, you will build a bounded web crawler that collects pages from a miniature documentation site. You will index the content in Meilisearch to power local full-text search in a browser.

What You'll Build

A search for timeouts returns a Crawling Safely result whose text snippet links to the matching documentation page.

By the end of this project, you'll have:

  • A visible reliability demo where you watch a naive crawl repeat URLs. You then run the bounded version to see each unique page fetched once.
  • A synchronized page index that reports Indexed 4 unique pages after a successful crawl. Pages missing from the latest crawl disappear from the stored results.
  • A browser search interface that returns matching titles with text snippets. Each result opens the original fixture page.
  • Secret Mission: Read robots.txt before crawling so disallowed private content disappears from refreshed search results.

Are there any prerequisites?

You should already have Node.js plus Docker installed. Comfort with JavaScript plus terminal basics prepares you for the crawler code.

Before We Start

Before any hands-on work begins, this step locks in the crawler's scope: unique HTML documentation pages from one local origin. Keeping that crawl bounded prevents cyclic links, fragments, failed requests, or external URLs from creating unbounded work.

Set Up the Local Search Stack

A bounded crawler behaves consistently only when its runtime is pinned. Its local search backend also needs a repeatable container configuration.

You'll verify Node.js at 24.22.0 LTS. You'll also confirm that Docker Compose is available through Docker Desktop.

Visual Studio Code will hold the project configuration. Meilisearch will run locally from the pinned getmeili/meilisearch:v1.55.0 image.

In this step, get ready to:
  • Confirm that your local toolchain is ready.
  • Define the project's pinned dependencies and container configuration.
  • Launch the local Meilisearch service.
Verify the tools and prepare the editor

A version check catches runtime drift before it turns into confusing crawler errors. Start with the runtime that executes every project script.

  • Check your installed Node.js release by running:
node --version

✔️ I see the current LTS version

Your terminal should show v24.22.0. The runtime now matches the project pin.

ⓧ I see an older or unsupported version

  • Open the official Node.js download page.
  • Download the installer for v24.22.0 LTS.
  • Complete the installer for your operating system.
  • Close every open terminal window.
  • Press Cmd+Space on macOS or the Windows key on Windows.
  • Type Terminal into the search bar.
  • Press Enter to open a fresh terminal.
  • Verify the new release by running:
node --version

You should now see v24.22.0.

ⓧ Command not found

  • Open the official Node.js download page.
  • Download the installer for v24.22.0 LTS.
  • Complete the installer for your operating system.
  • Close every open terminal window.
  • Press Cmd+Space on macOS or the Windows key on Windows.
  • Type Terminal into the search bar.
  • Press Enter to open a fresh terminal.
  • Confirm the installation by running:
node --version

You should now see v24.22.0.

Docker Compose turns the project configuration into a repeatable local service. Its version command confirms that the Compose plugin is available.

  • Check Docker Compose by running:
docker compose version

✔️ I see version information

Docker Compose is available. You can now start services from compose.yaml.

ⓧ The command is unavailable

Docker Desktop includes Docker Compose. Reinstalling or updating Docker Desktop restores the Compose plugin when it is unavailable.

  • Open the official Docker Compose installation guide.
  • Install Docker Desktop for your operating system.
  • Close every open terminal window.
  • Press Cmd+Space on macOS or the Windows key on Windows.
  • Type Terminal into the search bar.
  • Press Enter to open a fresh terminal.
  • Check Docker Compose again by running:
docker compose version

Still missing Docker Compose?

Ask for help: Why is Docker Compose unavailable after installing Docker Desktop?

The final tool is your editor. Check whether Visual Studio Code is already available before installing it.

  • Press Cmd+Space on macOS or the Windows key on Windows.
  • Type Visual Studio Code into the search bar.

✔️ Visual Studio Code is listed

  • Press Enter to open Visual Studio Code.

ⓧ Visual Studio Code is not listed

  • Open the official Visual Studio Code setup guide.
  • Choose the installer for your operating system.
  • Complete the installation.
  • Press Cmd+Space on macOS or the Windows key on Windows.
  • Type Visual Studio Code into the search bar.
  • Press Enter to open Visual Studio Code.

Your runtime, container tooling, and editor are ready. That removes the machine-specific drift before the crawler makes its first request.

Create the project configuration

The project needs one manifest for its JavaScript packages. A second file describes the local search container.

  • Select File from the Visual Studio Code menu.
  • Select Open Folder....
  • Select your Desktop in the folder dialog.
  • Select New Folder.
  • Enter bounded-crawler-search as the folder name.
  • Select Select Folder on Windows or Open on macOS.
  • Select Yes, I trust the authors if the Workspace Trust dialog appears.

You should see bounded-crawler-search at the top of the Explorer sidebar.

  • Select the New File icon in the Explorer sidebar.
  • Enter package.json as the filename.
  • Define the project scripts and pinned dependencies by pasting this configuration:
{
  "name": "bounded-crawler-search",
  "version": "1.0.0",
  "private": true,
  "type": "module",
  "scripts": {
    "start": "node app.js",
    "crawl": "node crawler.js"
  },
  "dependencies": {
    "cheerio": "1.2.0",
    "meilisearch": "0.62.0"
  }
}

What does this configuration do?

  • The type value enables ES modules for the JavaScript files created later.
  • The scripts object defines the server command and crawler command used in later steps.
  • The dependencies object pins Cheerio at 1.2.0.
  • The same object pins the Meilisearch JavaScript SDK at 0.62.0.
  • Save package.json.
  • Select View from the Visual Studio Code menu.
  • Select Terminal.
  • Install the pinned project dependencies by running:
npm install

When the command finishes, you should see node_modules in the Explorer sidebar. Your project now has the exact parser and search client it needs.

Did the dependency install fail?

  • Check that the terminal prompt ends inside the bounded-crawler-search folder.
  • Compare every comma and quote in package.json with the configuration above.

Ask for help: Why is npm install failing in my bounded-crawler-search folder?

  • Select the New File icon in the Explorer sidebar.
  • Enter compose.yaml as the filename.
  • Define the local Meilisearch service by pasting this configuration:
services:
  meilisearch:
    image: getmeili/meilisearch:v1.55.0
    ports:
      - "7700:7700"
    volumes:
      - meili_data:/meili_data

volumes:
  meili_data:

What does this configuration do?

  • The image value pins the Meilisearch engine to v1.55.0.
  • The ports mapping makes the local service available on port 7700.
  • The meili_data named volume preserves search data across container restarts.
  • Save compose.yaml.

The Explorer sidebar should now show package.json, compose.yaml, and node_modules inside bounded-crawler-search.

Missing a project file?

  • Check that both filenames appear directly under bounded-crawler-search in the Explorer sidebar.
  • Check that the Compose filename is compose.yaml.

Ask for help: Why are my project configuration files missing from the Explorer sidebar?

✔️ Awesome, I've got everything!

Both configuration files are saved in the project folder. Your dependency directory is also present.

ⓧ I'd like to double check the full code

  • Compare your two configuration files with these complete references.
{
  "name": "bounded-crawler-search",
  "version": "1.0.0",
  "private": true,
  "type": "module",
  "scripts": {
    "start": "node app.js",
    "crawl": "node crawler.js"
  },
  "dependencies": {
    "cheerio": "1.2.0",
    "meilisearch": "0.62.0"
  }
}
services:
  meilisearch:
    image: getmeili/meilisearch:v1.55.0
    ports:
      - "7700:7700"
    volumes:
      - meili_data:/meili_data

volumes:
  meili_data:
Start the local search service

Docker Compose can now turn compose.yaml into a running local search service. The detached mode keeps the service running after the command returns.

The first start can take a few minutes while Docker downloads the image. The download is complete when the command returns control to the terminal.

Before you run this, which pinned image do you expect Docker Compose to start?

  • Start the local Meilisearch container by running:
docker compose up -d

Docker Compose starts getmeili/meilisearch:v1.55.0 in the background. Port 7700 now connects your machine to the container.

Keep This Service Local

This Meilisearch instance has no master key. Keep port 7700 on your local development machine.

Did the container fail to start?

  • Confirm that Docker Desktop has finished starting.
  • Check the indentation in compose.yaml against the full-code reference.

Ask for help: Why is docker compose up failing for my Meilisearch service?

Before you load the local address, what kind of interface do you expect a development-mode search service to show?

  • Open a new browser tab.
  • Enter http://localhost:7700 in the address bar.

You should see the Meilisearch search preview. That confirms the pinned container is running and reachable through port 7700.

Can't reach the preview?

  • Check that the address uses port 7700.
  • Return to the Visual Studio Code terminal to confirm that the Compose command completed.

Ask for help: Why can't my browser reach the local Meilisearch preview?

Your local search backend is live and ready to receive documents. Next, you'll build a fixture site and watch a naive crawler revisit the same cyclic links.

Make the Crawler Fail Safely

Your local Meilisearch instance is running. You now need a predictable website that exposes unsafe crawler behavior without depending on an external site.

A crawler can look busy while repeatedly following the same cycle of links. You will use a deterministic fixture plus a naive breadth-first crawl powered by Cheerio to make that repeated work visible.

In this step, get ready to:
  • Serve a deterministic local documentation fixture.
  • Build a naive Cheerio crawler with an eight-fetch limit.
  • Run the crawler to expose repeated URLs.
Build the local documentation fixture

The fixture gives every learner the same link graph. Its starting page links to documentation pages plus a private page.

  • In the Visual Studio Code file sidebar, select the folder that contains package.json.
  • Use the new-file control above the sidebar to create app.js beside package.json.
  • Add the fixture port plus the starting page to app.js with this code:
import * as http from 'node:http';

const PORT = 3000;

const pages = {
  '/docs/start': `<!doctype html>
<html lang="en">
  <head><meta charset="utf-8"><title>Start Here</title></head>
  <body>
    <main>
      <h1>Start Here</h1>
      <p>This guide introduces a bounded crawler and a local search index.</p>
      <a href="/docs/crawling">Crawling safely</a>
      <a href="/docs/search#ranking">Search ranking</a>
      <a href="/private/notes">Private notes</a>
    </main>
  </body>
</html>`,
};

What does this code do?

  • The node:http import provides the local HTTP server used by the fixture.
  • The PORT value fixes the server address at port 3000.
  • The pages object maps each request path to a complete HTML document.
  • The links on /docs/start create the first branches in the crawl queue.
  • Save app.js.
  • Confirm app.js appears beside package.json in the file sidebar.

Can't find app.js?

Check that you created app.js in the folder containing package.json. A file created inside node_modules belongs to the wrong location.

Ask for help: Help me locate app.js in my Visual Studio Code project and place it beside package.json.

  • Add the response helper plus the fixture server below the pages object with this code:
function send(response, statusCode, contentType, body) {
  response.writeHead(statusCode, { 'Content-Type': contentType });
  response.end(body);
}

const server = http.createServer((request, response) => {
  const requestUrl = new URL(request.url || '/', 'http://localhost:3000');
  const page = pages[requestUrl.pathname];

  if (page) {
    send(response, 200, 'text/html; charset=utf-8', page);
    return;
  }

  send(response, 404, 'text/plain; charset=utf-8', 'Not found');
});

server.listen(PORT, '127.0.0.1', () => {
  console.log(`Fixture: http://localhost:${PORT}`);
});

How does the fixture respond?

  • The send() helper writes a status code plus a content type before returning a response body.
  • The server uses the request path to select a document from pages.
  • Unknown paths receive a 404 response.
  • The server.listen() call keeps the fixture available on the local machine.
  • Save app.js.
  • Return to the project terminal from the setup step.
  • Start the fixture server by running:
npm start

What does this command start?

The start script runs app.js. The terminal stays occupied because the server continues listening for browser requests.

  • Open http://localhost:3000/docs/start in your browser.

You should see the Start Here heading plus links for crawling, search ranking, and private notes.

Fixture page not loading?

  • Confirm the terminal running npm start remains open.
  • Check that server.listen() uses PORT.
  • Check that the browser address includes /docs/start.

Ask for help: Help me debug why my local fixture is not loading on port 3000.

The other fixture pages complete the crawl graph. Their fragment links create distinct URL strings that still point to familiar documents.

  • In app.js, find the closing }; line for the pages object.
  • Insert these documentation pages immediately above that closing line:
  '/docs/crawling': `<!doctype html>
<html lang="en">
  <head><meta charset="utf-8"><title>Crawling Safely</title></head>
  <body>
    <main>
      <h1>Crawling Safely</h1>
      <p>Visited sets, page limits, same-origin checks, and timeouts keep a crawler bounded.</p>
      <a href="/docs/start#top">Back to start</a>
      <a href="/docs/search">Search indexing</a>
    </main>
  </body>
</html>`,
  '/docs/search': `<!doctype html>
<html lang="en">
  <head><meta charset="utf-8"><title>Search Indexing</title></head>
  <body>
    <main>
      <h1>Search Indexing</h1>
      <p>Documents contain a stable id, source URL, title, and searchable page content.</p>
      <a href="/docs/crawling#timeouts">Crawler timeouts</a>
      <a href="/docs/start">Start again</a>
    </main>
  </body>
</html>`,

How does the cycle form?

  • The crawling page links back to the start page through the #top fragment.
  • The search page links back to the crawling page through the #timeouts fragment.
  • Both pages keep feeding earlier documents back into the queue.
  • Save app.js.
  • Confirm the /docs/crawling entry appears above the closing line of pages.
  • Confirm the /docs/search entry appears above the closing line of pages.

Seeing a syntax problem?

Check that the starting page ends with a comma. Check that the crawling page also ends with a comma.

Ask for help: Help me check the commas and template literal boundaries in my pages object.

  • In app.js, find the closing }; line for the pages object again.
  • Insert the private fixture page immediately above that closing line:
  '/private/notes': `<!doctype html>
<html lang="en">
  <head><meta charset="utf-8"><title>Private Notes</title></head>
  <body>
    <main>
      <h1>Private Notes</h1>
      <p>This fixture page demonstrates content excluded by robots.txt.</p>
      <a href="/docs/start">Return to start</a>
    </main>
  </body>
</html>`

Why include a private page?

The private page gives the fixture a fourth reachable document. Its return link adds another route back to the starting page.

  • Save app.js.
  • Stop the running fixture from the Visual Studio Code terminal panel.
  • Restart the fixture server by running:
npm start

Why restart the fixture?

The running Node.js process loaded the earlier version of app.js. Restarting it loads all four fixture pages into the server.

  • Open http://localhost:3000/docs/crawling in your browser.
  • Open http://localhost:3000/docs/search in your browser.
  • Open http://localhost:3000/private/notes in your browser.

You should see the Crawling Safely, Search Indexing, and Private Notes pages. Each page includes a link that feeds the crawl graph.

One page returning 404?

Compare the missing request path with its property name inside pages. A missing slash or changed spelling prevents the lookup from finding the page.

Ask for help: Help me find why one fixture route returns 404 from app.js.

✔️ Awesome, I've got everything!

Great. Your four-page fixture is running with its cyclic links ready for the crawler.

ⓧ I'd like to double check the full code

Compare your complete app.js file with this reference.

import * as http from 'node:http';

const PORT = 3000;

const pages = {
  '/docs/start': `<!doctype html>
<html lang="en">
  <head><meta charset="utf-8"><title>Start Here</title></head>
  <body>
    <main>
      <h1>Start Here</h1>
      <p>This guide introduces a bounded crawler and a local search index.</p>
      <a href="/docs/crawling">Crawling safely</a>
      <a href="/docs/search#ranking">Search ranking</a>
      <a href="/private/notes">Private notes</a>
    </main>
  </body>
</html>`,
  '/docs/crawling': `<!doctype html>
<html lang="en">
  <head><meta charset="utf-8"><title>Crawling Safely</title></head>
  <body>
    <main>
      <h1>Crawling Safely</h1>
      <p>Visited sets, page limits, same-origin checks, and timeouts keep a crawler bounded.</p>
      <a href="/docs/start#top">Back to start</a>
      <a href="/docs/search">Search indexing</a>
    </main>
  </body>
</html>`,
  '/docs/search': `<!doctype html>
<html lang="en">
  <head><meta charset="utf-8"><title>Search Indexing</title></head>
  <body>
    <main>
      <h1>Search Indexing</h1>
      <p>Documents contain a stable id, source URL, title, and searchable page content.</p>
      <a href="/docs/crawling#timeouts">Crawler timeouts</a>
      <a href="/docs/start">Start again</a>
    </main>
  </body>
</html>`,
  '/private/notes': `<!doctype html>
<html lang="en">
  <head><meta charset="utf-8"><title>Private Notes</title></head>
  <body>
    <main>
      <h1>Private Notes</h1>
      <p>This fixture page demonstrates content excluded by robots.txt.</p>
      <a href="/docs/start">Return to start</a>
    </main>
  </body>
</html>`
};

function send(response, statusCode, contentType, body) {
  response.writeHead(statusCode, { 'Content-Type': contentType });
  response.end(body);
}

const server = http.createServer((request, response) => {
  const requestUrl = new URL(request.url || '/', 'http://localhost:3000');
  const page = pages[requestUrl.pathname];

  if (page) {
    send(response, 200, 'text/html; charset=utf-8', page);
    return;
  }

  send(response, 404, 'text/plain; charset=utf-8', 'Not found');
});

server.listen(PORT, '127.0.0.1', () => {
  console.log(`Fixture: http://localhost:${PORT}`);
});
Create the naive crawler

The first crawler follows links in queue order. Every discovered URL goes back into the queue because there is no record of earlier work.

  • In the Visual Studio Code file sidebar, select the folder that contains app.js.
  • Use the new-file control above the sidebar to create crawler.js beside app.js.
  • Add the naive crawl loop plus its temporary safety guard with this code:
import * as cheerio from 'cheerio';

const SEED_URL = process.argv[2] || 'http://localhost:3000/docs/start';
const MAX_FETCHES = 8;

async function crawl(seedUrl) {
  const queue = [seedUrl];
  let fetches = 0;

  while (queue.length > 0 && fetches < MAX_FETCHES) {
    const current = queue.shift();
    const response = await fetch(current);
    const html = await response.text();
    const $ = cheerio.load(html);

    fetches += 1;
    console.log(`Fetched: ${current}`);

    $('a[href]').each((_, element) => {
      const href = $(element).attr('href');
      if (href) queue.push(new URL(href, current).href);
    });
  }

  console.log(`Stopped after ${fetches} fetches`);
}

await crawl(SEED_URL);

How does this crawler work?

  • The SEED_URL value gives the queue its first page.
  • The loop removes one URL from the front of queue before fetching its HTML.
  • Cheerio selects every a[href] link from the downloaded document.
  • The MAX_FETCHES guard stops the experiment after eight requests.
  • Every discovered link enters the queue because this version does not track completed URLs.
  • Save crawler.js.
  • Confirm crawler.js appears beside app.js in the file sidebar.

Crawler file showing a syntax error?

  • Check that each opening brace has a matching closing brace.
  • Check that the Cheerio selector remains a[href].
  • Check that await crawl(SEED_URL); remains below the function.

Ask for help: Help me compare my crawler.js structure with the naive crawler target.

✔️ Awesome, I've got everything!

Your naive crawler is ready. Keep the fixture server running before you test it.

ⓧ I'd like to double check the full code

Compare your complete crawler.js file with this reference.

import * as cheerio from 'cheerio';

const SEED_URL = process.argv[2] || 'http://localhost:3000/docs/start';
const MAX_FETCHES = 8;

async function crawl(seedUrl) {
  const queue = [seedUrl];
  let fetches = 0;

  while (queue.length > 0 && fetches < MAX_FETCHES) {
    const current = queue.shift();
    const response = await fetch(current);
    const html = await response.text();
    const $ = cheerio.load(html);

    fetches += 1;
    console.log(`Fetched: ${current}`);

    $('a[href]').each((_, element) => {
      const href = $(element).attr('href');
      if (href) queue.push(new URL(href, current).href);
    });
  }

  console.log(`Stopped after ${fetches} fetches`);
}

await crawl(SEED_URL);
Run the repeated-work experiment

The queue now has everything required to reveal the cycle. The eight-fetch guard contains the experiment if those links keep generating more work.

  • Keep the terminal running npm start active.
  • Add another terminal panel in Visual Studio Code for the crawler.

Before you run the crawler, predict whether eight fetches can produce eight unique page paths.

  • Run the naive crawler from the second terminal by running:
npm run crawl

What did the crawl reveal?

The output repeats documentation URLs before ending with Stopped after 8 fetches. Fragment variants such as #top plus #timeouts remain distinct queue entries.

The cycle keeps earlier pages reachable from later pages. The crawler therefore mistakes repeated work for progress.

You have exposed the exact failure safely. The temporary limit prevents the naive queue from running forever while preserving evidence of the problem.

Crawler not reaching eight fetches?

  • Confirm the fixture terminal still shows the server running.
  • Open http://localhost:3000/docs/start to confirm the seed page responds.
  • Check that every fixture link matches the reference app.js file.

Ask for help: Help me debug why my naive crawler does not print eight fetched URLs.

Your crawler now demonstrates why a queue needs explicit safety rules. Next, you will normalize URLs, suppress duplicates, enforce the origin boundary, isolate failures, and index the clean result.

Bound, Deduplicate, and Index the Crawl

The last run exposed the crawler's core weakness. A queue that accepts every discovered URL cannot distinguish fresh work from a cycle.

This step turns that queue into a bounded breadth-first crawl. It also converts each page into a document for Meilisearch.

In this step, get ready to:
  • Normalize discovered URLs and enforce a same-origin boundary.
  • Deduplicate pages and isolate individual request failures.
  • Synchronize the latest crawl with the local search index.
Replace the naive crawler

URL normalization converts equivalent links into one canonical address. Removing fragments means links such as /docs/search#ranking and /docs/search represent the same page.

A Set records every attempted URL. The origin comparison keeps discovered links inside the fixture that supplied the seed.

  • Switch back to crawler.js from earlier.
  • Select everything above the final await crawl(SEED_URL); line.
  • Replace the selected code with the following four sections in order:
  • Paste this first section at the top of crawler.js:
import { Buffer } from 'node:buffer';
import * as cheerio from 'cheerio';
import { Meilisearch } from 'meilisearch';

const SEED_URL = process.argv[2] || 'http://localhost:3000/docs/start';
const MAX_PAGES = 10;
const REQUEST_TIMEOUT_MS = 3000;
const USER_AGENT = 'NextWorkCrawler';
const client = new Meilisearch({ host: 'http://127.0.0.1:7700' });

function normalizeUrl(value, base) {
  const url = new URL(value, base);
  url.hash = '';
  return url.href;
}

What does this section do?

  • The MAX_PAGES value caps the crawl at ten unique attempts.
  • The REQUEST_TIMEOUT_MS value gives each request three seconds to finish.
  • The client connects the crawler to the local search service from earlier.
  • The normalizeUrl() function resolves relative links and removes their fragments.
  • Continue the replacement by pasting this section directly below normalizeUrl():
async function crawl(seedUrl) {
  const seed = normalizeUrl(seedUrl, seedUrl);
  const origin = new URL(seed).origin;
  const queue = [seed];
  const seen = new Set();
  const documents = [];
  const failures = [];

  while (queue.length > 0 && seen.size < MAX_PAGES) {
    const current = queue.shift();
    if (seen.has(current)) continue;
    seen.add(current);

    try {
      const response = await fetch(current, {
        headers: { 'user-agent': USER_AGENT },
        signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS)
      });

      if (!response.ok) throw new Error(`HTTP ${response.status}`);
      const contentType = response.headers.get('content-type') || '';
      if (!contentType.includes('text/html')) {
        console.log(`Skipped non-HTML: ${current}`);
        continue;
      }

How is the crawl bounded?

  • The origin value captures the seed's origin for later link checks.
  • The seen set records a URL before its request begins.
  • The loop stops when the queue is empty or ten unique URLs have been attempted.
  • The timeout prevents one slow page from holding the entire crawl open.
  • Add the document extraction and link discovery section directly below the content-type check:
      const html = await response.text();
      const $ = cheerio.load(html);
      $('script, style').remove();

      const title = $('title').text().trim() || current;
      const content = $('body').text().replace(/\s+/g, ' ').trim();
      documents.push({
        id: Buffer.from(current).toString('base64url'),
        url: current,
        title,
        content
      });
      console.log(`Fetched: ${current}`);

      $('a[href]').each((_, element) => {
        const href = $(element).attr('href');
        if (!href) return;
        const candidate = normalizeUrl(href, current);
        if (new URL(candidate).origin === origin && !seen.has(candidate)) {
          queue.push(candidate);
        }
      });

How does a page become a document?

  • Cheerio parses each HTML response so the crawler can select the title and body.
  • Removing script and style elements keeps their source code out of the searchable content.
  • The encoded URL becomes a stable document id.
  • Only normalized links from the seed origin enter the queue.
  • Complete the crawl() function by pasting this final section directly below the link discovery code:
    } catch (error) {
      failures.push({ url: current, reason: error.message });
      console.error(`Failed: ${current} (${error.message})`);
    }
  }

  return { documents, failures };
}

Why isolate each failure?

The try block covers one page request at a time. A failed page enters failures while the queue continues with the next URL.

The function returns successful documents separately from failed attempts. That separation makes the crawl useful even when one page is unavailable.

  • Save crawler.js.
Verify the bounded page set

The final line left from the previous step still invokes crawl(). This gives you a quick check before any documents reach the search index.

Before you run the crawler, which repeated fragment URLs do you expect normalization to remove?

  • Run the bounded crawl in the second terminal from earlier with this command:
npm run crawl

You should see four Fetched: lines. Each fixture URL appears once.

That is the reliability boundary working. The cycle now reaches a finite set of unique pages instead of generating repeated fetches.

Still seeing repeats or failed requests?

  • Confirm the fixture terminal from earlier is still running.
  • Check that url.hash = ''; appears inside normalizeUrl().
  • Check that seen.add(current); appears before the page request.

Ask for help: My bounded crawler still repeats pages or cannot fetch the fixture. Help me compare the queue, normalization, and seen-set logic with the expected crawler.

Synchronize the Meilisearch index

The crawl now returns a clean snapshot of the fixture. Synchronization writes that snapshot to the pages index and removes documents that disappeared from later crawls.

Meilisearch processes document changes asynchronously. The crawler waits for each indexing task before reading or deleting documents.

  • Replace the final await crawl(SEED_URL); line in crawler.js with this synchronization section:
async function synchronizeIndex(documents) {
  if (documents.length === 0) throw new Error('No HTML documents were crawled');

  const index = client.index('pages');
  const addTask = await index.addDocuments(documents);
  await client.waitForTask(addTask.taskUid);

  const stored = await index.getDocuments({ limit: MAX_PAGES + 10 });
  const currentIds = new Set(documents.map((document) => document.id));
  const staleIds = stored.results
    .filter((document) => !currentIds.has(document.id))
    .map((document) => document.id);

  if (staleIds.length > 0) {
    const deleteTask = await index.deleteDocuments(staleIds);
    await client.waitForTask(deleteTask.taskUid);
  }

  return index;
}

const { documents, failures } = await crawl(SEED_URL);
const index = await synchronizeIndex(documents);
const check = await index.search('timeouts');

console.log(`Indexed ${documents.length} unique pages`);
console.log(`Failures: ${failures.length}`);
console.log(`Search check: ${check.hits.map((hit) => hit.title).join(', ')}`);

How does synchronization work?

  • The function rejects an empty crawl so it cannot erase the useful index after a complete crawl failure.
  • The add task writes the latest documents and updates matching IDs.
  • The currentIds set identifies stored documents that are absent from the latest crawl.
  • The final search proves that indexing finished before the script exits.
  • Save crawler.js.

Before you run the complete pipeline, what count should a deduplicated crawl report for the four fixture pages?

  • Run the crawl and indexing pipeline with this command:
npm run crawl

You should see Indexed 4 unique pages and Failures: 0.

The final line should include Search check: Crawling Safely. This confirms that the indexed body text can retrieve its source page.

Indexing or search check failed?

  • Confirm the local Meilisearch container from earlier is still running.
  • Check that the client host remains http://127.0.0.1:7700.
  • Compare the synchronizeIndex() task waits with the code below.

Ask for help: My crawl fetches four pages but the Meilisearch indexing or timeouts search check fails. Help me inspect the local container connection and synchronization tasks.

✔️ Awesome, I've got everything!

Great. Your crawler now bounds the queue and synchronizes four searchable documents.

ⓧ I'd like to double check the full code

Compare your saved crawler.js with this complete version.

import { Buffer } from 'node:buffer';
import * as cheerio from 'cheerio';
import { Meilisearch } from 'meilisearch';

const SEED_URL = process.argv[2] || 'http://localhost:3000/docs/start';
const MAX_PAGES = 10;
const REQUEST_TIMEOUT_MS = 3000;
const USER_AGENT = 'NextWorkCrawler';
const client = new Meilisearch({ host: 'http://127.0.0.1:7700' });

function normalizeUrl(value, base) {
  const url = new URL(value, base);
  url.hash = '';
  return url.href;
}

async function crawl(seedUrl) {
  const seed = normalizeUrl(seedUrl, seedUrl);
  const origin = new URL(seed).origin;
  const queue = [seed];
  const seen = new Set();
  const documents = [];
  const failures = [];

  while (queue.length > 0 && seen.size < MAX_PAGES) {
    const current = queue.shift();
    if (seen.has(current)) continue;
    seen.add(current);

    try {
      const response = await fetch(current, {
        headers: { 'user-agent': USER_AGENT },
        signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS)
      });

      if (!response.ok) throw new Error(`HTTP ${response.status}`);
      const contentType = response.headers.get('content-type') || '';
      if (!contentType.includes('text/html')) {
        console.log(`Skipped non-HTML: ${current}`);
        continue;
      }

      const html = await response.text();
      const $ = cheerio.load(html);
      $('script, style').remove();

      const title = $('title').text().trim() || current;
      const content = $('body').text().replace(/\s+/g, ' ').trim();
      documents.push({
        id: Buffer.from(current).toString('base64url'),
        url: current,
        title,
        content
      });
      console.log(`Fetched: ${current}`);

      $('a[href]').each((_, element) => {
        const href = $(element).attr('href');
        if (!href) return;
        const candidate = normalizeUrl(href, current);
        if (new URL(candidate).origin === origin && !seen.has(candidate)) {
          queue.push(candidate);
        }
      });
    } catch (error) {
      failures.push({ url: current, reason: error.message });
      console.error(`Failed: ${current} (${error.message})`);
    }
  }

  return { documents, failures };
}

async function synchronizeIndex(documents) {
  if (documents.length === 0) throw new Error('No HTML documents were crawled');

  const index = client.index('pages');
  const addTask = await index.addDocuments(documents);
  await client.waitForTask(addTask.taskUid);

  const stored = await index.getDocuments({ limit: MAX_PAGES + 10 });
  const currentIds = new Set(documents.map((document) => document.id));
  const staleIds = stored.results
    .filter((document) => !currentIds.has(document.id))
    .map((document) => document.id);

  if (staleIds.length > 0) {
    const deleteTask = await index.deleteDocuments(staleIds);
    await client.waitForTask(deleteTask.taskUid);
  }

  return index;
}

const { documents, failures } = await crawl(SEED_URL);
const index = await synchronizeIndex(documents);
const check = await index.search('timeouts');

console.log(`Indexed ${documents.length} unique pages`);
console.log(`Failures: ${failures.length}`);
console.log(`Search check: ${check.hits.map((hit) => hit.title).join(', ')}`);

Your crawler now contains cycles without losing useful pages. Next, you'll connect the indexed documents to a browser search interface.

Search the Crawled Pages in a Browser

Your bounded crawler has already placed four fixture pages in a local Meilisearch search index. Those documents still need a browser path where someone can issue a query.

Now you'll connect the Node.js server to that index. A browser form will submit queries. Result cards will show each matching title with a snippet.

In this step, get ready to:
  • Connect the server to Meilisearch through a search route.
  • Build a browser search form with safely rendered result cards.
  • Verify the complete browser search path.
Connect the server to Meilisearch

The server needs its own Meilisearch client before it can accept browser queries. This client points to the same local search service that contains your four indexed documents.

  • In app.js, replace the imports plus the PORT declaration at the top with this code:
import * as http from 'node:http';
import { Meilisearch } from 'meilisearch';

const PORT = 3000;
const client = new Meilisearch({ host: 'http://127.0.0.1:7700' });

What does this code do?

  • The Meilisearch import gives your server access to the installed search SDK.
  • The client connects to the local service that stores the pages index.
  • Save app.js.
  • Return to the terminal running the server from earlier.
  • Press Ctrl+C to stop the current server process.
  • Restart the server with the new client by running:
npm start

What does this restart prove?

The server starts with the Meilisearch client loaded. Your fixture routes still work because the search client only adds a new connection point.

  • Return to the browser tab from earlier.
  • Load http://localhost:3000/docs/start to confirm the fixture still shows Start Here.

Server not starting?

  • Check that the Meilisearch import appears directly below the existing HTTP import.
  • Confirm the client declaration appears directly below PORT.

Ask for help: Help me debug the Meilisearch client setup in app.js.

An API route turns a browser query into a Meilisearch request. It returns matching documents as JSON so the browser can render them.

  • In app.js, find the server block beginning with const server = http.createServer.
  • Replace the start of that block through the const page line with this code:
const server = http.createServer(async (request, response) => {
  const requestUrl = new URL(request.url || '/', 'http://localhost:3000');

  if (requestUrl.pathname === '/api/search') {
    const query = (requestUrl.searchParams.get('q') || '').trim();
    try {
      const searchResult = await client.index('pages').search(query);
      send(
        response,
        200,
        'application/json; charset=utf-8',
        JSON.stringify({ hits: searchResult.hits })
      );
    } catch (error) {
      send(
        response,
        500,
        'application/json; charset=utf-8',
        JSON.stringify({ error: error.message })
      );
    }
    return;
  }

  const page = pages[requestUrl.pathname];

How does the search route work?

  • The asynchronous server callback allows the route to wait for a search response.
  • The query value comes from the q parameter in the request URL.
  • The route searches the pages index. It returns searchResult.hits to the browser.
  • The try block converts search failures into a JSON response that the browser can display.
  • Save app.js.
  • Return to the server terminal.
  • Press Ctrl+C to stop the current process.
  • Restart the server with the search route by running:
npm start

What does this restart activate?

The restarted process now recognizes requests to /api/search. It can query the documents your crawler indexed earlier.

  • Load http://localhost:3000/api/search?q=timeouts in the browser address bar.

You'll see a JSON response containing a hits array. One matching document has the title Crawling Safely.

Good progress. Your browser can now retrieve indexed pages through the server.

Search route not returning results?

  • Confirm the local Meilisearch container from earlier remains running.
  • Check that the route searches the pages index.
  • Confirm the request URL includes the q parameter.

Ask for help: Help me debug why my search API route does not return Crawling Safely.

Build the browser search interface

The search page gives the indexed documents a usable front end. Its first version creates the form plus the result area before the browser behavior is added.

  • In app.js, find the closing }; beneath the pages object.
  • Insert this search page directly below that closing line:
const searchPage = `<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Local Crawl Search</title>
    <style>
      body { font-family: system-ui, sans-serif; max-width: 760px; margin: 3rem auto; padding: 0 1rem; color: #172033; }
      form { display: flex; gap: 0.5rem; }
      input { flex: 1; padding: 0.75rem; }
      button { padding: 0.75rem 1rem; cursor: pointer; }
      article { border-top: 1px solid #d7dce5; padding: 1rem 0; }
      a { color: #2156a5; font-weight: 700; }
    </style>
  </head>
  <body>
    <h1>Local Crawl Search</h1>
    <p>Search the pages collected from the bounded fixture crawl.</p>
    <form id="search-form">
      <input id="query" name="q" placeholder="Try: timeouts" required>
      <button>Search</button>
    </form>
    <p id="status" class="status"></p>
    <section id="results"></section>
    <script>
    </script>
  </body>
</html>`;

What does this page contain?

  • The form collects a required query through the query input.
  • The status paragraph provides a place for progress messages plus errors.
  • The results section provides a container for matching page cards.
  • The styles keep the interface readable while leaving room between result cards.
  • Save app.js.
  • Return to the server terminal.
  • Press Ctrl+C to stop the current process.
  • Check that the new page template has valid syntax by running:
npm start

What does this check prove?

The server starts with the complete searchPage template loaded. Your existing fixture remains available while you add the root route next.

  • Return to app.js.
  • Find the requestUrl declaration inside the server callback.
  • Insert these root plus policy routes directly below that declaration:
  if (requestUrl.pathname === '/') {
    send(response, 200, 'text/html; charset=utf-8', searchPage);
    return;
  }

  if (requestUrl.pathname === '/robots.txt') {
    send(
      response,
      200,
      'text/plain; charset=utf-8',
      'User-agent: NextWorkCrawler\nDisallow: /private/\n'
    );
    return;
  }

What do these routes serve?

  • The root route sends searchPage as an HTML response.
  • The /robots.txt route exposes a deterministic crawl policy for the local fixture.
  • Save app.js.
  • Return to the server terminal.
  • Press Ctrl+C to stop the current process.
  • Restart the server with the browser page route by running:
npm start

What does this restart reveal?

The root address now serves the search page instead of a missing-page response. Your documentation fixture remains available through its existing paths.

  • Load http://localhost:3000 in the browser.

You'll see the Local Crawl Search heading. A search field plus a Search button appear below it.

Search page not appearing?

  • Confirm the root route appears before the fixture page lookup.
  • Check that the root route sends the searchPage value.
  • Verify that the server terminal remains active while you load the page.

Ask for help: Help me debug why the Local Crawl Search page does not load at the root address.

The page still needs browser logic to call the search route. The script will create result cards with DOM elements.

  • In the searchPage styles, find the line beginning with a {.
  • Place this status style directly below that line:
      .status { color: #5a6475; min-height: 1.5rem; }

Why reserve status space?

The minimum height reserves a stable line for progress messages plus errors. The result area stays in place when the message changes.

  • Find the empty script element near the bottom of searchPage.
  • Insert this browser logic between its opening plus closing tags:
      const form = document.querySelector('#search-form');
      const input = document.querySelector('#query');
      const status = document.querySelector('#status');
      const results = document.querySelector('#results');

      form.addEventListener('submit', async (event) => {
        event.preventDefault();
        results.replaceChildren();
        status.textContent = 'Searching...';

        try {
          const response = await fetch('/api/search?q=' + encodeURIComponent(input.value));
          const payload = await response.json();
          if (!response.ok) throw new Error(payload.error || 'Search failed');

          status.textContent = payload.hits.length + ' result(s)';
          for (const hit of payload.hits) {
            const card = document.createElement('article');
            const link = document.createElement('a');
            const snippet = document.createElement('p');
            link.href = hit.url;
            link.textContent = hit.title;
            snippet.textContent = hit.content.slice(0, 180);
            card.append(link, snippet);
            results.append(card);
          }
        } catch (error) {
          status.textContent = error.message;
        }
      });

How does the browser script work?

  • The first four constants hold references to the form plus its output elements.
  • The submit listener requests /api/search with the encoded input value.
  • Each hit becomes an article containing a link plus a short snippet.
  • The script assigns titles plus snippets through textContent. Search data is displayed as text instead of injected as page markup.

✔️ Awesome, I've got everything!

Great. Double check that you've saved app.js before testing the complete search path.

ⓧ I'd like to double check the full code

Compare your complete app.js file against this version.

import * as http from 'node:http';
import { Meilisearch } from 'meilisearch';

const PORT = 3000;
const client = new Meilisearch({ host: 'http://127.0.0.1:7700' });

const pages = {
  '/docs/start': `<!doctype html>
<html lang="en">
  <head><meta charset="utf-8"><title>Start Here</title></head>
  <body>
    <main>
      <h1>Start Here</h1>
      <p>This guide introduces a bounded crawler and a local search index.</p>
      <a href="/docs/crawling">Crawling safely</a>
      <a href="/docs/search#ranking">Search ranking</a>
      <a href="/private/notes">Private notes</a>
    </main>
  </body>
</html>`,
  '/docs/crawling': `<!doctype html>
<html lang="en">
  <head><meta charset="utf-8"><title>Crawling Safely</title></head>
  <body>
    <main>
      <h1>Crawling Safely</h1>
      <p>Visited sets, page limits, same-origin checks, and timeouts keep a crawler bounded.</p>
      <a href="/docs/start#top">Back to start</a>
      <a href="/docs/search">Search indexing</a>
    </main>
  </body>
</html>`,
  '/docs/search': `<!doctype html>
<html lang="en">
  <head><meta charset="utf-8"><title>Search Indexing</title></head>
  <body>
    <main>
      <h1>Search Indexing</h1>
      <p>Documents contain a stable id, source URL, title, and searchable page content.</p>
      <a href="/docs/crawling#timeouts">Crawler timeouts</a>
      <a href="/docs/start">Start again</a>
    </main>
  </body>
</html>`,
  '/private/notes': `<!doctype html>
<html lang="en">
  <head><meta charset="utf-8"><title>Private Notes</title></head>
  <body>
    <main>
      <h1>Private Notes</h1>
      <p>This fixture page demonstrates content excluded by robots.txt.</p>
      <a href="/docs/start">Return to start</a>
    </main>
  </body>
</html>`
};

const searchPage = `<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Local Crawl Search</title>
    <style>
      body { font-family: system-ui, sans-serif; max-width: 760px; margin: 3rem auto; padding: 0 1rem; color: #172033; }
      form { display: flex; gap: 0.5rem; }
      input { flex: 1; padding: 0.75rem; }
      button { padding: 0.75rem 1rem; cursor: pointer; }
      article { border-top: 1px solid #d7dce5; padding: 1rem 0; }
      a { color: #2156a5; font-weight: 700; }
      .status { color: #5a6475; min-height: 1.5rem; }
    </style>
  </head>
  <body>
    <h1>Local Crawl Search</h1>
    <p>Search the pages collected from the bounded fixture crawl.</p>
    <form id="search-form">
      <input id="query" name="q" placeholder="Try: timeouts" required>
      <button>Search</button>
    </form>
    <p id="status" class="status"></p>
    <section id="results"></section>
    <script>
      const form = document.querySelector('#search-form');
      const input = document.querySelector('#query');
      const status = document.querySelector('#status');
      const results = document.querySelector('#results');

      form.addEventListener('submit', async (event) => {
        event.preventDefault();
        results.replaceChildren();
        status.textContent = 'Searching...';

        try {
          const response = await fetch('/api/search?q=' + encodeURIComponent(input.value));
          const payload = await response.json();
          if (!response.ok) throw new Error(payload.error || 'Search failed');

          status.textContent = payload.hits.length + ' result(s)';
          for (const hit of payload.hits) {
            const card = document.createElement('article');
            const link = document.createElement('a');
            const snippet = document.createElement('p');
            link.href = hit.url;
            link.textContent = hit.title;
            snippet.textContent = hit.content.slice(0, 180);
            card.append(link, snippet);
            results.append(card);
          }
        } catch (error) {
          status.textContent = error.message;
        }
      });
    </script>
  </body>
</html>`;

function send(response, statusCode, contentType, body) {
  response.writeHead(statusCode, { 'Content-Type': contentType });
  response.end(body);
}

const server = http.createServer(async (request, response) => {
  const requestUrl = new URL(request.url || '/', 'http://localhost:3000');

  if (requestUrl.pathname === '/') {
    send(response, 200, 'text/html; charset=utf-8', searchPage);
    return;
  }

  if (requestUrl.pathname === '/robots.txt') {
    send(
      response,
      200,
      'text/plain; charset=utf-8',
      'User-agent: NextWorkCrawler\nDisallow: /private/\n'
    );
    return;
  }

  if (requestUrl.pathname === '/api/search') {
    const query = (requestUrl.searchParams.get('q') || '').trim();
    try {
      const searchResult = await client.index('pages').search(query);
      send(
        response,
        200,
        'application/json; charset=utf-8',
        JSON.stringify({ hits: searchResult.hits })
      );
    } catch (error) {
      send(
        response,
        500,
        'application/json; charset=utf-8',
        JSON.stringify({ error: error.message })
      );
    }
    return;
  }

  const page = pages[requestUrl.pathname];
  if (page) {
    send(response, 200, 'text/html; charset=utf-8', page);
    return;
  }

  send(response, 404, 'text/plain; charset=utf-8', 'Not found');
});

server.listen(PORT, '127.0.0.1', () => {
  console.log(`Fixture and search UI: http://localhost:${PORT}`);
});
Verify the browser search path

This final check follows the whole request path from the browser to Meilisearch. It also confirms that a returned document opens its original fixture page.

Before you test it, which indexed page do you expect the query timeouts to return?

  • Save app.js.
  • Return to the server terminal.
  • Press Ctrl+C to stop the current process.
  • Restart the completed server by running:
npm start

What does the completed server provide?

This process serves the search page plus the fixture routes. It also forwards browser queries to the local Meilisearch index.

  • Load http://localhost:3000 in the browser.
  • Enter timeouts in the search field.
  • Click Search.

You'll see Crawling Safely among the results. Its card includes a text snippet from the indexed page.

  • Click the Crawling Safely result link.

You'll see the fixture page with the Crawling Safely heading. That's the complete path working from browser query to indexed result to source page.

You made it work. Your bounded crawler now feeds a browser search experience with relevant links plus readable snippets.

Search form not returning Crawling Safely?

  • Confirm the server terminal shows the completed fixture plus search UI address.
  • Check that crawler.js previously indexed four documents into the pages index.
  • Verify that the browser script requests /api/search?q= with the entered query.

Ask for help: Help me debug the browser-to-server-to-Meilisearch search path.

Secret mission

Respect robots.txt

Your bounded crawler still collects every same-origin page it discovers. Add a robots.txt policy gate that blocks private fixture content and removes the stale private document from search.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to keep your local resources running, pause them for another session, or remove them entirely. Everything runs on your machine, so this project has no paid resources or ongoing service costs.

Resources you used:

  • The running Node.js fixture server that provides the browser search interface.
  • The Docker Compose stack for local Meilisearch.
  • The persistent named volume called meili_data.
  • The pages index stored inside the named volume.
  • The local project files shown in Visual Studio Code.
  • The local node_modules dependency directory.

Keep everything running

No action is needed. Choose this option if you are still testing the crawler or experimenting with search queries.

  • Keep the Node.js fixture server running to preserve access to http://localhost:3000.
  • Keep the Docker Compose stack running to preserve access to http://localhost:7700.
  • Keep the local project files for future crawler experiments.
  • Limit access to http://localhost:7700 to your local machine because this development instance is unprotected.

Pause - I'll come back to this later

Shut down the running processes to free local memory. Your project files plus indexed data remain available for your next session.

  • Switch back to the terminal running the Node.js fixture server from earlier.
  • Press Ctrl+C to stop the fixture server.
  • Return to the terminal panel positioned in the folder containing compose.yaml.
  • Stop the Compose stack without deleting its named volume by running:
docker compose stop

What Does This Command Do?

Docker Compose stops the Meilisearch container. The meili_data volume remains stored on your machine.

The preserved volume retains the pages index for the next time you start the stack.

  • Refresh http://localhost:7700 in your browser.

The search preview no longer loads. Your crawler code plus indexed documents remain ready for another session.

Does the Search Preview Still Load?

Confirm that you ran the command from the folder containing compose.yaml. A different folder may refer to another Compose project.

Ask for help if the local stack still appears active.

Delete - I don't want to use this again

Deleting these resources is permanent. The cleanup only affects the local project resources listed above.

  • Switch back to the terminal running the Node.js fixture server from earlier.
  • Press Ctrl+C if the fixture server is still running.
  • Return to the terminal panel positioned in the folder containing compose.yaml.
  • Remove the Compose resources plus stored Meilisearch data by running:
docker compose down -v

What Does This Command Remove?

This command removes the Meilisearch container. It also removes the Compose network.

The volume option removes meili_data. That permanently deletes the stored pages index.

  • Refresh http://localhost:7700 in your browser.

The search preview no longer loads because the local Meilisearch container has been removed.

Did the Compose Cleanup Fail?

Confirm that the terminal is still positioned in the folder containing compose.yaml. Confirm that Docker Desktop is available to Docker Compose.

Ask for help with the local cleanup.

  • Return to Visual Studio Code from earlier.
  • Reveal the folder containing app.js in Finder on macOS or File Explorer on Windows by using the folder's context menu in Visual Studio Code.
  • Delete the revealed folder using your operating system's file deletion control.
  • Search the folder's former location for app.js.

You no longer find app.js at that location. Your source files plus local dependencies have now been removed.

Nice Work!

Nice Work!

You made it! Your bounded crawler now powers a local search experience through Meilisearch.

You've learned how to:

  • Built a cyclic local fixture that makes repeated work visible. This proved why breadth-first crawling needs explicit safety limits.
  • Added crawler boundaries around every run. URL normalization removes fragments. Duplicate suppression prevents cyclic refetches. Same-origin enforcement rejects outside links. A strict page cap limits attempts. Request timeouts contain slow responses. Failure isolation lets the queue continue after one page fails.
  • Built a repeatable indexing pipeline for the latest crawl. It updates current documents in Meilisearch. It removes stale documents. The browser search returns matching titles. Each result shows a snippet. Each link opens the source fixture page.
  • Secret Mission: Added a robots.txt policy gate before every page request. Disallowed content disappears from the refreshed search index. Network or server failures stop the crawl.

Ready to quiz yourself?