Harden a Next.js SEO Site for Crawlers

Convert a Next.js site to on-demand ISR with request coalescing and rate-limit resilience.

Introduction

30 Second Summary

Search engines send crawlers to discover and index your pages. When you have thousands of programmatic SEO pages and a crawler like Googlebot hits them all at once, your backend can buckle under the sudden flood of requests.

In this project, you will build a resilience layer for a Next.js programmatic SEO site that prevents your backend from being overwhelmed during crawler stampedes.

What You'll Build

A production-hardening layer featuring on-demand ISR, request coalescing, concurrency limiting, 429 retry logic, and a post-deploy cache warming script.

By the end of this project, you'll have:

  • A lib/api-client.ts module with single-flight coalescing, concurrency limiting, and 429/Retry-After backoff.
  • On-demand ISR pages using getStaticProps with fallback: "blocking" and a webhook API route for cache invalidation.
  • A post-deploy cache warming script that fetches your sitemap and visits each URL at a controlled rate to pre-populate the ISR cache.
  • Secret Mission: An observability layer that logs coalescing ratio, 429 hit rate, and p95 backend latency as structured JSON to stdout.

Are there any prerequisites?

This project assumes you have an existing Next.js Pages Router site with dynamic routes and a backend API. You should be familiar with TypeScript and async patterns like Promises and async/await.

Before We Start

Your Next.js programmatic SEO site currently uses getServerSideProps for every page request. When crawlers like Googlebot hit thousands of URLs simultaneously, each request goes straight to your backend. The result is a stampede of concurrent API calls that triggers 429 rate-limit errors.

Before building the resilience layer, you need to install the packages that handle concurrency limiting and XML parsing. You also need the project directories and environment variables that the rest of the project depends on.

In this step, get ready to:
  • Install the concurrency-limiting and XML-parsing packages your project needs.
  • Create the lib/ and scripts/ directories for your API client and cache-warming script.
  • Configure environment variables for your backend URL and revalidation secret.
Install the project dependencies

Your resilience layer relies on two key packages. p-limit caps the number of concurrent outbound requests to your backend. xml2js parses your sitemap XML so the warming script knows which URLs to visit.

  • Install both packages by running this command in your project's root directory:
npm install p-limit xml2js
  • Install the TypeScript type definitions for xml2js as a dev dependency:
npm install --save-dev @types/xml2js

What do these packages do?

  • p-limit creates a concurrency limiter. You pass it a number (like 5), and it guarantees no more than 5 async tasks run at the same time.
  • xml2js converts XML strings into JavaScript objects. Your warming script uses it to extract URLs from your sitemap.
  • @types/xml2js provides TypeScript definitions so your editor gives you autocomplete and type checking for xml2js functions.
  • Confirm the install succeeded by checking your package.json. You should see p-limit and xml2js listed under dependencies, and @types/xml2js under devDependencies.

Seeing permission errors or EACCES?

This usually means your terminal does not have write access to the node_modules folder. Try closing and reopening your terminal, then re-running the install command from your project root.

help me fix npm permission errors when installing packages

Create the project directories

Your API client code will live in lib/, and your post-deploy warming script will live in scripts/. Creating these now gives you a clean structure before you start writing code.

  • Create both directories by running this command from your project root:
mkdir lib scripts
  • Confirm the directories exist by running:
ls
  • You should see both lib and scripts listed alongside your existing project folders like pages and node_modules.
Configure environment variables

Your API client needs to know where your backend lives. Your revalidation webhook needs a secret token to prevent unauthorized cache purges.

  • Create a file called .env.local in your project root by running:
touch .env.local
  • Open .env.local in your editor and add the following environment variables:
BACKEND_API_URL=http://localhost:4000
REVALIDATION_SECRET=your-secret-token-here
SITE_URL=http://localhost:3000

What do these variables control?

  • BACKEND_API_URL tells your API client where to send requests. Replace this with your actual backend URL if it differs.
  • REVALIDATION_SECRET protects your revalidation webhook. Only requests with this token can purge cached pages. Replace your-secret-token-here with a strong, unique value in production.
  • SITE_URL tells the warming script where your Next.js server is running. The script fetches your sitemap from this URL.
  • Save .env.local.

Variables not being read by Next.js?

Make sure the file is named exactly .env.local (with the leading dot) and sits in the project root alongside package.json. If your dev server is already running, restart it so it picks up the new file.

help me troubleshoot Next.js not reading my .env.local file

✔️ Awesome, I've got everything!

Great. Double check you have saved .env.local before moving on.

ⓧ I'd like to double check the full code

Here is what your .env.local should look like:

BACKEND_API_URL=http://localhost:4000
REVALIDATION_SECRET=your-secret-token-here
SITE_URL=http://localhost:3000

Your project is ready to go. Next up, you will convert your dynamic pages from getServerSideProps to on-demand ISR so pages are served from static cache instead of hitting your backend every time.

Convert to On-Demand ISR

Right now, every page on your site uses getServerSideProps. That means every single request from Googlebot, Bingbot, or any user triggers a fresh call to your backend. When crawlers hit thousands of URLs simultaneously, your backend drowns in requests.

The fix is Incremental Static Regeneration with on-demand invalidation. Instead of re-fetching on every request, pages are generated once, cached statically, and only regenerated when you explicitly tell Next.js to purge the cache via a webhook. This turns thousands of backend calls into just one per page.

In this step, get ready to:
  • Convert your dynamic page from getServerSideProps to getStaticProps with fallback: "blocking".
  • Create a revalidation webhook that purges specific pages from the ISR cache on demand.
  • Understand how the ISR caching flow protects your backend from crawler stampedes.
Migrate from getServerSideProps to getStaticProps + getStaticPaths

The key shift is moving from per-request rendering to generate-once-and-cache. With fallback: "blocking" and an empty paths array, Next.js generates each page on its first request and caches it forever until you explicitly invalidate it.

  • Create the pages/[category] directory inside your pages/ folder by running this command:
mkdir -p pages/[category]
  • Create the [slug].tsx file inside pages/[category]/ and paste the following code:
import type { GetStaticPaths, GetStaticProps, InferGetStaticPropsType } from "next";
import { fetchFromBackend } from "../../lib/api-client";

interface PageData {
  title: string;
  content: string;
  category: string;
  slug: string;
}

export const getStaticPaths: GetStaticPaths = async () => {
  return {
    paths: [],
    fallback: "blocking",
  };
};

export const getStaticProps: GetStaticProps<{ page: PageData }> = async (context) => {
  const { category, slug } = context.params as { category: string; slug: string };

  try {
    const page = await fetchFromBackend<PageData>(
      `/api/pages/${category}/${slug}`
    );

    if (!page) {
      return { notFound: true };
    }

    return {
      props: { page },
    };
  } catch (error) {
    console.error(`[ISR] Failed to generate /${category}/${slug}:`, error);
    throw error;
  }
};

export default function CategorySlugPage({
  page,
}: InferGetStaticPropsType<typeof getStaticProps>) {
  return (
    <main>
      <h1>{page.title}</h1>
      <div dangerouslySetInnerHTML={{ __html: page.content }} />
    </main>
  );
}

What does this code do?

getStaticPaths returns an empty paths array with fallback: "blocking". This tells Next.js to generate zero pages at build time. When a request arrives for any path, Next.js runs getStaticProps on the server, waits for it to finish (blocking), and then caches the result for all future requests.

There is no revalidate property in the return value. This means the page stays cached forever until you explicitly call res.revalidate() from an API route. No timer-based background regeneration happens.

The fetchFromBackend import references the API client module you will build in the next step. Your editor may show an error on this import until then.

Why no revalidate timer?

Timer-based ISR (e.g. revalidate: 60) regenerates pages in the background after a fixed interval. This works for content that changes unpredictably. Your programmatic SEO pages change only when the CMS publishes new content. On-demand invalidation via a webhook is more precise and puts zero unnecessary load on your backend.

  • Save pages/[category]/[slug].tsx.
  • Confirm the file exists by running:
ls pages/[category]/

You should see [slug].tsx listed in the output.

Don't see the file?

Make sure you created the [category] directory with the square brackets included in the folder name. The brackets are literal characters in the file system that Next.js uses to identify dynamic route segments.

help me create a dynamic route directory with brackets in Next.js Pages Router

Create the revalidation webhook API route

Your ISR pages need a way to be invalidated when upstream content changes. The revalidation webhook accepts a POST request with a secret token and a path, then tells Next.js to purge that specific page from the cache.

  • Create pages/api/revalidate.ts and paste the following code:
import type { NextApiRequest, NextApiResponse } from "next";

export default async function handler(
  req: NextApiRequest,
  res: NextApiResponse
) {
  if (req.method !== "POST") {
    return res.status(405).json({ message: "Method not allowed" });
  }

  const { secret, path } = req.body;

  if (secret !== process.env.REVALIDATION_SECRET) {
    return res.status(401).json({ message: "Invalid token" });
  }

  if (!path || typeof path !== "string") {
    return res.status(400).json({ message: "Missing path" });
  }

  try {
    await res.revalidate(path);
    return res.json({ revalidated: true, path });
  } catch (err) {
    return res.status(500).json({ message: "Error revalidating" });
  }
}

What does this code do?

  • Rejects anything that isn't a POST request with a 405 status.
  • Validates the secret field against your REVALIDATION_SECRET environment variable to prevent unauthorized cache purges.
  • Calls res.revalidate(path) to purge the ISR cache for the given path. The next visitor to that URL triggers a fresh getStaticProps call.
  • Returns { revalidated: true, path } on success so your CMS webhook can confirm the purge worked.
  • Save pages/api/revalidate.ts.
  • Confirm the file exists by running:
ls pages/api/

You should see revalidate.ts listed alongside any other API routes you already have.

Getting a "directory not found" error?

If pages/api/ does not exist, create it first with mkdir -p pages/api and then create the file. Most Next.js projects already have this directory.

help me set up an API route in my Next.js Pages Router project

✔️ Awesome, I've got everything!

Great. Make sure both files are saved before continuing.

ⓧ I'd like to double check the full code

Here is the complete code for both files:

import type { GetStaticPaths, GetStaticProps, InferGetStaticPropsType } from "next";
import { fetchFromBackend } from "../../lib/api-client";

interface PageData {
  title: string;
  content: string;
  category: string;
  slug: string;
}

export const getStaticPaths: GetStaticPaths = async () => {
  return {
    paths: [],
    fallback: "blocking",
  };
};

export const getStaticProps: GetStaticProps<{ page: PageData }> = async (context) => {
  const { category, slug } = context.params as { category: string; slug: string };

  try {
    const page = await fetchFromBackend<PageData>(
      `/api/pages/${category}/${slug}`
    );

    if (!page) {
      return { notFound: true };
    }

    return {
      props: { page },
    };
  } catch (error) {
    console.error(`[ISR] Failed to generate /${category}/${slug}:`, error);
    throw error;
  }
};

export default function CategorySlugPage({
  page,
}: InferGetStaticPropsType<typeof getStaticProps>) {
  return (
    <main>
      <h1>{page.title}</h1>
      <div dangerouslySetInnerHTML={{ __html: page.content }} />
    </main>
  );
}
import type { NextApiRequest, NextApiResponse } from "next";

export default async function handler(
  req: NextApiRequest,
  res: NextApiResponse
) {
  if (req.method !== "POST") {
    return res.status(405).json({ message: "Method not allowed" });
  }

  const { secret, path } = req.body;

  if (secret !== process.env.REVALIDATION_SECRET) {
    return res.status(401).json({ message: "Invalid token" });
  }

  if (!path || typeof path !== "string") {
    return res.status(400).json({ message: "Missing path" });
  }

  try {
    await res.revalidate(path);
    return res.json({ revalidated: true, path });
  } catch (err) {
    return res.status(500).json({ message: "Error revalidating" });
  }
}
Understand the ISR caching flow

Your two new files work together to create a generate-once, invalidate-on-demand caching pattern. Here is how the flow works end-to-end once the API client is in place.

  • First request to /seo/best-widgets: Next.js sees no cached page. It calls getStaticProps, which fetches data from your backend. The visitor waits (blocking). The generated HTML is cached.
  • All subsequent requests to /seo/best-widgets: Next.js serves the cached HTML instantly. Zero backend calls. Googlebot gets a fast response.
  • Content changes upstream: Your CMS sends a POST to /api/revalidate with the secret and path /seo/best-widgets. Next.js purges the cached page.
  • Next request after purge: The page is regenerated fresh from the backend, cached again, and served to all future visitors.

Why does this solve the stampede problem?

With getServerSideProps, 1000 simultaneous crawler requests means 1000 backend calls. With this ISR setup, only the very first request hits the backend. The other 999 either wait for that single generation to finish (blocking) or get the cached result immediately afterward.

The API client you build in the next step adds an extra safety layer. Even during cold-cache stampedes where many different pages are generated at once, it caps the number of outbound backend requests.

Testing will come alive in the next step

The fetchFromBackend import in your page file does not resolve yet. Once you build lib/api-client.ts in the next step, the full ISR flow will work end-to-end and you can test it with next build followed by next start.

Your ISR foundation is in place. Next up, you will build the single-flight and concurrency-limited API client that ensures your backend never gets more than 5 simultaneous requests, even when hundreds of pages are being generated at once.

Build the Single-Flight API Client

Your ISR page is set up with fallback: "blocking", which means the first visitor to any uncached URL triggers a backend fetch.

Now imagine Googlebot discovers 500 of your pages simultaneously. Each cold-cache hit fires off its own backend request, creating a thundering herd that overwhelms your API with 429 rate-limit errors. In this step, you will build a fetch layer that deduplicates concurrent requests, caps outbound concurrency, and gracefully retries when rate-limited.

In this step, get ready to:
  • Implement a SingleFlight class that coalesces duplicate in-flight requests into one shared Promise.
  • Build a retry-aware fetch utility with concurrency limiting and 429 backoff.
  • Export a unified fetchFromBackend function that ties single-flight, concurrency, and retry together.
Implement the SingleFlight class

The single-flight pattern ensures that if 50 ISR pages all need the same backend URL at the same instant, only one actual HTTP request is made. Every other caller receives the same Promise result.

Why build it custom instead of using a library?

The single-flight pattern is about 15 lines of TypeScript. Node.js is single-threaded, so the Map.has() and Map.set() sequence is atomic with no race condition.

External libraries add dependencies for trivial logic. A custom implementation has zero supply-chain risk and is easier to reason about.

  • Create a new file called lib/single-flight.ts and paste the following code:
/**
 * SingleFlight ensures only one in-flight request per key.
 * Concurrent callers for the same key share the same Promise.
 */
export class SingleFlight {
  private inflight = new Map<string, Promise<unknown>>();

  async do<T>(key: string, fn: () => Promise<T>): Promise<T> {
    const existing = this.inflight.get(key);
    if (existing) {
      return existing as Promise<T>;
    }

    const promise = fn().finally(() => {
      this.inflight.delete(key);
    });

    this.inflight.set(key, promise);
    return promise;
  }
}

What does this code do?

  • inflight is a Map that stores in-progress Promises keyed by a string (the request URL).
  • The do() method checks if a Promise already exists for the given key. If it does, every caller shares that same Promise instead of making a new request.
  • The .finally() callback removes the key from the map once the request resolves or rejects, so future calls start fresh.
  • Save lib/single-flight.ts.
  • Confirm the file was created by running:
ls lib/

You should see single-flight.ts listed in the output.

Don't see single-flight.ts?

  • Make sure you created the file inside the lib/ directory, not in the project root.
  • Check that the filename is exactly single-flight.ts (with the hyphen).

Why isn't my single-flight.ts file showing up in the lib directory?

✔️ Awesome, I've got everything!

Great. Make sure you've saved your file before moving on.

ⓧ I'd like to double check the full code

/**
 * SingleFlight ensures only one in-flight request per key.
 * Concurrent callers for the same key share the same Promise.
 */
export class SingleFlight {
  private inflight = new Map<string, Promise<unknown>>();

  async do<T>(key: string, fn: () => Promise<T>): Promise<T> {
    const existing = this.inflight.get(key);
    if (existing) {
      return existing as Promise<T>;
    }

    const promise = fn().finally(() => {
      this.inflight.delete(key);
    });

    this.inflight.set(key, promise);
    return promise;
  }
}
Build the retry-aware fetch utilities

The API client needs three capabilities: capping how many requests hit the backend at once, understanding the Retry-After header when a 429 arrives, and falling back to exponential backoff when no header is present.

  • Create a new file called lib/api-client.ts and paste the following setup code:
import pLimit from "p-limit";
import { SingleFlight } from "./single-flight";

const BACKEND_API_URL = process.env.BACKEND_API_URL || "http://localhost:4000";
const MAX_CONCURRENCY = 5;
const MAX_RETRIES = 3;

// Create a concurrency gate that allows at most 5 simultaneous outbound requests
const concurrencyLimit = pLimit(MAX_CONCURRENCY);
const singleFlight = new SingleFlight();

// Helper to pause execution for a given number of milliseconds
function sleep(ms: number): Promise<void> {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

// Parse the Retry-After header (supports both seconds and HTTP-date formats)
function parseRetryAfter(header: string | null): number | null {
  if (!header) return null;

  const seconds = Number(header);
  if (!Number.isNaN(seconds)) {
    return seconds * 1000;
  }

  const date = Date.parse(header);
  if (!Number.isNaN(date)) {
    return Math.max(0, date - Date.now());
  }

  return null;
}

What does this code do?

  • pLimit(MAX_CONCURRENCY) creates a gate that queues any request beyond 5 simultaneous calls. No matter how many ISR pages regenerate at once, only 5 outbound requests reach the backend.
  • parseRetryAfter() handles two formats the backend might send: a plain number of seconds (e.g. Retry-After: 2) or a full HTTP date string.
  • Both return values are converted to milliseconds so the sleep() helper can pause the right amount.

You won't see a change yet. The next chunk adds the retry loop that uses these utilities.

  • In lib/api-client.ts, add the fetchWithRetry function below parseRetryAfter:
async function fetchWithRetry(url: string): Promise<unknown> {
  let lastError: Error | null = null;

  for (let attempt = 0; attempt < MAX_RETRIES; attempt++) {
    const response = await fetch(url);

    if (response.ok) {
      return response.json();
    }

    if (response.status === 429) {
      // Use Retry-After header if available, otherwise exponential backoff with jitter
      const retryAfterMs =
        parseRetryAfter(response.headers.get("Retry-After")) ??
        Math.pow(2, attempt) * 1000 + Math.random() * 500;

      console.warn(
        `[api-client] 429 from backend, retrying in ${Math.round(retryAfterMs)}ms (attempt ${attempt + 1}/${MAX_RETRIES})`
      );

      await sleep(retryAfterMs);
      lastError = new Error(`HTTP 429 from ${url}`);
      continue;
    }

    // Non-429 errors are not retryable
    throw new Error(`HTTP ${response.status} from ${url}`);
  }

  throw lastError ?? new Error(`Max retries exceeded for ${url}`);
}

What does this code do?

  • The loop retries up to MAX_RETRIES (3) times, but only for 429 responses. Any other HTTP error throws immediately.
  • If the backend sends a Retry-After header, the function respects it. Otherwise, it falls back to exponential backoff (1s, 2s, 4s) plus random jitter (0-500ms).
  • The jitter prevents multiple pages that failed at the same instant from all retrying in sync, which would cause another stampede.

You won't see a change yet. The next chunk exports the public function that wires everything together.

Seeing a red squiggly under the import?

  • If your editor shows an error on import pLimit from "p-limit", make sure you installed the package earlier with npm install p-limit.
  • If the error is on import { SingleFlight } from "./single-flight", confirm that lib/single-flight.ts exists and the class is exported.

Help me debug an import error in my api-client.ts file.

Wire up the unified fetchFromBackend function

The final piece is the public function that your ISR page already imports. It combines single-flight deduplication and the concurrency limiter into one call.

  • At the bottom of lib/api-client.ts, add the exported fetchFromBackend function:
export async function fetchFromBackend<T>(path: string): Promise<T> {
  const url = `${BACKEND_API_URL}${path}`;

  // Single-flight ensures one in-flight request per URL
  // Concurrency limit caps total outbound requests to the backend
  return singleFlight.do(url, () =>
    concurrencyLimit(() => fetchWithRetry(url))
  ) as Promise<T>;
}

What does this code do?

  • singleFlight.do(url, ...) checks if there is already a request in flight for this exact URL. If so, the caller shares that existing Promise instead of creating a new one.
  • concurrencyLimit(() => fetchWithRetry(url)) queues the request if 5 are already in progress. The request only proceeds when a slot opens up.
  • The nesting order matters: single-flight wraps the limiter so that duplicate requests never even enter the concurrency queue.
  • Save lib/api-client.ts.
  • Start (or restart) your dev server by running:
npm run dev

Your dev server should start without any module-not-found errors. The pages/[category]/[slug].tsx file imports fetchFromBackend from ../../lib/api-client, and that module now exists.

Seeing a module not found error?

  • If the error mentions ../../lib/api-client, confirm your file is saved as lib/api-client.ts (not lib/apiClient.ts or lib/api_client.ts).
  • If the error mentions ./single-flight, verify that lib/single-flight.ts exists and exports the SingleFlight class.
  • If the error mentions p-limit, re-run npm install p-limit from your project root.

Help me fix a module not found error when starting my Next.js dev server.

✔️ Awesome, I've got everything!

Great. Double check you've saved both files before moving on.

ⓧ I'd like to double check the full code

import pLimit from "p-limit";
import { SingleFlight } from "./single-flight";

const BACKEND_API_URL = process.env.BACKEND_API_URL || "http://localhost:4000";
const MAX_CONCURRENCY = 5;
const MAX_RETRIES = 3;

const concurrencyLimit = pLimit(MAX_CONCURRENCY);
const singleFlight = new SingleFlight();

function sleep(ms: number): Promise<void> {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

function parseRetryAfter(header: string | null): number | null {
  if (!header) return null;

  const seconds = Number(header);
  if (!Number.isNaN(seconds)) {
    return seconds * 1000;
  }

  const date = Date.parse(header);
  if (!Number.isNaN(date)) {
    return Math.max(0, date - Date.now());
  }

  return null;
}

async function fetchWithRetry(url: string): Promise<unknown> {
  let lastError: Error | null = null;

  for (let attempt = 0; attempt < MAX_RETRIES; attempt++) {
    const response = await fetch(url);

    if (response.ok) {
      return response.json();
    }

    if (response.status === 429) {
      const retryAfterMs =
        parseRetryAfter(response.headers.get("Retry-After")) ??
        Math.pow(2, attempt) * 1000 + Math.random() * 500;

      console.warn(
        `[api-client] 429 from backend, retrying in ${Math.round(retryAfterMs)}ms (attempt ${attempt + 1}/${MAX_RETRIES})`
      );

      await sleep(retryAfterMs);
      lastError = new Error(`HTTP 429 from ${url}`);
      continue;
    }

    throw new Error(`HTTP ${response.status} from ${url}`);
  }

  throw lastError ?? new Error(`Max retries exceeded for ${url}`);
}

export async function fetchFromBackend<T>(path: string): Promise<T> {
  const url = `${BACKEND_API_URL}${path}`;

  return singleFlight.do(url, () =>
    concurrencyLimit(() => fetchWithRetry(url))
  ) as Promise<T>;
}

Your API client is ready. Every backend request now passes through single-flight deduplication, a 5-slot concurrency gate, and automatic retry with backoff. Next up, you will build a post-deploy script that proactively warms your ISR cache so crawlers never hit a cold page.

Add 429 Retry with Retry-After Backoff

Your API client can now coalesce duplicate requests and cap concurrency at 5. But what happens when the backend is under heavy load and starts returning HTTP 429 (Too Many Requests) responses?

Without retry logic, a single 429 response would crash the page generation entirely. In this step, you will add intelligent retry behavior that respects the backend's Retry-After header and falls back to exponential backoff with jitter when no header is present.

In this step, get ready to:
  • Understand how the Retry-After header is parsed to determine wait durations.
  • Implement exponential backoff with random jitter as a fallback retry strategy.
  • Wire retry logic into the concurrency-limited and single-flight-wrapped fetch path.
Parse the Retry-After header

When a backend returns HTTP 429, it often includes a Retry-After header that tells you exactly how long to wait before trying again. This header can be a number of seconds (like 5) or a full HTTP-date string.

  • In lib/api-client.ts, find the parseRetryAfter function. It should look like this:
function parseRetryAfter(header: string | null): number | null {
  if (!header) return null;

  const seconds = Number(header);
  if (!Number.isNaN(seconds)) {
    return seconds * 1000;
  }

  const date = Date.parse(header);
  if (!Number.isNaN(date)) {
    return Math.max(0, date - Date.now());
  }

  return null;
}

What does this function do?

This function handles two formats the Retry-After header can use.

  • If the header is a plain number (e.g. "5"), it converts seconds to milliseconds.
  • If the header is an HTTP-date (e.g. "Wed, 21 Oct 2026 07:28:00 GMT"), it calculates how many milliseconds remain until that timestamp.
  • If the header is missing or unparseable, it returns null so the caller knows to use a fallback strategy.
Implement exponential backoff with jitter

When the backend does not include a Retry-After header, you need a fallback. A naive approach (wait 1 second every time) causes all retrying pages to slam the backend simultaneously. Exponential backoff with jitter solves this by spreading retries over time.

Why add jitter?

Without jitter, all pages that got a 429 at the same moment would retry at exactly the same future moment. That creates another spike. Random jitter offsets each retry by a small random amount, distributing the load.

The formula is: 2^attempt * 1000ms + random(0-500ms). Attempt 0 waits about 1-1.5s. Attempt 1 waits about 2-2.5s. Attempt 2 waits about 4-4.5s.

Integrate retry logic into the API client

Now both pieces come together inside fetchWithRetry. The function loops up to MAX_RETRIES times, uses parseRetryAfter when a 429 arrives, and falls back to the exponential backoff formula.

  • In lib/api-client.ts, find your fetchWithRetry function. Confirm it matches this code:
async function fetchWithRetry(url: string): Promise<unknown> {
  let lastError: Error | null = null;

  for (let attempt = 0; attempt < MAX_RETRIES; attempt++) {
    const response = await fetch(url);

    if (response.ok) {
      return response.json();
    }

    if (response.status === 429) {
      const retryAfterMs =
        parseRetryAfter(response.headers.get("Retry-After")) ??
        Math.pow(2, attempt) * 1000 + Math.random() * 500;

      console.warn(
        `[api-client] 429 from backend, retrying in ${Math.round(retryAfterMs)}ms (attempt ${attempt + 1}/${MAX_RETRIES})`
      );

      await sleep(retryAfterMs);
      lastError = new Error(`HTTP 429 from ${url}`);
      continue;
    }

    throw new Error(`HTTP ${response.status} from ${url}`);
  }

  throw lastError ?? new Error(`Max retries exceeded for ${url}`);
}

What does this code do?

  • The for loop runs up to MAX_RETRIES (3) times. If the response is OK on any attempt, it returns immediately.
  • On a 429, it first tries parseRetryAfter to get the server's preferred wait time. The ?? operator means if that returns null, it uses the exponential backoff formula instead.
  • The console.warn logs every 429 with the wait duration and attempt number. This is critical for debugging production issues.
  • Any non-429, non-OK status (like 500 or 403) throws immediately without retrying. Only rate-limit errors are worth retrying.
  • After exhausting all retries, it throws the last error. This surfaces to getStaticProps as a build error for that page.

The key insight is where this function sits in the call chain. Look at fetchFromBackend at the bottom of the file:

export async function fetchFromBackend<T>(path: string): Promise<T> {
  const url = `${BACKEND_API_URL}${path}`;

  return singleFlight.do(url, () =>
    concurrencyLimit(() => fetchWithRetry(url))
  ) as Promise<T>;
}

How does the retry fit into the pipeline?

The call chain flows: single-flight dedup > concurrency limiter > retry logic > actual fetch. This means retries do NOT consume additional concurrency slots. A retrying request holds its one slot until it succeeds or gives up.

If 50 ISR pages request the same URL simultaneously, single-flight collapses them into 1 actual request. That request gets a concurrency slot. If it gets a 429, it retries inside that same slot. The other 49 waiters just share the final result.

  • Save lib/api-client.ts.

Seeing a TypeScript error?

  • Make sure sleep and parseRetryAfter are defined ABOVE fetchWithRetry in the file. TypeScript needs them declared before they are called.
  • Check that MAX_RETRIES is declared as a const at the top of the file, not inside a function.
  • help me debug a TypeScript error in my api-client.ts retry logic

✔️ Awesome, I've got everything!

Great. Double check you have saved lib/api-client.ts.

ⓧ I'd like to double check the full code

import pLimit from "p-limit";
import { SingleFlight } from "./single-flight";

const BACKEND_API_URL = process.env.BACKEND_API_URL || "http://localhost:4000";
const MAX_CONCURRENCY = 5;
const MAX_RETRIES = 3;

const concurrencyLimit = pLimit(MAX_CONCURRENCY);
const singleFlight = new SingleFlight();

function sleep(ms: number): Promise<void> {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

function parseRetryAfter(header: string | null): number | null {
  if (!header) return null;

  const seconds = Number(header);
  if (!Number.isNaN(seconds)) {
    return seconds * 1000;
  }

  const date = Date.parse(header);
  if (!Number.isNaN(date)) {
    return Math.max(0, date - Date.now());
  }

  return null;
}

async function fetchWithRetry(url: string): Promise<unknown> {
  let lastError: Error | null = null;

  for (let attempt = 0; attempt < MAX_RETRIES; attempt++) {
    const response = await fetch(url);

    if (response.ok) {
      return response.json();
    }

    if (response.status === 429) {
      const retryAfterMs =
        parseRetryAfter(response.headers.get("Retry-After")) ??
        Math.pow(2, attempt) * 1000 + Math.random() * 500;

      console.warn(
        `[api-client] 429 from backend, retrying in ${Math.round(retryAfterMs)}ms (attempt ${attempt + 1}/${MAX_RETRIES})`
      );

      await sleep(retryAfterMs);
      lastError = new Error(`HTTP 429 from ${url}`);
      continue;
    }

    throw new Error(`HTTP ${response.status} from ${url}`);
  }

  throw lastError ?? new Error(`Max retries exceeded for ${url}`);
}

export async function fetchFromBackend<T>(path: string): Promise<T> {
  const url = `${BACKEND_API_URL}${path}`;

  return singleFlight.do(url, () =>
    concurrencyLimit(() => fetchWithRetry(url))
  ) as Promise<T>;
}

Your API client is now fully resilient. It deduplicates concurrent requests, limits outbound concurrency to 5, and gracefully retries when the backend pushes back with 429s. Next up, you will build a cache warming script that pre-populates your ISR pages after every deploy.

Build the Post-Deploy Cache Warming Script

Your API client now protects the backend from thundering herds during organic traffic. But there is still a cold-cache problem at deploy time.

Every deploy wipes the ISR cache, meaning the next crawler sweep finds every page uncached and triggers simultaneous regeneration. In this step, you will build a script that proactively visits every URL in your sitemap right after deploy, populating the cache before crawlers arrive.

In this step, get ready to:
  • Fetch and parse your sitemap XML to extract all page URLs.
  • Visit each URL at a safe concurrency rate using p-limit.
  • Add progress reporting and error handling for deploy-time visibility.
Fetch and parse the sitemap

Your Next.js site already generates a sitemap.xml file listing every page URL. The warming script needs to fetch that XML, parse it, and extract each <loc> entry into a flat array of URLs to visit.

  • Create the file scripts/warm-cache.ts and add the imports, constants, and fetchSitemap function by pasting this code:
import pLimit from "p-limit";
import { parseStringPromise } from "xml2js";

const SITE_URL = process.env.SITE_URL || "http://localhost:3000";
const SITEMAP_URL = `${SITE_URL}/sitemap.xml`;
const WARM_CONCURRENCY = 3;

const limit = pLimit(WARM_CONCURRENCY);

interface SitemapUrl {
  loc: string[];
}

async function fetchSitemap(): Promise<string[]> {
  const response = await fetch(SITEMAP_URL);
  if (!response.ok) {
    throw new Error(`Failed to fetch sitemap: HTTP ${response.status}`);
  }

  const xml = await response.text();
  const parsed = await parseStringPromise(xml);

  const urls: string[] = [];
  const urlEntries: SitemapUrl[] = parsed.urlset?.url || [];

  for (const entry of urlEntries) {
    if (entry.loc && entry.loc[0]) {
      urls.push(entry.loc[0]);
    }
  }

  return urls;
}

What does this code do?

The script reads SITE_URL from your environment variables and constructs the sitemap URL. WARM_CONCURRENCY is set to 3 to avoid overwhelming your own server during warming.

The fetchSitemap function fetches the raw XML, passes it to parseStringPromise from xml2js, then loops through the parsed urlset.url array to extract each <loc> value into a flat string array.

Why pLimit(3) for warming?

Each warm request hits your Next.js server, which in turn calls the backend through your API client (capped at 5 concurrent requests). Using 3 concurrent warm requests keeps the pipeline comfortably below the backend cap. The two layers of concurrency control work together.

  • Save scripts/warm-cache.ts.
Visit URLs at a safe rate using p-limit

With the URL list ready, you need a function that visits a single URL and reports whether it succeeded. This keeps the warming logic clean and testable.

  • In scripts/warm-cache.ts, add the warmUrl function below the fetchSitemap function:
async function warmUrl(url: string): Promise<boolean> {
  try {
    const response = await fetch(url);
    return response.ok;
  } catch {
    return false;
  }
}

What does this code do?

The warmUrl function sends a GET request to a single page URL. If the response is successful, it returns true. If the request fails or returns an error status, it returns false without crashing the entire script. This lets the warming script skip broken pages and continue.

  • Save scripts/warm-cache.ts.
Add progress reporting and error handling

A warming script that processes thousands of URLs in silence is hard to debug during deploys. You need progress logs so you can see whether the script is stuck, how many URLs succeeded, and how many failed.

  • Below the warmUrl function, add the main function and its invocation by pasting this code:
async function main() {
  console.log(`[warm-cache] Fetching sitemap from ${SITEMAP_URL}`);
  const urls = await fetchSitemap();
  console.log(`[warm-cache] Found ${urls.length} URLs to warm`);

  let warmed = 0;
  let failed = 0;

  const tasks = urls.map((url) =>
    limit(async () => {
      const success = await warmUrl(url);
      if (success) {
        warmed++;
      } else {
        failed++;
        console.warn(`[warm-cache] Failed: ${url}`);
      }

      if ((warmed + failed) % 50 === 0) {
        console.log(
          `[warm-cache] Progress: ${warmed + failed}/${urls.length} (${warmed} ok, ${failed} failed)`
        );
      }
    })
  );

  await Promise.all(tasks);

  console.log(
    `[warm-cache] Complete: ${warmed} warmed, ${failed} failed out of ${urls.length} total`
  );
}

main().catch((err) => {
  console.error("[warm-cache] Fatal error:", err);
  process.exit(1);
});

What does this code do?

The main function orchestrates the entire warming flow. It fetches the URL list from the sitemap, then maps each URL through the limit wrapper so that at most 3 requests run concurrently.

Every 50 URLs, the script logs a progress line so you can monitor deploy pipelines. Failed URLs are logged individually but do not stop the script. The final summary gives you a quick pass/fail count.

The main().catch() at the bottom ensures any unhandled error (like a missing sitemap) logs a clear message and exits with a non-zero code, which fails your CI pipeline rather than silently succeeding.

  • Save scripts/warm-cache.ts.
  • To test the script, start your Next.js dev server in one terminal, then run the warming script in a second terminal. If you have a public/sitemap.xml file, you should see progress logs. If you do not have a sitemap yet, you will see a "Failed to fetch sitemap" error, confirming the error handling works.

Seeing a module error when running the script?

If you see an error about ESM imports, make sure xml2js is installed in your project. You can verify by checking your package.json dependencies.

Also confirm that p-limit is listed in your dependencies. Both packages must be installed for the script to run.

Why can't I run my warm-cache script?

✔️ Awesome, I've got everything!

Great. Double check you have saved scripts/warm-cache.ts.

ⓧ I'd like to double check the full code

Here is the complete scripts/warm-cache.ts file:

import pLimit from "p-limit";
import { parseStringPromise } from "xml2js";

const SITE_URL = process.env.SITE_URL || "http://localhost:3000";
const SITEMAP_URL = `${SITE_URL}/sitemap.xml`;
const WARM_CONCURRENCY = 3;

const limit = pLimit(WARM_CONCURRENCY);

interface SitemapUrl {
  loc: string[];
}

async function fetchSitemap(): Promise<string[]> {
  const response = await fetch(SITEMAP_URL);
  if (!response.ok) {
    throw new Error(`Failed to fetch sitemap: HTTP ${response.status}`);
  }

  const xml = await response.text();
  const parsed = await parseStringPromise(xml);

  const urls: string[] = [];
  const urlEntries: SitemapUrl[] = parsed.urlset?.url || [];

  for (const entry of urlEntries) {
    if (entry.loc && entry.loc[0]) {
      urls.push(entry.loc[0]);
    }
  }

  return urls;
}

async function warmUrl(url: string): Promise<boolean> {
  try {
    const response = await fetch(url);
    return response.ok;
  } catch {
    return false;
  }
}

async function main() {
  console.log(`[warm-cache] Fetching sitemap from ${SITEMAP_URL}`);
  const urls = await fetchSitemap();
  console.log(`[warm-cache] Found ${urls.length} URLs to warm`);

  let warmed = 0;
  let failed = 0;

  const tasks = urls.map((url) =>
    limit(async () => {
      const success = await warmUrl(url);
      if (success) {
        warmed++;
      } else {
        failed++;
        console.warn(`[warm-cache] Failed: ${url}`);
      }

      if ((warmed + failed) % 50 === 0) {
        console.log(
          `[warm-cache] Progress: ${warmed + failed}/${urls.length} (${warmed} ok, ${failed} failed)`
        );
      }
    })
  );

  await Promise.all(tasks);

  console.log(
    `[warm-cache] Complete: ${warmed} warmed, ${failed} failed out of ${urls.length} total`
  );
}

main().catch((err) => {
  console.error("[warm-cache] Fatal error:", err);
  process.exit(1);
});

Your warming script is ready. After every deploy, this script crawls your sitemap and pre-populates the ISR cache at a safe rate, so crawlers always find warm pages. Next up, tackle the Secret Mission to add structured observability logging across the entire resilience layer.

Secret mission

Add Observability

Your resilience layer is working, but you have no visibility into how well it performs. How often is the single-flight layer actually deduplicating requests? How many 429s are you absorbing? Build a structured logging layer that answers these questions with hard numbers.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs.

Resources you used:

  • Local files in lib/ directory (single-flight.ts, api-client.ts, metrics.ts).
  • Local file in scripts/ directory (warm-cache.ts).
  • ISR page at pages/[category]/[slug].tsx and API route at pages/api/revalidate.ts.
  • Environment variables in .env.local.
  • npm packages: p-limit and xml2js.

Keep everything running

No action needed. Choose this if you are still actively building on your hardened ISR layer or planning to deploy it to production.

  • Your files in lib/, scripts/, and pages/ stay in place and are ready to use.
  • The p-limit and xml2js packages remain installed in your project.
  • Your .env.local environment variables remain configured.

Pause - I'll come back to this later

Since this project is entirely local files with no running processes, there is nothing to pause. Your files will be here when you come back.

  • If your Next.js dev server is running, stop it with Ctrl+C in the terminal.
  • All your files and packages remain in place for when you return.

Delete - I don't want to use this again

Remove all project files and packages added during this project to return your Next.js site to its original state.

  • Stop the dev server if it is running by pressing Ctrl+C in the terminal.
  • Remove the lib/ files by running:
rm lib/single-flight.ts lib/api-client.ts lib/metrics.ts
  • Remove the warming script by running:
rm scripts/warm-cache.ts
  • Remove the revalidation API route by running:
rm pages/api/revalidate.ts
  • Remove the ISR page by running:
rm pages/[category]/[slug].tsx
  • Remove the environment variables file by running:
rm .env.local
  • Uninstall the npm packages by running:
npm uninstall p-limit xml2js
  • If the lib/ and scripts/ directories are now empty, remove them by running:
rmdir lib scripts

Nice Work!

Nice Work!

Nice work! You have hardened a Next.js programmatic SEO site against crawler stampedes. Your backend stays healthy even when Googlebot and Bingbot hit thousands of uncached URLs simultaneously.

You learned how to:

  • Convert dynamic pages from getServerSideProps to on-demand ISR with fallback: "blocking" and a webhook-based revalidation route that purges pages only when content changes upstream.
  • Build a resilient API client combining a SingleFlight coalescer to deduplicate concurrent requests, a pLimit(5) concurrency gate to cap outbound traffic, and Retry-After header parsing with exponential backoff for 429 responses.
  • Create a post-deploy cache warming script that fetches your sitemap, extracts all URLs, and visits each at a controlled concurrency to pre-populate the ISR cache before crawlers arrive.
  • Added structured observability logging that tracks coalescing ratio, 429 hit rate, and p95 backend latency, flushing metrics as JSON to stdout for ingestion by any log aggregation system.

Ready to quiz yourself?