Add Observability to a Next.js App

Build a structured logging and Prometheus metrics layer for a Next.js API.

Introduction

30 Second Summary

When your app breaks at 2am, console.log statements scattered through your codebase won't tell you which endpoint is slow, how many users are hitting rate limits, or whether auth failures are spiking. Production systems need structured, searchable signals that answer these questions in seconds.

In this project, you will build a complete observability layer for a Next.js API application. You'll create structured logging, custom Prometheus metrics, and a live Grafana dashboard that visualizes your app's health in real time.

What You'll Build

A production observability layer for a Next.js API application, combining structured JSON logging, Prometheus metrics collection, and a Grafana dashboard into a reusable instrumentation library.

By the end of this project, you'll have:

  • A Winston-based structured logger that outputs searchable JSON with request-scoped context across all API routes.
  • A Prometheus-compatible /api/metrics endpoint exposing custom counters, histograms, and gauges for rate-limit hits, auth failures, recommendations, and request duration.
  • A running Docker Compose stack with Prometheus and Grafana scraping your Next.js app and rendering a live observability dashboard.
  • Secret Mission: Add a slow-endpoint detection alert rule in Prometheus that fires when p95 latency exceeds a threshold, and surface it in Grafana.

Are there any prerequisites?

This project assumes you're comfortable with Next.js App Router, TypeScript, API routes, and Docker Compose. You should have Node.js, Docker Desktop, and VS Code or Cursor installed on your machine.

Set Up the Observability Libraries

Right now, if something goes wrong in your RunLayer app, you're flying blind. console.log statements scattered across your routes give you zero structure, no timestamps, no request context, and no way to search or filter.

Production apps need two pillars of observability: structured logs (so you can trace what happened during a request) and metrics (so you can measure how your system is performing over time). In this step, you'll install Winston for structured logging and prom-client for Prometheus-compatible metrics, then build the foundational libraries that all your routes will use.

In this step, get ready to:
  • Install Winston and prom-client into your Next.js project.
  • Create a structured logger with JSON output and request-scoped child loggers.
  • Define a Prometheus metrics registry with six custom metrics for your app domain.
Install Winston and prom-client

Winston is a logging library that outputs structured JSON in production, making your logs searchable by any log aggregator. prom-client is the official Prometheus client for Node.js, letting you define counters, histograms, and gauges that Prometheus can scrape.

  • In your terminal, make sure you're in your RunLayer project directory.
  • Install both libraries by running this command:
npm install winston@3.19.0 prom-client@15.1.3
  • Confirm the installation succeeded by running:
npm list winston prom-client

You should see winston@3.19.0 and prom-client@15.1.3 listed in the output with no errors.

Seeing npm errors?

Make sure your terminal is in the RunLayer project folder (the one containing package.json). If you see peer dependency warnings, those are informational and safe to ignore.

help me fix npm install errors for winston and prom-client

Create the structured logger

Your logger needs to do two things: output structured JSON in production (so log aggregators can parse it) and output colorized, human-readable text in development (so you can scan it quickly). It also needs to create child loggers that automatically attach request context to every log line.

  • Create a new file called lib/logger.ts in your project.
  • If the lib/ folder doesn't exist yet, create it first.
  • Paste the following code into lib/logger.ts:
import winston from 'winston';

const { combine, timestamp, json, colorize, printf } = winston.format;

// Development format: colorized, human-readable output
const devFormat = combine(
  colorize(),
  timestamp({ format: 'HH:mm:ss' }),
  printf(({ timestamp, level, message, ...meta }) => {
    const metaStr = Object.keys(meta).length ? ` ${JSON.stringify(meta)}` : '';
    return `${timestamp} ${level}: ${message}${metaStr}`;
  })
);

// Production format: structured JSON for log aggregators
const prodFormat = combine(
  timestamp(),
  json()
);

export const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || 'info',
  format: process.env.NODE_ENV === 'production' ? prodFormat : devFormat,
  defaultMeta: { service: 'runlayer' },
  transports: [
    new winston.transports.Console(),
  ],
});

// Creates a child logger with request context bound to every log line
export function createRequestLogger(requestId: string, route: string, method: string) {
  return logger.child({ requestId, route, method });
}

What does this code do?

  • devFormat produces colorized, timestamped output you can read at a glance during development.
  • prodFormat produces structured JSON lines that log aggregators like CloudWatch or Datadog can parse and search.
  • createLogger picks the format based on NODE_ENV and attaches a service name (runlayer) to every log line.
  • createRequestLogger returns a child logger that automatically includes the requestId, route, and method in every log line it produces. This lets you trace all logs from a single request.
  • Save lib/logger.ts.

You won't see a visible change yet. The logger is a utility that other files will import in the next step.

Seeing a red underline on the import?

If your editor shows a TypeScript error on import winston from 'winston', make sure npm install completed successfully in the previous substep. The winston package includes its own type definitions, so no separate @types package is needed.

help me fix TypeScript import errors for winston in my Next.js project

Create the metrics registry

Prometheus works by scraping an HTTP endpoint that exposes metrics in a specific text format. You need a registry that collects all your metrics and serves them on demand.

You'll define six custom metrics tailored to RunLayer's domain: request duration, request counts, rate-limit hits, auth failures, recommendation usage, and active in-flight requests.

  • Create a new file called lib/metrics.ts in your lib/ folder.
  • Start by pasting the registry setup and core HTTP metrics:
import client from 'prom-client';

// Create an isolated registry (not the global default)
export const registry = new client.Registry();

// Collect Node.js runtime metrics with a runlayer_ prefix
client.collectDefaultMetrics({ register: registry, prefix: 'runlayer_' });

// Histogram: tracks how long each request takes
export const httpRequestDuration = new client.Histogram({
  name: 'runlayer_http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['route', 'method'] as const,
  buckets: [0.01, 0.05, 0.1, 0.3, 0.5, 1, 2, 5],
  registers: [registry],
});

// Counter: total requests processed, broken down by route/method/status
export const httpRequestsTotal = new client.Counter({
  name: 'runlayer_http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['route', 'method', 'status_code'] as const,
  registers: [registry],
});

What does this code do?

A custom Registry isolates your metrics from any other library that might register metrics globally. This prevents name collisions.

collectDefaultMetrics automatically tracks Node.js runtime stats (memory, CPU, event loop lag) without you writing any code. The runlayer_ prefix namespaces them so they're easy to filter in Prometheus.

The Histogram records request duration with specific bucket boundaries (10ms to 5s). The Counter increments for every request and lets you slice data by route, method, and status code.

  • Below the httpRequestsTotal counter, add the four domain-specific metrics:
// Counter: tracks when rate limiting rejects a request
export const rateLimitHitsTotal = new client.Counter({
  name: 'runlayer_rate_limit_hits_total',
  help: 'Total number of rate limit hits',
  labelNames: ['route'] as const,
  registers: [registry],
});

// Counter: tracks authentication failures with a reason label
export const authFailuresTotal = new client.Counter({
  name: 'runlayer_auth_failures_total',
  help: 'Total number of authentication failures',
  labelNames: ['route', 'reason'] as const,
  registers: [registry],
});

// Counter: tracks how many recommendations are served
export const recommendationsTotal = new client.Counter({
  name: 'runlayer_recommendations_total',
  help: 'Total number of recommendations served',
  registers: [registry],
});

// Gauge: tracks how many requests are currently being processed
export const activeRequests = new client.Gauge({
  name: 'runlayer_active_requests',
  help: 'Number of currently active requests',
  registers: [registry],
});

Counters vs Gauges vs Histograms

A Counter only goes up. It counts events (requests, errors, rate-limit hits). You use rate() in PromQL to see how fast it's increasing.

A Gauge goes up and down. It measures current state (active connections, queue depth, memory usage).

A Histogram records observations into buckets. It's ideal for latency because you can calculate percentiles (p50, p95, p99) from the bucket boundaries.

  • Save lib/metrics.ts.
  • Confirm the registry loads without errors by running this command:
npx ts-node --eval "import('./lib/metrics')"

If the module loads successfully, you'll see no error output. The command will complete and return you to your prompt.

Seeing a module resolution error?

If you get an error about ESM or module resolution, try running with the --esm flag: npx ts-node --esm --eval "import('./lib/metrics')". Make sure both lib/logger.ts and lib/metrics.ts are saved, since metrics.ts doesn't import the logger but both must be valid TypeScript.

help me fix ts-node import errors when testing my metrics module

✔️ Awesome, I've got everything!

Great. Double check you've saved both lib/logger.ts and lib/metrics.ts.

ⓧ I'd like to double check the full code

Here's the complete lib/logger.ts:

import winston from 'winston';

const { combine, timestamp, json, colorize, printf } = winston.format;

const devFormat = combine(
  colorize(),
  timestamp({ format: 'HH:mm:ss' }),
  printf(({ timestamp, level, message, ...meta }) => {
    const metaStr = Object.keys(meta).length ? ` ${JSON.stringify(meta)}` : '';
    return `${timestamp} ${level}: ${message}${metaStr}`;
  })
);

const prodFormat = combine(
  timestamp(),
  json()
);

export const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || 'info',
  format: process.env.NODE_ENV === 'production' ? prodFormat : devFormat,
  defaultMeta: { service: 'runlayer' },
  transports: [
    new winston.transports.Console(),
  ],
});

export function createRequestLogger(requestId: string, route: string, method: string) {
  return logger.child({ requestId, route, method });
}

Here's the complete lib/metrics.ts:

import client from 'prom-client';

export const registry = new client.Registry();

client.collectDefaultMetrics({ register: registry, prefix: 'runlayer_' });

export const httpRequestDuration = new client.Histogram({
  name: 'runlayer_http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['route', 'method'] as const,
  buckets: [0.01, 0.05, 0.1, 0.3, 0.5, 1, 2, 5],
  registers: [registry],
});

export const httpRequestsTotal = new client.Counter({
  name: 'runlayer_http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['route', 'method', 'status_code'] as const,
  registers: [registry],
});

export const rateLimitHitsTotal = new client.Counter({
  name: 'runlayer_rate_limit_hits_total',
  help: 'Total number of rate limit hits',
  labelNames: ['route'] as const,
  registers: [registry],
});

export const authFailuresTotal = new client.Counter({
  name: 'runlayer_auth_failures_total',
  help: 'Total number of authentication failures',
  labelNames: ['route', 'reason'] as const,
  registers: [registry],
});

export const recommendationsTotal = new client.Counter({
  name: 'runlayer_recommendations_total',
  help: 'Total number of recommendations served',
  registers: [registry],
});

export const activeRequests = new client.Gauge({
  name: 'runlayer_active_requests',
  help: 'Number of currently active requests',
  registers: [registry],
});

Your observability foundation is in place. You have a structured logger that attaches request context to every log line and a metrics registry with six custom metrics covering request performance, rate limiting, auth failures, and recommendation usage. Next up, you'll build the instrumentation middleware that wires these into every API route automatically.

Build the Instrumentation Middleware

Your logger and metrics registry are ready, but they don't do anything automatically yet. Right now, you'd need to manually add timing code and counter increments to every single route handler.

In this step, you'll build a reusable instrumentation middleware that wraps any route handler and automatically collects structured logs, timing data, and metric counts. You'll also expose all your metrics through a dedicated endpoint that Prometheus can scrape.

In this step, get ready to:
  • Create the withInstrumentation higher-order function that wraps route handlers with logging and metrics.
  • Create domain-specific helper functions for recording rate-limit hits, auth failures, and recommendations.
  • Create the /api/metrics route to expose Prometheus-format output.
Create the withInstrumentation wrapper

A higher-order function takes a function as input and returns a new function with extra behaviour layered on top. This pattern lets you add instrumentation to every route handler without duplicating code.

  • Create a new file called lib/instrumentation.ts in the same lib/ folder where your logger and metrics live.
  • Paste the following code into the file:
import { NextRequest, NextResponse } from 'next/server';
import { createRequestLogger, logger } from './logger';
import {
  httpRequestDuration,
  httpRequestsTotal,
  rateLimitHitsTotal,
  authFailuresTotal,
  recommendationsTotal,
  activeRequests,
} from './metrics';

// Define the shape of a Next.js route handler
type RouteHandler = (request: NextRequest) => Promise<NextResponse>;

// Higher-order function that wraps any route handler with instrumentation
export function withInstrumentation(handler: RouteHandler, routeName: string): RouteHandler {
  return async (request: NextRequest) => {
    // Generate a unique ID to correlate all logs for this request
    const requestId = crypto.randomUUID();
    const method = request.method;
    const log = createRequestLogger(requestId, routeName, method);

    // Start a high-resolution timer for duration tracking
    const startTime = process.hrtime.bigint();
    activeRequests.inc();

    log.info('Request started');

    // Default to 500 so unhandled errors still get counted
    let statusCode = 500;

    try {
      const response = await handler(request);
      statusCode = response.status;
      return response;
    } catch (error) {
      log.error('Request failed', {
        error: error instanceof Error ? error.message : String(error),
        stack: error instanceof Error ? error.stack : undefined,
      });
      throw error;
    } finally {
      // Finally block runs whether the request succeeded or failed
      const durationNs = process.hrtime.bigint() - startTime;
      const durationSeconds = Number(durationNs) / 1e9;

      activeRequests.dec();
      httpRequestDuration.observe({ route: routeName, method }, durationSeconds);
      httpRequestsTotal.inc({ route: routeName, method, status_code: String(statusCode) });

      log.info('Request completed', { statusCode, durationMs: Math.round(durationSeconds * 1000) });
    }
  };
}

What does this code do?

The withInstrumentation function wraps any route handler and adds three layers of observability automatically:

  • crypto.randomUUID() generates a unique request ID that gets attached to every log line, letting you trace a single request across all its log entries.
  • process.hrtime.bigint() starts a nanosecond-precision timer. The finally block guarantees the duration is always recorded, even if the handler throws an error.
  • activeRequests.inc() and activeRequests.dec() track how many requests are in-flight at any moment. This gauge helps you spot concurrency bottlenecks.
  • Save lib/instrumentation.ts.

You won't see a change yet. The next chunk adds domain-specific helpers that complete this file.

Create domain-specific instrumentation helpers

Route handlers will call these helpers explicitly when domain-specific events happen, like a rate limit being hit or an authentication attempt failing.

  • In lib/instrumentation.ts, add the following code below the closing } of the withInstrumentation function:
// Domain-specific helpers that route handlers call explicitly
export function recordRateLimitHit(route: string): void {
  rateLimitHitsTotal.inc({ route });
  logger.warn('Rate limit hit', { route });
}

export function recordAuthFailure(route: string, reason: string): void {
  authFailuresTotal.inc({ route, reason });
  logger.warn('Authentication failure', { route, reason });
}

export function recordRecommendation(): void {
  recommendationsTotal.inc();
}

What do these helpers do?

Each helper increments a specific Prometheus counter and logs a warning with context. They keep your route handlers clean by providing one-line calls instead of repeated metric and logging boilerplate.

  • recordRateLimitHit tracks when a request gets throttled by your rate limiter.
  • recordAuthFailure tracks authentication failures with a reason label so you can see whether failures are from expired tokens, invalid credentials, or other causes.
  • recordRecommendation counts how many recommendations your API serves.
  • Save lib/instrumentation.ts.

You won't see a visible change yet. The next substep creates the metrics endpoint that exposes all of this data.

✔️ Awesome, I've got everything!

Great. Double check you've saved lib/instrumentation.ts before moving on.

ⓧ I'd like to double check the full code

Here is the complete lib/instrumentation.ts file:

import { NextRequest, NextResponse } from 'next/server';
import { createRequestLogger, logger } from './logger';
import {
  httpRequestDuration,
  httpRequestsTotal,
  rateLimitHitsTotal,
  authFailuresTotal,
  recommendationsTotal,
  activeRequests,
} from './metrics';

type RouteHandler = (request: NextRequest) => Promise<NextResponse>;

export function withInstrumentation(handler: RouteHandler, routeName: string): RouteHandler {
  return async (request: NextRequest) => {
    const requestId = crypto.randomUUID();
    const method = request.method;
    const log = createRequestLogger(requestId, routeName, method);

    const startTime = process.hrtime.bigint();
    activeRequests.inc();

    log.info('Request started');

    let statusCode = 500;

    try {
      const response = await handler(request);
      statusCode = response.status;
      return response;
    } catch (error) {
      log.error('Request failed', {
        error: error instanceof Error ? error.message : String(error),
        stack: error instanceof Error ? error.stack : undefined,
      });
      throw error;
    } finally {
      const durationNs = process.hrtime.bigint() - startTime;
      const durationSeconds = Number(durationNs) / 1e9;

      activeRequests.dec();
      httpRequestDuration.observe({ route: routeName, method }, durationSeconds);
      httpRequestsTotal.inc({ route: routeName, method, status_code: String(statusCode) });

      log.info('Request completed', { statusCode, durationMs: Math.round(durationSeconds * 1000) });
    }
  };
}

export function recordRateLimitHit(route: string): void {
  rateLimitHitsTotal.inc({ route });
  logger.warn('Rate limit hit', { route });
}

export function recordAuthFailure(route: string, reason: string): void {
  authFailuresTotal.inc({ route, reason });
  logger.warn('Authentication failure', { route, reason });
}

export function recordRecommendation(): void {
  recommendationsTotal.inc();
}
Create the /api/metrics route

Prometheus collects metrics by pulling them from your app at a regular interval. Your app needs a dedicated HTTP endpoint that returns all registered metrics in Prometheus exposition format.

  • Create the folder app/api/metrics/ inside your project's app/ directory.
  • Create a new file called route.ts inside app/api/metrics/ and paste the following code:
import { NextResponse } from 'next/server';
import { registry } from '@/lib/metrics';

// Prevent Next.js from caching the metrics response
export const revalidate = 0;
export const dynamic = 'force-dynamic';

export async function GET() {
  // Gather all registered metrics in Prometheus exposition format
  const metrics = await registry.metrics();
  return new NextResponse(metrics, {
    headers: {
      'Content-Type': registry.contentType,
    },
  });
}

Why revalidate = 0 and dynamic = 'force-dynamic'?

Next.js caches API route responses by default. If the metrics endpoint gets cached, Prometheus would scrape stale data instead of the current values.

Setting revalidate = 0 disables the response cache entirely. Adding dynamic = 'force-dynamic' ensures Next.js never statically renders the route, even in production builds.

  • Save app/api/metrics/route.ts.
  • Start your Next.js dev server by running:
npm run dev
  • Open your browser and navigate to http://localhost:3000/api/metrics.

You should see a wall of text in Prometheus exposition format. Look for lines prefixed with runlayer_ to confirm both the default Node.js metrics and your custom metrics are registered.

Seeing an empty page or an error?

  • Check that app/api/metrics/route.ts is inside the correct folder path. The route must be at app/api/metrics/route.ts for Next.js to register it as /api/metrics.
  • Make sure the import path uses @/lib/metrics (with the @ alias). If your project doesn't have a @ path alias configured, check tsconfig.json for a paths entry.
  • Confirm your dev server is running on port 3000. If another process is using that port, the dev server may start on a different one.

Help me debug my /api/metrics endpoint

✔️ Awesome, I've got everything!

Great. Double check you've saved app/api/metrics/route.ts and your dev server is running.

ⓧ I'd like to double check the full code

Here is the complete app/api/metrics/route.ts file:

import { NextResponse } from 'next/server';
import { registry } from '@/lib/metrics';

export const revalidate = 0;
export const dynamic = 'force-dynamic';

export async function GET() {
  const metrics = await registry.metrics();
  return new NextResponse(metrics, {
    headers: {
      'Content-Type': registry.contentType,
    },
  });
}

Your instrumentation layer is complete. Next up, you'll wrap your existing API route handlers with withInstrumentation and wire the domain helpers into your rate-limiting and auth logic to start collecting real data.

Instrument Your API Routes

Your observability library is ready, but it's sitting on the shelf. The instrumentation middleware, counters, and domain helpers you built in the last step only produce data when they're wired into live request handlers.

In this step, you'll connect your API routes to the instrumentation layer. Every request that passes through your handlers will automatically produce structured log lines, update duration histograms, and increment the appropriate counters.

In this step, get ready to:
  • Wrap route handlers with the withInstrumentation middleware.
  • Wire domain-specific helpers into rate-limiting and auth logic.
  • Verify all custom metrics show non-zero values end-to-end.
Wrap existing route handlers with the instrumentation middleware

The withInstrumentation function is a higher-order wrapper. You pass it your existing handler function and a route name string, and it returns a new function that automatically logs, times, and counts every request.

  • Open your recommendations route file (e.g. app/api/recommendations/route.ts).
  • Add an import for withInstrumentation and recordRecommendation from @/lib/instrumentation.
  • Wrap your existing GET handler with withInstrumentation, following this pattern:
// app/api/recommendations/route.ts
import { withInstrumentation, recordRecommendation } from '@/lib/instrumentation';

export const GET = withInstrumentation(async (request) => {
  // ... your existing recommendation logic ...

  // Record that a recommendation was served
  recordRecommendation();
  return NextResponse.json(recommendations);
}, '/api/recommendations');

What does this pattern do?

Instead of exporting your handler function directly, you export the result of withInstrumentation(handler, routeName). This wraps your handler so that every request is automatically timed, logged, and counted.

The second argument ('/api/recommendations') becomes the route label on your Prometheus metrics. This is how you'll filter dashboards by route later.

  • Apply the same wrapping pattern to your auth routes. In your auth handler, call recordAuthFailure(route, reason) when authentication fails:
// Example: inside your auth route handler
import { withInstrumentation, recordAuthFailure } from '@/lib/instrumentation';

export const POST = withInstrumentation(async (request) => {
  // ... your existing auth validation logic ...

  if (!isValid) {
    // Record the failure with a specific reason
    recordAuthFailure('/api/auth', 'invalid_token');
    return NextResponse.json({ error: 'Unauthorized' }, { status: 401 });
  }

  return NextResponse.json({ success: true });
}, '/api/auth');

What reasons should I use?

The reason parameter becomes a label on the runlayer_auth_failures_total counter. Use short, consistent strings like invalid_token, expired_session, or missing_credentials. These become filterable dimensions in your dashboard.

  • Wrap any other route handlers in your application with the same withInstrumentation pattern. Every wrapped route will automatically appear in your metrics.
Wire rate-limit and auth instrumentation into existing middleware

Some instrumentation calls belong in shared middleware rather than individual route handlers. Rate-limit rejections happen before your route logic runs, so the recording call goes where the rejection happens.

  • Open your rate-limiting middleware or utility file where requests are rejected.
  • Import recordRateLimitHit from @/lib/instrumentation.
  • Add a call to recordRateLimitHit(route) at the point where a request is rejected due to rate limiting:
// Inside your rate-limiting logic (e.g. middleware.ts or a utility)
import { recordRateLimitHit } from '@/lib/instrumentation';

// When a request exceeds the limit:
recordRateLimitHit('/api/recommendations');
return NextResponse.json({ error: 'Too many requests' }, { status: 429 });

Why record rate limits separately?

Rate-limited requests often get rejected before reaching your route handler. If you only rely on withInstrumentation inside the handler, those rejections would be invisible. The separate recordRateLimitHit call captures them at the middleware layer where the rejection actually happens.

  • Similarly, if your auth validation logic lives in a shared utility (outside of route handlers), import recordAuthFailure there and call it when authentication fails.
  • Save all modified files.

Import errors after adding instrumentation?

Make sure your import path uses the @/lib/instrumentation alias. If your project doesn't have the @/ path alias configured, use a relative path like ../../lib/instrumentation instead.

Help me fix import path errors for my instrumentation library

Test the instrumentation end-to-end

With all your routes instrumented, it's time to generate real traffic and confirm the metrics pipeline works from request to Prometheus output.

  • Make sure your Next.js dev server is running.
  • Make several requests to your instrumented API routes by running commands like this:
curl http://localhost:3000/api/recommendations
curl http://localhost:3000/api/recommendations
curl http://localhost:3000/api/recommendations
  • Trigger some auth failures by sending requests with invalid or missing credentials to your auth route.
  • If possible, trigger rate limiting by sending rapid repeated requests to a rate-limited route.
  • Check the metrics output by running:
curl http://localhost:3000/api/metrics
  • Confirm you see non-zero values for each of these metrics in the output:
  • runlayer_http_request_duration_seconds shows histogram buckets with observations.
  • runlayer_http_requests_total shows counts per route, method, and status.
  • runlayer_rate_limit_hits_total increments on rate-limited requests.
  • runlayer_auth_failures_total increments on failed auth.
  • runlayer_recommendations_total increments for each recommendation served.
  • runlayer_active_requests shows 0 when no requests are in flight.

What should the output look like?

Prometheus metrics are plain text. Each metric appears with its name, labels in curly braces, and a numeric value. For example:

runlayer_http_requests_total{route="/api/recommendations",method="GET",status_code="200"} 3
runlayer_http_request_duration_seconds_bucket{route="/api/recommendations",method="GET",le="0.01"} 2
runlayer_recommendations_total 3

The exact numbers depend on how many requests you made. The key confirmation is that values are greater than zero.

Metrics still showing zero?

If your custom metrics all show 0, check that you exported your handler using the withInstrumentation wrapper. A common mistake is wrapping the function but forgetting to assign it to export const GET (or POST).

Also confirm your dev server restarted after saving the route files. Next.js hot-reloads most changes, but occasionally a full restart is needed for new imports to resolve.

Help me debug why my metrics are showing zero values

Your routes are now fully instrumented. Every request produces structured logs and updates Prometheus counters. Next up, you'll deploy a Prometheus + Grafana stack to scrape these metrics and visualize them on a live dashboard.

Deploy the Prometheus + Grafana Stack

Your Next.js app is now producing structured logs and Prometheus-format metrics on every request. But metrics sitting on an endpoint nobody reads are about as useful as writing them on sticky notes and throwing them away.

In this step, you will deploy a local Prometheus + Grafana stack using Docker Compose. Prometheus will scrape your /api/metrics endpoint every 10 seconds, and Grafana will give you a live UI to query and visualize those numbers.

In this step, get ready to:
  • Create a Docker Compose file defining the Prometheus and Grafana services.
  • Configure Prometheus to scrape your Next.js app.
  • Provision Grafana with Prometheus as a pre-configured data source.
Create the Docker Compose configuration

Docker Compose lets you define multiple containers in a single YAML file and spin them all up with one command. You need two services: Prometheus (to collect metrics) and Grafana (to visualize them).

  • Create a new file called docker-compose.observability.yml at the root of your project (the same level as package.json).
  • Paste in the following configuration:
services:
  prometheus:
    image: prom/prometheus:v3.13.0
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    extra_hosts:
      - "host.docker.internal:host-gateway"

  grafana:
    image: grafana/grafana:13.1.0
    ports:
      - "3001:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - ./grafana/dashboards:/var/lib/grafana/dashboards:ro
    depends_on:
      - prometheus

What does this configuration do?

  • The prometheus service uses the official Prometheus image (v3.13.0), maps port 9090, and mounts a local config file into the container.
  • extra_hosts maps host.docker.internal to your host machine so Prometheus can reach your Next.js dev server running on localhost.
  • The grafana service maps port 3001 (avoiding a clash with Next.js on port 3000) and enables anonymous admin access for local development.
  • depends_on ensures Prometheus starts before Grafana, so the data source is available on first load.
  • Save docker-compose.observability.yml.

You won't see a change yet. This file defines the services, but they need a Prometheus config file before they can start.

Create the Prometheus scrape configuration

Prometheus needs a configuration file that tells it which endpoints to scrape and how often. You will point it at your /api/metrics route.

  • Create a new file called prometheus.yml at your project root (same level as docker-compose.observability.yml).
  • Paste in the following scrape configuration:
global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'nextjs-app'
    metrics_path: '/api/metrics'
    static_configs:
      - targets: ['host.docker.internal:3000']

What does this configuration do?

  • scrape_interval: 10s tells Prometheus to pull metrics every 10 seconds.
  • job_name labels all metrics from this target so you can filter them in queries.
  • metrics_path points at your /api/metrics route instead of the Prometheus default (/metrics).
  • targets uses host.docker.internal:3000 to reach your Next.js dev server running on the host machine from inside the Docker container.
  • Save prometheus.yml.

You won't see a change yet. One more config file to go before you start the stack.

Create Grafana provisioning files

Grafana supports file-based provisioning, which means you can pre-configure data sources without clicking through the UI. You will create a provisioning file that registers Prometheus as the default data source on startup.

  • Create the provisioning directory by running this command from your project root:
mkdir -p grafana/provisioning/datasources
  • Confirm the directory exists by running:
ls grafana/provisioning/datasources

You should see an empty directory listed with no errors.

  • Create a new file at grafana/provisioning/datasources/prometheus.yml and paste in the following:
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

What does this provisioning file do?

When Grafana starts, it reads files from /etc/grafana/provisioning inside the container. This file tells Grafana to create a Prometheus data source pointing at http://prometheus:9090 (the Prometheus container's hostname on the Docker network).

Setting isDefault: true means Grafana will use this data source automatically when you create new dashboard panels.

  • Save grafana/provisioning/datasources/prometheus.yml.

All three configuration files are in place. Time to start the stack and see everything connect.

  • Make sure your Next.js dev server is still running on port 3000.
  • Start the observability stack by running this command from your project root:
docker compose -f docker-compose.observability.yml up -d

What does this command do?

  • docker compose -f specifies a custom Compose file (instead of the default docker-compose.yml).
  • up -d creates and starts the containers in detached mode (running in the background).
  • Open your browser and go to http://localhost:9090.
  • Click **Status** in the top navigation bar, then click **Targets**.
  • Confirm you see the nextjs-app job with state UP and a last-scrape time within the last 10 seconds.
  • Open a new tab and go to http://localhost:3001.
  • Click the hamburger menu (three horizontal lines) in the top left, then click **Connections**, then **Data sources**.
  • Confirm you see Prometheus listed as the default data source.

Target showing as DOWN or not appearing?

  • Make sure your Next.js dev server is running on port 3000. Prometheus cannot scrape a server that is not running.
  • If you are on Linux, the extra_hosts entry in docker-compose.observability.yml is what enables host.docker.internal. Confirm it is present and not commented out.
  • Check that prometheus.yml uses host.docker.internal:3000 in the targets list, not localhost:3000 (localhost inside the container refers to the container itself, not your machine).
  • help me debug why Prometheus is not scraping my Next.js app

✔️ Awesome, I've got everything!

Great. Double check you have saved all three files before moving on.

ⓧ I'd like to double check the full code

Here are the complete contents of all three files created in this step.

services:
  prometheus:
    image: prom/prometheus:v3.13.0
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    extra_hosts:
      - "host.docker.internal:host-gateway"

  grafana:
    image: grafana/grafana:13.1.0
    ports:
      - "3001:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - ./grafana/dashboards:/var/lib/grafana/dashboards:ro
    depends_on:
      - prometheus
global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'nextjs-app'
    metrics_path: '/api/metrics'
    static_configs:
      - targets: ['host.docker.internal:3000']
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

Your metrics pipeline is now live. Prometheus is scraping your app every 10 seconds and Grafana is ready to query that data. Next up, you will build a dashboard in Grafana that turns those raw numbers into charts you can actually read.

Build the Grafana Dashboard

Your Prometheus + Grafana stack is running, and Prometheus is happily scraping your Next.js metrics every 10 seconds. But right now, Grafana has no dashboards configured.

In this step, you will create a pre-built dashboard JSON and configure Grafana to auto-provision it on startup. This gives you a repeatable, code-driven dashboard that lives in version control alongside your application.

In this step, get ready to:
  • Create a Grafana dashboard JSON with six visualization panels.
  • Configure Grafana to auto-load dashboards from a provisioned directory.
  • Verify the full observability pipeline with live data.
Create the dashboard JSON

Grafana dashboards can be defined as JSON files. Instead of clicking around in a GUI, you will define your panels in code so they are reproducible and version-controlled.

  • Create the file grafana/dashboards/observability.json in your project root. The grafana/dashboards/ directory should already exist from Step 4.
  • Paste the following dashboard definition into that file:
{
  "dashboard": {
    "title": "RunLayer API Observability",
    "uid": "runlayer-observability",
    "timezone": "browser",
    "refresh": "10s",
    "time": { "from": "now-30m", "to": "now" },
    "panels": [
      {
        "id": 1,
        "title": "Request Rate by Route",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
        "targets": [
          {
            "expr": "sum(rate(runlayer_http_requests_total[5m])) by (route)",
            "legendFormat": "{{route}}"
          }
        ]
      },
      {
        "id": 2,
        "title": "Latency Percentiles (p50 / p95 / p99)",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
        "targets": [
          {
            "expr": "histogram_quantile(0.50, sum(rate(runlayer_http_request_duration_seconds_bucket[5m])) by (le, route))",
            "legendFormat": "p50 {{route}}"
          },
          {
            "expr": "histogram_quantile(0.95, sum(rate(runlayer_http_request_duration_seconds_bucket[5m])) by (le, route))",
            "legendFormat": "p95 {{route}}"
          },
          {
            "expr": "histogram_quantile(0.99, sum(rate(runlayer_http_request_duration_seconds_bucket[5m])) by (le, route))",
            "legendFormat": "p99 {{route}}"
          }
        ]
      },
      {
        "id": 3,
        "title": "Rate Limit Hits",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 8, "x": 0, "y": 8 },
        "targets": [
          {
            "expr": "sum(rate(runlayer_rate_limit_hits_total[5m])) by (route)",
            "legendFormat": "{{route}}"
          }
        ]
      },
      {
        "id": 4,
        "title": "Auth Failures",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 8, "x": 8, "y": 8 },
        "targets": [
          {
            "expr": "sum(rate(runlayer_auth_failures_total[5m])) by (reason)",
            "legendFormat": "{{reason}}"
          }
        ]
      },
      {
        "id": 5,
        "title": "Active Requests",
        "type": "gauge",
        "gridPos": { "h": 8, "w": 4, "x": 16, "y": 8 },
        "targets": [
          {
            "expr": "runlayer_active_requests",
            "legendFormat": "active"
          }
        ]
      },
      {
        "id": 6,
        "title": "Recommendations Served (1h)",
        "type": "stat",
        "gridPos": { "h": 4, "w": 4, "x": 20, "y": 8 },
        "targets": [
          {
            "expr": "increase(runlayer_recommendations_total[1h])",
            "legendFormat": "recommendations"
          }
        ]
      }
    ]
  }
}

What does this dashboard define?

Each panel in the JSON maps to a visualization in Grafana. The targets field contains PromQL queries that pull data from Prometheus.

  • Request Rate by Route uses rate() to calculate requests per second, grouped by route.
  • Latency Percentiles uses histogram_quantile() to calculate p50, p95, and p99 latency from the histogram buckets.
  • Active Requests is a gauge panel showing concurrent in-flight requests at this instant.
  • Recommendations Served uses increase() to show total recommendations over the last hour as a stat counter.
  • Save grafana/dashboards/observability.json.

JSON syntax errors?

JSON is strict about commas and brackets. If your editor shows a red squiggly, check for trailing commas after the last item in any array or object. Also confirm every opening { has a matching }.

Help me fix a JSON syntax error in my Grafana dashboard file

Provision the dashboard automatically

Grafana needs a provisioning config to know where to find dashboard JSON files on disk. Without this, it will not load your dashboard automatically on startup.

  • Create the file grafana/provisioning/dashboards/dashboards.yml inside your project. You will need to create the dashboards/ subdirectory inside grafana/provisioning/.
  • Paste the following provisioning configuration:
apiVersion: 1

providers:
  - name: 'default'
    orgId: 1
    folder: ''
    type: file
    disableDeletion: false
    editable: true
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: false

What does this provisioning file do?

This tells Grafana to look for dashboard JSON files in /var/lib/grafana/dashboards inside the container. Your docker-compose.observability.yml already mounts your local grafana/dashboards/ folder to that path.

The editable: true setting means you can tweak panels in the Grafana UI without restarting the container. Changes made in the UI are temporary though. The JSON file on disk is the source of truth.

  • Save grafana/provisioning/dashboards/dashboards.yml.

Why no Docker Compose update?

Your docker-compose.observability.yml already mounts both ./grafana/provisioning and ./grafana/dashboards from Step 4. No changes needed there. Grafana will pick up the new provisioning file and dashboard JSON when you restart.

Verify the full pipeline

With the dashboard JSON and provisioning config in place, you need to restart the Grafana container so it picks up the new files.

  • Restart the Docker Compose stack by running this command:
docker compose -f docker-compose.observability.yml restart
  • Wait about 10 seconds for the containers to finish restarting.
  • Generate some traffic on your instrumented API routes. Hit a few routes using curl or your browser to produce metrics data:
curl http://localhost:3000/api/recommendations
curl http://localhost:3000/api/recommendations
curl http://localhost:3000/api/recommendations

Hit multiple routes

If you have other instrumented routes (auth, rate-limited endpoints), hit those too. The more variety you generate, the more panels will light up with data.

  • Open Grafana in your browser at http://localhost:3001.
  • Click the hamburger menu icon in the top left corner.
  • Click Dashboards.
  • Click on RunLayer API Observability to open your provisioned dashboard.

You should see all six panels rendered with live data. The Request Rate and Latency Percentiles panels on the top row will show lines corresponding to the routes you hit.

Dashboard not appearing in Grafana?

Make sure grafana/provisioning/dashboards/dashboards.yml exists and the path in options.path is exactly /var/lib/grafana/dashboards. This must match the volume mount in your Docker Compose file.

If you still do not see it, try a full stop and start instead of a restart:

docker compose -f docker-compose.observability.yml down
docker compose -f docker-compose.observability.yml up -d

Help me troubleshoot why my Grafana dashboard is not loading

Panels showing 'No data'?

If the dashboard loads but panels are empty, Prometheus has not scraped enough data yet. Wait at least 30 seconds after hitting your API routes (the dashboard time range is 30 minutes, and Prometheus scrapes every 10 seconds).

Also confirm your Next.js dev server is still running on port 3000. Check the Prometheus targets page at http://localhost:9090/targets to verify the nextjs-app target shows UP.

Help me figure out why my Grafana panels show no data

✔️ Awesome, I've got everything!

Great. Double check you have saved both files.

ⓧ I'd like to double check the full code

Here are the two new files you created in this step:

{
  "dashboard": {
    "title": "RunLayer API Observability",
    "uid": "runlayer-observability",
    "timezone": "browser",
    "refresh": "10s",
    "time": { "from": "now-30m", "to": "now" },
    "panels": [
      {
        "id": 1,
        "title": "Request Rate by Route",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
        "targets": [
          {
            "expr": "sum(rate(runlayer_http_requests_total[5m])) by (route)",
            "legendFormat": "{{route}}"
          }
        ]
      },
      {
        "id": 2,
        "title": "Latency Percentiles (p50 / p95 / p99)",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
        "targets": [
          {
            "expr": "histogram_quantile(0.50, sum(rate(runlayer_http_request_duration_seconds_bucket[5m])) by (le, route))",
            "legendFormat": "p50 {{route}}"
          },
          {
            "expr": "histogram_quantile(0.95, sum(rate(runlayer_http_request_duration_seconds_bucket[5m])) by (le, route))",
            "legendFormat": "p95 {{route}}"
          },
          {
            "expr": "histogram_quantile(0.99, sum(rate(runlayer_http_request_duration_seconds_bucket[5m])) by (le, route))",
            "legendFormat": "p99 {{route}}"
          }
        ]
      },
      {
        "id": 3,
        "title": "Rate Limit Hits",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 8, "x": 0, "y": 8 },
        "targets": [
          {
            "expr": "sum(rate(runlayer_rate_limit_hits_total[5m])) by (route)",
            "legendFormat": "{{route}}"
          }
        ]
      },
      {
        "id": 4,
        "title": "Auth Failures",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 8, "x": 8, "y": 8 },
        "targets": [
          {
            "expr": "sum(rate(runlayer_auth_failures_total[5m])) by (reason)",
            "legendFormat": "{{reason}}"
          }
        ]
      },
      {
        "id": 5,
        "title": "Active Requests",
        "type": "gauge",
        "gridPos": { "h": 8, "w": 4, "x": 16, "y": 8 },
        "targets": [
          {
            "expr": "runlayer_active_requests",
            "legendFormat": "active"
          }
        ]
      },
      {
        "id": 6,
        "title": "Recommendations Served (1h)",
        "type": "stat",
        "gridPos": { "h": 4, "w": 4, "x": 20, "y": 8 },
        "targets": [
          {
            "expr": "increase(runlayer_recommendations_total[1h])",
            "legendFormat": "recommendations"
          }
        ]
      }
    ]
  }
}
apiVersion: 1

providers:
  - name: 'default'
    orgId: 1
    folder: ''
    type: file
    disableDeletion: false
    editable: true
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: false

Your full observability pipeline is live. Metrics flow from your Next.js app through Prometheus and into Grafana panels that update every 10 seconds. Next up, you will push this further with alerting in the Secret Mission.

Secret mission

Slow Endpoint Detection with Prometheus Alerting

Your dashboard shows what's happening right now. But what about when something goes wrong at 3am and nobody's watching? In this mission, you'll configure Prometheus to automatically detect slow endpoints and surface alerts in Grafana.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

This project runs entirely locally via Docker, so there are no ongoing cloud costs. The Prometheus and Grafana containers use minimal system resources (under 200MB RAM combined).

Resources you used:

  • Prometheus container running on port 9090.
  • Grafana container running on port 3001.

Keep everything running

No action needed. Choose this if you're still actively building or want continuous observability during development.

  • Your Prometheus and Grafana containers will continue running in the background.
  • Prometheus scrapes your Next.js app every 10 seconds while the dev server is running.
  • If you added an artificial delay to a route handler during the Secret Mission testing, remove it now so your app runs at normal speed.

Pause - I'll come back to this later

Shut down the Docker containers to free up memory. Your configuration and data stay intact for next time.

  • Stop the observability stack by running:
docker compose -f docker-compose.observability.yml stop
  • When you're ready to resume, start the containers again by running:
docker compose -f docker-compose.observability.yml start
  • If you added an artificial delay to a route handler during the Secret Mission testing, remove it now.

Delete - I don't want to use this again

Remove all Docker containers, volumes, and configuration files to start completely fresh.

  • Stop and remove the containers and their volumes by running:
docker compose -f docker-compose.observability.yml down -v
  • Remove the Docker Compose and Prometheus configuration files by running:
rm docker-compose.observability.yml prometheus.yml prometheus-rules.yml
  • Remove the Grafana provisioning and dashboard directory by running:
rm -rf grafana
  • Remove the observability library files by running:
rm lib/logger.ts lib/metrics.ts lib/instrumentation.ts app/api/metrics/route.ts
  • Uninstall the npm packages by running:
npm uninstall winston prom-client
  • If you added an artificial delay to a route handler during the Secret Mission testing, remove that code as well.
  • Remove any withInstrumentation wrappers and instrumentation helper calls from your route handlers if you want to fully revert.

That's a Wrap!

That's a Wrap!

Nice work! You've built a production-grade observability layer for your Next.js application, all running locally at zero cost.

You've learned how to:

  • Built a structured logging layer using Winston with request-scoped child loggers, JSON output for production log aggregators, and colorized dev formatting.
  • Created a reusable instrumentation middleware that automatically collects Prometheus metrics (counters, histograms, gauges) and structured logs for every API route handler.
  • Deployed a zero-cost Prometheus + Grafana Docker Compose stack with auto-provisioned data sources and a live dashboard showing request rates, latency percentiles, and failure trends.
  • Secret Mission: Implemented Prometheus alerting rules for slow-endpoint detection and surfaced active alerts in a Grafana dashboard panel.

Ready to quiz yourself?