Evaluator Design and Custom Stop Hooks

Build a multi-signal evaluation pipeline that catches quality issues tests miss.

Introduction

30 Second Summary

You run all your tests and see green checkmarks across the board. Then a real user sends an empty request and your app crashes.

In this project, you will build custom quality gates for Claude Code that catch the issues your tests miss. You will start with a deliberately naive URL shortener where all tests pass, then wire evaluation hooks that block Claude from stopping until real integration checks go green.

What You'll Build

You will prompt Claude Code to improve your URL shortener, watch it try to stop after tests pass, and see your quality gate send it back to work until every integration check passes.

By the end of this project, you'll have:

  • A deterministic Stop hook that sends real HTTP requests to your running server and blocks Claude Code from stopping until every integration check passes.
  • A prompt-based Stop hook that reviews Claude's work for quality issues scripted checks cannot catch, using session evidence instead of runtime verification.
  • A multi-signal gate where both hooks fire in parallel and Claude keeps working until every evaluator gives the green light.
  • Secret Mission: Deliberately create a misleading test suite that triggers a false PASS, then build an evaluator that catches the gap.

Are there any prerequisites?

This is Project 2 in the Claude Code Loop Engineering series. Complete Project 1 (Autonomous Loops with Claude Code) before starting, and make sure you have Node.js and Claude Code already installed and configured.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up the Project and Hook Infrastructure

Every Stop hook you write needs a project to evaluate. Before you can build a single evaluator, you need a Node.js project with Express installed, a directory where hook scripts live, and a set of documented quality standards that define what "production-ready" means.

Getting this structure right now means every hook you write later will just work. In this step, you will scaffold the project, install dependencies, and lay down the rules your evaluators will enforce.

In this step, get ready to:
  • Create a Node.js project with Express installed.
  • Set up the hook directory and write quality standards for Claude Code.
  • Configure and verify the built-in test runner.
Create the project and install Express
  • Press Cmd+Space to open your search bar.
  • Type Terminal and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir url-shortener
cd url-shortener
  • Initialize your Node.js project by running:
npm init -y

You should see a summary of the new package.json printed to your terminal.

  • Install Express by running:
npm install express@5.2.1

You should see added followed by a package count in the output, confirming Express is installed.

  • Verify Express loads correctly by running:
node -e "require('express'); console.log('Express loaded')"

You should see Express loaded printed. This confirms Node.js can find and import the Express package from your node_modules folder.

Seeing 'Cannot find module' or 'command not found'?

  • If you see Cannot find module 'express', make sure you ran npm install express@5.2.1 from inside the url-shortener folder.
  • If you see node: command not found, Node.js may not be on your PATH. Close and reopen Terminal, then try again.
  • Why can't Node find my Express module?

Your hook scripts will use jq to parse JSON input from Claude Code. Check whether it is installed by running:

jq --version

✔️ I see a version number

You should see something like jq-1.7.1. The exact version does not matter. jq is ready to go.

ⓧ Command not found

  • Install jq using Homebrew by running:
brew install jq
  • After the install finishes, verify by running:
jq --version

You should now see a version number like jq-1.7.1.

Create the hook directory and write quality standards

Claude Code looks for hook scripts inside your project's .claude/hooks/ directory. You also need a CLAUDE.md file that tells Claude what quality standards your project follows.

  • Create the hook directory by running:
mkdir -p .claude/hooks
  • Confirm the directory exists by running:
ls .claude/hooks/

The command should run without errors. The directory is empty for now, so you will not see any files listed.

  • Create the CLAUDE.md file with the project's quality standards by running this command:
cat > CLAUDE.md << 'EOF'
# URL Shortener API

This is a simple URL shortener API built with Express.

## Tech Stack

- Node.js with Express
- In-memory storage (no database required)

## API Endpoints

- POST /shorten - Create a short URL. Requires `url` field in JSON body.
- GET /:code - Redirect to the original URL
- GET /health - Health check endpoint

## Quality Standards

- All inputs must be validated before processing
- Error responses must use consistent JSON format: { "error": "description" }
- Configuration values (port) must use environment variables with defaults
- Use semantically correct HTTP status codes (201 for creation, 400 for bad input)
- Export the app for testing. Only call app.listen() when run directly.

## Testing

- Run unit tests: npm test
- Run integration quality checks: node quality-checks.js
EOF

What does CLAUDE.md do?

CLAUDE.md is a special file that Claude Code reads automatically when you start a session. It gives Claude context about your project and defines the rules it should follow.

The four Quality Standards are key. They define what "production-ready" means for this project: input validation, consistent error format ({ "error": "message" }), a configurable port, and semantically correct HTTP status codes. Your Stop hooks will enforce these exact standards later.

  • Confirm the file was created by running:
ls

You should see CLAUDE.md, node_modules, package-lock.json, and package.json listed.

Don't see CLAUDE.md listed?

  • Make sure you pressed Enter after pasting the full cat command. The shell waits for the closing EOF line before writing the file.
  • If you see a > prompt instead of your normal terminal prompt, the heredoc is still open. Type EOF on its own line and press Enter to close it.
  • Why isn't my CLAUDE.md file being created by the cat command?

✔️ Awesome, I've got everything!

Great. Your CLAUDE.md is ready and your .claude/hooks/ directory is waiting for your first hook script.

ⓧ I'd like to double check the full code

Run this command to print the contents of CLAUDE.md and compare them to what you see below:

cat CLAUDE.md
# URL Shortener API

This is a simple URL shortener API built with Express.

## Tech Stack

- Node.js with Express
- In-memory storage (no database required)

## API Endpoints

- POST /shorten - Create a short URL. Requires `url` field in JSON body.
- GET /:code - Redirect to the original URL
- GET /health - Health check endpoint

## Quality Standards

- All inputs must be validated before processing
- Error responses must use consistent JSON format: { "error": "description" }
- Configuration values (port) must use environment variables with defaults
- Use semantically correct HTTP status codes (201 for creation, 400 for bad input)
- Export the app for testing. Only call app.listen() when run directly.

## Testing

- Run unit tests: npm test
- Run integration quality checks: node quality-checks.js
Configure the test script and verify

The npm init -y command created a default package.json with a placeholder test script. You need to replace it with one that uses Node's built-in test runner.

  • Overwrite package.json with the correct project configuration by running:
cat > package.json << 'EOF'
{
  "name": "url-shortener",
  "version": "1.0.0",
  "description": "URL shortener API for Claude Code hook engineering",
  "main": "server.js",
  "scripts": {
    "test": "node --test",
    "start": "node server.js"
  },
  "dependencies": {
    "express": "5.2.1"
  }
}
EOF

What changed in package.json?

  • "test": "node --test" replaces the placeholder script with Node's built-in test runner. It automatically discovers *.test.js files in your project.
  • "start": "node server.js" gives you a convenient way to start the server later.
  • "main": "server.js" points to the entry file you will create in the next step.

Before you run the test script, what do you think will happen? There are no *.test.js files in the project yet.

  • Verify the test runner is wired up correctly by running:
npm test

The command should exit cleanly with no failures. You will see zero tests reported because no test files exist yet.

Seeing an error when running npm test?

  • If you see a message about the test script not being found, run cat package.json and confirm the "test" entry inside "scripts" reads "node --test".
  • If you see a JSON syntax error, check for missing commas or mismatched braces in your package.json. Compare it against the double-check tab below.
  • Why is npm test failing in my url-shortener project?

✔️ Awesome, I've got everything!

Great. Double check you have saved your file, then move on to the next step.

ⓧ I'd like to double check the full code

Run this command to print the contents of package.json and compare them to the reference below:

cat package.json
{
  "name": "url-shortener",
  "version": "1.0.0",
  "description": "URL shortener API for Claude Code hook engineering",
  "main": "server.js",
  "scripts": {
    "test": "node --test",
    "start": "node server.js"
  },
  "dependencies": {
    "express": "5.2.1"
  }
}

Your project scaffolding is complete. You have Express installed, a hook directory ready, quality standards documented, and a test runner configured. Next up, you will build the deliberately naive URL shortener that exposes the gap between passing tests and production quality.

Build the Naive API and Watch Tests Pass

Your project structure is ready. Express is installed, the hooks directory is in place, and CLAUDE.md spells out the quality bar your API needs to meet.

In this step, you'll write a URL shortener that passes every test you throw at it. Then you'll discover that passing tests and being production-ready are two very different things.

In this step, get ready to:
  • Write a deliberately naive URL shortener API with no input validation.
  • Write three happy-path tests that all pass.
  • Expose the gap between passing tests and real production quality.
Write the URL shortener server

The URL shortener needs two endpoints. POST /shorten takes a URL and returns a short code. GET /:code redirects to the original URL.

You're going to write this server deliberately without input validation, error handling, or a health endpoint. The naivety is the point.

  • Create a file called server.js in your project folder and paste the following code:
// Set up Express and JSON body parsing
const express = require('express');
const app = express();

app.use(express.json());

// In-memory storage for shortened URLs
const urls = {};
let counter = 1;

// Accept a URL and return a short code
app.post('/shorten', (req, res) => {
  const short = 's' + counter++;
  urls[short] = req.body.url;
  res.json({ short });
});

What does this code do?

  • express.json() is middleware that parses incoming JSON request bodies automatically.
  • urls is a plain object acting as an in-memory database. Short codes are keys, original URLs are values.
  • The POST /shorten handler generates codes like s1, s2, stores the mapping, and returns the code. It does not check whether req.body.url actually exists or is a valid URL.
  • Save server.js.
  • Confirm the file was created by running:
ls server.js

You should see server.js listed in the output.

Don't see server.js?

  • Make sure you created the file in the same directory where package.json lives. Run ls package.json to confirm you're in the right folder.
  • Check that you saved the file after pasting the code.

Why isn't my server.js file showing up?

  • Add the redirect endpoint and server startup below the existing code in server.js:
// Look up a short code and redirect to the original URL
app.get('/:code', (req, res) => {
  const url = urls[req.params.code];
  if (url) {
    res.redirect(url);
  } else {
    res.status(404).json({ error: 'Not found' });
  }
});

// Only start the server when run directly, not when imported
if (require.main === module) {
  app.listen(3000, () => console.log('Running on port 3000'));
}

module.exports = app;

What does this code do?

  • GET /:code looks up the short code in the urls object and redirects if found. Otherwise it returns a 404.
  • The require.main === module check means the server only starts listening when you run the file directly. When your tests import it, the app loads without starting a listener.
  • module.exports = app exports the Express app so your test file can import it.
  • Save server.js.
  • Verify the server loads correctly by running:
node -e "require('./server'); console.log('Server loaded')"

You should see Server loaded printed with no errors.

Seeing a syntax error?

  • Make sure both code chunks are in server.js in order, with no duplicate const express = require('express') lines.
  • Check that every opening brace { has a matching closing brace } and every opening parenthesis has a matching close.

Help me debug my server.js syntax error.

✔️ Awesome, I've got everything!

Great. Make sure you've saved server.js before moving on.

ⓧ I'd like to double check the full code

Here is the full server.js file. Your version may include the comments from the instructions above. The important thing is that the code structure matches.

const express = require('express');
const app = express();

app.use(express.json());

const urls = {};
let counter = 1;

app.post('/shorten', (req, res) => {
  const short = 's' + counter++;
  urls[short] = req.body.url;
  res.json({ short });
});

app.get('/:code', (req, res) => {
  const url = urls[req.params.code];
  if (url) {
    res.redirect(url);
  } else {
    res.status(404).json({ error: 'Not found' });
  }
});

if (require.main === module) {
  app.listen(3000, () => console.log('Running on port 3000'));
}

module.exports = app;
Write the happy-path tests

Three tests cover the happy path: creating a short URL, following a redirect, and hitting an unknown code. These tests use Node's built-in node:test runner and the global fetch API. No extra testing libraries needed.

  • Create a file called server.test.js in your project folder and paste the following code:
const { describe, it, before, after } = require('node:test');
const assert = require('node:assert');
const app = require('./server');

describe('URL Shortener', () => {
  let server;
  let base;

  // Start the server on a random port before tests run
  before(() => {
    return new Promise((resolve) => {
      server = app.listen(0, () => {
        base = `http://localhost:${server.address().port}`;
        resolve();
      });
    });
  });

  // Shut down the server after tests complete
  after(() => {
    return new Promise((resolve) => server.close(resolve));
  });

What does this setup do?

  • require('node:test') imports Node's built-in test runner. No external test libraries needed.
  • app.listen(0) starts the server on a random available port. This prevents port conflicts when tests run.
  • before and after hooks start and stop the server for the test suite so each run is clean.
  • Save server.test.js.
  • Confirm the file was created by running:
ls server.test.js

You should see server.test.js listed in the output.

  • Add the three test cases below the existing code in server.test.js:
  // Test: creating a short URL returns a code
  it('should shorten a URL', async () => {
    const res = await fetch(`${base}/shorten`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ url: 'https://example.com' })
    });
    assert.strictEqual(res.status, 200);
    const body = await res.json();
    assert.ok(body.short);
  });

  // Test: following a short code redirects to the original URL
  it('should redirect to the original URL', async () => {
    const create = await fetch(`${base}/shorten`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ url: 'https://example.com/redirect-test' })
    });
    const { short } = await create.json();
    const res = await fetch(`${base}/${short}`, { redirect: 'manual' });
    assert.strictEqual(res.status, 302);
  });

  // Test: unknown codes return 404
  it('should return 404 for unknown codes', async () => {
    const res = await fetch(`${base}/nonexistent`);
    assert.strictEqual(res.status, 404);
  });
});

What do these tests check?

  • The first test creates a short URL and verifies the response includes a short code.
  • The second test creates a short URL, then follows the redirect and checks for a 302 status.
  • The third test requests a nonexistent code and checks for a 404.
  • All three test the happy path only. None of them test what happens with bad input.
  • Save server.test.js.

Before you run the tests, how many of the 3 do you expect to pass?

  • Run the test suite by running:
npm test

All 3 tests pass with green checkmarks. The server works perfectly for every case your tests cover.

Tests failing or not found?

  • If no tests are found, make sure your file is named exactly server.test.js (not server.tests.js or test.js). The node --test runner discovers files matching *.test.js.
  • If you see a syntax error, check that both chunks of code are present in server.test.js and the closing }); from the second chunk is the last line.
  • If a test fails with a connection error, make sure server.js has module.exports = app at the bottom so the tests can import the app.

Help me figure out why my tests aren't passing.

✔️ Awesome, I've got everything!

Great. Make sure you've saved server.test.js before moving on.

ⓧ I'd like to double check the full code

Scroll up and confirm your server.test.js has these sections in order:

  • The three imports (node:test, node:assert, and ./server).
  • The describe block with before and after hooks that start and stop the server.
  • Three it() blocks: shorten a URL, redirect to the original URL, and return 404 for unknown codes.
  • A closing }); as the final line that closes the describe block.
Expose the quality gap

All three tests pass. The server works perfectly for every case those tests cover. But open CLAUDE.md and look at the quality standards: input validation, consistent error responses, a health endpoint, and a configurable port. The server has none of those.

Your tests are all green. What do you think the server returns when you POST a request with no URL in the body?

  • Start the server in the background by running:
node server.js &

Wait until you see Running on port 3000 printed. The server is now listening in the background.

  • Send an empty JSON body to the shorten endpoint by running:
curl -X POST http://localhost:3000/shorten -H "Content-Type: application/json" -d '{}'

The server returns something like {"short":"s1"} with a 200 status. It cheerfully created a short code that points to nothing. According to your quality standards, this should return a 400 error.

  • Try sending an invalid URL by running:
curl -X POST http://localhost:3000/shorten -H "Content-Type: application/json" -d '{"url": "not-a-url"}'

Same result. The server happily creates a short code for a string that is not a real URL.

  • Check for a health endpoint by running:
curl http://localhost:3000/health

The server returns a 404 with {"error":"Not found"}. There is no health endpoint at all, even though CLAUDE.md lists one as required.

  • Stop the background server by running:
kill %1

Why do the tests miss all of this?

The tests only check what happens when everything goes right. None of them send bad input. None check for a health endpoint. None verify that the port is configurable.

This is the gap between "tests pass" and "production-ready." Your Stop hooks in the next step will close this gap by running exactly these kinds of checks as a quality gate that Claude must pass before it can stop working.

You have a server that passes every test and fails every quality check. In the next step, you'll write a deterministic Stop hook that catches these gaps and forces Claude to fix them before it can stop working.

Write and Wire the Deterministic Stop Hook

In the last step, you saw the gap firsthand. All three unit tests passed, but the server happily accepted empty requests, nonsense URLs, and returned no health endpoint. Your CLAUDE.md file spells out exactly what "production-ready" means, but those standards are advisory. Claude can read them and move on without acting on them.

A deterministic Stop hook changes the equation. It runs real HTTP requests against your live server every time Claude tries to stop working. If the integration checks fail, Claude gets the failure output piped back as its next instruction and keeps iterating until the code genuinely passes.

In this step, get ready to:
  • Write an integration check script that tests the server with real HTTP requests.
  • Build a Stop hook shell script that runs both unit tests and integration checks as a quality gate.
  • Wire the hook into Claude Code and watch the key loop: tests PASS, quality gate FAILS, Claude iterates, quality gate PASSES.
Write the integration check script

Your unit tests only check happy paths. The integration check script will start the actual server, send it deliberately bad input, and verify that the server rejects it correctly.

  • Create quality-checks.js in your project root and add the server setup and first integration check:
// Import the Express app to start it on a random port
const app = require('./server');

async function runChecks() {
  // Start the server on a random available port
  const server = await new Promise((resolve) => {
    const s = app.listen(0, () => resolve(s));
  });

  const port = server.address().port;
  const base = `http://localhost:${port}`;
  const failures = [];

  // Check 1: POST /shorten with empty body should return 400
  const res1 = await fetch(`${base}/shorten`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({})
  });
  if (res1.status !== 400) {
    failures.push(
      `POST /shorten with missing url returned ${res1.status}, expected 400`
    );
  }

What does this code do?

  • app.listen(0) starts the server on a random available port. This avoids conflicts with anything already running on port 3000.
  • failures is an array that collects human-readable descriptions of every check that fails. These descriptions are what Claude reads when the hook blocks it.
  • The first check sends a POST request with an empty body. A production-ready server should return 400 for missing input. Your current server returns 200 and silently stores undefined.
  • Still in quality-checks.js, add the remaining two checks, teardown, and the function call below the first check:
  // Check 2: POST /shorten with an invalid URL should return 400
  const res2 = await fetch(`${base}/shorten`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ url: 'not-a-valid-url' })
  });
  if (res2.status !== 400) {
    failures.push(
      `POST /shorten with invalid url returned ${res2.status}, expected 400`
    );
  }

  // Check 3: GET /health should return 200
  const res3 = await fetch(`${base}/health`);
  if (res3.status !== 200) {
    failures.push(
      `GET /health returned ${res3.status}, expected 200`
    );
  }

  // Shut down the server and report results
  await new Promise((resolve) => server.close(resolve));

  if (failures.length > 0) {
    console.error(failures.join('\n'));
    process.exit(1);
  }
}

runChecks();

What do these checks cover?

  • Check 2 sends a URL value of "not-a-valid-url". A production server should reject this with 400.
  • Check 3 hits GET /health. Your server has no health endpoint at all, so this returns 404.
  • process.exit(1) signals failure to any parent process. The Stop hook script will read this exit code to decide whether Claude can stop.
  • Save quality-checks.js.

✔️ Awesome, I've got everything!

Great. Make sure you have saved the file before continuing.

ⓧ I'd like to double check the full code

Scroll up and confirm both code blocks are present in quality-checks.js, in this order:

  • The server setup and first integration check (POST with empty body).
  • The remaining two checks (invalid URL, health endpoint), teardown, and the runChecks() call at the bottom.

Before you run this script, think about what you expect to happen. The naive server has no input validation and no health endpoint. How many of the three checks do you think will pass?

  • Run the integration checks against your naive server:
node quality-checks.js

All three checks fail. You should see output listing each failure: the empty body returned 200 instead of 400, the invalid URL returned 200 instead of 400, and the health endpoint returned 404 instead of 200.

Seeing a different error?

  • If you see Cannot find module './server', make sure quality-checks.js is in the same folder as server.js (your project root, not inside .claude/hooks/).
  • If you see a port-in-use error, make sure you do not have the server running in another terminal tab. Stop it with Ctrl+C first.

Help me debug my quality-checks.js script

Why do the checks fail?

This is exactly the gap from Step 2 made measurable. Your unit tests only verify happy paths. The integration checks verify what should not work. Right now, everything "works" because the server never says no.

Write the Stop hook shell script

The integration checks can catch problems, but right now they are just a script you run manually. A Stop hook runs them automatically every time Claude tries to finish its work. If the checks fail, the hook feeds the failure output back to Claude and blocks it from stopping.

  • Create .claude/hooks/quality-gate.sh and add the full hook script:
#!/bin/bash
INPUT=$(cat)
STOP_HOOK_ACTIVE=$(echo "$INPUT" | jq -r '.stop_hook_active')

if [ "$STOP_HOOK_ACTIVE" = "true" ]; then  # Respect 8-block override
  exit 0
fi

cd "$CLAUDE_PROJECT_DIR"

FAILURES=""

TEST_OUTPUT=$(npm test 2>&1)  # Run unit tests
if [ $? -ne 0 ]; then
  FAILURES="${FAILURES}\n- Unit tests failing:\n${TEST_OUTPUT}"
fi

CHECK_OUTPUT=$(node quality-checks.js 2>&1)  # Run integration checks
if [ $? -ne 0 ]; then
  FAILURES="${FAILURES}\n- Integration quality checks failing:\n${CHECK_OUTPUT}"
fi

if [ -n "$FAILURES" ]; then  # Block Claude if anything failed
  printf "Quality gate FAILED. Fix these issues:%b\n" "$FAILURES" >&2
  exit 2
fi

exit 0

What does this script do?

  • Claude Code pipes a JSON object to the hook's stdin every time Claude tries to stop. The script reads it with INPUT=$(cat) and uses jq to extract stop_hook_active.
  • The stop_hook_active check is a safety valve. Claude Code overrides a Stop hook after 8 consecutive blocks without progress. When that happens, stop_hook_active is true and the script exits cleanly so Claude can stop.
  • Exit code 2 tells Claude Code to block the stop. Anything printed to stderr gets fed back to Claude as its next instruction, so Claude knows exactly what to fix.
  • Save .claude/hooks/quality-gate.sh.
  • Make the script executable by running this command:
chmod +x .claude/hooks/quality-gate.sh
  • Confirm the file is executable by running:
ls -la .claude/hooks/quality-gate.sh

You should see -rwxr-xr-x (or similar) at the start of the line. The x characters confirm the file is executable.

✔️ Awesome, I've got everything!

Great. Confirm the file is saved and executable before moving on.

ⓧ I'd like to double check the full code

#!/bin/bash
INPUT=$(cat)
STOP_HOOK_ACTIVE=$(echo "$INPUT" | jq -r '.stop_hook_active')

if [ "$STOP_HOOK_ACTIVE" = "true" ]; then
  exit 0
fi

cd "$CLAUDE_PROJECT_DIR"

FAILURES=""

TEST_OUTPUT=$(npm test 2>&1)
if [ $? -ne 0 ]; then
  FAILURES="${FAILURES}\n- Unit tests failing:\n${TEST_OUTPUT}"
fi

CHECK_OUTPUT=$(node quality-checks.js 2>&1)
if [ $? -ne 0 ]; then
  FAILURES="${FAILURES}\n- Integration quality checks failing:\n${CHECK_OUTPUT}"
fi

if [ -n "$FAILURES" ]; then
  printf "Quality gate FAILED. Fix these issues:%b\n" "$FAILURES" >&2
  exit 2
fi

exit 0

Not seeing the x permission?

  • Run chmod +x .claude/hooks/quality-gate.sh again and check the path. Make sure you are in your project root, not inside the .claude/hooks/ directory.
  • If the file does not exist at that path, confirm you created it inside .claude/hooks/ and not in the project root.

Help me make my quality-gate.sh script executable

Wire the hook and watch Claude iterate

The integration check script and the hook script are ready. Now you need to tell Claude Code to run the hook every time Claude tries to stop. This configuration lives in .claude/settings.json.

  • Create .claude/settings.json and add the Stop hook configuration:
{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/quality-gate.sh"
          }
        ]
      }
    ]
  }
}

What does this configuration do?

  • The "Stop" key tells Claude Code which event triggers the hook. The Stop event fires every time Claude finishes responding and is about to end its turn.
  • type: "command" means this is a deterministic hook that runs a shell command. Claude Code also supports "prompt" hooks (you will build one in the next step).
  • $CLAUDE_PROJECT_DIR is an environment variable that Claude Code exports to every hook process. It resolves to your project root so the hook can find its scripts.
  • Save .claude/settings.json.

✔️ Awesome, I've got everything!

Great. Double check you have saved the file before continuing.

ⓧ I'd like to double check the full code

{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/quality-gate.sh"
          }
        ]
      }
    ]
  }
}

Everything is wired. Time to see the loop in action. You will open Claude Code, give it a task, and watch the quality gate do its job.

  • Open Claude Code in your project directory by running:
claude

What if Claude Code asks about the hook?

When Claude Code starts, it detects the new Stop hook in .claude/settings.json. It may ask you to approve the hook. Accept it to continue.

  • Verify your hook is registered by typing /hooks in Claude Code. You should see your Stop hook listed under the Stop event.

Before you send the prompt below, think about what will happen. Claude has your CLAUDE.md quality standards and the quality gate hook is active. Will the naive server pass the quality gate on Claude's first attempt, or will it take multiple iterations?

  • Type this prompt into Claude Code and press Enter:
Make this URL shortener production-ready. Follow the quality standards in CLAUDE.md.

Watch the sequence unfold. Claude makes initial changes to the server code (adding validation, a health endpoint, error handling). Then it tries to stop.

What should I see?

This is the key moment of the project. When Claude tries to stop, the quality gate runs your unit tests and integration checks against the actual server. If any check fails, Claude gets the failure output and keeps working.

You should see at least one cycle where the quality gate reports specific failures (like a check still returning the wrong status code). Claude reads that feedback, makes targeted fixes, and tries to stop again. When all checks pass, the quality gate exits 0 and Claude stops successfully.

Claude stopped without hitting the quality gate?

  • Check that .claude/settings.json is saved and the JSON is valid. A missing comma or bracket silently disables the hook.
  • Confirm quality-gate.sh is executable. Run ls -la .claude/hooks/quality-gate.sh and look for the x permission.
  • Type /hooks inside Claude Code to verify the hook is registered. If it does not appear, exit Claude Code, fix the config, and run claude again.

Help me debug why my Stop hook is not running

You just witnessed the core loop that makes Stop hooks powerful. Tests alone gave you a green checkmark on code that was missing validation, error handling, and a health endpoint. The deterministic hook forced Claude to fix every gap before it could stop. Next up, you will build a prompt-based hook that catches quality issues that scripted checks cannot.

Add a Prompt-Based Stop Hook

Your deterministic Stop hook proved its worth in the last step. Claude tried to stop, the quality gate blocked it, and Claude iterated until the code passed your integration checks.

Deterministic hooks only catch failure patterns you script for. Some quality issues require judgment that a shell script cannot express. A prompt-based Stop hook evaluates Claude's work using an LLM, assessing session evidence like last_assistant_message rather than running code against the live server.

In this step, get ready to:
  • Replace the deterministic hook with a prompt-based Stop hook in .claude/settings.json.
  • Prompt Claude to add a /stats endpoint and observe the prompt hook evaluating session evidence.
  • Identify the false PASS and false FAIL risks of prompt-based evaluation.
Swap the command hook for a prompt hook

The deterministic hook in quality-gate.sh runs shell commands and checks exit codes. A prompt-based hook works differently. It sends the hook's input JSON to a fast evaluator model, which returns a judgment: {"ok": true} or {"ok": false, "reason": "..."}.

  • Open .claude/settings.json in your editor.
  • Replace the entire contents with this prompt-based hook configuration:
{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          {
            "type": "prompt",
            "prompt": "You are a code quality evaluator for a URL shortener API. Based on Claude's last response, evaluate whether the implementation meets these standards:\n\n1. Input validation: Does the code validate that required fields exist and have correct formats before processing?\n2. Error consistency: Do all error responses use the format { \"error\": \"description\" }?\n3. Configuration: Is the port configurable via an environment variable with a default?\n4. HTTP semantics: Does POST /shorten return 201 (not 200) on success and 400 on bad input?\n\nContext: $ARGUMENTS\n\nIf all standards appear met, respond {\"ok\": true}.\nIf any standard is not met, respond {\"ok\": false, \"reason\": \"list the specific issues\"}."
          }
        ]
      }
    ]
  }
}

What does this configuration do?

Instead of "type": "command" running a shell script, "type": "prompt" sends the prompt text to a fast evaluator model. The model receives the hook input through the $ARGUMENTS placeholder, which includes last_assistant_message containing Claude's summary of what it changed.

The prompt defines four quality dimensions to evaluate: input validation, consistent error format, configurable port, and correct HTTP status codes. If the evaluator model judges any dimension unmet, it returns ok: false with a reason, and Claude continues working.

How is this different from /goal?

The /goal command from Project 1 is a built-in shortcut for a session-scoped prompt-based Stop hook. Writing your own prompt hook gives you full control: you define the exact rubric, choose which quality dimensions to evaluate, and persist the configuration across sessions in .claude/settings.json.

  • Save .claude/settings.json.
  • Switch back to Claude Code and type /hooks to open the hooks browser. You should see your prompt-based hook listed under the Stop event.

Don't see the prompt hook listed?

  • Make sure you saved .claude/settings.json after pasting the new content.
  • Check that the JSON is valid. A missing comma or unmatched bracket will silently break the configuration.
  • Confirm the old command hook entry is completely gone. You should only see the "type": "prompt" entry now.

Why isn't my prompt hook showing up in the /hooks browser?

Prompt Claude and observe the prompt hook

The prompt hook is now the only Stop evaluator. There is no deterministic gate running integration checks. Everything depends on the evaluator model's judgment of what Claude reports.

Before you prompt Claude, think about this: the prompt hook evaluates last_assistant_message, which is Claude's own summary of what it changed. What could go wrong if the only quality gate reads Claude's self-report instead of testing the actual code?

  • In Claude Code, enter this prompt:
Add a GET /stats endpoint that returns the total number of shortened URLs and the most recent short code created.

Watch Claude's output. After Claude finishes its response, the prompt hook fires. The evaluator model reads Claude's last_assistant_message and decides whether the work meets the four quality dimensions in your rubric.

What is the evaluator model actually reading?

The prompt hook receives the Stop event's JSON input, which includes last_assistant_message, transcript_path, and other session metadata. The evaluator model sees what Claude said it did. It does not start the server, send HTTP requests, or verify that the code actually works.

Compare this with the deterministic hook from Step 3, which started the actual server, sent real curl-style requests, and checked HTTP status codes. The deterministic hook tested runtime behavior. The prompt hook evaluates reported evidence.

  • Once Claude stops, confirm the /stats endpoint exists by starting the server and testing it:
node server.js &
curl http://localhost:3000/stats
kill %1

What does this command do?

The & starts the server in the background so you can run curl in the same terminal. After verifying the response, kill %1 stops the background server.

You should see a JSON response with a count of shortened URLs and the most recent short code. The prompt hook approved Claude's work based on what Claude reported, and in this case the code does work.

Server won't start or curl returns an error?

  • If you see an address-in-use error, another process is using port 3000. Kill it with kill %1 or find and stop the process on that port.
  • If /stats returns a 404, Claude may not have added the route. Check server.js for a GET /stats handler. If it is missing, prompt Claude again to add it.

Help me debug why my /stats endpoint isn't responding.

Identify the false PASS and false FAIL risks

The prompt hook approved Claude's work. But think about what it actually verified. It read Claude's description of the changes and judged whether that description sounded like it met the rubric.

This creates two risks that do not exist with deterministic hooks.

What is a false PASS?

A false PASS happens when the prompt hook returns ok: true but the code has real issues. If Claude says "I added input validation" but the validation has a subtle bug, the prompt hook has no way to catch it. It evaluates the claim, not the code.

What is a false FAIL?

A false FAIL happens when the prompt hook returns ok: false but the code is actually fine. If Claude is modest about solid work or does not mention a feature it implemented, the evaluator model may judge the rubric unmet. Claude then wastes time "fixing" something that already works.

  • Review the prompt hook evaluation you observed. Consider whether the evaluator model's judgment was based on evidence of what Claude reported or on verified runtime behavior.
  • Note any case where the prompt hook's decision felt like it could have gone either way. This is the ambiguity that deterministic hooks eliminate for the specific patterns they test.

Why not just use deterministic hooks for everything?

Deterministic hooks only catch what you script for. You cannot write a shell script that checks "is this code well-organized" or "does this error message make sense to a user." Prompt hooks handle the judgment calls. Deterministic hooks handle the verifiable facts.

Neither is sufficient alone. That is exactly why the next step combines both into a multi-signal gate.

✔️ Awesome, I've got everything!

Double check you have saved .claude/settings.json with the prompt hook configuration.

ⓧ I'd like to double check the full code

Here is the complete .claude/settings.json at this point in the project. Make sure yours matches exactly.

{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          {
            "type": "prompt",
            "prompt": "You are a code quality evaluator for a URL shortener API. Based on Claude's last response, evaluate whether the implementation meets these standards:\n\n1. Input validation: Does the code validate that required fields exist and have correct formats before processing?\n2. Error consistency: Do all error responses use the format { \"error\": \"description\" }?\n3. Configuration: Is the port configurable via an environment variable with a default?\n4. HTTP semantics: Does POST /shorten return 201 (not 200) on success and 400 on bad input?\n\nContext: $ARGUMENTS\n\nIf all standards appear met, respond {\"ok\": true}.\nIf any standard is not met, respond {\"ok\": false, \"reason\": \"list the specific issues\"}."
          }
        ]
      }
    ]
  }
}

You have seen the prompt hook in action and identified its fundamental limitation: it evaluates what Claude reports, not what the code actually does. In the next step, you will wire both the deterministic hook and the prompt hook together so that Claude must satisfy every evaluator before the loop ends.

Wire the Multi-Signal Gate

In Step 4, you tested the prompt-based Stop hook in isolation and discovered its core limitation. It evaluates what Claude reported doing, not the actual behavior of the running code.

A deterministic hook has the opposite gap: it catches exactly what you script for but nothing else. Combining both into a single gate gives you runtime verification from the command hook AND judgment-based evaluation from the prompt hook. In this step, you'll wire them together so Claude must satisfy every evaluator before the loop ends.

In this step, get ready to:
  • Add the command hook back alongside the prompt hook in .claude/settings.json.
  • Prompt Claude to add URL validation and watch both hooks fire in parallel.
  • Confirm that Claude stops only when both evaluators pass.
Add the command hook back to settings

Right now, .claude/settings.json only has the prompt hook you configured in Step 4. The command hook that runs quality-gate.sh is missing. You need both in the same hooks array so they fire in parallel on every Stop event.

  • Open .claude/settings.json in your editor.
  • Replace the entire file contents with this configuration that includes both hooks:
{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/quality-gate.sh"
          },
          {
            "type": "prompt",
            "prompt": "You are a code quality evaluator for a URL shortener API. Based on Claude's last response, evaluate whether the implementation meets these standards:\n\n1. Input validation: Does the code validate that required fields exist and have correct formats before processing?\n2. Error consistency: Do all error responses use the format { \"error\": \"description\" }?\n3. Configuration: Is the port configurable via an environment variable with a default?\n4. HTTP semantics: Does POST /shorten return 201 (not 200) on success and 400 on bad input?\n\nContext: $ARGUMENTS\n\nIf all standards appear met, respond {\"ok\": true}.\nIf any standard is not met, respond {\"ok\": false, \"reason\": \"list the specific issues\"}."
          }
        ]
      }
    ]
  }
}

What changed from the Step 4 version?

You added the command hook object back into the hooks array, right before the existing prompt hook. Both handlers sit inside the same array under the Stop event.

When Claude finishes a turn, all matching hooks run in parallel. The most restrictive answer wins. If either hook blocks, Claude continues working.

  • Save .claude/settings.json.
  • Switch back to Claude Code and type /hooks to open the hooks browser.
  • Under the Stop event, you should see two hooks listed: one command hook pointing to quality-gate.sh and one prompt hook.

Only seeing one hook?

Make sure both hook objects are inside the same hooks array, separated by a comma. A missing comma between the two objects is the most common cause.

Also check that your JSON is valid. A stray bracket or missing quote will prevent Claude Code from loading the hooks at all.

Help me debug my .claude/settings.json hook configuration

Prompt Claude and watch both hooks fire

With both evaluators active, you can give Claude a task that will truly test the gate. The URL shortener currently accepts any string as a URL. That is a quality gap the deterministic hook can verify with real requests AND the prompt hook can evaluate against the quality standards.

  • Before you send this prompt, think about what you expect to happen. Will Claude get it right on the first try, or will one of the hooks catch something? Which hook is more likely to block?
  • Type this prompt into Claude Code:
The /shorten endpoint currently accepts any string as a URL. Add validation so it only accepts valid http:// and https:// URLs. Return 400 with an error message for invalid URLs.
  • Press Enter and watch Claude work.

After Claude finishes its changes and tries to stop, both hooks fire in parallel. Watch the output carefully. You should see the deterministic hook running npm test and node quality-checks.js against the live server, and the prompt hook evaluating Claude's reported changes against the quality rubric.

What does "fire in parallel" mean here?

Claude Code runs all matching hooks for the same event at the same time, not one after another. The most restrictive answer wins. If the command hook exits with code 2 (blocking) but the prompt hook returns {"ok": true}, Claude keeps working because the command hook blocked.

Claude stops only when every evaluator passes. This is inherently an "all must pass" gate.

Observe the multi-signal gate in action

As Claude works through the URL validation task, pay close attention to the feedback from each hook. This is where you see the real power of combining both evaluator types.

  • Watch for a moment where the two hooks disagree. The deterministic hook verifies that validation actually works by sending real requests with invalid URLs. The prompt hook evaluates whether the implementation looks complete and consistent based on Claude's reported changes.
  • If Claude's URL validation has a subtle bug (for example, it accepts http:// with no domain), the deterministic hook catches it by sending that exact request, even if the prompt hook said "looks good."
  • Note the final sequence: which hook passed first, which blocked, and how many iterations Claude needed before both went green.

Why is this stronger than either hook alone?

The deterministic hook guarantees runtime behavior. If POST /shorten with not-a-valid-url does not return 400, the hook blocks. No false PASS is possible for that specific check.

The prompt hook catches quality dimensions that are harder to script: consistent error formatting, proper HTTP semantics, configurable ports. Together, each hook covers the other's blind spot.

Once both hooks pass and Claude stops, verify the URL validation is actually working by running a quick test from your terminal.

  • Start the server by running:
node server.js
  • Open a second terminal tab or window. Before you run the next command, predict: will the server accept a plain string like not-a-url or reject it?
  • Test with an invalid URL by running:
curl -s -X POST http://localhost:3000/shorten -H "Content-Type: application/json" -d '{"url": "not-a-url"}' | jq

You should see a 400 response with an error message. The server now rejects URLs that do not start with http:// or https://.

  • Now test with a valid URL by running:
curl -s -X POST http://localhost:3000/shorten -H "Content-Type: application/json" -d '{"url": "https://example.com"}' | jq

You should see a successful response with a short code. Valid URLs pass through. Invalid ones get rejected.

  • Stop the server with Ctrl+C in the terminal where it is running.

Server accepting invalid URLs?

If the server returns a 200 or 201 for not-a-url, the multi-signal gate may not have fully passed. Check Claude Code's output for any remaining hook failures.

You can re-prompt Claude with: "The quality gate is still failing. Fix the URL validation so only http:// and https:// URLs are accepted."

Help me debug why my multi-signal gate is not catching invalid URLs

✔️ Awesome, I've got everything!

Great. Double check you have saved .claude/settings.json with both hooks configured.

ⓧ I'd like to double check the full code

Here is the complete .claude/settings.json file. This is the only file you manually edited in this step.

{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/quality-gate.sh"
          },
          {
            "type": "prompt",
            "prompt": "You are a code quality evaluator for a URL shortener API. Based on Claude's last response, evaluate whether the implementation meets these standards:\n\n1. Input validation: Does the code validate that required fields exist and have correct formats before processing?\n2. Error consistency: Do all error responses use the format { \"error\": \"description\" }?\n3. Configuration: Is the port configurable via an environment variable with a default?\n4. HTTP semantics: Does POST /shorten return 201 (not 200) on success and 400 on bad input?\n\nContext: $ARGUMENTS\n\nIf all standards appear met, respond {\"ok\": true}.\nIf any standard is not met, respond {\"ok\": false, \"reason\": \"list the specific issues\"}."
          }
        ]
      }
    ]
  }
}

You now have a working multi-signal evaluation pipeline. Two evaluators with different strengths fire together, and Claude cannot stop until both are satisfied. Next up: the Secret Mission, where you will deliberately break this gate to see its most dangerous failure mode.

Secret mission

Expose and Catch a False PASS

You have been building evaluators that catch quality issues. Now you will deliberately create the most dangerous failure mode: a test suite that looks comprehensive but misses a meaningful bug. You will watch the entire quality gate pass with green lights while a real error goes undetected. Then you will strengthen the evaluator to catch the gap.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing cloud costs or active services to shut down. The Anthropic API credits you used during Claude Code sessions have already been consumed.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • The url-shortener project folder on your Desktop (source files, hook scripts, and node_modules).
  • jq command-line tool (installed via Homebrew, if you did not already have it).

Keep everything running

No action needed. Choose this if you are still actively building on your Stop hook pipeline or plan to revisit the project soon.

  • Your url-shortener folder stays on your Desktop, ready to open in Claude Code whenever you want to experiment with new hooks.
  • jq remains installed on your system. It is a lightweight utility used by many CLI tools, so keeping it is generally useful.

Pause - I'll come back to this later

Close any open terminals and editor windows to free up memory. Your project files stay on disk and are ready to pick up again later.

  • Close any terminal tabs that have the server running. If the server is active, press Ctrl+C to stop it first.
  • Close your code editor if you have the project open.
  • To resume later, open your terminal, run cd ~/Desktop/url-shortener, and launch Claude Code.

Delete - I don't want to use this again

Remove all project files and optionally uninstall tools you no longer need.

  • Delete the project folder by running this command in your terminal:
rm -rf ~/Desktop/url-shortener
  • Confirm the folder is gone by running:
ls ~/Desktop/url-shortener

You should see a "No such file or directory" message.

  • If you installed jq specifically for this project and no longer need it, uninstall it by running:
brew uninstall jq

Nice Work!

Nice Work!

You just built a multi-signal evaluation pipeline that catches quality issues your test suite misses. Your Claude Code projects now have hard quality gates that force real fixes before Claude can stop working.

You've learned how to:

  • Built a deterministic Stop hook that starts the actual server, sends real HTTP requests with malformed input, and blocks Claude from stopping until every integration check passes.
  • Built a prompt-based Stop hook that evaluates code quality by assessing available model and session evidence, and learned its key limitation: it evaluates what Claude reported doing, not the actual running code.
  • Wired both hooks into a multi-signal gate where Claude must satisfy every evaluator before the loop ends, and experienced the key loop firsthand: tests PASS, quality gate FAILS, Claude iterates, final PASS.
  • Secret Mission: Deliberately created a false PASS by crafting a misleading test suite, observed the entire gate pass while a real bug remained, then built an evaluator that caught the gap.

Ready to quiz yourself?