Stop an AI Agent Loop

Build a TypeScript supervisor that detects and halts looping AI agents.

Introduction

30 Second Summary

When a task keeps repeating the same failed move, activity can look like progress. The longer it runs unattended, the more time and money it burns.

In this project, you will build a TypeScript supervisor that checks whether an AI agent is progressing, looping, or stuck. It uses OpenRouter's Decisions API with ScaleDown Classify to choose between continuing the run, nudging the agent, or halting it.

What You'll Build

Your terminal shows a forced loop stopping as soon as the supervisor detects repeated tool calls.

By the end of this project, you'll have:

  • A transcript supervisor that returns one clear action from continue, nudge, or halt.
  • A three-case test that shows healthy, looping, and stuck transcripts producing different actions.
  • A live safeguard that halts a looping run after repeated tool calls.
  • Secret Mission: An optional challenge to push your loop-supervisor skills further.

What do I need before starting?

You need an OpenRouter account with credits and an API key. The project makes only a few paid classifier calls, so a credit limit keeps the experiment contained.

You also need Node.js 20+ or Bun to run the TypeScript project.

Before We Start

A loop supervisor cannot judge an agent until it has a funded model connection. Your OpenRouter API key supplies that connection while its credit limit caps the risk.

Your local Node.js runtime gives the supervisor somewhere to run. A TypeScript project with Zod gives later classifier responses a typed boundary.

In this step, get ready to:
  • Fund an OpenRouter account and create a limited API key.
  • Set up a Node.js TypeScript project with Zod.
  • Load OPENROUTER_API_KEY from local configuration and run a TypeScript entry point.
Create your funded OpenRouter key

OpenRouter deducts each model request from your credit balance. A key-level credit limit creates a separate spending boundary for this project.

Adding credits requires a payment card. You choose the available checkout amount before any charge is confirmed.

  • Open the OpenRouter website in your browser.
  • Select an available sign-in method.
  • Complete the sign-in flow for that method.
  • Open the Credits page.
  • Choose an available credit amount that fits your budget.
  • Complete the checkout confirmation.

How credits and key limits work

Credits fund requests across your OpenRouter account. The API key limit controls how much this specific key can spend.

This project's test calls should stay well below five dollars. A 5 USD key limit leaves room for the walkthrough while capping the key.

  • Open the API Keys page.
  • Create a key named loop-supervisor.
  • Set its credit limit to 5 USD.

Protect the one-time key

OpenRouter shows the plaintext key only when it is created. Never paste it into project notes or messages.

Keep the key out of screenshots. You store it in the project's local environment configuration later in this step.

  • Copy the new key from the one-time key view.
  • Store the key temporarily in a trusted password manager.
  • Close the one-time key view.

The sensitive part is handled. Your funded key now has its own spending boundary.

Prepare the TypeScript project

This project needs Node.js 20 or newer. The environment-file option used below requires version 20.6 or newer.

  • Check your installed Node.js version by running this command:
node --version

✔️ I see version 20.6 or higher

Your Node.js version supports the runtime features used in this project. Continue with the project folder setup.

ⓧ I see an older version

Your installed version is below 20.6. Install the current LTS release to get a supported runtime.

  • Open the official Node.js download page.
  • Select the LTS installer for your operating system.
  • Run the downloaded installer with its default options.
  • Press Cmd+Space on macOS or the Windows key on Windows.
  • Type Terminal into the search bar.
  • Press Enter to start a new terminal session.
  • Check the upgraded version by running this command:
node --version

You should see version 20.6 or higher.

ⓧ Command not found

Node.js is missing from your command path. The official LTS installer adds the runtime and npm.

  • Open the official Node.js download page.
  • Select the LTS installer for your operating system.
  • Run the downloaded installer with its default options.
  • Press Cmd+Space on macOS or the Windows key on Windows.
  • Type Terminal into the search bar.
  • Press Enter to start a new terminal session.
  • Confirm the installation by running this command:
node --version

You should see version 20.6 or higher.

Still seeing the old version?

Close every terminal window after the installer finishes. A new terminal session reloads the command path.

Confirm that you downloaded the current LTS installer from the official Node.js page.

Help me fix my Node.js version check.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create the agent-loop-supervisor folder and move into it by running these commands:
mkdir agent-loop-supervisor
cd agent-loop-supervisor
pwd

What should the path show?

The final line ends with Desktop/agent-loop-supervisor. This confirms that later files land in the intended project folder.

  • Create the default package.json file by running this command:
npm init --yes

The terminal prints a path ending in agent-loop-supervisor/package.json.

  • Install Zod 4 and the TypeScript runtime dependencies by running these commands:
npm install zod@4
npm install -D tsx @types/node

What do these packages provide?

  • The zod@4 package validates data against schemas.
  • The tsx package runs TypeScript files without a separate compile command.
  • The @types/node package provides types for Node.js globals.
  • Confirm the installed package folders by running this command:
ls node_modules

You should see entries for zod and tsx. You should also see an @types entry.

Did the dependency installation fail?

Confirm that the terminal path ends with agent-loop-supervisor. The install commands need the package.json file in the current folder.

Check your internet connection if the package download stops before completion.

Help me diagnose my npm installation.

Load the key and run TypeScript

An environment variable keeps credentials outside application code. A local .env file provides OPENROUTER_API_KEY only when you load that file.

  • Create .env inside agent-loop-supervisor by adding this content in a plain-text editor:
OPENROUTER_API_KEY=your-openrouter-api-key-here

What does this configuration do?

The OPENROUTER_API_KEY name matches the environment variable expected by OpenRouter integrations. The value stays in local configuration instead of application code.

  • Replace your-openrouter-api-key-here with the key stored in your password manager.
  • Save .env.

Keep .env out of source control

The .env file remains local and uncommitted in this project. A future Git repository must ignore it before its first commit.

Never upload or share this file. Rotate the OpenRouter key immediately if it becomes exposed.

  • Check that Node.js can load the key without printing it by running this command:
node --env-file=.env --eval "console.log(Boolean(process.env.OPENROUTER_API_KEY))"

What should you see?

The terminal prints true. This proves that .env provides a non-empty value without revealing the key.

Did the check print false?

Confirm that .env sits directly inside agent-loop-supervisor. Check that the variable name is exactly OPENROUTER_API_KEY.

Confirm that the value appears after the equals sign on the same line.

Help me load my OpenRouter key safely.

  • Create script.ts inside agent-loop-supervisor by adding this code in your plain-text editor:
import { z } from "zod";

// Validate that local configuration contains a non-empty API key.
const apiKey = z.string().min(1).safeParse(
  process.env.OPENROUTER_API_KEY,
);

// Print readiness signals without exposing the credential.
console.log({
  zodReady: true,
  apiKeyReady: apiKey.success,
});

What does this code check?

  • The z import proves that the project can load Zod.
  • The safeParse() call checks for a non-empty key without printing its value.
  • The readiness object gives you two safe signals for the runtime and local configuration.
  • Save script.ts.

Before you run the final check, do you expect both readiness values to be true?

  • Run the TypeScript entry point with local environment configuration by running this command:
npx tsx --env-file=.env ./script.ts

What should the result show?

The terminal prints an object containing zodReady: true and apiKeyReady: true. Your TypeScript runtime can now load Zod and read the local key.

Is one readiness value false?

A false API key value points to the .env file or its variable name. A package error points to the dependency installation.

Run the command from inside agent-loop-supervisor so it can find both files.

Help me debug the TypeScript setup check.

✔️ Awesome, I've got everything!

Your TypeScript entry point is saved. Both readiness checks now pass without exposing your credential.

ⓧ I'd like to double check the full code

import { z } from "zod";

// Validate that local configuration contains a non-empty API key.
const apiKey = z.string().min(1).safeParse(
  process.env.OPENROUTER_API_KEY,
);

// Print readiness signals without exposing the credential.
console.log({
  zodReady: true,
  apiKeyReady: apiKey.success,
});

Setup checkpoint

  • Confirm your signed-in OpenRouter dashboard shows available credits.
  • Confirm the loop-supervisor API key has a 5 USD credit limit.
  • Confirm package.json lists the installed project dependencies.
  • Confirm the terminal shows both readiness values as true.
  • Keep .env closed before capturing your result.
  • Keep the final terminal output visible without showing the API key.

Your funded connection and local TypeScript runtime are ready. Next, you send an agent transcript to the Decisions API and inspect its typed classification response.

Classify an Agent Transcript

Your OpenRouter account is funded. Your TypeScript project can run.

The supervisor now needs measurable evidence from an agent transcript. The Decisions API uses three questions to return labels with probabilities.

In this step, get ready to:
  • Define the supervisor's three classification questions.
  • Send a looping sample transcript to ScaleDown Classify.
  • Confirm the typed JSON decision response.
Define the classification inputs

The request needs complementary signals about the same transcript. Together they distinguish a healthy run from repeated behavior or a blocked run.

  • Create src/classifyTranscript.ts inside the existing TypeScript project's src folder with your code editor.
  • Add the classification inputs by pasting this code:
const DECISIONS_MODEL = 'scaledown/classify-1';

// Ask for three complementary signals about the run.
const QUESTIONS = {
  run_state: {
    type: 'choice',
    instructions: 'Which single state best describes this agent run?',
    criteria: {
      healthy: 'The agent is making new progress and moving closer to the goal.',
      stuck: 'The agent is blocked by an error or missing information it cannot resolve.',
      looping: 'The agent is repeating the same tool calls or reasoning with the same outcome.',
    },
  },
  is_repeating: {
    type: 'noul',
    instructions: 'Does the agent repeat a tool call with the same arguments and get the same result?',
  },
  progress: {
    type: 'score',
    instructions: "How much progress is the agent making toward the user's task?",
    criteria: ['No progress', 'Some progress', 'Clear progress toward completion'],
  },
} as const;

// Build a transcript that repeats the same unsuccessful lookup.
const REPEATED_LOOKUP =
  '[assistant]\nI will look up the order.\n[tool_call lookup_order] {"order_id":1042}\n[tool_result]\nOrder not found.';
const SAMPLE_TRANSCRIPT = ['[user]\nFind order 1042.', ...Array.from({ length: 8 }, () => REPEATED_LOOKUP)].join('\n\n');

What Do These Inputs Measure?

  • The choice question selects one run-state label. It also returns a probability for every label.
  • The noul question returns the probability that the agent is repeating itself.
  • The score question places the agent's progress on an ordered scale.
  • The sample repeats the same failed lookup eight times. This gives the classifier a clear loop to detect.
  • Save src/classifyTranscript.ts.
  • Confirm your editor shows no TypeScript syntax errors in the question object or sample transcript.

Seeing TypeScript Syntax Errors?

Check that each question closes with the same braces shown above. Confirm that the transcript strings keep their surrounding quotes.

Ask for help with the input definitions.

Send the classification request

The request places the transcript in state.text. It sends the question object beside the transcript for one typed decision.

  • Add the request function below SAMPLE_TRANSCRIPT by pasting this code:
async function classifyTranscript() {
  // Stop before the paid request if the local credential is unavailable.
  const apiKey = process.env.OPENROUTER_API_KEY;
  if (!apiKey) {
    throw new Error('OPENROUTER_API_KEY is not set.');
  }

  // Submit the transcript and questions for one typed decision.
  const response = await fetch('https://openrouter.ai/api/alpha/decisions', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${apiKey}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      model: DECISIONS_MODEL,
      state: { text: SAMPLE_TRANSCRIPT },
      questions: QUESTIONS,
    }),
    signal: AbortSignal.timeout(60_000),
  });

  if (!response.ok) {
    throw new Error(`decisions ${response.status}: ${await response.text()}`);
  }

  // Print the untouched response so its complete shape is visible.
  const decision = await response.json();
  console.log(JSON.stringify(decision, null, 2));
}

await classifyTranscript();

What Does This Code Do?

  • The environment lookup keeps OPENROUTER_API_KEY outside the source file.
  • The request sends the model name with the transcript and questions.
  • The response check surfaces an unsuccessful request before the script tries to read its JSON.
  • The final log preserves the raw decision response for inspection.
  • Save src/classifyTranscript.ts.

The next run uses one paid classification request from the credits you already added.

Before you run it, which run-state label do you expect the repeated lookup to receive?

  • Run src/classifyTranscript.ts with the TypeScript entry-point command you verified in Step 1.

What Should You See?

  • The top-level id identifies the decision.
  • The answers.run_state.choice value contains the selected run-state label.
  • The answers.run_state.probabilities object contains values for healthy, stuck, and looping.
  • The repeated lookup should make looping the strongest run-state probability.
  • The is_repeating and progress answers contain their own numeric results.

Request Did Not Return a Decision?

If the script reports OPENROUTER_API_KEY is not set. return to the terminal session where your environment configuration is loaded. Keep the key itself out of screenshots and source files.

If the request returns another error, check that your OpenRouter account still has available credits. Confirm that the environment variable belongs to the same terminal session running the script.

Help me diagnose the failed Decisions API request.

✔️ Awesome, I've got everything!

Your classifier now sends a looping transcript to OpenRouter. The terminal shows the raw decision ID with labelled answers and probabilities.

ⓧ I'd like to double check the full code

const DECISIONS_MODEL = 'scaledown/classify-1';

// Ask for three complementary signals about the run.
const QUESTIONS = {
  run_state: {
    type: 'choice',
    instructions: 'Which single state best describes this agent run?',
    criteria: {
      healthy: 'The agent is making new progress and moving closer to the goal.',
      stuck: 'The agent is blocked by an error or missing information it cannot resolve.',
      looping: 'The agent is repeating the same tool calls or reasoning with the same outcome.',
    },
  },
  is_repeating: {
    type: 'noul',
    instructions: 'Does the agent repeat a tool call with the same arguments and get the same result?',
  },
  progress: {
    type: 'score',
    instructions: "How much progress is the agent making toward the user's task?",
    criteria: ['No progress', 'Some progress', 'Clear progress toward completion'],
  },
} as const;

// Build a transcript that repeats the same unsuccessful lookup.
const REPEATED_LOOKUP =
  '[assistant]\nI will look up the order.\n[tool_call lookup_order] {"order_id":1042}\n[tool_result]\nOrder not found.';
const SAMPLE_TRANSCRIPT = ['[user]\nFind order 1042.', ...Array.from({ length: 8 }, () => REPEATED_LOOKUP)].join('\n\n');

async function classifyTranscript() {
  // Stop before the paid request if the local credential is unavailable.
  const apiKey = process.env.OPENROUTER_API_KEY;
  if (!apiKey) {
    throw new Error('OPENROUTER_API_KEY is not set.');
  }

  // Submit the transcript and questions for one typed decision.
  const response = await fetch('https://openrouter.ai/api/alpha/decisions', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${apiKey}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      model: DECISIONS_MODEL,
      state: { text: SAMPLE_TRANSCRIPT },
      questions: QUESTIONS,
    }),
    signal: AbortSignal.timeout(60_000),
  });

  if (!response.ok) {
    throw new Error(`decisions ${response.status}: ${await response.text()}`);
  }

  // Print the untouched response so its complete shape is visible.
  const decision = await response.json();
  console.log(JSON.stringify(decision, null, 2));
}

await classifyTranscript();

Your classifier now turns a looping transcript into typed evidence. Next, you'll validate that response with zod. Then you'll map the probabilities to a supervisor action.

Choose a Supervisor Action

Your Decisions API classifier now returns a decision ID plus probabilities for each signal. That raw response is useful for inspection.

An agent loop needs one small decision it can act on. This step adds a Zod boundary around the JSON response before mapping its probabilities to a typed TypeScript action.

In this step, get ready to:
  • Validate the classifier response with schemas.
  • Configure thresholds for the three supervisor actions.
  • Print one clear action for the sample transcript.
Validate the decision response

A schema creates a boundary between an external response and your supervisor logic. Invalid labels or out-of-range probabilities stop at that boundary.

  • Return to the TypeScript project in your editor from earlier.
  • Create src/supervisor.ts inside the existing src folder.
  • Add the schemas for the three classifier answers by copying this code into src/supervisor.ts:
import { z } from 'zod';

const RUN_STATES = ['progressing', 'stuck', 'looping'] as const;

// Describe answers from the three classification questions.
const AnswersSchema = z.object({
  run_state: z.object({
    type: z.literal('choice'),
    choice: z.enum(RUN_STATES),
    probabilities: z.record(z.enum(RUN_STATES), z.number().min(0).max(1)),
  }),
  is_repeating: z.object({
    type: z.literal('noul'),
    noul: z.number().min(0).max(1),
  }),
  completion: z.object({
    type: z.literal('score'),
    score: z.number().min(0).max(2),
  }),
});

What does this schema validate?

  • The run_state answer must use one of the three labels defined in RUN_STATES.
  • Every run-state probability must stay between 0 and 1.
  • The is_repeating answer must contain a noul probability between 0 and 1.
  • The completion score must stay on the configured scale from 0 to 2.
  • Save src/supervisor.ts.
  • Check the TypeScript diagnostics for AnswersSchema in your editor.

Good progress. You should see no type or syntax errors around the answer schema.

Seeing schema errors?

Confirm that every opening brace has a matching closing brace. Check that zod remains installed in the project from the earlier setup.

Help me fix the answer schema in my supervisor module.

The answer schema checks the classifier signals. A second schema validates the response envelope that carries those signals.

  • Extend src/supervisor.ts by adding this response schema below AnswersSchema:
// Validate the response envelope before using its answers.
export const DecisionsResponseSchema = z.object({
  id: z.string(),
  model: z.string(),
  answers: AnswersSchema,
  usage: z.object({
    input_tokens: z.number(),
    output_tokens: z.number(),
    cost: z.number(),
  }),
});

What does the response schema protect?

  • The id field identifies the classification decision.
  • The answers field must satisfy the answer schema you just created.
  • The usage field records the token counts plus the request cost.
  • Save src/supervisor.ts.
  • Check the TypeScript diagnostics for DecisionsResponseSchema in your editor.

You should see the response schema resolve without type or syntax errors.

Is the response schema underlined?

Place the response schema below AnswersSchema in the same file. Confirm that the field names match the raw response printed by your classifier.

Help me diagnose my Decisions response schema.

Map probabilities to an action

Thresholds turn uncertain model probabilities into predictable control flow. A halt requires two strong loop signals to agree.

A single loop signal or stuck signal produces a nudge. Every response below those thresholds continues.

  • Add the action type plus its threshold logic below DecisionsResponseSchema in src/supervisor.ts:
// Name the only actions an agent loop can receive.
export type SupervisorAction = 'continue' | 'nudge' | 'halt';

// Keep halting stricter than nudging to reduce false stops.
const HALT_LOOPING = 0.8;
const HALT_REPEATING = 0.8;
const NUDGE_LOOPING = 0.5;
const NUDGE_STUCK = 0.4;

export function chooseSupervisorAction(response: unknown): SupervisorAction {
  // Validate untrusted response data before reading its probabilities.
  const decision = DecisionsResponseSchema.parse(response);
  const looping = decision.answers.run_state.probabilities.looping;
  const stuck = decision.answers.run_state.probabilities.stuck;
  const repeating = decision.answers.is_repeating.noul;

  // Halt only when both loop signals are strong.
  if (
    looping >= HALT_LOOPING && repeating >= HALT_REPEATING
  ) {
    return 'halt';
  }

  // Use one concerning signal to redirect the agent.
  if (looping >= NUDGE_LOOPING || stuck >= NUDGE_STUCK) {
    return 'nudge';
  }

  return 'continue';
}

How are the thresholds applied?

  • The supervisor returns halt when both looping and repetition reach 0.8.
  • The supervisor returns nudge when looping reaches 0.5.
  • The supervisor also returns nudge when stuck reaches 0.4.
  • The supervisor returns continue when no action threshold is reached.
  • Save src/supervisor.ts.
  • Check the TypeScript diagnostics for chooseSupervisorAction in your editor.

You should see no type errors. The function can only return a value allowed by SupervisorAction.

Seeing probability property errors?

Confirm that DecisionsResponseSchema.parse(response) runs before the function reads any probabilities. Check each property name against the schemas above.

Help me trace the probability types in my supervisor function.

Print the sample action

Your classifier already holds the parsed response used by its raw JSON log. Passing that same value into chooseSupervisorAction keeps classification separate from control logic.

  • Return to src/classifyTranscript.ts in your editor.
  • Add an import for chooseSupervisorAction from the local src/supervisor.ts module using the import style already used by your project.

Your editor should resolve chooseSupervisorAction without showing a missing-module diagnostic.

Is the supervisor import unresolved?

Confirm that both TypeScript files sit directly inside the src folder. Match the local file-extension style used by the other imports in your project.

Help me resolve the local supervisor import.

  • Create an action value by passing the existing parsed decision response to chooseSupervisorAction.
  • Check the inline TypeScript diagnostics for the new action value.

You should see action inferred as SupervisorAction.

  • Print the returned action with the label Supervisor action: immediately after the existing raw JSON log.
  • Save src/classifyTranscript.ts.

Before you rerun the classifier, which action do you expect from the sample transcript's probabilities?

  • Rerun src/classifyTranscript.ts with the same project command you used earlier.

You should first see the raw decision response. Beneath it, you should see Supervisor action: followed by exactly one of continue, nudge, or halt.

No supervisor action printed?

Confirm that the action chooser receives the parsed response object used by the raw JSON log. Place the action log after that response has been created.

Help me debug why my classifier does not print a supervisor action.

  • Keep any environment configuration or API key value outside the screenshot.

That is the control boundary working. Your classifier now produces one typed instruction that an agent loop can follow.

✔️ Awesome, I've got everything!

Your schemas and thresholds are saved. Your sample transcript now produces one clear supervisor action.

ⓧ I'd like to double check the full code

import { z } from 'zod';

const RUN_STATES = ['progressing', 'stuck', 'looping'] as const;

// Describe answers from the three classification questions.
const AnswersSchema = z.object({
  run_state: z.object({
    type: z.literal('choice'),
    choice: z.enum(RUN_STATES),
    probabilities: z.record(z.enum(RUN_STATES), z.number().min(0).max(1)),
  }),
  is_repeating: z.object({
    type: z.literal('noul'),
    noul: z.number().min(0).max(1),
  }),
  completion: z.object({
    type: z.literal('score'),
    score: z.number().min(0).max(2),
  }),
});

// Validate the response envelope before using its answers.
export const DecisionsResponseSchema = z.object({
  id: z.string(),
  model: z.string(),
  answers: AnswersSchema,
  usage: z.object({
    input_tokens: z.number(),
    output_tokens: z.number(),
    cost: z.number(),
  }),
});

// Name the only actions an agent loop can receive.
export type SupervisorAction = 'continue' | 'nudge' | 'halt';

// Keep halting stricter than nudging to reduce false stops.
const HALT_LOOPING = 0.8;
const HALT_REPEATING = 0.8;
const NUDGE_LOOPING = 0.5;
const NUDGE_STUCK = 0.4;

export function chooseSupervisorAction(response: unknown): SupervisorAction {
  // Validate untrusted response data before reading its probabilities.
  const decision = DecisionsResponseSchema.parse(response);
  const looping = decision.answers.run_state.probabilities.looping;
  const stuck = decision.answers.run_state.probabilities.stuck;
  const repeating = decision.answers.is_repeating.noul;

  // Halt only when both loop signals are strong.
  if (
    looping >= HALT_LOOPING && repeating >= HALT_REPEATING
  ) {
    return 'halt';
  }

  // Use one concerning signal to redirect the agent.
  if (looping >= NUDGE_LOOPING || stuck >= NUDGE_STUCK) {
    return 'nudge';
  }

  return 'continue';
}

Your supervisor can now turn one classifier response into an action. Next, you'll test those decisions against healthy, looping, and stuck transcripts.

Test Healthy and Failing Runs

Your supervisor can now turn one classifier response into an action. A single transcript cannot prove that its thresholds separate productive work from repetition or a genuine block.

A small TypeScript test runner gives the supervisor the same three patterns every time. It sends each fixture through your existing OpenRouter classifier so you can compare the returned actions.

In this step, get ready to:
  • Prepare a three-state transcript matrix.
  • Call the supervisor once per transcript.
  • Confirm that the classifier returns three distinct actions.
Compare the three run patterns

How the scenarios differ

A healthy transcript shows changing actions plus new results that move the task forward.

A looping transcript repeats the same tool call with the same failed result.

A stuck transcript tries different approaches before reporting that every path remains blocked.

  • Identify the healthy pattern by its varied work plus successful test result.
  • Identify the looping pattern by its repeated command plus unchanged failure.
  • Identify the stuck pattern by its varied failures plus final blocked state.
Run the transcript matrix

The test runner keeps its fixtures separate from your classifier logic. It imports supervise() from src/supervisor.ts so every scenario uses the thresholds you already configured.

  • Create src/testSupervisor.ts with your code editor's file-creation control.

You should see testSupervisor.ts listed inside the existing src folder.

  • Paste the following test runner into src/testSupervisor.ts:
import { supervise } from './supervisor.js';

// This fixture shows varied work reaching a successful result.
const healthyTranscript = '[user]\nMake the failing test in packages/billing/invoice.test.ts pass and open a PR.\n\n[assistant]\nReading the failing test first.\n[tool_result]\nexpect(renderDueDate(invoice)).toBe("2026-09-30") // line 41\n\n[assistant]\nRefactoring renderDueDate to take a clock parameter.\n[tool_result]\nedited packages/billing/invoice.ts (+3 -2)\n\n[assistant]\nRunning the billing tests.\n[tool_result]\n42 pass, 0 fail\n\n[assistant]\nPushing and opening the PR.\n[tool_result]\nhttps://github.com/example/repo/pull/4821';

// This fixture repeats one failing tool call enough times to form a loop.
const loopingStep = '[assistant]\nRunning the test suite again.\n[tool_call run_command] {"cmd":"bun test packages/billing/invoice.test.ts"}\n[tool_result]\nFAIL renders due date: expected "2026-09-30" received "2026-10-12"';
const loopingTranscript = ['[user]\nMake the failing test in packages/billing/invoice.test.ts pass and open a PR.', ...Array.from({ length: 60 }, () => loopingStep)].join('\n\n');

// This fixture tries several routes before reporting an unresolved block.
const stuckTranscript = '[user]\nMake the failing test in packages/billing/invoice.test.ts pass and open a PR.\n\n[assistant]\nReading the fixture the test loads.\n[tool_result]\nEACCES: permission denied, open fixtures/invoice.json\n\n[assistant]\nTrying to relax the permissions.\n[tool_result]\nchmod: changing permissions of fixtures/invoice.json: Operation not permitted\n\n[assistant]\nTrying sudo.\n[tool_result]\nsudo: a terminal is required to read the password\n\n[assistant]\nCopying it somewhere readable.\n[tool_result]\ncp: cannot open fixtures/invoice.json for reading: Permission denied\n\n[assistant]\nFetching the missing object.\n[tool_result]\nfatal: unable to access remote: Could not resolve host: github.com\n\n[assistant]\nI have no way to read fixtures/invoice.json from this environment. Every route to it is blocked by permissions and the network is unavailable.';

// Each case records the label printed beside the returned supervisor action.
const testCases = [
  { name: 'healthy', transcript: healthyTranscript },
  { name: 'looping', transcript: loopingTranscript },
  { name: 'stuck', transcript: stuckTranscript },
] as const;

for (const { name, transcript } of testCases) {
  const action = await supervise(transcript);
  console.log(`${name}: ${action}`);
}

What does this test runner do?

  • The healthyTranscript fixture shows varied actions reaching a passing test plus a completed pull request.
  • The loopingTranscript fixture uses Array.from() to repeat the same failed command sixty times.
  • The stuckTranscript fixture records several failed recovery attempts plus a final blocked statement.
  • The final loop passes each transcript to supervise() before printing its returned action.
  • Save src/testSupervisor.ts.

This run sends three paid classifier requests against the OpenRouter credits you already added. You remain in control because the runner makes one request per fixture.

Before you run the matrix, predict which fixture will produce the strictest action.

  • Execute src/testSupervisor.ts with the same TypeScript runner you used for src/classifyTranscript.ts.

What should you see?

  • The healthy summary line reads healthy: continue.
  • The looping summary line reads looping: halt.
  • The stuck summary line reads stuck: nudge.

That is the key proof from this step. Your supervisor now distinguishes productive work from repetition plus a recoverable block.

Seeing a missing import or different action?

Confirm that src/supervisor.ts exports supervise. Check that src/testSupervisor.ts sits beside it inside src.

A different action can come from a changed fixture or threshold. Compare the transcript strings with the full-file reference below.

Help me diagnose my transcript supervisor test results.

✔️ Awesome, I've got everything!

Your test runner now demonstrates separate actions for healthy, looping, and stuck agent transcripts.

ⓧ I'd like to double check the full code

import { supervise } from './supervisor.js';

// This fixture shows varied work reaching a successful result.
const healthyTranscript = '[user]\nMake the failing test in packages/billing/invoice.test.ts pass and open a PR.\n\n[assistant]\nReading the failing test first.\n[tool_result]\nexpect(renderDueDate(invoice)).toBe("2026-09-30") // line 41\n\n[assistant]\nRefactoring renderDueDate to take a clock parameter.\n[tool_result]\nedited packages/billing/invoice.ts (+3 -2)\n\n[assistant]\nRunning the billing tests.\n[tool_result]\n42 pass, 0 fail\n\n[assistant]\nPushing and opening the PR.\n[tool_result]\nhttps://github.com/example/repo/pull/4821';

// This fixture repeats one failing tool call enough times to form a loop.
const loopingStep = '[assistant]\nRunning the test suite again.\n[tool_call run_command] {"cmd":"bun test packages/billing/invoice.test.ts"}\n[tool_result]\nFAIL renders due date: expected "2026-09-30" received "2026-10-12"';
const loopingTranscript = ['[user]\nMake the failing test in packages/billing/invoice.test.ts pass and open a PR.', ...Array.from({ length: 60 }, () => loopingStep)].join('\n\n');

// This fixture tries several routes before reporting an unresolved block.
const stuckTranscript = '[user]\nMake the failing test in packages/billing/invoice.test.ts pass and open a PR.\n\n[assistant]\nReading the fixture the test loads.\n[tool_result]\nEACCES: permission denied, open fixtures/invoice.json\n\n[assistant]\nTrying to relax the permissions.\n[tool_result]\nchmod: changing permissions of fixtures/invoice.json: Operation not permitted\n\n[assistant]\nTrying sudo.\n[tool_result]\nsudo: a terminal is required to read the password\n\n[assistant]\nCopying it somewhere readable.\n[tool_result]\ncp: cannot open fixtures/invoice.json for reading: Permission denied\n\n[assistant]\nFetching the missing object.\n[tool_result]\nfatal: unable to access remote: Could not resolve host: github.com\n\n[assistant]\nI have no way to read fixtures/invoice.json from this environment. Every route to it is blocked by permissions and the network is unavailable.';

// Each case records the label printed beside the returned supervisor action.
const testCases = [
  { name: 'healthy', transcript: healthyTranscript },
  { name: 'looping', transcript: loopingTranscript },
  { name: 'stuck', transcript: stuckTranscript },
] as const;

for (const { name, transcript } of testCases) {
  const action = await supervise(transcript);
  console.log(`${name}: ${action}`);
}

Your supervisor has passed its three-state test. Next, you will place that check inside an agent loop so a repeated run can stop itself.

Stop a Looping Agent

Your OpenRouter classifier can now assign each transcript a run state. The missing safeguard is a live agent loop that obeys its decision.

A repeated tool call can keep consuming steps after progress stops. A scheduled supervisor check gives the loop a clear stopping point.

In this step, get ready to:
  • Accumulate a transcript from a forced repeating tool call.
  • Run the supervisor every few agent steps.
  • Stop the forced run when the supervisor returns halt.
Build the forced agent loop

The loop needs a transcript that grows one step at a time. An identical failed tool call gives the classifier a clear repetition pattern.

  • In your local TypeScript project, create agentLoop.ts inside the src folder.
  • Paste the code below into src/agentLoop.ts.
import { supervise } from './supervisor.js';

const CHECK_EVERY = 5;
const MAX_STEPS = 60;
const history = ['[user]\nMake the failing billing test pass.'];

// Keep the same failed tool call visible throughout the transcript.
const repeatedToolCall = [
  '[assistant]\nRunning the test suite again.',
  '[tool_call run_command] {"cmd":"bun test packages/billing/invoice.test.ts"}',
  '[tool_result]\nFAIL invoice.test.ts > renders due date\n0 pass, 1 fail',
].join('\n');

// Check the growing transcript and stop when the supervisor halts.
for (let step = 1; step <= MAX_STEPS; step++) {
  history.push(repeatedToolCall);
  console.log(`agent step ${step}: repeated failing test`);
  if (step % CHECK_EVERY !== 0) continue;

  const action = await supervise(history.join('\n\n'));
  console.log(`supervisor check at step ${step}: ${action}`);
  if (action === 'halt') {
    console.log('supervisor halted the looping agent');
    break;
  }
  if (action === 'nudge') history.push('[user]\nYou appear to be repeating yourself or blocked. Try a different approach, or stop and explain what is blocking you.');
}

How does the loop stop?

  • The CHECK_EVERY value schedules a supervisor check after every five agent steps.
  • The repeatedToolCall value preserves the same failure pattern for every simulated step.
  • The history array accumulates each identical failure record.
  • A halt action triggers break before another agent step can run.
  • Save src/agentLoop.ts.
  • Confirm agentLoop.ts appears inside the src folder in your project sidebar.
Prove the supervisor stops the run

Each supervisor check sends one paid request to OpenRouter. The 60-step cap allows at most 12 checks in this demonstration.

Before you run the demo, which action do you expect once the transcript contains the same failed tool call several times?

  • Return to the terminal from the earlier transcript test.
  • Run src/agentLoop.ts with the same TypeScript runner you used for src/testSupervisor.ts.

You'll see repeated agent-step logs followed by supervisor checks at five-step intervals. Once the action is halt, the terminal prints supervisor halted the looping agent.

No later agent-step log appears. That proves the supervisor stopped execution before the looping agent could issue another tool call.

Does the loop keep running?

  • Confirm src/supervisor.ts exports supervise when the import fails.
  • Compare the import path with the supervisor file name inside your src folder.
  • Confirm repeatedToolCall still contains the same tool call arguments on every step.

Help me diagnose why my supervisor does not halt the forced repeating tool-call loop.

That is the control boundary working. The agent cannot issue another tool call after the supervisor logs halt.

✔️ Awesome, I've got everything!

Your agent loop now accumulates a transcript, checks it on a fixed cadence, and stops when the supervisor detects repetition.

ⓧ I'd like to double check the full code

  • Compare src/agentLoop.ts with this reference file:
import { supervise } from './supervisor.js';

const CHECK_EVERY = 5;
const MAX_STEPS = 60;
const history = ['[user]\nMake the failing billing test pass.'];

// Keep the same failed tool call visible throughout the transcript.
const repeatedToolCall = [
  '[assistant]\nRunning the test suite again.',
  '[tool_call run_command] {"cmd":"bun test packages/billing/invoice.test.ts"}',
  '[tool_result]\nFAIL invoice.test.ts > renders due date\n0 pass, 1 fail',
].join('\n');

// Check the growing transcript and stop when the supervisor halts.
for (let step = 1; step <= MAX_STEPS; step++) {
  history.push(repeatedToolCall);
  console.log(`agent step ${step}: repeated failing test`);
  if (step % CHECK_EVERY !== 0) continue;

  const action = await supervise(history.join('\n\n'));
  console.log(`supervisor check at step ${step}: ${action}`);
  if (action === 'halt') {
    console.log('supervisor halted the looping agent');
    break;
  }
  if (action === 'nudge') history.push('[user]\nYou appear to be repeating yourself or blocked. Try a different approach, or stop and explain what is blocking you.');
}

Your looping run now stops itself. Next, you'll confirm that each supervisor check appears in your OpenRouter activity history.

Review Classification Usage

The forced loop proved that your supervisor can stop repeated work. The final question is what those checks consumed.

Every Decisions API classification uses tokens. Each request also consumes credits.

OpenRouter's Activity dashboard connects that usage to the requests you ran. You can use the evidence to choose a sensible check frequency.

In this step, get ready to:
  • Open the Activity dashboard for your existing OpenRouter account.
  • Filter the dashboard to the ScaleDown Classify model.
  • Confirm the usage attached to the classification requests.
Open the activity dashboard

The Activity dashboard aggregates usage by time window. Filtering to the classifier isolates the supervisor's footprint from other OpenRouter traffic.

  • In the signed-in OpenRouter dashboard from earlier, select Activity from the navigation.
  • Select Overview to show the headline usage metrics.
  • Set the date range to include the classifier tests from this project.
  • Add a model filter for scaledown/classify-1.

Spend updates to the filtered cost. Tokens updates to the classifier's token volume.

Requests shows how many matching calls OpenRouter recorded.

What do these metrics show?

  • Requests counts the matching API calls.
  • Tokens reports the combined token volume for prompts plus completions.
  • Spend shows the credits used by the matching requests.

Can't find the classification usage?

  • Widen the date range until it includes the project runs.
  • Remove the model filter to confirm recent requests appear.
  • Reapply scaledown/classify-1 after recent requests appear.

Help me find my Decisions API classification requests in OpenRouter Activity.

Trace the classification calls

Aggregate metrics show the total footprint. Request details connect that total to the classifier checks that produced it.

Before you open the request details, make a prediction about whether the transcript prompt or the classified output uses more tokens.

  • Select View logs on a filtered activity card.
  • Open the newest matching request row.
  • Check its token breakdown to test your prediction.
  • Check its cost to see what one classification added to your spend.
  • Review the other matching request rows.

You'll see the Decisions API classification calls listed with recorded token counts. Each call also shows its cost.

Why does usage vary?

Each question reads the transcript. Longer transcripts therefore raise input-token usage.

More frequent checks create more paid requests. These records help you choose a cadence from evidence.

  • Use the browser's Back control to return to the filtered Activity page.

The filtered Activity page is open again with classification usage visible. You now have a cost trail for the supervisor checks that protected your loop.

✔️ Awesome, I've got everything!

Your OpenRouter activity now connects the supervisor's classification requests to their token usage and cost.

ⓧ I'd like to double check the final state

  • Confirm the OpenRouter Activity page is open.
  • Confirm the model filter uses scaledown/classify-1.
  • Confirm Requests reflects the project's Decisions API calls.
  • Confirm Tokens reflects the classification token volume.
  • Confirm Spend reflects the classification costs.
  • Confirm no local project files were changed during this review.

That's the full control loop in place. You can now stop repetitive runs with usage evidence in view.

Secret mission

Design a Cost-Aware Escalation Policy

Use the real cost from one classifier call to project a longer agent run. Turn that projection into a clear escalation rule for future agents.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to keep the supervisor available, pause access to paid calls, or remove it. The local TypeScript project has no ongoing compute cost until it makes another OpenRouter Decisions API call.

Cost warning

Each supervisor check sends one paid Decisions API classification request. A stopped local project sends no new requests.

An active API key can authorize more paid calls. The Pause and Delete options below limit that risk.

Resources you used:

  • An OpenRouter API key funded with available credits.
  • A local uncommitted environment configuration that supplies OPENROUTER_API_KEY.
  • A local TypeScript project containing src/agentLoop.ts, src/classifyTranscript.ts, src/supervisor.ts, and src/testSupervisor.ts.
  • The project's installed dependencies, including zod.

Keep everything running

No cleanup is needed while you continue testing the supervisor. Future classifications continue using your available OpenRouter credits.

  • Keep the local TypeScript project folder for future agent projects.
  • Keep the local environment configuration out of source control.
  • Retain the OpenRouter API key only while you still need authenticated classification calls.
  • Review the activity page from earlier after future tests to track token counts.
  • Review the activity page from earlier after future tests to track costs.

Pause - I'll come back to this later

Pause access to paid calls while preserving the TypeScript project. This gives you a restart point without leaving unrestricted API access in the project.

  • Return to the signed-in OpenRouter dashboard from earlier to locate the API key created for this project.
  • Set a credit limit on that API key.
  • Confirm the credit limit is saved for the key.
  • Remove the key value from the same local uncommitted environment configuration where you stored it.
  • Confirm OPENROUTER_API_KEY no longer exposes the credential in that configuration.
  • Keep the local TypeScript project folder for your next session.

Delete - I don't want to use this again

Choose this when you are finished with the supervisor. Deleting the API key can feel final because it permanently blocks that credential from authorizing future calls.

  • Return to the signed-in OpenRouter dashboard from earlier to locate the API key created for this project.
  • Delete that API key using the deletion control attached to it.
  • Confirm the deleted key no longer appears in the key management area.
  • Remove the deleted key value from the same local uncommitted environment configuration where you stored it.
  • Confirm OPENROUTER_API_KEY no longer exposes the deleted credential in that configuration.
  • Locate src/agentLoop.ts in Finder on macOS or File Explorer on Windows.
  • Select the parent project folder that contains the src directory.
  • Delete that parent project folder using your operating system's delete action.
  • Confirm the project folder no longer appears in its previous location.

Nice Work!

Nice Work!

Mission complete! You've built a TypeScript supervisor that uses OpenRouter's Decisions API to classify agent transcripts as progressing, looping, or stuck. It now protects an agent loop by choosing when to continue, nudge, or halt.

What you learned:

  • Designed three classification questions for run state, repetition, and progress. Read labelled probabilities from the ScaleDown Classify responses.
  • Validated each response with Zod. Mapped chosen thresholds to typed continue, nudge, or halt actions. Proved the mapping with healthy, looping, and stuck transcripts.
  • Wired scheduled supervisor checks into a live agent loop. Stopped a forced repeating-tool-call run when the supervisor selected halt. Confirmed token counts and costs on the OpenRouter activity page.
  • Secret Mission: Extended the transcript supervisor through an optional challenge that pushed your loop-detection skills further.

Ready to quiz yourself?