Build an AI Ticket Triage System

Build a validated Gemini API ticket triage CLI with evaluation and human review.

Introduction

30 Second Summary

Support tickets often hide the real issue inside a rushed customer message. A wrong routing decision can delay an urgent incident.

In this project, you will build a command-line system that sends synthetic support tickets directly to the Gemini API for triage. The finished workflow constrains each response before measuring category and urgency decisions against labeled cases.

What You'll Build

You will demo a Windows PowerShell tool that turns a messy ticket into a validated triage decision with a visible four-case evaluation score.

By the end of this project, you'll have:

  • A validated terminal result that displays category, urgency, summary, recommended action, confidence, plus human-review routing.
  • A machine-enforced response contract that uses JSON Schema to constrain fields before application validation checks every value.
  • A four-case regression report that shows a PASS, FAIL, or ERROR result for each ticket before displaying the total score.
  • Secret Mission: Test prompt injection with an adversarial ticket that tries to override the triage instructions. Harden the boundary between instructions and ticket data.

Are there any prerequisites?

You need a Windows computer, internet access, plus a Google account.

The guide includes complete setup for Node.js 24.21.0 LTS, Visual Studio Code, Google AI Studio, plus your Gemini API key.

Before We Start

Before any hands-on work, take a moment to commit to building a reliable support ticket triage system. You will focus on why accurate category and urgency decisions matter to a support team.

Set Up the Windows AI Development Workspace

A direct API project needs Node.js to run JavaScript. Visual Studio Code provides the editor where you will build the triage system.

The model connection also needs a Gemini API credential stored outside your project files. Windows PowerShell must expose that credential before you test any model behavior.

In this step, get ready to:
  • Prepare Node.js v24.21.0 LTS plus Visual Studio Code.
  • Create a Gemini API key as a Windows user environment variable.
  • Create the ai-ticket-triage workspace in PowerShell.
Install and verify your development tools

Node.js is the runtime that executes your JavaScript outside a browser. This project requires the v24.21.0 LTS release.

  • Press the Windows key to open search.
  • Type PowerShell into the search field.
  • Press Enter to open Windows PowerShell.
  • Check the installed Node.js version by running this command:
node --version

The node --version command asks the runtime to identify its installed release. The output places your computer in one of the three setup states below.

✔️ I see the required Node.js version

Your runtime is ready. PowerShell should show v24.21.0.

  • Keep this PowerShell session open for the Visual Studio Code check.

ⓧ I see an older version

The installed runtime is below the project requirement. Upgrade it before creating the workspace.

  • Open the official Node.js download page in your browser.
  • Download the Windows installer for Node.js v24.21.0 LTS.
  • Run the downloaded installer.
  • Keep the recommended installer options.
  • Close every open PowerShell session after the installation completes.
  • Press the Windows key to open search.
  • Type PowerShell into the search field.
  • Press Enter to open a new Windows PowerShell session.
  • Verify the upgraded runtime by running this command:
node --version

You should now see v24.21.0 in PowerShell.

Still seeing the older version?

Close every PowerShell window after the installer finishes. A new session reloads the updated executable path.

Still stuck? Help me verify which Node.js installation PowerShell is using.

ⓧ The command is unavailable

PowerShell cannot find a Node.js installation. The Windows installer adds the runtime to the executable path.

  • Open the official Node.js download page in your browser.
  • Download the Windows installer for Node.js v24.21.0 LTS.
  • Run the downloaded installer.
  • Keep the recommended installer options.
  • Close every open PowerShell session after the installation completes.
  • Press the Windows key to open search.
  • Type PowerShell into the search field.
  • Press Enter to open a new Windows PowerShell session.
  • Verify the new installation by running this command:
node --version

You should now see v24.21.0 in PowerShell.

PowerShell still cannot find Node.js?

Confirm that the installer finished before opening the new PowerShell session. Existing sessions keep their previous executable path.

Need another pair of eyes? Help me troubleshoot why PowerShell cannot find Node.js.

Visual Studio Code gives the project folder an editor plus an integrated terminal. A Windows User setup also makes the code . command available to new console sessions.

  • Press the Windows key to open search.
  • Type Visual Studio Code into the search field.
  • Choose the tab that matches the search result.

✔️ Visual Studio Code is installed

The editor is already available. The final workspace check confirms that PowerShell can launch it.

  • Press Esc to close Windows search.

ⓧ Visual Studio Code is missing

The recommended Windows User setup installs the editor without administrator permissions. It also adds the editor command to the executable path.

  • Open the official Visual Studio Code Windows setup guide in your browser.
  • Download the Windows User setup installer.
  • Run the downloaded installer.
  • Keep the recommended installer options.
  • Complete the installation.
  • Close every open PowerShell session so the next session receives the updated executable path.

Installation did not finish?

Run the Windows User setup installer again. Confirm that the setup reaches its completion screen.

Need help with the installer? Help me troubleshoot my Visual Studio Code Windows User setup.

Create and protect your Gemini API key

An API key authorizes requests to the model. This credential step is sensitive, but the key stays in your Windows user environment instead of a project file.

  • Open the official Gemini API key guide in your browser.
  • Follow the guide into Google AI Studio.
  • Sign in with your Google account.
  • Accept the Google AI Studio terms if your account prompts you.
  • Create a Gemini API key if your account has no reusable key.
  • Copy the Gemini API key you plan to use.

The copied key needs a persistent user variable. That storage makes the credential available to new terminal sessions without placing it in ai-ticket-triage.

  • Press the Windows key to open search.
  • Search for the environment-variable settings for your account.
  • Select the matching system result.
  • Locate the user variables section in the environment settings window.
  • Choose the option to add a user variable.

Why use a user variable?

A user variable persists beyond the current PowerShell session. Node.js can later read it through process.env without exposing the key in source control.

  • Enter GEMINI_API_KEY as the variable name.
  • Paste your copied Gemini API key as the variable value.
  • Save the new user variable.
  • Close the environment settings windows.

Your key is now stored outside the project. A fresh terminal session is required before PowerShell can see the new user variable.

  • Close every open PowerShell session.
  • Press the Windows key to open search.
  • Type PowerShell into the search field.
  • Press Enter to open a new Windows PowerShell session.
  • Confirm that the key exists without displaying its value by running this command:
Test-Path Env:GEMINI_API_KEY

Why is this check safe?

PowerShell checks whether the environment-variable path exists. The command returns a Boolean result without printing your API key.

You should see True. Your new PowerShell session can now access the credential.

Seeing a false result?

Confirm that the user variable name is exactly GEMINI_API_KEY. Close every PowerShell window after saving the variable.

Still missing the variable? Help me troubleshoot my Gemini API key environment variable.

Create and open the project workspace

A workspace keeps every source file for one project together. The ai-ticket-triage folder becomes that boundary for your triage system.

Before you run the setup commands, what two signals would prove that both the runtime and editor are ready?

  • Create the workspace plus run the final tool checks by running these commands in the new PowerShell session:
New-Item -ItemType Directory -Path "ai-ticket-triage" -Force
Set-Location -Path "ai-ticket-triage"
node --version
code .

What do these commands do?

  • New-Item creates the ai-ticket-triage folder. The force option safely reuses it if it already exists.
  • Set-Location moves PowerShell into the new folder.
  • node --version confirms that the required runtime is available in this session.
  • code . opens the current folder in Visual Studio Code.

You should see v24.21.0 in PowerShell. You should also see Visual Studio Code open with ai-ticket-triage as the current folder.

That setup work has paid off. Your runtime, editor, workspace, and protected credential are ready for the first model request.

Did the final check fall short?

If PowerShell shows the wrong Node.js version, return to the Node.js setup tab above. Reinstall the required LTS release before opening another PowerShell session.

If Visual Studio Code stays closed, confirm that you installed the Windows User setup. Open a new PowerShell session after the installation.

Need help with the workspace? Help me troubleshoot my Windows AI development workspace.

Your Windows AI development workspace is ready. Next, you will send a synthetic support ticket directly to the model.

Send a Ticket to the Model

Your Windows workspace now has Node.js v24.21.0 LTS ready to run JavaScript. The ai-ticket-triage folder is already open in Visual Studio Code.

A visible response from the Gemini API proves that your runtime can reach the model. However, readable labeled prose still gives your application no machine-enforceable contract.

In this step, get ready to:
  • Build a direct API request with JavaScript.
  • Send a synthetic duplicate-charge support ticket to the model.
  • Print the final model response as four readable labeled lines.
Create the direct API script

A direct model call needs an endpoint, authentication headers, instructions, ticket data, and response handling. You will keep that entire path visible in one script.

  • Use the file-creation control in the Explorer sidebar to create naive.mjs inside ai-ticket-triage.

You will see naive.mjs listed in the Explorer sidebar.

  • Fill naive.mjs with the complete request-to-output flow by pasting this code:
const API_URL = "https://generativelanguage.googleapis.com/v1beta/interactions";

const response = await fetch(API_URL, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "x-goog-api-key": process.env.GEMINI_API_KEY,
  },
  body: JSON.stringify({
    model: "gemini-3.8-flash",
    store: false,
    system_instruction:
      "You triage support tickets. Reply with exactly four labeled plain-text lines: Category:, Urgency:, Summary:, and Action:.",
    input:
      "I was charged twice for our subscription and payroll closes today. Please correct it today.",
  }),
});

const interaction = await response.json();
const outputStep = [...interaction.steps].reverse().find(
  (step) => step.type === "model_output",
);
const outputText = outputStep.content.find((block) => block.type === "text").text;
console.log(outputText);

What does this code do?

  • The API_URL constant holds the Interactions API endpoint.
  • The fetch() call sends the ticket through an HTTP POST request.
  • The process.env.GEMINI_API_KEY value reads your credential from the Windows environment.
  • The system_instruction field defines the four-line response format.
  • The response logic searches the steps array for the final model_output text.
  • Save naive.mjs.
  • Confirm naive.mjs stays listed under ai-ticket-triage in the Explorer sidebar.

File missing or named incorrectly?

  • Confirm the file name ends with .mjs.
  • Remove any hidden .txt extension from the file name.
  • Compare your file with the full-code reference below.

Still stuck? Help me check why my naive.mjs file is missing or saved with the wrong extension.

✔️ Awesome, I've got everything!

Your saved naive.mjs now contains one complete request-to-output path.

ⓧ I'd like to double check the full code

const API_URL = "https://generativelanguage.googleapis.com/v1beta/interactions";

const response = await fetch(API_URL, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "x-goog-api-key": process.env.GEMINI_API_KEY,
  },
  body: JSON.stringify({
    model: "gemini-3.8-flash",
    store: false,
    system_instruction:
      "You triage support tickets. Reply with exactly four labeled plain-text lines: Category:, Urgency:, Summary:, and Action:.",
    input:
      "I was charged twice for our subscription and payroll closes today. Please correct it today.",
  }),
});

const interaction = await response.json();
const outputStep = [...interaction.steps].reverse().find(
  (step) => step.type === "model_output",
);
const outputText = outputStep.content.find((block) => block.type === "text").text;
console.log(outputText);
Trace the request contract

Why use a direct REST request?

A direct REST request keeps the model boundary visible. You can inspect the exact endpoint and headers.

The request body separates the system instruction from the ticket text. This makes later validation work easier to reason about.

  • Find the API_URL line in naive.mjs.
  • Trace the fetch() call to find the POST method.
  • Locate x-goog-api-key in the headers object.
  • Locate process.env.GEMINI_API_KEY to confirm the credential comes from Windows.
  • Locate gemini-3.8-flash in the request body.
  • Read the system_instruction field to identify the triage policy.
  • Read the input field to identify the synthetic customer ticket.

The API key never appears as a literal value in naive.mjs. The script reads it from your Windows environment when Node.js runs.

Run the first triage

The API returns a JSON interaction with a steps array. The script searches backward for the final model_output step.

The script takes the text block from that step. PowerShell then shows the model's readable decision.

Before you run the script, make a quick prediction about whether the result will be readable labeled prose or a JSON object.

  • Send the synthetic ticket to Gemini by running this command:
node naive.mjs

What should I see?

The terminal prints four readable labeled lines.

  • A line beginning with Category: identifies the ticket type.
  • A line beginning with Urgency: states the operational priority.
  • A line beginning with Summary: condenses the customer problem.
  • A line beginning with Action: recommends the next support response.

Your first live model connection now prints a triage answer in PowerShell.

No four labeled lines?

  • Switch back to the new PowerShell session from Step 1.
  • Confirm naive.mjs matches the full-file reference if the script stops before printing output.
  • Check that the PowerShell terminal is using the ai-ticket-triage workspace.

Need another pair of eyes? Help me debug why node naive.mjs does not print four labeled triage lines.

Your direct Gemini connection is working. Next, you will test whether this readable response can survive machine parsing.

Break the Free-Form Parser

Your naive.mjs script already proves the Node.js runtime can print a readable triage response from the model. That visible result confirms the connection works.

Readable output still needs to prove it can satisfy the application's parser contract. This step sends the same labeled prose into JSON.parse() to test whether prompt wording created a dependable JSON interface.

In this step, get ready to:
  • Preserve the raw model response for comparison.
  • Test the labeled response with JSON.parse() inside try...catch.
  • Compare human readability with the parser's outcome.
Preserve the raw response

The raw response is your control sample. Keeping it visible lets you compare what a person can read with what the parser can consume.

  • Switch back to naive.mjs in Visual Studio Code.
  • Locate console.log(outputText); at the bottom of the file.
  • Confirm that this line still prints the raw labeled response.
Add the parser test

The parser test sends the exact same model text through a second consumer. The surrounding error handler keeps the failure visible without crashing before you can inspect it.

  • Select everything from the const outputText line through console.log(outputText);.
  • Replace the selected ending with this code:
const outputText = outputStep.content.find((block) => block.type === "text").text;

console.log(outputText);

try {
  JSON.parse(outputText);
} catch (error) {
  const message = error instanceof Error ? error.message : String(error);
  console.error(`Parser failure: ${message}`);
}

What does this code do?

  • The outputText variable keeps the final model text extracted from the response.
  • The console.log(outputText); line prints the untouched response before the parser test.
  • The JSON.parse(outputText); line tests whether the response follows JSON syntax.
  • The catch block reports any parsing error with the Parser failure: prefix.
  • Save naive.mjs.

✔️ Awesome, I've got everything!

  • Confirm that naive.mjs is saved.

ⓧ I'd like to double check the full code

const API_URL = "https://generativelanguage.googleapis.com/v1beta/interactions";

const response = await fetch(API_URL, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "x-goog-api-key": process.env.GEMINI_API_KEY,
  },
  body: JSON.stringify({
    model: "gemini-3.8-flash",
    store: false,
    system_instruction:
      "You triage support tickets. Reply with exactly four labeled plain-text lines: Category:, Urgency:, Summary:, and Action:.",
    input:
      "I was charged twice for our subscription and payroll closes today. Please correct it today.",
  }),
});

const interaction = await response.json();
const outputStep = [...interaction.steps].reverse().find(
  (step) => step.type === "model_output",
);
const outputText = outputStep.content.find((block) => block.type === "text").text;

console.log(outputText);

try {
  JSON.parse(outputText);
} catch (error) {
  const message = error instanceof Error ? error.message : String(error);
  console.error(`Parser failure: ${message}`);
}
Run the parser check

The PowerShell terminal now gives you one view of both consumers. Running the script tests whether the same response can serve a person and a JSON parser.

  • Before you run the script, predict whether the same output can satisfy both consumers.
  • Test the parser assumption from the PowerShell terminal in Visual Studio Code by running:
node naive.mjs

What should you see?

You will first see four readable lines beginning with category, urgency, summary, and action labels.

After those lines, you will see Parser failure: followed by a JSON parsing error.

This failure is intentional. The model returned useful prose while the application required fixed JSON syntax.

Do you only see the readable lines?

Confirm that the try...catch block is saved beneath console.log(outputText); in naive.mjs.

If the request stops before printing the labels, confirm that the API key remains available in this PowerShell session.

Still stuck? Help me debug why naive.mjs does not print the parser failure.

That is a valuable failure. Your terminal now exposes the contract gap clearly. Next, you will make the API produce constrained JSON and validate every returned field.

Enforce and Validate a Triage Schema

Your earlier script exposed the weak point in the first design. The model produced readable labels that failed as soon as JSON.parse() treated them as structured data.

This step fixes that contract with two layers. A JSON Schema constrains the response shape. Application validation independently rejects missing fields or unsafe values before the result reaches the support workflow.

In this step, get ready to:
  • Define the allowed triage fields with a schema.
  • Validate model output before applying the human-review policy.
  • Run a command-line ticket through the structured triage path.
Define the schema contract

The schema describes the JSON object that the model must produce. Its enums narrow category and urgency to values your application knows how to handle.

  • In the VS Code Explorer sidebar, create a file named triage.mjs inside the ai-ticket-triage folder.
  • Add the API configuration and allowed classification values by pasting this code into triage.mjs:
const API_URL = "https://generativelanguage.googleapis.com/v1beta/interactions";
const MODEL = "gemini-3.8-flash";
export const REVIEW_THRESHOLD = 0.7;

const CATEGORIES = new Set([
  "billing",
  "technical",
  "account",
  "feature_request",
  "other",
]);

const URGENCIES = new Set(["low", "medium", "high", "critical"]);

What does this configuration control?

  • API_URL identifies the Gemini Interactions API endpoint used for each request.
  • MODEL keeps the gemini-3.8-flash model choice in one place.
  • REVIEW_THRESHOLD defines the project policy that routes confidence below 0.7 to human review.
  • CATEGORIES and URGENCIES provide deterministic allowlists for application validation.
  • Save triage.mjs.
  • Confirm that triage.mjs appears beneath the ai-ticket-triage folder in the Explorer sidebar.

Don't see the new file?

Check that you created triage.mjs inside the open ai-ticket-triage folder. A file created outside that folder does not appear in this workspace.

Still stuck? Help me find triage.mjs in my VS Code workspace.

The response object needs five model-generated fields. Each field carries a type plus any limits that the API should enforce.

  • Add the initial TRIAGE_SCHEMA below the urgency values by pasting this code:
export const TRIAGE_SCHEMA = {
  type: "object",
  properties: {
    category: {
      type: "string",
      enum: ["billing", "technical", "account", "feature_request", "other"],
      description: "The single best category for the support ticket.",
    },
    urgency: {
      type: "string",
      enum: ["low", "medium", "high", "critical"],
      description: "The operational urgency of the support ticket.",
    },
    summary: {
      type: "string",
      description: "A concise summary of the customer's problem.",
    },
    recommended_action: {
      type: "string",
      description: "The next action a support team should take.",
    },
    confidence: {
      type: "number",
      minimum: 0,
      maximum: 1,
      description: "Confidence in the category and urgency from 0 to 1.",
    },
  },
};

How does the schema shape the response?

  • category and urgency use enums so the model chooses from known routing values.
  • summary and recommended_action capture the ticket meaning and the next support action as strings.
  • confidence uses a numeric range from 0 to 1.
  • The descriptions tell the model what each field represents.
  • Save triage.mjs.
  • Confirm that VS Code shows matching braces around TRIAGE_SCHEMA with no syntax marker beside the object.

Seeing a schema syntax marker?

Check the comma after each property object. Also check that the schema ends with one brace for properties and one brace for TRIAGE_SCHEMA.

Need another pair of eyes? Help me compare the braces and commas in TRIAGE_SCHEMA.

The property definitions describe valid fields. The final schema rules make every field mandatory and reject extra keys that your application does not understand.

  • At the end of TRIAGE_SCHEMA in triage.mjs, find these closing lines:
  },
};

What are these lines closing?

The first brace closes properties. The second brace closes the complete TRIAGE_SCHEMA object.

  • Replace those closing lines with this required-field policy:
  },
  required: [
    "category",
    "urgency",
    "summary",
    "recommended_action",
    "confidence",
  ],
  additionalProperties: false,
};

Why require fields and reject extras?

required prevents a superficially valid object from omitting a decision your application needs. additionalProperties blocks unexpected keys from silently expanding the contract.

  • Save triage.mjs.
  • Confirm that the schema now lists all five required fields beneath properties.

Is the required list outside the schema?

Check the indentation around the final braces. required and additionalProperties belong inside TRIAGE_SCHEMA after the properties object closes.

Still unsure? Help me place required and additionalProperties correctly.

Validate and request structured output

Schema-constrained output gives the parser valid JSON. Independent validation protects the application when a field still carries an unusable value.

The request also needs robust response handling. Your module checks the HTTP result before parsing the response body or trusting the model output.

  • Add the response extraction helper below TRIAGE_SCHEMA in triage.mjs:
function extractOutputText(interaction) {
  for (let stepIndex = interaction.steps.length - 1; stepIndex >= 0; stepIndex -= 1) {
    const step = interaction.steps[stepIndex];
    if (step.type !== "model_output" || !Array.isArray(step.content)) continue;

    for (const block of step.content) {
      if (block.type === "text" && typeof block.text === "string") {
        return block.text;
      }
    }
  }

  throw new Error("Gemini response did not contain a model text output.");
}

How does output extraction work?

  • The outer loop searches the steps array from the newest step toward the oldest.
  • The type checks ignore steps that do not contain model output.
  • The inner loop returns the first text block from a matching model-output step.
  • The final error prevents missing model text from becoming an undefined parsing failure later.
  • Save triage.mjs.
  • Confirm that extractOutputText() sits after the closing brace of TRIAGE_SCHEMA.

Is the helper inside the schema?

Look for the closing }; after additionalProperties. extractOutputText() starts on the next line outside the schema object.

Need help locating the boundary? Help me separate TRIAGE_SCHEMA from extractOutputText().

The validator starts with the object shape plus the two enum checks. It collects every detected problem so one failure can explain multiple invalid fields.

  • Add the first version of validateTriage() below extractOutputText():
export function validateTriage(value) {
  if (value === null || typeof value !== "object" || Array.isArray(value)) {
    throw new Error("Triage result must be an object.");
  }

  const errors = [];

  if (!CATEGORIES.has(value.category)) {
    errors.push("category is not allowed");
  }

  if (!URGENCIES.has(value.urgency)) {
    errors.push("urgency is not allowed");
  }

  if (errors.length > 0) {
    throw new Error(`Invalid triage result: ${errors.join("; ")}`);
  }

  return value;
}

What does the first validator cover?

  • The opening guard rejects null values, arrays, and primitive values.
  • CATEGORIES rejects category values outside the routing allowlist.
  • URGENCIES rejects urgency values outside the operational scale.
  • The function throws one combined validation error or returns the accepted object.
  • Save triage.mjs.
  • Confirm that validateTriage() appears as a separate exported function beneath the extraction helper.

Category and urgency cover routing. The remaining checks ensure that summaries and actions contain useful text while confidence remains a finite number within the schema range.

  • Inside validateTriage(), find the final error check:
  if (errors.length > 0) {
    throw new Error(`Invalid triage result: ${errors.join("; ")}`);
  }

Why use this line as the anchor?

Every field check must run before this block examines errors. Placing the new checks directly above it preserves that order.

  • Replace that error check with the expanded field checks and the same final error block:
  if (typeof value.summary !== "string" || value.summary.trim() === "") {
    errors.push("summary must be a non-empty string");
  }

  if (
    typeof value.recommended_action !== "string" ||
    value.recommended_action.trim() === ""
  ) {
    errors.push("recommended_action must be a non-empty string");
  }

  if (
    typeof value.confidence !== "number" ||
    !Number.isFinite(value.confidence) ||
    value.confidence < 0 ||
    value.confidence > 1
  ) {
    errors.push("confidence must be a number from 0 to 1");
  }

  if (errors.length > 0) {
    throw new Error(`Invalid triage result: ${errors.join("; ")}`);
  }

What do the added checks reject?

  • The summary check rejects missing text plus strings that contain only whitespace.
  • The recommended-action check applies the same requirement to the support team's next step.
  • The confidence check rejects non-numeric values, non-finite numbers, and values outside the range from 0 to 1.
  • Save triage.mjs.
  • Confirm that the added checks appear before if (errors.length > 0).

Did a check land after return?

Move every field check above the final errors.length block. Code placed after return value never validates the returned result.

Need help checking the order? Help me arrange the checks inside validateTriage().

The public triage function begins at the application boundary. It rejects empty tickets and confirms that the existing API key reached the current terminal session.

  • Add the input and credential guards below validateTriage() in triage.mjs:
export async function triageTicket(ticket) {
  if (typeof ticket !== "string" || ticket.trim() === "") {
    throw new Error("Ticket text is required.");
  }

  const apiKey = process.env.GEMINI_API_KEY;
  if (!apiKey) {
    throw new Error("GEMINI_API_KEY is not available in this terminal session.");
  }
}

Why check input before the API call?

The ticket guard stops an empty request before it consumes an API call. The environment check keeps the Gemini API key outside the project files while producing a clear failure when the variable is unavailable.

  • Save triage.mjs.
  • Confirm that triageTicket() appears as the final exported function in the file.

The request now supplies the schema through response_format. It also sends the ticket as the model input and keeps response storage disabled.

  • Inside triageTicket(), add this request after the API key guard:
  const response = await fetch(API_URL, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "x-goog-api-key": apiKey,
    },
    body: JSON.stringify({
      model: MODEL,
      store: false,
      system_instruction:
        "You triage support tickets. Use critical only for an active security incident, a widespread outage, or immediate serious risk. Use high for major impact without a current widespread outage. Base confidence on how clearly the ticket supports the classification.",
      input: ticket,
      response_format: {
        type: "text",
        mime_type: "application/json",
        schema: TRIAGE_SCHEMA,
      },
    }),
  });

How does this request enforce structure?

  • The headers send JSON and authenticate the request with the API key from process.env.
  • system_instruction defines how operational impact maps to urgency.
  • response_format requests JSON that conforms to TRIAGE_SCHEMA.
  • store remains false for the synthetic ticket request.

A structured request still needs defensive response handling. The remaining code preserves the raw response for HTTP errors before parsing and validating the successful result.

  • Add the response checks and review-routing result after the fetch() call:
  const rawResponse = await response.text();

  if (!response.ok) {
    throw new Error(`Gemini API error ${response.status}: ${rawResponse}`);
  }

  const interaction = JSON.parse(rawResponse);
  if (!Array.isArray(interaction.steps)) {
    throw new Error("Gemini response did not contain a steps array.");
  }

  const outputText = extractOutputText(interaction);
  const parsed = JSON.parse(outputText);
  const validated = validateTriage(parsed);

  return {
    ...validated,
    needs_review:
      validated.confidence < REVIEW_THRESHOLD || validated.urgency === "critical",
  };

How does the response become a safe decision?

  • response.text() preserves the full body so an unsuccessful HTTP response can include useful context.
  • JSON.parse(rawResponse) parses the API envelope only after the HTTP status succeeds.
  • extractOutputText() locates the JSON text produced by the model.
  • validateTriage() enforces the application's own field rules before the result is returned.
  • needs_review becomes true for confidence below 0.7 or any critical ticket.
  • Save triage.mjs.
  • Confirm that the final response-handling code remains inside triageTicket() before its closing brace.

Seeing braces outside triageTicket()?

Check that the request and response blocks sit between the API key guard and the function's final brace. A premature closing brace leaves response outside the function.

Need help checking the function boundary? Help me find a misplaced brace in triageTicket().

✔️ Awesome, I've got everything!

Great. Save triage.mjs before you connect it to the command-line entry point.

ⓧ I'd like to double check the full code

const API_URL = "https://generativelanguage.googleapis.com/v1beta/interactions";
const MODEL = "gemini-3.8-flash";
export const REVIEW_THRESHOLD = 0.7;

const CATEGORIES = new Set([
  "billing",
  "technical",
  "account",
  "feature_request",
  "other",
]);

const URGENCIES = new Set(["low", "medium", "high", "critical"]);

export const TRIAGE_SCHEMA = {
  type: "object",
  properties: {
    category: {
      type: "string",
      enum: ["billing", "technical", "account", "feature_request", "other"],
      description: "The single best category for the support ticket.",
    },
    urgency: {
      type: "string",
      enum: ["low", "medium", "high", "critical"],
      description: "The operational urgency of the support ticket.",
    },
    summary: {
      type: "string",
      description: "A concise summary of the customer's problem.",
    },
    recommended_action: {
      type: "string",
      description: "The next action a support team should take.",
    },
    confidence: {
      type: "number",
      minimum: 0,
      maximum: 1,
      description: "Confidence in the category and urgency from 0 to 1.",
    },
  },
  required: [
    "category",
    "urgency",
    "summary",
    "recommended_action",
    "confidence",
  ],
  additionalProperties: false,
};

function extractOutputText(interaction) {
  for (let stepIndex = interaction.steps.length - 1; stepIndex >= 0; stepIndex -= 1) {
    const step = interaction.steps[stepIndex];
    if (step.type !== "model_output" || !Array.isArray(step.content)) continue;

    for (const block of step.content) {
      if (block.type === "text" && typeof block.text === "string") {
        return block.text;
      }
    }
  }

  throw new Error("Gemini response did not contain a model text output.");
}

export function validateTriage(value) {
  if (value === null || typeof value !== "object" || Array.isArray(value)) {
    throw new Error("Triage result must be an object.");
  }

  const errors = [];

  if (!CATEGORIES.has(value.category)) {
    errors.push("category is not allowed");
  }

  if (!URGENCIES.has(value.urgency)) {
    errors.push("urgency is not allowed");
  }

  if (typeof value.summary !== "string" || value.summary.trim() === "") {
    errors.push("summary must be a non-empty string");
  }

  if (
    typeof value.recommended_action !== "string" ||
    value.recommended_action.trim() === ""
  ) {
    errors.push("recommended_action must be a non-empty string");
  }

  if (
    typeof value.confidence !== "number" ||
    !Number.isFinite(value.confidence) ||
    value.confidence < 0 ||
    value.confidence > 1
  ) {
    errors.push("confidence must be a number from 0 to 1");
  }

  if (errors.length > 0) {
    throw new Error(`Invalid triage result: ${errors.join("; ")}`);
  }

  return value;
}

export async function triageTicket(ticket) {
  if (typeof ticket !== "string" || ticket.trim() === "") {
    throw new Error("Ticket text is required.");
  }

  const apiKey = process.env.GEMINI_API_KEY;
  if (!apiKey) {
    throw new Error("GEMINI_API_KEY is not available in this terminal session.");
  }

  const response = await fetch(API_URL, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "x-goog-api-key": apiKey,
    },
    body: JSON.stringify({
      model: MODEL,
      store: false,
      system_instruction:
        "You triage support tickets. Use critical only for an active security incident, a widespread outage, or immediate serious risk. Use high for major impact without a current widespread outage. Base confidence on how clearly the ticket supports the classification.",
      input: ticket,
      response_format: {
        type: "text",
        mime_type: "application/json",
        schema: TRIAGE_SCHEMA,
      },
    }),
  });

  const rawResponse = await response.text();

  if (!response.ok) {
    throw new Error(`Gemini API error ${response.status}: ${rawResponse}`);
  }

  const interaction = JSON.parse(rawResponse);
  if (!Array.isArray(interaction.steps)) {
    throw new Error("Gemini response did not contain a steps array.");
  }

  const outputText = extractOutputText(interaction);
  const parsed = JSON.parse(outputText);
  const validated = validateTriage(parsed);

  return {
    ...validated,
    needs_review:
      validated.confidence < REVIEW_THRESHOLD || validated.urgency === "critical",
  };
}

What should match?

Compare the constants, schema, extraction helper, validator, and triageTicket() in order. Your saved triage.mjs should match this reference exactly.

Run the structured command-line path

The reusable module now handles the model boundary. A small command-line entry point supplies a ticket and prints the validated object for a person or another program to consume.

  • In the VS Code Explorer sidebar, create app.mjs inside the ai-ticket-triage folder.
  • Add the command-line entry point by pasting this code into app.mjs:
import { triageTicket } from "./triage.mjs";

const defaultTicket =
  "I was charged twice for our subscription and payroll closes today. Please correct it today.";

const ticket = process.argv.slice(2).join(" ").trim() || defaultTicket;

try {
  const result = await triageTicket(ticket);
  console.log(JSON.stringify(result, null, 2));
} catch (error) {
  const message = error instanceof Error ? error.message : String(error);
  console.error(`Triage failed: ${message}`);
  process.exitCode = 1;
}

What does the entry point do?

  • The import reuses triageTicket() without duplicating the API logic.
  • process.argv combines the command-line words into one ticket.
  • defaultTicket provides the synthetic duplicate-charge example when no argument is supplied.
  • JSON.stringify() formats the validated result for readable terminal output.
  • process.exitCode reports a failed run while allowing Node.js to finish cleanly.
  • Save app.mjs.
  • Confirm that app.mjs appears beside triage.mjs in the Explorer sidebar.

Can't resolve triage.mjs?

Check that both files sit in the same ai-ticket-triage folder. Also compare the import path with the exact filename triage.mjs.

Need help with the import? Help me fix the app.mjs import path.

✔️ Awesome, I've got everything!

Your command-line entry point is ready. Save app.mjs before running the structured request.

ⓧ I'd like to double check the full code

import { triageTicket } from "./triage.mjs";

const defaultTicket =
  "I was charged twice for our subscription and payroll closes today. Please correct it today.";

const ticket = process.argv.slice(2).join(" ").trim() || defaultTicket;

try {
  const result = await triageTicket(ticket);
  console.log(JSON.stringify(result, null, 2));
} catch (error) {
  const message = error instanceof Error ? error.message : String(error);
  console.error(`Triage failed: ${message}`);
  process.exitCode = 1;
}

What should match?

Compare the import, default ticket, command-line argument handling, output formatting, and error handling. Your saved app.mjs should match this reference exactly.

Before you run the new path, predict what replaces the parser failure. Which fields should a validated triage object contain?

  • Return to the PowerShell session in the ai-ticket-triage folder.
  • Send the sample ticket through the structured path by running this command:
node app.mjs "I was charged twice and payroll closes today."

What does this command test?

Node.js passes the quoted ticket to app.mjs. The entry point sends it through schema-constrained generation, application validation, and human-review routing.

You should see formatted JSON containing category, urgency, summary, recommended_action, confidence, and needs_review.

That is the contract working. The application now receives a validated object that deterministic code can inspect before any support action occurs.

Seeing Triage failed?

If the message mentions GEMINI_API_KEY, return to the new PowerShell session that received the Windows user environment variable. If the message identifies an invalid triage field, compare your schema and validator with the full-code reference.

Still blocked? Help me diagnose my structured triage failure.

The structured path has now replaced the temporary parser experiment. The old file can leave the workspace without affecting the reusable module.

  • Right-click naive.mjs in the VS Code Explorer sidebar.
  • Use the context menu's delete action to remove naive.mjs.
  • Confirm that the Explorer sidebar now shows triage.mjs and app.mjs without naive.mjs.

Your triage system now turns model output into a constrained and validated software object. Next, you will measure that system against several labeled support tickets instead of trusting one successful example.

Measure the Triage System

Your structured triage path now turns one support ticket into validated JSON. That proves the contract works for a single example.

One successful demo cannot reveal regressions across different ticket types.

An evaluation harness turns model behavior into repeatable checks. It keeps decisions for human review visible.

In this step, get ready to:
  • Create four labeled synthetic regression cases.
  • Build a sequential evaluator that compares model output with expected labels.
  • Report each result with confidence, review routing, and an overall score.
Create the regression cases

Each case pairs a synthetic ticket with expected category and urgency labels. These fixed labels create a baseline for future runs.

  • Create cases.mjs inside the ai-ticket-triage folder from the VS Code file sidebar.
  • Add the four labeled tickets to cases.mjs by pasting this code:
export const cases = [
  {
    name: "duplicate subscription charge",
    ticket:
      "We were charged twice for the same subscription invoice. Payroll closes today and we need the duplicate corrected today.",
    expectedCategory: "billing",
    expectedUrgency: "high",
  },
  {
    name: "single-user CSV crash",
    ticket:
      "The desktop app crashes whenever one user uploads a CSV. They can enter the records manually, so work is slower but not blocked.",
    expectedCategory: "technical",
    expectedUrgency: "medium",
  },
  {
    name: "dark mode request",
    ticket:
      "Please add a dark mode in a future release. Nothing is broken and there is no deadline.",
    expectedCategory: "feature_request",
    expectedUrgency: "low",
  },
  {
    name: "administrator account takeover",
    ticket:
      "An attacker changed our company administrator email and locked out every admin. Unauthorized access is active now.",
    expectedCategory: "account",
    expectedUrgency: "critical",
  },
];

What does this dataset do?

  • Each ticket contains synthetic wording for a distinct support scenario.
  • The expectedCategory field records the intended category.
  • The expectedUrgency field records the intended urgency.
  • The administrator account takeover expects critical. This also exercises the existing human-review rule.
  • Save cases.mjs.
  • Confirm cases.mjs is listed beside app.mjs in the VS Code file sidebar.

Good progress. Your triage system now has four consistent examples that can expose behavior changes.

Seeing a problem in cases.mjs?

Check that every case has a name, ticket, expected category, and expected urgency. Compare the opening and closing brackets with the reference below.

Still stuck? Help me find the syntax problem in my cases.mjs dataset.

✔️ Awesome, I've got everything!

Great. Double-check that cases.mjs is saved before you build the evaluator.

ⓧ I'd like to double check the full code

export const cases = [
  {
    name: "duplicate subscription charge",
    ticket:
      "We were charged twice for the same subscription invoice. Payroll closes today and we need the duplicate corrected today.",
    expectedCategory: "billing",
    expectedUrgency: "high",
  },
  {
    name: "single-user CSV crash",
    ticket:
      "The desktop app crashes whenever one user uploads a CSV. They can enter the records manually, so work is slower but not blocked.",
    expectedCategory: "technical",
    expectedUrgency: "medium",
  },
  {
    name: "dark mode request",
    ticket:
      "Please add a dark mode in a future release. Nothing is broken and there is no deadline.",
    expectedCategory: "feature_request",
    expectedUrgency: "low",
  },
  {
    name: "administrator account takeover",
    ticket:
      "An attacker changed our company administrator email and locked out every admin. Unauthorized access is active now.",
    expectedCategory: "account",
    expectedUrgency: "critical",
  },
];

What should I compare?

Your file should contain exactly four case objects. Check every ticket and expected label character by character.

Build the evaluator

The evaluator sends each ticket through the existing triageTicket() function. It compares only category and urgency against the fixed labels.

Sequential requests keep the report easy to follow. Each result is printed before the next case starts.

  • Create evaluate.mjs inside the ai-ticket-triage folder from the VS Code file sidebar.
  • Add the sequential evaluation loop to evaluate.mjs by pasting this code:
import { cases } from "./cases.mjs";
import { triageTicket } from "./triage.mjs";

let passed = 0;

for (const testCase of cases) {
  try {
    const result = await triageTicket(testCase.ticket);
    const categoryMatch = result.category === testCase.expectedCategory;
    const urgencyMatch = result.urgency === testCase.expectedUrgency;
    const casePassed = categoryMatch && urgencyMatch;

    if (casePassed) passed += 1;

    console.log(
      `${casePassed ? "PASS" : "FAIL"} | ${testCase.name} | ` +
        `expected=${testCase.expectedCategory}/${testCase.expectedUrgency} | ` +
        `actual=${result.category}/${result.urgency} | ` +
        `confidence=${result.confidence} | review=${result.needs_review}`,
    );
  } catch (error) {
    const message = error instanceof Error ? error.message : String(error);
    console.log(`ERROR | ${testCase.name} | ${message}`);
  }
}

const percentage = Math.round((passed / cases.length) * 100);
console.log(`\nScore: ${passed}/${cases.length} (${percentage}%)`);

What does this evaluator do?

  • The imports connect the labeled cases to the existing triageTicket() function.
  • The loop sends one ticket at a time to the triage system.
  • The match checks require both category and urgency to equal the expected labels.
  • The catch block converts a failed request into an ERROR line. Later cases can still run.
  • The final calculation reports the passing count as a rounded percentage.
  • Save evaluate.mjs.
  • Confirm evaluate.mjs is listed beside cases.mjs in the VS Code file sidebar.

Seeing an import or syntax problem?

Confirm that evaluate.mjs, cases.mjs, and triage.mjs are inside the same ai-ticket-triage folder.

Need a second pair of eyes? Help me debug my evaluation harness.

✔️ Awesome, I've got everything!

Your evaluator is ready. Save evaluate.mjs before running the four cases.

ⓧ I'd like to double check the full code

import { cases } from "./cases.mjs";
import { triageTicket } from "./triage.mjs";

let passed = 0;

for (const testCase of cases) {
  try {
    const result = await triageTicket(testCase.ticket);
    const categoryMatch = result.category === testCase.expectedCategory;
    const urgencyMatch = result.urgency === testCase.expectedUrgency;
    const casePassed = categoryMatch && urgencyMatch;

    if (casePassed) passed += 1;

    console.log(
      `${casePassed ? "PASS" : "FAIL"} | ${testCase.name} | ` +
        `expected=${testCase.expectedCategory}/${testCase.expectedUrgency} | ` +
        `actual=${result.category}/${result.urgency} | ` +
        `confidence=${result.confidence} | review=${result.needs_review}`,
    );
  } catch (error) {
    const message = error instanceof Error ? error.message : String(error);
    console.log(`ERROR | ${testCase.name} | ${message}`);
  }
}

const percentage = Math.round((passed / cases.length) * 100);
console.log(`\nScore: ${passed}/${cases.length} (${percentage}%)`);

What should I compare?

Check both import paths first. Then compare the loop, output fields, error handling, and final score calculation.

Interpret the evaluation report

The overall score summarizes performance on this small dataset. The individual case lines carry the diagnostic detail.

The evaluator makes four requests in sequence, so the terminal can pause between result lines.

Before you run the evaluator, consider which ticket is most likely to be classified consistently.

  • Run all four regression cases from the PowerShell terminal by running:
node evaluate.mjs

How do I read the report?

  • A PASS means both the category and urgency matched the expected labels.
  • A FAIL preserves the disagreement for inspection.
  • An ERROR records a request or execution problem for that case.
  • The confidence value shows the model's reported certainty.
  • The review value shows whether the deterministic policy routed the result to a person.

You should see four lines beginning with PASS, FAIL, or ERROR. The final line begins with Score: and includes a count plus a percentage.

No complete evaluation report?

Confirm that the current PowerShell location is the ai-ticket-triage folder. Confirm that GEMINI_API_KEY remains available in this terminal session.

If one case reports ERROR, read the message on that line before rerunning the evaluator.

Still blocked? Help me diagnose my incomplete evaluation report.

  • Compare each actual category with its expected category.
  • Compare each actual urgency with its expected urgency.
  • Classify each disagreement as ambiguous data, prompt policy, or model behavior.
  • Treat each ERROR as an execution issue separate from a classification disagreement.
  • Choose one passing case if every case passes.
  • Identify the ticket wording that made the selected case unambiguous.

Well done. Your triage system now reports measurable behavior across four labeled tickets. Its confidence and review fields keep risky decisions visible.

Secret mission

Test a Prompt-Injection Ticket

Find out whether your model follows the outage evidence or the malicious instruction embedded beside it. You will turn the result into a stronger trust boundary and prove it across all five evaluation cases.

Clean Up Your Resources

Clean Up Your Resources

The Gemini API Free Tier is free of charge, so there is no paid resource to keep running. Choose whether to retain the local project, pause your work, or delete its files and credential.

Resources you used:

  • A local ai-ticket-triage folder containing app.mjs, triage.mjs, cases.mjs, and evaluate.mjs.
  • A Gemini API key associated with a Google Cloud project through Google AI Studio.
  • A Windows user environment variable named GEMINI_API_KEY that makes the key available to PowerShell.

Keep everything running

No action needed. Choose this if you want to run more synthetic tickets or continue improving the evaluator.

  • Your ai-ticket-triage folder keeps the complete application and five evaluation cases.
  • Your Gemini API key remains available for future requests.
  • Your GEMINI_API_KEY user variable remains available in new PowerShell sessions.
  • Your scripts remain idle until you run them.
  • Use only synthetic, non-sensitive tickets because Free Tier content may be used to improve Google products.

Pause - I'll come back to this later

Shut down the project tools to free up memory. Your folder and credential remain available for later.

  • Close VS Code.
  • Close the PowerShell session used for this project.

The scripts make no Gemini API requests while they are idle.

Delete - I don't want to use this again

The Delete option gives you a clean restart. Deletion is permanent, so the steps below target only the resources listed above.

  • Close VS Code.
  • Delete the ai-ticket-triage folder from the location where you created it by using File Explorer.
  • Open the official Gemini API key guide.
  • Follow its Google AI Studio link to reach the key-management page.
  • Select the API key associated with this project.
  • Confirm that no other project relies on the same API key.
  • Delete the key in Google AI Studio.
  • Open the user environment-variable editor through Windows search.
  • Select GEMINI_API_KEY under your user variables.
  • Remove the selected user variable.

Nice Work!

Nice Work!

You did it! Your command-line AI triage system now converts messy support tickets into structured decisions that pass application checks.

You've learned how to:

  • Build a direct REST integration with the Gemini API from Node.js. You authenticated through an environment variable. You also surfaced API failures before parsing model output.
  • Turn a readable but fragile model response into a JSON Schema contract. You added application validation for every triage field. You applied a confidence threshold that routes uncertain or critical tickets to human review.
  • Measure category and urgency behavior with a repeatable evaluation harness. You made every case visible as PASS, FAIL, or ERROR. You calculated a final count and percentage while preserving disagreements for diagnosis.
  • Secret Mission: Hardened the instruction boundary against prompt injection inside an untrusted support ticket. The evaluator now checks the adversarial outage case across the complete five-ticket set.

Ready to quiz yourself?