Build a Claude Code PR Risk Coach

Build and evaluate a Claude Code plugin for structured PR risk reviews.

Introduction

30 Second Summary

A code review can sound convincing while missing a serious security flaw. Without a repeatable test, you cannot tell whether the result came from your workflow or from a lucky response.

In this project, you will build PR Risk Coach as a local Claude Code plugin that reviews proposed code changes. A specialized read-only subagent handles security analysis in an isolated context.

What You'll Build

From Windows PowerShell, you invoke /pr-risk-coach:review-change against an unsafe login change to see a structured risk report backed by isolated security analysis.

By the end of this project, you'll have:

A reusable review command that reports the risk level, blocking issues, security findings, test gaps, and merge recommendation.
An isolated security review that delegates focused analysis to a read-only specialist.
A repeatable baseline eval that compares behavior with the plugin against behavior without it.
Secret Mission: Add a negative test proving ordinary writing requests do not trigger the review workflow.

What do I need before starting?

You need access to a Claude Code-supported account on Windows. Familiarity with Claude Code, Skills, and plugins helps you focus on workflow design.

Plugin evals make real model calls. They count against your existing Claude Code plan usage or API bill.

Before We Start

First, lock in what you are building and why it matters. This commitment keeps the project focused on reusable developer tooling with measurable behavior instead of a disposable prompt.

Verify Claude Code and Prepare the Workspace

PR Risk Coach depends on Claude Code plugin evals. A stale installation can turn a plugin problem into a version or authentication failure.

This step confirms version 2.1.310 in Windows PowerShell. You will also authenticate Claude Code before creating the empty workspace for your plugin.

In this step, get ready to:

Verify Claude Code version 2.1.310 on Windows.
Prepare a current authenticated Claude Code session.
Create the required empty my-plugin workspace.

Confirm your local toolchain

The version check separates a ready installation from an older build or a missing command. Each outcome has a direct path to 2.1.310.

Press the Windows key to open the search bar.
Type PowerShell into the search bar.
Press Enter to open Windows PowerShell.
Check the installed Claude Code version by running this command:

claude --version

What does this check?

The command asks your local Claude Code executable to print its installed version. The result tells you which setup path to follow.

✔️ I see the required version

You are in good shape. Your output includes 2.1.310, so this installation supports the plugin eval workflow used in this project.

ⓧ I see an older version

An older build needs an update before you create the plugin. Updating now prevents unsupported eval behavior later.

Bring Claude Code up to date by running this command:

claude update

What does this command do?

The claude update command updates the existing Claude Code installation. Keeping the CLI current gives the plugin commands the release they expect.

Confirm the updated version by running this command:

claude --version

What confirms the update?

The new output should include 2.1.310. That version confirms the update reached the required release.

Still seeing an older version?

Close any other Claude Code sessions before retrying the update.
Review the official Claude Code installation guide for the supported Windows installation path.
Use this prompt to help me diagnose why Claude Code still reports an older version.

ⓧ Command not found

PowerShell cannot find a Claude Code installation yet. The native Windows installer adds the CLI to your computer.

Installing a command-line tool can feel fiddly. This command comes from the official Claude Code installation instructions for Windows PowerShell.

Install the native Windows build by running this command:

irm https://claude.ai/install.ps1 | iex

What does this command do?

PowerShell downloads the official Claude Code installation script. It passes that script to PowerShell for execution on your computer.

Close the current Windows PowerShell window.
Press the Windows key to open the search bar.
Type PowerShell into the search bar.
Press Enter to open a new Windows PowerShell window.
Check the installed version by running this command:

claude --version

What confirms the installation?

The command should now print a version containing 2.1.310. That output proves PowerShell can locate the new installation.

Installation command failed?

Open a fresh Windows PowerShell window before retrying the version check.
Review the official Claude Code installation guide for current Windows requirements.
Use this prompt to help me troubleshoot the native Windows installation.

That clears the environment check: your machine now reports the required Claude Code release.

Update and authenticate Claude Code

A current executable solves the version requirement. An authenticated session proves the CLI can use your Claude Code-supported account before you ask it to create local files.

Check for any available Claude Code update by running this command:

claude update

Why run the update again?

This command gives every setup path the same final update check. It also confirms that the installed CLI can run its built-in update workflow.

Confirm the final version by running this command:

claude --version

What should the version show?

Your output should include 2.1.310. This is the version required by the workspace state for this project.

Update or version check failed?

Close any running Claude Code sessions before retrying the update.
Open a new Windows PowerShell window before checking the version again.
Use this prompt to help me debug the Claude Code update check.

The sign-in flow connects this CLI session to your Claude Code-supported account. If authentication is already active, the session opens directly in PowerShell.

Start a normal Claude Code session by running this command:

claude

What happens when Claude Code starts?

Claude Code checks whether the current session is authenticated. It opens a browser sign-in flow when your account still needs to be connected.

Complete the browser sign-in flow if it appears.
Switch back to Windows PowerShell after the browser confirms sign-in.

You are through the account checkpoint. The Claude Code prompt in PowerShell confirms that your session is authenticated.

Unable to reach the Claude Code prompt?

Finish any browser sign-in page that remains open.
Start a fresh Claude Code session after the browser completes authentication.
Use this prompt to help me troubleshoot Claude Code authentication.

Create the empty plugin workspace

A Claude Code plugin uses a predictable directory structure to separate its manifest, reusable instructions, specialist agents, and evaluation cases. Starting with empty directories makes the structure visible before any component content is added.

Claude Code may request permission before creating directories on your Desktop. That confirmation keeps local file changes under your control.

Create the empty workspace by entering this prompt at the Claude Code prompt:

Create an empty my-plugin workspace on my Desktop. Include these directories: .claude-plugin, skills/review-change, agents, and evals/review-unsafe-change/graders. Do not create any files.

Why this folder shape?

The .claude-plugin directory will hold the plugin manifest. The skills/review-change directory reserves a home for the reusable Skill.

The agents directory will hold the specialist subagent. The evals/review-unsafe-change/graders directory reserves the evaluation case and its graders.

Approve each requested directory operation if Claude asks for permission.

Claude confirms that it created the directory structure on your Desktop. That response is your first visible workspace result.

Before you display the tree, which four top-level branches do you expect to see under my-plugin?

Ask Claude Code to display the workspace by entering this prompt:

Print the complete directory tree for my-plugin, including empty directories.

What does this check prove?

Printing the tree confirms the directory names and their nesting. Including empty directories ensures that every required branch appears in the result.

You should see my-plugin containing .claude-plugin, skills/review-change, agents, and evals/review-unsafe-change/graders. Each directory should still be empty.

Is a directory missing from the tree?

Ask Claude Code to create the missing directory inside my-plugin.
Ask Claude Code to print the complete tree again after the correction.
Use this prompt to help me fix my empty plugin directory tree.

The empty my-plugin tree is now visible from an authenticated Claude Code session. Next, you will turn that scaffold into the first working PR Risk Coach Skill.

Build the Skill-Only Reviewer

You now have a clean plugin workspace ready for its first working component. Claude Code can discover the folders that you prepared in the previous step.

In this step, you will build a deliberately small reviewer through a reusable Skill. You will run it against an unsafe login change to establish a visible baseline.

In this step, get ready to:

Define the plugin identity in its manifest.
Add a temporary Skill that returns three general review bullets.
Demonstrate the temporary reviewer against an unsafe login change.

Define the plugin identity

A plugin manifest gives Claude Code the plugin's identity. Its metadata makes the local folder loadable under a stable name.

In the authenticated Claude Code session from earlier, ask it to save the following content as .claude-plugin/plugin.json inside my-plugin:

{
  "name": "pr-risk-coach",
  "description": "Reviews proposed code changes for correctness, security, and test risk using a structured workflow.",
  "version": "1.0.0",
  "author": {
    "name": "Your Name"
  }
}

What does this manifest do?

The JSON object identifies the plugin as pr-risk-coach.
The description tells other users what the plugin reviews.
The 1.0.0 value gives this initial plugin state a version.
The author object records who created the plugin.

Confirm Claude Code reports creating .claude-plugin/plugin.json.
Ask Claude Code to read back .claude-plugin/plugin.json.
Compare the displayed manifest with the block above.

Manifest missing or different?

Confirm the Claude Code session is still working in the directory that contains my-plugin. Check that the file path begins with .claude-plugin.

If the content still differs, help me create the exact plugin manifest.

✔️ Awesome, I've got everything!

Great. Your plugin now has its identity and metadata.

ⓧ I'd like to double check the full code

{
  "name": "pr-risk-coach",
  "description": "Reviews proposed code changes for correctness, security, and test risk using a structured workflow.",
  "version": "1.0.0",
  "author": {
    "name": "Your Name"
  }
}

Add the temporary review Skill

A Skill stores reusable instructions that Claude Code can invoke for a specific task. This temporary version uses $ARGUMENTS to receive the proposed change from the command message.

In the same Claude Code session, ask it to save the following content as skills/review-change/SKILL.md inside my-plugin:

---
name: review-change
description: Reviews a proposed code change or pull request. Use when the user asks for a code review, risk review, pull request review, or change assessment.
---

# PR Risk Coach

Review the proposed change supplied in `$ARGUMENTS` or in the user's message.

Return exactly three concise general review bullets. Focus on correctness, maintainability, and tests. Do not use a security subagent or a structured report template yet.

How does this Skill work?

The frontmatter names the Skill review-change.
The description tells Claude Code which review requests match this Skill.
The $ARGUMENTS placeholder receives the change supplied after the Skill command.
The final instruction limits the response to exactly three general bullets.

Confirm Claude Code reports creating skills/review-change/SKILL.md.
Ask Claude Code to read back skills/review-change/SKILL.md.
Compare the displayed Skill with the block above.

Skill file missing or incomplete?

Check that SKILL.md uses uppercase letters in its filename. Confirm it sits inside skills/review-change.

If Claude Code omitted part of the file, help me verify the temporary review Skill.

✔️ Awesome, I've got everything!

Your temporary Skill is ready to receive a proposed change.

ⓧ I'd like to double check the full code

---
name: review-change
description: Reviews a proposed code change or pull request. Use when the user asks for a code review, risk review, pull request review, or change assessment.
---

# PR Risk Coach

Review the proposed change supplied in `$ARGUMENTS` or in the user's message.

Return exactly three concise general review bullets. Focus on correctness, maintainability, and tests. Do not use a security subagent or a structured report template yet.

Validate and test the reviewer

Validation checks whether Claude Code can recognize the plugin structure before loading it. The temporary Skill can then prove that the command routing works.

Switch back to PowerShell from earlier.
Confirm its current folder contains my-plugin.
Validate the plugin by running this command:

claude plugin validate ./my-plugin --strict

What does this command check?

The command validates the plugin inside ./my-plugin. The --strict flag treats warnings as errors.

You should see a successful validation message. That result proves the manifest and Skill follow the expected plugin structure.

Validation did not pass?

Check both files against the full-code tabs above. Confirm the manifest sits at .claude-plugin/plugin.json.

If validation still fails, help me diagnose my plugin validation result.

Your first validation is complete. The manifest and Skill now form a loadable plugin.

Load the plugin for a new Claude Code session by running this command:

claude --plugin-dir ./my-plugin

What does session loading do?

The --plugin-dir option loads ./my-plugin for this session. You can now invoke its namespaced Skill from the Claude Code prompt.

You will see the Claude Code prompt inside the new session. Keep this session running for the review test.

Plugin session did not start?

Confirm PowerShell is still in the directory that contains my-plugin. Check that the validation command passed before trying the session-load command again.

If the session still does not start, help me load my local plugin.

Before you run the review, form a prediction about whether three general bullets can make the merge risk easy to judge.

At the Claude Code prompt, type this command without pressing Enter:

/pr-risk-coach:review-change

How is the Skill selected?

The pr-risk-coach namespace identifies the plugin. The review-change suffix identifies the Skill inside it.

Paste the following unsafe login change into the same command message:

function login(username, password) {
  const query = "SELECT * FROM users WHERE username = '" + username + "' AND password = '" + password + "'";
  return db.query(query);
}

What is the reviewer receiving?

The username value is concatenated directly into the SQL query. The password value follows the same pattern.

This gives the Skill concrete code to assess through $ARGUMENTS.

Press Enter to run the review.

You will receive exactly three concise general review bullets. The response proves that the plugin command routes the proposed change into the temporary Skill.

The output has no stable risk level. It also lacks dedicated security findings, test gaps, or a merge recommendation.

Review command missing or output incorrect?

Confirm the session was started with the local plugin directory. Check that the Skill file contains the instruction to return exactly three bullets.

If the command still does not work, help me debug the temporary review Skill.

That baseline is working. Next, you will add a specialized security reviewer and turn the shallow bullets into a dependable risk report.

Add a Specialized Security Reviewer

Your Claude Code plugin already reviews proposed changes. Its temporary Skill keeps every result to three general bullets.

That shallow review cannot separate security analysis from general feedback. In this step, a read-only subagent takes ownership of security analysis while the Skill enforces a dependable report contract.

In this step, get ready to:

Create a read-only security specialist with a focused finding format.
Build a delegated review workflow with an exact report contract.
Reload the plugin to verify the structured review against the unsafe login change.

Isolate the security analysis

A specialist context keeps security review focused on evidence from the proposed change. Read-only tools also prevent the specialist from modifying project files during analysis.

Return to the Claude Code session from earlier.
Tell Claude Code to create agents/security-reviewer.md with the exact content below:

---
name: security-reviewer
description: Reviews proposed code changes for security risks. Use during PR Risk Coach reviews involving authentication, authorization, input handling, secrets, queries, or sensitive data.
tools: Read, Grep, Glob
---

You are a read-only application security reviewer.

Analyze only the proposed change supplied in the delegation message. Check for:

- Injection risks
- Authentication or session weaknesses
- Missing authorization checks
- Exposed secrets or sensitive data
- Missing input validation
- Unsafe error handling or data disclosure

For every finding, return:

- Severity: LOW, MEDIUM, HIGH, or CRITICAL
- Evidence: the exact behavior or line from the proposed change
- Impact: what could go wrong
- Fix: the smallest concrete remediation
- Test: one test that would verify the fix

If the supplied change does not provide enough evidence, label the item as an assumption. Do not edit files and do not claim a vulnerability without evidence.

What does this agent definition do?

The frontmatter gives the specialist its security-reviewer identity.
The tools field limits the specialist to Read, Grep, and Glob.
The checklist focuses analysis on six common security risk areas.
The finding format requires evidence plus a concrete remediation.

Ask Claude Code to report the path of the file it created.
Ask Claude Code to list the tools allowed by the new agent.

You should see agents/security-reviewer.md plus the tools Read, Grep, and Glob. That gives PR Risk Coach a focused specialist without file-editing access.

Agent file missing or misplaced?

Confirm the file sits directly inside the existing agents directory.
Check that the filename is exactly security-reviewer.md.

If Claude Code created the file elsewhere, help me place the security reviewer in the correct plugin directory.

✔️ Awesome, I've got everything!

The security specialist is ready. Its scope stays focused on the proposed change.

ⓧ I'd like to double check the full code

---
name: security-reviewer
description: Reviews proposed code changes for security risks. Use during PR Risk Coach reviews involving authentication, authorization, input handling, secrets, queries, or sensitive data.
tools: Read, Grep, Glob
---

You are a read-only application security reviewer.

Analyze only the proposed change supplied in the delegation message. Check for:

- Injection risks
- Authentication or session weaknesses
- Missing authorization checks
- Exposed secrets or sensitive data
- Missing input validation
- Unsafe error handling or data disclosure

For every finding, return:

- Severity: LOW, MEDIUM, HIGH, or CRITICAL
- Evidence: the exact behavior or line from the proposed change
- Impact: what could go wrong
- Fix: the smallest concrete remediation
- Test: one test that would verify the fix

If the supplied change does not provide enough evidence, label the item as an assumption. Do not edit files and do not claim a vulnerability without evidence.

How to compare this file

Check the frontmatter first. Then confirm the six risk areas plus the five finding fields match your file exactly.

Build the report contract

The main Skill now needs two responsibilities. It delegates the complete proposed change to the specialist before assembling both reviews into one predictable structure.

Tell Claude Code to replace the contents of skills/review-change/SKILL.md with this first part:

---
name: review-change
description: Reviews a proposed code change, patch, or pull request for correctness, security, and missing tests. Use when the user asks for a code review, risk review, pull request review, or change assessment.
---

# PR Risk Coach

Review the proposed change supplied in `$ARGUMENTS` or in the user's message.

Follow this workflow:

1. Identify the intended behavior and the most important correctness risks.
2. Delegate the security analysis to the `pr-risk-coach:security-reviewer` subagent. Give it the complete proposed change and ask it to return concise findings with evidence and fixes.
3. Combine your correctness review and the subagent's security review into exactly this structure:

# PR RISK COACH REPORT

## Risk Level
State `LOW`, `MEDIUM`, `HIGH`, or `CRITICAL` and give one sentence of evidence.

## Blocking Issues
List defects that should block merging. Write `None found` when there are none.

## Security Findings
List each security finding with severity, evidence from the change, and a concrete fix. Write `None found` when there are none.

What does the first part establish?

The updated description broadens routing to proposed patches plus missing-test reviews.
The workflow delegates the complete proposed change to pr-risk-coach:security-reviewer.
The report begins with a stable marker plus sections for risk level, blockers, and security findings.

Tell Claude Code to append the remaining content after one blank line:

## Test Gaps
List the tests needed before merging.

## Recommendation
State `MERGE`, `MERGE AFTER FIXES`, or `DO NOT MERGE`, followed by the next action.

Rules:

- Base every finding on the supplied change. Do not invent surrounding code.
- Distinguish confirmed defects from assumptions.
- Keep the report concise and actionable.
- Never edit files while performing a review.

What completes the contract?

The final sections identify missing tests plus the next merge action.
The rules keep findings grounded in the supplied change.
The read-only rule prevents review work from changing files.

Ask Claude Code to print the report headings from the completed Skill in order.

You should see PR RISK COACH REPORT followed by Risk Level, Blocking Issues, Security Findings, Test Gaps, and Recommendation.

Headings missing or out of order?

Confirm the second chunk starts beneath the Security Findings instructions.
Check that every report heading begins with two hash characters.
Check that the report marker begins with one hash character.

If the assembled structure still differs, help me compare my review-change Skill with the required report contract.

✔️ Awesome, I've got everything!

The Skill now delegates specialist analysis before producing one consistent review report.

ⓧ I'd like to double check the full code

---
name: review-change
description: Reviews a proposed code change, patch, or pull request for correctness, security, and missing tests. Use when the user asks for a code review, risk review, pull request review, or change assessment.
---

# PR Risk Coach

Review the proposed change supplied in `$ARGUMENTS` or in the user's message.

Follow this workflow:

1. Identify the intended behavior and the most important correctness risks.
2. Delegate the security analysis to the `pr-risk-coach:security-reviewer` subagent. Give it the complete proposed change and ask it to return concise findings with evidence and fixes.
3. Combine your correctness review and the subagent's security review into exactly this structure:

# PR RISK COACH REPORT

## Risk Level
State `LOW`, `MEDIUM`, `HIGH`, or `CRITICAL` and give one sentence of evidence.

## Blocking Issues
List defects that should block merging. Write `None found` when there are none.

## Security Findings
List each security finding with severity, evidence from the change, and a concrete fix. Write `None found` when there are none.

## Test Gaps
List the tests needed before merging.

## Recommendation
State `MERGE`, `MERGE AFTER FIXES`, or `DO NOT MERGE`, followed by the next action.

Rules:

- Base every finding on the supplied change. Do not invent surrounding code.
- Distinguish confirmed defects from assumptions.
- Keep the report concise and actionable.
- Never edit files while performing a review.

How to compare this Skill

Confirm the delegation uses the namespaced specialist. Then compare the report marker, heading order, fallback text, and final rules.

Verify the delegated workflow

Plugin validation checks the package before you reload it. The final review then proves that the new Skill contract changes the visible result.

Return to the Windows PowerShell window in the directory that contains my-plugin.
Validate the completed plugin by running this command:

claude plugin validate ./my-plugin --strict

What does this command check?

The validator checks the plugin package in my-plugin. The --strict flag treats warnings as errors.

You should see ✔ Validation passed. Excellent work. Your plugin package now includes a valid specialist plus the final review Skill.

Validation did not pass?

Check that security-reviewer.md is directly inside my-plugin/agents.
Check that SKILL.md is directly inside my-plugin/skills/review-change.
Check each frontmatter block for matching opening plus closing separators.

If validation still fails, help me diagnose my PR Risk Coach plugin validation result.

Switch back to the authenticated Claude Code session that already loaded my-plugin.
Reload the updated plugin by entering:

/reload-plugins

What does reloading change?

This refreshes plugin components inside the running session. The session can now use the updated Skill plus the new security specialist.

Before you run the review, do you expect the same three general bullets or a report with stable sections?

Begin one Claude Code message with this Skill invocation:

/pr-risk-coach:review-change

What does this invocation select?

The namespaced command selects the review-change Skill from the pr-risk-coach plugin.

Paste this unsafe login change directly beneath the Skill invocation in the same message:

function login(username, password) {
  const query = "SELECT * FROM users WHERE username = '" + username + "' AND password = '" + password + "'";
  return db.query(query);
}

Why reuse this change?

This is the same proposed login change used for the shallow review. Reusing it keeps the comparison focused on the plugin behavior.

Send the completed review message.

You should see a response beginning with PR RISK COACH REPORT. The response should contain all five required sections.

What changed from the shallow review?

The earlier result contained three general bullets without a dependable structure. The updated result separates risk level, blocking issues, security findings, test gaps, and recommendation.

Security analysis now comes from the specialized read-only agent. The main Skill combines that analysis with its correctness review.

Still seeing three general bullets?

Run /reload-plugins again in the loaded Claude Code session.
Confirm skills/review-change/SKILL.md no longer contains the temporary three-bullet instruction.
Confirm the delegation names pr-risk-coach:security-reviewer exactly.

If the old behavior remains, help me find why my reloaded plugin still uses the temporary Skill.

Your reviewer now separates specialist security analysis from the main conversation. Next, you will measure this behavior against a no-plugin baseline.

Evaluate the Plugin Behavior

Your structured review proves that the plugin works in one conversation. A single successful demo cannot show which behavior comes from the plugin.

In this step, Claude Code runs the same unsafe change with the plugin loaded. It also runs the change without the plugin. Deterministic graders turn that comparison into repeatable evidence.

In this step, get ready to:

Define a repeatable unsafe-login eval case.
Add deterministic graders for the report contract and workflow routing.
Produce a documented baseline comparison with a local HTML report.

Create the repeatable eval case

An eval case gives Claude Code the same prompt on every run. This case uses the unsafe login change you reviewed earlier so the plugin must identify a concrete SQL injection risk.

Create evals/review-unsafe-change/prompt.md inside the existing my-plugin workspace by giving Claude Code the content below:

---
max_turns: 10
allowed_tools: [Skill, Agent]
---

Review this proposed login change before it is merged:

```javascript
function login(username, password) {
  const query = "SELECT * FROM users WHERE username = '" + username + "' AND password = '" + password + "'";
  return db.query(query);
}
```

What does this eval prompt do?

The frontmatter limits the case to ten turns so the run stays focused.
The allowed tools let the plugin invoke its Skill and security Agent.
The login function concatenates untrusted credentials into a SQL query. This gives the review workflow specific evidence to assess.

Ask Claude Code to read back evals/review-unsafe-change/prompt.md from the saved workspace.

You should see the ten-turn limit, the two allowed tools, and the complete unsafe login function.

Is the prompt incomplete?

Check that the file is named prompt.md inside evals/review-unsafe-change. Confirm that both frontmatter fences use three hyphens.

If the nested code fence was truncated, help me compare my eval prompt with the required file.

The first grader checks the observable report contract. Its pattern requires the marker and all five section headings to appear in order.

Create evals/review-unsafe-change/graders/report-contract.md by giving Claude Code the content below:

---
type: regex
pattern: 'PR RISK COACH REPORT[\s\S]*Risk Level[\s\S]*Blocking Issues[\s\S]*Security Findings[\s\S]*Test Gaps[\s\S]*Recommendation'
flags: i
target: last_message
---

How does the contract grader work?

The regex grader inspects the final response from the case.
The pattern accepts any content between the required headings.
The case passes this check only when every report section appears in the required order.

Ask Claude Code to read back evals/review-unsafe-change/graders/report-contract.md from the saved workspace.

You should see one pattern containing the report marker and all five required section names.

Does the pattern look different?

Check every [\s\S]* segment carefully. A missing backslash changes what the grader can match.

If the pattern changed while it was pasted, help me find the mismatch in my report-contract grader.

Why use deterministic graders?

The report grader checks a fixed output contract. The tool graders inspect recorded workflow calls. These checks avoid an additional judge-model grader while still testing output and routing behavior.

The next grader proves that the reusable review Skill handled the request. It accepts either the namespaced Skill name or the unqualified Skill name.

Create evals/review-unsafe-change/graders/skill-fired.md by giving Claude Code the content below:

---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?review-change"'
---

What does the Skill grader prove?

The tool_used grader examines the recorded tool calls.
The input pattern looks specifically for the review-change Skill.
A polished report cannot pass this routing check unless the expected Skill was invoked.

Ask Claude Code to read back evals/review-unsafe-change/graders/skill-fired.md from the saved workspace.

You should see Skill as the tool and review-change at the end of the input pattern.

Is the Skill grader missing its pattern?

Confirm that input_match remains inside the frontmatter. Check that the pattern still includes the optional namespace group.

If the escaped characters are unclear, help me check my Skill invocation grader.

The final grader checks the specialist boundary. It requires the plugin arm to delegate through the exact namespaced security Agent.

Create evals/review-unsafe-change/graders/security-agent-fired.md by giving Claude Code the content below:

---
type: tool_used
tool: Agent
input_match: '"subagent_type"\s*:\s*"pr-risk-coach:security-reviewer"'
arm: with-only
---

What does the Agent grader prove?

The grader checks that the Agent tool receives pr-risk-coach:security-reviewer as its subagent type.
The with-only setting keeps this plugin-specific requirement out of the no-plugin score.
Together with the Skill grader, this verifies the intended delegation path.

Ask Claude Code to read back evals/review-unsafe-change/graders/security-agent-fired.md from the saved workspace.

You should see the namespaced security Agent in the input pattern and with-only as the arm.

Is the Agent grader targeting the wrong name?

Confirm that the input pattern uses the full pr-risk-coach:security-reviewer identity. The plugin namespace is part of the expected tool input.

If the grader points at a different Agent, help me fix my security Agent delegation grader.

Document the plugin workflow

A local tool becomes reusable when another developer can validate it, load it, and reproduce its evidence. The README captures that complete path without requiring Git or GitHub.

Create README.md inside my-plugin by giving Claude Code the first part below:

# PR Risk Coach

PR Risk Coach is a Claude Code plugin that reviews proposed code changes through a reusable Skill and a specialized, read-only security subagent.

## Requirements

- Claude Code 2.1.269 or later
- An authenticated Claude Code session
- Windows PowerShell for the documented local workflow

## Validate

From the directory that contains `my-plugin`:

```powershell
claude plugin validate ./my-plugin --strict
```

A valid plugin prints `Validation passed`.

## Load for one session

From the directory that contains `my-plugin`:

```powershell
claude --plugin-dir ./my-plugin
```

What does this README section cover?

The requirements state the minimum Claude Code version as 2.1.269.
The validation section gives a strict structural check for the plugin.
The loading section keeps the plugin scoped to one Claude Code session.

Ask Claude Code to read back the headings in the saved README.md.

You should see Requirements, Validate, and Load for one session beneath the project introduction.

Is the README formatting broken?

Check that the PowerShell commands remain inside fenced code blocks. Confirm that every heading begins on a new line.

If the first section renders as one paragraph, help me repair the Markdown structure in my README.

The remaining sections connect the usage instructions to the eval evidence. They also explain the engineering capabilities demonstrated by the project.

Append the remaining content below to README.md in the Claude Code session:

## Run the review

At the Claude Code prompt, run:

```text
/pr-risk-coach:review-change
```

Paste a proposed code change in the same message. The plugin returns a risk level, blocking issues, security findings, test gaps, and a merge recommendation.

## Run the eval

Open PowerShell in the `my-plugin` root and run:

```powershell
claude plugin eval .
```

The suite runs with and without the plugin. The graders check that the report follows the required contract, that the review Skill was invoked, and that the security subagent received the specialized analysis.

## What this project demonstrates

- Reusable Claude Code plugin packaging
- Skill routing and structured output design
- Isolated specialist analysis with a custom subagent
- Behavioral testing against a no-plugin baseline
- Deterministic grading without an additional judge-model grader

What does the completed README explain?

The review section shows how to invoke the namespaced Skill.
The eval section shows how to run the suite from the plugin root.
The final list connects the artifact to reusable AI workflow engineering.

Ask Claude Code to read back the complete saved README.md.

You should see six sections ending with five project-demonstration bullets.

Is the README missing its final sections?

Confirm that the appended content starts with ## Run the review. Check that it ends with the deterministic grading bullet.

If part of the file was replaced during the append, help me reconstruct my complete PR Risk Coach README.

Before running the suite, compare the cumulative project files with the expected end state.

✔️ Awesome, I've got everything!

Great. Double check that every new file is saved inside my-plugin.

ⓧ I'd like to double check the full code

{
  "name": "pr-risk-coach",
  "description": "Reviews proposed code changes for correctness, security, and test risk using a structured workflow.",
  "version": "1.0.0",
  "author": {
    "name": "Your Name"
  }
}

# PR Risk Coach

PR Risk Coach is a Claude Code plugin that reviews proposed code changes through a reusable Skill and a specialized, read-only security subagent.

## Requirements

- Claude Code 2.1.269 or later
- An authenticated Claude Code session
- Windows PowerShell for the documented local workflow

## Validate

From the directory that contains `my-plugin`:

```powershell
claude plugin validate ./my-plugin --strict
```

A valid plugin prints `Validation passed`.

## Load for one session

From the directory that contains `my-plugin`:

```powershell
claude --plugin-dir ./my-plugin
```

## Run the review

At the Claude Code prompt, run:

```text
/pr-risk-coach:review-change
```

Paste a proposed code change in the same message. The plugin returns a risk level, blocking issues, security findings, test gaps, and a merge recommendation.

## Run the eval

Open PowerShell in the `my-plugin` root and run:

```powershell
claude plugin eval .
```

The suite runs with and without the plugin. The graders check that the report follows the required contract, that the review Skill was invoked, and that the security subagent received the specialized analysis.

## What this project demonstrates

- Reusable Claude Code plugin packaging
- Skill routing and structured output design
- Isolated specialist analysis with a custom subagent
- Behavioral testing against a no-plugin baseline
- Deterministic grading without an additional judge-model grader

---
name: security-reviewer
description: Reviews proposed code changes for security risks. Use during PR Risk Coach reviews involving authentication, authorization, input handling, secrets, queries, or sensitive data.
tools: Read, Grep, Glob
---

You are a read-only application security reviewer.

Analyze only the proposed change supplied in the delegation message. Check for:

- Injection risks
- Authentication or session weaknesses
- Missing authorization checks
- Exposed secrets or sensitive data
- Missing input validation
- Unsafe error handling or data disclosure

For every finding, return:

- Severity: LOW, MEDIUM, HIGH, or CRITICAL
- Evidence: the exact behavior or line from the proposed change
- Impact: what could go wrong
- Fix: the smallest concrete remediation
- Test: one test that would verify the fix

If the supplied change does not provide enough evidence, label the item as an assumption. Do not edit files and do not claim a vulnerability without evidence.

---
type: regex
pattern: 'PR RISK COACH REPORT[\s\S]*Risk Level[\s\S]*Blocking Issues[\s\S]*Security Findings[\s\S]*Test Gaps[\s\S]*Recommendation'
flags: i
target: last_message
---

---
type: tool_used
tool: Agent
input_match: '"subagent_type"\s*:\s*"pr-risk-coach:security-reviewer"'
arm: with-only
---

---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?review-change"'
---

---
max_turns: 10
allowed_tools: [Skill, Agent]
---

Review this proposed login change before it is merged:

```javascript
function login(username, password) {
  const query = "SELECT * FROM users WHERE username = '" + username + "' AND password = '" + password + "'";
  return db.query(query);
}
```

---
name: review-change
description: Reviews a proposed code change, patch, or pull request for correctness, security, and missing tests. Use when the user asks for a code review, risk review, pull request review, or change assessment.
---

# PR Risk Coach

Review the proposed change supplied in `$ARGUMENTS` or in the user's message.

Follow this workflow:

1. Identify the intended behavior and the most important correctness risks.
2. Delegate the security analysis to the `pr-risk-coach:security-reviewer` subagent. Give it the complete proposed change and ask it to return concise findings with evidence and fixes.
3. Combine your correctness review and the subagent's security review into exactly this structure:

# PR RISK COACH REPORT

## Risk Level
State `LOW`, `MEDIUM`, `HIGH`, or `CRITICAL` and give one sentence of evidence.

## Blocking Issues
List defects that should block merging. Write `None found` when there are none.

## Security Findings
List each security finding with severity, evidence from the change, and a concrete fix. Write `None found` when there are none.

## Test Gaps
List the tests needed before merging.

## Recommendation
State `MERGE`, `MERGE AFTER FIXES`, or `DO NOT MERGE`, followed by the next action.

Rules:

- Base every finding on the supplied change. Do not invent surrounding code.
- Distinguish confirmed defects from assumptions.
- Keep the report concise and actionable.
- Never edit files while performing a review.

Run the baseline comparison

The eval now has a prompt, deterministic graders, and documentation. A strict validation catches structural mistakes before real model calls begin.

Switch back to Windows PowerShell from earlier.
Move to the directory that contains my-plugin.
Validate the completed plugin by running:

claude plugin validate ./my-plugin --strict

What should validation confirm?

Strict validation checks the completed plugin structure. You should see ✔ Validation passed before starting the eval.

Does validation fail?

Check that prompt.md sits directly inside evals/review-unsafe-change. Confirm that all three grader files sit inside its graders directory.

If the file layout looks correct, help me diagnose my strict plugin validation failure.

Eval runs consume usage

The default eval runs this case three times with the plugin. It also runs the case three times without the plugin. These are real model calls that count against your Claude Code plan usage or API bill.

The deterministic graders do not add judge-model calls. The six agent runs still consume usage.

Move into the my-plugin root in Windows PowerShell.

Before you run the suite, which arm do you expect to follow the full report contract and specialist delegation path?

Run every eval case from the plugin root by running:

claude plugin eval .

What does the eval reveal?

The completed summary shows the review-unsafe-change case with WITH, W/OUT, and Δ columns. The delta isolates the behavior added by your plugin.

Claude Code also writes an HTML report under evals/results/<timestamp>/report.html. That local report contains the case-level evidence behind the summary.

Open the generated report.html path shown in the eval output.
Select the review-unsafe-change case card.
Confirm that the report contract grader passes in the plugin arm.
Confirm that the Skill invocation grader records review-change.
Confirm that the Agent grader records pr-risk-coach:security-reviewer.

Is the eval unavailable or failing?

If the eval feature is unavailable server-side, no local setting can enable it. Confirm Claude Code is still on 2.1.310. Retry later in a fresh authenticated session.

If the case loads but a grader fails, inspect that grader in the HTML report. Help me interpret my PR Risk Coach eval failure.

You now have repeatable evidence that PR Risk Coach changes Claude Code's behavior. The local report shows the structured output and specialist routing that disappear from the no-plugin baseline.

Secret mission

Teach the Plugin When to Stay Silent

Your plugin already proves when it should review a risky change. Now test the opposite behavior by giving it an ordinary writing request. You will add a negative eval that proves the review workflow stays silent.

Clean Up Your Resources

Keeping the local my-plugin folder creates no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

Local my-plugin workspace.

Keep everything running

No action needed. Choose this if you are still actively building or want to keep the plugin as a local demonstration project.

Retain the my-plugin folder.
Review the existing local report.html when you want to compare the two eval cases.
Rerun the eval suite only when you want fresh behavioral evidence.

Existing files do not consume model usage. Each new eval run counts against your Claude Code plan usage or API bill.

Pause - I'll come back to this later

Stop the active session to free up memory. Your local project remains ready for later.

End the active Claude Code session in the PowerShell window from earlier.
Leave the my-plugin folder on your computer.
Follow the validation and session-load sections in README.md when you return.

All project files remain ready for your next session.

Delete - I don't want to use this again

Removing my-plugin is permanent. Your Claude Code setup stays intact.

End the active Claude Code session in the PowerShell window from earlier.
Leave the my-plugin directory and remove it by running these commands:

Set-Location ..
Remove-Item -Recurse -Force .\my-plugin

What do these commands do?

Set-Location moves PowerShell to the parent directory so the project folder is no longer the current location.
Remove-Item recursively deletes the entire my-plugin folder.
-Force includes hidden metadata such as .claude-plugin.

Display the directory that previously contained my-plugin in your file browser.
Confirm that my-plugin is no longer listed.

Folder still present?

Close any editor that has a file from my-plugin open.
End any terminal process still running from inside my-plugin.
Repeat the deletion commands above.

Help me troubleshoot this folder deletion.

Nice Work!

Excellent work. Your evaluated local Claude Code plugin now turns proposed changes into consistent risk reports.

You've learned how to:

Package a reusable Skill inside PR Risk Coach. Invoke /pr-risk-coach:review-change to receive the complete risk-report contract.
Expose the visible limit of a shallow three-bullet review. Strengthen the workflow with an isolated read-only subagent named security-reviewer.
Measure plugin behavior against a no-plugin baseline. Use deterministic graders to prove Skill routing. Inspect the local eval report for evidence of security delegation.
Complete the optional Secret Mission by proving routing precision. An unrelated calculator README request makes zero review-change Skill calls. The PR RISK COACH REPORT marker stays absent.

Ready to quiz yourself?