Build an AI CI Doctor

Use GitHub Copilot CLI to diagnose and repair a failing Python CI release gate.

Introduction

30 Second Summary

A tiny calculation mistake can stop a software release while the useful clue stays buried in a wall of failed checks. The real skill is proving what broke before changing the code.

In this project, you will build a Python release gate that turns failed checks into a visible DEPLOY or BLOCK decision. You will trace a hidden percentage bug from a red GitHub Actions run to a human-reviewed repair proposed by GitHub Copilot CLI.

What You'll Build

You'll show a green GitHub Actions run whose incident report traces a once-hidden 5 percent boundary defect to the reviewed fix.

By the end of this project, you'll have:

  • A runnable release gate that prints DEPLOY or BLOCK from the number of failed checks.
  • A public red-to-green pipeline where boundary tests expose the defect before your reviewed fix clears every check.
  • An evidence-backed incident report that documents the failed tests through a clear root cause, resolution, and prevention plan.
  • Secret Mission: a tested guard that rejects an empty check set with an explicit error.

Are there any prerequisites?

You should be comfortable with Python plus CI/CD. You also need a Windows computer with a GitHub account that includes GitHub Copilot access.

Before We Start

Before the investigation begins, lock in the standard that guides every decision. Your release gate protects deployments from exceeding a 5 percent failed-check budget. Any AI-proposed repair needs your review before you push it.

Prepare Your Investigation Workspace

A reliable investigation starts from a known local environment. Otherwise, missing tools can look like defects in the release gate.

In this step, you will prepare a dedicated Windows workspace. You will also authenticate the investigation assistant before collecting any pipeline evidence.

In this step, get ready to:
  • Verify PowerShell 7 on your Windows device.
  • Create the local investigation workspace.
  • Verify the command-line tools required for the investigation.
Verify PowerShell 7

PowerShell 7 provides the command-line environment for this investigation. The pwsh command reveals whether the required shell is available.

  • Press the Windows key to open the Windows search bar.
  • Type PowerShell into the search bar.
  • Press Enter to open the terminal.
  • Check whether PowerShell 7 is available by running this command:
pwsh

What does this command do?

The command starts PowerShell 7 when it is installed. The startup banner shows which version opened.

Use the startup result to choose the matching path below.

✔️ PowerShell 7 opens

The startup banner identifies PowerShell version 7 or later. Your required shell is ready.

  • Keep this PowerShell 7 window open for the workspace commands.

ⓧ An older version opens

The startup banner shows a version below 7. Install the current stable PowerShell package before continuing.

  • Install PowerShell 7 by running this command:
winget install --id Microsoft.PowerShell --source winget

What does this command do?

WinGet installs the current stable PowerShell package from its configured source. Follow any source agreement prompt shown in the terminal.

  • Start the newly installed shell by running this command:
pwsh

What does this command do?

The command starts the installed PowerShell shell. The new startup banner should identify version 7 or later.

Still seeing the older version?

Close the existing terminal after the installation completes. Start a fresh terminal from the Windows search bar before retrying pwsh.

Ask for help with the active installation: Help me verify why PowerShell 7 is not opening after installation.

ⓧ The command is unavailable

The unavailable command means PowerShell 7 needs to be installed before the investigation continues.

  • Install PowerShell 7 by running this command:
winget install --id Microsoft.PowerShell --source winget

What does this command do?

WinGet installs the current stable PowerShell package from its configured source. Follow any source agreement prompt shown in the terminal.

  • Start the installed shell by running this command:
pwsh

What does this command do?

The command starts PowerShell 7 after the installation completes. The startup banner should identify version 7 or later.

PowerShell still unavailable?

Close the existing terminal after the installation completes. A fresh terminal session can detect the newly installed command.

Ask for help with the missing command: Help me diagnose why pwsh is unavailable after installing PowerShell.

Your PowerShell environment is ready. Later commands now have a consistent Windows shell.

Create the workspace structure

The ai-ci-doctor folder keeps the release gate beside its tests and workflow definition. A predictable structure gives later investigation commands a known place to work.

  • Create the project structure on your Desktop by running these commands in PowerShell 7:
Set-Location -Path "$HOME\Desktop"
New-Item -Path "ai-ci-doctor" -ItemType Directory
Set-Location -Path "ai-ci-doctor"
New-Item -Path "tests" -ItemType Directory
New-Item -Path ".github" -ItemType Directory
New-Item -Path ".github\workflows" -ItemType Directory
New-Item -Path "tests\__init__.py" -ItemType File

What do these commands create?

  • The first command moves PowerShell to your Desktop.
  • The next commands create the ai-ci-doctor folder plus its nested test and workflow directories.
  • The final command creates an empty tests/__init__.py package file.

PowerShell lists the directories and file as it creates them. The terminal prompt now points to the ai-ci-doctor folder on your Desktop.

Workspace item already exists?

An existing item message means part of this workspace is already on your Desktop. Inspect that folder before replacing anything.

Ask for help without deleting existing work: Help me inspect my existing ai-ci-doctor folder safely.

Visual Studio Code treats an opened folder as a workspace. This gives the editor and later command-line tools the same project context.

  • Press the Windows key to open the Windows search bar.
  • Type Visual Studio Code into the search bar.
  • Press Enter to open Visual Studio Code.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Select the ai-ci-doctor folder on your Desktop.
  • Click Select Folder.
  • Confirm trust for the folder you created if Visual Studio Code shows a Workspace Trust prompt.

The Explorer view should show tests plus .github/workflows. Expanding tests should reveal the empty __init__.py file.

✔️ Awesome, I've got everything!

Your workspace structure matches the investigation plan. Keep the ai-ci-doctor folder open in Visual Studio Code.

ⓧ I'd like to double check the full code

The complete tests/__init__.py file is empty at this stage.

Why is this file blank?

The empty file marks tests as a Python package. A blank editor is the correct result for this artifact.

Verify and authenticate Copilot CLI

Version checks establish which tools are available before the investigation begins. This baseline keeps tool availability separate from the release-gate defect.

Why verify these tools?

  • Python runs the release gate and its tests.
  • Git records each evidence-backed change.
  • GitHub Copilot CLI connects local files with workflow evidence from GitHub.
  • Switch back to the PowerShell 7 window inside ai-ci-doctor.
  • Verify Python, Git, and GitHub Copilot CLI by running these commands:
python --version
git --version
copilot version

What do these checks prove?

  • The first command prints the installed Python version.
  • The second command prints the installed Git version.
  • The third command prints the installed GitHub Copilot CLI version.

You should see installed version information for all three tools. Use the Copilot CLI result to choose the matching path below.

Python or Git unavailable?

Open a fresh PowerShell 7 window after confirming that your existing Python or Git installation includes command-line access. A new terminal session can detect an updated command path.

Ask for help with the failing prerequisite: Help me diagnose why Python or Git is unavailable in PowerShell 7.

✔️ I see version 1.0.92 or higher

GitHub Copilot CLI reports version 1.0.92 or later. The CLI is ready for authentication.

ⓧ I see an older version

The installed release is below version 1.0.92. Install the stable WinGet package before continuing.

  • Install the stable GitHub Copilot CLI package by running this command:
winget install GitHub.Copilot

What does this command do?

WinGet installs or updates the stable GitHub Copilot CLI package. Follow any source agreement prompt shown in the terminal.

  • Verify the active GitHub Copilot CLI version by running this command:
copilot version

What does this command confirm?

The command prints the active CLI version. You should now see version 1.0.92 or later.

Still seeing the older version?

Close PowerShell after the installation completes. Open a fresh PowerShell 7 window before checking the version again.

Ask for help with the active executable: Help me verify why Copilot CLI still reports an older version.

ⓧ Command not found

GitHub Copilot CLI needs to be installed before it can investigate workflow evidence.

  • Install GitHub Copilot CLI by running this command:
winget install GitHub.Copilot

What does this command do?

WinGet installs the stable GitHub Copilot CLI package. Follow any source agreement prompt shown in the terminal.

  • Verify that PowerShell can find the installed CLI by running this command:
copilot version

What does this command confirm?

The command prints the installed CLI version. You should see version 1.0.92 or later.

Copilot still unavailable?

Close PowerShell after the installation completes. Open a fresh PowerShell 7 window before retrying the version check.

Ask for help with the missing command: Help me diagnose why Copilot CLI is unavailable after installation.

The version checks now establish a reproducible local baseline. The final check connects the CLI to the GitHub identity that gathers workflow evidence.

  • Launch the interactive GitHub Copilot CLI from the ai-ci-doctor folder by running this command:
copilot

What does this command do?

The command starts the interactive Copilot interface in the current project folder. That folder context lets later prompts reference the investigation files.

  • Select the trust option limited to the current ai-ci-doctor folder when the trust prompt appears.
  • Start authentication with this slash command if Copilot asks you to sign in:
/login

What does this command do?

The slash command starts GitHub authentication from the interactive Copilot interface. The browser authorization connects the CLI to your existing GitHub account.

  • Complete the browser authorization with the GitHub account that has Copilot access.
  • Return to the interactive Copilot terminal after authorization succeeds.

Before you check, which GitHub account do you expect Copilot to report?

  • Display the authenticated account by entering this slash command:
/user

What does this command confirm?

The slash command displays the GitHub account associated with the current Copilot session. This ties future evidence gathering to a known identity.

You should see your authenticated GitHub account in the interactive interface. Your earlier checks should show installed version information for Python, Git, and GitHub Copilot CLI.

Wrong account or no account shown?

Restart the login flow from the interactive interface. Confirm that the browser authorization uses the GitHub account with Copilot access.

Ask for help without sharing credentials: Help me correct the authenticated account in GitHub Copilot CLI.

Your local investigation workspace is ready for evidence-driven work. Next, you will build the release gate that produces the first visible deployment decisions.

Build the Release Gate

Your investigation workspace can now capture the evidence that a future CI pipeline needs. A release gate turns failed checks into a visible deployment decision.

The first demo uses inputs far from the 5 percent boundary. Their distance from the boundary conceals a percentage-versus-fraction mismatch.

In this step, get ready to:
  • Create the Python release gate calculation.
  • Print readable decisions for two extreme inputs.
  • Document the local run command plus the test command.
Create the release gate logic

The Python file starts with a configured failure limit. The decision function compares each calculated result with that limit.

  • In the Explorer sidebar of Visual Studio Code, click the New File icon.
  • Type release_gate.py in the filename field.
  • Press Enter to create the file.

You should see release_gate.py directly inside the ai-ci-doctor folder.

  • Add the configured limit plus the first decision function by pasting this code into release_gate.py:
MAX_FAILURE_RATE = 0.05


def should_deploy(failed_checks, total_checks, max_failure_rate=MAX_FAILURE_RATE):
    failure_percent = failed_checks / total_checks * 100
    return failure_percent <= max_failure_rate

What does this code do?

  • The MAX_FAILURE_RATE constant stores the configured release limit as the decimal fraction 0.05.
  • The failure_percent variable converts the failed-check ratio into percentage points.
  • The should_deploy() function compares the calculated value with the configured limit.
  • Save release_gate.py.
  • Confirm that the unsaved indicator disappears from the release_gate.py tab.

File not saving?

Confirm that release_gate.py appears directly inside the ai-ci-doctor folder. Check that the filename ends with .py.

Help me check the location and saved state of release_gate.py.

Add the readable demo

A Boolean value works inside the gate. An investigation needs output that connects each input with a visible DEPLOY or BLOCK decision.

  • In release_gate.py, place your cursor after return failure_percent <= max_failure_rate.
  • Add two blank lines below the function.
  • Paste the readable decision demo below those blank lines:
def describe_decision(failed_checks, total_checks):
    failure_percent = failed_checks / total_checks * 100
    decision = "DEPLOY" if should_deploy(failed_checks, total_checks) else "BLOCK"
    return (
        f"{failed_checks}/{total_checks} checks failed "
        f"({failure_percent:.1f}%): {decision}"
    )


def main():
    for failed_checks, total_checks in ((0, 100), (6, 100)):
        print(describe_decision(failed_checks, total_checks))


if __name__ == "__main__":
    main()

How does the demo work?

  • The describe_decision() function formats the failed-check count as a percentage with one decimal place.
  • The decision variable converts the Boolean result into a readable deployment decision.
  • The main() function evaluates 0 failures out of 100 checks plus 6 failures out of 100 checks.
  • The __name__ == "__main__" entry point starts the demo when you run the file directly.
  • Save release_gate.py.

Before you run the script, do you expect the two extreme inputs to produce the same decision?

  • Run the visible baseline in the PowerShell terminal with this command:
python release_gate.py

What should I see?

The first line should be 0/100 checks failed (0.0%): DEPLOY.

The second line should be 6/100 checks failed (6.0%): BLOCK.

That is your first visible win: both extreme examples now produce plausible release decisions.

Output different from the baseline?

Check that main() contains (0, 100) plus (6, 100). Confirm that the entry point calls main().

Help me compare my release gate with the expected baseline.

✔️ Awesome, I've got everything!

Your saved script contains the configured limit, decision logic, readable output, demonstration inputs, and runnable entry point.

ⓧ I'd like to double check the full code

MAX_FAILURE_RATE = 0.05


def should_deploy(failed_checks, total_checks, max_failure_rate=MAX_FAILURE_RATE):
    failure_percent = failed_checks / total_checks * 100
    return failure_percent <= max_failure_rate


def describe_decision(failed_checks, total_checks):
    failure_percent = failed_checks / total_checks * 100
    decision = "DEPLOY" if should_deploy(failed_checks, total_checks) else "BLOCK"
    return (
        f"{failed_checks}/{total_checks} checks failed "
        f"({failure_percent:.1f}%): {decision}"
    )


def main():
    for failed_checks, total_checks in ((0, 100), (6, 100)):
        print(describe_decision(failed_checks, total_checks))


if __name__ == "__main__":
    main()
Document and verify the baseline

A reproducible investigation gives another person the commands needed to run the same evidence. The README.md file records the visible demo command plus the upcoming test command.

  • In the Explorer sidebar, click the New File icon.
  • Type README.md in the filename field.
  • Press Enter to create the file.

You should see README.md beside release_gate.py in the Explorer sidebar.

  • Document the project purpose plus both local commands by pasting this content into README.md:
# AI CI Doctor

A small Python release gate and GitHub Actions pipeline for practicing evidence-driven, AI-assisted CI failure investigation.

## Run the release gate

```powershell
python release_gate.py
```

## Run the tests

```powershell
python -m unittest
```

What does the README capture?

  • The opening sentence states the repository's investigation purpose.
  • The first command runs the visible release-gate demo.
  • The second command records the standard way to run the boundary tests.
  • Save README.md.
  • Confirm that the unsaved indicator disappears from the README.md tab.

README missing from the Explorer?

Confirm that README.md appears directly inside the ai-ci-doctor folder. Check that the filename ends with .md.

Help me find or correct README.md in my workspace.

✔️ Awesome, I've got everything!

Your README now records the project purpose plus both local commands.

ⓧ I'd like to double check the full code

# AI CI Doctor

A small Python release gate and GitHub Actions pipeline for practicing evidence-driven, AI-assisted CI failure investigation.

## Run the release gate

```powershell
python release_gate.py
```

## Run the tests

```powershell
python -m unittest
```

Before you run the documented command, which decision do you expect for 6 failed checks out of 100?

  • Verify the documented baseline in the PowerShell terminal with this command:
python release_gate.py

What should I see?

You should see 0/100 checks failed (0.0%): DEPLOY on the first line.

You should see 6/100 checks failed (6.0%): BLOCK on the second line.

Your documented command now reproduces the same visible baseline.

Your release gate now looks convincing at both extremes. Next, you will use focused unit tests to challenge the boundary that this demo skips.

Expose the Unit Mismatch

Your release gate now prints sensible decisions for zero failed checks and six failed checks. Those examples sit far from the five percent boundary, so they cannot prove the threshold behaves correctly.

Focused boundary cases use unit testing to inspect values below the threshold, at the threshold, and above the threshold. These cases turn the hidden unit mismatch into evidence you can inspect.

In this step, get ready to:
  • Create boundary tests below the five percent budget, at the budget, and above the budget.
  • Run the full test suite to expose the unit mismatch.
  • Record the invariant that explains both failures.
Create the boundary tests

A boundary test checks behavior at the exact point where a decision changes. Testing one percent, five percent, and six percent gives the release policy a precise contract.

  • In the Explorer sidebar in Visual Studio Code, select the tests folder inside ai-ci-doctor.
  • Create test_release_gate.py with the file creation control at the top of the Explorer sidebar.
  • Add the three boundary cases to tests/test_release_gate.py by copying the code below:
import unittest

from release_gate import should_deploy


class ReleaseGateTests(unittest.TestCase):
    def test_allows_release_below_budget(self):
        self.assertTrue(should_deploy(1, 100))

    def test_allows_release_at_budget(self):
        self.assertTrue(should_deploy(5, 100))

    def test_blocks_release_above_budget(self):
        self.assertFalse(should_deploy(6, 100))

What do these tests prove?

  • The class inherits from unittest.TestCase. This gives each test access to assertions that the test runner evaluates.
  • The first two cases use assertTrue to protect releases below the budget and exactly at the budget.
  • The final case uses assertFalse to protect the block decision above the budget.
  • Save tests/test_release_gate.py.
  • Confirm test_release_gate.py appears under the tests folder in Explorer.

Cannot find the test file?

  • Check that test_release_gate.py sits inside the tests folder.
  • Confirm the filename starts with test_ so test discovery can find it.

Help me check why my boundary test file is missing or undiscoverable.

✔️ Awesome, I've got everything!

Your test file now contains all three boundary cases.

ⓧ I'd like to double check the full code

import unittest

from release_gate import should_deploy


class ReleaseGateTests(unittest.TestCase):
    def test_allows_release_below_budget(self):
        self.assertTrue(should_deploy(1, 100))

    def test_allows_release_at_budget(self):
        self.assertTrue(should_deploy(5, 100))

    def test_blocks_release_above_budget(self):
        self.assertFalse(should_deploy(6, 100))
Run the tests and record the invariant

The suite now asks the release gate to decide at one percent, five percent, and six percent. Before you run it, do you expect all three decisions to match the five percent policy?

  • Run the full boundary suite from the PowerShell terminal in ai-ci-doctor by using this command:
python -m unittest

You'll see three tests run. The one percent and five percent allowance tests fail.

The six percent blocking test passes. That failing result is the evidence you were looking for.

Seeing a different test result?

  • Check that PowerShell is still in the ai-ci-doctor folder.
  • Confirm the file is named test_release_gate.py inside tests.
  • Confirm the three calls use (1, 100), (5, 100), and (6, 100).

Help me understand why my unittest result differs from the expected two failures.

The failed cases reveal a unit mismatch inside should_deploy. failure_percent holds percentage points while MAX_FAILURE_RATE holds the decimal fraction 0.05.

A valid threshold comparison uses matching units on both sides. This invariant explains why values near the boundary expose the defect.

  • Return to the release_gate.py editor tab from earlier.
  • Locate the failure_percent calculation inside should_deploy.
  • Compare the unit produced by failure_percent with the unit stored in MAX_FAILURE_RATE.
  • Record this invariant in your notes: MAX_FAILURE_RATE is a decimal fraction. The computed rate must use the same unit.

Your PowerShell evidence should show the below-budget test and at-budget test as the only failures. The above-budget test should pass.

You have turned a misleading happy path into repeatable evidence. Next, you will capture the same deterministic failure in GitHub Actions.

Reproduce the Failure in CI

Your boundary tests now expose the release gate's hidden unit mismatch. The same evidence needs to survive outside your machine.

GitHub Actions gives the defect a reproducible CI record with searchable logs. In this step, you will publish the failing baseline before inspecting the hosted test evidence.

In this step, get ready to:
  • Define a workflow with a pinned Python 3.14.8 runtime.
  • Commit the failing baseline before publishing it to a public repository.
  • Inspect the hosted workflow logs for the same boundary-test evidence.
Create the CI workflow

A workflow file tells GitHub Actions when to run your tests. Pinning its runtime makes the hosted result reproducible.

  • Switch back to the Explorer sidebar in Visual Studio Code.
  • Select the existing .github/workflows folder.
  • Use the new-file control at the top of the Explorer sidebar to create ci.yml.

You should now see ci.yml beneath the .github/workflows folder.

  • Populate .github/workflows/ci.yml by pasting this workflow:
name: CI

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository
        uses: actions/checkout@v7.0.1
      - name: Set up Python
        uses: actions/setup-python@v7.0.0
        with:
          python-version: "3.14.8"
      - name: Run unit tests
        run: python -m unittest

What does this workflow do?

This YAML file turns the local test command into an automated release check.

  • The trigger section runs the workflow for pushes to main.
  • The trigger section also runs the workflow for pull requests targeting main.
  • The contents: read permission limits repository access to reading files.
  • The ubuntu-latest runner provides the hosted test machine.
  • The actions/checkout@v7.0.1 step makes your repository files available to the job.
  • The actions/setup-python@v7.0.0 step installs Python 3.14.8.
  • The python -m unittest step repeats your local test discovery in CI.
  • Save .github/workflows/ci.yml.
  • Confirm the Explorer sidebar lists ci.yml inside .github/workflows.

The workflow now captures the exact environment that reproduces your local test run.

Workflow file in the wrong place?

  • Check that the full path is .github/workflows/ci.yml.
  • Move ci.yml beneath the existing .github/workflows folder if it appears elsewhere in Explorer.
  • Compare the indentation with the complete reference below because YAML uses spaces to represent structure.

Help me check why GitHub Actions cannot detect or parse my CI workflow.

✔️ Awesome, I've got everything!

Your workflow matches the target configuration. The pinned test environment is ready to publish.

ⓧ I'd like to double check the full code

name: CI

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository
        uses: actions/checkout@v7.0.1
      - name: Set up Python
        uses: actions/setup-python@v7.0.0
        with:
          python-version: "3.14.8"
      - name: Run unit tests
        run: python -m unittest

How to use this reference

This is the complete workflow file for this step. Use it to compare every trigger, permission, version pin, and test command.

Commit and publish the failing baseline

A Git commit freezes the failing baseline before the repair begins. That snapshot lets the later green result prove exactly what changed.

  • Initialize the repository on main before committing every current project file by running these commands:
git init -b main
git add .
git commit -m "Add failing release gate CI"

What do these commands record?

  • Initialization creates the local repository metadata.
  • The main branch becomes the current branch.
  • Staging selects the current project files for the snapshot.
  • The commit records the workflow beside the deliberately failing tests.
  • Confirm the terminal reports that the commit was created on main.

Your failing baseline is now preserved as one reviewable commit.

Commit not created?

  • Read the terminal message for any missing Git identity details.
  • Complete the requested Git identity setup outside the project files.
  • Run the three commands again after Git accepts your identity.

Help me understand why Git could not create my failing baseline commit.

The remote repository must begin empty because your complete project already exists locally. GitHub can then accept the baseline without creating unrelated files.

  • Go to GitHub in your browser.
  • Sign in with your existing account if prompted.
  • Click New repository.
  • Enter ai-ci-doctor as the repository name.
  • Select Public as the visibility.

The form should now identify a public repository named ai-ci-doctor.

  • Confirm that every repository initialization option remains empty.
  • Click Create repository.

That public repository is now the hosted home for your failing baseline.

The first push may trigger GitHub authentication. Your existing account completes the connection without placing credentials in the project.

  • Copy the repository's remote URL here: REMOTE-URL.
  • Use your existing GitHub account if Git opens an authentication prompt during the push.
  • Connect the local repository to GitHub before publishing main by running these commands:
git remote add origin [[REMOTE_URL="REMOTE-URL"]]
git push -u origin main

What do these commands publish?

  • The remote named origin stores the URL of your public repository.
  • The push publishes the local main branch.
  • The upstream setting connects future pushes from main to origin.
  • Confirm the terminal reports that objects were sent to GitHub.
  • Confirm main now tracks origin.

The workflow file is now on GitHub. Its push trigger starts the hosted test job from the exact failing commit.

Push not reaching GitHub?

  • Confirm the copied remote URL belongs to the public ai-ci-doctor repository.
  • Complete any GitHub authentication prompt shown during the push.
  • Check that the terminal still shows main as the branch being published.

Help me diagnose why my first push to the public GitHub repository failed.

Inspect the failed Actions run

The hosted run is valuable only if it reproduces the local evidence. Matching test names prove the defect is deterministic across environments.

Before you inspect the run, do you expect the hosted test job to agree with your local boundary tests?

  • Return to the public ai-ci-doctor repository on GitHub.
  • Click the Actions tab.
  • Select the newest CI workflow run.
  • Select the test job.
  • Expand the Run unit tests step.

You should see Run unit tests marked red. Its logs show failures for test_allows_release_below_budget and test_allows_release_at_budget.

What does this prove?

The hosted job fails on the same two boundary cases as your local suite. The public run now preserves the test evidence beside the exact source revision that produced it.

This gives the investigation a stable starting point. Every proposed repair can be judged against the red workflow before it reaches main.

Workflow run missing or different?

  • Refresh the Actions tab after the push finishes.
  • Confirm the repository contains .github/workflows/ci.yml.
  • Confirm the newest run uses the commit named Add failing release gate CI.
  • Open the Run unit tests logs to compare their test names with your local result.

Help me diagnose why my GitHub Actions run is missing or does not show the expected boundary-test failures.

You have reproduced the deterministic defect in a public CI run with searchable logs. Next, you will use that evidence to guide a constrained AI diagnosis before reviewing the proposed repair.

Diagnose and Repair with AI

The red GitHub Actions run reproduces the two local boundary failures. It now provides evidence that an investigation can trace.

That evidence gives GitHub Copilot CLI a concrete starting point. You will constrain its role to diagnosis before reviewing the smallest supported repair.

In this step, get ready to:
  • Correlate the failed workflow logs with the responsible Python code.
  • Review the proposed repair before accepting the calculation change.
  • Document the incident before proving the repair locally and in CI.
Diagnose the red workflow

A useful root-cause analysis connects the first meaningful failure to the code that produced it. Copilot can retrieve the workflow evidence while your prompt prevents premature file changes.

  • Switch back to the PowerShell 7 terminal in Visual Studio Code.
  • Start GitHub Copilot CLI by running this command:
copilot

What does this command do?

The command starts the interactive Copilot CLI interface in your existing ai-ci-doctor workspace. The authenticated session can use the built-in GitHub integration to retrieve workflow runs and logs.

  • Submit this investigation prompt in the Copilot CLI interface:
My GitHub Actions CI is failing on this branch. Pull the latest workflow run logs, identify the earliest meaningful failure, correlate it with @release_gate.py and @tests/test_release_gate.py, classify the root cause, cite the evidence, and propose the smallest fix. Do not change files until I approve.

What does this investigation prompt do?

  • The workflow request grounds the diagnosis in the latest failed CI run.
  • The two file references focus the investigation on the implementation and its boundary tests.
  • The approval constraint keeps Copilot in diagnosis mode until you review its evidence.
  • Approve read-only requests that retrieve the latest workflow run.
  • Approve read-only requests that inspect the workflow logs.
  • Review the response for the earliest meaningful test failure.
  • Confirm that the response cites test_allows_release_below_budget as failing.
  • Confirm that the response cites test_allows_release_at_budget as failing.
  • Confirm that the diagnosis connects those failures to should_deploy() in release_gate.py.

Copilot's wording can vary. The evidence should converge on a percentage-versus-fraction unit mismatch.

Copilot cannot retrieve the workflow logs?

Confirm that the interactive session is using your authenticated GitHub account. Check that the public ai-ci-doctor repository still contains the failed CI run.

If the repository evidence is still unavailable, help me troubleshoot Copilot CLI workflow access.

Review and apply the smallest fix

AI-generated changes need a human review boundary. The accepted diff should correct the calculation units while preserving the three tests and the surrounding program structure.

  • Limit the permitted edit to release_gate.py if Copilot asks for file access.
  • Decline any edit request for a different file.
  • Submit this focused repair request in the Copilot CLI interface:
Apply only the unit correction in release_gate.py. Rename failure_percent to failure_rate in both functions. Remove * 100 from both calculations. Format the displayed rate with .1%. Do not edit any other file.

Why is this the smallest repair?

The threshold already stores five percent as the decimal fraction 0.05. Computing the failed-check rate as a fraction puts both sides of the comparison in the same unit.

The display still presents the result as a percentage through the formatting rule. This keeps the terminal output readable without changing the decision logic.

  • Review Copilot's current changes by entering this command:
/diff

What should the diff contain?

  • The should_deploy() function should calculate failure_rate without multiplying by 100.
  • The describe_decision() function should use the same fractional rate.
  • The displayed rate should use .1% formatting.
  • The test file should remain unchanged.
  • Inspect every file path shown in the diff.
  • Reject unrelated edits by asking Copilot to revert them.
  • Enter /diff again after any revision.
  • Confirm that the final diff changes only release_gate.py.
Document and verify the repair

An incident report turns a one-time fix into reusable operational knowledge. Its evidence should connect the failing tests to the unit mismatch before recording the repair and prevention rule.

  • Use the file-creation control in Visual Studio Code's Explorer sidebar to create INCIDENT_REPORT.md inside the open ai-ci-doctor folder.
  • Paste this incident record into INCIDENT_REPORT.md.
# Incident Report: Release Gate Unit Mismatch

## Summary

The CI release gate blocked deployments that were still within the allowed 5 percent failure budget.

## Evidence

- `test_allows_release_below_budget` failed for 1 failed check out of 100.
- `test_allows_release_at_budget` failed for 5 failed checks out of 100.
- `test_blocks_release_above_budget` passed for 6 failed checks out of 100.
- The local test output and GitHub Actions logs showed the same deterministic failures.

## Root Cause

The implementation converted the failed-check ratio into percentage points by multiplying by 100, then compared that value with `0.05`, which represents a decimal fraction.

## Resolution

The implementation now compares `failed_checks / total_checks` directly with `MAX_FAILURE_RATE`.

## Prevention

Keep boundary tests below, at, and above the configured error budget, and require a green CI run before accepting future release-gate changes.

How does this report help?

  • The evidence section records the exact boundary behavior that exposed the defect.
  • The root cause section names the incompatible units.
  • The resolution section records the focused code change.
  • The prevention section keeps the boundary tests in the release process.
  • Save INCIDENT_REPORT.md.
  • Confirm that the saved file contains the Evidence section.
  • Confirm that the saved file contains the Root Cause section.
  • Confirm that the saved file contains the Resolution section.
  • Confirm that the saved file contains the Prevention section.

✔️ Awesome, I've got everything!

Your reviewed repair and incident record now match the workflow evidence.

  • Save every open file in Visual Studio Code.

ⓧ I'd like to double check the full code

.github/workflows/ci.yml should remain unchanged:

name: CI

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository
        uses: actions/checkout@v7.0.1
      - name: Set up Python
        uses: actions/setup-python@v7.0.0
        with:
          python-version: "3.14.8"
      - name: Run unit tests
        run: python -m unittest

This workflow still runs the unit tests for pushes and pull requests to main.

INCIDENT_REPORT.md should match this reference:

# Incident Report: Release Gate Unit Mismatch

## Summary

The CI release gate blocked deployments that were still within the allowed 5 percent failure budget.

## Evidence

- `test_allows_release_below_budget` failed for 1 failed check out of 100.
- `test_allows_release_at_budget` failed for 5 failed checks out of 100.
- `test_blocks_release_above_budget` passed for 6 failed checks out of 100.
- The local test output and GitHub Actions logs showed the same deterministic failures.

## Root Cause

The implementation converted the failed-check ratio into percentage points by multiplying by 100, then compared that value with `0.05`, which represents a decimal fraction.

## Resolution

The implementation now compares `failed_checks / total_checks` directly with `MAX_FAILURE_RATE`.

## Prevention

Keep boundary tests below, at, and above the configured error budget, and require a green CI run before accepting future release-gate changes.

This report preserves the evidence trail behind the reviewed repair.

README.md should remain unchanged:

# AI CI Doctor

A small Python release gate and GitHub Actions pipeline for practicing evidence-driven, AI-assisted CI failure investigation.

## Run the release gate

```powershell
python release_gate.py
```

## Run the tests

```powershell
python -m unittest
```

The README still documents the local release-gate command and test command.

release_gate.py should match this repaired version:

MAX_FAILURE_RATE = 0.05


def should_deploy(failed_checks, total_checks, max_failure_rate=MAX_FAILURE_RATE):
    failure_rate = failed_checks / total_checks
    return failure_rate <= max_failure_rate


def describe_decision(failed_checks, total_checks):
    failure_rate = failed_checks / total_checks
    decision = "DEPLOY" if should_deploy(failed_checks, total_checks) else "BLOCK"
    return (
        f"{failed_checks}/{total_checks} checks failed "
        f"({failure_rate:.1%}): {decision}"
    )


def main():
    for failed_checks, total_checks in ((0, 100), (6, 100)):
        print(describe_decision(failed_checks, total_checks))


if __name__ == "__main__":
    main()

Both calculations now use a decimal fraction. The display converts that fraction into percentage text.

The tests/__init__.py file should remain empty.

tests/test_release_gate.py should remain unchanged:

import unittest

from release_gate import should_deploy


class ReleaseGateTests(unittest.TestCase):
    def test_allows_release_below_budget(self):
        self.assertTrue(should_deploy(1, 100))

    def test_allows_release_at_budget(self):
        self.assertTrue(should_deploy(5, 100))

    def test_blocks_release_above_budget(self):
        self.assertFalse(should_deploy(6, 100))

These three boundary tests still cover rates below the budget, at the budget, and above the budget.

Before you run the test suite, do you expect the two boundary failures to remain or disappear?

  • Test the reviewed repair from the PowerShell 7 terminal by running this command:
python -m unittest

What should you see?

The test runner should execute three tests. The final result should be OK.

That result proves the below-budget case and exact-boundary case now pass. The above-budget case remains blocked.

Still seeing failed tests?

  • Check that should_deploy() compares failure_rate directly with max_failure_rate.
  • Check that neither calculation multiplies the failed-check ratio by 100.
  • Check that tests/test_release_gate.py still contains all three original boundary tests.

If the failures continue, help me compare my release-gate calculation with the failing boundary tests.

The local suite is green. Your reviewed calculation now satisfies every release boundary.

  • Stage the reviewed files and create the repair commit by running these commands:
git add .
git commit -m "Fix release gate rate calculation"

What do these commands record?

Git stages the repaired release gate plus the incident report. The commit records them together under the focused repair message.

An authentication prompt may appear when Git pushes. It uses your existing GitHub sign-in, so no credential belongs in the project files.

Before you push, do you expect the same boundary tests to stay red in CI or turn green?

  • Push the repair commit to main by running this command:
git push

What does this push trigger?

The push updates the public repository. The existing workflow starts a new CI run against the repaired code.

  • Switch back to the CI run list in GitHub Actions.
  • Refresh the run list.
  • Select the newest CI run.
  • Select the test job.
  • Confirm that Run unit tests is green.

That completes the red-to-green repair. Your CI evidence now supports the human-reviewed fix recorded in the incident report.

Secret mission

Guard Against an Empty Check Set

A release gate cannot calculate a failure rate when no checks ran. Add a tested error contract for an empty check set. Review the focused change before pushing a green CI run.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to keep your resources available, pause the CI workflow, or delete the project completely. Keeping the local folder costs nothing, while standard runners from GitHub Actions are free for this public repository.

Resources you used:

  • The local ai-ci-doctor project folder containing your release gate, tests, workflow, README, and incident report.
  • The public ai-ci-doctor repository on GitHub, including its source files and run history.
  • The enabled CI workflow that tests every push to main.

Keep everything running

Everything can remain in place if you want to rerun the tests or continue the investigation later. GitHub Copilot CLI only uses more of your plan allowance when you submit more prompts.

  • Retain the local ai-ci-doctor project folder.
  • Leave the public ai-ci-doctor repository available.
  • Keep the CI workflow enabled so future pushes run the test suite.

Pause - I'll come back to this later

Pause automatic CI runs while preserving the repository and its previous evidence. Your local project files remain untouched.

  • Switch back to the signed-in GitHub Actions interface from earlier.
  • Select the CI workflow in the left sidebar.
  • Open the workflow dropdown menu.
  • Click Disable workflow.

Your investigation evidence stays available because the code and earlier run history remain in the repository. You should see the CI workflow marked as disabled.

Delete - I don't want to use this again

This is the permanent option, so it is completely reasonable to pause before confirming it. The repository's code and run history disappear together.

Delete the GitHub repository:

  • Switch back to the public ai-ci-doctor repository on GitHub.
  • Open the repository's Settings tab.
  • Scroll to Danger Zone.
  • Click Delete this repository.
  • Acknowledge each deletion warning.
  • Enter ai-ci-doctor as the repository name.
  • Confirm the repository deletion.

The public repository is now removed. Its enabled CI workflow and previous run history are removed with it.

Delete the local project folder:

  • Return to the open ai-ci-doctor workspace in Visual Studio Code.
  • Right-click the ai-ci-doctor folder in the Explorer sidebar.
  • Choose the context-menu option that opens the folder's location in File Explorer.
  • Close Visual Studio Code.
  • Delete the ai-ci-doctor folder from File Explorer.

That's the cleanup complete. The remote repository and local project folder are no longer available.

Nice Work!

Nice Work!

You did it! Your AI-assisted release gate now protects deployments with a tested 5 percent failure budget.

What you learned:

  • Built a Python release gate that converts the failure budget into visible DEPLOY or BLOCK decisions.
  • Created boundary tests across both sides of the threshold. These tests exposed the percentage-versus-fraction mismatch.
  • Reproduced the defect in a pinned GitHub Actions workflow. Used GitHub Copilot CLI to connect workflow evidence to local code before reviewing the focused repair. Captured the root cause in INCIDENT_REPORT.md after the pipeline returned to green.
  • Secret Mission: Hardened the release gate with a tested ValueError for an empty check set. The newest CI run stayed green.

Ready to quiz yourself?