Engineer Reliable AI Solutions

Introduction

30 Second Summary

A triage team can look fast until one urgent customer issue reaches the wrong people. A missed security report can turn a tidy queue into a client crisis.

In this project, you will harden an existing FastAPI support issue triage service so urgent cases, human overrides, and provider failures produce safe outcomes. You will use failing evaluations to repair deterministic rules, test Gemini-assisted behavior, and block unreliable changes with GitHub Actions.

What You'll Build

When you send a checkout outage to POST /triage, you'll receive a guarded P0 decision that the model cannot silently downgrade, while the automated gate proves the same safeguards before code merges.

By the end of this project, you'll have:

• A guarded triage API that routes urgent cases safely, honors human overrides, and falls back when the model provider fails.
• A two-tier evaluation suite that reduces five known policy failures to zero and tests model downgrades, provider failures, and human overrides without live model calls.
• An automated merge gate that blocks unreliable changes through GitHub Actions without storing a Gemini credential.
• Secret Mission: Extend the evaluation workflow so one controlled failure opens a GitHub Issue automatically without leaving the gate broken.

Are there any prerequisites?

• Starting code: Complete First Steps in AI Solutions Engineering and keep its ai-architecture-nextwork repository ready, or use the shared starter repository in Step 1.
• Confirm your Google account can open Google AI Studio.
• Install Git if it is not already available on your computer.
• Install Python 3.11 or higher if your computer uses an older version.
• Install Visual Studio Code on macOS, Windows, or Linux.

Before We Start

In this project, you will build an evaluation gate for a support-triage classifier that assigns each client issue a priority, a routing team, and a human-review decision. The gate must stop exposed production keys or widespread outages from being labeled as routine because a delayed response could leave client systems vulnerable or unavailable.

Set Up the Evaluation Extension

One of your teammates has built on the project from First Steps in AI Solutions Engineering. They have shared an updated starter project with the pieces needed for this next phase.

Your task is to take this new version and make it reliable. In this step, you'll make your own copy, get it running, and confirm that it is ready for testing.

In this step, get ready to:

• Fork the complete Engineer Reliable AI Solutions starter.
• Create a working branch without exposing your Gemini key.
• Install the pinned dependencies and launch the supplied FastAPI service.

Fork and clone the starter

A fork gives you the complete Part 2 scaffold while keeping your independent Part 1 repository untouched. It also connects this new repository to the starter, so you can sync future updates.

• Open the Engineer Reliable AI Solutions starter.
• Click Fork in the top-right corner.

• Select your GitHub account from the Owner menu.
• Enter ai-reliability-nextwork in the Repository name field.
• Click Create fork.

Already forked this starter?

• Open the existing fork linked by GitHub.
• Click Sync fork above the file list.
• Click Update branch if GitHub shows that your fork is behind.
• Click the Settings tab.
• Enter ai-reliability-nextwork in the Repository name field.
• Click Rename.

Help me prepare my existing fork.

Your Part 2 fork now contains the full scaffold. Bring it onto your computer so you can run the supplied code and make changes on a dedicated branch.

• Click Code above the fork's file list.
• Select HTTPS.
• Click the copy button beside the repository URL.

• Record the copied URL here: https://github.com/YOUR_USERNAME/ai-reliability-nextwork.git.

macOS

• Press Cmd+Space to open Spotlight Search.
• Type Terminal.
• Press Enter to open Terminal.
• Move to your Desktop, clone your fork, and enter the repository folder by running these commands:

cd ~/Desktop
git clone [[YOUR_REPO_URL="https://github.com/YOUR_USERNAME/ai-reliability-nextwork.git"]]
cd ai-reliability-nextwork

Windows

• Press the Windows key to open system search.
• Type PowerShell.
• Press Enter to open PowerShell.
• Move to your Desktop, clone your fork, and enter the repository folder by running these commands:

cd ~/Desktop
git clone [[YOUR_REPO_URL="https://github.com/YOUR_USERNAME/ai-reliability-nextwork.git"]]
cd ai-reliability-nextwork

Linux

• Press the Super key to open application search.
• Type Terminal.
• Press Enter to open Terminal.
• Move to your Desktop, clone your fork, and enter the repository folder by running these commands:

cd ~/Desktop
git clone [[YOUR_REPO_URL="https://github.com/YOUR_USERNAME/ai-reliability-nextwork.git"]]
cd ai-reliability-nextwork

Clone did not complete?

Confirm the copied URL ends with .git and belongs to the repository you just created. If GitHub asks you to authenticate, complete the browser sign-in before retrying.

Help me clone my GitHub fork.

• Confirm Git recognizes your repository and the supplied folders by running:

git status
ls app evals scripts

You should see Git report the main branch. The output should also list files inside app/, evals/, and scripts/.

• Create your working branch from the supplied scaffold by running:

git switch -c part-2-evaluation-gate

Good start. Your evaluation work now has its own branch while the starter remains unchanged on main.

Protect your Gemini credential

Credentials deserve a deliberate setup. The repository includes .env.example as a safe template, while .env stays ignored by Git.

✔️ I already have a Gemini API key

• Keep your existing key private.
• Continue below to store it in .env as GEMINI_API_KEY.

ⓧ I need a Gemini API key

A Gemini API key is a live credential, so anyone who has it can use your access.

• Open the Google AI Studio API keys page.
• Click Create API key.
• Copy the generated key to a password manager or another private location.

• Create your local credential file from the supplied template by running the command for your operating system:

macOS

cp .env.example .env

Windows

Copy-Item .env.example .env

Linux

cp .env.example .env

• Press Cmd+Space on macOS or the Windows key on Windows to open system search.
• Type Visual Studio Code and press Enter.
• Click File in the top menu.
• Click Open Folder.
• Select the ai-reliability-nextwork folder on your Desktop.

• Open .env from the Explorer sidebar.
• Replace your-api-key-here with your copied key.
• Save .env.

Prepare the Python environment

A virtual environment isolates this repository's packages from the rest of your computer. Python 3.11 or higher is required by the pinned dependency set.

• Click Terminal from the tool tab
• Open New Terminal
• Check your Python version by running the command for your operating system:

macOS

python3 --version

Windows

py --version

Linux

python3 --version

✔️ I see version 3.11 or higher

Your interpreter meets the project requirement. Continue below to create the isolated environment.

ⓧ I see an older version

The installed interpreter is below the required version. Use the official Python downloads page to install Python 3.11 or higher.

• Install Python 3.11 or higher using the official installer for your operating system.
• Restart your terminal after installation.
• Run the version check again before continuing.

ⓧ Command not found

Python is unavailable through the current terminal command. Use the official Python downloads page to install Python 3.11 or higher.

• Install Python using the official installer for your operating system.
• Restart your terminal after installation.
• Run the version check again before continuing.

The dependency installation can take two or three minutes while packages download. A quiet terminal during part of that process does not mean it has stalled.

• Create the environment, activate it, and install the supplied dependency manifest by running the commands for your operating system:

macOS

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Windows

py -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Linux

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

You should see (.venv) at the start of your prompt. The installation finishes with the pinned packages available inside that environment.

Before you start the service, do you think the supplied scaffold can load the FastAPI application without changing any code?

• Start the supplied API with your local credential file by running:

python -m uvicorn app.main:app --reload --env-file .env

• Open the interactive API documentation in your browser.
• Return to the terminal running Uvicorn.
• Stop the local service by pressing Ctrl+C.

Why run the service now?

Starting the service checks that your setup works before you begin changing the evaluation logic.

The /docs page confirms that the supplied POST /triage endpoint loaded successfully.

This project is an API service, so it is expected not to have a home page. A 404 Not Found message at the main address is normal and does not mean your setup failed.

Your prepared scaffold is ready. Next, you will run its broad deterministic policy and use the named failures to repair three client-facing decisions.

Build the Baseline and Expose Its Gaps

Expose and Repair the Deterministic Baseline

Your Part 2 scaffold now imports cleanly. Its frozen fixtures and Tier A test are ready to challenge the broad policy supplied in app/triage_rules.py.

A classifier can return plausible labels while breaking urgent client rules. You will expose those gaps first, then make three focused policy edits until the same ten expectations pass.

What do these terms mean?

Tier A is the project's deterministic test layer. It checks the classifier against ten frozen, client-approved outcomes.

A classifier turns each support issue into structured labels, such as a priority, routing team, and human-review decision. In this project, it follows the ordered rules in app/triage_rules.py.

In this step, get ready to:

• Run the supplied baseline against ten frozen support issues.
• Preserve explicit human judgment before automation.
• Order urgent and widespread rules ahead of routine categories.

Expose the baseline's policy gaps

Each record in evals/issues.json contains an issue object with the classifier's input. Its expected object records the client-approved priority, routing team, and human-review decision.

• Open evals/issues.json from the Visual Studio Code Explorer sidebar.
• Press Cmd+F (macOS) or Ctrl+F (Windows) to open the file search.
• Enter ISSUE-001 in the search field.
• Read its issue and expected objects.

• Replace the search value with ISSUE-010.
• Read its issue, human_override, and expected objects.

• Open app/triage_rules.py from the Explorer sidebar.
• Press Cmd+F (macOS) or Ctrl+F (Windows) to open the file search.
• Enter TODO in the search field.
• Confirm the search shows three matches in the file.

What should you notice?

ISSUE-001 reports exposed production keys with write access. Its approved result is priority P0, routing team security, and requires_human_review: true.

ISSUE-010 looks routine until its human_override sets priority P1 and routing team security. Its expected result also requires human review, so automation must preserve the reviewer’s decision.

These expected objects are the client-approved answer key. Keep them frozen while you change the classifier, or weak logic could redefine what counts as success.

Before you run Tier A, do you think broad payment, integration, and performance categories can protect urgent security, outage, and override cases?

• Switch Back to your Integrated Terminal
• Ctrl + c, if the server is still running from step 1
• Expose the incomplete policy by running the supplied Tier A evaluation:

python -m unittest -v evals.test_tier_a

The failures are intentional

You should see the fixture-count test pass. At the bottom of the terminal, the run should end with FAILED (failures=5).

The command and test infrastructure worked. The red result is evidence that the broad policy lacks precedence.

🙋‍♀️ Tier A did not reach the assertions?

An import or syntax error is separate from the expected assertion failures. Confirm your terminal is inside ai-reliability-nextwork and that (.venv) appears in the prompt.

Help me reach the intended Tier A result.

Preserve explicit human judgment

Automated matching should never replace an explicit reviewer decision. The first TODO belongs before searchable text is built or any rule is checked.

• Open app/triage_rules.py from the Explorer sidebar in Visual Studio Code.
• Press Cmd+F (macOS) or Ctrl+F (Windows) to open the file search.
• Enter TODO 1 in the search field.
• Confirm the match appears at the start of classify_issue().

• Add an early return that calls _override_decision(issue) when issue.human_override is present.

✔️ I'll try it first

Use the existing helper so the override remains typed and receives the correct decision source. Save app/triage_rules.py when your early return is ready.

ⓧ Let me check my work

Place this focused guard at the start of classify_issue() in app/triage_rules.py, before the searchable text is built.

# Preserve a person's explicit decision before automation
if issue.human_override is not None:
    return _override_decision(issue)

What does this code solve?

This guard preserves a reviewer's explicit decision before the automated rules run. It prevents the classifier from overwriting a human override.

Before you rerun Tier A, do you expect the failure count to stay at five or decrease?

• Measure the first repair by rerunning Tier A:

python -m unittest -v evals.test_tier_a

You should see FAILED (failures=4) at the bottom of the terminal, down from five failures. Your first edit repaired one mismatch without changing the frozen fixture.

Still seeing five failures?

If the summary still shows FAILED (failures=5), check that the override guard appears before text = _searchable_text(issue). Confirm the return calls the supplied _override_decision() helper.

Help me reduce the Tier A failure count.

Order urgent rules before routine categories

The _RULES tuple stops at the first match. Critical security, global outages, and investigations must therefore appear before the general payment and routine category rules.

• Open app/triage_rules.py from the Explorer sidebar in Visual Studio Code.
• Press Cmd+F (macOS) or Ctrl+F (Windows) to open the file search.
• Enter TODO 2 in the search field.
• Confirm the match appears near the top of _RULES.

• Add four ordered rules for critical security, global payment outages, global service outages, and security investigations.
• Reuse the supplied phrase sets instead of adding new keywords.

✔️ I'll try it first

Start with the most specific safety condition. Keep the general payment rule below all four additions.

ⓧ Let me check my work

Place these four rules near the top of _RULES in app/triage_rules.py, above the routine category rules.

# Handle the most urgent and specific policies first
_Rule("Critical security exposure", (CRITICAL_SECURITY_PHRASES,), "P0", "security"),
_Rule("Global payment outage", (PAYMENT_PHRASES, GLOBAL_IMPACT_PHRASES, OUTAGE_PHRASES), "P0", "payments"),
_Rule("Global service outage", (GLOBAL_IMPACT_PHRASES, OUTAGE_PHRASES), "P0", "reliability"),
_Rule("Security investigation", (SECURITY_INVESTIGATION_PHRASES,), "P1", "security"),

What does this code solve?

These rules give urgent security incidents and global outages priority over routine categories. The classifier stops at the first match, so this order prevents critical issues from receiving a lower priority.

• Save app/triage_rules.py.
• Measure the urgent-rule repair by rerunning Tier A:

python -m unittest -v evals.test_tier_a

You should now see FAILED (failures=1) at the bottom of the terminal. The urgent security, global outage, and override cases now follow their approved labels.

The remaining case describes impact across many customers without a global outage. That signal needs a P1 reliability rule before the routine categories.

• Open app/triage_rules.py from the Explorer sidebar in Visual Studio Code.
• Press Cmd+F (macOS) or Ctrl+F (Windows) to open the file search.
• Enter TODO 3 in the search field.
• Confirm the match appears directly below the urgent rules.
• Add a widespread-impact rule using WIDESPREAD_IMPACT_PHRASES.

✔️ I'll try it first

Keep this rule above the general payment, integration, performance, documentation, and feature-request rules.

ⓧ Let me check my work

Place this rule in app/triage_rules.py below the urgent rules and above the routine categories.

# Escalate broad customer impact before routine categories
_Rule("Widespread customer impact", (WIDESPREAD_IMPACT_PHRASES,), "P1", "reliability"),

What does this code solve?

This rule catches issues that affect many customers before routine categories handle them. It routes widespread impact to reliability as P1, even when the issue is not a global outage.

Before you run the complete gate, do you expect the failure count to stay at one or reach zero?

• Run the repaired deterministic gate:

python -m unittest -v evals.test_tier_a

What should you see?

You should see both Tier A test methods pass and the run end with OK.

That is the deterministic foundation locked in. Ten frozen decisions now follow one ordered policy.

🙋‍♀️ Tier A still showing a mismatch?

If the final run still ends with FAILED, note the number in failures=N before changing code. Check that critical security precedes global rules, global payment precedes global service, and widespread impact precedes routine categories.

Help me diagnose the remaining Tier A failures.

Your baseline now protects every frozen decision. Next, you will challenge its model-assisted path with a forced downgrade, a provider failure, and a human override.

Test Model-Assisted Resilience

Your deterministic policy now passes all ten frozen cases. That trusted baseline gives model assistance a boundary it cannot silently weaken.

The starter already supplies the Gemini provider and a controlled model stub. You will use Tier B to force three risky outcomes, then repair one guardrail at a time.

In this step, get ready to:

• Run the supplied stub through the incomplete orchestration path.
• Protect human overrides and provider failures.
• Accept escalations while rejecting unsafe downgrades.

Expose the missing guardrails

Before you run Tier B, do you think the incomplete orchestrator can stop the forced downgrade, provider failure, and override call?

• Expose the missing guardrails by running Tier B:

python -m unittest -v evals.test_tier_b

The Tier B failures are intentional

Tier B uses a deterministic model stub, so these tests need no Gemini credentials or network access. Each failure exposes one missing guardrail.

• ISSUE-001 tests whether an unsafe priority downgrade is accepted.
• ISSUE-006 tests what happens when the provider fails.
• ISSUE-010 tests whether a human override reaches the provider.

The red result confirms that orchestration still allows these outcomes. You will close the override and provider-failure gaps next.

Protect overrides and provider failures

The deterministic baseline already marks an explicit reviewer decision with human_override. Orchestration should return that decision before the provider sees the issue.

• Open app/orchestrator.py from the Explorer sidebar in Visual Studio Code.
• Press Cmd+F (macOS) or Ctrl+F (Windows) to open the file search.
• Enter TODO 1 in the search field.

• Confirm the match appears inside triage().
• Return baseline immediately when its decision source is human_override.

✔️ I'll try it first

Place the guard before self._provider.suggest(issue) so the stub cannot record the override call.

ⓧ Let me check my work

Replace the TODO 1 gap with this early return.

# Preserve a reviewed decision without consulting the model
if baseline.decision_source == "human_override":
    return baseline

• Save app/orchestrator.py.
• In your terminal, run the focused test for the override guardrail:

python -m unittest -v evals.test_tier_b.TierBGuardedOrchestrationTest.test_human_override_bypasses_provider

You should see the focused test pass and the run end with OK. The assertion also proves ISSUE-010 never reached the stub.

A provider outage should reduce capability without erasing the trusted decision. The second TODO converts that exception into a typed baseline fallback.

• Press Cmd+F (macOS) or Ctrl+F (Windows) to open the file search.
• Enter TODO 2 in the search field.
• Confirm the match appears immediately before the provider call.
• Wrap self._provider.suggest(issue) in a try and except ModelProviderError block.

• Return a copy of the baseline with source baseline_fallback and the supplied unavailable note.

✔️ I'll try it first

Use baseline.model_copy(update={...}) so priority, routing, and review status remain unchanged.

ⓧ Let me check my work

Replace the direct provider call with this guarded version.

# Fall back to the trusted baseline when the provider is unavailable
try:
    suggestion = self._provider.suggest(issue)
except ModelProviderError:
    return baseline.model_copy(update={
        "decision_source": "baseline_fallback",
        "summary": f"{baseline.summary} {PROVIDER_UNAVAILABLE_NOTE}",
    })

• Save app/orchestrator.py.
• Run the focused test for the provider-fallback guardrail:

python -m unittest -v evals.test_tier_b.TierBGuardedOrchestrationTest.test_provider_failure_returns_baseline_fallback

You should see the focused test pass and the run end with OK. ISSUE-006 keeps its frozen labels and records baseline_fallback as the source.

A focused guardrail still failing?

For the override test, confirm the early return appears before the provider call. For the fallback test, catch ModelProviderError and keep the supplied PROVIDER_UNAVAILABLE_NOTE in the summary.

Help me diagnose the focused guardrail test.

Reject unsafe priority downgrades

Lower numeric ranks represent greater urgency. A suggestion can move a decision from P2 to P1, but it cannot move a guarded P0 decision to P3.

• Press Cmd+F (macOS) or Ctrl+F (Windows) to open the file search.
• Enter TODO 3 in the search field.
• Confirm the match appears inside _apply_suggestion().
• Compare the suggestion and baseline through _PRIORITY_RANK.

• Use the suggested priority and team only when the suggestion is more urgent.

✔️ I'll try it first

Remember that a smaller rank is more urgent. Equal priority is not an escalation, so it keeps the baseline team.

ⓧ Let me check my work

Replace the two unconditional assignments below TODO 3 with this comparison.

# Accept only suggestions that increase urgency
is_escalation = _PRIORITY_RANK[suggestion.priority] < _PRIORITY_RANK[baseline.priority]
priority = suggestion.priority if is_escalation else baseline.priority
routing_team = suggestion.routing_team if is_escalation else baseline.routing_team

• Save app/orchestrator.py.
• Run the focused test for the downgrade guardrail:

python -m unittest -v evals.test_tier_b.TierBGuardedOrchestrationTest.test_unsafe_downgrade_is_rejected

You should see the focused test pass and the run end with OK. The stub still proposes P3 for ISSUE-001, while the guarded result remains P0.

Before you run the complete gate, can any of the three forced outcomes now weaken a frozen decision?

• Run every deterministic and model-assisted evaluation:

python -m unittest -v evals.test_tier_a evals.test_tier_b

What should you see?

You should see all six test methods pass and the run end with OK.

That is the resilience layer working. The override bypasses the provider, failure falls back safely, and the forced downgrade cannot reduce urgency.

🙋‍♀️ The complete gate still failing?

Run the failing focused test on its own before changing another guardrail. Check its assertion message for the issue ID, decision source, or expected label that still differs.

Help me diagnose the remaining Tier B failure.

Your guarded decision flow now survives unsafe suggestions and provider failures. Next, you will call that flow through HTTP and turn the same tests into an automatic repository gate.

Wire the API, Architecture, and CI Gate

Expose the API and Automate the Eval Gate

You’ve tested how your support system handles urgent issues, human decisions, and unexpected failures. Now you’ll prove the finished system works from beginning to end, then protect it from unreliable changes.

By the end of this step, you’ll have sent a sample support issue through the system and confirmed that it receives the right priority and team. You’ll also have a GitHub check that automatically runs your tests whenever the code changes.

In this step, get ready to:

• Send a live support issue through POST /triage.
• Run both evaluation tiers automatically in GitHub Actions.

Start the API and send a request

• Start the API with the local credential file by running:

python -m uvicorn app.main:app --reload --env-file .env

What does this command start?

Uvicorn loads the FastAPI application from app/main.py. It keeps this terminal occupied while the server waits for requests.

You should see the server listening on http://127.0.0.1:8000. Press Ctrl+C when you need to stop it.

• Open a split terminal by clicking split icon

• Enter the ai-reliability-nextwork folder and activate its existing environment by running the commands for your operating system:

macOS

cd ~/Desktop/ai-reliability-nextwork
source .venv/bin/activate

Windows

cd ~/Desktop/ai-reliability-nextwork
.\.venv\Scripts\Activate.ps1

Linux

cd ~/Desktop/ai-reliability-nextwork
source .venv/bin/activate

This check sends one request through the Gemini API. If billing is enabled, review the official Gemini API pricing page before continuing.

Before you send the request, can any model suggestion lower a guarded P0 checkout outage?

• Send the supplied live request from the second terminal:

python scripts/call_triage.py

What should you see?

You should see formatted JSON for DEMO-001 with priority P0, routing team payments, and human review set to true.

The HTTP path reached the guarded orchestrator. Model assistance can enrich the summary, but it cannot downgrade the global outage.

🙋‍♀️ Live request not returning JSON?

Confirm the first terminal still runs Uvicorn. Keep the API key out of screenshots and help requests.

Help me debug the missing JSON response.

• Return to the first terminal.
• Stop Uvicorn by pressing Ctrl+C.

Complete the automated evaluation gate

The supplied workflow is manual while the starter evaluations are intentionally red. Now that both tiers pass, two small edits can turn it into a gate for every push and pull request.

• Open .github/workflows/evaluation-gate.yml in your editor.
• Press Cmd+F (macOS) or Ctrl+F (Windows) to open the file search.
• Enter TODO in the search field.

• Confirm the first match sits directly above this manual trigger block
• Replace that trigger block so pushes, pull requests, and manual runs all start the gate:

# Run the completed gate for code changes and manual checks
on:
  push:
  pull_request:
  workflow_dispatch:

• Press Cmd+F (macOS) or Ctrl+F (Windows) to open the file search.
• Enter TODO in the search field.
• Press Enter once to move to the second match.

• Replace it with the complete credential-free gate:

# Run deterministic and stubbed model-assisted evaluations
- name: Run evaluations
  run: python -m unittest -v evals.test_tier_a evals.test_tier_b

What does the completed workflow enforce?

The workflow installs the pinned packages on a hosted Python 3.11 runner. Both suites use frozen fixtures and the stub, so the workflow receives no Gemini credential.

• Save .github/workflows/evaluation-gate.yml.

Pushing starts a remote check, which can feel risky when a project uses credentials. This workflow contains no secret and makes no live model call.

• Commit the application, evaluations, and workflow by running:

git add app evals .github/workflows/evaluation-gate.yml
git commit -m "Add guarded AI evaluation gate"
git push -u origin part-2-evaluation-gate

The first hosted run can take a few minutes while GitHub prepares the runner and installs packages. A queued or in-progress status during that wait is expected.

• Open your fork on GitHub.
• Click Actions below the repository name.
• Select Evaluation Gate in the left sidebar.

• Open the latest workflow run.
• Open the evaluate job.

What should you see?

You should see a green check beside the job. Its final log step shows Tier A and Tier B completing successfully.

Your repository now rejects policy or orchestration regressions before they can be merged.

🙋‍♀️ Evaluation Gate showing a failure?

Open the failed workflow step to identify whether dependency installation, Tier A, or Tier B stopped the job. Run that exact command locally before pushing another change.

Help me diagnose the failed evaluation workflow.

You now have a callable AI-assisted triage solution with deterministic policy, repeatable resilience tests, and an automated evaluation gate.

Secret mission

Turn Evaluation Failures into Work Items

Extend the evaluation gate so a failed GitHub Actions run automatically opens a GitHub Issue. You will trigger one controlled failure and confirm the handoff without leaving the workflow broken.

Clean Up Your Resources

Clean Up Your Resources

Your local evals use the supplied model stub. Your GitHub Actions gate uses the same stub without creating ongoing model cost.

Decide whether to keep the Part 2 fork ready for more client work, pause its local processes, or delete the fork and local clone.

Cost warning

The project makes one live Gemini API request with your own key. Google lists free and paid tiers on the official Gemini API pricing page, so review the model and billing status before sending another request.

The local server and stubbed GitHub Actions tests create no idle Gemini charge. Charges can only arise from live API requests on a paid tier.

Resources you used:

• The ai-reliability-nextwork fork on GitHub, including your Part 2 branch and workflow history.
• The local ai-reliability-nextwork clone on your Desktop, including .venv and .env.
• A local Uvicorn process serving the FastAPI API if it is still running.

Keep everything running

No action is needed. Choose this if you plan to keep testing or extending the guarded triage solution.

• Keep the ai-reliability-nextwork fork so its branches and workflow history remain available.
• Keep the local clone so its source files and pinned environment remain ready.
• Remember that another run of scripts/call_triage.py sends another live model request.

Pause - I'll come back to this later

Shut down local processes to free memory while preserving the fork, branches, fixtures, tests, and workflow.

• Switch to the terminal running Uvicorn if the server is still active.
• Press Ctrl+C to stop the local server.

You should see the shell prompt return when Uvicorn stops.

If (.venv) still prefixes the prompt, the virtual environment remains active.

• Leave the active virtual environment by running:

deactivate

You should no longer see (.venv) before the prompt. Your fork and local files remain available.

Delete - I don't want to use this again

Deleting the fork and local clone is irreversible. Your independent Part 1 repository remains untouched.

• Switch to the terminal running Uvicorn if the server is still active.
• Press Ctrl+C to stop the local server.

The server is stopped when your shell prompt returns.

If (.venv) appears before the prompt, leave the environment before deleting its folder.

• Leave the active virtual environment by running:

deactivate

• Open the ai-reliability-nextwork fork on GitHub.
• Click the Settings tab.
• Select General in the left sidebar.
• Scroll to Danger Zone.
• Click Delete this repository.
• Follow GitHub's confirmation prompts to delete the fork.

Use GitHub's repository deletion guide if the confirmation screen differs from these labels.

Deleting the local folder also removes its .env credential file. The commands below target only the ai-reliability-nextwork folder on your Desktop.

macOS

• Delete the local clone from your Desktop by running:

cd ~/Desktop
rm -rf ai-reliability-nextwork

You should no longer see ai-reliability-nextwork on your Desktop.

Windows

• Delete the local clone from your Desktop by running:

cd ~/Desktop
Remove-Item -Recurse -Force ai-reliability-nextwork

You should no longer see ai-reliability-nextwork on your Desktop.

Linux

• Delete the local clone from your Desktop by running:

cd ~/Desktop
rm -rf ai-reliability-nextwork

You should no longer see ai-reliability-nextwork on your Desktop.

Nice Work!

Nice Work!

You did it! You built a guarded AI support-triage API that returns live decisions and survives twenty case-level evaluations before changes merge.

You've learned how to:

• Build a live triage endpoint that returns guarded priority, routing, review, source, and summary fields through POST /triage.
• Prove twenty case-level outcomes across deterministic checks plus stubbed orchestration checks for downgrade protection, provider fallback, and override bypass.
• Automate the two-tier evaluation gate with GitHub Actions without storing a Gemini credential.
• Complete the optional Secret Mission by granting least-privilege issue permissions and turning one controlled failure into a GitHub Issue without leaving the workflow broken.

Ready to quiz yourself?