Build an Observable AI Alert Router

Build a local AI alert router with safety gates, telemetry, and evaluation.

Introduction

30 Second Summary

Security teams often receive more alerts than analysts can review at once. A weak automated decision can quietly send an urgent incident to the wrong queue.

In this project, you will build a terminal-based incident triage control plane for synthetic security alerts. It uses a local decision model with human review as its safe fallback.

What You'll Build

You run the finished program and watch each alert reveal its model-selected queue, guarded route, confidence, latency, and evaluation result in your terminal.

By the end of this project, you'll have:

  • A local typed decision workflow that lets you batch-route synthetic alerts. You can inspect the model's queue, urgency, and maliciousness answers.
  • A deterministic fault injection that lets you watch the naive router trust a low-confidence network decision. You can then prove the guarded router sends the same alert to human_review.
  • Machine-readable JSONL telemetry that exposes route quality, latency, and fallback behavior after each run. A labeled evaluation shows how many final routes match the expected outcomes.
  • Secret Mission: Redact sensitive user_email and api_key fields before inference. Then prove the local model received the sanitized alert.

Are there any prerequisites?

You'll need a Mac running macOS 14 or newer with administrator access.

The downloads require internet access. You'll also need enough local storage for Ollama and its local model.

Before We Start

This checkpoint locks in the observable alert router you are building. It also captures why unsafe AI decisions need a safe fallback before any hands-on work begins.

Prepare the Local AI Runtime

An observable alert router only works when its local decision endpoint is available. A missing runtime would turn every later routing test into an environment problem.

Python sends alerts to the model. Visual Studio Code gives you a place to build the control plane.

Ollama runs the decision model on your Mac. This step proves each layer is ready before you write application code.

In this step, get ready to:
  • Confirm macOS 14 compatibility before preparing Python 3.14.8.
  • Enable folder opening from Terminal through Visual Studio Code.
  • Run Ollama 0.35.1 locally with the tev1:0.8b model.
Confirm macOS and Python

Ollama requires macOS 14 or newer. Python 3.14.8 gives this project a consistent interpreter for every routing test.

  • Select the Apple menu in the top-left corner of your screen.
  • Select About This Mac.

You will see the installed macOS version in the window.

✔️ I See macOS 14 or Newer

Your Mac clears the compatibility gate. Ollama can run on this version of macOS.

ⓧ I See macOS 13 or Older

This Mac needs macOS 14 or newer before Ollama can run.

  • Update the Mac to macOS 14 or newer before continuing.
  • Return to About This Mac after the update.
  • Confirm that the displayed version is macOS 14 or newer.
  • Press Cmd+Space to open macOS search.
  • Type Terminal and press Enter to open it.
  • Check the installed Python interpreter by running this command:
python3.14 -VV

What Does This Command Check?

  • The python3.14 executable selects the exact Python interpreter used throughout this project.
  • The -VV option prints the full interpreter version plus build details.

✔️ I See Python 3.14.8

Python 3.14.8 is ready. Keep this Terminal window available for the remaining checks.

ⓧ I See an Older Version

The official Python installer may request your Mac administrator password. That request authorizes the local installation.

  • Open the official Python 3.14.8 release page.
  • Select macOS installer.
  • Open the downloaded .pkg file from your Downloads folder.
  • Complete the standard macOS installer.
  • Return to the Terminal window from earlier.
  • Check the new interpreter by running this command:
python3.14 -VV

Why Use the Full Command Name?

The python3.14 command selects the newly installed interpreter directly. An older python3 shortcut can remain elsewhere on the Mac.

ⓧ The Command Is Unavailable

An administrator prompt can appear during installation. It gives the official installer permission to add Python to your Mac.

  • Open the official Python 3.14.8 release page.
  • Select macOS installer.
  • Open the downloaded .pkg file from your Downloads folder.
  • Complete the standard macOS installer.
  • Return to the Terminal window from earlier.
  • Confirm the installation by running this command:
python3.14 -VV

What Proves Python Is Ready?

The command should begin its output with Python 3.14.8. That result confirms the required executable is available.

Python uses trusted TLS certificates when it connects to secure services. The official macOS installation includes a script that prepares this certificate store.

  • Press Cmd+Space to open macOS search.
  • Type Finder and press Enter to open it.

You will see a Finder window where you can navigate directly to the Python installation.

  • Press Cmd+Shift+G to open the folder location field.
  • Type /Applications/Python 3.14/ into the field.
  • Press Return to open the folder.

You will see the certificate setup file inside the Python folder.

  • Double-click Install Certificates.command.

A Terminal window runs the certificate setup. Wait until it reports that the process has finished.

Certificate Setup Does Not Finish?

Confirm that you opened Install Certificates.command from /Applications/Python 3.14/. Reinstall Python 3.14.8 if that file is missing.

Help me troubleshoot the Python certificate setup on macOS.

Enable Visual Studio Code from Terminal

The code shell command lets Terminal open the current folder in Visual Studio Code. This keeps the project workflow anchored to one known folder.

  • Confirm trust for your own local folder if Visual Studio Code shows a first-launch trust prompt.
  • Test the Visual Studio Code shell command by running:
code .

What Does This Command Do?

  • The code command starts Visual Studio Code from Terminal.
  • The . value represents the folder at Terminal's current location.

✔️ Visual Studio Code Opens

Visual Studio Code can now open folders directly from Terminal. This is the workflow you will use for observable-ai-triage in the next step.

ⓧ Terminal Cannot Find code

  • Press Cmd+Space to open macOS search.
  • Type Visual Studio Code and press Enter to open it.

Visual Studio Code is installed. The missing piece is its Terminal command.

  • Press Cmd+Shift+P to open the Command Palette.
  • Type Shell Command: Install 'code' command in PATH.

macOS may request administrator approval. The approval creates the local command link used by Terminal.

  • Select Shell Command: Install 'code' command in PATH from the results.
  • Return to Terminal.
  • Press Cmd+Q to quit Terminal.

Terminal closes so its next session can load the updated command path.

  • Press Cmd+Space to open macOS search.
  • Type Terminal and press Enter to reopen it.
  • Test the installed command by running:
code .

What Should Open?

Visual Studio Code should open the folder represented by Terminal's current location. That result confirms code is available in your command path.

ⓧ Visual Studio Code Is Missing

The disk image contains the Visual Studio Code application.

  • Open the downloaded .dmg file from your Downloads folder.
  • Drag Visual Studio Code.app into the Applications folder.
  • Press Cmd+Space to open macOS search.
  • Type Visual Studio Code and press Enter to launch it.

macOS may request administrator approval when you add the Terminal command. The approval creates a local command link.

  • Press Cmd+Shift+P to open the Command Palette.
  • Type Shell Command: Install 'code' command in PATH.
  • Select Shell Command: Install 'code' command in PATH from the results.
  • Return to Terminal.
  • Press Cmd+Q to quit Terminal.

Terminal closes with the previous command path.

  • Press Cmd+Space to open macOS search.
  • Type Terminal and press Enter to reopen it.
  • Confirm the shell command by running:
code .

What Proves the Setup Worked?

Visual Studio Code should open Terminal's current folder. This proves the application plus its shell command are ready.

Still Unable to Use code?

Restart Terminal after installing the shell command. Confirm that you selected the command whose name ends with in PATH.

Help me troubleshoot the Visual Studio Code shell command.

Install Ollama and the decision model

Ollama runs the model through a local service at http://localhost:11434. Checking that service reveals both its availability and its installed version.

  • Check the local Ollama service by running:
curl http://localhost:11434/api/version

What Does This Request Check?

  • The curl command sends a request to the Ollama service running on your Mac.
  • The /api/version endpoint returns the version of the running service.

✔️ I See Version 0.35.1

Ollama 0.35.1 is running locally. The service is ready to receive a model.

ⓧ I See an Older Version

Ollama must be updated before it can run the decision model used in this project.

  • Quit the currently running Ollama application.
  • Open the official Ollama release page.
  • Download the macOS disk image for v0.35.1.
  • Open the downloaded disk image.
  • Drag the Ollama application into the system-wide Applications folder.

macOS may ask whether to replace the existing application. Confirm the replacement so the installed copy uses 0.35.1.

  • Press Cmd+Space to open macOS search.
  • Type Ollama and press Enter to launch it.
  • Allow creation of the CLI link in /usr/local/bin if macOS requests permission.
  • Confirm the running version by running:
curl http://localhost:11434/api/version

What Confirms the Update?

The response should contain a version field with the value 0.35.1. That response comes from the running local service.

ⓧ The Local Service Is Unavailable

Installing Ollama adds the application plus its Terminal command. The first launch starts the local service.

  • Open the official Ollama release page.
  • Download the macOS disk image for v0.35.1.
  • Open the downloaded disk image.
  • Drag the Ollama application into the system-wide Applications folder.

The first launch may request permission to create a CLI link in /usr/local/bin. Allowing it makes the ollama command available in Terminal.

  • Press Cmd+Space to open macOS search.
  • Type Ollama and press Enter to launch it.
  • Allow creation of the CLI link if macOS requests permission.
  • Confirm that the local service is running by executing:
curl http://localhost:11434/api/version

What Confirms Ollama Is Running?

A JSON response containing version 0.35.1 proves that the local service is listening at http://localhost:11434.

The model download can take several minutes because Ollama stores the model weights on your Mac. A quiet period during the download is expected.

  • Download the decision model by running:
ollama pull tev1:0.8b

What Does This Command Do?

  • The pull command downloads a model into Ollama's local model store.
  • The tev1:0.8b identifier pins the decision model used by this project.
  • Local inference requires no API key.

Before you run the final check, do you expect both required runtimes to identify themselves successfully?

  • Verify Python plus the installed Ollama model by running these commands:
python3.14 -VV
ollama ls

What Does the Final Check Prove?

  • The first command confirms that Python 3.14.8 is available through python3.14.
  • The second command asks the running Ollama service for its locally installed models.

The first output begins with Python 3.14.8. The model list includes tev1:0.8b.

Missing a Runtime or Model?

Reopen Ollama if the model list cannot reach the local service. Run the model download command again if tev1:0.8b is absent.

Help me diagnose the final local runtime check.

Your local AI runtime is ready to serve decisions from tev1:0.8b. Next, you will create observable-ai-triage in Visual Studio Code and route the first security alert.

Route the First Security Alert

Your local Ollama runtime is ready. That setup gives your alert router a reliable place to make its first decision.

A Python application still needs to prove that one synthetic alert can cross the local HTTP boundary. This step connects that alert to a typed decision endpoint.

The first visible route confirms that the local model can receive application data. It also confirms that your code can read the structured response.

In this step, get ready to:
  • Create the observable-ai-triage folder in Visual Studio Code.
  • Store the first labeled synthetic alert in alerts.json.
  • Run the first local typed routing decision.
Create the project and first alert

A synthetic alert gives the model a realistic security case. Its human-authored label records the route that an evaluator expects.

You will keep this alert in JSON so Python can load its fields without parsing free-form text. Visual Studio Code keeps the project files together as you build.

  • Press Cmd+Space to open Spotlight.
  • Type Visual Studio Code into Spotlight.
  • Press Enter to open Visual Studio Code.

You will see the Visual Studio Code start screen. The next actions give your project a fixed location on your Desktop.

  • Select File from the top menu bar.
  • Select Open Folder.
  • Select Desktop in the folder dialog.
  • Select New Folder.
  • Enter observable-ai-triage as the folder name.
  • Select Create.
  • Select Open to load the folder in Visual Studio Code.

You will see observable-ai-triage at the top of the Explorer sidebar. This confirms that every file you add lands in the correct folder.

  • Select the New File icon in the Explorer sidebar.
  • Enter alerts.json as the file name.
  • Add the first labeled alert by pasting this code:
[
  {
    "id": "alert-001",
    "source": "siem",
    "title": "Impossible travel login",
    "description": "The same account authenticated from two distant regions within twelve minutes, and both sessions completed an MFA challenge.",
    "user_email": "alex@example.com",
    "api_key": "your-api-key-here",
    "expected_route": "identity"
  }
]

What does this alert contain?

  • The id gives the alert a stable identifier for terminal output.
  • The source identifies the system that produced the alert.
  • The title provides a compact incident summary.
  • The description gives the model the evidence it needs for routing.
  • The expected_route stores the human-authored evaluation label.
  • Press Cmd+S to save alerts.json.
  • Confirm that alerts.json appears beneath observable-ai-triage in the Explorer sidebar.

Is the alert file showing an error?

  • Check that the file name ends with .json.
  • Check that every property name uses double quotes.
  • Check that the closing square bracket remains at the bottom of the file.

Ask for help with the file structure: troubleshoot my alert JSON

✔️ Awesome, I've got everything!

Your first labeled alert is saved inside the project folder.

ⓧ I'd like to double check the full code

[
  {
    "id": "alert-001",
    "source": "siem",
    "title": "Impossible travel login",
    "description": "The same account authenticated from two distant regions within twelve minutes, and both sessions completed an MFA challenge.",
    "user_email": "alex@example.com",
    "api_key": "your-api-key-here",
    "expected_route": "identity"
  }
]
Build the typed decision request

The local System One endpoint accepts a model name. It also accepts alert state plus typed questions.

Your script needs connection settings before it can build that request. These values keep the endpoint address separate from the request logic.

  • Select the New File icon in the Explorer sidebar.
  • Enter triage.py as the file name.
  • Configure the imports plus local model settings by pasting this code:
import json
import urllib.request

OLLAMA_URL = "http://localhost:11434/v1/systemone"
MODEL = "tev1:0.8b"

How does the connection setup work?

  • The json module converts Python data into the request body.
  • The urllib.request module sends the local HTTP request without an extra package.
  • The OLLAMA_URL value points to the local decision endpoint.
  • The MODEL value selects tev1:0.8b for the decision.
  • Press Cmd+S to save triage.py.
  • Confirm that Visual Studio Code shows no red syntax error underlines beneath the imports or settings.

Are the imports or settings underlined?

  • Check that both import statements begin at the left edge of the file.
  • Check that the endpoint remains inside matching double quotes.
  • Check that the model value is exactly tev1:0.8b.

Ask for help with the connection setup: diagnose my Python configuration.

Typed questions give the model a fixed decision shape. Each question asks for a different operational signal.

  • Define the routing plus assessment questions by adding this code below MODEL = "tev1:0.8b".
QUESTIONS = {
    "queue": {
        "type": "choice",
        "instructions": "Which response queue should investigate this security alert?",
        "criteria": {
            "identity": "Authentication, account, access, MFA, or identity activity.",
            "endpoint": "Host, process, malware, persistence, or workstation activity.",
            "network": "Firewall, DNS, proxy, connection, or traffic activity.",
            "cloud": "Cloud control-plane, resource, role, or permission activity.",
            "none": "The evidence is too ambiguous or none of the queues fits."
        }
    },
    "urgency": {
        "type": "score",
        "instructions": "How urgently should an analyst investigate this alert?",
        "criteria": ["Low", "Medium", "High"]
    },
    "malicious": {
        "type": "noul",
        "instructions": "Does the available evidence indicate malicious activity?",
        "criteria": {
            "true": "The evidence indicates malicious activity.",
            "false": "The evidence does not indicate malicious activity."
        }
    }
}

What do the question types return?

  • The choice question selects one response queue from the supplied criteria.
  • The score question places urgency on the supplied scale.
  • The noul question returns a probability for whether the evidence indicates malicious activity.
  • The none choice gives the model a valid answer when the evidence does not fit another queue.
  • The returned confidence measures how concentrated the choice probabilities are. It does not measure the chance that the answer is correct.
  • Press Cmd+S to save triage.py.
  • Confirm that the complete QUESTIONS dictionary has no red syntax error underlines.

Is the questions dictionary underlined?

  • Check that each opening brace has a matching closing brace.
  • Check that the three question entries remain inside QUESTIONS.
  • Check that commas remain between neighboring properties.

Ask for help checking the structure: find the syntax mismatch.

The request function is the boundary between your application and the local model. It packages the alert with the typed questions before decoding the response.

  • Create the model request by adding this function below the closing brace of QUESTIONS.
def call_model(alert):
    payload = {"model": MODEL, "state": alert, "questions": QUESTIONS}
    request = urllib.request.Request(
        OLLAMA_URL,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST"
    )
    with urllib.request.urlopen(request, timeout=30) as response:
        return json.loads(response.read())

How does the request cross the model boundary?

  • The payload groups the model name with the alert state plus typed questions.
  • The request encodes that payload as JSON for the local endpoint.
  • The request uses POST because it sends an alert body to the endpoint.
  • The timeout=30 limit prevents the request from waiting forever.
  • The returned response is decoded into Python data for routing.
  • Press Cmd+S to save triage.py.
  • Confirm that call_model(alert) has no red syntax error underlines.

Is the request function underlined?

  • Check that every line inside call_model(alert) uses four spaces of indentation.
  • Check that urlopen remains inside the with statement.
  • Check that return remains inside the response block.

Ask for help with the request function: debug my local model call.

✔️ Awesome, I've got everything!

Your typed decision request is ready. Double-check that you saved triage.py.

ⓧ I'd like to double check the full code

import json
import urllib.request

OLLAMA_URL = "http://localhost:11434/v1/systemone"
MODEL = "tev1:0.8b"

QUESTIONS = {
    "queue": {
        "type": "choice",
        "instructions": "Which response queue should investigate this security alert?",
        "criteria": {
            "identity": "Authentication, account, access, MFA, or identity activity.",
            "endpoint": "Host, process, malware, persistence, or workstation activity.",
            "network": "Firewall, DNS, proxy, connection, or traffic activity.",
            "cloud": "Cloud control-plane, resource, role, or permission activity.",
            "none": "The evidence is too ambiguous or none of the queues fits."
        }
    },
    "urgency": {
        "type": "score",
        "instructions": "How urgently should an analyst investigate this alert?",
        "criteria": ["Low", "Medium", "High"]
    },
    "malicious": {
        "type": "noul",
        "instructions": "Does the available evidence indicate malicious activity?",
        "criteria": {
            "true": "The evidence indicates malicious activity.",
            "false": "The evidence does not indicate malicious activity."
        }
    }
}


def call_model(alert):
    payload = {"model": MODEL, "state": alert, "questions": QUESTIONS}
    request = urllib.request.Request(
        OLLAMA_URL,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST"
    )
    with urllib.request.urlopen(request, timeout=30) as response:
        return json.loads(response.read())
Run the first route

The terminal run now tests the whole path. Your alert crosses the local boundary before the typed response becomes a visible routing result.

  • Select Terminal from the Visual Studio Code menu bar.
  • Select New Terminal.
  • Confirm that the terminal prompt ends inside observable-ai-triage.

Before you run the route, which typed answer do you expect the terminal to surface as the operational queue?

  • Run the first alert through the naive router by entering this command:
python3.14 triage.py naive

What should you see?

  • You will see alert-001 as the processed alert ID.
  • You will see a selected response queue after route=.
  • You will see a numeric value after confidence=.
  • You will see status=routed for the completed naive route.
  • The model response also carries the typed score plus noul answers used for urgency and malicious activity.

You have your first live typed decision. The local model can now receive a security alert and return structured routing evidence.

Did the route fail to print?

  • Return to the Ollama app from the previous step to confirm that the local service is still running.
  • Confirm that the model list from earlier still includes tev1:0.8b.
  • Compare alerts.json plus triage.py with the full-code tabs above.

Ask for help with the local route: diagnose my first decision request.

Your first alert now reaches the local decision model through Python. Next, you will make the naive router trust a deliberately unsafe low-confidence decision.

Expose the Naive Router

The first alert proved your local decision endpoint can return a typed route. A production-shaped control plane needs evidence across several alert patterns.

A naive router forwards every structurally valid answer. It has no policy for an unsafe low-confidence decision.

In this step, get ready to:
  • Expand the labeled evaluation set to three synthetic security alerts.
  • Process every alert through the naive routing path.
  • Force a repeatable low-confidence decision that the naive router trusts.
Expand the labeled alert set

A labeled evaluation set pairs each synthetic alert with the queue a human expects. The third label creates a concrete standard for reviewing the router's behavior.

  • In alerts.json from earlier, replace its current contents with the complete labeled set below:
[
  {
    "id": "alert-001",
    "source": "siem",
    "title": "Impossible travel login",
    "description": "The same account authenticated from two distant regions within twelve minutes, and both sessions completed an MFA challenge.",
    "user_email": "alex@example.com",
    "api_key": "your-api-key-here",
    "expected_route": "identity"
  },
  {
    "id": "alert-002",
    "source": "endpoint",
    "title": "Encoded command launched by a document process",
    "description": "A document process started an encoded shell command that created a scheduled task on one workstation.",
    "user_email": "sam@example.com",
    "api_key": "your-api-key-here",
    "expected_route": "endpoint"
  },
  {
    "id": "alert-003",
    "source": "pipeline",
    "title": "Unusual outbound connection",
    "description": "A build host made a short encrypted connection to a newly observed destination. No process context or threat intelligence match is available.",
    "user_email": "casey@example.com",
    "api_key": "your-api-key-here",
    "expected_route": "human_review"
  }
]

What does this alert set measure?

  • Each expected_route stores the human-authored routing label for one synthetic alert.
  • The first alert represents identity activity.
  • The second alert represents endpoint activity.
  • The third alert contains limited evidence that calls for human_review.
  • Save alerts.json.
  • Confirm the file contains three alert objects ending with alert-003.

Seeing a JSON warning?

Check the comma after the alert-002 object. Confirm that the file has one opening square bracket and one closing square bracket.

Help me find the JSON formatting problem in my alert set.

Build the naive batch path

Batch processing applies the same routing logic to every labeled alert. The command arguments select the routing mode and activate controlled test behavior.

  • In triage.py from earlier, replace the imports and constants at the top with this block:
import json
import sys
import urllib.request

OLLAMA_URL = "http://localhost:11434/v1/systemone"
MODEL = "tev1:0.8b"
ALERTS_FILE = "alerts.json"

What does this setup add?

  • The sys import gives the script access to its command-line arguments.
  • The ALERTS_FILE constant keeps the input filename in one place.
  • The existing local endpoint and model values stay unchanged.
  • Save triage.py.
  • Confirm the editor shows no red error underline beneath sys or ALERTS_FILE.

Seeing an import or constant warning?

Keep the imports at the top of triage.py. Place ALERTS_FILE directly below MODEL.

Help me check the imports and constants in triage.py.

The naive routing function preserves the model's selected queue as the operational route. Its output record keeps the raw decision visible for the experiment.

  • Add the following function below call_model(alert) in triage.py to create one naive routing record:
def route_alert(alert, mode, inject_low_confidence):
    decision = call_model(alert)
    queue_answer = decision["answers"]["queue"]
    raw_queue = queue_answer["choice"]
    queue_confidence = float(queue_answer["confidence"])

    routed_queue = raw_queue
    status = "routed"
    record = {
        "alert_id": alert["id"],
        "mode": mode,
        "raw_queue": raw_queue,
        "queue_confidence": queue_confidence,
        "routed_queue": routed_queue,
        "status": status
    }
    print(
        f"{record['alert_id']} route={record['routed_queue']} "
        f"raw={record['raw_queue']} confidence={record['queue_confidence']:.3f} "
        f"status={record['status']}"
    )
    return record

How does naive routing work?

  • The decision value holds the typed response from the local model.
  • The raw_queue value stores the model-selected queue.
  • The routed_queue = raw_queue assignment gives that selection direct operational effect.
  • The formatted output displays confidence with three decimal places.
  • Save triage.py.
  • Confirm route_alert starts below the completed call_model function.

Seeing an indentation warning?

Keep route_alert at the left edge of the file. Indent every statement inside the function with four spaces.

Help me fix the route_alert indentation in triage.py.

The entry point loads every object from alerts.json. It sends each object through the same route function.

  • Add the following entry point below route_alert in triage.py to process the complete batch:
def main():
    arguments = sys.argv[1:]
    mode = "naive" if "naive" in arguments else "guarded"
    inject_low_confidence = "--inject-low-confidence" in arguments

    with open(ALERTS_FILE, "r", encoding="utf-8") as handle:
        alerts = json.load(handle)

    records = [
        route_alert(alert, mode, inject_low_confidence)
        for alert in alerts
    ]


if __name__ == "__main__":
    main()

What does the entry point do?

  • The arguments list captures the values supplied after the script filename.
  • The mode value selects the naive path when that argument is present.
  • The inject_low_confidence value records whether the controlled fault flag was supplied.
  • The list comprehension creates one routing record per alert.
  • Save triage.py.

Before you run the batch, how many alert IDs do you expect the terminal to print?

  • Run the naive batch from the Terminal panel by using this command:
python3.14 triage.py naive

What does this command test?

The command selects naive mode without activating the controlled fault. Every alert passes through the local decision endpoint.

You should see one output line for each alert from alert-001 through alert-003. Each successful response ends with status=routed.

Good. Your one-alert path now processes the complete labeled set.

Batch stopped before all three alerts?

Confirm the Ollama app from earlier is still running. Check that all three objects in alerts.json contain valid quoted keys.

Help me diagnose why the naive batch stopped early.

Inject the low-confidence decision

Fault injection makes a reliability test repeatable. This override runs after a genuine local model call, so the surrounding HTTP path still executes.

  • In route_alert, find the line that assigns queue_confidence from the model response.
  • Insert this conditional block directly below that assignment:
    if inject_low_confidence and alert["id"] == "alert-003":
        raw_queue = "network"
        queue_confidence = 0.10

How does the injection stay deterministic?

  • The condition activates only when the command includes --inject-low-confidence.
  • The alert ID check confines the override to alert-003.
  • The override assigns the same raw queue and confidence on every injected run.
  • Save triage.py.

Before you run this check, do you think naive routing will question the injected decision or pass it through?

  • Activate the deterministic fault in naive mode by running this command:
python3.14 triage.py naive --inject-low-confidence

What does this command test?

The naive argument keeps direct model-to-route behavior active. The --inject-low-confidence flag activates the controlled override for the third alert.

Find the alert-003 line in the output. You should see route=network, confidence=0.100, and status=routed.

Why is this result unsafe?

The synthetic label expects human_review for the ambiguous third alert. The naive router gives the injected network choice direct operational effect.

The 0.100 confidence remains visible in the output. The routed status proves that visibility alone does not create a safety policy.

The cumulative files for this step are shown below:

✔️ Awesome, I've got everything!

  • Save alerts.json.
  • Save triage.py.
  • Keep the successful injected terminal output available for the checkpoint.

ⓧ I'd like to double check the full code

Your complete alerts.json file should match this reference:

[
  {
    "id": "alert-001",
    "source": "siem",
    "title": "Impossible travel login",
    "description": "The same account authenticated from two distant regions within twelve minutes, and both sessions completed an MFA challenge.",
    "user_email": "alex@example.com",
    "api_key": "your-api-key-here",
    "expected_route": "identity"
  },
  {
    "id": "alert-002",
    "source": "endpoint",
    "title": "Encoded command launched by a document process",
    "description": "A document process started an encoded shell command that created a scheduled task on one workstation.",
    "user_email": "sam@example.com",
    "api_key": "your-api-key-here",
    "expected_route": "endpoint"
  },
  {
    "id": "alert-003",
    "source": "pipeline",
    "title": "Unusual outbound connection",
    "description": "A build host made a short encrypted connection to a newly observed destination. No process context or threat intelligence match is available.",
    "user_email": "casey@example.com",
    "api_key": "your-api-key-here",
    "expected_route": "human_review"
  }
]

This reference contains the three synthetic alerts and their human-authored labels.

Your cumulative triage.py file should match this reference:

import json
import sys
import urllib.request

OLLAMA_URL = "http://localhost:11434/v1/systemone"
MODEL = "tev1:0.8b"
ALERTS_FILE = "alerts.json"

QUESTIONS = {
    "queue": {
        "type": "choice",
        "instructions": "Which response queue should investigate this security alert?",
        "criteria": {
            "identity": "Authentication, account, access, MFA, or identity activity.",
            "endpoint": "Host, process, malware, persistence, or workstation activity.",
            "network": "Firewall, DNS, proxy, connection, or traffic activity.",
            "cloud": "Cloud control-plane, resource, role, or permission activity.",
            "none": "The evidence is too ambiguous or none of the queues fits."
        }
    },
    "urgency": {
        "type": "score",
        "instructions": "How urgently should an analyst investigate this alert?",
        "criteria": ["Low", "Medium", "High"]
    },
    "malicious": {
        "type": "noul",
        "instructions": "Does the available evidence indicate malicious activity?",
        "criteria": {
            "true": "The evidence indicates malicious activity.",
            "false": "The evidence does not indicate malicious activity."
        }
    }
}


def call_model(alert):
    payload = {"model": MODEL, "state": alert, "questions": QUESTIONS}
    request = urllib.request.Request(
        OLLAMA_URL,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST"
    )
    with urllib.request.urlopen(request, timeout=30) as response:
        return json.loads(response.read())


def route_alert(alert, mode, inject_low_confidence):
    decision = call_model(alert)
    queue_answer = decision["answers"]["queue"]
    raw_queue = queue_answer["choice"]
    queue_confidence = float(queue_answer["confidence"])

    if inject_low_confidence and alert["id"] == "alert-003":
        raw_queue = "network"
        queue_confidence = 0.10

    routed_queue = raw_queue
    status = "routed"
    record = {
        "alert_id": alert["id"],
        "mode": mode,
        "raw_queue": raw_queue,
        "queue_confidence": queue_confidence,
        "routed_queue": routed_queue,
        "status": status
    }
    print(
        f"{record['alert_id']} route={record['routed_queue']} "
        f"raw={record['raw_queue']} confidence={record['queue_confidence']:.3f} "
        f"status={record['status']}"
    )
    return record


def main():
    arguments = sys.argv[1:]
    mode = "naive" if "naive" in arguments else "guarded"
    inject_low_confidence = "--inject-low-confidence" in arguments

    with open(ALERTS_FILE, "r", encoding="utf-8") as handle:
        alerts = json.load(handle)

    records = [
        route_alert(alert, mode, inject_low_confidence)
        for alert in alerts
    ]


if __name__ == "__main__":
    main()

This reference retains the local model call while adding batch loading and deterministic fault injection.

You now have repeatable evidence that a typed response can still produce an unsafe operational route. Next, you will turn that evidence into a deterministic human-review safety gate.

Add a Human-Review Safety Gate

Your naive router now exposes the exact reliability gap this project was designed to reveal. A low-confidence model decision still becomes an operational route.

Model output cannot serve as an authorization decision on its own. Confidence gating turns your routing policy into fail-safe routing. Malformed fields or an HTTP failure also send the alert to a human.

In this step, get ready to:
  • Define the confidence policy plus the allowed model queues.
  • Validate typed responses while preserving the raw model decision.
  • Prove that guarded mode sends the injected decision to human review.
Define the safety policy

The router needs a clear boundary between accepted output and human escalation. The project uses a confidence threshold of 0.60 as its teaching policy.

  • In triage.py, find the import block at the top of the file.
  • Add the error import by making the block match this code:
import json
import sys
import urllib.error
import urllib.request

Why import the error module?

The request library reports connection failures through urllib.error.URLError. Importing the error module lets the router catch that failure at its control boundary.

  • Find the constants directly above QUESTIONS.
  • Add the confidence policy by making the constants match this code:
OLLAMA_URL = "http://localhost:11434/v1/systemone"
MODEL = "tev1:0.8b"
CONFIDENCE_THRESHOLD = 0.60
ALERTS_FILE = "alerts.json"

What does the threshold mean?

The threshold is a deterministic routing rule owned by your application. A queue confidence below 0.60 triggers human review in guarded mode.

Confidence measures how concentrated the model probabilities are. It does not measure the chance that the answer is correct.

  • Save triage.py.
  • Confirm that naive behavior remains unchanged by running this command:
python3.14 triage.py naive --inject-low-confidence

What does this check prove?

You should still see alert-003 routed to network with confidence 0.100. Its status remains routed because the safety gate is not active in naive mode.

Router cannot reach the model?

Confirm that the Ollama app from earlier is still running. A stopped local service prevents call_model() from receiving a decision.

Help me diagnose why the local router cannot reach Ollama.

Build guarded routing

A valid decision needs a supported queue plus readable typed fields. The router also keeps the model's raw answer separate from the final operational route.

  • In triage.py, find the closing brace of QUESTIONS.
  • Add the allowed queue set directly below QUESTIONS using this code:
ALLOWED_QUEUES = set(QUESTIONS["queue"]["criteria"])

Why derive the allowed queues?

The allowlist comes from the same criteria used in the model request. This keeps response validation aligned with the queues your application actually offered.

  • Find the existing route_alert() function in triage.py.
  • Replace that function with the guarded version below:
def route_alert(alert, mode, inject_low_confidence):
    raw_queue = "unavailable"
    queue_confidence = 0.0
    try:
        decision = call_model(alert)
        queue_answer = decision["answers"]["queue"]
        raw_queue = queue_answer["choice"]
        queue_confidence = float(queue_answer["confidence"])
        urgency_score = float(decision["answers"]["urgency"]["score"])
        malicious_probability = float(decision["answers"]["malicious"]["noul"])
        if raw_queue not in ALLOWED_QUEUES:
            raise ValueError("Model returned an unsupported queue")
        if inject_low_confidence and alert["id"] == "alert-003":
            raw_queue = "network"
            queue_confidence = 0.10
        if mode == "guarded" and raw_queue == "none":
            routed_queue = "human_review"
            status = "fallback_none"
        elif mode == "guarded" and queue_confidence < CONFIDENCE_THRESHOLD:
            routed_queue = "human_review"
            status = "fallback_low_confidence"
        else:
            routed_queue = raw_queue
            status = "routed"
    except (urllib.error.URLError, TimeoutError, KeyError, TypeError, ValueError) as exc:
        routed_queue = "human_review"
        status = "fallback_error"
    print(f"{alert['id']} route={routed_queue} raw={raw_queue} confidence={queue_confidence:.3f} status={status}")

How does the safety gate work?

  • The response lookups validate that the queue answer plus its typed fields exist.
  • The float() conversions reject confidence or score values that cannot be read as numbers.
  • The allowlist rejects any queue that was not defined in QUESTIONS.
  • Guarded mode converts none or confidence below 0.60 into human_review.
  • Request failures plus malformed responses produce fallback_error.

Keep both routing views

The raw_queue value shows what the model selected. The routed_queue value shows what the control plane allowed.

That separation lets an operator diagnose a fallback without treating the rejected answer as an approved route.

  • Save triage.py.
  • Compare your cumulative file with the reference below if you want to check every line.

✔️ Awesome, I've got everything!

  • Keep triage.py saved in the observable-ai-triage folder.

ⓧ I'd like to double check the full code

Use this reference to compare the complete guarded router at the end of this step.

import json
import sys
import urllib.error
import urllib.request

OLLAMA_URL = "http://localhost:11434/v1/systemone"
MODEL = "tev1:0.8b"
CONFIDENCE_THRESHOLD = 0.60
ALERTS_FILE = "alerts.json"

QUESTIONS = {
    "queue": {
        "type": "choice",
        "instructions": "Which response queue should investigate this security alert?",
        "criteria": {
            "identity": "Authentication, account, access, MFA, or identity activity.",
            "endpoint": "Host, process, malware, persistence, or workstation activity.",
            "network": "Firewall, DNS, proxy, connection, or traffic activity.",
            "cloud": "Cloud control-plane, resource, role, or permission activity.",
            "none": "The evidence is too ambiguous or none of the queues fits."
        }
    },
    "urgency": {
        "type": "score",
        "instructions": "How urgently should an analyst investigate this alert?",
        "criteria": ["Low", "Medium", "High"]
    },
    "malicious": {
        "type": "noul",
        "instructions": "Does the available evidence indicate malicious activity?",
        "criteria": {
            "true": "The evidence indicates malicious activity.",
            "false": "The evidence does not indicate malicious activity."
        }
    }
}

ALLOWED_QUEUES = set(QUESTIONS["queue"]["criteria"])


def call_model(alert):
    payload = {
        "model": MODEL,
        "state": alert,
        "questions": QUESTIONS
    }
    request = urllib.request.Request(
        OLLAMA_URL,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST"
    )
    with urllib.request.urlopen(request, timeout=30) as response:
        return json.loads(response.read())


def route_alert(alert, mode, inject_low_confidence):
    raw_queue = "unavailable"
    queue_confidence = 0.0
    try:
        decision = call_model(alert)
        queue_answer = decision["answers"]["queue"]
        raw_queue = queue_answer["choice"]
        queue_confidence = float(queue_answer["confidence"])
        urgency_score = float(decision["answers"]["urgency"]["score"])
        malicious_probability = float(decision["answers"]["malicious"]["noul"])

        if raw_queue not in ALLOWED_QUEUES:
            raise ValueError("Model returned an unsupported queue")

        if inject_low_confidence and alert["id"] == "alert-003":
            raw_queue = "network"
            queue_confidence = 0.10

        if mode == "guarded" and raw_queue == "none":
            routed_queue = "human_review"
            status = "fallback_none"
        elif mode == "guarded" and queue_confidence < CONFIDENCE_THRESHOLD:
            routed_queue = "human_review"
            status = "fallback_low_confidence"
        else:
            routed_queue = raw_queue
            status = "routed"
    except (urllib.error.URLError, TimeoutError, KeyError, TypeError, ValueError) as exc:
        routed_queue = "human_review"
        status = "fallback_error"

    print(
        f"{alert['id']} route={routed_queue} "
        f"raw={raw_queue} confidence={queue_confidence:.3f} status={status}"
    )


def main():
    arguments = sys.argv[1:]
    mode = "naive" if "naive" in arguments else "guarded"
    inject_low_confidence = "--inject-low-confidence" in arguments

    with open(ALERTS_FILE, "r", encoding="utf-8") as handle:
        alerts = json.load(handle)

    for alert in alerts:
        route_alert(alert, mode, inject_low_confidence)


if __name__ == "__main__":
    main()

What should match?

Confirm that your file contains the confidence threshold plus the derived queue allowlist. Confirm that route_alert() preserves the raw values before selecting the final route.

Prove the fallback works

The injected decision still claims that network is the best queue. Before you run the guarded router, do you expect the operational route to trust that answer?

  • Run the guarded router with deterministic fault injection using this command:
python3.14 triage.py guarded --inject-low-confidence

What should you see?

Find the output line for alert-003. You should see route=human_review plus raw=network.

The same line should show confidence=0.100 plus status=fallback_low_confidence.

Still seeing the network route?

Confirm that the command includes guarded. Check that the low-confidence branch compares queue_confidence with CONFIDENCE_THRESHOLD.

Help me debug why alert-003 is not falling back to human review.

That is the control plane doing its job. The model's answer remains visible for diagnosis while the unsafe route is blocked.

Before you compare modes, do you expect the naive router to enforce the new confidence policy?

  • Run the same injected decision through naive mode using this command:
python3.14 triage.py naive --inject-low-confidence

What does the comparison show?

Naive mode should still print route=network with status=routed for alert-003. Guarded mode applies the policy that converts the same raw decision into human review.

Your router now treats model output as evidence instead of authority. Next, you will capture each decision as telemetry and measure how the guarded routes compare with the human-authored labels.

Measure Reliability with Telemetry

Your safety gate now protects the local Ollama decision path. Low-confidence decisions reach human review instead of an operational queue.

A guarded system also needs evidence about its decisions. This step measures latency, fallback behavior, and label agreement after every run.

In this step, get ready to:
  • Measure each model request with a high-resolution timer.
  • Write one JSON Lines telemetry record for every alert.
  • Compare final routes with human-authored labels across guarded runs.
Add the telemetry foundation

Telemetry needs a stable clock plus a predictable output file. Python's high-resolution performance clock measures request duration without depending on the system clock.

  • Switch back to triage.py in Visual Studio Code.
  • Update the imports plus configuration constants by replacing the top of triage.py with:
import json
import sys
import time
import urllib.error
import urllib.request

OLLAMA_URL = "http://localhost:11434/v1/systemone"
MODEL = "tev1:0.8b"
CONFIDENCE_THRESHOLD = 0.60
ALERTS_FILE = "alerts.json"
TELEMETRY_FILE = "telemetry.jsonl"

What does this setup add?

  • The time module provides the clock used to measure each request.
  • TELEMETRY_FILE gives every telemetry write the same destination.

Each telemetry record also needs a timestamp. A dedicated append helper keeps the JSON serialization separate from the routing policy.

  • Add these telemetry helpers immediately above route_alert():
def utc_timestamp():
    return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())


def append_telemetry(record):
    with open(TELEMETRY_FILE, "a", encoding="utf-8") as handle:
        handle.write(json.dumps(record, sort_keys=True) + "\n")

How do the helpers work?

  • utc_timestamp() formats the current UTC time for each routing event.
  • append_telemetry() serializes one record as JSON.
  • The trailing newline separates the records inside telemetry.jsonl.
  • Save triage.py.

Before you run the router, do you expect these new helpers to preserve the existing safety-gate result?

  • Check the guarded route by running:
python3.14 triage.py guarded --inject-low-confidence

What should you see?

You should still see alert-003 routed to human_review with status=fallback_low_confidence. This confirms the telemetry foundation loads without weakening the safety gate.

Seeing an import or name error?

Check that import time sits with the other imports. Confirm that TELEMETRY_FILE appears above QUESTIONS.

Make sure both helper functions sit at the left edge of triage.py. They must remain outside every other function.

Help me fix the telemetry helper setup in triage.py.

Record each routing outcome

A useful routing event connects the model response to the final operational decision. It also records how long that decision took plus whether the route matched the human-authored label.

  • In route_alert(), replace the opening variable setup with:
def route_alert(alert, mode, inject_low_confidence):
    started = time.perf_counter()
    raw_queue = "unavailable"
    queue_confidence = 0.0
    urgency_score = None
    malicious_probability = None
    error = None

Why start the timer here?

time.perf_counter() starts immediately before the model request. The measured duration therefore covers the decision boundary plus the routing logic that follows it.

The default fields also give failed requests a complete telemetry shape. Operators can compare successful decisions with fallback events using the same keys.

  • Replace the existing output section beneath the except block with:
    latency_ms = round((time.perf_counter() - started) * 1000, 2)
    record = {
        "timestamp": utc_timestamp(),
        "alert_id": alert["id"],
        "model": MODEL,
        "mode": mode,
        "raw_queue": raw_queue,
        "queue_confidence": queue_confidence,
        "routed_queue": routed_queue,
        "status": status,
        "latency_ms": latency_ms,
        "urgency_score": urgency_score,
        "malicious_probability": malicious_probability,
        "expected_route": alert["expected_route"],
        "correct": routed_queue == alert["expected_route"],
        "error": error
    }
    append_telemetry(record)
    print(
        f"{record['alert_id']} route={record['routed_queue']} "
        f"raw={record['raw_queue']} confidence={record['queue_confidence']:.3f} "
        f"latency_ms={record['latency_ms']:.2f} status={record['status']}"
    )
    return record

What does the routing record capture?

The record preserves the model's raw queue plus the final routed queue. The status field explains whether the control plane routed normally or used a fallback.

The correct field compares routed_queue with expected_route. The record excludes the raw alert body plus its sensitive fields.

Returning the record lets main() calculate batch-level results after every alert has finished.

  • Save triage.py.

Before you run the updated router, which output field should prove that request timing is now active?

  • Generate routing telemetry by running:
python3.14 triage.py guarded --inject-low-confidence

What changed in the output?

Each alert line now includes a numeric latency_ms value. You should also see alert-003 reach human_review through fallback_low_confidence.

That is the first observable event from the complete control path. It connects model behavior, policy behavior, and runtime performance.

  • Validate the generated telemetry.jsonl records by running:
python3.14 -m json --json-lines telemetry.jsonl

What should the JSON validator show?

You should see one formatted JSON object per alert. Each object should contain latency_ms, status, routed_queue, and correct.

You should not see alert titles, descriptions, email addresses, or API key values. The telemetry records operational evidence without copying the raw alert body.

Missing or invalid telemetry records?

Confirm that append_telemetry(record) appears after the record dictionary. Check that the helper writes "\n" after each serialized record.

If telemetry.jsonl is missing, confirm that you ran the command from the observable-ai-triage folder.

Help me diagnose invalid or missing JSON Lines telemetry.

Compare guarded runs

Individual telemetry events explain one alert. A labeled evaluation summarizes whether the complete batch matched its expected operational routes.

  • Replace the existing main() function plus the script entry point with:
def main():
    arguments = sys.argv[1:]
    mode = "naive" if "naive" in arguments else "guarded"
    inject_low_confidence = "--inject-low-confidence" in arguments

    with open(ALERTS_FILE, "r", encoding="utf-8") as handle:
        alerts = json.load(handle)

    with open(TELEMETRY_FILE, "w", encoding="utf-8"):
        pass

    records = [
        route_alert(alert, mode, inject_low_confidence)
        for alert in alerts
    ]
    matched = sum(record["correct"] for record in records)
    fallbacks = sum(record["status"].startswith("fallback") for record in records)
    print(
        f"\nEvaluation: {matched}/{len(records)} routes matched labels; "
        f"{fallbacks} fallbacks; telemetry={TELEMETRY_FILE}"
    )


if __name__ == "__main__":
    main()

How does the evaluation work?

main() clears telemetry.jsonl before processing the batch. Every execution therefore produces one fresh set of three records.

The matched count totals records whose final route agrees with the human-authored label. The fallbacks count totals every status beginning with fallback.

Confidence describes how concentrated the model's probabilities are. Label agreement shows whether the final route matched the expected operational outcome.

  • Save triage.py.

Before you run the baseline, which alerts do you expect the guarded policy to route directly?

  • Create a guarded baseline without fault injection by running:
python3.14 triage.py guarded

Read the baseline evidence

You should see three routing lines followed by an Evaluation: summary. Note the final route, status, and latency for each alert.

Local model decisions can vary with the available evidence. Use the expected labels plus fallback statuses to interpret the run.

  • Inspect the fresh baseline records by running:
python3.14 -m json --json-lines telemetry.jsonl

What does the baseline prove?

The file should contain exactly three valid JSON objects. Their mode values should be guarded.

Compare each raw_queue with its routed_queue plus expected_route. These fields separate model behavior from policy behavior.

Before the final run, do you expect the injected low-confidence decision to remain in the network queue?

  • Run the deterministic fault-injection comparison with:
python3.14 triage.py guarded --inject-low-confidence

The safety policy is measurable

You should see alert-003 with route=human_review, raw=network, confidence=0.100, and status=fallback_low_confidence.

The evaluation summary now reports matched routes plus fallback count plus the telemetry location. The injected decision remains available for diagnosis while the operational route stays safe.

  • Validate the final fault-injection telemetry by running:
python3.14 -m json --json-lines telemetry.jsonl

What should the final records confirm?

You should see three fresh JSON records containing latency_ms, status, routed_queue, and correct.

The alert-003 record should preserve network as the raw queue. Its final route should be human_review.

Missing the evaluation summary?

Confirm that every route_alert() call returns its record. Check that the records list appears before the matched and fallbacks calculations.

If old telemetry remains, confirm that main() opens TELEMETRY_FILE in write mode before processing alerts.

Help me fix the missing evaluation summary or stale telemetry.

✔️ Awesome, I've got everything!

Your guarded router now writes fresh telemetry plus a labeled evaluation after every run. Save triage.py before completing the checkpoint.

ⓧ I'd like to double check the full code

import json
import sys
import time
import urllib.error
import urllib.request

OLLAMA_URL = "http://localhost:11434/v1/systemone"
MODEL = "tev1:0.8b"
CONFIDENCE_THRESHOLD = 0.60
ALERTS_FILE = "alerts.json"
TELEMETRY_FILE = "telemetry.jsonl"

QUESTIONS = {
    "queue": {
        "type": "choice",
        "instructions": "Which response queue should investigate this security alert?",
        "criteria": {
            "identity": "Authentication, account, access, MFA, or identity activity.",
            "endpoint": "Host, process, malware, persistence, or workstation activity.",
            "network": "Firewall, DNS, proxy, connection, or traffic activity.",
            "cloud": "Cloud control-plane, resource, role, or permission activity.",
            "none": "The evidence is too ambiguous or none of the queues fits."
        }
    },
    "urgency": {
        "type": "score",
        "instructions": "How urgently should an analyst investigate this alert?",
        "criteria": ["Low", "Medium", "High"]
    },
    "malicious": {
        "type": "noul",
        "instructions": "Does the available evidence indicate malicious activity?",
        "criteria": {
            "true": "The evidence indicates malicious activity.",
            "false": "The evidence does not indicate malicious activity."
        }
    }
}

ALLOWED_QUEUES = set(QUESTIONS["queue"]["criteria"])


def call_model(alert):
    payload = {
        "model": MODEL,
        "state": alert,
        "questions": QUESTIONS
    }
    request = urllib.request.Request(
        OLLAMA_URL,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST"
    )
    with urllib.request.urlopen(request, timeout=30) as response:
        return json.loads(response.read())


def utc_timestamp():
    return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())


def append_telemetry(record):
    with open(TELEMETRY_FILE, "a", encoding="utf-8") as handle:
        handle.write(json.dumps(record, sort_keys=True) + "\n")


def route_alert(alert, mode, inject_low_confidence):
    started = time.perf_counter()
    raw_queue = "unavailable"
    queue_confidence = 0.0
    urgency_score = None
    malicious_probability = None
    error = None

    try:
        decision = call_model(alert)
        queue_answer = decision["answers"]["queue"]
        raw_queue = queue_answer["choice"]
        queue_confidence = float(queue_answer["confidence"])
        urgency_score = float(decision["answers"]["urgency"]["score"])
        malicious_probability = float(decision["answers"]["malicious"]["noul"])

        if raw_queue not in ALLOWED_QUEUES:
            raise ValueError("Model returned an unsupported queue")

        if inject_low_confidence and alert["id"] == "alert-003":
            raw_queue = "network"
            queue_confidence = 0.10

        if mode == "guarded" and raw_queue == "none":
            routed_queue = "human_review"
            status = "fallback_none"
        elif mode == "guarded" and queue_confidence < CONFIDENCE_THRESHOLD:
            routed_queue = "human_review"
            status = "fallback_low_confidence"
        else:
            routed_queue = raw_queue
            status = "routed"
    except (urllib.error.URLError, TimeoutError, KeyError, TypeError, ValueError) as exc:
        routed_queue = "human_review"
        status = "fallback_error"
        error = type(exc).__name__

    latency_ms = round((time.perf_counter() - started) * 1000, 2)
    record = {
        "timestamp": utc_timestamp(),
        "alert_id": alert["id"],
        "model": MODEL,
        "mode": mode,
        "raw_queue": raw_queue,
        "queue_confidence": queue_confidence,
        "routed_queue": routed_queue,
        "status": status,
        "latency_ms": latency_ms,
        "urgency_score": urgency_score,
        "malicious_probability": malicious_probability,
        "expected_route": alert["expected_route"],
        "correct": routed_queue == alert["expected_route"],
        "error": error
    }
    append_telemetry(record)
    print(
        f"{record['alert_id']} route={record['routed_queue']} "
        f"raw={record['raw_queue']} confidence={record['queue_confidence']:.3f} "
        f"latency_ms={record['latency_ms']:.2f} status={record['status']}"
    )
    return record


def main():
    arguments = sys.argv[1:]
    mode = "naive" if "naive" in arguments else "guarded"
    inject_low_confidence = "--inject-low-confidence" in arguments

    with open(ALERTS_FILE, "r", encoding="utf-8") as handle:
        alerts = json.load(handle)

    with open(TELEMETRY_FILE, "w", encoding="utf-8"):
        pass

    records = [
        route_alert(alert, mode, inject_low_confidence)
        for alert in alerts
    ]
    matched = sum(record["correct"] for record in records)
    fallbacks = sum(record["status"].startswith("fallback") for record in records)
    print(
        f"\nEvaluation: {matched}/{len(records)} routes matched labels; "
        f"{fallbacks} fallbacks; telemetry={TELEMETRY_FILE}"
    )


if __name__ == "__main__":
    main()

How to use this reference

Compare this file with your saved triage.py from top to bottom. The telemetry helpers, routing record, and evaluation logic should match exactly.

You have turned a guarded AI router into an observable control plane. Its routing policy, latency, failures, and label agreement now leave evidence after every run.

Secret mission

Redact Sensitive Fields Before Inference

Protect the model boundary with recursive input redaction. Prove that every email address and API key becomes `[REDACTED]` before inference. Keep the existing routing, telemetry, and evaluation workflow intact.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to keep the local setup, pause its running process, or remove it. The project creates no paid cloud or API resources, so leaving the files plus model on your Mac has no ongoing service cost.

Resources you used:

  • Local project folder containing alerts.json, triage.py, and telemetry.jsonl.
  • Ollama 0.35.1 application with the local tev1:0.8b model.

Keep everything running

No action is needed. Choose this if you want to rerun the evaluation or continue extending the control plane.

  • Keep the project folder so the alert set, router, redaction logic, telemetry, and evaluation workflow remain available.
  • Leave Ollama installed so the local decision endpoint remains ready for future runs.
  • Retain the tev1:0.8b model so you do not need to download it again.
  • Expect no ongoing service cost while these local resources sit idle.

Pause - I'll come back to this later

Pause the local server while preserving every file for a later session. This frees the running process without deleting your model.

  • Quit the Ollama app to stop the local server.
  • Keep the project folder on your Mac.
  • Keep Ollama's local data so tev1:0.8b remains available.
  • Reopen the Ollama app before your next run of triage.py.

Delete - I don't want to use this again

Deletion is permanent. The sequence below gives you a final chance to preserve evidence before it removes the local files.

  • Save any telemetry evidence you want to retain outside the project folder.
  • Use Finder to locate the project folder that contains alerts.json, triage.py, and telemetry.jsonl.
  • Move the project folder to Trash.
  • Empty your Mac's Trash to remove the project files permanently.
  • Open the official Ollama macOS guide.
  • Follow the official uninstall section to remove the Ollama application, CLI link, local models, configuration, caches, and logs.
  • Confirm that /Applications/Ollama.app, /usr/local/bin/ollama, and ~/.ollama no longer exist.

Nice Work!

Nice Work!

You did it! You built an observable AI incident triage control plane that turns local model output into guarded operational routes.

You've learned how to:

  • Built a local typed AI alert router that sends synthetic security alerts to Ollama. The terminal displays the queue chosen for each alert.
  • Exposed the risk of naive AI automation with a deterministic low-confidence fault. Added a human-review safety gate for uncertain or failed model decisions.
  • Recorded routing evidence as JSONL telemetry without raw alert bodies. Evaluated final routes against human-authored labels.
  • Secret Mission: Added recursive input redaction that replaces sensitive fields before inference. Proved the model receives the sanitized payload. Preserved the local evaluation workflow.

Ready to quiz yourself?