Build a Clinical-AI Gateway

Route clinical AI apps through one self-hosted, offline-first LLM gateway.

Introduction

30 Second Summary

AI diagnostic support pilots in global health stall for the same reasons every time. They cost too much per patient, they go dark during outbreaks, and they are useless offline.

In this project, you will build a self-hosted clinical-AI gateway that routes any clinical app through one vendor-agnostic endpoint. You will prove it works offline, respects data sovereignty, and shows funders the cost per patient served.

What You'll Build

Run a single Docker Compose command, send a clinical triage question with the network cut, and watch your local model still answer. Then open the Grafana dashboard and show a funder exactly what each patient costs.

By the end of this project, you'll have:

  • A vendor-agnostic gateway that serves every clinical app through one OpenAI-compatible endpoint, letting you swap models by editing a config file with zero app code changes.
  • An offline-first floor proven by cutting connectivity and watching the local Ollama model still answer a triage question.
  • A funder dashboard showing per-capita cost, with five scripted validation drills passing sovereignty, resilience, multilingual access, and clinical safety bars in one run.
  • Secret Mission: Write the population-impact briefing, Mermaid system diagram, and two-audience walkthrough that gets the gateway funded at the coalition steering meeting.

Are there any prerequisites?

This project assumes you have Cursor, Docker Desktop, and Git installed, along with a GitHub account, a Linear account, and an Anthropic API key with existing credits.

You will also need at least 8 GB of RAM available for the local model. Step 1 will walk you through verifying everything is ready.

Before We Start

A ministry-of-health AI pilot just stalled because it was too costly per patient, went dark during an outbreak, and was useless offline. Before you build the gateway that fixes this, lock in what you're solving and why it matters that it works offline, in-country, and at a cost a ministry can defend.

Stand Up the Gateway Stack

Every validation drill in this project needs a running platform underneath it. Offline reach, sovereignty, failover, cost governance. None of these can be tested until the gateway stack is live.

Design comes before build. You will commit architecture decision records, an implementation plan, and a topology diagram before writing a single line of infrastructure config. Then you will bring up all eight services in one command and prove the contract works end-to-end.

In this step, get ready to:
  • Set up delivery infrastructure with a GitHub repo, Linear project, and design-first commit.
  • Write the gateway configuration files for eight containerized services.
  • Bring the stack up and verify the OpenAI-compatible contract works.
Set up delivery infrastructure

The gateway ships through a private GitHub repo and a Linear project. You will create both, then commit all design artifacts before any infrastructure code.

  • In your browser, go to GitHub and create a new private repository named clinical-ai-gateway.
  • Leave the repository empty (no README, no .gitignore, no license) so you control the initial commit.
  • In a new browser tab, go to Linear and create a new project named Clinical AI Gateway.
  • Create five tickets in Linear, one for each build step: Gateway Stack, Contract + Offline Floor, Sovereignty + Safety, Resilience + Cost, Dashboard + Drills.

What is Linear?

Linear is a project-tracking tool used by engineering teams. Each ticket represents a unit of work. You will close tickets by referencing them in pull requests as you complete each step.

  • Open your terminal in Cursor by pressing Ctrl+` (Windows) or Cmd+` (macOS).
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Clone your empty repo and move into it by running these commands:
git clone https://github.com/[[GITHUB_USERNAME="your-github-username"]]/clinical-ai-gateway.git
cd clinical-ai-gateway
  • Create the project directory structure by running:
mkdir -p docs/adr
  • Confirm the folder structure exists by running:
ls docs

You should see adr listed in the output.

Don't see the adr folder?

Make sure you ran mkdir -p docs/adr from inside the clinical-ai-gateway folder. Run pwd to check your current directory.

help me figure out why my directory structure wasn't created

Now create the five architecture decision records. Each ADR documents a key design choice and its consequences.

  • Create docs/adr/ADR-001-vendor-agnostic-routing.md in Cursor with this content:
# ADR-001: Vendor-Agnostic Model Routing via LiteLLM

## Status: Accepted

## Context
Clinical AI pilots stall when tied to a single vendor. Cost spikes, outages, and data-residency requirements all require model portability.

## Decision
Use LiteLLM Proxy as the single OpenAI-compatible gateway. All clinical apps call one endpoint; model selection is config-driven.

## Consequences
Positive: Apps never change when models change. Negative: LiteLLM is a dependency; pin to a tested version.
  • Create docs/adr/ADR-002-offline-first-local-model.md:
# ADR-002: Offline-First Local Model via Ollama

## Status: Accepted

## Context
Clinics in low-resource settings lose connectivity during outbreaks — exactly when AI support is most needed.

## Decision
Run Ollama with llama3.2:3b as the guaranteed offline floor. LiteLLM falls back to it automatically when cloud models are unreachable.

## Consequences
Positive: Clinic always gets an answer. Negative: Local model trades capability for reach; responses must be bounded by safety guardrails.
  • Create docs/adr/ADR-003-phi-redaction-sovereignty.md:
# ADR-003: PHI Redaction and Data Sovereignty via Presidio

## Status: Accepted

## Context
Ministries of health cannot send patient identifiers across borders. Pilots that cannot guarantee this cannot be deployed.

## Decision
All outbound requests pass through Microsoft Presidio for PHI detection and redaction. A region-routing assertion blocks cross-border requests at the middleware layer.

## Consequences
Positive: Hard sovereignty guarantee. Negative: Adds latency; Presidio must be kept updated with local identifier patterns.
  • Create docs/adr/ADR-004-per-capita-cost-governance.md:
# ADR-004: Per-Capita Cost Governance via LiteLLM Teams and Redis

## Status: Accepted

## Context
Funders measure impact per patient served, not per API call. Budget overruns kill programs.

## Decision
Use LiteLLM virtual keys scoped to teams representing health programs. Set max_budget per team. When budget is exhausted, budget_fallbacks route to the zero-marginal-cost local model.

## Consequences
Positive: Cost is governed the way funders measure it. Negative: Budget state lives in Redis; Redis failure could allow overspend.
  • Create docs/adr/ADR-005-clinical-safety-guardrails.md:
# ADR-005: Clinical Safety Guardrails via System Prompt Injection

## Status: Accepted

## Context
A local model with bounded capability must not overreach its clinical scope. Overconfident responses in triage contexts are a patient-safety risk.

## Decision
Inject a safety system prompt for clinical-risk patterns (diagnosis, prescribe, emergency). The prompt instructs the model to flag uncertainty, provide only bounded support, and escalate to a clinician.

## Consequences
Positive: Reduces overreach risk. Negative: System prompt injection is not a certified medical device control; this is deployment infrastructure, not a diagnostic device.
  • Verify all five ADR files exist by running:
ls docs/adr/

You should see all five ADR files listed.

What is an ADR?

An Architecture Decision Record captures a design choice, its context, and its consequences in a short document. Committing ADRs before code establishes a design-first discipline. Anyone joining the project later can read these to understand why the system is built this way.

  • Create docs/implementation-plan.md with this content:
# Implementation Plan: Clinical-AI Gateway

## Phases

### Phase 1: Gateway Stack (Step 1)
- Acceptance: docker compose up -d healthy, curl /health returns 200, local model pulled
- Risk: Docker resource limits on 8GB RAM machine

### Phase 2: Contract and Offline Floor (Step 2)
- Acceptance: Offline drill passes, cloud model confirmed via headers, failback confirmed
- Risk: Ollama pull time on slow connections

### Phase 3: Sovereignty and Safety (Step 3)
- Acceptance: PHI redacted, cross-border blocked with 403, safety guardrail triggers
- Risk: Presidio custom pattern configuration complexity

### Phase 4: Resilience, Language, Cost (Step 4)
- Acceptance: Failover drill passes, French response confirmed, spend tracked
- Risk: LiteLLM budget_fallbacks config syntax

### Phase 5: Dashboard and Drills (Step 5)
- Acceptance: All five bars pass in validate.py, Grafana panels visible
- Risk: Prometheus metric names may change across LiteLLM versions
  • Create docs/architecture.mmd with the Mermaid topology diagram:
graph TD
    Client[Clinical App] -->|POST /chat/completions| Gateway[LiteLLM Proxy :4000]
    Gateway -->|PHI check| Presidio[Presidio Analyzer+Anonymizer]
    Gateway -->|online| Claude[Anthropic Claude API]
    Gateway -->|offline fallback| Ollama[Ollama llama3.2:3b :11434]
    Gateway -->|budget state| Redis[Redis :6379]
    Gateway -->|spend log| Postgres[PostgreSQL]
    Gateway -->|metrics| Prometheus[Prometheus :9090]
    Prometheus -->|datasource| Grafana[Grafana Dashboard :3000]
    Grafana -->|per-capita cost view| Funder[Funder / Ministry]
  • Create README.md at the project root:
# Clinical-AI Gateway

A self-hosted, vendor-agnostic AI gateway for global health. Routes any clinical app through one OpenAI-compatible endpoint. Proven offline, sovereign, multilingual, and cost-governed.

## Quick Start

```bash
cp .env.example .env  # add your ANTHROPIC_API_KEY
docker compose up -d
docker compose exec ollama ollama pull llama3.2:3b
curl http://localhost:4000/health
```

## What's Inside
- LiteLLM: routing brain
- Ollama: offline model floor
- Redis: rate-limit and budget state
- PostgreSQL: spend store
- Prometheus + Grafana: funder dashboard
- Presidio: PHI redaction

See docs/ for ADRs, implementation plan, architecture diagram, and ops guide.
  • Commit and push the design-first initial commit by running:
git add .
git commit -m "docs: design-first initial commit (ADRs, impl plan, architecture)"
git push -u origin main

You should see the push succeed and your files appear on GitHub.

Push rejected or permission denied?

Make sure the repo you created on GitHub matches the clone URL exactly. If you see a credentials error, confirm your Git authentication is set up (SSH key or GitHub CLI).

help me fix a git push permission error to my clinical-ai-gateway repo

Write the gateway configuration

The gateway runs eight services orchestrated by Docker Compose. You will write the configuration on a feature branch so it can be merged via pull request.

  • Create and switch to a feature branch by running:
git checkout -b feat/gateway-stack
  • Create docker-compose.yml in Cursor. Start with the LiteLLM, Ollama, and Redis services:
version: '3.8'
services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    ports:
      - '4000:4000'
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    environment:
      - DATABASE_URL=postgresql://litellm:litellm@postgres:5432/litellm
      - REDIS_URL=redis://redis:6379
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
    command: ['--config', '/app/config.yaml', '--port', '4000']
    depends_on:
      - redis
      - postgres
  ollama:
    image: ollama/ollama:latest
    ports:
      - '11434:11434'
    volumes:
      - ollama_data:/root/.ollama
  redis:
    image: redis:7-alpine
    ports:
      - '6379:6379'
    volumes:
      - redis_data:/data

What do these three services do?

LiteLLM is the routing brain. It exposes a single OpenAI-compatible endpoint on port 4000 and decides which model handles each request based on config. Ollama runs the local open-weight model that guarantees offline reach. Redis holds rate-limit counters and budget state so LiteLLM can enforce cost caps.

  • Add the PostgreSQL, Prometheus, and Grafana services below the Redis service in the same file:
  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: litellm
      POSTGRES_DB: litellm
    volumes:
      - postgres_data:/var/lib/postgresql/data
  prometheus:
    image: prom/prometheus:latest
    ports:
      - '9090:9090'
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
  grafana:
    image: grafana/grafana:latest
    ports:
      - '3000:3000'
    volumes:
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

What do these three services do?

PostgreSQL stores spend logs so LiteLLM can enforce per-team budgets persistently. Prometheus scrapes metrics from LiteLLM for the funder dashboard. Grafana visualizes those metrics as panels showing cost per patient served.

  • Add the Presidio services and the volumes section below Grafana to complete the file:
  presidio-analyzer:
    image: mcr.microsoft.com/presidio-analyzer:latest
    ports:
      - '5002:5002'
  presidio-anonymizer:
    image: mcr.microsoft.com/presidio-anonymizer:latest
    ports:
      - '5001:5001'
volumes:
  ollama_data:
  redis_data:
  postgres_data:
  prometheus_data:
  grafana_data:

What does Presidio do?

Microsoft Presidio detects and redacts protected health information (PHI) before any request leaves the local network. The analyzer identifies entities like names and patient IDs. The anonymizer replaces them with redaction tags.

✔️ Awesome, I've got everything!

Great. Make sure you have saved docker-compose.yml.

ⓧ I'd like to double check the full code

Scroll up and confirm your docker-compose.yml contains all three chunks in order: (1) the version header with litellm, ollama, and redis services, (2) postgres, prometheus, and grafana services, (3) presidio-analyzer, presidio-anonymizer, and the volumes block. The complete file should have eight services and five named volumes.

  • Create litellm_config.yaml with the model routing and fallback configuration:
model_list:
  - model_name: claude-primary
    litellm_params:
      model: anthropic/claude-3-5-haiku-20241022
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: ollama-local
    litellm_params:
      model: ollama/llama3.2:3b
      api_base: http://ollama:11434

router_settings:
  fallbacks:
    - claude-primary:
        - ollama-local
  num_retries: 2
  request_timeout: 5
  allowed_fails: 2
  cooldown_time: 30

litellm_settings:
  callbacks: ['prometheus']
  success_callback: ['prometheus']
  failure_callback: ['prometheus']

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  redis_url: os.environ/REDIS_URL

What does this config do?

The model_list defines two model groups: claude-primary (your Anthropic Claude credits) and ollama-local (the offline floor). The fallbacks setting means if Claude fails after two retries, LiteLLM automatically routes to Ollama. The os.environ/ prefix tells LiteLLM to read values from environment variables at startup.

  • Create prometheus.yml for the metrics scraper:
global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'litellm'
    static_configs:
      - targets: ['litellm:4000']
  • Create .env with your environment variables. Replace the placeholder with your actual Anthropic API key:
ANTHROPIC_API_KEY=your_anthropic_api_key_here
LITELLM_MASTER_KEY=sk-master-key-change-me
DATABASE_URL=postgresql://litellm:litellm@postgres:5432/litellm
REDIS_URL=redis://redis:6379

Why a .env file?

Docker Compose reads .env automatically and injects its values into container environments via the ${VARIABLE} syntax in docker-compose.yml. This keeps secrets out of committed code. Never commit your real API key.

  • Save all four files. Verify them by running:
ls *.yml *.yaml .env

You should see docker-compose.yml, litellm_config.yaml, prometheus.yml, and .env listed.

Missing files in the listing?

Make sure you saved each file at the project root (inside clinical-ai-gateway), not inside the docs folder. Run pwd to confirm your terminal is in the right directory.

help me find where my config files ended up

Bring the stack up and verify

All configuration is in place. Time to bring up all eight services and prove the gateway contract works.

  • Start all containers by running:
docker compose up -d

What does this command do?

Docker Compose reads docker-compose.yml, pulls all container images, creates the network, and starts the eight services in detached mode. The first run takes a few minutes while images download.

  • Wait for the images to download and containers to start. Check that all services are healthy by running:
docker compose ps

You should see eight containers listed, all with a status of Up.

Containers exiting or restarting?

If litellm shows as restarting, it may have started before postgres was ready. Run docker compose logs litellm to check. Usually waiting 30 seconds and running docker compose ps again resolves it.

If you see a port conflict error, another service on your machine is using that port. Stop the conflicting service or change the host port in docker-compose.yml.

help me debug why my containers aren't all running

  • Pull the local model into Ollama by running:
docker compose exec ollama ollama pull llama3.2:3b

How long does this take?

The llama3.2:3b model is approximately 2GB. Download time depends on your connection. You will see a progress bar in the terminal. This model is the guaranteed offline floor that ensures the clinic always gets an answer.

  • Confirm the model is available by running:
curl http://localhost:11434/api/tags

You should see a JSON response with a models array containing an entry with llama3.2:3b in the name.

Now for the critical test. Before you run the next command, predict: do you think the gateway will respond successfully, or will it fail?

  • Send a test completion request through the gateway by running:
curl http://localhost:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-master-key-change-me" \
  -d '{
    "model": "claude-primary",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}]
  }'

You should see a JSON response with a choices array containing a completion from Claude. This proves the OpenAI-compatible contract works end-to-end: one endpoint, model selected by config, response in the standard format.

Getting a 401 or connection refused?

A 401 means the Authorization header value does not match LITELLM_MASTER_KEY in your .env file. Confirm the Bearer token matches exactly.

Connection refused means LiteLLM is not ready yet. Wait 30 seconds after docker compose up -d and try again.

If you see an Anthropic authentication error, open .env and replace the placeholder with your real Anthropic API key. Then restart with docker compose up -d.

help me debug why my gateway test request failed

  • Commit the infrastructure code and push the feature branch:
git add .
git commit -m "feat: gateway stack with 8 services"
git push -u origin feat/gateway-stack
  • In GitHub, open a pull request from feat/gateway-stack to main.
  • In the PR description, reference your Linear ticket (e.g., "Closes GAT-1").
  • Merge the pull request.
  • In Linear, move the Gateway Stack ticket to Done.

Your gateway stack is running. Eight services, one endpoint, vendor-agnostic by contract. Next up, you will prove this contract survives a connectivity cut.

Prove the Contract and the Offline Floor

Your gateway stack is running, but a running gateway is not a proven gateway. The pilot this system replaces stalled because clinics went dark during an outbreak. The cloud model was the only option, and when connectivity dropped, the clinic got nothing.

This step proves two things: the contract works (one endpoint, model choice in config) and the offline floor holds (cut the network, clinic still gets an answer). You will send the same clinical question three times and watch the gateway route it differently each time, all without changing a single line of application code.

In this step, get ready to:
  • Send a clinical triage question and confirm the cloud model answers via response headers.
  • Simulate a connectivity cut and prove the local model serves the same question automatically.
  • Restore connectivity and confirm the gateway routes back to the cloud model with no code changes.
Send a triage question through the gateway

The gateway exposes a single POST /chat/completions endpoint. Every clinical app calls this one URL. The model that answers is decided by config, not by the app.

  • Create a feature branch for this validation work by running this command:
git checkout -b feat/offline-floor
  • Send a clinical triage question through the gateway and inspect the response headers by running this command:
curl -s -D - http://localhost:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-master-key-change-me" \
  -d '{"model": "claude-primary", "messages": [{"role": "user", "content": "A patient presents with fever and dehydration in a remote clinic. What are the immediate triage priorities?"}]}'

What does this command do?

  • -s -D - tells curl to suppress the progress bar but dump all response headers to stdout before the body.
  • The Authorization header carries the master key you set in litellm_config.yaml.
  • The model field targets the claude-primary model group. LiteLLM resolves it to the Anthropic Claude deployment defined in your config.

Look at the response headers in your terminal output. You should see several headers prefixed with x-litellm-.

  • Find the x-litellm-model-api-base header. It should show the Anthropic API URL, confirming the cloud model served this request.
  • Find the x-litellm-model-id header. Note this value down. It identifies the specific deployment that handled your request.

The response body contains a standard OpenAI-compatible completion with triage guidance. This proves the contract: one endpoint, model choice in config, clinical app unchanged.

Not getting a response?

  • Confirm your ANTHROPIC_API_KEY in .env is valid and has credits available.
  • Check that the litellm container is running with docker compose ps.
  • If you see a timeout, increase request_timeout in litellm_config.yaml and restart litellm.

Help me debug my LiteLLM gateway request

Simulate a connectivity cut

The pilot died when the network went down. This is the test that matters: can the gateway still serve a triage question when the cloud model is unreachable?

You will simulate a connectivity cut by temporarily pointing the cloud model at an unreachable endpoint. This mimics what happens when a clinic loses internet access.

  • In Cursor, open litellm_config.yaml.
  • Find the claude-primary entry under model_list. Find this line:
  - model_name: claude-primary
    litellm_params:
      model: anthropic/claude-3-5-haiku-20241022
      api_key: os.environ/ANTHROPIC_API_KEY
  • Add an api_base line pointing to an unreachable address, so the entry looks like this:
  - model_name: claude-primary
    litellm_params:
      model: anthropic/claude-3-5-haiku-20241022
      api_key: os.environ/ANTHROPIC_API_KEY
      api_base: http://unreachable:9999

What does this simulate?

Setting api_base to an unreachable address forces every request to the claude-primary model group to fail with a connection error. This is functionally identical to what happens when a clinic's internet goes down. LiteLLM's fallback chain should kick in and route the request to ollama-local instead.

  • Save litellm_config.yaml.
  • Restart the litellm container to pick up the config change by running:
docker compose restart litellm
  • Wait about 10 seconds for the container to fully restart.

Before you send the next request, think: with the cloud model unreachable, do you expect this request to succeed or fail?

  • Send the same triage question again by running:
curl -s -D - http://localhost:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-master-key-change-me" \
  -d '{"model": "claude-primary", "messages": [{"role": "user", "content": "A patient presents with fever and dehydration in a remote clinic. What are the immediate triage priorities?"}]}'

The request succeeds. Look at the response headers now.

  • Find x-litellm-model-api-base. It should show http://ollama:11434 instead of the Anthropic URL.
  • Find x-litellm-model-id. Compare it to the value you noted earlier. It is a different deployment ID, confirming the gateway routed to Ollama.

This is the reach bar: the cloud model is gone, and the clinic still got a triage answer from the local model. No code change, no redeployment. The fallback happened automatically.

Why does the response take longer?

The gateway first attempts the cloud model, hits the configured request_timeout (5 seconds) and num_retries (2 retries), then falls back to Ollama. Plus the local llama3.2:3b model runs on CPU, so inference takes a few seconds. In production you would tune these timeouts down for faster failover.

Still seeing an error instead of a response?

  • Run docker compose ps and confirm both litellm and ollama show healthy status.
  • Verify llama3.2:3b is pulled by running docker compose exec ollama ollama list and checking it appears in the output.
  • Check litellm logs with docker compose logs litellm --tail 30 and look for fallback routing messages.

Help me debug why my LiteLLM fallback to Ollama is not working

Restore connectivity and confirm the route-back

The offline floor is proven. Now restore the cloud model and confirm the gateway automatically routes back, with no intervention from the clinical app.

  • In Cursor, open litellm_config.yaml.
  • Remove the api_base: http://unreachable:9999 line you added to the claude-primary entry.
  • Save litellm_config.yaml.
  • Restart litellm to pick up the restored config by running:
docker compose restart litellm
  • Wait about 10 seconds for the container to restart fully.

Before you send this final request, which model do you expect to answer now that connectivity is restored?

  • Send the same triage question one more time:
curl -s -D - http://localhost:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-master-key-change-me" \
  -d '{"model": "claude-primary", "messages": [{"role": "user", "content": "A patient presents with fever and dehydration in a remote clinic. What are the immediate triage priorities?"}]}'
  • Check x-litellm-model-api-base in the response headers. It should show the Anthropic API URL again, confirming the gateway routed back to Claude automatically.

Same endpoint, same request, same app code. The model changed twice and the clinical app never knew. That is the contract: vendor-agnostic routing, proven.

What did this prove?

  • The contract: one OpenAI-compatible endpoint serves any clinical app. Model choice lives in config.
  • The offline floor: when the cloud is unreachable, the local model answers automatically. No app-side code change required.
  • The route-back: when connectivity returns, the gateway routes back to the stronger model. The swap is seamless and config-driven.
  • Commit the validation record and push by running:
git add -A
git commit -m "validate: offline floor proven, contract confirmed"
git push -u origin feat/offline-floor
  • Open your clinical-ai-gateway repository on GitHub.
  • Create a Pull Request from feat/offline-floor to main.
  • Merge the PR.
  • In Linear, close the ticket for the offline floor validation.

Git push rejected?

  • Make sure you're on the feat/offline-floor branch by running git branch.
  • If the push is rejected due to authentication, confirm your GitHub credentials are configured in Cursor's terminal.

Help me push my feature branch to GitHub

The gateway contract holds and the offline floor is proven. Next, you will add the controls that make this system defensible to a ministry: PHI redaction before egress and a hard block on cross-border routing.

Implement Sovereignty and Clinical Safety

The offline floor is proven. The gateway serves a clinic when connectivity drops. But the original pilot had a second fatal flaw: patient data crossed borders unchecked.

A ministry of health cannot deploy an AI system that sends patient identifiers to servers in another jurisdiction. This step adds the two controls that make the gateway defensible: PHI redaction before any request leaves the local network, and a region routing assertion that blocks cross-border requests outright. You will also add a clinical safety guardrail so the local model never overreaches its scope.

In this step, get ready to:
  • Write a sovereignty middleware that redacts PHI using Presidio before egress.
  • Implement a region routing assertion that blocks cross-border requests with a 403.
  • Add a clinical safety guardrail to the local model via system prompt injection.
Write the PHI redaction middleware

The Presidio containers are already running, but nothing connects them to the request pipeline yet. You will write a Python module that intercepts outbound messages, detects personal health identifiers, and replaces them with redaction tags before any text leaves your network.

  • Create the feature branch by running this command:
git checkout -b feat/sovereignty
  • Before writing the middleware, confirm that the Presidio analyzer is responding. Test it directly by running:
curl -X POST http://localhost:5002/analyze \
  -H "Content-Type: application/json" \
  -d '{"text": "Patient John Smith, ID PAT-123456, phone 555-0123", "language": "en"}'

You should see a JSON array listing detected entities with their positions, scores, and types like PERSON and PHONE_NUMBER.

What does this response tell you?

Presidio returns the start and end character positions of each detected entity, a confidence score, and the entity type. Your middleware will feed these results into the anonymizer service to replace them with redaction tags.

Getting an empty array or connection refused?

Make sure all containers are running with docker compose ps. The presidio-analyzer container must show a healthy status. If the container restarted, give it 10-15 seconds to load its NLP model before retrying.

Help me debug Presidio connectivity

  • Create a new file called sovereignty.py in the root of your clinical-ai-gateway project folder. Start with the imports and configuration constants by pasting this code:
import re
import requests
from fastapi import Request, Response
from fastapi.responses import JSONResponse

PRESIDIO_ANALYZER_URL = 'http://localhost:5002'
PRESIDIO_ANONYMIZER_URL = 'http://localhost:5001'

PATIENT_ID_PATTERN = r'PAT-[0-9]{6}'

REGION_MODEL_MAP = {
    'claude-primary': 'us',
    'ollama-local': 'local'
}

What does this code set up?

  • PRESIDIO_ANALYZER_URL and PRESIDIO_ANONYMIZER_URL point at the two Presidio containers running on their default ports.
  • PATIENT_ID_PATTERN is a custom regex for health-system patient IDs in the format PAT-XXXXXX. Presidio does not know this pattern natively, so you handle it with regex before calling Presidio for standard PII.
  • REGION_MODEL_MAP declares where each model deployment lives. The local Ollama model is always safe (data never leaves the machine). The cloud model is in the US region.
  • Below the constants, add the redact_phi function that handles the actual redaction logic:
def redact_phi(text: str) -> str:
    # Custom patient ID redaction
    text = re.sub(PATIENT_ID_PATTERN, '[REDACTED-PATIENT-ID]', text)
    # Presidio redaction
    analyze_resp = requests.post(
        f'{PRESIDIO_ANALYZER_URL}/analyze',
        json={'text': text, 'language': 'en', 'entities': ['PERSON', 'PHONE_NUMBER', 'EMAIL_ADDRESS']}
    )
    results = analyze_resp.json()
    if results:
        anonymize_resp = requests.post(
            f'{PRESIDIO_ANONYMIZER_URL}/anonymize',
            json={'text': text, 'analyzer_results': results}
        )
        return anonymize_resp.json().get('text', text)
    return text

What does redact_phi do?

  • First, it strips custom patient IDs (PAT-123456) with a regex substitution. This catches health-system identifiers that Presidio would not recognize.
  • Then it calls the Presidio analyzer to detect standard PII entities: names, phone numbers, and emails.
  • If entities are found, it passes them to the anonymizer which returns the text with redaction tags in place of the sensitive spans.
  • Update docker-compose.yml to mount sovereignty.py into the LiteLLM container and declare the Presidio dependency.
  • In the litellm service, find the volumes section. It currently looks like this:
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
  • Add the sovereignty mount so it becomes:
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
      - ./sovereignty.py:/app/sovereignty.py
  • In the same litellm service, find the depends_on section. It currently lists:
    depends_on:
      - redis
      - postgres
  • Add the Presidio services so it becomes:
    depends_on:
      - redis
      - postgres
      - presidio-analyzer
      - presidio-anonymizer
  • Save sovereignty.py and docker-compose.yml.
  • Restart the stack to pick up the new volume mount by running:
docker compose up -d
  • Now test the full redaction pipeline. Run this Python snippet from your project directory to call redact_phi directly:
python3 -c "
import sys
sys.path.insert(0, '.')
from sovereignty import redact_phi
result = redact_phi('Patient John Smith, ID PAT-123456, phone 555-0123')
print(result)
"

You should see the patient ID replaced with [REDACTED-PATIENT-ID] and the person name replaced with a Presidio redaction tag. The phone number should also be anonymized.

Seeing a connection error to localhost:5002?

The sovereignty.py module uses localhost URLs because you are running the test from your host machine where Docker ports are forwarded. Confirm ports 5002 and 5001 are mapped with docker compose ps.

Help me fix Presidio connectivity from my host machine

Implement the region routing assertion

PHI redaction prevents identifiers from leaking. But what if a request is routed to a model hosted in a different jurisdiction entirely? The sovereignty bar demands a hard block: if the data region does not match the model region, the request is rejected.

  • Open sovereignty.py and add the following two functions below redact_phi:
def check_region(request_region: str, model_name: str) -> bool:
    model_region = REGION_MODEL_MAP.get(model_name, 'unknown')
    if model_region == 'local':
        return True
    return request_region == model_region

def sovereignty_middleware(request_body: dict, model_name: str, data_region: str) -> dict:
    # Check cross-border routing
    if not check_region(data_region, model_name):
        raise ValueError(f'SOVEREIGNTY_VIOLATION: Request region {data_region} cannot route to model region {REGION_MODEL_MAP.get(model_name)}')
    # Redact PHI in messages
    if 'messages' in request_body:
        for msg in request_body['messages']:
            if 'content' in msg and isinstance(msg['content'], str):
                msg['content'] = redact_phi(msg['content'])
    return request_body

What do these functions do?

  • check_region compares the request's declared data region against the model's deployment region. Local models always pass because data never leaves the machine.
  • sovereignty_middleware is the orchestrator. It first checks the region assertion. If the regions do not match, it raises a ValueError with a sovereignty violation message. If the check passes, it redacts PHI from every message before forwarding.
  • Save sovereignty.py.

Before you run the next test, predict: a request with data region eu targeting the claude-primary model (deployed in us). Will it pass or be blocked?

  • Test the cross-border block by running this Python snippet:
python3 -c "
import sys
sys.path.insert(0, '.')
from sovereignty import sovereignty_middleware
try:
    sovereignty_middleware(
        {'messages': [{'role': 'user', 'content': 'Hello'}]},
        'claude-primary',
        'eu'
    )
except ValueError as e:
    print(e)
"

You should see: SOVEREIGNTY_VIOLATION: Request region eu cannot route to model region us. The EU request tried to route to a US-hosted model and was blocked. This is the sovereignty bar: hard zero cross-border.

Why does the local model always pass?

When a request targets ollama-local, the region map returns local. The check_region function treats local as always valid because data on the local model never leaves the machine, regardless of which jurisdiction the request originates from.

Add the clinical safety guardrail

The local model has bounded capability. It must never provide a definitive diagnosis or prescribe treatment. A system prompt guardrail instructs the model to flag uncertainty and escalate to a clinician whenever it encounters clinical-risk patterns.

  • Open litellm_config.yaml and find the ollama-local entry. It currently looks like this:
  - model_name: ollama-local
    litellm_params:
      model: ollama/llama3.2:3b
      api_base: http://ollama:11434
  • Add the system_prompt field below api_base so the section now looks like this:
  - model_name: ollama-local
    litellm_params:
      model: ollama/llama3.2:3b
      api_base: http://ollama:11434
      system_prompt: |
        CLINICAL SAFETY NOTICE: You are a bounded AI support tool, not a diagnostic device.
        When responding to any question involving diagnosis, prescribing, or emergency situations:
        1. FLAG your uncertainty explicitly: 'I am not certain and this requires clinical judgment.'
        2. Provide only general, bounded informational support.
        3. ALWAYS direct the user to a qualified clinician or, in emergencies, to the offline emergency protocol.
        You must never provide a definitive diagnosis or prescribe treatment.

How does this guardrail work?

LiteLLM prepends the system_prompt to every request sent to the ollama-local model. When the prompt contains clinical-risk language (diagnosis, prescribe, emergency), the model's response is steered toward flagging uncertainty and directing users to a clinician.

This is not a certified medical device control. It is deployment infrastructure that reduces overreach risk.

  • Save litellm_config.yaml.
  • Restart LiteLLM to pick up the config change by running:
docker compose restart litellm

Before you test, predict: when you send a clinical-risk question to the local model, what language do you expect in the response?

  • Send a clinical-risk prompt through the gateway targeting the local model by running:
curl http://localhost:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-master-key-change-me" \
  -d '{"model": "ollama-local", "messages": [{"role": "user", "content": "Can you diagnose my chest pain and prescribe medication?"}]}'

The response should include language about uncertainty, bounded support, and a directive to consult a clinician. Look for phrases like "I am not certain" or "requires clinical judgment" and a reference to the offline emergency protocol.

Not seeing the safety language in the response?

Make sure you restarted the litellm container after editing the config. Run docker compose restart litellm and wait 10 seconds for it to reload. Also confirm you are targeting ollama-local in the request body, not claude-primary.

Help me verify my system_prompt is being applied

  • Commit all changes, push the branch, and open a PR by running:
git add sovereignty.py docker-compose.yml litellm_config.yaml
git commit -m "feat: add sovereignty middleware, region routing, and clinical safety guardrail"
git push origin feat/sovereignty
  • Open the PR on GitHub, merge it to main, and close the corresponding Linear ticket.

What did this step prove?

Three controls now compose into the sovereignty layer. PHI is redacted before any request can leave the local network. A hard region assertion blocks cross-border routing. The local model flags clinical uncertainty and escalates to a clinician. Each control is independently testable and the middleware is config-driven.

✔️ Awesome, I've got everything!

Great. Double check you have saved sovereignty.py, docker-compose.yml, and litellm_config.yaml.

ⓧ I'd like to double check the full code

The complete sovereignty.py exceeds 30 lines, so scroll through and confirm each section is present in order:

  • Imports and constants (PRESIDIO_ANALYZER_URL, PRESIDIO_ANONYMIZER_URL, PATIENT_ID_PATTERN, REGION_MODEL_MAP).
  • The redact_phi function (regex sub, then Presidio analyze + anonymize call).
  • The check_region function (region comparison with local bypass).
  • The sovereignty_middleware function (region check, then PHI redaction on messages).

For litellm_config.yaml, confirm that the ollama-local entry now has a system_prompt block starting with CLINICAL SAFETY NOTICE.

For docker-compose.yml, confirm the litellm service has ./sovereignty.py:/app/sovereignty.py in volumes and presidio-analyzer plus presidio-anonymizer in depends_on.

The sovereignty layer is locked in. Patient data cannot leak across borders, identifiers are stripped before egress, and the local model stays within its clinical scope. Next up: you will configure health-aware failover to survive a vendor outage, add multilingual access, and set up per-capita cost governance so a funder can see exactly what each patient served costs.

Add Resilience, Language Routing, and Per-Capita Cost

Your gateway can now redact patient data and block cross-border routes. But the original pilot failed for a different reason: during a cholera surge, the vendor throttled requests and the entire system went dark.

This step adds three capabilities that prevent that failure mode. Health-aware failover keeps triage alive when a vendor goes down. Language-aware routing answers patients in their own language. Per-capita budget metering lets a funder see cost per patient served and enforces a cap.

In this step, get ready to:
  • Prove the resilience bar by simulating a vendor outage and observing automatic failover.
  • Configure language-aware routing so a French clinical question is answered in French.
  • Set up per-capita cost governance with team budgets and budget fallbacks.
Prove the resilience bar

The failover chain from claude-primary to ollama-local is already configured in your litellm_config.yaml with retry and cooldown settings. Now you need to prove it actually works under a real vendor outage.

  • Create a new feature branch by running this command:
git checkout -b feat/resilience-cost

To simulate a vendor outage, you will temporarily point the cloud model at an unreachable endpoint. This forces LiteLLM to exhaust its retries and trigger the fallback chain.

  • In litellm_config.yaml, find the claude-primary model entry.
  • Find this line:
      model: anthropic/claude-3-5-haiku-20241022
  • Add an api_base line directly below it pointing to an unreachable endpoint, so the entry now looks like this:
      model: anthropic/claude-3-5-haiku-20241022
      api_base: http://unreachable:9999
  • Save litellm_config.yaml.
  • Restart the LiteLLM container to pick up the config change by running:
docker compose restart litellm

Before you run the next command, what do you think will happen when you send a triage question to claude-primary now that its endpoint is unreachable?

  • Send a clinical triage question through the gateway by running:
curl -s http://localhost:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-master-key-change-me" \
  -d '{
    "model": "claude-primary",
    "messages": [{"role": "user", "content": "A patient presents with acute dehydration and fever. What is the triage priority?"}]
  }'

You still get a completion back. The gateway exhausted its retries against the unreachable endpoint, then automatically fell over to the local Ollama model. This is the resilience bar: triage stays alive during a vendor outage.

How does LiteLLM know to failover?

The router_settings in your config set num_retries: 2 and allowed_fails: 2. After two failed attempts, the router marks claude-primary as unhealthy and enters a cooldown_time: 30 second cooldown.

During cooldown, all requests route to the next model in the fallbacks list: ollama-local. This happens transparently to the calling application.

Now restore the cloud model by removing the unreachable endpoint.

  • In litellm_config.yaml, find this line you just added:
      api_base: http://unreachable:9999
  • Delete that line entirely so the claude-primary entry is back to its original state.
  • Save litellm_config.yaml and restart LiteLLM:
docker compose restart litellm
  • Send the same triage question again and confirm the response now comes from Claude (the response will be more detailed than the local model's answer).

Still getting responses from the local model?

The cooldown period may not have expired yet. Wait 30 seconds after restarting and try again. Also confirm you fully removed the api_base: http://unreachable:9999 line and saved the file.

help me debug why my LiteLLM gateway is still routing to the fallback model after I restored the config

Configure language-aware routing

The pilot also excluded patients who did not speak the dominant language. A gateway that only answers in English fails the access bar. You will add a multilingual model entry that responds in the same language as the user's message.

  • In litellm_config.yaml, find the model_list section. Add the following entry between claude-primary and ollama-local:
  - model_name: claude-multilingual
    litellm_params:
      model: anthropic/claude-3-5-haiku-20241022
      api_key: os.environ/ANTHROPIC_API_KEY
      system_prompt: 'Always respond in the same language as the user\'s message.'

What does this model entry do?

This creates a separate model alias claude-multilingual that uses the same Claude model but injects a system prompt directing responses in the user's language. Clinical apps targeting non-English populations route through this alias.

  • In the same file, find the fallbacks section under router_settings. Add a fallback entry for the new multilingual model so it looks like this:
  fallbacks:
    - claude-primary:
        - ollama-local
    - claude-multilingual:
        - ollama-local
  • Save litellm_config.yaml.
  • Now update the sovereignty middleware so it knows the new model's region. In sovereignty.py, find the REGION_MODEL_MAP dictionary:
REGION_MODEL_MAP = {
    'claude-primary': 'us',
    'ollama-local': 'local'
}
  • Add 'claude-multilingual': 'us' so the dictionary now looks like this:
REGION_MODEL_MAP = {
    'claude-primary': 'us',
    'claude-multilingual': 'us',
    'ollama-local': 'local'
}
  • Save sovereignty.py and restart LiteLLM:
docker compose restart litellm
  • Send a clinical question in French through the multilingual model by running:
curl -s http://localhost:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-master-key-change-me" \
  -d '{
    "model": "claude-multilingual",
    "messages": [{"role": "user", "content": "Un patient presente une deshydratation aigue. Quelle est la priorite de triage?"}]
  }'

The response comes back in French. This is the access bar: patients are answered in their language through the same gateway endpoint.

Response in English instead of French?

Confirm your request uses "model": "claude-multilingual" (not claude-primary). The system prompt directive is only attached to the multilingual alias. Also confirm you restarted the LiteLLM container after editing the config.

help me debug why my LiteLLM multilingual model is responding in English instead of French

Set up per-capita cost governance

A funder measures cost per patient served. LiteLLM models this as a team with a max budget. When the budget is exhausted, requests fall back to the zero-marginal-cost local model. This is cost governance the way a ministry can defend.

  • First, add budget_fallbacks to the config. In litellm_config.yaml, find the router_settings section. After the cooldown_time: 30 line, add:
  budget_fallbacks:
    - claude-primary:
        - ollama-local

What are budget fallbacks?

When a key's budget is exhausted for a given model, budget_fallbacks reroutes subsequent requests to the fallback model instead of returning a budget-exceeded error. This means the clinic still gets an answer from the local model at zero marginal cost.

  • Save litellm_config.yaml and restart LiteLLM:
docker compose restart litellm
  • Create a team representing a health program by running:
curl -s http://localhost:4000/team/new \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-master-key-change-me" \
  -d '{
    "team_alias": "moh-program-alpha",
    "max_budget": 0.50
  }'

The response includes a team_id value. Copy it for the next command.

What does max_budget mean here?

The max_budget: 0.50 sets a $0.50 spending cap for this health program. In a real deployment, this number comes from dividing the program's total AI budget by the patient population served. That ratio is the per-capita cost a funder sees on the dashboard.

  • Generate a virtual key scoped to that team (replace the team_id with the value from the previous response):
curl -s http://localhost:4000/key/generate \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-master-key-change-me" \
  -d '{"team_id": "[[TEAM_ID="your team_id from the previous response"]]"}'

The response includes a key value. This is the virtual API key that any clinical app for this program would use.

  • Send a request using the team key to test spend tracking (replace the key value with the one you just generated):
curl -si http://localhost:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer [[TEAM_KEY="your generated team key"]]" \
  -d '{
    "model": "claude-primary",
    "messages": [{"role": "user", "content": "What are the signs of cholera in a child under 5?"}]
  }'

Look at the response headers. You should see x-litellm-response-cost with a dollar value. This is the per-request cost tracked against the team's budget.

  • Verify spend is tracked by querying the spend logs endpoint:
curl -s http://localhost:4000/spend/logs \
  -H "Authorization: Bearer sk-master-key-change-me"

The response shows spend entries attributed to your team key. Cost is visible, tracked, and capped. When spend exceeds the team budget, budget_fallbacks routes requests to ollama-local at zero marginal cost.

Spend logs returning empty?

Spend logging is a batch process. Wait 10-15 seconds after your request and query again. Also confirm your PostgreSQL container is healthy by checking docker compose ps. The database_url in general_settings must point to the running Postgres instance.

help me debug why LiteLLM spend logs are returning empty results

  • Commit all changes and push the branch:
git add -A
git commit -m "feat: add resilience failover, multilingual routing, per-capita cost governance"
git push origin feat/resilience-cost
  • Open a PR from feat/resilience-cost to main on GitHub and merge it.
  • Close the corresponding ticket in your Linear project.

Before you run the final check, predict: if you now send a request with the team key to claude-primary, what two things should you see that prove cost governance is working?

  • Verify the full system by sending one more request with your team key and confirming both the x-litellm-response-cost header is present and the /spend/logs endpoint shows cumulative spend for the team.

You should see cost in the response header and cumulative spend in the logs. The budget cap, fallback chain, and per-request cost tracking compose into per-capita governance.

✔️ Awesome, I've got everything!

Great. Confirm you have saved both litellm_config.yaml and sovereignty.py and merged your PR.

ⓧ I'd like to double check the full code

Your litellm_config.yaml and sovereignty.py each exceed 30 lines. Scroll up and confirm the following pieces are present in order:

  • litellm_config.yaml: three model entries (claude-primary, claude-multilingual, ollama-local) in model_list.
  • litellm_config.yaml: router_settings with fallbacks for both claude-primary and claude-multilingual, plus budget_fallbacks routing claude-primary to ollama-local.
  • sovereignty.py: REGION_MODEL_MAP includes 'claude-multilingual': 'us'.

Three bars now pass. The gateway survives a vendor outage, answers patients in their language, and tracks cost per patient with a budget cap. Next up, you will wire these metrics into a Grafana dashboard and run all five validation bars as a single scripted drill.

Build the Funder Dashboard and Run Validation Drills

Your gateway routes, redacts, fails over, and meters cost. But the funder in the steering meeting cannot SSH into your machine and check response headers.

The gateway must show a funder that it works, not just tell them. This step wires Prometheus metrics into a Grafana dashboard showing cost per patient served, runs all five validation bars as a scripted drill, and commits production-ready deployment configs so the gateway is portable beyond Docker Compose.

In this step, get ready to:
  • Wire Grafana to Prometheus and build the funder dashboard with four panels.
  • Write and run a five-bar validation script that proves the gateway end-to-end.
  • Commit Kubernetes, Terraform, and an ops guide as production-readiness artifacts.
Configure the Grafana datasource and build the funder dashboard

Grafana is running but has no idea where to pull metrics from. You need to point it at the Prometheus instance scraping LiteLLM, then create the panels a funder would see.

  • Create and switch to a new feature branch by running this command:
git checkout -b feat/dashboard-drills
  • Open your browser and navigate to http://localhost:3000.
  • Log in with username admin and password admin.
  • If Grafana prompts you to change the password, skip or set a new one.

Now add the Prometheus datasource so Grafana can query LiteLLM metrics.

  • In the left sidebar, click the gear icon (Configuration), then click Data sources.
  • Click Add data source.
  • Select Prometheus from the list.
  • In the URL field, enter http://prometheus:9090.
  • Scroll down and click Save & test.

You should see a green banner confirming the datasource is working.

Seeing a connection error?

Make sure you typed http://prometheus:9090 (the Docker service name), not localhost. Grafana runs inside Docker and reaches Prometheus via the internal network.

Confirm Prometheus is healthy by running docker compose ps prometheus in your terminal.

Help me debug this

Now create the funder dashboard with four panels.

  • Click the + icon in the left sidebar, then click Dashboard.
  • Click Add new panel.
  • In the query editor at the bottom, enter the metric litellm_remaining_team_budget_metric.
  • In the panel title field (top right), type Per-Capita Budget Remaining.
  • Change the visualization type to Gauge using the dropdown in the top-right corner.
  • Click Apply in the top-right corner to save this panel.

What is this metric?

The litellm_remaining_team_budget_metric tracks the remaining budget for each LiteLLM team. For the funder, this is the per-capita cost view: divide the team's max budget by the patient population served and you have cost per patient.

  • Click Add panel again (top-right icon bar).
  • Set the query to litellm_proxy_total_requests_metric.
  • Title this panel Total Requests by Model.
  • Leave the visualization as Time series (the default).
  • Click Apply.
  • Add a third panel with the query litellm_provider_remaining_budget_metric.
  • Title it Spend per Provider.
  • Click Apply.
  • Add a fourth panel and change the visualization type to Text.
  • In the content area, type: Coalition Validation Status: Run validate.py for PASS/FAIL results.
  • Title it Validation Status.
  • Click Apply.
  • Click the save icon (floppy disk) at the top of the dashboard.
  • Name the dashboard Clinical AI Gateway and click Save.

You should now see all four panels on the dashboard. The per-capita gauge shows the remaining budget for moh-program-alpha (the team you created in the previous step).

Why a dashboard for a funder?

A ministry of health cannot evaluate AI infrastructure by reading logs. The dashboard translates system metrics into a question a funder can answer at a glance: how much does this cost per patient, and is it working?

Write the five-bar validation script

The dashboard shows ongoing state, but the steering meeting needs a binary answer: does the gateway pass every bar or not? A scripted drill runs all five tests sequentially and prints PASS or FAIL for each.

  • Create a new file called validate.py in the root of your clinical-ai-gateway folder.
  • Add the script setup and the first validation bar (offline clinic drill) by pasting this code:
#!/usr/bin/env python3
import requests
import json
import sys

GATEWAY_URL = 'http://localhost:4000'
MASTER_KEY = 'sk-master-key-change-me'
TEAM_KEY = 'sk-team-key-moh-program-alpha'

HEADERS = {
    'Authorization': f'Bearer {MASTER_KEY}',
    'Content-Type': 'application/json'
}

results = {}

# Bar 1: Offline clinic drill
print('Running Bar 1: Offline Clinic Drill...')
try:
    resp = requests.post(
        f'{GATEWAY_URL}/chat/completions',
        headers={**HEADERS, 'x-litellm-model': 'ollama-local'},
        json={'model': 'ollama-local', 'messages': [{'role': 'user', 'content': 'A patient has a fever of 39C. What are the triage priorities?'}]},
        timeout=30
    )
    assert resp.status_code == 200
    assert 'ollama' in resp.headers.get('x-litellm-model-id', '').lower() or resp.json().get('model', '').startswith('ollama')
    results['offline_clinic'] = 'PASS'
except Exception as e:
    results['offline_clinic'] = f'FAIL: {e}'

What does this code do?

  • The constants define your gateway URL and authentication key. The results dict collects PASS/FAIL for each bar.
  • Bar 1 sends a triage question directly to ollama-local and asserts the response came from Ollama. This proves the offline floor works.
  • Save validate.py and verify the first bar works by running:
python validate.py

You should see Running Bar 1: Offline Clinic Drill... printed. The script exits silently after Bar 1 because no results summary exists yet.

Seeing a connection error or timeout?

Confirm all containers are running with docker compose ps. If Ollama shows unhealthy, the model may not be pulled. Re-run docker compose exec ollama ollama pull llama3.2:3b.

Help me debug this connection issue

  • Add the surge failover drill (Bar 2) below the Bar 1 code in validate.py:
# Bar 2: Surge failover drill
print('Running Bar 2: Surge Failover Drill...')
try:
    resp = requests.post(
        f'{GATEWAY_URL}/chat/completions',
        headers=HEADERS,
        json={'model': 'claude-primary', 'messages': [{'role': 'user', 'content': 'Triage question: patient with chest pain.'}]},
        timeout=30
    )
    assert resp.status_code == 200
    results['surge_failover'] = 'PASS'
except Exception as e:
    results['surge_failover'] = f'FAIL: {e}'

What does Bar 2 test?

Bar 2 sends a request to claude-primary and asserts a 200 response. With your failover config, if the cloud model is unreachable, LiteLLM automatically routes to the local model. The assertion proves triage stays alive either way.

  • Add the sovereignty audit (Bar 3) below Bar 2:
# Bar 3: Sovereignty audit
print('Running Bar 3: Sovereignty Audit...')
try:
    # Cross-border block
    resp_403 = requests.post(
        f'{GATEWAY_URL}/chat/completions',
        headers={**HEADERS, 'X-Data-Region': 'eu'},
        json={'model': 'claude-primary', 'messages': [{'role': 'user', 'content': 'Patient query.'}]},
        timeout=10
    )
    assert resp_403.status_code == 403
    # PHI redaction
    phi_resp = requests.post(
        'http://localhost:5002/analyze',
        json={'text': 'Patient PAT-123456 John Smith called about chest pain.', 'language': 'en', 'entities': ['PERSON', 'PHONE_NUMBER']},
        timeout=10
    )
    assert phi_resp.status_code == 200
    assert len(phi_resp.json()) > 0
    results['sovereignty'] = 'PASS'
except Exception as e:
    results['sovereignty'] = f'FAIL: {e}'

What does Bar 3 test?

Two assertions in one bar. First, a request tagged with X-Data-Region: eu is blocked because the cloud model is deployed in us. Second, Presidio detects PII in a synthetic patient string. Together these prove sovereignty: hard zero cross-border violations and PHI never leaves unredacted.

  • Save validate.py and run it again to confirm the first three bars execute:
python validate.py

You should see three print statements: Running Bar 1..., Running Bar 2..., and Running Bar 3.... No crash means all assertions passed.

  • Add the cost bar and access bar (Bars 4 and 5) below Bar 3:
# Bar 4: Cost bar
print('Running Bar 4: Cost Bar...')
try:
    spend_resp = requests.get(
        f'{GATEWAY_URL}/spend/logs',
        headers=HEADERS,
        timeout=10
    )
    assert spend_resp.status_code == 200
    results['cost'] = 'PASS'
except Exception as e:
    results['cost'] = f'FAIL: {e}'

# Bar 5: Access bar (multilingual)
print('Running Bar 5: Access Bar (Multilingual)...')
try:
    fr_resp = requests.post(
        f'{GATEWAY_URL}/chat/completions',
        headers=HEADERS,
        json={'model': 'claude-multilingual', 'messages': [{'role': 'user', 'content': 'Un patient a de la fièvre. Quelles sont les priorités de triage?'}]},
        timeout=30
    )
    assert fr_resp.status_code == 200
    content = fr_resp.json()['choices'][0]['message']['content']
    french_words = ['patient', 'fièvre', 'triage', 'priorité', 'clinicien', 'médecin']
    assert any(w in content.lower() for w in french_words)
    results['access'] = 'PASS'
except Exception as e:
    results['access'] = f'FAIL: {e}'

What do Bars 4 and 5 test?

  • Bar 4 queries /spend/logs and asserts a 200 response. This proves cost tracking is accessible for the funder dashboard.
  • Bar 5 sends a French triage question to claude-multilingual and checks that the response contains French words. This is the access bar: patients answered in their language.
  • Finally, add the results summary at the end of validate.py:
print('\n=== VALIDATION RESULTS ===')
all_pass = True
for bar, result in results.items():
    print(f'{bar}: {result}')
    if not result.startswith('PASS'):
        all_pass = False

if all_pass:
    print('\nALL BARS PASS - Gateway validated.')
    sys.exit(0)
else:
    print('\nSOME BARS FAILED - Review output above.')
    sys.exit(1)

What does the summary do?

The loop prints each bar's result, then exits with code 0 (success) if all pass or code 1 (failure) if any fail. A CI pipeline or the learner at the terminal can use the exit code to gate a deployment.

  • Save validate.py.

Before you run the full script, what do you expect to see? Do you think all five bars will pass on the first try?

  • Run the validation drill end-to-end:
python validate.py

You should see all five bars print their status, followed by a structured results table and the message ALL BARS PASS - Gateway validated.

Some bars failing?

If Bar 3 (sovereignty) fails, confirm your sovereignty middleware is still configured. The X-Data-Region: eu header must trigger a 403 for the claude-primary model mapped to us.

If Bar 5 (access) fails, the multilingual model may not have returned French. Increase the timeout value or verify your claude-multilingual entry in litellm_config.yaml has the system prompt directing French responses.

Help me debug my failing validation bars

✔️ Awesome, I've got everything!

Great. Double check you saved validate.py and all five bars printed PASS.

ⓧ I'd like to double check the full code

Your validate.py is longer than 30 lines, so confirm it has these sections in order:

  • Shebang line, imports (requests, json, sys), constants (GATEWAY_URL, MASTER_KEY, TEAM_KEY, HEADERS, results).
  • Bar 1: Offline clinic drill (POST to ollama-local, assert 200).
  • Bar 2: Surge failover drill (POST to claude-primary, assert 200).
  • Bar 3: Sovereignty audit (cross-border 403 + Presidio analyze 200).
  • Bar 4: Cost bar (GET /spend/logs, assert 200).
  • Bar 5: Access bar (French request, assert French response).
  • Results summary: prints each bar, exits 0 if all pass, exits 1 otherwise.
Write deployment configs and commit the ops guide

The gateway runs on Docker Compose locally. For production, the coalition needs portable deployment targets. The Kubernetes manifest and Terraform config prove the gateway can deploy to any cluster or infrastructure-as-code pipeline without rewriting the application.

  • Create the k8s directory by running:
mkdir -p k8s
  • Create k8s/gateway.yaml in Cursor and paste the Deployment resource:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm-gateway
  namespace: clinical-ai
spec:
  replicas: 2
  selector:
    matchLabels:
      app: litellm-gateway
  template:
    metadata:
      labels:
        app: litellm-gateway
    spec:
      containers:
      - name: litellm
        image: ghcr.io/berriai/litellm:main-latest
        ports:
        - containerPort: 4000
        env:
        - name: ANTHROPIC_API_KEY
          valueFrom:
            secretKeyRef:
              name: gateway-secrets
              key: anthropic-api-key
        - name: LITELLM_MASTER_KEY
          valueFrom:
            secretKeyRef:
              name: gateway-secrets
              key: master-key

What does this manifest do?

This defines a Kubernetes Deployment with 2 replicas of the LiteLLM gateway. Secrets are pulled from a Kubernetes Secret object (gateway-secrets) instead of environment variables. The ops team creates that Secret separately when deploying to a real cluster.

  • Add the remaining environment variables, volume mounts, and the Service resource below the existing code in k8s/gateway.yaml:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: gateway-secrets
              key: database-url
        - name: REDIS_URL
          valueFrom:
            secretKeyRef:
              name: gateway-secrets
              key: redis-url
        volumeMounts:
        - name: config
          mountPath: /app/config.yaml
          subPath: litellm_config.yaml
      volumes:
      - name: config
        configMap:
          name: litellm-config
---
apiVersion: v1
kind: Service
metadata:
  name: litellm-gateway
  namespace: clinical-ai
spec:
  selector:
    app: litellm-gateway
  ports:
  - port: 4000
    targetPort: 4000
  type: ClusterIP

What does the Service do?

The Service exposes the gateway inside the cluster on port 4000 as a ClusterIP. Any clinical app in the same namespace can reach it at http://litellm-gateway:4000.

  • Save k8s/gateway.yaml.
  • Confirm the file exists by running:
ls k8s/

You should see gateway.yaml listed.

  • Create the terraform directory by running:
mkdir -p terraform
  • Create terraform/main.tf in Cursor and paste the provider and network config:
terraform {
  required_providers {
    docker = {
      source  = "kreuzwerker/docker"
      version = "~> 3.0"
    }
  }
}

provider "docker" {}

resource "docker_network" "gateway" {
  name = "clinical-ai-gateway"
}

resource "docker_volume" "ollama_data" {
  name = "ollama_data"
}

resource "docker_volume" "postgres_data" {
  name = "postgres_data"
}

resource "docker_volume" "redis_data" {
  name = "redis_data"
}
  • Add the remaining volumes and output below the existing code in terraform/main.tf:
resource "docker_volume" "grafana_data" {
  name = "grafana_data"
}

resource "docker_volume" "prometheus_data" {
  name = "prometheus_data"
}

output "gateway_endpoint" {
  value = "http://localhost:4000"
  description = "LiteLLM gateway OpenAI-compatible endpoint"
}

Why Terraform for a Docker project?

Terraform is included as a production-readiness artifact. It proves the gateway's infrastructure can be declared as code and provisioned reproducibly. The Docker provider mirrors what a team would extend to AWS, GCP, or Azure when moving to production.

  • Save terraform/main.tf.
  • Confirm both deployment directories exist by running:
ls k8s/ terraform/

You should see gateway.yaml in k8s/ and main.tf in terraform/.

  • Create docs/ops-guide.md in Cursor and paste the operations guide:
# Clinical-AI Gateway: Operations Guide

## Stand the Gateway Up

```bash
cp .env.example .env
# Edit .env: add ANTHROPIC_API_KEY, set LITELLM_MASTER_KEY
docker compose up -d
docker compose exec ollama ollama pull llama3.2:3b
curl http://localhost:4000/health
```

## Add a Clinic

1. Create a team: `POST /team/new` with `team_alias` and `max_budget`
2. Generate a virtual key: `POST /key/generate` scoped to the team
3. Give the clinic the virtual key and the gateway endpoint
  • Add the remaining sections below in docs/ops-guide.md:
## Run It Offline

1. Disconnect the machine from the internet
2. Send requests to `http://localhost:4000/chat/completions` with `model: ollama-local`
3. The local llama3.2:3b model responds without any cloud dependency

## Run Validation Drills

```bash
python validate.py
```

All five bars should print PASS.

## Tear Down

```bash
# Stop without deleting data:
docker compose stop

# Delete everything including volumes:
docker compose down -v
```

Who is this ops guide for?

The ops guide is written for a colleague who did not build the gateway. It answers four questions in plain language: how do I start it, how do I add a clinic, how do I run it offline, and how do I tear it down. This is the handover artifact.

  • Save docs/ops-guide.md.
  • Run the validation script one final time to confirm the full system still passes:
python validate.py

All five bars should print PASS and the script exits with ALL BARS PASS - Gateway validated.

  • Stage, commit, and push your feature branch:
git add .
git commit -m "feat: add funder dashboard, validation drills, and deployment configs"
git push origin feat/dashboard-drills
  • Open a Pull Request on GitHub from feat/dashboard-drills into main.
  • Merge the PR and close the corresponding Linear ticket.

Validation bars passing locally but PR checks fail?

The validation script runs against your local Docker stack, not a CI runner. If your PR has CI checks configured, they may not have access to the running containers. For this project, the local PASS is your proof. CI integration is a production-readiness extension.

Help me debug my git workflow

Your gateway is now validated end-to-end with passing drills, a funder-visible dashboard, and production-portable configs. Next up, you have the option to produce the population-impact briefing that gets this gateway funded.

Secret mission

The Population-Impact Briefing

The gateway works. But a working gateway that nobody funds is infrastructure that never deploys. Your mission: produce the artifact that gets it funded. Write the plain-language briefing, render the annotated system diagram, and deliver the two-audience walkthrough for the coalition steering meeting.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs on Docker containers locally, so the only ongoing cost is Anthropic API token usage if you continue sending requests through the cloud model.

Cost warning

If you used pay-per-token API calls during testing, check your Anthropic dashboard. Typical usage for this project is less than $0.10. Leaving the gateway running does not incur charges on its own. Charges only apply when requests route through the cloud model (Anthropic Claude).

Resources you used:

  • Docker containers: litellm, ollama, redis, postgres, prometheus, grafana, presidio-analyzer, presidio-anonymizer.
  • Docker volumes: postgres data, ollama model cache, redis data, grafana data, prometheus data.
  • Docker network: gateway bridge network.
  • GitHub repo: clinical-ai-gateway (private, tagged v1.0.0-gateway-validated).
  • Linear project with closed tickets.
  • Local project folder: clinical-ai-gateway on your Desktop.

Keep everything running

No action needed. Choose this if you are still actively building or want to demo the gateway to others.

  • Your Docker containers remain running and accessible on their configured ports.
  • Requests through the cloud model will continue to use your Anthropic credits.
  • The Ollama local model stays loaded in memory (uses approximately 4-6 GB RAM).
  • Your GitHub repo and Linear project remain intact.

Pause - I'll come back to this later

Shut down running containers to free up memory and stop any potential API usage. Your data volumes and project files are preserved.

  • Stop all containers by running:
docker compose stop
  • This halts containers without deleting volumes or data.
  • To resume later, navigate back to your clinical-ai-gateway folder and run docker compose up -d.

Delete - I don't want to use this again

Remove all project resources and start fresh. This is irreversible for local data.

  • Remove all Docker containers and volumes by running:
docker compose down -v
  • This deletes all eight containers, the gateway bridge network, and all named volumes (postgres data, ollama model cache, redis data, grafana data, prometheus data).
  • Delete the local project folder. On macOS or Linux, run:
cd ~/Desktop
rm -rf clinical-ai-gateway
  • On Windows, run:
cd ~\Desktop
rmdir /s /q clinical-ai-gateway
  • Delete the GitHub repo: open the clinical-ai-gateway repository on GitHub, click Settings, scroll down to the Danger Zone section, and click Delete this repository. Confirm by typing the repository name.
  • Delete the Linear project: open the project, click the three-dot menu next to the project name (beside the Overview and Issues tabs), and select Delete. Confirm the deletion when prompted.

That's a Wrap!

That's a Wrap!

Nice work! You just built a self-hosted clinical-AI gateway that proves AI can reach the last clinic. One Docker Compose command brings up an entire vendor-agnostic routing layer, and five scripted validation drills prove it works under the exact conditions that stalled the original pilot.

You've learned how to:

  • Build a vendor-agnostic AI gateway that routes clinical apps through one OpenAI-compatible endpoint, swapping models by config alone, with LiteLLM as the routing brain and Ollama as the offline floor.
  • Prove offline reach, data sovereignty, health-aware failover, multilingual access, and per-capita cost governance compose into one deployable substrate by running a scripted validation drill that passes all five bars end-to-end.
  • Ship infrastructure through a professional delivery pipeline: GitHub (private repo, feature branches, PRs, tagged release), Linear (planned tickets closed on merge), five ADRs, Kubernetes manifests, Terraform config, and a plain-language ops guide.
  • Secret Mission: Produced the population-impact briefing, annotated Mermaid system diagram, and two-audience walkthrough that gets the gateway funded at the coalition steering meeting.

Ready to quiz yourself?