Route OpenHands Models Across Providers

Route OpenHands agent roles through a local multi-provider LiteLLM gateway.

Introduction

30 Second Summary

A good team gives different jobs to people with different strengths. Using one decision-maker for every task creates avoidable blind spots.

In this project, you will build a local model control plane that routes four OpenHands roles through a LiteLLM Gateway. You will use free local mock inference APIs to prove each role can reach its selected provider without exposing provider credentials to agents.

What You'll Build

Your final demo prints a routing transcript that proves each role reached its selected provider through its own model profile.

By the end of this project, you'll have:

  • A deterministic local provider test that shows whether Provider A or Provider B handled each request.
  • A gateway boundary where agents use stable model aliases while provider credentials stay outside their definitions.
  • Reusable per-role profiles that let each specialist load its selected model from routing.json.
  • Secret Mission: Build a provider policy that blocks any configuration where the implementer shares its provider with the auditor.

Are there any prerequisites?

No prior setup or provider account is required. Step 1 covers every Windows installation required for the free local path.

Before We Start

Before any tools enter the picture, lock in the boundary that keeps this project focused. You are committing to route each agent role independently so provider diversity can support an independent audit.

Set Up the Windows Workspace

Your LiteLLM Gateway needs a reproducible Python environment before its routing behavior can be trusted. The OpenHands SDK relies on that same foundation.

You will prepare the Windows workspace. You will then create a uv environment from pinned dependencies.

In this step, get ready to:
  • Confirm PowerShell can launch Visual Studio Code.
  • Prepare the pinned Python toolchain.
  • Build the project environment from pyproject.toml.
Confirm PowerShell and Visual Studio Code

Windows PowerShell runs every setup command in this project. Visual Studio Code gives you one place to edit the project files.

  • Press the Windows key to open the search bar.
  • Type Windows PowerShell into the search bar.
  • Press Enter to open PowerShell.

Before you run this check, consider whether Visual Studio Code can open the folder represented by your current PowerShell location.

  • Test the Visual Studio Code command by running:
code .

What does this command do?

The code . command asks Visual Studio Code to open your current PowerShell folder. A new editor window confirms that the command is available through your system path.

✔️ Visual Studio Code opens

Your command path is ready. Visual Studio Code can now open a folder directly from PowerShell.

  • Close the Visual Studio Code window to return to PowerShell.

ⓧ The command does not open Visual Studio Code

The User setup adds the code command to your path. Reopening PowerShell loads that updated path.

  • Open the Visual Studio Code Windows setup guide in your browser.
  • Download the Visual Studio Code User setup for Windows.
  • Run the downloaded installer.
  • Close PowerShell after the installation finishes.
  • Press the Windows key to reopen the search bar.
  • Type Windows PowerShell into the search bar.
  • Press Enter to open a fresh PowerShell window.
  • Test the command again by running:
code .

Why reopen PowerShell?

The Visual Studio Code installer updates your system path. A fresh PowerShell session reads the updated value.

You should now see Visual Studio Code open your current folder.

  • Close the Visual Studio Code window to return to PowerShell.

Visual Studio Code still does not open?

Confirm that you installed the User setup. Restart Windows if a fresh PowerShell window still cannot find the command.

Ask for help with the command path: Why does code . fail after I installed Visual Studio Code User setup on Windows?

Install uv and Python 3.12

uv manages the project environment from one configuration file. It also installs the exact Python runtime required by the pinned packages.

  • Check whether uv is already available by running:
uv --version

What does this command do?

The uv --version command prints the installed uv release. This project uses uv 0.13.1.

✔️ I see uv 0.13.1

That version is ready for the project environment.

ⓧ I see an older version

Your installed release is older than the project pin. The official installer updates uv to the current release used here.

  • Update uv by running:
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

What does this command do?

PowerShell downloads the official uv installation script. It then installs uv for your Windows user.

  • Close PowerShell after the installer finishes.
  • Reopen PowerShell through the Windows search bar.
  • Confirm the installed release by running:
uv --version

What should this confirm?

The command should now report uv 0.13.1.

ⓧ Command not found

uv is not installed yet. The official installer adds it for your Windows user.

  • Install uv by running:
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

What does this command do?

PowerShell downloads the official uv installation script. It then installs uv for your Windows user.

  • Close PowerShell after the installer finishes.
  • Reopen PowerShell through the Windows search bar.
  • Confirm the installation by running:
uv --version

What should this confirm?

The command should report uv 0.13.1.

PowerShell still cannot find uv?

Close every PowerShell window before opening a new one. The installer updates your path for future terminal sessions.

Get help with the installation: Why is uv missing after I ran the Windows installer?

Python 3.12 satisfies the package requirements for this project. uv can install it without replacing another Python release already on your computer.

  • Check the Python version currently available in PowerShell by running:
python --version

What does this command do?

The python --version command prints the Python release currently available through your system path. The result determines which setup state applies to you.

✔️ I see Python 3.12

Your system already has a compatible release. uv will ensure that a managed Python 3.12 installation is available for the project.

  • Prepare the uv-managed runtime by running:
uv python install 3.12

What does this command do?

uv finds or installs the latest Python 3.12 patch release. The project can then select that managed runtime.

ⓧ I see another version

Your existing Python release can remain installed. uv adds Python 3.12 for this project.

  • Install the required runtime by running:
uv python install 3.12

What does this command do?

uv installs the latest Python 3.12 patch release. Your other Python installation stays available.

ⓧ Command not found

No system Python command is available. uv can provide the required runtime directly.

  • Install the required runtime by running:
uv python install 3.12

What does this command do?

uv downloads the latest Python 3.12 patch release. This gives the project a managed interpreter without a separate Python installer.

Your pinned toolchain is now available. The project can use uv 0.13.1 with Python 3.12.

Create the project environment

The pyproject.toml file records the Python range plus the exact package versions. uv reads this file to create the same environment on future runs.

  • Move to your Desktop by running:
Set-Location -Path "$HOME\Desktop"

What does this command do?

The command changes PowerShell's current location to your Desktop. The project folder will be easy to find there.

  • Create the openhands-multi-provider-router folder and move into it by running:
New-Item -Path "openhands-multi-provider-router" -ItemType Directory
Set-Location -Path "openhands-multi-provider-router"

What do these commands do?

  • The first command creates the openhands-multi-provider-router directory on your Desktop.
  • The second command makes that directory your current PowerShell location.
  • Open the project folder in Visual Studio Code by running:
code .

What should open?

Visual Studio Code opens the openhands-multi-provider-router folder. New files created from PowerShell will appear in its file tree.

  • Create an empty pyproject.toml file from PowerShell by running:
New-Item -Path "pyproject.toml" -ItemType File

What does this command do?

PowerShell creates pyproject.toml inside the current project folder. Visual Studio Code detects the new file automatically.

  • Switch back to Visual Studio Code.
  • Select pyproject.toml in the file tree.
  • Add the project configuration by pasting:
[project]
name = "openhands-multi-provider-router"
version = "0.1.0"
description = "A local multi-provider model control plane for OpenHands agents"
requires-python = ">=3.12,<3.15"
dependencies = [
    "litellm[proxy]==1.105.1",
    "openhands-sdk==1.55.1",
]

What does this configuration do?

  • The project metadata gives the environment a stable name plus a local version.
  • The Python range accepts Python 3.12 through releases below 3.15.
  • The dependency list pins litellm[proxy]==1.105.1 plus openhands-sdk==1.55.1 for reproducible installs.
  • Save pyproject.toml.
  • Confirm the file exists from PowerShell by running:
Get-ChildItem -Path .

What does this command do?

PowerShell lists the contents of your current project folder. You should see pyproject.toml in the output.

Do you not see pyproject.toml?

Confirm that Visual Studio Code saved the file inside openhands-multi-provider-router. Check that the filename ends with .toml.

Get help locating the file: Why is pyproject.toml missing from my PowerShell project folder?

✔️ Awesome, I've got everything!

Great. Double check that pyproject.toml is saved inside openhands-multi-provider-router.

ⓧ I'd like to double check the full code

Compare your complete pyproject.toml file with this reference.

[project]
name = "openhands-multi-provider-router"
version = "0.1.0"
description = "A local multi-provider model control plane for OpenHands agents"
requires-python = ">=3.12,<3.15"
dependencies = [
    "litellm[proxy]==1.105.1",
    "openhands-sdk==1.55.1",
]

What should match?

Every project value plus dependency pin should match this reference. The final comma after the OpenHands dependency is valid TOML.

The first environment sync can pause while uv downloads the pinned packages. A quiet terminal during that download is normal.

  • Create the uv-managed environment from pyproject.toml by running:
uv sync

What does this command do?

The uv sync command creates or updates the project environment. It installs the exact dependencies recorded in pyproject.toml.

The command returns you to the PowerShell prompt after the environment is synchronized.

Does the dependency sync fail?

Confirm that PowerShell is inside the openhands-multi-provider-router folder. Compare every quote plus comma in pyproject.toml with the reference above.

Ask for help with the sync: Why does uv sync fail for my pinned OpenHands project?

Before you run the first final check, predict which Python release the project environment will select.

  • Check the project interpreter by running:
uv run python --version

What does this command verify?

The uv run command executes Python inside the synchronized project environment. This check proves that the project selected its compatible interpreter.

You should see Python 3.12 in the output.

Before you run the final check, predict whether both pinned libraries can be imported inside that same environment.

  • Verify the installed dependencies by running:
uv run python -c "import openhands.sdk, litellm; print('Dependencies ready')"

What does this command verify?

Python imports openhands.sdk plus litellm from the uv-managed environment. The print statement runs only after both imports succeed.

That is the workspace locked in. You should see Dependencies ready in PowerShell.

Your Windows workspace now has a pinned runtime plus a synchronized environment. Next, you will place two local inference APIs behind one LiteLLM Gateway.

Route Mock Providers Through LiteLLM

Your Windows workspace is ready to run the control plane. This step adds the local inference boundary that every agent uses later.

Direct provider calls would force each agent to understand provider-specific endpoints. A LiteLLM Gateway gives every agent one stable endpoint with model aliases that can route to different providers.

In this step, get ready to:
  • Create two local inference APIs that identify which provider handled each request.
  • Map four stable model aliases across the two providers.
  • Start the providers and verify the gateway routes to them.
Build the mock inference providers

The mock providers expose the same request shape that an OpenAI-compatible API uses. Their responses include a provider label so you can observe every routing decision without paid credentials.

  • In the VS Code Explorer sidebar, select the new-file control.
  • Name the file mock_providers.py.
  • Add the imports and message-extraction helper by pasting this code:
from __future__ import annotations

import json
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
from typing import Any


def last_message_text(payload: dict[str, Any]) -> str:
    messages = payload.get("messages", [])
    if not messages or not isinstance(messages[-1], dict):
        return ""

    content = messages[-1].get("content", "")
    if isinstance(content, str):
        return content
    if isinstance(content, list):
        parts = []
        for item in content:
            if isinstance(item, dict) and isinstance(item.get("text"), str):
                parts.append(item["text"])
        return " ".join(parts)
    return str(content)

What does this helper do?

  • The imports provide JSON handling and local HTTP servers.
  • The last_message_text() helper reads the latest message from a completion request.
  • The helper supports plain text content plus content supplied as a list.
  • Add the provider-specific request handler below last_message_text() by pasting this code:
def handler_for(provider_name: str) -> type[BaseHTTPRequestHandler]:
    class MockProviderHandler(BaseHTTPRequestHandler):
        def write_json(self, status: int, payload: dict[str, Any]) -> None:
            body = json.dumps(payload).encode("utf-8")
            self.send_response(status)
            self.send_header("Content-Type", "application/json")
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)

How are responses written?

  • The handler_for() function creates a handler tied to one provider name.
  • The write_json() method serializes each response body.
  • The response headers tell clients how to interpret the returned bytes.
  • Inside MockProviderHandler, add the health and model-discovery routes by pasting this code below write_json():
        def do_GET(self) -> None:
            if self.path == "/health":
                self.write_json(200, {"status": "healthy", "provider": provider_name})
                return
            if self.path == "/v1/models":
                self.write_json(
                    200,
                    {
                        "object": "list",
                        "data": [
                            {
                                "id": f"{provider_name.lower().replace(' ', '-')}-local",
                                "object": "model",
                                "created": 0,
                                "owned_by": provider_name,
                            }
                        ],
                    },
                )
                return
            self.write_json(404, {"error": "not found"})

What do these routes expose?

  • The /health route reports whether the selected provider is available.
  • The /v1/models route returns a local model record owned by that provider.
  • Unknown paths receive a missing-resource response.
  • Inside MockProviderHandler, begin the completion route by pasting this code below do_GET():
        def do_POST(self) -> None:
            if not self.path.endswith("/chat/completions"):
                self.write_json(404, {"error": "not found"})
                return

            content_length = int(self.headers.get("Content-Length", "0"))
            raw_body = self.rfile.read(content_length)
            payload = json.loads(raw_body or b"{}")
            model = str(payload.get("model", "unknown-model"))
            prompt = last_message_text(payload)
            answer = f"{provider_name} handled {model}: {prompt}"

How is a completion request identified?

  • The handler accepts requests whose paths end with /chat/completions.
  • The request body supplies the model name plus the message payload.
  • The answer records the provider name with the requested model.
  • Complete do_POST() by pasting this response directly below the answer line:
            self.write_json(
                200,
                {
                    "id": f"chatcmpl-{provider_name.lower().replace(' ', '-')}",
                    "object": "chat.completion",
                    "created": int(time.time()),
                    "model": model,
                    "choices": [
                        {
                            "index": 0,
                            "message": {"role": "assistant", "content": answer},
                            "finish_reason": "stop",
                        }
                    ],
                    "usage": {
                        "prompt_tokens": 1,
                        "completion_tokens": 1,
                        "total_tokens": 2,
                    },
                },
            )

Why use this response shape?

  • The response contains an assistant message that LiteLLM can pass back to its caller.
  • The model field preserves the selected mock model.
  • The fixed usage values keep local responses deterministic.
  • Finish handler_for() by adding request logging plus the completed handler below do_POST():
        def log_message(self, message_format: str, *args: object) -> None:
            print(f"[{provider_name}] {message_format % args}")

    return MockProviderHandler

What does the final handler code add?

  • The request log prefixes each line with the provider name.
  • The completed handler can now be assigned to a local server.
  • Add the server startup logic at the bottom of mock_providers.py by pasting this code:
def main() -> None:
    servers = [
        ThreadingHTTPServer(("127.0.0.1", 8001), handler_for("Provider A")),
        ThreadingHTTPServer(("127.0.0.1", 8002), handler_for("Provider B")),
    ]

    for server in servers:
        thread = Thread(target=server.serve_forever, daemon=True)
        thread.start()

    print("Provider A listening on http://127.0.0.1:8001/v1")
    print("Provider B listening on http://127.0.0.1:8002/v1")
    print("Press Ctrl+C to stop both providers.")

    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        for server in servers:
            server.shutdown()
            server.server_close()
        print("Mock providers stopped.")


if __name__ == "__main__":
    main()

How do both providers run together?

  • Provider A listens on http://127.0.0.1:8001/v1.
  • Provider B listens on http://127.0.0.1:8002/v1.
  • A background thread keeps each local HTTP server available.
  • The main loop keeps the process running until you stop it.
  • Save mock_providers.py.
  • In VS Code, select Terminal from the top menu.
  • Select New Terminal.

Before you start the providers, what two listening addresses do you expect the terminal to print?

  • Start both mock providers by running this command:
uv run python mock_providers.py

What should you see?

You should see Provider A listening on http://127.0.0.1:8001/v1. You should also see Provider B listening on http://127.0.0.1:8002/v1.

Keep this terminal running. It is now serving both local inference APIs.

Providers do not start?

  • Confirm that the terminal path ends with openhands-multi-provider-router.
  • Check that mock_providers.py is saved in that folder.
  • Stop any earlier process using either provider port before retrying.

I cannot start the mock providers.

✔️ Both providers are listening

Both local providers are running. Keep their PowerShell terminal active while you configure the gateway.

ⓧ I'd like to double check the full code

Compare your saved mock_providers.py with this complete file.

from __future__ import annotations

import json
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
from typing import Any


def last_message_text(payload: dict[str, Any]) -> str:
    messages = payload.get("messages", [])
    if not messages or not isinstance(messages[-1], dict):
        return ""

    content = messages[-1].get("content", "")
    if isinstance(content, str):
        return content
    if isinstance(content, list):
        parts = []
        for item in content:
            if isinstance(item, dict) and isinstance(item.get("text"), str):
                parts.append(item["text"])
        return " ".join(parts)
    return str(content)


def handler_for(provider_name: str) -> type[BaseHTTPRequestHandler]:
    class MockProviderHandler(BaseHTTPRequestHandler):
        def write_json(self, status: int, payload: dict[str, Any]) -> None:
            body = json.dumps(payload).encode("utf-8")
            self.send_response(status)
            self.send_header("Content-Type", "application/json")
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)

        def do_GET(self) -> None:
            if self.path == "/health":
                self.write_json(200, {"status": "healthy", "provider": provider_name})
                return
            if self.path == "/v1/models":
                self.write_json(
                    200,
                    {
                        "object": "list",
                        "data": [
                            {
                                "id": f"{provider_name.lower().replace(' ', '-')}-local",
                                "object": "model",
                                "created": 0,
                                "owned_by": provider_name,
                            }
                        ],
                    },
                )
                return
            self.write_json(404, {"error": "not found"})

        def do_POST(self) -> None:
            if not self.path.endswith("/chat/completions"):
                self.write_json(404, {"error": "not found"})
                return

            content_length = int(self.headers.get("Content-Length", "0"))
            raw_body = self.rfile.read(content_length)
            payload = json.loads(raw_body or b"{}")
            model = str(payload.get("model", "unknown-model"))
            prompt = last_message_text(payload)
            answer = f"{provider_name} handled {model}: {prompt}"

            self.write_json(
                200,
                {
                    "id": f"chatcmpl-{provider_name.lower().replace(' ', '-')}",
                    "object": "chat.completion",
                    "created": int(time.time()),
                    "model": model,
                    "choices": [
                        {
                            "index": 0,
                            "message": {"role": "assistant", "content": answer},
                            "finish_reason": "stop",
                        }
                    ],
                    "usage": {
                        "prompt_tokens": 1,
                        "completion_tokens": 1,
                        "total_tokens": 2,
                    },
                },
            )

        def log_message(self, message_format: str, *args: object) -> None:
            print(f"[{provider_name}] {message_format % args}")

    return MockProviderHandler


def main() -> None:
    servers = [
        ThreadingHTTPServer(("127.0.0.1", 8001), handler_for("Provider A")),
        ThreadingHTTPServer(("127.0.0.1", 8002), handler_for("Provider B")),
    ]

    for server in servers:
        thread = Thread(target=server.serve_forever, daemon=True)
        thread.start()

    print("Provider A listening on http://127.0.0.1:8001/v1")
    print("Provider B listening on http://127.0.0.1:8002/v1")
    print("Press Ctrl+C to stop both providers.")

    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        for server in servers:
            server.shutdown()
            server.server_close()
        print("Mock providers stopped.")


if __name__ == "__main__":
    main()
Map stable aliases through LiteLLM

A model alias is the stable name that callers select. LiteLLM resolves that alias to a provider-specific model plus its API base.

  • In the VS Code Explorer sidebar, select the new-file control.
  • Name the file gateway.yaml.
  • Define the four model aliases by pasting this configuration:
model_list:
  - model_name: chat-fast
    litellm_params:
      model: openai/mock-chat
      api_base: http://127.0.0.1:8001/v1
      api_key: none

  - model_name: implementer-coding
    litellm_params:
      model: openai/mock-code
      api_base: http://127.0.0.1:8001/v1
      api_key: none

  - model_name: orchestrator-strong
    litellm_params:
      model: openai/mock-orchestrator
      api_base: http://127.0.0.1:8002/v1
      api_key: none

  - model_name: auditor-independent
    litellm_params:
      model: openai/mock-auditor
      api_base: http://127.0.0.1:8002/v1
      api_key: none

litellm_settings:
  drop_params: true

How does the alias map work?

  • The model_name values are the aliases that gateway clients can select.
  • The chat-fast alias routes to Provider A.
  • The implementer-coding alias also routes to Provider A.
  • The other two aliases route to Provider B.
  • Save gateway.yaml.
  • In VS Code, select Terminal from the top menu.
  • Select New Terminal to create a separate PowerShell terminal.
  • Start LiteLLM Gateway on port 4000 by running this command:
uv run litellm --config gateway.yaml --port 4000

What does this command start?

LiteLLM reads gateway.yaml and exposes its model aliases through http://127.0.0.1:4000.

Keep this second terminal running. The gateway needs the mock-provider terminal to remain active too.

Gateway does not stay running?

  • Confirm that the new terminal path ends with openhands-multi-provider-router.
  • Check that gateway.yaml is saved in that folder.
  • Confirm that the mock-provider terminal is still running.

My LiteLLM Gateway does not start.

✔️ The gateway is running

The provider process plus the gateway process are both active. Leave both terminals running for the final checks.

ⓧ I'd like to double check the full code

Compare your saved gateway.yaml with this complete configuration.

model_list:
  - model_name: chat-fast
    litellm_params:
      model: openai/mock-chat
      api_base: http://127.0.0.1:8001/v1
      api_key: none

  - model_name: implementer-coding
    litellm_params:
      model: openai/mock-code
      api_base: http://127.0.0.1:8001/v1
      api_key: none

  - model_name: orchestrator-strong
    litellm_params:
      model: openai/mock-orchestrator
      api_base: http://127.0.0.1:8002/v1
      api_key: none

  - model_name: auditor-independent
    litellm_params:
      model: openai/mock-auditor
      api_base: http://127.0.0.1:8002/v1
      api_key: none

litellm_settings:
  drop_params: true
Verify gateway readiness and aliases

A readiness check proves that the gateway can accept requests. The model inventory proves that clients can discover all four stable aliases.

  • In VS Code, select Terminal from the top menu.
  • Select New Terminal to create a third PowerShell terminal for checks.

Before you run the first check, do you expect the gateway to report that it is ready or unavailable?

  • Check gateway readiness by running this command:
Invoke-RestMethod http://127.0.0.1:4000/health/readiness

What does readiness prove?

You should see a response with a healthy status. That result proves the gateway process is ready to receive requests.

Now predict how many aliases the model inventory should return before you query it.

  • List the model aliases exposed by the gateway by running this command:
Invoke-RestMethod http://127.0.0.1:4000/v1/models

What should the model inventory contain?

You should find chat-fast plus implementer-coding in the response. You should also find orchestrator-strong plus auditor-independent.

The four names are now the provider-neutral interface that later agent code uses.

Cannot reach the gateway?

  • Return to the LiteLLM terminal to confirm that its process is still running.
  • Return to the mock-provider terminal to confirm that both local APIs are still running.
  • Check that the readiness request uses port 4000.

My readiness check or model inventory request cannot reach LiteLLM.

You have the control-plane boundary working. Four stable aliases now route through one gateway to two independently identifiable providers.

Your local inference layer is ready. Next, you will send every role through one shared model so the single-model bottleneck becomes visible.

Expose the Single-Model Bottleneck

Your LiteLLM Gateway already exposes four aliases across two providers. Each alias responds through one stable gateway address.

A useful control plane must let each role choose its own model. This step deliberately sends every role through one shared OpenHands SDK profile so you can see the constraint before changing the design.

In this step, get ready to:
  • Create the initial OpenHands router.
  • Send four role-labeled requests through one shared model.
  • Confirm Provider B handles every request.
Create the router foundation

The router needs a fixed set of role names. It also needs the gateway address that every model request crosses.

  • Switch back to Visual Studio Code from earlier.
  • Use the Explorer sidebar's new-file control to create role_router.py inside the open project folder.
  • Add the imports plus the shared router constants by pasting this code into role_router.py:
from __future__ import annotations

import argparse
import json
import os
from pathlib import Path
from typing import Any

from openhands.sdk import LLM, LLMProfileStore, Message, TextContent

ROLES = ("chat", "orchestrator", "implementer", "auditor")
CONFIG_PATH = Path("routing.json")
PROFILE_DIR = Path("profiles")
PROXY_URL = os.getenv("LITELLM_PROXY_API_BASE", "http://127.0.0.1:4000")
os.environ.setdefault("LITELLM_PROXY_API_KEY", "local-proxy-key")

What does this code establish?

  • The OpenHands imports provide the model client plus the message types used for each request.
  • The ROLES tuple fixes the four role labels used throughout the demonstration.
  • The PROXY_URL value points every model call at the local gateway.
  • The proxy key stays at the gateway boundary instead of entering a prompt.
  • Save role_router.py.
  • Confirm role_router.py appears in the Explorer sidebar.

Cannot find the new file?

Check that the Explorer sidebar still shows the Windows project folder from the earlier setup. Create role_router.py directly inside that folder.

Ask for help with the file location: Help me confirm that role_router.py is inside my open Visual Studio Code project folder.

  • Add the response helpers directly below the proxy key line by pasting this code:
def response_text(response: Any) -> str:
    message = response.message
    texts = [
        content.text
        for content in message.content
        if isinstance(content, TextContent)
    ]
    return " ".join(texts) if texts else str(message)


def ask_llm(llm: LLM, prompt: str) -> str:
    response = llm.completion(
        messages=[Message(role="user", content=[TextContent(text=prompt)])]
    )
    return response_text(response)


def ask_role(store: LLMProfileStore, role: str, prompt: str) -> str:
    return ask_llm(store.load(role), prompt)

How do requests become readable text?

  • The response_text() helper collects text content from an OpenHands response.
  • The ask_llm() helper wraps a prompt in the message format expected by the model client.
  • The ask_role() helper supports named profiles in the router's general command path.
  • Save role_router.py.
  • Confirm the saved file contains response_text() above ask_llm().

Seeing incomplete helper code?

Check the indentation inside the list comprehension. Every line between the opening and closing square brackets must match the code block.

Ask for a focused comparison: Help me compare my response helpers with the expected role_router.py code.

Route all roles through one model

One shared LLM profile makes the bottleneck concrete. Every role label changes while the selected model stays fixed.

  • Add the naive demonstration directly below ask_role() by pasting this function:
def run_naive_demo(message: str) -> None:
    shared_llm = LLM(
        usage_id="shared",
        model="litellm_proxy/orchestrator-strong",
        base_url=PROXY_URL,
    )
    print("Naive routing: every role uses orchestrator-strong")
    for role in ROLES:
        answer = ask_llm(shared_llm, f"[{role}] {message}")
        print(f"{role:12} -> {answer}")

Where is the bottleneck?

  • The shared_llm object has one usage ID.
  • Its model is permanently set to litellm_proxy/orchestrator-strong.
  • The loop changes only the role label in each prompt.
  • All four calls therefore cross the same alias boundary.
  • Save role_router.py.
  • Confirm run_naive_demo() appears below ask_role().

Is the shared model unclear?

Look for the single LLM( call inside run_naive_demo(). The loop reuses that object for every role.

Ask for a walkthrough: Explain why run_naive_demo sends every role to the same model.

  • Add the command-line entry point directly below run_naive_demo() by pasting this function:
def main() -> None:
    parser = argparse.ArgumentParser(description="Route OpenHands roles to model profiles")
    actions = parser.add_mutually_exclusive_group(required=True)
    actions.add_argument("--demo-naive", action="store_true")
    actions.add_argument("--show-bindings", action="store_true")
    actions.add_argument("--role", choices=ROLES)
    parser.add_argument(
        "--message",
        default="Report which provider and model handled this request.",
    )
    args = parser.parse_args()

    if args.demo_naive:
        run_naive_demo(args.message)
        return

    config = load_routing_config()
    store = build_profiles(config)

    if args.show_bindings:
        show_bindings(config)
        return

    role = str(args.role)
    alias = config["roles"][role]
    print(f"{role} -> {alias}")
    print(ask_role(store, role, args.message))

How does the naive action stay isolated?

  • The parser selects one router action at a time.
  • The --demo-naive branch calls the shared-model demonstration.
  • The immediate return stops that branch before the general routing configuration is loaded.
  • Save role_router.py.
  • Confirm the --demo-naive branch appears before the configuration loader call.

Does the parser look incomplete?

Check that the closing parenthesis for the --message argument appears before args = parser.parse_args().

Ask for help with the entry point: Help me check the main function in role_router.py for missing parentheses or indentation.

  • Make the command-line entry point run when the file is executed by adding this code at the bottom:
if __name__ == "__main__":
    main()

What launches the router?

This guard calls main() when PowerShell runs role_router.py as a script.

  • Save role_router.py.
  • Confirm the launcher remains at the bottom of the file.

Does the script return immediately?

Check that main() is indented beneath the script guard. Also check that the guard sits outside every function.

Ask for help with the launcher: Help me verify the script guard at the bottom of role_router.py.

Before you run the shared router, do you expect the four role labels to change the selected provider?

  • Run the naive demonstration in the PowerShell terminal for the project by executing:
uv run python role_router.py --demo-naive

What does this run prove?

The first line states that every role uses orchestrator-strong. The four result lines identify chat, orchestrator, implementer, and auditor separately.

Each result says Provider B handled the request. Changing the prompt label did not create a separate model choice.

Did the naive demonstration fail?

  • Confirm the mock-provider terminal from earlier is still running.
  • Confirm the gateway terminal from earlier is still running.
  • Check that orchestrator-strong still appears in gateway.yaml.

Ask for help with the request path: Help me trace why my naive OpenHands request is not reaching Provider B.

Complete the cumulative router file

The shared run has exposed the intended shortfall. The cumulative router file also retains the configuration helpers used by its non-naive command path.

  • Find the LITELLM_PROXY_API_KEY environment line near the top of role_router.py.
  • Insert the routing configuration loader directly below that line by pasting this function:
def load_routing_config() -> dict[str, Any]:
    config = json.loads(CONFIG_PATH.read_text(encoding="utf-8"))
    models = config.get("models")
    roles = config.get("roles")
    if not isinstance(models, dict) or not isinstance(roles, dict):
        raise ValueError("routing.json must contain 'models' and 'roles' objects")

    for role in ROLES:
        alias = roles.get(role)
        if not isinstance(alias, str):
            raise ValueError(f"Missing model alias for role: {role}")
        if alias not in models:
            raise ValueError(f"Role {role} references unknown model alias: {alias}")
    return config

What does the loader protect?

  • The loader reads the project routing configuration as JSON.
  • It requires a model alias for every role in ROLES.
  • It rejects any role that points to an unknown alias.
  • Save role_router.py.
  • Confirm load_routing_config() appears before response_text().

Is the loader nested accidentally?

Check that def load_routing_config() begins at the far left of the editor. A leading indent would place it inside another block.

Ask for help with the function boundary: Help me check whether load_routing_config is at the top level of role_router.py.

  • Add the profile builder directly below load_routing_config() by pasting this function:
def build_profiles(config: dict[str, Any]) -> LLMProfileStore:
    PROFILE_DIR.mkdir(exist_ok=True)
    store = LLMProfileStore(base_dir=PROFILE_DIR)
    roles = config["roles"]

    for role in ROLES:
        alias = roles[role]
        store.save(
            role,
            LLM(
                usage_id=role,
                model=f"litellm_proxy/{alias}",
                base_url=PROXY_URL,
            ),
        )
    return store

How are named profiles materialized?

  • The profile store uses the local profiles directory.
  • Each role receives an OpenHands model whose alias comes from the supplied configuration.
  • The gateway URL remains shared across all saved profiles.
  • Save role_router.py.
  • Confirm build_profiles() appears before response_text().

Is the profile loop misaligned?

Check that store.save( sits inside the role loop. Check that return store sits outside the loop.

Ask for an indentation review: Help me check the build_profiles indentation in role_router.py.

  • Confirm the naive path still reaches Provider B by running the demonstration again:
uv run python role_router.py --demo-naive

Why does the naive path stay unchanged?

The --demo-naive branch returns before the configuration loader runs. The same shared model still handles all four calls.

Did the second run stop early?

Check the two inserted functions for missing indentation or unmatched parentheses. The PowerShell output identifies the line where Python stopped parsing the file.

Ask for help with that line: Help me diagnose the line reported when role_router.py stopped running.

  • Find ask_role() in role_router.py.
  • Insert the binding display helper directly below ask_role() by pasting this function:
def show_bindings(config: dict[str, Any]) -> None:
    print("Role bindings:")
    for role in ROLES:
        alias = config["roles"][role]
        provider = config["models"][alias]["provider"]
        print(f"  {role:12} -> {alias:24} -> {provider}")

What does the binding display reveal?

The helper formats each role beside its selected alias plus provider metadata. The naive demonstration does not call it.

  • Save role_router.py.
  • Confirm show_bindings() appears immediately before run_naive_demo().

Is the display helper in the wrong place?

Check that show_bindings() begins at the far left. Its final print line must remain inside the role loop.

Ask for help with the placement: Help me place show_bindings between ask_role and run_naive_demo.

✔️ Awesome, I've got everything!

Great. Double check that role_router.py is saved before the final demonstration.

ⓧ I'd like to double check the full code

Compare your complete role_router.py file with this cumulative version.

from __future__ import annotations

import argparse
import json
import os
from pathlib import Path
from typing import Any

from openhands.sdk import LLM, LLMProfileStore, Message, TextContent

ROLES = ("chat", "orchestrator", "implementer", "auditor")
CONFIG_PATH = Path("routing.json")
PROFILE_DIR = Path("profiles")
PROXY_URL = os.getenv("LITELLM_PROXY_API_BASE", "http://127.0.0.1:4000")
os.environ.setdefault("LITELLM_PROXY_API_KEY", "local-proxy-key")


def load_routing_config() -> dict[str, Any]:
    config = json.loads(CONFIG_PATH.read_text(encoding="utf-8"))
    models = config.get("models")
    roles = config.get("roles")
    if not isinstance(models, dict) or not isinstance(roles, dict):
        raise ValueError("routing.json must contain 'models' and 'roles' objects")

    for role in ROLES:
        alias = roles.get(role)
        if not isinstance(alias, str):
            raise ValueError(f"Missing model alias for role: {role}")
        if alias not in models:
            raise ValueError(f"Role {role} references unknown model alias: {alias}")
    return config


def build_profiles(config: dict[str, Any]) -> LLMProfileStore:
    PROFILE_DIR.mkdir(exist_ok=True)
    store = LLMProfileStore(base_dir=PROFILE_DIR)
    roles = config["roles"]

    for role in ROLES:
        alias = roles[role]
        store.save(
            role,
            LLM(
                usage_id=role,
                model=f"litellm_proxy/{alias}",
                base_url=PROXY_URL,
            ),
        )
    return store


def response_text(response: Any) -> str:
    message = response.message
    texts = [
        content.text
        for content in message.content
        if isinstance(content, TextContent)
    ]
    return " ".join(texts) if texts else str(message)


def ask_llm(llm: LLM, prompt: str) -> str:
    response = llm.completion(
        messages=[Message(role="user", content=[TextContent(text=prompt)])]
    )
    return response_text(response)


def ask_role(store: LLMProfileStore, role: str, prompt: str) -> str:
    return ask_llm(store.load(role), prompt)


def show_bindings(config: dict[str, Any]) -> None:
    print("Role bindings:")
    for role in ROLES:
        alias = config["roles"][role]
        provider = config["models"][alias]["provider"]
        print(f"  {role:12} -> {alias:24} -> {provider}")


def run_naive_demo(message: str) -> None:
    shared_llm = LLM(
        usage_id="shared",
        model="litellm_proxy/orchestrator-strong",
        base_url=PROXY_URL,
    )
    print("Naive routing: every role uses orchestrator-strong")
    for role in ROLES:
        answer = ask_llm(shared_llm, f"[{role}] {message}")
        print(f"{role:12} -> {answer}")


def main() -> None:
    parser = argparse.ArgumentParser(description="Route OpenHands roles to model profiles")
    actions = parser.add_mutually_exclusive_group(required=True)
    actions.add_argument("--demo-naive", action="store_true")
    actions.add_argument("--show-bindings", action="store_true")
    actions.add_argument("--role", choices=ROLES)
    parser.add_argument(
        "--message",
        default="Report which provider and model handled this request.",
    )
    args = parser.parse_args()

    if args.demo_naive:
        run_naive_demo(args.message)
        return

    config = load_routing_config()
    store = build_profiles(config)

    if args.show_bindings:
        show_bindings(config)
        return

    role = str(args.role)
    alias = config["roles"][role]
    print(f"{role} -> {alias}")
    print(ask_role(store, role, args.message))


if __name__ == "__main__":
    main()

Before the final run, do you expect the implementer or auditor labels to override the shared model selection?

  • Run the completed naive demonstration from the project PowerShell terminal by executing:
uv run python role_router.py --demo-naive

What should you see?

The terminal prints four role lines. Every line reports Provider B because run_naive_demo() reuses orchestrator-strong.

The implementer cannot select implementer-coding. The auditor cannot select auditor-independent while the shared object owns the model decision.

Do the four roles show different providers?

Check the model value inside run_naive_demo(). It must match the cumulative code exactly.

Ask for help tracing unexpected routing: Help me explain why my naive demo is not sending all four roles through Provider B.

You found the control plane's first hard limit: one shared model sends every role to Provider B. Next, you will separate the model catalog from the role bindings so each role can choose independently.

Bind Profiles to Individual Roles

The last run proved that the LiteLLM Gateway accepts requests. It also exposed the bottleneck: one shared model sends every role to Provider B.

Role bindings move those model choices into a user-editable file. In this step, you'll materialize each choice as a saved OpenHands profile before testing the implementer route.

In this step, get ready to:
  • Separate the model catalog from the four role bindings.
  • Generate one saved LLM profile for each role.
  • Prove that the implementer reaches its selected provider.
Create the routing catalog

The control plane needs a source of truth for available aliases. It also needs a separate place where each role selects an alias.

  • Use the file-creation control in the Visual Studio Code Explorer sidebar to create routing.json beside role_router.py.
  • Define the catalog and role bindings by pasting this configuration into routing.json:
{
  "models": {
    "chat-fast": {
      "provider": "provider-a"
    },
    "orchestrator-strong": {
      "provider": "provider-b"
    },
    "implementer-coding": {
      "provider": "provider-a"
    },
    "auditor-independent": {
      "provider": "provider-b"
    }
  },
  "roles": {
    "chat": "chat-fast",
    "orchestrator": "orchestrator-strong",
    "implementer": "implementer-coding",
    "auditor": "auditor-independent"
  }
}

How does the routing file work?

  • The models object records provider metadata for each stable alias.
  • The roles object selects one catalog alias for each role.
  • The aliases match the model names exposed by the gateway.
  • Provider credentials stay outside this file and outside agent prompts.
  • Save routing.json.
  • Confirm that the Explorer sidebar lists routing.json beside role_router.py.

Is the routing file showing a JSON error?

Check that every property name uses double quotes. Confirm that the final item in each object has no trailing comma.

Compare each opening brace with its closing brace. Help me find the syntax problem in my routing.json file.

Generate the role profiles

The router from the previous step already knows how to validate aliases through load_routing_config(). Its build_profiles() helper converts each valid binding into a reusable profile under profiles/.

Before you run the binding check, which two roles do you expect to resolve to Provider A?

  • Validate the bindings and generate the four profiles by running:
uv run python role_router.py --show-bindings

What does this command prove?

The command loads routing.json before checking that every role has a known alias. A missing role or unknown alias stops profile creation with a clear validation error.

After validation passes, LLMProfileStore saves one profile per role. Each profile points to a litellm_proxy/ alias through PROXY_URL.

What should you see?

  • The chat role maps to chat-fast with provider-a.
  • The orchestrator role maps to orchestrator-strong with provider-b.
  • The implementer role maps to implementer-coding with provider-a.
  • The auditor role maps to auditor-independent with provider-b.
  • Expand the generated profiles/ folder in the Visual Studio Code Explorer sidebar.
  • Confirm that it contains profiles named chat, orchestrator, implementer, and auditor.

Did profile generation stop?

A missing-role error means the roles object does not contain all four names from ROLES. An unknown-alias error means a selected role value does not match a key under models.

Check the spelling and letter case in both objects. Help me diagnose why role_router.py cannot build my profiles.

✔️ Awesome, I've got everything!

Your four bindings are valid. The generated profiles are ready for role-specific requests.

ⓧ I'd like to double check the full code

Compare your complete routing.json file with this reference.

{
  "models": {
    "chat-fast": {
      "provider": "provider-a"
    },
    "orchestrator-strong": {
      "provider": "provider-b"
    },
    "implementer-coding": {
      "provider": "provider-a"
    },
    "auditor-independent": {
      "provider": "provider-b"
    }
  },
  "roles": {
    "chat": "chat-fast",
    "orchestrator": "orchestrator-strong",
    "implementer": "implementer-coding",
    "auditor": "auditor-independent"
  }
}

Compare your complete role_router.py file with this reference.

from __future__ import annotations

import argparse
import json
import os
from pathlib import Path
from typing import Any

from openhands.sdk import LLM, LLMProfileStore, Message, TextContent

ROLES = ("chat", "orchestrator", "implementer", "auditor")
CONFIG_PATH = Path("routing.json")
PROFILE_DIR = Path("profiles")
PROXY_URL = os.getenv("LITELLM_PROXY_API_BASE", "http://127.0.0.1:4000")
os.environ.setdefault("LITELLM_PROXY_API_KEY", "local-proxy-key")


def load_routing_config() -> dict[str, Any]:
    config = json.loads(CONFIG_PATH.read_text(encoding="utf-8"))
    models = config.get("models")
    roles = config.get("roles")
    if not isinstance(models, dict) or not isinstance(roles, dict):
        raise ValueError("routing.json must contain 'models' and 'roles' objects")

    for role in ROLES:
        alias = roles.get(role)
        if not isinstance(alias, str):
            raise ValueError(f"Missing model alias for role: {role}")
        if alias not in models:
            raise ValueError(f"Role {role} references unknown model alias: {alias}")
    return config


def build_profiles(config: dict[str, Any]) -> LLMProfileStore:
    PROFILE_DIR.mkdir(exist_ok=True)
    store = LLMProfileStore(base_dir=PROFILE_DIR)
    roles = config["roles"]

    for role in ROLES:
        alias = roles[role]
        store.save(
            role,
            LLM(
                usage_id=role,
                model=f"litellm_proxy/{alias}",
                base_url=PROXY_URL,
            ),
        )
    return store


def response_text(response: Any) -> str:
    message = response.message
    texts = [
        content.text
        for content in message.content
        if isinstance(content, TextContent)
    ]
    return " ".join(texts) if texts else str(message)


def ask_llm(llm: LLM, prompt: str) -> str:
    response = llm.completion(
        messages=[Message(role="user", content=[TextContent(text=prompt)])]
    )
    return response_text(response)


def ask_role(store: LLMProfileStore, role: str, prompt: str) -> str:
    return ask_llm(store.load(role), prompt)


def show_bindings(config: dict[str, Any]) -> None:
    print("Role bindings:")
    for role in ROLES:
        alias = config["roles"][role]
        provider = config["models"][alias]["provider"]
        print(f"  {role:12} -> {alias:24} -> {provider}")


def run_naive_demo(message: str) -> None:
    shared_llm = LLM(
        usage_id="shared",
        model="litellm_proxy/orchestrator-strong",
        base_url=PROXY_URL,
    )
    print("Naive routing: every role uses orchestrator-strong")
    for role in ROLES:
        answer = ask_llm(shared_llm, f"[{role}] {message}")
        print(f"{role:12} -> {answer}")


def main() -> None:
    parser = argparse.ArgumentParser(description="Route OpenHands roles to model profiles")
    actions = parser.add_mutually_exclusive_group(required=True)
    actions.add_argument("--demo-naive", action="store_true")
    actions.add_argument("--show-bindings", action="store_true")
    actions.add_argument("--role", choices=ROLES)
    parser.add_argument(
        "--message",
        default="Report which provider and model handled this request.",
    )
    args = parser.parse_args()

    if args.demo_naive:
        run_naive_demo(args.message)
        return

    config = load_routing_config()
    store = build_profiles(config)

    if args.show_bindings:
        show_bindings(config)
        return

    role = str(args.role)
    alias = config["roles"][role]
    print(f"{role} -> {alias}")
    print(ask_role(store, role, args.message))


if __name__ == "__main__":
    main()
Test the implementer binding

A binding becomes useful when a request follows it to the selected alias. The implementer profile should load litellm_proxy/implementer-coding before the gateway routes the request to Provider A.

Before you run the request, do you expect Provider A or Provider B to answer?

  • Send a request through the implementer profile by running:
uv run python role_router.py --role implementer --message "Return the provider and model handling this task"

How does the request reach Provider A?

  • The CLI selects the implementer role.
  • The ask_role() helper loads the saved profile with that name.
  • The loaded profile requests implementer-coding through the gateway.
  • The gateway resolves that alias to Provider A without exposing a provider credential to the prompt.

That is the control-plane boundary working: you should see implementer paired with implementer-coding. Provider A should identify itself in the response.

Did the implementer request fail?

Confirm that the mock-provider terminal still shows both local providers running. Confirm that the gateway terminal is still listening on port 4000.

If the response comes from Provider B, check that the implementer binding selects implementer-coding in routing.json. Help me trace why my implementer request reached the wrong provider.

Your four roles now have independent model profiles. Next, you'll attach the implementer and auditor profiles to OpenHands subagents.

Run Agents with Their Own Models

Your four saved profiles already prove that each role can reach its selected alias. The remaining gap appears when an OpenHands subagent is created from the orchestrator.

Without a model override, the implementer or auditor can inherit the orchestrator model. In this step, each subagent definition names its own saved LLM profile so the selected role binding controls its request.

In this step, get ready to:
  • Define a reusable subagent runner that loads a named profile.
  • Send direct requests through the chat profile and the orchestrator profile.
  • Run implementer and auditor subagents with profile overrides.
Create the profile-aware subagent runner

An AgentDefinition describes a subagent before OpenHands creates it. Its model value can name one of the role profiles saved in profiles/.

  • Switch back to the Visual Studio Code workspace from earlier.
  • Use the file creation control in the file sidebar to create demo.py beside role_router.py.
  • Add the import block below to demo.py:
from __future__ import annotations

from openhands.sdk import agent_definition_to_factory
from openhands.sdk.subagent import AgentDefinition

from role_router import (
    PROFILE_DIR,
    ask_llm,
    ask_role,
    build_profiles,
    load_routing_config,
    show_bindings,
)

What do these imports provide?

  • agent_definition_to_factory converts a subagent definition into a callable factory.
  • AgentDefinition holds the role description plus its profile selection.
  • The role_router imports reuse the routing configuration plus the saved profile directory.
  • Save demo.py.
  • Confirm that the imports load by running this command in PowerShell:
uv run python demo.py

What should I see?

PowerShell returns to its prompt without a traceback. This confirms that demo.py can import OpenHands plus your router helpers.

Do the imports fail?

  • Check that PowerShell is still inside the project folder containing pyproject.toml.
  • Compare every imported name with the block above.

Help me debug the OpenHands imports in demo.py.

The runner receives an orchestrator LLM as its parent. The named role profile determines the model loaded into the resulting agent.

  • Add the reusable runner below the import block in demo.py:
def run_subagent(
    role: str,
    description: str,
    system_prompt: str,
    parent_llm: object,
) -> None:
    definition = AgentDefinition(
        name=role,
        description=description,
        tools=[],
        system_prompt=system_prompt,
        model=role,
        profile_store_dir=str(PROFILE_DIR.resolve()),
    )
    factory = agent_definition_to_factory(definition)
    agent = factory(parent_llm)
    prompt = f"Act as the {role}. Report which provider and model handled this request."
    answer = ask_llm(agent.llm, prompt)
    print(f"{role:12} -> {agent.llm.model} -> {answer}")

How does the runner select a model?

  • model=role makes the role name double as the saved profile name.
  • profile_store_dir points OpenHands at the resolved profiles/ directory.
  • agent_definition_to_factory() prepares the definition for agent creation.
  • factory(parent_llm) provides the orchestrator LLM as the parent.
  • agent.llm exposes the profile OpenHands loaded for the subagent.
  • Save demo.py.
  • Confirm that Python can define the runner by executing the file again:
uv run python demo.py

What should I see now?

PowerShell returns to its prompt without a traceback. Your reusable subagent runner is now valid Python.

Is the runner invalid?

  • Check that every line inside run_subagent() uses consistent indentation.
  • Confirm that every opening parenthesis has a matching closing parenthesis.

Help me find the syntax or indentation problem in run_subagent().

Route the direct chat and orchestrator requests

The main flow rebuilds the role profiles from routing.json. Chat plus orchestration then use their saved profiles directly.

  • Add the main routing flow below run_subagent() in demo.py:
def main() -> None:
    config = load_routing_config()
    store = build_profiles(config)
    show_bindings(config)
    print("\nRouting transcript:")

    chat_llm = store.load("chat")
    chat_answer = ask_role(store, "chat", "Handle this user chat request.")
    print(f"{'chat':12} -> {chat_llm.model} -> {chat_answer}")

    orchestrator_llm = store.load("orchestrator")
    orchestrator_answer = ask_role(
        store,
        "orchestrator",
        "Plan how implementation and audit work should be delegated.",
    )
    print(
        f"{'orchestrator':12} -> {orchestrator_llm.model} -> "
        f"{orchestrator_answer}"
    )


if __name__ == "__main__":
    main()

How does the direct routing work?

  • load_routing_config() loads the validated model catalog plus its role bindings.
  • build_profiles() refreshes the four profile files from those bindings.
  • store.load("chat") exposes the model selected for the chat role.
  • ask_role() sends each direct request through its named saved profile.
  • main() runs when PowerShell executes demo.py.
  • Save demo.py.
  • Run the direct routing flow with this command:
uv run python demo.py

What should the first transcript show?

That first routing slice is live. You should see all four bindings followed by chat on litellm_proxy/chat-fast through Provider A.

The orchestrator line should show litellm_proxy/orchestrator-strong through Provider B.

Does the transcript stop before the answers?

  • Confirm that the mock provider PowerShell terminal from earlier is still running.
  • Confirm that the LiteLLM Gateway PowerShell terminal from earlier is still running.
  • Check that routing.json remains beside demo.py.

Help me diagnose why the direct routing transcript cannot reach the local providers.

Add the subagent profile overrides

The implementer and auditor both receive the orchestrator LLM as their parent. Their AgentDefinition.model values make OpenHands load the matching role profile.

  • Insert the implementer call inside main() immediately above the if __name__ == "__main__": line:
    run_subagent(
        role="implementer",
        description="Implements the requested code changes.",
        system_prompt="You are the implementation specialist.",
        parent_llm=orchestrator_llm,
    )

Why does the implementer override the parent?

The role value is implementer. The runner uses that value to load the saved implementer profile.

The parent_llm value still carries the orchestrator LLM into the factory. The named profile controls the resulting agent model.

  • Save demo.py.
  • Run the expanded transcript with this command:
uv run python demo.py

What should the implementer show?

You should now see an implementer line after the orchestrator. It prints litellm_proxy/implementer-coding before Provider A identifies itself.

Does the implementer use the wrong model?

  • Check that the call sets role="implementer".
  • Check that the runner sets model=role.
  • Confirm that the profiles/implementer file still exists.

Help me trace why the implementer subagent is loading the wrong profile.

  • Add the auditor call directly below the implementer call inside main():
    run_subagent(
        role="auditor",
        description="Independently audits the implementation.",
        system_prompt="You are an independent final auditor.",
        parent_llm=orchestrator_llm,
    )

How is the auditor kept independent?

The auditor definition names the saved auditor profile. That profile points to litellm_proxy/auditor-independent.

The agent factory receives the same orchestrator parent used for the implementer. The profile name gives the auditor its own model selection.

  • Save demo.py.

Before you run the complete demo, predict which provider each of the four roles will reach.

  • Run the end-to-end routing transcript with this command:
uv run python demo.py

What should the full transcript show?

  • The chat line prints litellm_proxy/chat-fast with a Provider A response.
  • The orchestrator line prints litellm_proxy/orchestrator-strong with a Provider B response.
  • The implementer line prints litellm_proxy/implementer-coding with a Provider A response.
  • The auditor line prints litellm_proxy/auditor-independent with a Provider B response.

Is a subagent inheriting the orchestrator model?

  • Confirm that each subagent call passes a role matching a profile filename.
  • Check that profile_store_dir uses str(PROFILE_DIR.resolve()).
  • Confirm that both calls pass orchestrator_llm as the parent.

Help me debug a subagent that inherits the parent model instead of loading its named profile.

✔️ Awesome, I've got everything!

Great work. Save demo.py and keep both local services running while you compare the complete transcript.

ⓧ I'd like to double check the full code

Compare your complete demo.py file with this reference:

from __future__ import annotations

from openhands.sdk import agent_definition_to_factory
from openhands.sdk.subagent import AgentDefinition

from role_router import (
    PROFILE_DIR,
    ask_llm,
    ask_role,
    build_profiles,
    load_routing_config,
    show_bindings,
)


def run_subagent(
    role: str,
    description: str,
    system_prompt: str,
    parent_llm: object,
) -> None:
    definition = AgentDefinition(
        name=role,
        description=description,
        tools=[],
        system_prompt=system_prompt,
        model=role,
        profile_store_dir=str(PROFILE_DIR.resolve()),
    )
    factory = agent_definition_to_factory(definition)
    agent = factory(parent_llm)
    prompt = f"Act as the {role}. Report which provider and model handled this request."
    answer = ask_llm(agent.llm, prompt)
    print(f"{role:12} -> {agent.llm.model} -> {answer}")


def main() -> None:
    config = load_routing_config()
    store = build_profiles(config)
    show_bindings(config)
    print("\nRouting transcript:")

    chat_llm = store.load("chat")
    chat_answer = ask_role(store, "chat", "Handle this user chat request.")
    print(f"{'chat':12} -> {chat_llm.model} -> {chat_answer}")

    orchestrator_llm = store.load("orchestrator")
    orchestrator_answer = ask_role(
        store,
        "orchestrator",
        "Plan how implementation and audit work should be delegated.",
    )
    print(
        f"{'orchestrator':12} -> {orchestrator_llm.model} -> "
        f"{orchestrator_answer}"
    )

    run_subagent(
        role="implementer",
        description="Implements the requested code changes.",
        system_prompt="You are the implementation specialist.",
        parent_llm=orchestrator_llm,
    )
    run_subagent(
        role="auditor",
        description="Independently audits the implementation.",
        system_prompt="You are an independent final auditor.",
        parent_llm=orchestrator_llm,
    )


if __name__ == "__main__":
    main()

How should I use this reference?

Compare the imports first. Check the two function definitions next.

Pay close attention to indentation inside main(). Both run_subagent() calls belong inside that function.

You did it. Each role now reaches its selected provider through a saved profile.

Secret mission

Enforce Independent Auditing

A binding mistake can place implementation and final auditing on the same provider. Add a policy that rejects this conflict before any OpenHands profiles are built.

Clean Up Your Resources

Clean Up Your Resources

Your local mock-inference path has no ongoing provider charges. Choose whether to keep the project available, pause its processes, or remove its resources.

Resources you used:

  • The mock provider process serving Provider A plus Provider B.
  • The local LiteLLM Gateway process on port 4000.
  • The Windows project folder containing demo.py, role_router.py, routing.json, gateway.yaml, mock_providers.py, pyproject.toml, the virtual environment, plus the profiles/ directory.
  • The shared package cache populated by uv during dependency installation.

Keep everything running

Keep the complete project so you can continue experimenting with role bindings. Stop the temporary services while they are idle so their local ports remain free.

  • Switch back to the PowerShell terminal running mock_providers.py.
  • Press Ctrl+C to stop both mock inference APIs.
  • Switch back to the PowerShell terminal running the LiteLLM Gateway.
  • Press Ctrl+C to stop the gateway.
  • Leave the complete project folder in place.

Your routing catalog, independent-auditor policy, saved OpenHands profiles, plus application files remain ready for your next session.

Pause - I'll come back to this later

Pausing keeps every project file while freeing ports 8001, 8002, plus 4000. The same two commands restart the control plane later.

  • Switch back to the PowerShell terminal running mock_providers.py.
  • Press Ctrl+C to stop both mock inference APIs.
  • Switch back to the PowerShell terminal running the LiteLLM Gateway.
  • Press Ctrl+C to stop the gateway.

When you return, use the project folder from earlier in two separate PowerShell terminals.

  • Restart both mock providers in the first PowerShell terminal by running:
uv run python mock_providers.py

What does this command start?

The script starts both local mock servers in one PowerShell process. Keeping that terminal active makes Provider A plus Provider B reachable.

  • Confirm the first terminal shows Provider A listening.
  • Confirm the same terminal shows Provider B listening.

Mock providers do not restart?

  • Confirm the first PowerShell terminal is using the project folder that contains mock_providers.py.
  • Ask for help with the startup failure.
  • Restart the gateway in the second PowerShell terminal by running:
uv run litellm --config gateway.yaml --port 4000

What does this command start?

The gateway loads the four aliases from gateway.yaml. It listens on local port 4000 for OpenHands requests.

  • Confirm the second terminal remains occupied by the running gateway.

Gateway does not restart?

  • Confirm the second PowerShell terminal is using the project folder that contains gateway.yaml.
  • Ask for help with the gateway startup failure.

Delete - I don't want to use this again

Deleting the project removes your routing catalog, policy code, saved profiles, plus virtual environment. Your external provider accounts remain untouched because the default project used local mock APIs.

  • Switch back to the PowerShell terminal running mock_providers.py.
  • Press Ctrl+C to stop both mock inference APIs.
  • Switch back to the PowerShell terminal running the LiteLLM Gateway.
  • Press Ctrl+C to stop the gateway.
  • Clear the shared uv package cache from either stopped PowerShell terminal by running this optional command:
uv cache clean

What does this command remove?

The command clears packages stored in uv's shared cache. Deleting the project folder remains a separate step.

  • Confirm PowerShell reports that cached files were removed.

Cache cleanup does not finish?

  • Confirm the PowerShell terminal can still run uv commands.
  • Ask for help with the cache cleanup failure.
  • Close both stopped PowerShell terminals.
  • Close Visual Studio Code.
  • Locate the Windows project folder that contains demo.py in File Explorer.
  • Select that project folder.
  • Use File Explorer to delete the selected project folder from the computer.
  • Confirm the folder no longer appears in File Explorer.

That closes the loop. Your mock services, routing profiles, project files, plus cached packages have been removed.

Nice Work!

Nice Work!

You did it! Your local model control plane now routes four OpenHands roles through stable aliases exposed by LiteLLM Gateway. It also rejects an auditor that shares the implementer's provider.

What you learned:

  • Run two OpenAI-compatible inference APIs that reveal which provider handles each request.
  • Expose four stable model aliases through one LiteLLM Gateway. Each role now has a provider-neutral target.
  • Create reusable LLM profiles for all four roles. Named subagents now override the parent orchestrator model.
  • Secret Mission: Add an independent-auditor policy that rejects any binding where the implementer and auditor share one provider.

Ready to quiz yourself?