Build a Reliable Local AI Gateway

Route local AI requests through retries and fallback with LiteLLM.

Introduction

30 Second Summary

AI features can feel dependable until the service behind them stops answering. Without a backup path, one outage leaves every request unanswered.

In this project, you will build a local reliability lab around a LiteLLM Gateway. One OpenAI-compatible API request will reach a healthy backup after the primary provider fails.

What You'll Build

Your finished lab will send one chat request through a gateway that recovers from a planned provider outage without changing the request.

By the end of this project, you'll have:

  • A healthy backup provider that returns a visible chat response containing backup-provider handled.
  • A deliberate HTTP 503 failure visible in the client output. Docker Compose logs will identify the failing primary provider.
  • An ordered fallback that leaves the failing primary model group before returning a successful response from the backup.
  • Secret Mission: Replace the explicit outage with a slow primary provider. Prove that a one-second timeout still sends the request to the backup.

What do I need for this lab?

You need Docker Desktop plus Visual Studio Code already working on your Windows computer.

Python runs inside Linux containers. You do not need a local Python installation, cloud account, model download, or API key.

Before We Start

This checkpoint defines the reliability behavior you are about to prove before you build the local lab. The target is a successful OpenAI-compatible API response from the backup provider after LiteLLM applies its retry policy to the unavailable primary provider while the client request stays unchanged.

Set Up the Windows Workspace

A reliability test is easier to trust when every file lives in one known workspace. That boundary keeps machine-specific Python versions out of the gateway behavior.

In this step, you will create the litellm-reliability-lab workspace in Visual Studio Code. You will use its PowerShell terminal to confirm Docker Compose from Docker Desktop is ready.

In this step, get ready to:
  • Create the litellm-reliability-lab workspace in Visual Studio Code.
  • Add the provider folder with two empty configuration files.
  • Verify Docker Compose from the integrated PowerShell terminal.
Create the project workspace

Visual Studio Code treats an opened folder as a workspace. Its Explorer and terminal share that project location.

  • Press the Windows key to open the Windows search bar.
  • Type Visual Studio Code into the search bar.
  • Press Enter to launch Visual Studio Code.

You will see the Visual Studio Code window. The folder picker creates the workspace on your Desktop.

The folder picker is a little fiddly because you create the new folder inside the same dialog used to open it.

  • Select File in the top menu bar.
  • Select Open Folder....
  • Choose your Desktop as the folder location.
  • Select New Folder.
  • Enter litellm-reliability-lab as the folder name.
  • Select Select Folder.
  • Select Yes, I trust the authors if the Workspace Trust dialog appears.

Your Explorer now shows litellm-reliability-lab as the open workspace.

Folder name missing from Explorer?

  • Select File in the top menu bar.
  • Select Open Folder....
  • Choose the litellm-reliability-lab folder on your Desktop.
  • Select Select Folder.

Help me open my project folder as a Visual Studio Code workspace on Windows.

Create the empty project structure

The provider folder will hold the simulator code in the next step. Two empty YAML files reserve stable paths for the Compose stack and gateway configuration.

  • Select the Explorer view in the left Activity Bar.
  • Select the litellm-reliability-lab folder in the Explorer sidebar.
  • Select New Folder in the Explorer toolbar.
  • Enter provider as the folder name.
  • Press Enter to create the folder.

You will see provider directly inside litellm-reliability-lab.

  • Select the litellm-reliability-lab folder in the Explorer sidebar.
  • Select New File... in the Explorer toolbar.
  • Enter compose.yaml as the file name.
  • Press Enter to create the empty file.

A blank compose.yaml editor opens. The file also appears beside the provider folder.

  • Select the litellm-reliability-lab folder in the Explorer sidebar.
  • Select New File... in the Explorer toolbar.
  • Enter litellm-config.yaml as the file name.
  • Press Enter to create the empty file.

A blank litellm-config.yaml editor opens. Your project skeleton now has every path needed for the next step.

✔️ Awesome, I've got everything!

Good progress. Your Explorer shows provider, compose.yaml, and litellm-config.yaml inside litellm-reliability-lab.

ⓧ I'd like to double check the full code

At this checkpoint, compose.yaml has no content.

At this checkpoint, litellm-config.yaml also has no content.

These two blank files are the complete cumulative code for this step.

Check the container toolchain

The integrated terminal starts inside the open workspace. The version check confirms that the Compose command included with Docker Desktop is available.

  • Select View in the top menu bar.
  • Select Terminal.
  • Confirm the terminal tab shows PowerShell as the active shell.

Before you run the check, what do you expect the terminal to identify?

  • Confirm that Docker Compose is available by running this command:
docker compose version

What does this command check?

This command asks Docker Compose to print its installed version information. It does not create or start any containers.

You will see a line identifying Docker Compose with its installed version. That confirms your Windows container toolchain is ready for the lab.

Docker Compose version missing?

  • Choose PowerShell from the terminal shell selector if another shell is active.
  • Press the Windows key to open the Windows search bar.
  • Type Docker Desktop into the search bar.
  • Press Enter to launch Docker Desktop.
  • Wait for Docker Desktop to finish starting.
  • Return to the Visual Studio Code terminal.
  • Repeat the version check above.

Help me diagnose why Docker Compose does not print version information in my Visual Studio Code PowerShell terminal.

Your workspace is ready. Next up, you will build two local provider simulators that produce a visible backup response.

Build the Provider Simulators

Your verified Visual Studio Code workspace gives this lab a reliable base. You can now focus on provider behavior without local shell differences getting in the way.

Before the gateway can recover from a failure, you need a visible working response from an OpenAI-compatible API. You will build two deterministic providers with Docker Compose. The backup provider gives you the first successful response.

In this step, get ready to:
  • Create the provider server plus its matching client.
  • Define the primary service plus the backup service.
  • Build both providers before sending a request to the backup.
Create the provider files

The provider image contains a small Python HTTP server plus a reusable request client. Running both inside Docker keeps the lab independent from any Python installation on Windows.

  • Select the existing provider folder in the Visual Studio Code file sidebar.
  • Create a file named Dockerfile inside the provider folder.
  • Define the provider image by pasting this code into provider/Dockerfile:
FROM python:3.14.10-slim

WORKDIR /app
COPY server.py client.py ./

CMD ["python", "server.py"]

What does this image contain?

  • The base image supplies Python 3.14.10 inside a small Linux container.
  • The working directory places the application files under /app.
  • The copy instruction gives the image both scripts.
  • The final instruction starts server.py whenever a provider container launches.
  • Save provider/Dockerfile.
  • Confirm the file ends with the instruction that starts server.py.

Image file look different?

Check that the file is named exactly Dockerfile without a file extension. Confirm that it sits inside the existing provider folder.

Ask for help if the image file is not appearing in the right place. Help me check my provider Dockerfile location in Visual Studio Code.

The server exposes a health route plus a chat-completions route. Its environment values let the same image behave as either a failing primary provider or a healthy backup provider.

  • Select the existing provider folder in the file sidebar.
  • Create a file named server.py inside that folder.
  • Add the provider server by pasting this code into provider/server.py:
import json
import os
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

PROVIDER_NAME = os.getenv("PROVIDER_NAME", "simulated-provider")
PROVIDER_MODE = os.getenv("PROVIDER_MODE", "success")
PORT = int(os.getenv("PORT", "8000"))


class ProviderHandler(BaseHTTPRequestHandler):
    def send_json(self, status_code, payload):
        body = json.dumps(payload).encode("utf-8")
        self.send_response(status_code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        try:
            self.wfile.write(body)
        except BrokenPipeError:
            print(f"{PROVIDER_NAME}: client disconnected before the response", flush=True)

    def do_GET(self):
        if self.path == "/health":
            self.send_json(200, {"status": "ok", "provider": PROVIDER_NAME})
            return
        self.send_json(404, {"error": "not_found"})

    def do_POST(self):
        if self.path != "/v1/chat/completions":
            self.send_json(404, {"error": "not_found"})
            return

        content_length = int(self.headers.get("Content-Length", "0"))
        request_body = json.loads(self.rfile.read(content_length) or b"{}")
        requested_model = request_body.get("model", "unknown")
        print(
            f"{PROVIDER_NAME}: request for {requested_model} in {PROVIDER_MODE} mode",
            flush=True,
        )

        if PROVIDER_MODE == "fail":
            self.send_json(
                503,
                {
                    "error": {
                        "message": f"{PROVIDER_NAME} is deliberately unavailable",
                        "type": "server_error",
                        "code": "simulated_outage",
                    }
                },
            )
            return

        messages = request_body.get("messages", [])
        prompt = next(
            (
                str(message.get("content", ""))
                for message in reversed(messages)
                if message.get("role") == "user"
            ),
            "",
        )
        created = int(time.time())
        self.send_json(
            200,
            {
                "id": f"chatcmpl-{PROVIDER_NAME}-{created}",
                "object": "chat.completion",
                "created": created,
                "model": requested_model,
                "choices": [
                    {
                        "index": 0,
                        "message": {
                            "role": "assistant",
                            "content": f"{PROVIDER_NAME} handled: {prompt}",
                        },
                        "finish_reason": "stop",
                    }
                ],
                "usage": {
                    "prompt_tokens": 0,
                    "completion_tokens": 0,
                    "total_tokens": 0,
                },
            },
        )

    def log_message(self, format_string, *args):
        print(f"{PROVIDER_NAME}: {format_string % args}", flush=True)


if __name__ == "__main__":
    print(f"{PROVIDER_NAME}: listening on port {PORT} in {PROVIDER_MODE} mode", flush=True)
    ThreadingHTTPServer(("0.0.0.0", PORT), ProviderHandler).serve_forever()

How does the provider behave?

  • The environment values control the provider name plus its deterministic mode.
  • The /health route gives Docker a lightweight readiness signal.
  • The /v1/chat/completions route accepts the same payload shape used by an OpenAI-compatible client.
  • The failing mode returns an HTTP 503 response with the project-defined simulated_outage code.
  • The successful mode returns the provider name in the assistant message.
  • Save provider/server.py.
  • Confirm the file begins with the four import lines.
  • Confirm the file ends by starting ThreadingHTTPServer on PORT.

Server file showing syntax warnings?

Confirm that every indented method remains inside ProviderHandler. Check that the final server startup lines begin at the left edge.

Use this guided check if Python indentation is difficult to compare. Help me compare the indentation in provider/server.py.

The client sends one fixed chat request to whichever endpoint you provide. It also prints error bodies so later steps can make provider failures visible.

  • Select the existing provider folder in the file sidebar.
  • Create a file named client.py inside that folder.
  • Add the request client by pasting this code into provider/client.py:
import json
import sys
from urllib.error import HTTPError
from urllib.request import Request, urlopen

if len(sys.argv) != 3:
    print("Usage: python client.py ENDPOINT MODEL")
    sys.exit(2)

endpoint = sys.argv[1]
model = sys.argv[2]
payload = {
    "model": model,
    "messages": [
        {
            "role": "user",
            "content": "Explain why provider fallback matters in one sentence.",
        }
    ],
}
request = Request(
    endpoint,
    data=json.dumps(payload).encode("utf-8"),
    headers={"Content-Type": "application/json"},
    method="POST",
)

try:
    with urlopen(request, timeout=15) as response:
        result = json.loads(response.read())
        print(json.dumps(result, indent=2))
except HTTPError as error:
    error_body = error.read().decode("utf-8")
    print(f"HTTP {error.code}")
    print(error_body)
    sys.exit(1)

How does the client work?

  • The first argument supplies the request endpoint.
  • The second argument supplies the model name.
  • The payload keeps one fixed user message for every reliability test.
  • The success path prints formatted JSON from the selected provider.
  • The error path prints the HTTP status plus the response body before exiting with a nonzero status.
  • Save provider/client.py.
  • Confirm the file contains the fixed prompt about provider fallback.
  • Confirm the final line exits with status 1 after an HTTP error.

Client file showing syntax warnings?

Check the braces around payload first. Confirm that the except block lines up with try.

Use this comparison prompt if a bracket or indent still looks wrong. Help me find a syntax mismatch in provider/client.py.

✔️ Awesome, I've got everything!

Great. Save all three files before you define the services that use them.

ⓧ I'd like to double check the full code

Compare each file with the complete versions below.

FROM python:3.14.10-slim

WORKDIR /app
COPY server.py client.py ./

CMD ["python", "server.py"]

What should match?

Your provider/Dockerfile should match this five-instruction image definition exactly.

import json
import os
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

PROVIDER_NAME = os.getenv("PROVIDER_NAME", "simulated-provider")
PROVIDER_MODE = os.getenv("PROVIDER_MODE", "success")
PORT = int(os.getenv("PORT", "8000"))


class ProviderHandler(BaseHTTPRequestHandler):
    def send_json(self, status_code, payload):
        body = json.dumps(payload).encode("utf-8")
        self.send_response(status_code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        try:
            self.wfile.write(body)
        except BrokenPipeError:
            print(f"{PROVIDER_NAME}: client disconnected before the response", flush=True)

    def do_GET(self):
        if self.path == "/health":
            self.send_json(200, {"status": "ok", "provider": PROVIDER_NAME})
            return
        self.send_json(404, {"error": "not_found"})

    def do_POST(self):
        if self.path != "/v1/chat/completions":
            self.send_json(404, {"error": "not_found"})
            return

        content_length = int(self.headers.get("Content-Length", "0"))
        request_body = json.loads(self.rfile.read(content_length) or b"{}")
        requested_model = request_body.get("model", "unknown")
        print(
            f"{PROVIDER_NAME}: request for {requested_model} in {PROVIDER_MODE} mode",
            flush=True,
        )

        if PROVIDER_MODE == "fail":
            self.send_json(
                503,
                {
                    "error": {
                        "message": f"{PROVIDER_NAME} is deliberately unavailable",
                        "type": "server_error",
                        "code": "simulated_outage",
                    }
                },
            )
            return

        messages = request_body.get("messages", [])
        prompt = next(
            (
                str(message.get("content", ""))
                for message in reversed(messages)
                if message.get("role") == "user"
            ),
            "",
        )
        created = int(time.time())
        self.send_json(
            200,
            {
                "id": f"chatcmpl-{PROVIDER_NAME}-{created}",
                "object": "chat.completion",
                "created": created,
                "model": requested_model,
                "choices": [
                    {
                        "index": 0,
                        "message": {
                            "role": "assistant",
                            "content": f"{PROVIDER_NAME} handled: {prompt}",
                        },
                        "finish_reason": "stop",
                    }
                ],
                "usage": {
                    "prompt_tokens": 0,
                    "completion_tokens": 0,
                    "total_tokens": 0,
                },
            },
        )

    def log_message(self, format_string, *args):
        print(f"{PROVIDER_NAME}: {format_string % args}", flush=True)


if __name__ == "__main__":
    print(f"{PROVIDER_NAME}: listening on port {PORT} in {PROVIDER_MODE} mode", flush=True)
    ThreadingHTTPServer(("0.0.0.0", PORT), ProviderHandler).serve_forever()

What should match?

Your provider/server.py should contain the health handler plus both deterministic chat outcomes exactly as shown.

import json
import sys
from urllib.error import HTTPError
from urllib.request import Request, urlopen

if len(sys.argv) != 3:
    print("Usage: python client.py ENDPOINT MODEL")
    sys.exit(2)

endpoint = sys.argv[1]
model = sys.argv[2]
payload = {
    "model": model,
    "messages": [
        {
            "role": "user",
            "content": "Explain why provider fallback matters in one sentence.",
        }
    ],
}
request = Request(
    endpoint,
    data=json.dumps(payload).encode("utf-8"),
    headers={"Content-Type": "application/json"},
    method="POST",
)

try:
    with urlopen(request, timeout=15) as response:
        result = json.loads(response.read())
        print(json.dumps(result, indent=2))
except HTTPError as error:
    error_body = error.read().decode("utf-8")
    print(f"HTTP {error.code}")
    print(error_body)
    sys.exit(1)

What should match?

Your provider/client.py should contain the fixed payload plus both response paths exactly as shown.

Define the provider services

Each service builds from the same provider image. Environment values turn one container into the failing primary while the other becomes the healthy backup.

The client uses the same image with a different entry point. Its tools profile keeps the client container on demand until you run a request.

  • Select the existing compose.yaml file in the project file sidebar.
  • Replace its empty contents with this service configuration:
services:
  primary:
    build:
      context: ./provider
    environment:
      PROVIDER_NAME: primary-provider
      PROVIDER_MODE: fail
    healthcheck:
      test:
        - CMD
        - python
        - -c
        - "import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=2).close()"
      interval: 5s
      timeout: 3s
      retries: 10
      start_period: 5s

  backup:
    build:
      context: ./provider
    environment:
      PROVIDER_NAME: backup-provider
      PROVIDER_MODE: success
    healthcheck:
      test:
        - CMD
        - python
        - -c
        - "import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=2).close()"
      interval: 5s
      timeout: 3s
      retries: 10
      start_period: 5s

  client:
    build:
      context: ./provider
    entrypoint: ["python", "/app/client.py"]
    profiles: ["tools"]

How do these services fit together?

  • The primary service starts in fail mode for the planned outage.
  • The backup service starts in success mode for a deterministic response.
  • Each health check calls the provider's local /health route inside its container.
  • The client service starts client.py only when you run it.
  • Compose service discovery lets the client reach a provider through its service name.
  • Save compose.yaml.
  • Confirm the file lists primary as the first service.
  • Confirm the file lists client as the final service.

Compose file showing indentation warnings?

Use spaces consistently throughout compose.yaml. Confirm that all three service names line up beneath services.

Use this guided check if the nested health checks look misaligned. Help me check my compose.yaml indentation.

✔️ Awesome, I've got everything!

Your initial Compose file is ready. Keep litellm-config.yaml empty for now.

ⓧ I'd like to double check the full code

services:
  primary:
    build:
      context: ./provider
    environment:
      PROVIDER_NAME: primary-provider
      PROVIDER_MODE: fail
    healthcheck:
      test:
        - CMD
        - python
        - -c
        - "import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=2).close()"
      interval: 5s
      timeout: 3s
      retries: 10
      start_period: 5s

  backup:
    build:
      context: ./provider
    environment:
      PROVIDER_NAME: backup-provider
      PROVIDER_MODE: success
    healthcheck:
      test:
        - CMD
        - python
        - -c
        - "import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=2).close()"
      interval: 5s
      timeout: 3s
      retries: 10
      start_period: 5s

  client:
    build:
      context: ./provider
    entrypoint: ["python", "/app/client.py"]
    profiles: ["tools"]

What should match?

This is the complete initial compose.yaml file. It contains the two providers plus the on-demand client.

Build the providers and test the backup

The configuration now has enough information to build both provider containers. Starting them creates the shared Compose network that makes service-name URLs work.

  • Switch back to the integrated PowerShell terminal from the previous step.
  • Build the image plus start both provider services by running this command:
docker compose up --build -d primary backup

What does this command do?

Docker Compose builds the provider image from provider/Dockerfile. Detached mode leaves both provider containers running in the background.

The shared network lets containers reach primary or backup by service name.

The first build downloads the pinned Python image if it is missing locally. You will see build progress before both provider services report that they started.

Provider build did not complete?

Confirm Docker Desktop is still running. Check that the terminal remains inside the open litellm-reliability-lab folder.

Ask for a targeted review if the build points to a file or configuration problem. Help me diagnose my provider container build.

The backup provider is your first end-to-end proof. The client reaches it through the internal service name before printing the JSON response.

  • Prepare to send one chat request to the healthy backup by using the command below.

Before you run the check, which provider name do you expect to see inside the assistant response?

docker compose run --rm client http://backup:8000/v1/chat/completions simulated-backup

What does this request prove?

  • The one-off client container joins the same Compose network as the providers.
  • The hostname backup resolves to the healthy provider service.
  • The request uses the project model name simulated-backup.
  • The client container is removed after the request finishes.

You should see formatted JSON containing backup-provider handled in the assistant message. That is your first complete reliability-lab response from a containerized provider.

Backup response missing?

Confirm the request URL uses the service name backup with container port 8000. Check that PROVIDER_MODE is set to success for the backup service.

Use this prompt if the terminal shows a connection or response problem. Help me debug the missing backup-provider response.

Your provider simulators are live. Next, you will call the primary directly so its planned HTTP 503 failure exposes the weakness of direct routing.

Prove Direct Routing Is Fragile

Your healthy backup response proves that requests can cross the Docker Compose network.

A client fixed to one provider has no recovery path when that provider fails. This step makes that weakness visible through a deliberate outage.

In this step, get ready to:
  • Trigger the primary provider's planned HTTP 503 response.
  • Trace the failed request in the primary provider logs.
Call the primary provider directly

The client has a single route to the primary provider. An HTTP 503 response ends that request because no fallback destination exists.

Before you run the request, do you expect a direct client to recover when its only provider returns an error?

  • Send the client request from the PowerShell terminal by running this command:
docker compose run --rm client http://primary:8000/v1/chat/completions simulated-primary

What does this command do?

  • The docker compose run --rm portion starts an on-demand client container. It removes that container after the command finishes.
  • The http://primary:8000/v1/chat/completions endpoint sends the request to the primary service through the Compose network.
  • The simulated-primary value becomes the model name in the fixed chat payload.

You will see HTTP 503 followed by JSON containing simulated_outage. The client then exits with a nonzero status.

This Failure Is Intentional

The primary provider is configured in fail mode. Every chat request receives the planned outage.

The client prints the response body before exiting with a nonzero status. This is the fragility you needed to expose before adding a recovery policy.

That expected failure gives you a clear baseline for the recovery test ahead.

Seeing a Different Result?

  • Confirm Docker Desktop is still running.
  • Check that PROVIDER_MODE: fail remains under primary in compose.yaml.
  • Compare your endpoint with the command shown above.

Help me diagnose an unexpected result from the direct primary provider request.

Inspect the primary provider logs

The client output shows the failure from the requester's side. The provider log reveals which service produced it.

Before you inspect the logs, which provider name and mode do you expect the request line to show?

  • Display the primary service logs by running this command:
docker compose logs primary

What Does This Command Show?

  • The docker compose logs primary command reads the captured output from the primary service.
  • The request line identifies the provider. It also records the requested model plus the configured mode.

You will see a line naming primary-provider. The same line reports a request for simulated-primary in fail mode.

You now have both halves of the proof: a failed client response plus a provider log naming the source.

Missing the Fail-Mode Request?

  • Confirm the PowerShell terminal is still inside litellm-reliability-lab.
  • Repeat the direct client request from the previous substep.
  • Run the primary log command again.

Help me find the missing primary provider request in my Docker Compose logs.

The weak point is now visible. Next, you will place LiteLLM in front of both providers so its routing policy can recover from the same failure.

Add LiteLLM Retries and Fallback

Your provider lab has exposed the weakness in direct routing. The client stopped at the planned HTTP 503 because no routing policy could move the request elsewhere.

You will now place a LiteLLM Gateway between the client and both providers. The gateway retries the primary model group before an ordered fallback sends the request to the backup group.

In this step, get ready to:
  • Define the primary and backup model groups.
  • Configure the retry policy plus ordered fallback.
  • Start the gateway to prove recovery through the backup.
Configure the model groups and routing policy

A model group gives the client one stable name while LiteLLM handles the provider behind it. The router policy decides how long to wait before leaving a failing group.

Why use a gateway?

A client-side retry loop would spread recovery rules across every application that calls a model. LiteLLM keeps those rules in one gateway while the client keeps one request shape.

Retries stay inside the failed model group. Fallbacks leave that group for the next configured model group.

This local lab permits LiteLLM to start without a safe master key. The gateway port remains bound to your own computer in the next configuration change.

Keep This Setting Local

The dangerously_permit_weak_or_unset_master_key setting is intended for isolated local development. Do not carry it into a shared gateway or production deployment.

  • Select litellm-config.yaml in the Visual Studio Code Explorer sidebar.
  • Fill the empty file by pasting this configuration:
model_list:
  - model_name: primary-chat
    litellm_params:
      model: openai/simulated-primary
      api_base: http://primary:8000/v1
      api_key: none
  - model_name: backup-chat
    litellm_params:
      model: openai/simulated-backup
      api_base: http://backup:8000/v1
      api_key: none

router_settings:
  num_retries: 1
  timeout: 5
  fallbacks:
    - primary-chat:
        - backup-chat

general_settings:
  dangerously_permit_weak_or_unset_master_key: true

What does this configuration do?

  • The model_list section maps primary-chat to the primary provider.
  • The same section maps backup-chat to the backup provider.
  • Each api_base uses a Compose service name to reach its provider.
  • The router retries primary-chat once inside the same model group.
  • The fallback mapping sends unresolved failures to backup-chat.
  • The timeout value gives each request five seconds before the router treats it as timed out.
  • Save litellm-config.yaml.
  • Confirm the file ends with dangerously_permit_weak_or_unset_master_key: true.

Configuration indentation looks uneven?

YAML uses spaces to show which values belong together. Compare the indentation beneath litellm_params and fallbacks with the complete reference below.

Help me check my LiteLLM YAML indentation.

✔️ Awesome, I've got everything!

Your LiteLLM routing configuration is saved. The gateway can now load both model groups plus the recovery policy.

ⓧ I'd like to double check the full code

model_list:
  - model_name: primary-chat
    litellm_params:
      model: openai/simulated-primary
      api_base: http://primary:8000/v1
      api_key: none
  - model_name: backup-chat
    litellm_params:
      model: openai/simulated-backup
      api_base: http://backup:8000/v1
      api_key: none

router_settings:
  num_retries: 1
  timeout: 5
  fallbacks:
    - primary-chat:
        - backup-chat

general_settings:
  dangerously_permit_weak_or_unset_master_key: true

How to use this reference

Compare this complete file with your saved litellm-config.yaml. Match every value plus each indentation level.

Add the gateway service

Services in the same Docker Compose project can reach each other through their service names. The gateway uses primary plus backup without publishing either provider to your Windows host.

The gateway publishes port 4000 only on 127.0.0.1. Its readiness check prevents dependent clients from starting before LiteLLM can accept requests.

  • Select compose.yaml in the Visual Studio Code Explorer sidebar.
  • Find the end of the backup service.
  • Insert this gateway service immediately before the existing client service:
  gateway:
    image: ghcr.io/berriai/litellm:v1.108.0
    command: ["--config", "/app/config.yaml"]
    volumes:
      - ./litellm-config.yaml:/app/config.yaml:ro
    ports:
      - "127.0.0.1:4000:4000"
    depends_on:
      primary:
        condition: service_healthy
      backup:
        condition: service_healthy
    healthcheck:
      test:
        - CMD
        - python
        - -c
        - "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/readiness', timeout=2).close()"
      interval: 5s
      timeout: 3s
      retries: 20
      start_period: 20s

What does the gateway service do?

  • The image line pins LiteLLM Gateway to v1.108.0.
  • The command loads the mounted configuration from /app/config.yaml.
  • The read-only volume protects the mounted configuration from container changes.
  • The port binding limits host access to your own computer.
  • The dependency conditions wait for both providers to become healthy.
  • The readiness check proves LiteLLM can accept traffic on port 4000.
  • Save compose.yaml.
  • Start the gateway from the existing PowerShell terminal by running this command:
docker compose up -d --wait gateway

What should this command prove?

Docker Compose starts the gateway after both providers pass their health checks. The command returns after the gateway reaches its ready state.

The first gateway start may pause while Docker downloads the pinned image. Once the command returns, your gateway is healthy on the Compose network.

Gateway does not become healthy?

Check that litellm-config.yaml is saved beside compose.yaml. A misplaced file prevents the read-only mount from supplying the gateway configuration.

Help me diagnose an unhealthy LiteLLM gateway container.

The on-demand client also needs to wait for this readiness check. Its existing service definition currently has no gateway dependency.

  • Find this existing client block in compose.yaml:
  client:
    build:
      context: ./provider
    entrypoint: ["python", "/app/client.py"]
    profiles: ["tools"]

What is missing from this block?

The client can already run as an on-demand tool. It does not currently wait for the gateway readiness check.

  • Replace the existing block with this final version:
  client:
    build:
      context: ./provider
    entrypoint: ["python", "/app/client.py"]
    profiles: ["tools"]
    depends_on:
      gateway:
        condition: service_healthy

What changed in the client service?

The depends_on condition makes the client wait for a healthy gateway. This removes a startup race from the test path.

  • Save compose.yaml.
  • Confirm the final line reads condition: service_healthy.

✔️ Awesome, I've got everything!

Your Compose file now defines the two providers, the gateway, plus the on-demand client dependency.

ⓧ I'd like to double check the full code

services:
  primary:
    build:
      context: ./provider
    environment:
      PROVIDER_NAME: primary-provider
      PROVIDER_MODE: fail
    healthcheck:
      test:
        - CMD
        - python
        - -c
        - "import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=2).close()"
      interval: 5s
      timeout: 3s
      retries: 10
      start_period: 5s

  backup:
    build:
      context: ./provider
    environment:
      PROVIDER_NAME: backup-provider
      PROVIDER_MODE: success
    healthcheck:
      test:
        - CMD
        - python
        - -c
        - "import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=2).close()"
      interval: 5s
      timeout: 3s
      retries: 10
      start_period: 5s

  gateway:
    image: ghcr.io/berriai/litellm:v1.108.0
    command: ["--config", "/app/config.yaml"]
    volumes:
      - ./litellm-config.yaml:/app/config.yaml:ro
    ports:
      - "127.0.0.1:4000:4000"
    depends_on:
      primary:
        condition: service_healthy
      backup:
        condition: service_healthy
    healthcheck:
      test:
        - CMD
        - python
        - -c
        - "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/readiness', timeout=2).close()"
      interval: 5s
      timeout: 3s
      retries: 20
      start_period: 20s

  client:
    build:
      context: ./provider
    entrypoint: ["python", "/app/client.py"]
    profiles: ["tools"]
    depends_on:
      gateway:
        condition: service_healthy

How to use this reference

Compare this complete file with your saved compose.yaml. Check that gateway appears between backup and client.

Prove the fallback path

The client still sends the same OpenAI-compatible API payload that failed during direct routing. Only the endpoint plus model group change for this gateway test.

Before you run the request, do you expect the client to receive another HTTP 503 or a valid response? Which provider should answer?

  • Send the request through the gateway from the existing PowerShell terminal by running:
docker compose run --rm client http://gateway:4000/v1/chat/completions primary-chat

What should you see?

The client should print a successful JSON response containing backup-provider handled. The client receives a valid answer even though it requested primary-chat.

LiteLLM first retries inside the primary model group. The ordered fallback then sends the unresolved request to backup-chat.

That is the recovery path working. Your client now receives a valid chat response while the primary provider remains in fail mode.

Request does not reach the backup?

Check that the request uses primary-chat as its model. Also confirm the fallback target in litellm-config.yaml is backup-chat.

Help me trace why my LiteLLM fallback request is not returning the backup response.

The response proves recovery from the client's perspective. The provider logs reveal the order behind that result.

  • Trace the primary failure plus backup response by running:
docker compose logs primary backup gateway

What do the logs prove?

You should see primary-provider handle the request in fail mode before backup-provider handles the fallback. The primary line can appear more than once because num_retries is set to 1.

The gateway logs sit beside both provider histories. Together they show one request crossing the planned failure boundary.

Logs do not show both providers?

Run the gateway client request once more before reading the logs. Confirm that the log command names primary, backup, plus gateway.

Help me understand my provider and gateway logs.

That closes the loop. Your client kept one payload while the gateway handled provider recovery.

Secret mission

Fail Over on a Slow Provider

Replace the explicit provider outage with a three-second delay. Give the gateway one second to wait, then prove the same backup path recovers from a timeout without changing the client request.

Clean Up Your Resources

Clean Up Your Resources

Your reliability lab runs entirely on your Windows machine, so it costs $0 to leave the files in place. Choose whether to keep the containers active, pause them for later, or remove the running lab while keeping the project files.

Resources you used:

  • Two local provider containers named primary and backup.
  • One LiteLLM Gateway container named gateway.
  • A Docker Compose network connecting the provider and gateway services.

The on-demand client container has already been removed after each request, so there is nothing to tear down for it.

Keep everything running

Keeping the lab active is useful if you want to repeat the failure tests. No cleanup is required.

  • Leave Docker Desktop running.
  • Keep primary, backup, and gateway active for another gateway request.

Pause - I'll come back to this later

Pausing frees the RAM used by the running services. Your containers remain available for a later restart.

  • Stop the lab services from the PowerShell terminal you already have open by running this command:
docker compose stop

What does this command do?

Docker Compose stops the existing service containers without removing them. Their configuration remains available for a later restart.

  • Confirm that primary, backup, and gateway are no longer running in Docker Desktop.

Your lab is now paused, which frees its runtime memory without discarding the setup.

Containers still running?

Confirm that the PowerShell terminal is still inside litellm-reliability-lab. Check that Docker Desktop is running.

Help me troubleshoot why Docker Compose cannot stop these containers.

  • Restart the preserved containers when you return by running this command:
docker compose start

How does the restart work?

Docker Compose starts the existing service containers again. The lab returns without rebuilding the provider images.

  • Confirm that primary, backup, and gateway are running again in Docker Desktop.

Containers not restarting?

Confirm that Docker Desktop is running. Check that you did not remove the containers after pausing them.

Help me diagnose why my Compose services will not restart.

Delete - I don't want to use this again

Removing the running lab can feel final. Your litellm-reliability-lab files and cached images stay on your machine for a future rebuild.

  • Remove the lab containers and Compose network from the PowerShell terminal you already have open by running this command:
docker compose down --remove-orphans

What gets removed?

Docker Compose stops and removes the project containers and network. The option also removes orphaned service containers.

  • Confirm that primary, backup, and gateway no longer appear as running lab containers in Docker Desktop.

That clears the active lab while preserving the code you built.

Cleanup command failing?

Confirm that the PowerShell terminal is inside litellm-reliability-lab. Check that Docker Desktop is running.

Help me troubleshoot this Compose cleanup command.

Nice Work!

Nice Work!

You made it. Your local reliability lab now routes one unchanged chat request through a LiteLLM Gateway that recovers through a backup provider when the primary cannot answer.

You've learned how to:

  • Build two Python provider simulators that return responses through an OpenAI-compatible API. Connect them with Docker Compose health checks. Reach each container through service-name networking.
  • Trigger a deterministic HTTP 503 outage from the primary provider. Confirm the planned failure in the client output. Trace the same event in the container logs.
  • Configure retries inside the primary model group. Apply an ordered fallback to the backup model group. Keep the client payload unchanged while backup-provider handled proves recovery.
  • Secret Mission: Replaced the explicit outage with a slow primary provider. Set a one-second timeout. Proved the backup still responds after the primary exceeds the deadline.

Ready to quiz yourself?