Route AI Requests with LiteLLM

Deploy a LiteLLM gateway that routes Gemini requests with Jev on Google Cloud.

Introduction

30 Second Summary

A company can launch a new AI feature, then watch model costs climb as customer traffic grows. The engineer who can measure and manage spend can turn that tradeoff into a strong portfolio case study.

In this project, you will deploy LiteLLM on Google Cloud Run as a cost-aware AI gateway. LiteLLM uses Jev to classify each request and route it to a suitable Gemini model, giving you measured evidence for cost optimization.

What You'll Build

A live gateway where every model choice is backed by visible latency and spend evidence.

By the end of this project, you'll have:

  • 🚪 Live, authenticated AI gateway: Send requests through the deployed gateway and see the model responses it returns.
  • 📊 Repeatable cost benchmark: Compare fixed-model requests with routed requests using the same prompt set.
  • 🕵️‍♂️ Evidence-backed case study: Connect each routing choice to model selection, latency, classifier cost, and total spend.
  • 💎 Secret Mission: Set a small virtual-key budget, exhaust it, and observe LiteLLM reject the next request with HTTP 422.

Are there any prerequisites?

  • Access to Google Cloud Shell from a billing-enabled Google Cloud project
  • A TypeSafe AI account for your Jev API key.
  • Basic familiarity with the gcloud CLI, Terraform, and Docker

Is this project right for me?

Before We Start

Before any work, define the business result you want to prove. You are building evidence that you can manage AI spend while protecting response quality, a tradeoff AI systems engineers need to measure and communicate.

Set Up Cloud Shell and Credentials

A reliable deployment starts in a secure workspace tied to the correct cloud project. Google Cloud Shell provides that authenticated environment without requiring local cloud tools.

In this step, you will verify the workspace, install Terraform, and protect your TypeSafe AI API key before deployment.

In this step, get ready to:
  • Start an authenticated Cloud Shell session for your billed Google Cloud project.
  • Verify the built-in tools and install Terraform 1.16.5.
  • Verify the TypeSafe API key and store it in Secret Manager.
Open Cloud Shell and verify the project

Cloud Shell inherits the active project from the Google Cloud console. Its persistent $HOME directory keeps your project files between sessions.

I have a billing account

Use your existing Cloud Billing account when you create the project. Google Cloud links the new project to the selected account during creation.

  • Open the Manage resources page.
  • Click Create Project.
  • Enter LiteLLM Routing Lab in the Project name field.
  • Select your active account in the Billing account field.
  • Select the appropriate parent in the Location field if Google Cloud shows it.
  • Click Create.

You should see LiteLLM Routing Lab in the project selector with billing enabled.

I need to set up a billing account

This setup enables paid services, but the lab stays within its $5 target when you remove the resources in the cleanup section.

  • Open the Manage billing accounts page.
  • Click Create account.
  • Enter LiteLLM Routing Billing in the Name field.
  • Select your organization if Google Cloud shows an organization list.
  • Click Continue.
  • Select the Google payments profile that pays for this account.
  • Complete the payment-profile fields shown for your country if Google asks you to create a profile.
  • Click Submit and enable billing.

Your billing account is ready. Now create the project that uses it.

  • Open the Manage resources page.
  • Click Create Project.
  • Enter LiteLLM Routing Lab in the Project name field.
  • Select LiteLLM Routing Billing in the Billing account field.
  • Click Create.

You should see LiteLLM Routing Lab in the project selector with billing enabled.

  • Open the project selector in the Google Cloud console header.
  • Select LiteLLM Routing Lab.
  • Click Activate Cloud Shell in the console header.
  • Click Authorize if Cloud Shell asks to use your Google Cloud credentials.
  • Wait for the terminal prompt to finish loading. This usually takes a few seconds.
  • Confirm the active Google Cloud project by running this command:
gcloud config get-value project

You should see the ID of your billed Google Cloud project.

Seeing the wrong project ID?

Return to the project selector and choose the billed project. Restart Cloud Shell before running the check again.

Help me connect Cloud Shell to my billed Google Cloud project.

  • Record the verified project ID for the rest of the project: your-project-id.
Verify the tools and install Terraform

Cloud Shell includes the gcloud CLI and Docker. Terraform requires a separate installation in your persistent home directory.

  • Confirm the built-in Google Cloud and Docker tools by running these commands:
gcloud version; docker --version

You should see version details for both tools.

  • Install and activate Terraform 1.16.5 by running these commands:
mkdir -p "$HOME/bin"
curl -fsSLo "$HOME/terraform_1.16.5_linux_amd64.zip" "https://releases.hashicorp.com/terraform/1.16.5/terraform_1.16.5_linux_amd64.zip"
echo "2bc2fcfff033265c9e02ca0351f01794eb122f62a9b2a49a3294b9e49eaab5e4  $HOME/terraform_1.16.5_linux_amd64.zip" | sha256sum -c -
unzip -o "$HOME/terraform_1.16.5_linux_amd64.zip" -d "$HOME/bin"
rm "$HOME/terraform_1.16.5_linux_amd64.zip"
grep -qxF 'export PATH="$HOME/bin:$PATH"' "$HOME/.bashrc" || echo 'export PATH="$HOME/bin:$PATH"' >> "$HOME/.bashrc"
export PATH="$HOME/bin:$PATH"
terraform version

What does this installation do?

The commands download the official archive and verify its checksum before extraction. They add $HOME/bin to your shell path so Terraform remains available in later sessions.

You should see the checksum end with OK and the Terraform version report 1.16.5.

Terraform command still unavailable?

Restart Cloud Shell if the download failed or the updated path did not load.

Help me finish the Terraform installation.

Verify and store the TypeSafe API key

Your API key needs a durable home before the gateway can use it. Store it in Secret Manager first, then retrieve the stored value to verify access.

  • In the CloudShell terminal, enable Secret Manager for your active project by running this command:
gcloud services enable secretmanager.googleapis.com

A short pause during the command means Google Cloud is enabling the API.

I have a TypeSafe account set up

Your existing account gives you access to the TypeSafe console.

I need to create a TypeSafe account

A TypeSafe account is required to create an API key. TypeSafe handles account access through its login page.

  • Create an API key with the key creation control.
  • Copy the generated key.
  • Return to the Cloud Shell terminal.
  • Start secret creation from standard input by running this command:
gcloud secrets create typesafe-api-key --replication-policy=automatic --data-file=-
  • Paste the copied TypeSafe API key when the terminal waits for input.
  • Press Ctrl+D twice without pressing Enter.

What does standard input do?

The --data-file=- flag tells gcloud to read the secret payload directly from standard input. The pasted value becomes secret version 1 without being added to your shell history.

Your TypeSafe key now has a durable home in Secret Manager. Next, retrieve that stored value to test Jev access.

Before you run the check, do you expect the stored key to return the available models?

  • Retrieve the stored key and verify TypeSafe access by running these commands:
export TYPESAFE_API_KEY="$(gcloud secrets versions access latest --secret=typesafe-api-key)"
curl https://api.typesafe.ai/v1/models -H "Authorization: Bearer $TYPESAFE_API_KEY"
unset TYPESAFE_API_KEY
gcloud secrets versions list typesafe-api-key

You should see a JSON response that lists an available Jev model such as jev-latest.

✔️ Request succeeded

Your stored TypeSafe credential works!

ⓧ Request failed

A rejected request usually means the stored credential is no longer valid. Add a fresh TypeSafe API key as a new secret version.

  • Create a fresh TypeSafe API key.
  • Start a new version for the existing secret by running this command:
gcloud secrets versions add typesafe-api-key --data-file=-
  • Paste the fresh TypeSafe API key.
  • Press Ctrl+D twice without pressing Enter.

Still unable to verify the key?

Help me troubleshoot the stored TypeSafe credential.

Your authenticated workspace and stored credential are ready. Next, you will connect the secret to a live LiteLLM gateway deployment.

Deploy the Gateway and Admin UI

Your authenticated Google Cloud Shell session now gives you a controlled place to build the system. That preparation matters when you need to explain how a production deployment protects credentials and limits mistakes.

The next milestone is a live LiteLLM Admin UI backed by durable data. Terraform creates the supporting services before Cloud Run launches the gateway.

In this step, get ready to:
  • Create a reproducible gateway stack and validate it before deployment.
  • Connect the stored TypeSafe credential and publish the LiteLLM image.
  • Deploy the gateway and capture a live Admin UI result.
Create and validate the project

The project combines container configuration with infrastructure as code. The files define the same stack that Terraform later creates in your billed project.

  • Switch back to the Cloud Shell terminal from earlier.
  • Set up the litellm-gateway folder and its starter files by running these commands:
mkdir litellm-gateway
cd litellm-gateway
touch .gitignore Dockerfile config.yaml main.tf outputs.tf terraform.tfvars variables.tf versions.tf
ls -a

Why use terraform.tfvars?

Terraform loads terraform.tfvars automatically. The file stores your project-specific value, while .gitignore keeps it outside version control.

You should see Dockerfile, config.yaml, the Terraform files, terraform.tfvars, and the hidden .gitignore file in the output.

.gitignore

  • Open .gitignore in the Cloud Shell Editor.
cloudshell open .gitignore
  • Add the ignore rules by pasting this code:
.terraform/
*.tfstate
*.tfstate.*
terraform.tfvars
benchmark-*.csv

What do these rules protect?

The rules keep Terraform state, local project values, and benchmark output out of version control. The TypeSafe key stays in Secret Manager instead of a project file.

Dockerfile

  • Open Dockerfile in the Cloud Shell Editor.
  • Define the gateway image by pasting this code:
FROM docker.litellm.ai/berriai/litellm:1.104.0
WORKDIR /app
COPY config.yaml /app/config.yaml
EXPOSE 8080/tcp
CMD ["--port", "8080", "--config", "/app/config.yaml"]

What does this image run?

The image pins LiteLLM to 1.104.0. It copies the gateway configuration into the image and starts the proxy on port 8080.

config.yaml

  • Open config.yaml in the Cloud Shell Editor.
  • Define the two model aliases and gateway settings by pasting this code:
model_list:
  - model_name: gemini-flash-lite
    litellm_params:
      model: vertex_ai/gemini-3.5-flash-lite
      vertex_project: os.environ/VERTEXAI_PROJECT
      vertex_location: os.environ/VERTEXAI_LOCATION
  - model_name: gemini-flash
    litellm_params:
      model: vertex_ai/gemini-3.5-flash
      vertex_project: os.environ/VERTEXAI_PROJECT
      vertex_location: os.environ/VERTEXAI_LOCATION

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

What does this configuration define?

The two aliases connect LiteLLM to Gemini 3.5 Flash-Lite and Gemini 3.5 Flash through Vertex AI. The general settings read authentication and persistent data settings from the deployed environment.

versions.tf

  • Open versions.tf in the Cloud Shell Editor.
  • Pin Terraform and its providers by pasting this code:
terraform {
  required_version = "= 1.16.5"
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "8.5.0"
    }
    random = {
      source  = "hashicorp/random"
      version = "3.9.1"
    }
  }
}
provider "google" {
  project = var.project_id
  region  = var.region
}

Why pin these versions?

The file requires Terraform 1.16.5. It fixes the Google provider at 8.5.0 and the Random provider at 3.9.1 so the configuration uses known resource schemas.

variables.tf

  • Open variables.tf in the Cloud Shell Editor.
  • Define the deployment inputs by pasting this code:
variable "project_id" {
  description = "The billed Google Cloud project ID."
  type        = string
}
variable "region" {
  description = "The region for Cloud Run, Cloud SQL, and Artifact Registry."
  type        = string
  default     = "us-central1"
}
variable "vertex_location" {
  description = "The Vertex AI location used by both Gemini deployments."
  type        = string
  default     = "global"
}

What do these variables control?

The variables select your billed project. They place the cloud resources in us-central1 while Gemini requests use the global location.

main.tf

The main infrastructure file has several sections. Add each visible block in order so you can check the boundary before moving on.

  • Open main.tf in the Cloud Shell Editor.
  • Add the shared values, required APIs, and existing TypeSafe secret by pasting this code at the top:
# Define names and connection values reused by the stack.
locals {
  required_services = toset([
    "aiplatform.googleapis.com",
    "artifactregistry.googleapis.com",
    "cloudresourcemanager.googleapis.com",
    "iam.googleapis.com",
    "run.googleapis.com",
    "secretmanager.googleapis.com",
    "sqladmin.googleapis.com",
  ])
  repository_id  = "litellm"
  service_name   = "litellm-gateway"
  image_uri      = "${var.region}-docker.pkg.dev/${var.project_id}/${local.repository_id}/litellm:1.104.0"
  runtime_member = "serviceAccount:${google_service_account.gateway.email}"
  database_url   = "postgresql://litellm:${random_password.runtime["database"].result}@localhost/litellm?host=/cloudsql/${google_sql_database_instance.postgres.connection_name}&connection_limit=2"
}

# Enable every Google Cloud API used by the stack.
resource "google_project_service" "apis" {
  for_each           = local.required_services
  project            = var.project_id
  service            = each.value
  disable_on_destroy = false
}

data "google_secret_manager_secret" "typesafe_api_key" {
  project   = var.project_id
  secret_id = "typesafe-api-key"
}

What does this section establish?

The local values keep names and connection strings consistent. The resources enable the required APIs and look up the TypeSafe secret created during setup.

  • Add the image repository, runtime identity, and generated passwords below the existing code:
# Create the image destination and the gateway identity.
resource "google_artifact_registry_repository" "images" {
  project       = var.project_id
  location      = var.region
  repository_id = local.repository_id
  description   = "LiteLLM gateway images"
  format        = "DOCKER"
  depends_on    = [google_project_service.apis]
}

resource "google_service_account" "gateway" {
  project      = var.project_id
  account_id   = "litellm"
  display_name = "LiteLLM gateway"
  depends_on   = [google_project_service.apis]
}

resource "google_project_iam_member" "runtime_roles" {
  for_each = toset(["roles/aiplatform.user", "roles/cloudsql.client"])
  project  = var.project_id
  role     = each.value
  member   = local.runtime_member
}

# Generate credentials without storing plaintext values in files.
resource "random_password" "runtime" {
  for_each = { database = 32, master = 48, salt = 48 }
  length   = each.value
  special  = false
}

What does this section create?

Artifact Registry receives the LiteLLM image. The service account receives the runtime roles, while Terraform generates the database and gateway credentials.

  • Add the PostgreSQL instance below the password resources:
# Provision the persistent database used by the Admin UI.
resource "google_sql_database_instance" "postgres" {
  project          = var.project_id
  name             = "litellm-postgres"
  region           = var.region
  database_version = "POSTGRES_15"
  settings {
    tier                  = "db-f1-micro"
    edition               = "ENTERPRISE"
    disk_type             = "PD_SSD"
    disk_size             = 10
    disk_autoresize       = false
    connector_enforcement = "REQUIRED"
    backup_configuration { enabled = false }
    ip_configuration { ipv4_enabled = true }
  }
  deletion_protection = false
  depends_on          = [google_project_service.apis]
}

What does this database section configure?

Cloud SQL runs PostgreSQL 15 on a small instance with a fixed 10 GB disk. Connector enforcement keeps the application connection on the managed Cloud SQL path.

  • Add the LiteLLM database, database user, and secret containers below the instance:
# Create the application database and its user.
resource "google_sql_database" "litellm" {
  project  = var.project_id
  name     = "litellm"
  instance = google_sql_database_instance.postgres.name
}
resource "google_sql_user" "litellm" {
  project  = var.project_id
  name     = "litellm"
  instance = google_sql_database_instance.postgres.name
  password = random_password.runtime["database"].result
}

# Create separate containers for the generated runtime values.
resource "google_secret_manager_secret" "runtime" {
  for_each = {
    master_key   = "litellm-master-key"
    salt_key     = "litellm-salt-key"
    database_url = "litellm-database-url"
  }
  project   = var.project_id
  secret_id = each.value
  replication {
    auto {}
  }
  depends_on = [google_project_service.apis]
}

Why separate the secret containers?

Each runtime value gets its own Secret Manager resource. Cloud Run can then receive only the values it needs through named environment variables.

  • Add the generated secret versions below the secret containers:
# Store generated values as the current secret versions.
resource "google_secret_manager_secret_version" "master_key" {
  secret      = google_secret_manager_secret.runtime["master_key"].id
  secret_data = "sk-${random_password.runtime["master"].result}"
}
resource "google_secret_manager_secret_version" "salt_key" {
  secret      = google_secret_manager_secret.runtime["salt_key"].id
  secret_data = "sk-${random_password.runtime["salt"].result}"
}
resource "google_secret_manager_secret_version" "database_url" {
  secret      = google_secret_manager_secret.runtime["database_url"].id
  secret_data = local.database_url
  depends_on  = [google_sql_database.litellm, google_sql_user.litellm]
}

What do these versions store?

The versions hold the generated master key, salt key, and database connection URL. Terraform passes the values directly into Secret Manager.

  • Grant the gateway access to its four runtime secrets by adding this code:
# Map the plain and secret-backed runtime environment variables.
locals {
  runtime_values = {
    VERTEXAI_PROJECT  = var.project_id
    VERTEXAI_LOCATION = var.vertex_location
    STORE_MODEL_IN_DB = "True"
  }
  runtime_secrets = {
    LITELLM_MASTER_KEY = google_secret_manager_secret.runtime["master_key"].secret_id
    LITELLM_SALT_KEY   = google_secret_manager_secret.runtime["salt_key"].secret_id
    DATABASE_URL       = google_secret_manager_secret.runtime["database_url"].secret_id
    TYPESAFE_API_KEY   = data.google_secret_manager_secret.typesafe_api_key.secret_id
  }
}

resource "google_secret_manager_secret_iam_member" "runtime_secret_access" {
  for_each  = local.runtime_secrets
  project   = var.project_id
  secret_id = each.value
  role      = "roles/secretmanager.secretAccessor"
  member    = local.runtime_member
}

How does the gateway get secret access?

The map collects the three generated secrets and the existing TypeSafe secret. Terraform grants the gateway service account access to each one.

  • Start the Cloud Run service definition below the secret access resource:
# Define the public gateway service and its scaling limits.
resource "google_cloud_run_v2_service" "gateway" {
  project             = var.project_id
  name                = local.service_name
  location            = var.region
  ingress             = "INGRESS_TRAFFIC_ALL"
  deletion_protection = false
  template {
    service_account = google_service_account.gateway.email
    timeout         = "300s"
    scaling {
      min_instance_count = 0
      max_instance_count = 1
    }
    containers {
      image = local.image_uri
      ports { container_port = 8080 }
      resources {
        limits           = { cpu = "1", memory = "2Gi" }
        cpu_idle          = true
        startup_cpu_boost = true
      }
      dynamic "env" {
        for_each = local.runtime_values
        content {
          name  = env.key
          value = env.value
        }
      }

What does this service section control?

The service scales from zero to one instance and sends traffic to port 8080. Its resource limits give LiteLLM one CPU and 2 GiB of memory.

  • Continue the open container block with the secret-backed environment settings:
      # Inject runtime secrets without placing them in the image.
      dynamic "env" {
        for_each = local.runtime_secrets
        content {
          name = env.key
          value_source {
            secret_key_ref {
              secret  = env.value
              version = "latest"
            }
          }
        }
      }

Why inject these values?

Cloud Run reads each value from its latest Secret Manager version. The container receives the settings at runtime without storing them in the image.

  • Finish the Cloud Run service by adding its Cloud SQL mount, traffic rule, and dependencies:
      # Mount the managed Cloud SQL connection inside the container.
      volume_mounts {
        name       = "cloudsql"
        mount_path = "/cloudsql"
      }
    }
    volumes {
      name = "cloudsql"
      cloud_sql_instance {
        instances = [google_sql_database_instance.postgres.connection_name]
      }
    }
  }
  traffic {
    type    = "TRAFFIC_TARGET_ALLOCATION_TYPE_LATEST"
    percent = 100
  }
  depends_on = [
    google_project_iam_member.runtime_roles,
    google_secret_manager_secret_iam_member.runtime_secret_access,
    google_secret_manager_secret_version.master_key,
    google_secret_manager_secret_version.salt_key,
    google_secret_manager_secret_version.database_url,
    google_sql_user.litellm,
  ]
}

What completes the service?

The volume exposes the Cloud SQL connection at /cloudsql. The traffic rule sends every request to the latest revision after the identities, secrets, and database user exist.

  • Make the deployed gateway publicly invokable by adding the final resource:
# Allow unauthenticated HTTP requests to reach the gateway.
resource "google_cloud_run_v2_service_iam_member" "public" {
  project  = var.project_id
  location = google_cloud_run_v2_service.gateway.location
  name     = google_cloud_run_v2_service.gateway.name
  role     = "roles/run.invoker"
  member   = "allUsers"
}

What does the final resource allow?

The invoker binding lets public HTTP requests reach the Cloud Run service. LiteLLM still protects gateway operations with its own key.

✔️ Awesome, I've got everything!

Great, you're ready to deploy your infrastructure.

ⓧ I'd like to double check the full code

# Define names and connection values reused by the stack.
locals {
  required_services = toset([
    "aiplatform.googleapis.com",
    "artifactregistry.googleapis.com",
    "cloudresourcemanager.googleapis.com",
    "iam.googleapis.com",
    "run.googleapis.com",
    "secretmanager.googleapis.com",
    "sqladmin.googleapis.com",
  ])
  repository_id  = "litellm"
  service_name   = "litellm-gateway"
  image_uri      = "${var.region}-docker.pkg.dev/${var.project_id}/${local.repository_id}/litellm:1.104.0"
  runtime_member = "serviceAccount:${google_service_account.gateway.email}"
  database_url   = "postgresql://litellm:${random_password.runtime["database"].result}@localhost/litellm?host=/cloudsql/${google_sql_database_instance.postgres.connection_name}&connection_limit=2"
}

# Enable every Google Cloud API used by the stack.
resource "google_project_service" "apis" {
  for_each           = local.required_services
  project            = var.project_id
  service            = each.value
  disable_on_destroy = false
}

data "google_secret_manager_secret" "typesafe_api_key" {
  project   = var.project_id
  secret_id = "typesafe-api-key"
}

# Create the image destination and the gateway identity.
resource "google_artifact_registry_repository" "images" {
  project       = var.project_id
  location      = var.region
  repository_id = local.repository_id
  description   = "LiteLLM gateway images"
  format        = "DOCKER"
  depends_on    = [google_project_service.apis]
}

resource "google_service_account" "gateway" {
  project      = var.project_id
  account_id   = "litellm"
  display_name = "LiteLLM gateway"
  depends_on   = [google_project_service.apis]
}

resource "google_project_iam_member" "runtime_roles" {
  for_each = toset(["roles/aiplatform.user", "roles/cloudsql.client"])
  project  = var.project_id
  role     = each.value
  member   = local.runtime_member
}

# Generate credentials without storing plaintext values in files.
resource "random_password" "runtime" {
  for_each = { database = 32, master = 48, salt = 48 }
  length   = each.value
  special  = false
}

# Provision the persistent database used by the Admin UI.
resource "google_sql_database_instance" "postgres" {
  project          = var.project_id
  name             = "litellm-postgres"
  region           = var.region
  database_version = "POSTGRES_15"
  settings {
    tier                  = "db-f1-micro"
    edition               = "ENTERPRISE"
    disk_type             = "PD_SSD"
    disk_size             = 10
    disk_autoresize       = false
    connector_enforcement = "REQUIRED"
    backup_configuration { enabled = false }
    ip_configuration { ipv4_enabled = true }
  }
  deletion_protection = false
  depends_on          = [google_project_service.apis]
}

# Create the application database and its user.
resource "google_sql_database" "litellm" {
  project  = var.project_id
  name     = "litellm"
  instance = google_sql_database_instance.postgres.name
}
resource "google_sql_user" "litellm" {
  project  = var.project_id
  name     = "litellm"
  instance = google_sql_database_instance.postgres.name
  password = random_password.runtime["database"].result
}

# Create separate containers for the generated runtime values.
resource "google_secret_manager_secret" "runtime" {
  for_each = {
    master_key   = "litellm-master-key"
    salt_key     = "litellm-salt-key"
    database_url = "litellm-database-url"
  }
  project   = var.project_id
  secret_id = each.value
  replication {
    auto {}
  }
  depends_on = [google_project_service.apis]
}

# Store generated values as the current secret versions.
resource "google_secret_manager_secret_version" "master_key" {
  secret      = google_secret_manager_secret.runtime["master_key"].id
  secret_data = "sk-${random_password.runtime["master"].result}"
}
resource "google_secret_manager_secret_version" "salt_key" {
  secret      = google_secret_manager_secret.runtime["salt_key"].id
  secret_data = "sk-${random_password.runtime["salt"].result}"
}
resource "google_secret_manager_secret_version" "database_url" {
  secret      = google_secret_manager_secret.runtime["database_url"].id
  secret_data = local.database_url
  depends_on  = [google_sql_database.litellm, google_sql_user.litellm]
}

# Map the plain and secret-backed runtime environment variables.
locals {
  runtime_values = {
    VERTEXAI_PROJECT  = var.project_id
    VERTEXAI_LOCATION = var.vertex_location
    STORE_MODEL_IN_DB = "True"
  }
  runtime_secrets = {
    LITELLM_MASTER_KEY = google_secret_manager_secret.runtime["master_key"].secret_id
    LITELLM_SALT_KEY   = google_secret_manager_secret.runtime["salt_key"].secret_id
    DATABASE_URL       = google_secret_manager_secret.runtime["database_url"].secret_id
    TYPESAFE_API_KEY   = data.google_secret_manager_secret.typesafe_api_key.secret_id
  }
}

resource "google_secret_manager_secret_iam_member" "runtime_secret_access" {
  for_each  = local.runtime_secrets
  project   = var.project_id
  secret_id = each.value
  role      = "roles/secretmanager.secretAccessor"
  member    = local.runtime_member
}

# Define the public gateway service and its scaling limits.
resource "google_cloud_run_v2_service" "gateway" {
  project             = var.project_id
  name                = local.service_name
  location            = var.region
  ingress             = "INGRESS_TRAFFIC_ALL"
  deletion_protection = false
  template {
    service_account = google_service_account.gateway.email
    timeout         = "300s"
    scaling {
      min_instance_count = 0
      max_instance_count = 1
    }
    containers {
      image = local.image_uri
      ports { container_port = 8080 }
      resources {
        limits           = { cpu = "1", memory = "2Gi" }
        cpu_idle          = true
        startup_cpu_boost = true
      }
      dynamic "env" {
        for_each = local.runtime_values
        content {
          name  = env.key
          value = env.value
        }
      }
      # Inject runtime secrets without placing them in the image.
      dynamic "env" {
        for_each = local.runtime_secrets
        content {
          name = env.key
          value_source {
            secret_key_ref {
              secret  = env.value
              version = "latest"
            }
          }
        }
      }
      # Mount the managed Cloud SQL connection inside the container.
      volume_mounts {
        name       = "cloudsql"
        mount_path = "/cloudsql"
      }
    }
    volumes {
      name = "cloudsql"
      cloud_sql_instance {
        instances = [google_sql_database_instance.postgres.connection_name]
      }
    }
  }
  traffic {
    type    = "TRAFFIC_TARGET_ALLOCATION_TYPE_LATEST"
    percent = 100
  }
  depends_on = [
    google_project_iam_member.runtime_roles,
    google_secret_manager_secret_iam_member.runtime_secret_access,
    google_secret_manager_secret_version.master_key,
    google_secret_manager_secret_version.salt_key,
    google_secret_manager_secret_version.database_url,
    google_sql_user.litellm,
  ]
}

# Allow unauthenticated HTTP requests to reach the gateway.
resource "google_cloud_run_v2_service_iam_member" "public" {
  project  = var.project_id
  location = google_cloud_run_v2_service.gateway.location
  name     = google_cloud_run_v2_service.gateway.name
  role     = "roles/run.invoker"
  member   = "allUsers"
}

Does main.tf look incomplete?

Ask for help with the section around the reported line: help me check the structure of main.tf.

outputs.tf

  • Open outputs.tf in the Cloud Shell Editor.
  • Expose the deployed addresses and image destination by pasting this code:
output "service_url" {
  description = "LiteLLM gateway base URL."
  value       = google_cloud_run_v2_service.gateway.uri
}
output "admin_url" {
  description = "LiteLLM Admin UI URL."
  value       = "${google_cloud_run_v2_service.gateway.uri}/ui"
}
output "image_uri" {
  description = "Artifact Registry image URI used by Cloud Run."
  value       = local.image_uri
}

Why expose these outputs?

The outputs provide the gateway address, Admin UI address, and exact container image destination after Terraform creates them.

terraform.tfvars

  • Open terraform.tfvars in the Cloud Shell Editor.
  • Set the billed project ID by pasting this code:
project_id = "[[PROJECT_ID="your-project-id"]]"

What belongs in this file?

The terraform.tfvars file supplies your billed project ID to Terraform. The matching .gitignore rule keeps this local value out of version control.

  • Switch back to the Cloud Shell terminal from earlier.
  • Enable the Cloud Resource Manager API and validate the Terraform project by running these commands:
gcloud services enable cloudresourcemanager.googleapis.com
gcloud services list --enabled | grep cloudresourcemanager.googleapis.com
terraform init
terraform fmt -check
terraform validate

Why enable this API first?

Cloud Resource Manager exposes the project-level IAM operations that Terraform uses for the gateway roles. Enabling it before the apply prevents those resources from reaching an unavailable API.

You should see successful initialization followed by successful validation. Your gateway now has a repeatable infrastructure definition.

What do these checks prove?

  • `terraform init` downloads Google provider `8.5.0` and Random provider `3.9.1`.
  • `terraform fmt -check` confirms that the Terraform files use canonical formatting.
  • `terraform validate` checks the configuration structure and provider arguments.

Terraform validation failed?

Ask for help with the exact file and line Terraform reports: help me diagnose this Terraform validation error.

Bootstrap the repository and image

Cloud Run needs a container image before Terraform can create the service. The first targeted apply creates the Artifact Registry repository that receives that image.

  • Create the image repository by running:
terraform apply -target=google_artifact_registry_repository.images

Why use a targeted apply?

Cloud Run references an image that does not exist yet. This targeted apply creates the destination first so Docker has somewhere to push the image.

  • Review the resources in the Terraform plan.
  • Confirm the apply with "yes" when Terraform prompts you.

You should see Terraform report that the litellm repository was created. The TypeSafe credential remains in Secret Manager from the setup step.

Targeted apply failed?

help me troubleshoot this targeted Terraform apply.

  • Evaluate the exact image destination from the Terraform configuration by running:
echo 'local.image_uri' | terraform console

Why use Terraform console here?

The terraform console command evaluates local.image_uri from your current configuration and terraform.tfvars. You should see the complete Artifact Registry URI surrounded by quotation marks.

🙋‍♀️ Image URI missing?

Confirm that main.tf and terraform.tfvars are saved inside litellm-gateway. Run pwd and make sure your terminal is still in that folder before retrying the command.help me retrieve the image URI.

  • Record the printed destination without its surrounding quotation marks here: your exact Artifact Registry image URI.
  • Configure Docker authentication for the us-central1 Artifact Registry host by running:
gcloud auth configure-docker us-central1-docker.pkg.dev

What does this command change?

The command registers the Google Cloud credential helper for the regional registry host. Docker can then use your authenticated Cloud Shell identity when it pushes the image.

  • Approve the credential-helper configuration if the command asks for confirmation.

The image build can take a few minutes on its first run because Docker downloads the pinned LiteLLM base image. A stream of downloaded layers means the build is progressing.

  • Build the image and push it to the recorded destination by running:
docker build -t [[IMAGE_URI="your exact Artifact Registry image URI"]] .
docker push [[IMAGE_URI="your exact Artifact Registry image URI"]]

What happens during the build?

Docker builds from `Dockerfile` and adds `config.yaml` to the image. The push uploads the resulting image layers to the litellm repository.

You should see the push finish with an image digest. That repository now contains the `litellm:1.104.0` image required by Cloud Run.

Image build or push failed?

Confirm that Dockerfile and config.yaml are saved inside litellm-gateway. Check that the recorded image URI begins with the regional Artifact Registry hostname.

If the push cannot authenticate, rerun the Docker authentication command for the same regional hostname.

Share the non-secret Docker error for targeted help: help me troubleshoot this LiteLLM image push.

Deploy and verify the gateway

The full apply now connects the image to managed compute, persistent PostgreSQL, and secret injection. The resulting gateway keeps its Admin UI data even when Cloud Run replaces an instance.

This step creates billed resources

The apply creates usage-billed Google Cloud resources. The project targets under $5 when you destroy the stack within two hours, but the virtual-key budget does not cap Google Cloud charges.

Cloud SQL provisioning commonly takes about 15 minutes. A long period of status updates during this apply is expected.

  • Start the complete infrastructure deployment by running:
terraform apply

What does the full apply create?

Terraform creates the PostgreSQL `15` database instance, the `litellm` database, and the `litellm` user. It also creates the gateway service account with access to Vertex AI, Cloud SQL, and Secret Manager.

The apply creates the three LiteLLM secrets and reads the TypeSafe secret from setup. It then deploys a public Cloud Run service that scales from zero to one instance and mounts the Cloud SQL connection at /cloudsql.

  • Review the complete resource plan when Terraform displays it.
  • Confirm the apply when Terraform prompts you.
  • Wait until Terraform reports that the apply completed.

You should see outputs for `service_url`, `admin_url`, and `image_uri`. The infrastructure is ready for a direct health check.

Deployment stopped before Cloud Run?

Retry the apply once if an API was still activating. Terraform resumes from the resources already stored in state.

Get help with the failed Terraform resource: help me diagnose this gateway deployment failure.

  • Print the Admin UI address by running:
terraform output -raw admin_url

What does this output show?

The output combines the Cloud Run service address with `/ui`. LiteLLM serves its database-backed administration interface at that path.

  • Record the printed address here: your LiteLLM Admin UI URL.
  • Open your LiteLLM Admin UI URL in a new browser tab.

You should see the LiteLLM sign-in page at an address ending in /ui. That page is visible proof that you can deploy a persistent, production-style AI gateway.

Your cloud-hosted gateway is live and its Admin UI preserves configuration in PostgreSQL. Next, you will build the benchmark client and run it against one fixed model to capture the before-state.

Measure the Fixed-Model Baseline

Your deployed LiteLLM Admin UI proves that the gateway is live. Now you need a repeatable way to measure how one fixed model handles prompts with different levels of complexity.

You will create a budget-limited virtual key and build a Python benchmark client. The client sends four prompts through gemini-flash so the fixed-model limitation becomes measurable.

In this step, get ready to:
  • Create a $1 virtual key restricted to gemini-flash.
  • Build a reusable benchmark.py client.
  • Run four benchmark cases and capture the fixed-model baseline.
Sign in and create the benchmark key

The litellm-master-key secret is the bootstrap password for the Admin UI. Secret Manager keeps the value outside your source files.

This command reveals a live credential in your terminal. The value only needs to stay visible long enough for you to copy it.

  • Switch back to the Cloud Shell terminal from the previous step.
  • Keep screenshots away from the terminal while the master key is visible.
  • Read the current master key by running this command:
gcloud secrets versions access latest --secret=litellm-master-key

What does this command retrieve?

The command reads the latest stored version of litellm-master-key. LiteLLM accepts this value as the initial Admin UI password.

  • Copy the returned master key without adding spaces.
  • Return to the Admin UI browser tab from the previous step.
  • Enter admin in the username field.
  • Paste the copied master key into the password field.
  • Complete the sign-in from the login form.

You will see the authenticated Admin UI navigation. The available sections include Virtual Keys.

Master key rejected?

Confirm that the username is admin. Copy the newest secret value directly from the terminal without including surrounding whitespace.

If access still fails, help me troubleshoot the LiteLLM Admin UI login.

A virtual key gives the benchmark its own model permissions and budget. Restricting this key to gemini-flash creates a controlled baseline.

  • Select Virtual Keys in the Admin UI sidebar.
  • Start creating a virtual key from the Virtual Keys page.

The virtual-key configuration form is now ready for the baseline settings.

  • Enter benchmark-key as the key name.
  • Set the allowed model to gemini-flash.
  • Set the maximum budget to $1.
  • Create the virtual key.
  • Copy the generated plaintext key before leaving the confirmation view.
  • Return to the Cloud Shell terminal.
  • Store the copied virtual key in Secret Manager by running this command:
gcloud secrets create litellm-api-key --replication-policy=automatic --data-file=-
  • Paste the copied virtual key when the terminal waits for input.
  • Press Ctrl+D twice without pressing Enter.

Your first cost guardrail is ready. Secret Manager now protects the virtual key that can call gemini-flash within its $1 gateway budget.

  • In the Cloud Shell terminal, set the gateway base URL from the existing Terraform output by running this command:
export LITELLM_BASE_URL="$(terraform output -raw service_url)"

What does this variable hold?

The Terraform output supplies the deployed gateway address. LITELLM_BASE_URL gives benchmark.py the base URL for its requests.

The benchmark still needs the virtual key as an environment variable. Retrieve it from Secret Manager so you do not have to paste the credential again.

  • Load the stored virtual key into the current shell by running this command:
export LITELLM_API_KEY="$(gcloud secrets versions access latest --secret=litellm-api-key)"

Your current Cloud Shell session now has the gateway URL and stored virtual key available without printing the credential.

Unable to load the stored key?

Help me load the stored LiteLLM virtual key.

Build the benchmark client

The benchmark client gives both the fixed model and the future router the same workload. That shared test makes the comparison fair.

  • Create benchmark.py inside the litellm-gateway folder and confirm it exists by running these commands:
touch benchmark.py
cloudshell open benchmark.py

This command creates and opens benchmark.py in the Cloud Shell Editor.

  • Add the imports, benchmark cases, and model aliases by pasting this code at the top:
# Import standard-library tools for HTTP calls and CSV reporting.
import csv
import json
import os
import sys
import time
import urllib.error
import urllib.request
from collections import Counter

# Cover requests with different complexity levels.
CASES = [
    ("simple", "SIMPLE", "Reply with only the word blue."),
    ("short_explanation", "MEDIUM", "Explain in two sentences why HTTPS is safer than HTTP."),
    ("architecture", "COMPLEX", "Design a resilient event-driven order pipeline. Include idempotency, retries, and observability in four bullets."),
    ("tradeoff", "REASONING", "Reason step by step about when a team should choose queues over synchronous APIs, considering latency, failure isolation, ordering, and operations."),
]
MODELS = {"baseline": "gemini-flash", "routed": "jev-router"}

What does this section prepare?

The four cases exercise simple, explanatory, architectural, and reasoning requests. The model map switches the same cases between the fixed baseline and routed gateway.

  • Add the request function below the model aliases:
def run_case(base_url, api_key, requested_model, case_name, expected_tier, prompt):
    # Build one OpenAI-compatible chat request for the gateway.
    payload = json.dumps({
        "model": requested_model,
        "messages": [{"role": "user", "content": prompt}],
    }).encode("utf-8")
    request = urllib.request.Request(
        f"{base_url}/v1/chat/completions",
        data=payload,
        headers={
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json",
        },
        method="POST",
    )
    # Measure the end-to-end request and capture routing evidence.
    started = time.perf_counter()
    with urllib.request.urlopen(request, timeout=120) as response:
        body = json.loads(response.read().decode("utf-8"))
        return {
            "case": case_name,
            "expected_tier": expected_tier,
            "requested_model": requested_model,
            "returned_model": body.get("model", "unknown"),
            "latency_ms": round((time.perf_counter() - started) * 1000, 1),
            "classifier_cost_header": response.headers.get("x-litellm-classifier-cost", ""),
        }

What does run_case() record?

The function sends one chat request and measures its latency. It reads returned_model from the response's model field. The baseline returns gemini-flash, while routed requests return the resolved Gemini deployment after you enable return_raw_model_name: true.

  • Finish the benchmark by adding the main workflow below run_case():
def main():
    if len(sys.argv) != 2 or sys.argv[1] not in MODELS:
        raise SystemExit("Usage: python3 benchmark.py baseline|routed")
    base_url = os.environ["LITELLM_BASE_URL"].rstrip("/")
    api_key = os.environ["LITELLM_API_KEY"]
    mode = sys.argv[1]
    rows = []
    # Run every case and print its routing result immediately.
    try:
        for case_name, expected_tier, prompt in CASES:
            row = run_case(base_url, api_key, MODELS[mode], case_name, expected_tier, prompt)
            rows.append(row)
            print(f"{case_name:18} expected={expected_tier:9} returned={row['returned_model']:28} latency_ms={row['latency_ms']:8} classifier_cost={row['classifier_cost_header'] or 'n/a'}")
    except urllib.error.HTTPError as error:
        detail = error.read().decode("utf-8", errors="replace")
        print(f"HTTP {error.code}: {detail}", file=sys.stderr)
        raise SystemExit(1) from error
    output_path = f"benchmark-{mode}.csv"
    # Save a reusable report and print the aggregate result.
    with open(output_path, "w", newline="", encoding="utf-8") as output_file:
        writer = csv.DictWriter(output_file, fieldnames=rows[0].keys())
        writer.writeheader()
        writer.writerows(rows)
    counts = Counter(row["returned_model"] for row in rows)
    average_latency = round(sum(row["latency_ms"] for row in rows) / len(rows), 1)
    print(f"\nWrote {output_path}")
    print(f"Model counts: {dict(counts)}")
    print(f"Average latency: {average_latency} ms")
if __name__ == "__main__":
    main()

What does the main workflow produce?

The workflow runs all four cases and prints each result. It writes a baseline or routed CSV file, then reports model counts and average latency.

✔️ Awesome, I've got everything!

Great, your benchmarking script is ready to go!

ⓧ I'd like to double check the full code

# Import standard-library tools for HTTP calls and CSV reporting.
import csv
import json
import os
import sys
import time
import urllib.error
import urllib.request
from collections import Counter

# Cover requests with different complexity levels.
CASES = [
    ("simple", "SIMPLE", "Reply with only the word blue."),
    ("short_explanation", "MEDIUM", "Explain in two sentences why HTTPS is safer than HTTP."),
    ("architecture", "COMPLEX", "Design a resilient event-driven order pipeline. Include idempotency, retries, and observability in four bullets."),
    ("tradeoff", "REASONING", "Reason step by step about when a team should choose queues over synchronous APIs, considering latency, failure isolation, ordering, and operations."),
]
MODELS = {"baseline": "gemini-flash", "routed": "jev-router"}

def run_case(base_url, api_key, requested_model, case_name, expected_tier, prompt):
    # Build one OpenAI-compatible chat request for the gateway.
    payload = json.dumps({
        "model": requested_model,
        "messages": [{"role": "user", "content": prompt}],
    }).encode("utf-8")
    request = urllib.request.Request(
        f"{base_url}/v1/chat/completions",
        data=payload,
        headers={
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json",
        },
        method="POST",
    )
    # Measure the end-to-end request and capture routing evidence.
    started = time.perf_counter()
    with urllib.request.urlopen(request, timeout=120) as response:
        body = json.loads(response.read().decode("utf-8"))
        return {
            "case": case_name,
            "expected_tier": expected_tier,
            "requested_model": requested_model,
            "returned_model": body.get("model", "unknown"),
            "latency_ms": round((time.perf_counter() - started) * 1000, 1),
            "classifier_cost_header": response.headers.get("x-litellm-classifier-cost", ""),
        }

def main():
    if len(sys.argv) != 2 or sys.argv[1] not in MODELS:
        raise SystemExit("Usage: python3 benchmark.py baseline|routed")
    base_url = os.environ["LITELLM_BASE_URL"].rstrip("/")
    api_key = os.environ["LITELLM_API_KEY"]
    mode = sys.argv[1]
    rows = []
    # Run every case and print its routing result immediately.
    try:
        for case_name, expected_tier, prompt in CASES:
            row = run_case(base_url, api_key, MODELS[mode], case_name, expected_tier, prompt)
            rows.append(row)
            print(f"{case_name:18} expected={expected_tier:9} returned={row['returned_model']:28} latency_ms={row['latency_ms']:8} classifier_cost={row['classifier_cost_header'] or 'n/a'}")
    except urllib.error.HTTPError as error:
        detail = error.read().decode("utf-8", errors="replace")
        print(f"HTTP {error.code}: {detail}", file=sys.stderr)
        raise SystemExit(1) from error
    output_path = f"benchmark-{mode}.csv"
    # Save a reusable report and print the aggregate result.
    with open(output_path, "w", newline="", encoding="utf-8") as output_file:
        writer = csv.DictWriter(output_file, fieldnames=rows[0].keys())
        writer.writeheader()
        writer.writerows(rows)
    counts = Counter(row["returned_model"] for row in rows)
    average_latency = round(sum(row["latency_ms"] for row in rows) / len(rows), 1)
    print(f"\nWrote {output_path}")
    print(f"Model counts: {dict(counts)}")
    print(f"Average latency: {average_latency} ms")

if __name__ == "__main__":
    main()

Does benchmark.py show a syntax error?

Check that the three visible sections appear once and in the same order. Python indentation inside run_case() and main() must remain four spaces per level.

For help with the reported line, ask me to diagnose the benchmark.py syntax error.

Run the baseline benchmark

Your benchmark.py script sends the four entries in CASES to the gateway. It records the requested model and returned model for each response. The script also records latency and the classifier-cost header. It writes those measurements to a CSV file named benchmark-baseline.csv.

Before you run the benchmark, do you expect prompt complexity to change the returned model?

  • Run the fixed-model baseline from the litellm-gateway folder by running this command:
python3 benchmark.py baseline

What does the baseline measure?

  • The baseline argument selects gemini-flash from the MODELS mapping.
  • Each request goes to /v1/chat/completions with the same virtual key.
  • Each result captures returned_model, latency_ms, and classifier_cost_header.

You will see four successful rows for simple, short_explanation, architecture, and tradeoff. Every row returns gemini-flash despite the change in prompt complexity.

  • Select benchmark-baseline.csv in the Cloud Shell Editor Explorer.
  • Confirm that the file contains four data rows.
  • Check that every requested_model value is gemini-flash.
  • Check that every returned_model value is gemini-flash.
  • Read the measured latency_ms value for each case.
  • Confirm that classifier_cost_header is empty for the fixed-model rows.

You now have a repeatable baseline showing that simple work and difficult work consume the same strong model. That evidence gives your portfolio a clear example of identifying waste before changing an architecture.

Benchmark request failed?

help me diagnose my fixed-model benchmark.

Your baseline turns a cost concern into measured business evidence. Next, you will configure LiteLLM to use Jev for complexity-aware routing and match model capacity to each request.

Add Jev Routing in the Admin UI

Configure Complexity-based Auto Routing

Your fixed-model baseline showed the cost of sending every prompt to Gemini 3.5 Flash. Now you can route each request according to its complexity.

Your AI gateway needs a version-controlled decision layer between the prompt and the model. You will define that layer in config.yaml with LiteLLM and the Jev classifier.

In this step, get ready to:
  • Map each complexity tier to a cost-appropriate model alias in config.yaml.
  • Build and deploy a new gateway image containing the Jev router.
  • Run the benchmark and capture the resolved model names.
Add the Jev router configuration

The router belongs beside the two Gemini model aliases in model_list. This keeps the routing policy in the same file as the deployments it can select.

  • Return to the Cloud Shell Editor
  • Open config.yaml inside the litellm-gateway folder.
  • Add the router entry to the model_list above general_settings: by pasting this configuration:
  - model_name: jev-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_default_model: gemini-flash
      complexity_router_config:
        tiers:
          SIMPLE: gemini-flash-lite
          MEDIUM: gemini-flash-lite
          COMPLEX: gemini-flash
          REASONING: gemini-flash
        classifier_type: jev
        jev_classifier_config:
          model: jev-latest
          timeout_ms: 3000
          circuit_breaker_enabled: true
          circuit_breaker_cooldown_seconds: 30
        classifier_fallback: default_model
        classifier_context_window_size: 3
        classifier_context_budget_chars: 8000
        classifier_context_include_assistant_turns: false
        return_raw_model_name: true

What does this router configuration do?

Jev classifies each request as SIMPLE, MEDIUM, COMPLEX, or REASONING. LiteLLM sends the first two tiers to gemini-flash-lite and the last two tiers to gemini-flash.

✔️ Awesome, I've got everything!

Great, your config.yaml is ready to go.

ⓧ I'd like to double check the full code

model_list:
  - model_name: gemini-flash-lite
    litellm_params:
      model: vertex_ai/gemini-3.5-flash-lite
      vertex_project: os.environ/VERTEXAI_PROJECT
      vertex_location: os.environ/VERTEXAI_LOCATION
  - model_name: gemini-flash
    litellm_params:
      model: vertex_ai/gemini-3.5-flash
      vertex_project: os.environ/VERTEXAI_PROJECT
      vertex_location: os.environ/VERTEXAI_LOCATION
  - model_name: jev-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_default_model: gemini-flash
      complexity_router_config:
        tiers:
          SIMPLE: gemini-flash-lite
          MEDIUM: gemini-flash-lite
          COMPLEX: gemini-flash
          REASONING: gemini-flash
        classifier_type: jev
        jev_classifier_config:
          model: jev-latest
          timeout_ms: 3000
          circuit_breaker_enabled: true
          circuit_breaker_cooldown_seconds: 30
        classifier_fallback: default_model
        classifier_context_window_size: 3
        classifier_context_budget_chars: 8000
        classifier_context_include_assistant_turns: false
        return_raw_model_name: true

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

Router block missing or misaligned?

Keep the new entry indented beneath model_list:. Place general_settings: back at the left edge after the router block.

help me check my Jev router YAML indentation.

Build and deploy the updated image

The gateway image contains config.yaml, so the running service still has the earlier configuration. A new image tag gives Terraform a visible change to deploy.

  • Open main.tf inside the litellm-gateway folder.
  • Find this image_uri line:
image_uri = "${var.region}-docker.pkg.dev/${var.project_id}/${local.repository_id}/litellm:1.104.0"
  • Replace the image tag with 1.104.0-jev so the line looks like this:
image_uri = "${var.region}-docker.pkg.dev/${var.project_id}/${local.repository_id}/litellm:1.104.0-jev"
  • Validate the Terraform files and print the new image destination by running these commands:
terraform fmt -check
terraform validate
echo 'local.image_uri' | terraform console

You should see successful validation followed by an Artifact Registry URI ending in :1.104.0-jev.

  • Replace the earlier saved image destination with the newly printed value here: your exact Artifact Registry image URI.

The first build with the new tag may take a few minutes while Docker assembles and uploads the image layers. A stream of layer statuses means it is still progressing.

  • Build and publish the updated gateway image by running these commands:
docker build -t [[IMAGE_URI="your exact Artifact Registry image URI"]] .
docker push [[IMAGE_URI="your exact Artifact Registry image URI"]]

You should see the push finish with a digest for the 1.104.0-jev image.

Image build or push failed?

Confirm that config.yaml and Dockerfile are saved inside litellm-gateway. Check that the image URI ends in 1.104.0-jev.

Ask for help without sharing credentials: help me troubleshoot this LiteLLM image build.

Before you apply the change, do you expect Terraform to replace the database or update only the gateway service?

  • Create the deployment plan by running this command:
terraform apply
  • Review the plan for the litellm-gateway Cloud Run service.
  • Enter yes when Terraform asks for confirmation.

Expect the deployment to take a few minutes while Cloud Run creates a revision from the new image. You should then see Apply complete with the gateway service changed.

Gateway revision failed to deploy?

Ask for help with the deployment output: help me diagnose this Cloud Run revision failure.

Allow the router and run the benchmark

A virtual key can only call models on its allow list. Give the benchmark key access to the config-defined router before sending the same four prompts.

  • Return to Virtual Keys in the LiteLLM Admin UI.
  • Select benchmark-key.
  • Add jev-router to the allowed models.
  • Keep gemini-flash in the allowed models.
  • Save the virtual key changes.
  • Return to the Cloud Shell terminal inside litellm-gateway.

Before you run the benchmark, which resolved model do you expect for the simplest prompt and the reasoning prompt?

  • Run the routed benchmark by using this command:
python3 benchmark.py routed

You should see four successful rows. After the final response, the script writes benchmark-routed.csv.

Each returned_model value now shows the resolved Gemini model instead of jev-router. That is return_raw_model_name: true making the routing decision visible.

Routed benchmark failed?

Confirm that benchmark-key allows jev-router. Check that the deployed image tag ends in 1.104.0-jev and that the TypeSafe secret still has an active version.

Ask for help without sharing either key: help me diagnose my config-defined Jev router.

Your routed benchmark now exposes the model chosen for every prompt. Next, you will compare model choice, latency, and spend against the fixed baseline.

Compare Routing, Latency, and Spend

Your routed AI gateway now sends each benchmark prompt to a suitable model. The final question is whether that choice improves model fit without hiding its cost.

The baseline and routed CSV files show changes in model choice and latency. The LiteLLM Admin UI connects those results to classification and completion spend.

In this step, get ready to:
  • Compare the baseline and routed benchmark results.
  • Trace one routed request through its classification records.
  • Connect the routing decision to virtual-key spend.
Compare the baseline and routed results

The baseline file is your control. Matching each case across both files isolates the effect of routing.

  • Return to the Cloud Shell Editor from earlier.
  • Open benchmark-baseline.csv from the litellm-gateway file tree.
  • Open benchmark-routed.csv in a second editor tab.
  • Match the rows using the case column.
  • Record each case where returned_model changed.
  • Calculate routed latency_ms minus baseline latency_ms for each case.
  • Record each routed classifier_cost_header value.

You should see gemini-flash in every baseline row. In benchmark-routed.csv, returned_model shows the resolved deployment selected by Jev, such as gemini-3.5-flash-lite or gemini-3.5-flash, because config.yaml enables return_raw_model_name: true.

How to read the comparison

A positive latency difference means routing took longer. Keep an empty classifier_cost_header value blank when LiteLLM does not return the header.

Verify routing and spend in the Admin UI

LiteLLM records the Jev decision that selected each completion's tier. Matching the completion to its classifier call is fiddly, so use their timestamps and key attribution.

  • Return to the LiteLLM Admin UI.
  • Select Logs in the navigation menu.
  • Filter the visible records to requests attributed to benchmark-key.
  • Select one successful jev-router completion from the routed benchmark.
  • Inspect the request details for cause: jev_classifier.
  • Compare the selected tier with the returned model.
  • Locate the separate TypeSafe AI classifier call in the same time window.

You should see cause: jev_classifier with the selected tier and returned model. A separate TypeSafe classifier call should appear near the completion.

Can't match the log records?

A narrow time window makes the completion and classifier records easier to pair.

  • Adjust the visible time window to include the routed benchmark.
  • Compare the completion timestamp with nearby TypeSafe classifier calls.
  • Confirm that the completion is attributed to benchmark-key.

Help me match these LiteLLM log records.

Before you check Usage, do you expect classification and model completion to appear as one spend record or two?

  • Select Usage in the Admin UI navigation menu.
  • Select the benchmark-key in Top Virtual Keys
  • Locate the TypeSafe classifier spend record.
  • Locate the model completion spend record.
  • Record the total spend attributed to benchmark-key.

You should see separate classifier and completion spend records for benchmark-key. Together, they confirm that LiteLLM tracked both parts of the routed workload.

Can't find the spend records?

The x-litellm-classifier-cost header reports the classifier charge on the routed response. Adding it to a Usage total that already includes the classifier record counts the same charge twice.

  • Set the Usage time window to the routed benchmark.
  • Confirm that the records are attributed to benchmark-key.
  • Compare the Usage timestamps with the routed entries in Logs.

Help me trace spend for benchmark-key.

That connects the full request path. You can now explain how routing changed model choice, latency, and total spend.

Secret mission

Prove the Budget Guardrail

Your gateway already records spend. Now prove that you can enforce a hard allowance, block further traffic, and explain the control as a production cost-safety pattern.

Clean Up Your Resources

Clean Up Your Resources

Your Google Cloud stack can continue to incur charges until you destroy it.

Decide whether to keep your resources running, pause your requests to come back later, or delete the stack entirely.

Cost warning

Cloud SQL compute remains billed while the instance runs. Its storage remains billable while it exists.

Cloud Run, Artifact Registry, Vertex AI, and TypeSafe AI remain usage billed. The $1 LiteLLM virtual-key budget limits traffic through that key. Google Cloud billing remains separate from this gateway limit.

Pause keeps the retained infrastructure billable. Delete is the safest choice when your demo is finished.

Resources you used:

  • The Cloud Run service litellm-gateway.
  • The Cloud SQL instance litellm-postgres. It contains the PostgreSQL database litellm plus the persisted router, virtual-key, log, and spend data.
  • The Artifact Registry repository litellm. It contains the image litellm:1.104.0.
  • Four Secret Manager secrets named litellm-master-key, litellm-salt-key, litellm-database-url, and typesafe-api-key.
  • The litellm service account.
  • The IAM bindings that grant runtime access to Vertex AI, Cloud SQL, and Secret Manager.
  • The litellm-gateway folder in Cloud Shell. It contains your Terraform state, configuration files, and benchmark evidence.

Keep everything running

No action is needed. Choose this option while you are actively demonstrating the gateway or continuing to test its routing.

  • Keep the deployed litellm-gateway stack available only while you are using it.
  • Monitor Billing in Google Cloud for continuing infrastructure charges.
  • Monitor Usage in the LiteLLM Admin UI for virtual-key spend.
  • Keep the litellm-gateway folder in Cloud Shell to retain both benchmark CSV files.

Pause - I'll come back to this later

Stop sending requests while keeping the cloud stack and project files ready for a later session. Cloud SQL compute and storage continue to incur charges during this pause.

  • Stop sending benchmark requests through benchmark-key.
  • Leave the Cloud Run minimum instance count at 0.
  • Keep the litellm-gateway folder in Cloud Shell.
  • Monitor Google Cloud Billing while the Cloud SQL instance remains available.

Delete - I don't want to use this again

Remove the deployed stack to stop its continuing infrastructure charges. You will also remove the local project folder after Terraform completes.

Deleting the stack is a big step. The saved Terraform state keeps the teardown scoped to resources managed by this project.

  • Switch back to the Cloud Shell terminal in the litellm-gateway folder from earlier.
  • Destroy the Terraform-managed stack by running this command:
terraform destroy

What does this command remove?

Terraform reads terraform.tfstate to identify the resources created by this project. It presents a destruction plan before deleting anything.

  • The plan removes the Cloud Run service litellm-gateway.
  • The plan removes the Cloud SQL instance litellm-postgres with its persisted LiteLLM data.
  • The plan removes the Artifact Registry repository litellm with its container image.
  • The plan removes the three Terraform-managed Secret Manager secrets.
  • The plan removes the litellm service account plus its Terraform-managed IAM bindings.
  • The TypeSafe secret remains until you delete it with the separate cleanup command below.
  • Review the destruction plan in the terminal.
  • Approve the destruction when Terraform asks for confirmation.

The teardown can take several minutes while Google Cloud removes the managed resources. A quiet terminal during this process does not mean the command has stopped.

  • Confirm the final Terraform summary reports completed destruction with no failed deletions.

That summary confirms the Cloud Run service, Cloud SQL instance, Artifact Registry repository, three Terraform-managed secrets, service account, and IAM bindings were removed.

Terraform destroy not finishing?

  • Confirm the Cloud Shell terminal is still inside the litellm-gateway folder.
  • Confirm your Google Cloud account still has permission to delete the resources from the billed project.
  • Ask for help with the failing resource before repeating the destroy command.

The TypeSafe secret was created during setup, so Terraform leaves it behind. Deleting it is irreversible, and the command asks for confirmation before removing every stored version.

  • Delete the TypeSafe secret by running this command:
gcloud secrets delete typesafe-api-key
  • Confirm the deletion when the Google Cloud CLI prompts you.

You should see confirmation that typesafe-api-key was deleted with all of its versions.

TypeSafe secret was not deleted?

Confirm that Cloud Shell still targets the billed project and that your account can delete Secret Manager resources.

Help me troubleshoot the TypeSafe secret deletion.

The next command permanently removes Terraform state plus both benchmark CSV files from Cloud Shell.

  • Remove the local litellm-gateway folder by running these commands:
cd ..
rm -rf litellm-gateway

What do these commands remove?

The first command moves the terminal to the folder above litellm-gateway. The second command permanently deletes the project folder.

This also removes terraform.tfstate, benchmark-baseline.csv, and benchmark-routed.csv from Cloud Shell.

  • Check the file list in Cloud Shell Editor.

You should no longer see the litellm-gateway folder.

You have closed the billing loop by removing the cloud stack, the separately stored TypeSafe secret, and the local Terraform state.

Nice Work!

Nice Work!

Nice work! You built a live LiteLLM 1.104.0 gateway on Google Cloud Run and turned its routing data into a cost-optimization case study.

You now have a portfolio project grounded in business context, architectural choices, measurable tradeoffs, and a tested financial guardrail.

You've learned how to:

  • Built and deployed a production-style LiteLLM gateway with Docker, Terraform, Cloud SQL for PostgreSQL, and Google Secret Manager.
  • Created an evidence-based cost baseline that exposes when a fixed model spends more than the workload requires.
  • Implemented LiteLLM complexity-aware routing with TypeSafe AI Jev and connected model choice to latency, classifier cost, logs, and virtual-key spend.
  • Secret Mission: Enforced a temporary key's tiny budget, captured the HTTP 422 boundary, and removed the credential after the test.

Ready to quiz yourself?