Deploy an AI Gateway on AWS

Route LLM traffic across Bedrock models with a Terraform-provisioned gateway.

Introduction

30 Second Summary

Every team that uses AI models faces the same problem: each provider has its own SDK, its own authentication, and its own billing dashboard. When you add a second or third model, you double or triple the integration work. You lose visibility into what each team is spending.

In this project, you will deploy a self-hosted AI gateway that sits in front of Amazon Bedrock and exposes a single, unified API endpoint for multiple LLM models. You will provision the entire AWS access layer using Terraform, route requests to different models based on aliases, and track token spend per team.

What You'll Build

You send a request to "fast" and it routes to the cheapest model. You send to "smart" and it routes to a more capable one. One endpoint, multiple models, full cost visibility.

By the end of this project, you'll have:

  • A single curl command that talks to any Bedrock model through your gateway, no AWS SDK required on the client side.
  • A spend tracking dashboard showing exactly how many tokens each request consumed and what it cost, broken down per model and per virtual key.
  • A budget alarm that fires an email if spending ever approaches your $1 threshold, with Terraform able to tear down every AWS resource in one command when you are done.
  • Secret Mission: Add a fallback routing strategy so requests automatically retry on a cheaper model when the primary model is throttled or unavailable.

Are there any prerequisites?

You need an AWS account on a Paid plan, Docker Desktop installed, Terraform v1.15.1 installed, and the AWS CLI configured with credentials. Step 1 walks you through enabling the specific Bedrock model access you need.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters.

Provision AWS Infrastructure with Terraform

Every AI workload needs a secure foundation before it can serve its first request. Without proper IAM policies, encrypted credential storage, and cost controls, even a single misconfigured request could expose credentials or generate surprise bills.

In this step, you will use Terraform to provision an AWS access layer purpose-built for AI workloads. This includes a locked-down IAM user, a secret store for credentials, and a budget alarm that fires before you ever spend a dollar.

In this step, get ready to:
  • Write a Terraform configuration that provisions IAM, Secrets Manager, and a Budgets alarm.
  • Apply the configuration to create all AWS resources in one command.
  • Enable Amazon Bedrock model access for the LLMs your gateway will route to.
Create the Terraform configuration
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir ai-gateway-terraform
cd ai-gateway-terraform
  • Create the main.tf file by running:
touch main.tf
  • Confirm the file exists by running:
ls

You should see main.tf listed in the output.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type VS Code and press Enter to open it.
  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Select the ai-gateway-terraform folder on your Desktop, then click Open.
  • Click on main.tf in the Explorer sidebar to open it.

Your Terraform configuration starts with a provider block that tells Terraform which cloud platform and region to use.

  • Paste the following code into main.tf:
terraform {
  required_version = ">= 1.5.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

resource "aws_iam_user" "litellm_gateway" {
  name = "litellm-bedrock-gateway"
}

What does this code do?

The terraform block pins your project to the AWS provider version ~> 6.0 and requires Terraform 1.5.0 or higher.

The provider block sets your deployment region to us-east-1 where both Bedrock models are available. The aws_iam_user resource creates a dedicated IAM user that your gateway will use to call Bedrock.

Next, you need to give this IAM user permission to invoke Bedrock models. The principle of least privilege means granting only the exact actions and resources the gateway needs.

  • Add the IAM policy below the IAM user resource in main.tf:
resource "aws_iam_user_policy" "bedrock_invoke" {
  name = "bedrock-invoke-only"
  user = aws_iam_user.litellm_gateway.name

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "bedrock:InvokeModel",
          "bedrock:InvokeModelWithResponseStream"
        ]
        Resource = [
          "arn:aws:bedrock:us-east-1::foundation-model/amazon.nova-lite-v1:0",
          "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-haiku-20240307-v1:0"
        ]
      }
    ]
  })
}

What does this policy do?

This inline policy allows only two actions: bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream. No listing models, no creating endpoints, no admin operations.

The Resource array locks access to exactly two model ARNs: Amazon Nova Lite and Claude 3 Haiku. If someone compromises this user, they can only call these two models in us-east-1.

  • Save main.tf.
  • Initialize the Terraform workspace by running this command in your terminal:
terraform init

You should see Terraform download the AWS provider and print the message: Terraform has been successfully initialized!

Seeing an error about provider versions?

Make sure the terraform block at the top of main.tf has no typos in hashicorp/aws or ~> 6.0. Also confirm your terminal is inside the ai-gateway-terraform folder where main.tf lives.

help me fix a Terraform init error

Now you will add the resources that store credentials securely and create an access key for the gateway user.

  • Add the access key and Secrets Manager resources below the policy in main.tf:
resource "aws_iam_access_key" "litellm_gateway" {
  user = aws_iam_user.litellm_gateway.name
}

resource "aws_secretsmanager_secret" "litellm_aws_credentials" {
  name                    = "litellm-gateway/aws-credentials"
  recovery_window_in_days = 0
}

resource "aws_secretsmanager_secret_version" "litellm_aws_credentials" {
  secret_id = aws_secretsmanager_secret.litellm_aws_credentials.id
  secret_string = jsonencode({
    aws_access_key_id     = aws_iam_access_key.litellm_gateway.id
    aws_secret_access_key = aws_iam_access_key.litellm_gateway.secret
  })
}

What does this code do?

The aws_iam_access_key resource generates programmatic credentials for your gateway user. The aws_secretsmanager_secret creates an encrypted vault to store those credentials.

Setting recovery_window_in_days = 0 allows immediate deletion when you run terraform destroy later. Without this, AWS would hold the secret for 7-30 days before deleting it.

  • Add the budget alarm, the email variable, and the outputs below the Secrets Manager resources in main.tf:
resource "aws_budgets_budget" "ai_gateway" {
  name         = "ai-gateway-monthly"
  budget_type  = "COST"
  limit_amount = "1"
  limit_unit   = "USD"
  time_unit    = "MONTHLY"

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 80
    threshold_type             = "PERCENTAGE"
    notification_type          = "FORECASTED"
    subscriber_email_addresses = [var.alert_email]
  }
}

variable "alert_email" {
  description = "Email address for budget alerts"
  type        = string
}

What does the budget alarm do?

This AWS Budgets alarm watches your monthly spend. When the forecasted cost exceeds 80% of your $1 limit, AWS sends an email to the address you provide.

The variable block declares an input that Terraform will ask you for when you apply. This keeps your email address out of the code itself.

  • Run terraform validate to confirm your configuration is syntactically valid:
terraform validate

You should see: Success! The configuration is valid.

  • Finally, add the output blocks at the bottom of main.tf:
output "access_key_id" {
  value     = aws_iam_access_key.litellm_gateway.id
  sensitive = true
}

output "secret_access_key" {
  value     = aws_iam_access_key.litellm_gateway.secret
  sensitive = true
}

output "secrets_manager_arn" {
  value = aws_secretsmanager_secret.litellm_aws_credentials.arn
}

What do outputs do?

Outputs expose values after Terraform finishes applying. You will use the access_key_id and secret_access_key in the next step to configure your AI gateway.

Marking them sensitive = true prevents Terraform from printing the raw values in the terminal unless you explicitly request them.

  • Save main.tf.

✔️ Awesome, I've got everything!

Great. Double check you have saved main.tf with all five sections: provider config, IAM user, IAM policy, access key with Secrets Manager, and budget alarm with outputs.

ⓧ I'd like to double check the full code

Scroll through your main.tf and confirm these sections appear in order:

  • The terraform block with required_version and required_providers for hashicorp/aws.
  • The provider "aws" block with region = "us-east-1".
  • The aws_iam_user resource named litellm-bedrock-gateway.
  • The aws_iam_user_policy resource with bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream actions scoped to both model ARNs.
  • The aws_iam_access_key resource.
  • The aws_secretsmanager_secret and aws_secretsmanager_secret_version resources.
  • The aws_budgets_budget resource with the notification block.
  • The variable "alert_email" block.
  • Three output blocks: access_key_id, secret_access_key, and secrets_manager_arn.
Apply the Terraform configuration

Terraform now has everything it needs to provision your AWS resources. The apply command will show you a plan of what it intends to create, then ask for confirmation before making changes.

  • Create a terraform.tfvars file to provide your email address for budget alerts by running:
echo 'alert_email = "[[ALERT_EMAIL="your-email@example.com"]]"' > terraform.tfvars

What is a .tfvars file?

A .tfvars file supplies values for declared variables so Terraform does not prompt you interactively. This keeps variable values separate from your infrastructure code.

Before you apply, think about what you expect to happen. How many AWS resources do you think Terraform will create?

  • Apply the Terraform configuration by running:
terraform apply

What will Terraform ask?

Terraform shows a plan listing every resource it will create (you should see 5 resources: the IAM user, the policy, the access key, the secret, the secret version, and the budget). It then prompts Do you want to perform these actions?

Type yes and press Enter to confirm.

After a few seconds, Terraform prints Apply complete! with a count of the resources it created.

Now verify that everything was created successfully. What do you expect to see when you query the outputs?

  • Check the Terraform outputs by running:
terraform output

You should see three outputs listed. The access_key_id and secret_access_key show <sensitive> because you marked them as sensitive. The secrets_manager_arn shows the full ARN of your new secret.

Retrieving sensitive output values

You will need the access key values in the next step. To reveal them, run terraform output -raw access_key_id or terraform output -raw secret_access_key. Keep these values private.

Seeing an error during apply?

If you see a permissions error, confirm your AWS CLI credentials have admin or IAM-write permissions. Run aws sts get-caller-identity to verify your identity.

If the budget resource fails, ensure you have not exceeded the free budget limit (2 budgets with no actions are free per AWS account).

help me debug a Terraform apply error

Enable Amazon Bedrock model access

Your IAM user now has permission to invoke Bedrock models, but Amazon Bedrock also requires you to explicitly enable access to each model in the console.

  • Open the AWS Management Console in your browser and sign in.
  • In the top navigation bar, confirm the region is set to US East (N. Virginia) us-east-1.
  • Search for Bedrock in the search bar and select Amazon Bedrock.
  • In the left sidebar, expand Bedrock configurations.
  • Click Model access.
  • Click Enable specific models.
  • Search for and select Amazon Nova Lite and Claude 3 Haiku.
  • Click Next.
  • Click Submit.

Why enable models separately?

IAM controls who can call Bedrock APIs, but model access controls which models are available in your account. Some models (like Claude) require accepting a third-party end-user license agreement before AWS makes them accessible.

After submitting, the model access status for both models should show Access granted within a few seconds.

Model access stuck on pending?

Claude 3 Haiku may require you to fill in a brief use-case form. Complete it and resubmit. Access is usually granted within a minute.

If you do not see the models listed, confirm your region is us-east-1 in the top navigation bar.

help me enable Bedrock model access

Your AWS foundation is locked down and ready. Next up, you will spin up the AI gateway itself and prove it can reach Bedrock through a single API call.

Run the AI Gateway with a Single Model

Your AWS foundation is in place. You have an IAM user with least-privilege permissions, encrypted credentials in Secrets Manager, and a budget alarm standing guard.

Now you need to prove the gateway can actually reach Amazon Bedrock. A single-model setup isolates authentication issues from routing issues. If something goes wrong, you know the problem is credentials, not routing logic.

In this step, get ready to:
  • Create the LiteLLM configuration files and environment variables.
  • Start the gateway containers with Docker Compose.
  • Send a test request to confirm Bedrock responds through the gateway.
Create the gateway configuration files

The gateway needs three things: your AWS credentials, a model routing config, and a Docker Compose file to run everything. You will start by pulling the credentials out of Terraform.

  • Retrieve your AWS access key ID by running this command:
terraform output -raw access_key_id
  • Copy the value that prints. Then retrieve your secret access key:
terraform output -raw secret_access_key
  • Copy this value too. You will paste both into an environment file next.

Why are these outputs marked sensitive?

Terraform hides sensitive outputs from regular terraform output listings. The -raw flag prints the actual value so you can copy it. Never commit these values to version control.

  • Generate a random salt key for LiteLLM by running:
openssl rand -hex 16
  • Copy this value as well. Now create the .env file by running:
touch .env
  • Open .env in your editor and paste the following content, replacing the placeholder values with the real values you just copied:
AWS_ACCESS_KEY_ID=your-access-key-from-terraform-output
AWS_SECRET_ACCESS_KEY=your-secret-key-from-terraform-output
LITELLM_MASTER_KEY=sk-gateway-master-1234
LITELLM_SALT_KEY=a-long-random-string-generate-one

What does each variable do?

  • AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY authenticate the gateway to Amazon Bedrock using the IAM user Terraform created.
  • LITELLM_MASTER_KEY is the admin password for the gateway API. Any request must include this as a Bearer token.
  • LITELLM_SALT_KEY encrypts provider API keys stored in the database. Changing it later prevents decryption of existing keys.
  • Save .env.
  • Create the LiteLLM routing config by running:
touch config.yaml
  • Open config.yaml and paste the following:
model_list:
  - model_name: fast
    litellm_params:
      model: bedrock/amazon.nova-lite-v1:0
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: us-east-1

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  store_model_in_db: true

What does this config do?

The model_list defines a model alias called fast that routes to Amazon Nova Lite in us-east-1. The os.environ/ prefix tells LiteLLM to read values from environment variables instead of hardcoding them.

The general_settings block configures the master key for API authentication, the database connection for tracking usage, and store_model_in_db: true so models can be managed from the Admin UI later.

  • Save config.yaml.
  • Confirm both new files exist by running:
ls

You should see .env, config.yaml, and main.tf in the output.

  • Create the Docker Compose file by running:
touch docker-compose.yml
  • Open docker-compose.yml and paste the following:
services:
  litellm:
    image: docker.litellm.ai/berriai/litellm:latest
    ports:
      - "4000:4000"
    volumes:
      - ./config.yaml:/app/config.yaml
    environment:
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
      - LITELLM_MASTER_KEY=sk-gateway-master-1234
      - LITELLM_SALT_KEY=${LITELLM_SALT_KEY}
      - DATABASE_URL=postgresql://llmproxy:dbpassword9090@db:5432/litellm
    command: ["--config", "/app/config.yaml"]
    depends_on:
      - db

  db:
    image: postgres:16
    environment:
      - POSTGRES_USER=llmproxy
      - POSTGRES_PASSWORD=dbpassword9090
      - POSTGRES_DB=litellm
    volumes:
      - pgdata:/var/lib/postgresql/data

volumes:
  pgdata:

What does this compose file do?

This defines two services. The litellm service runs the AI gateway on port 4000, mounts your config.yaml, and reads credentials from the .env file.

The db service runs PostgreSQL 16. LiteLLM uses this database to store virtual keys, spend logs, and model configurations. The pgdata volume persists data between container restarts.

  • Save docker-compose.yml.

✔️ Awesome, I've got everything!

Great. Make sure all three files are saved before moving on.

ⓧ I'd like to double check the full code

Here is each file in full. Compare yours against these:

AWS_ACCESS_KEY_ID=your-access-key-from-terraform-output
AWS_SECRET_ACCESS_KEY=your-secret-key-from-terraform-output
LITELLM_MASTER_KEY=sk-gateway-master-1234
LITELLM_SALT_KEY=a-long-random-string-generate-one
model_list:
  - model_name: fast
    litellm_params:
      model: bedrock/amazon.nova-lite-v1:0
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: us-east-1

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  store_model_in_db: true
services:
  litellm:
    image: docker.litellm.ai/berriai/litellm:latest
    ports:
      - "4000:4000"
    volumes:
      - ./config.yaml:/app/config.yaml
    environment:
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
      - LITELLM_MASTER_KEY=sk-gateway-master-1234
      - LITELLM_SALT_KEY=${LITELLM_SALT_KEY}
      - DATABASE_URL=postgresql://llmproxy:dbpassword9090@db:5432/litellm
    command: ["--config", "/app/config.yaml"]
    depends_on:
      - db

  db:
    image: postgres:16
    environment:
      - POSTGRES_USER=llmproxy
      - POSTGRES_PASSWORD=dbpassword9090
      - POSTGRES_DB=litellm
    volumes:
      - pgdata:/var/lib/postgresql/data

volumes:
  pgdata:
Start the LiteLLM gateway
  • Start both containers in detached mode by running:
docker compose up -d

Docker pulls the LiteLLM image and the PostgreSQL image, then starts both containers. This may take a minute on the first run.

  • Confirm both containers are running by checking their status:
docker compose ps

You should see two services listed with a State of running: one for litellm and one for db.

  • Verify the gateway is ready to accept traffic by hitting its health endpoint:
curl http://localhost:4000/health/liveliness

You should see the response "I'm alive!". This confirms the LiteLLM process is up and listening on port 4000.

Not seeing both containers running?

  • If the litellm container shows exited, check its logs with docker compose logs litellm. Look for credential or config errors.
  • If db is not running, port 5432 may be in use by a local PostgreSQL install. Stop the local instance or change the compose port mapping.
  • Make sure Docker Desktop is running and has enough memory allocated (at least 4 GB).

Help me troubleshoot my LiteLLM containers not starting

Send a test request to Bedrock

The gateway is up. Time to prove it can actually reach Amazon Bedrock and get a response from Nova Lite through the fast model alias.

Before you run the next command, what do you expect to happen? Will the request succeed and return an AI-generated answer, or will it fail with an authentication error?

  • Send a chat completion request through the gateway by running:
curl http://localhost:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-gateway-master-1234" \
  -d '{"model": "fast", "messages": [{"role": "user", "content": "What is cloud computing in one sentence?"}]}'

You should see a JSON response with choices[0].message.content containing a one-sentence answer about cloud computing, generated by Nova Lite. The response also includes a usage object showing how many tokens were consumed.

What just happened?

Your curl command hit the local gateway at localhost:4000. LiteLLM looked up the fast alias in config.yaml, authenticated to Bedrock with your IAM credentials, and forwarded the request to amazon.nova-lite-v1:0. The response came back through the same path.

The client (your terminal) never touched AWS directly. It only knows about the gateway's OpenAI-compatible API. This is the core value of an AI gateway: one interface, any model behind it.

Getting a 401 or connection refused?

  • A 401 Unauthorized means the Bearer token does not match LITELLM_MASTER_KEY. Confirm your curl header uses sk-gateway-master-1234 exactly.
  • A connection refused means the container is not running or not ready yet. Wait 15 seconds and try again, or check docker compose ps.
  • A Bedrock AccessDeniedException means your IAM credentials do not have invoke permission. Confirm you pasted the correct values from terraform output into .env.

Help me debug my LiteLLM gateway request to Bedrock

Your gateway is live and authenticated to Bedrock. You proved the full path works: local curl to gateway to AWS and back. Next up, you will add a second model so the same gateway can route to both cheap and capable models depending on the request.

Add Multi-Model Routing

Your gateway currently sends every request to the same model, regardless of complexity. A one-word greeting and a multi-paragraph analysis both hit Nova Lite. Simple prompts waste money on expensive models, and complex prompts get poor results from cheap ones.

Multi-model routing solves this by assigning friendly aliases to different Amazon Bedrock models. You send to "fast" for quick tasks and "smart" for harder ones. The gateway decides which foundation model actually receives the request.

In this step, get ready to:
  • Add a second model alias pointing to Claude 3 Haiku.
  • Restart the gateway and confirm both routes are live.
  • Compare response fields to prove traffic hits different models.
Add the "smart" model alias to your configuration

Right now config.yaml has a single entry in model_list. You need a second entry that maps the alias "smart" to Claude 3 Haiku on Bedrock.

  • In config.yaml, find the blank line between the fast model entry and the general_settings block.
  • Add the following second model entry below the fast block:
  - model_name: smart
    litellm_params:
      model: bedrock/anthropic.claude-3-haiku-20240307-v1:0
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: us-east-1

What does this code do?

model_name: smart creates a new alias that clients can reference in their requests.

The litellm_params block tells the gateway which Bedrock model to invoke when it receives a request for "smart". It uses the same AWS credentials and region as your "fast" alias because both models live in the same account.

  • Save config.yaml.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before restarting the gateway.

ⓧ I'd like to double check the full code

Your complete config.yaml should now look like this:

model_list:
  - model_name: fast
    litellm_params:
      model: bedrock/amazon.nova-lite-v1:0
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: us-east-1

  - model_name: smart
    litellm_params:
      model: bedrock/anthropic.claude-3-haiku-20240307-v1:0
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: us-east-1

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  store_model_in_db: true
Restart the gateway

The gateway reads config.yaml at startup. A restart picks up the new "smart" route without tearing down the database container.

  • Restart only the LiteLLM container by running this command:
docker compose restart litellm
  • Wait about 10 seconds for the container to finish restarting.
  • Confirm the gateway is healthy by running:
docker compose ps

You should see both litellm and db containers with a status of Up.

Container exited or shows unhealthy?

Check that your YAML indentation is correct. Each model_name line should start with exactly two spaces followed by a dash. Run docker compose logs litellm --tail 20 to see the exact error.

help me debug my LiteLLM container not starting after I updated config.yaml

Test both model routes

Now you can prove that the same endpoint routes to different models depending on the alias you send.

  • Send a test request to the "fast" alias by running this command:
curl -s http://localhost:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-gateway-master-1234" \
  -d '{
    "model": "fast",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}]
  }'

You should see a JSON response with "model" showing the Nova Lite model ID and a "usage" object showing prompt_tokens, completion_tokens, and total_tokens.

Before you send the next request, take a look at the model field in that response. Do you think the "smart" request will return the same model ID or a different one?

  • Send a test request to the "smart" alias by running this command:
curl -s http://localhost:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-gateway-master-1234" \
  -d '{
    "model": "smart",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}]
  }'

Compare the model field across both responses. The "fast" response shows a Nova Lite model identifier, while the "smart" response shows a Claude 3 Haiku model identifier. The usage object in each response also shows different token counts, because the two models tokenize text differently.

Why do the model fields differ?

The model field in the response tells you which actual foundation model served the request. Your gateway translated the friendly alias ("fast" or "smart") into the real Bedrock model ID behind the scenes.

This is the core value of multi-model routing. Client code never needs to know about provider-specific model IDs. You can swap the underlying model without changing any client code.

Getting a 401 or model not found error on "smart"?

Make sure you enabled model access for anthropic.claude-3-haiku-20240307-v1:0 in the Bedrock console (Step 1). Also verify that the model_name: smart entry in your config.yaml uses the exact model ID bedrock/anthropic.claude-3-haiku-20240307-v1:0.

help me troubleshoot why my smart model alias is returning an error from LiteLLM

You now have a working multi-model gateway. One endpoint, two models, zero SDK changes on the client side. Next up, you will add virtual keys and per-team spend tracking so different consumers get independent budgets.

Add Virtual Keys and Spend Tracking

Your gateway routes requests to multiple models, but every request uses the same master key with no spending limit. In production, different teams consume tokens at different rates and need independent budgets.

Without per-key tracking, you cannot attribute costs to a specific team or prevent one team from exhausting shared capacity. In this step, you will create virtual keys with dollar-denominated budgets and watch the gateway enforce those limits.

In this step, get ready to:
  • Create two virtual keys with independent spend caps.
  • Send requests using team-specific keys instead of the master key.
  • Verify per-key spend tracking and budget enforcement in the Admin UI.
Create virtual keys with budget caps

A virtual key is a proxy API key that LiteLLM issues on top of your master key. Each virtual key can have its own spending limit, model access list, and rate limits.

  • Generate the first virtual key named "team-backend" with a $0.10 max budget by running this command:
curl http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-gateway-master-1234" \
  -H "Content-Type: application/json" \
  -d '{"key_alias": "team-backend", "max_budget": 0.10}'

What does this command do?

The /key/generate endpoint creates a new virtual key with whatever constraints you pass in the body.

The key_alias gives the key a human-readable name for the Usage dashboard. The max_budget sets a hard dollar cap. Once spend crosses this limit, the gateway rejects further requests from that key.

The response includes a key field with your new virtual key (starts with sk-). Copy it somewhere safe. You will use it as a Bearer token in the next substep.

  • Generate the second virtual key named "team-frontend" with a $0.05 max budget by running:
curl http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-gateway-master-1234" \
  -H "Content-Type: application/json" \
  -d '{"key_alias": "team-frontend", "max_budget": 0.05}'
  • Copy the key value from this response as well.

You now have two team keys, each with its own spending ceiling.

  • Confirm both keys exist by opening the LiteLLM Admin UI at http://localhost:4000/ui in your browser.
  • Log in with username admin and password sk-gateway-master-1234.
  • Click Virtual Keys in the left sidebar.

You should see both team-backend and team-frontend listed with their respective max budgets of $0.10 and $0.05.

Don't see your keys listed?

Make sure you used the master key sk-gateway-master-1234 in the Authorization header when generating keys. Only the master key has permission to create virtual keys.

If you see a connection error, confirm your gateway is still running with docker compose ps. Both the litellm and db containers should show a running state.

help me troubleshoot why my virtual keys are not appearing in the LiteLLM Admin UI

Send requests using team virtual keys

Instead of authenticating every request with the master key, each team now uses its own virtual key. The gateway tracks spend per key automatically.

  • Send a request as the backend team by running this command (replace the Bearer token with the team-backend key you copied earlier):
curl http://localhost:4000/chat/completions \
  -H "Authorization: Bearer your-team-backend-key-here" \
  -H "Content-Type: application/json" \
  -d '{"model": "fast", "messages": [{"role": "user", "content": "Summarize what an API gateway does in one sentence."}]}'

You should see a JSON response with choices[0].message.content containing the model's answer. The response also includes a usage object showing prompt_tokens and completion_tokens.

  • Send a request as the frontend team using the team-frontend key and the "smart" model:
curl http://localhost:4000/chat/completions \
  -H "Authorization: Bearer your-team-frontend-key-here" \
  -H "Content-Type: application/json" \
  -d '{"model": "smart", "messages": [{"role": "user", "content": "What is infrastructure as code?"}]}'

Why separate keys per team?

Each key accumulates spend independently. If the frontend team burns through its budget, the backend team is unaffected.

This is the same pattern production AI platforms use to prevent a single runaway service from exhausting a shared token budget.

Both requests should return valid model responses. The model field in each response confirms which Bedrock model served it.

Getting a 401 or invalid key error?

Double-check that you replaced your-team-backend-key-here with the actual key value from the /key/generate response. Virtual keys start with sk- and are different from your master key.

help me debug why my LiteLLM virtual key is being rejected with a 401 error

Verify spend tracking and budget enforcement

The gateway has been quietly logging the cost of every request against each key. Now you will see that spend in the Admin UI and then test what happens when a key exceeds its budget.

  • Switch back to the LiteLLM Admin UI in your browser at http://localhost:4000/ui.
  • Click Usage in the left sidebar.

You should see per-key spend breakdowns showing token counts and estimated cost for the requests you sent. Each key's spend is tracked separately.

Now test budget enforcement. The team-frontend key has a tiny $0.05 budget. A few more requests should push it over the limit.

  • Send several requests using the team-frontend key to exhaust its budget. Run this command 3-4 times:
curl http://localhost:4000/chat/completions \
  -H "Authorization: Bearer your-team-frontend-key-here" \
  -H "Content-Type: application/json" \
  -d '{"model": "smart", "messages": [{"role": "user", "content": "Explain cloud computing in 200 words."}]}'

Before you run it the final time, ask yourself: what do you think will happen once the key's $0.05 budget is exceeded?

Once the key crosses its max_budget, you will see an error response instead of a model answer. The response indicates the budget has been exceeded and the request is rejected. This is the gateway enforcing per-key cost governance.

What does budget enforcement prove?

A single team cannot accidentally rack up unbounded spend. The gateway acts as a financial circuit breaker, cutting off access the moment a key crosses its ceiling.

In production, you would set these budgets per-team or per-application and alert on threshold crossings well before the hard cutoff.

Not seeing the budget error?

Bedrock token costs are very small per request (fractions of a cent). You may need to send 5-10 requests to cross the $0.05 limit, depending on how many tokens each response uses.

Check the Usage page in the Admin UI to see the key's current spend. Once it shows $0.05 or more, the next request will be rejected.

help me understand why my LiteLLM virtual key budget is not being enforced after multiple requests

You have now proved that your gateway can issue independent keys, track per-key spend, and enforce budget limits automatically. Next up, you will extend this gateway with automatic fallback routing so it stays available even when a model is throttled.

Secret mission

Add Fallback Routing

In production, models get throttled or go down. Without fallback logic, your entire AI pipeline fails when one provider hiccups. Can you configure automatic retries so requests gracefully reroute to a backup model when the primary is unavailable?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. The AWS Secrets Manager secret is the only resource with an ongoing cost ($0.40/month). All other AWS resources are free or pay-per-use with no idle charge.

Cost warning

If you leave the Secrets Manager secret running, it costs $0.40/month. The Budgets alarm and IAM user are free. If you are done experimenting, choose Pause or Delete to avoid any charges.

Resources you used:

  • AWS IAM user litellm-bedrock-gateway with Bedrock invoke policy.
  • AWS Secrets Manager secret litellm-gateway/aws-credentials ($0.40/month ongoing).
  • AWS Budgets alarm ai-gateway-monthly (free).
  • Docker containers (LiteLLM gateway and PostgreSQL) with a pgdata volume.
  • Local project files: main.tf, docker-compose.yml, config.yaml, .env.

Keep everything running

No action needed. Choose this if you are still actively building on your gateway or plan to experiment with more models.

  • Your Docker containers stay running on port 4000.
  • The Secrets Manager secret continues at $0.40/month.
  • Bedrock model access remains enabled in your account. Since you deleted the IAM user's access key via Terraform, no requests can be made unless you recreate credentials.

Pause - I'll come back to this later

Shut down running Docker containers to free up local memory and CPU. Your AWS resources remain but incur minimal cost.

  • Stop the Docker containers by running:
docker compose down

This stops the LiteLLM and PostgreSQL containers. The pgdata volume is preserved, so your virtual keys and spend history remain intact when you start up again.

  • To resume later, run docker compose up -d from your project folder.
  • AWS resources stay in place. The Secrets Manager secret still costs $0.40/month.

Delete - I don't want to use this again

Remove all project resources completely. This cannot be undone.

  • Stop Docker containers and remove the database volume by running:
docker compose down -v
  • Destroy all AWS resources (IAM user, Secrets Manager secret, Budgets alarm) by running:
terraform destroy

Terraform prompts you to confirm. Type yes and press Enter. It also asks for your alert email variable again. Enter any value to proceed.

  • Delete the local project files. On macOS/Linux, run:
cd ~/Desktop
rm -rf ai-gateway
  • On Windows, run:
cd ~\Desktop
Remove-Item -Recurse -Force ai-gateway

What about Bedrock model access?

Bedrock model access cannot be disabled once enabled for most foundation models. This is safe to leave as-is. The IAM user that had permission to invoke those models has been deleted by Terraform, so no requests can be made.

Nice Work!

Nice Work!

Nice work! You just built a production-style AI gateway that routes, tracks, and governs LLM traffic across multiple Amazon Bedrock models.

You've learned how to:

  • Provision least-privilege AWS infrastructure for AI workloads using Terraform, including IAM policies scoped to specific Bedrock models, encrypted credential storage in Secrets Manager, and a budget alarm that fires before costs escalate.
  • Deploy LiteLLM as a self-hosted AI gateway with multi-model routing, sending simple prompts to a cheap model and complex prompts to a capable one through a single OpenAI-compatible API endpoint.
  • Implement per-team virtual keys with independent budget caps and spend tracking, attributing token usage and cost to individual consumers of your gateway.
  • Built automatic fallback routing so your gateway retries on a cheaper model when the primary model is throttled, maintaining uptime without manual intervention (Secret Mission).

Ready to quiz yourself?