GenAI Cost Optimization Control Plane
Deploy an AI gateway to route, cache, and govern LLM costs across providers.
Introduction
30 Second Summary
Every time you send a prompt to an AI model, someone is paying for it. As organizations scale from a handful of AI experiments to hundreds of production workflows, those costs can spiral out of control fast.
In this project, you will build an AI Cost Optimization Control Plane. It is a centralized gateway that routes requests across multiple LLM providers, caches repeated responses, tracks every dollar spent, and enforces budget limits. Your architecture directly implements six of Gartner's 10 Best Practices for Optimizing Generative and Agentic AI Costs.
What You'll Build
A production-style AI gateway using LiteLLM that manages multi-provider model routing, response caching, cost tracking, and budget guardrails across Groq and Google Gemini free tiers.
By the end of this project, you'll have:
- A working LiteLLM proxy gateway routing requests across four models from two free-tier providers, with automated model routing, response caching, and budget guardrails.
- A cost comparison report showing per-model spend, cache hit savings, and a normalized cost-per-quality analysis grounded in Gartner's Practice 1.
- A prompt optimization toolkit that quantifies token savings from efficient prompting, implementing Gartner's Practice 9.
- A 97-mission secret challenge catalog with a scoring framework and 5 fully playable missions that unlock based on your cost optimization performance.
Are there any prerequisites?
You should be comfortable with APIs, Python, Docker, and basic LLM concepts. You will also need free API keys from Groq and Google Gemini. Neither requires a credit card or billing setup.
Set Up Your AI Gateway Foundation
Every enterprise AI system needs a centralized control layer that sits between your applications and the LLM providers they call.
Without one, each team picks its own model, builds its own integration, and racks up costs nobody can see.
In this step, you will install LiteLLM as your AI gateway, register models from two free-tier providers (Groq and Google Gemini), and send your first routed request through the gateway.
In This Step, Get Ready To:
- Install LiteLLM and verify your Groq and Gemini API keys are working.
- Build a multi-provider config.yaml that acts as your AI model sandbox.
- Start the gateway and confirm requests route correctly to both providers.
Install LiteLLM and Verify Your API Keys
LiteLLM is an open-source AI gateway (MIT license) that gives you a single OpenAI-compatible API in front of 100+ LLM providers. Your applications talk to LiteLLM, and LiteLLM handles the provider-specific details for Groq, Gemini, and any other provider you add later.
- Install the LiteLLM proxy package by running the following command in your terminal.
pip install 'litellm[proxy]'Why LiteLLM over other AI gateways?
Gartner's report names several AI gateway vendors (Airia, F5, Helicone, Kong, Portkey, TrueFoundry). LiteLLM is the best fit for this project because it is fully open-source, requires no account or external service to run, and has built-in cost tracking, caching, routing, and budget management.
Other gateways like Portkey and Helicone require accounts and have usage limits on their free tiers.
- Verify the installation succeeded.
litellm --versionWhat should I see?
You should see a version number printed to the terminal. Make sure you are on version 1.83 or higher.
Versions 1.82.7 and 1.82.8 had known issues, so avoid those.
Now you need to set your API keys as environment variables so LiteLLM can authenticate with each provider.
- Set your Groq API key as an environment variable.
export GROQ_API_KEY="your-groq-api-key-here"- Set your Gemini API key as an environment variable.
export GEMINI_API_KEY="your-gemini-api-key-here"Where do I get these keys?
Your Groq API key comes from the Groq console. It starts with gsk_. No credit card is required.
Your Gemini API key comes from Google AI Studio. No billing setup is required for the free tier.
- Verify your Groq key works by sending a quick test request with LiteLLM's CLI.
litellm --model groq/llama-3.1-8b-instant --test- Verify your Gemini key works.
litellm --model gemini/gemini-2.5-flash-preview-09-2025 --testBoth commands should return a successful LLM response in your terminal.
Getting an authentication error?
Double-check that you copied the full API key with no trailing spaces. Make sure you exported the variables in the same terminal session where you are running the litellm command.
If you are still stuck, help me debug my API key authentication error with LiteLLM.
Create Your Multi-Provider Config File
Gartner's Practice 2 recommends creating an AI model sandbox that gives teams access to multiple models with full price transparency. Your config.yaml file is that sandbox. It registers models from both providers, sets rate limits matching each provider's free tier, and defines a master key for gateway authentication.
- Create a new file called config.yaml in your project directory and add the following configuration.
model_list:
# High capability model - Groq free tier
- model_name: llama-70b
litellm_params:
model: groq/llama-3.3-70b-versatile
api_key: os.environ/GROQ_API_KEY
rpm: 30 # Groq free tier: 30 requests per minute
# Lightweight, fast model - Groq free tier
- model_name: llama-8b
litellm_params:
model: groq/llama-3.1-8b-instant
api_key: os.environ/GROQ_API_KEY
rpm: 30
# Mid-tier model - Gemini free tier
- model_name: gemini-flash
litellm_params:
model: gemini/gemini-2.0-flash
api_key: os.environ/GEMINI_API_KEY
rpm: 10 # Gemini free tier: 10 requests per minute
# Latest model - Gemini free tier
- model_name: gemini-flash-25
litellm_params:
model: gemini/gemini-2.5-flash-preview-09-2025
api_key: os.environ/GEMINI_API_KEY
rpm: 10
general_settings:
master_key: sk-1234 # Gateway authentication key
litellm_settings:
drop_params: true # Silently drop unsupported params per providerWhat does this configuration do?
The model_list section registers four models across two providers. Each entry maps a simple user-facing name (like llama-70b) to the provider-specific model string (like groq/llama-3.3-70b-versatile).
The rpm values match each provider's free tier limits. Groq allows 30 requests per minute. Gemini allows 10.
The master_key secures your gateway. It must start with sk-.
What is model routing?
Model routing is the process of directing an LLM request to a specific model based on rules you define. Your applications send requests to the gateway using the simple model name. The gateway looks up which provider and model string to use, then forwards the request.
Your applications never need to know which provider is behind the scenes.
Gemini 2.0 Flash deprecation notice
Gemini 2.0 Flash is scheduled for shutdown on June 1, 2026. Gemini 2.5 Flash is the recommended replacement.
This project includes both models so you can compare them. If you are working through this project after June 2026, replace gemini/gemini-2.0-flash with another available Gemini model from Google AI Studio.
Start the Gateway and Send Your First Routed Request
With your config file ready, you can start the LiteLLM proxy. It launches a local server that exposes an OpenAI-compatible API on port 4000.
- Start the LiteLLM proxy with your config file.
litellm --config config.yamlWhat should I see?
You should see startup logs showing your four models being loaded, followed by a message that the proxy is running on http://0.0.0.0:4000.
Keep this terminal window open. The proxy needs to stay running while you send requests.
Now open a second terminal window to send test requests. Your gateway is live and ready to route traffic.
- Send a test request to the Groq-backed llama-70b model through the gateway.
curl -i http://0.0.0.0:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "llama-70b",
"messages": [{"role": "user", "content": "What is an AI gateway in one sentence?"}]
}'What does this command do?
The -i flag tells curl to include response headers in the output. This is important because LiteLLM returns the calculated cost of each request in the x-litellm-response-cost header.
The Authorization: Bearer sk-1234 header authenticates with the master key you set in your config.
Notice you are requesting llama-70b (your simple name), not groq/llama-3.3-70b-versatile (the provider-specific name). The gateway handles the translation.
- Send a second test request, this time to the Gemini-backed gemini-flash-25 model.
curl -i http://0.0.0.0:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "gemini-flash-25",
"messages": [{"role": "user", "content": "What is an AI gateway in one sentence?"}]
}'Both requests should return a successful JSON response with the model's answer. Look at the response headers for each request. You should see the x-litellm-response-cost header showing the calculated cost of that request.
This is Gartner's Practice 7 in action. You now have automated model routing through a single API endpoint, with built-in cost visibility on every request.
Getting a connection refused error?
Make sure the LiteLLM proxy is still running in your first terminal window. If the proxy crashed, check the logs for error messages.
Common causes include malformed YAML (check indentation) or missing API key environment variables in the terminal session running the proxy.
If you need more help, help me troubleshoot my LiteLLM proxy startup.
Your AI gateway is live and routing requests to two different LLM providers through a single endpoint. Next up, you will add cost tracking and budget guardrails so you can see exactly where money is going and stop it from going too far.
Implement Cost Tracking and Budget Guardrails
Your gateway is routing requests across two providers. Now you need to answer the critical question: how much is each request actually costing you?
Gartner's Practice 10 warns that GenAI costs split into visible costs (inference charges, token usage) and hidden costs (wasted tokens, redundant calls, uncontrolled spending). In this step, you'll build the tracking and guardrails that make both types visible and controllable.
In This Step, Get Ready To:
- Capture per-request cost data from the gateway and log it to a local file.
- Set budget guardrails that reject requests when spending exceeds a threshold.
- Build a terminal-based cost dashboard that breaks down spending by model.
Enable Spend Tracking With a Cost Logger Script
LiteLLM automatically calculates the cost of every request and returns it in the x-litellm-response-cost response header. Without a database, this cost data only lives in the response headers. You need to capture it yourself.
You'll build a Python script that sends requests to your gateway and logs the cost data from each response into a local JSON file. This becomes your cost ledger.
- Make sure your LiteLLM proxy is still running from Step 1. If not, restart it:
litellm --config config.yaml- Create a new file called cost_logger.py and add the following code:
import requests
import json
import os
from datetime import datetime
# Gateway connection settings
GATEWAY_URL = "http://0.0.0.0:4000/chat/completions"
API_KEY = "sk-1234" # Your master key from config.yaml
LOG_FILE = "cost_log.json"
# Load existing log or start fresh
if os.path.exists(LOG_FILE):
with open(LOG_FILE, "r") as f:
cost_log = json.load(f)
else:
cost_log = []
def send_and_log(model, prompt):
"""Send a request to the gateway and log the cost data."""
headers = {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
}
payload = {
"model": model,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 150
}
response = requests.post(GATEWAY_URL, headers=headers, json=payload)
response_data = response.json()
# Extract cost from response header
cost = float(response.headers.get("x-litellm-response-cost", 0))
# Extract token usage from response body
usage = response_data.get("usage", {})
# Build the log entry
entry = {
"timestamp": datetime.now().isoformat(),
"model": model,
"prompt": prompt[:80],
"cost": cost,
"input_tokens": usage.get("prompt_tokens", 0),
"output_tokens": usage.get("completion_tokens", 0),
"total_tokens": usage.get("total_tokens", 0)
}
cost_log.append(entry)
print(f" Model: {model} | Cost: ${cost:.6f} | Tokens: {entry['total_tokens']}")
return entry
# Define test requests across different models
test_requests = [
("groq-llama-70b", "What is cloud computing in one sentence?"),
("groq-llama-8b", "What is cloud computing in one sentence?"),
("gemini-flash", "What is cloud computing in one sentence?"),
("gemini-flash-preview", "What is cloud computing in one sentence?"),
("groq-llama-70b", "Explain containerization briefly."),
("groq-llama-8b", "Explain containerization briefly."),
("gemini-flash", "Explain containerization briefly."),
("gemini-flash-preview", "Explain containerization briefly."),
]
print("Sending requests and logging costs...\n")
for model, prompt in test_requests:
send_and_log(model, prompt)
# Save the log to disk
with open(LOG_FILE, "w") as f:
json.dump(cost_log, f, indent=2)
print(f"\nLogged {len(test_requests)} requests to {LOG_FILE}")
print(f"Total entries in log: {len(cost_log)}")What does this script do?
This script sends the same two prompts to all four of your configured models. For each response, it extracts the cost from the x-litellm-response-cost header and the token usage from the response body. It saves everything to a cost_log.json file that accumulates data across multiple runs.
Why capture costs into a local file?
LiteLLM's full spend tracking features (per-key budgets, per-team reporting) require a PostgreSQL database. Without one, cost data only exists in the response headers of each individual request. By logging to a JSON file, you create a lightweight cost ledger that you can analyze later without any database setup.
- Open a new terminal window (keep the proxy running in the first one) and run the script:
python cost_logger.pyYou should see output showing the cost and token count for each request. The model names here (like groq-llama-70b) are the user-facing names you defined in your config.yaml in Step 1.
Pro tip: watch the rate limits
Groq's free tier allows 30 requests per minute but only 1,000 requests per day for the Llama 3.3 70B model. Gemini 2.0 Flash allows 10 requests per minute and 250 per day. If you get rate limit errors, wait a minute and run the script again. The script sends 8 requests total, well within these limits.
- Verify that cost_log.json was created in your project directory.
- Open it and confirm it contains entries with timestamps, model names, costs, and token counts.
Set Budget Limits and Rate Controls
Cost tracking tells you what you've spent. Budget guardrails prevent you from spending more than you should. This is Gartner's Practice 5 in action: proactively managing your AI spend rather than discovering overruns after the fact.
You'll configure a simulated daily budget of $0.10. When total spend exceeds this threshold, the gateway will reject new requests with a clear error message. On free-tier providers the actual dollar amounts are tiny, but the enforcement mechanism is identical to what you'd use at enterprise scale.
- Stop the LiteLLM proxy (press Ctrl+C in the terminal where it's running).
- Open your config.yaml and add a general_settings section with budget configuration:
general_settings:
max_budget: 0.10 # Maximum daily budget in USD
budget_duration: "1d" # Reset budget every day
alerting:
- "log" # Log alerts to terminal when budget thresholds are hitWhat does this configuration do?
The max_budget setting tells LiteLLM to track cumulative spend and reject any request that would push the total past $0.10. The budget_duration of 1d resets the counter every 24 hours. In production, you'd set this to match your billing cycle.
- Restart the proxy with the updated config:
litellm --config config.yamlNow test that the budget enforcement actually works. You'll temporarily set a very low budget to trigger the rejection.
- Stop the proxy again.
- Change max_budget to 0.0001 in your config.yaml (a fraction of a cent).
- Restart the proxy.
- Send two requests using curl:
curl http://0.0.0.0:4000/chat/completions \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
"model": "groq-llama-70b",
"messages": [{"role": "user", "content": "What is Kubernetes?"}]
}'The first request should succeed. Run the same command again. The second request should be rejected with a budget exceeded error because the first request's cost already surpassed the $0.0001 limit.
What should I see?
When the budget is exceeded, LiteLLM returns an error response with a message indicating that the proxy budget has been reached. The HTTP status code will be 400 or 429. This is exactly how budget enforcement would work in production to prevent cost overruns.
Not seeing a budget rejection?
Budget enforcement depends on LiteLLM's internal cost calculation. If your model's cost is calculated as $0.00 (which can happen with some free-tier models), the budget will never be exceeded. Try sending more requests or temporarily switching to a model with known pricing. Help me troubleshoot budget enforcement in LiteLLM.
- Once you've confirmed the budget rejection works, stop the proxy.
- Change max_budget back to 0.10 so you can continue the project without hitting the limit.
- Restart the proxy.
Build a Cost Dashboard Script
Raw cost data in a JSON file is useful, but a dashboard makes it actionable. Gartner's Practice 10 emphasizes analyzing both visible costs (what you pay per request) and hidden costs (wasted tokens, inefficient model choices). Your dashboard will surface both.
- Create a new file called cost_dashboard.py:
import json
import os
from collections import defaultdict
LOG_FILE = "cost_log.json"
# Load the cost log
if not os.path.exists(LOG_FILE):
print(f"No cost log found at {LOG_FILE}. Run cost_logger.py first.")
exit(1)
with open(LOG_FILE, "r") as f:
cost_log = json.load(f)
print("=" * 65)
print(" AI COST OPTIMIZATION DASHBOARD")
print(" Gartner Practice 10: Analyze Visible and Hidden Costs")
print("=" * 65)
# Aggregate costs by model
model_stats = defaultdict(lambda: {
"total_cost": 0.0,
"request_count": 0,
"input_tokens": 0,
"output_tokens": 0,
"total_tokens": 0
})
for entry in cost_log:
model = entry["model"]
model_stats[model]["total_cost"] += entry["cost"]
model_stats[model]["request_count"] += 1
model_stats[model]["input_tokens"] += entry.get("input_tokens", 0)
model_stats[model]["output_tokens"] += entry.get("output_tokens", 0)
model_stats[model]["total_tokens"] += entry.get("total_tokens", 0)
# Per-model breakdown (Visible Costs)
print("\n--- PER-MODEL SPEND BREAKDOWN (Visible Costs) ---\n")
print(f"{'Model':<28} {'Requests':>8} {'Cost ($)':>10} {'Avg Cost':>10} {'Tokens':>8}")
print("-" * 65)
total_cost = 0.0
total_requests = 0
total_tokens = 0
for model, stats in sorted(model_stats.items()):
avg_cost = stats["total_cost"] / stats["request_count"] if stats["request_count"] > 0 else 0
print(f"{model:<28} {stats['request_count']:>8} {stats['total_cost']:>10.6f} {avg_cost:>10.6f} {stats['total_tokens']:>8}")
total_cost += stats["total_cost"]
total_requests += stats["request_count"]
total_tokens += stats["total_tokens"]
print("-" * 65)
print(f"{'TOTAL':<28} {total_requests:>8} {total_cost:>10.6f} {'':>10} {total_tokens:>8}")
# Token efficiency analysis (Hidden Costs)
print("\n--- TOKEN EFFICIENCY ANALYSIS (Hidden Costs) ---\n")
for model, stats in sorted(model_stats.items()):
if stats["total_tokens"] > 0:
cost_per_1k_tokens = (stats["total_cost"] / stats["total_tokens"]) * 1000
output_ratio = stats["output_tokens"] / stats["total_tokens"] * 100
print(f"{model}:")
print(f" Cost per 1K tokens: ${cost_per_1k_tokens:.6f}")
print(f" Input tokens: {stats['input_tokens']:>6} | Output tokens: {stats['output_tokens']:>6}")
print(f" Output ratio: {output_ratio:.1f}% (higher = more value per request)")
print()
# Monthly projection
print("--- PROJECTED MONTHLY COST ---\n")
if total_requests > 0:
avg_cost_per_request = total_cost / total_requests
# Project based on 100 requests/day enterprise usage
daily_projection = avg_cost_per_request * 100
monthly_projection = daily_projection * 30
print(f" Current avg cost/request: ${avg_cost_per_request:.6f}")
print(f" At 100 requests/day: ${daily_projection:.4f}/day")
print(f" Projected monthly cost: ${monthly_projection:.4f}/month")
print(f" Projected annual cost: ${monthly_projection * 12:.4f}/year")
print("\n" + "=" * 65)
print(" Gartner Insight: Inference costs account for at least 70% of")
print(" total model lifetime costs. Track them continuously.")
print("=" * 65)What does this script do?
The dashboard reads your cost_log.json and produces three reports. The first is a per-model spend breakdown showing total cost, average cost per request, and token usage for each model. This surfaces your visible costs.
The second is a token efficiency analysis that calculates cost per 1,000 tokens and the output-to-total token ratio for each model. A low output ratio means you're spending tokens on input but not getting much output back. This is a hidden cost.
The third is a monthly projection that extrapolates your current usage to enterprise scale (100 requests per day). This helps you understand what costs would look like in production.
- Run the dashboard:
python cost_dashboard.pyYou should see a formatted table showing each model's cost and token usage, followed by the efficiency analysis and monthly projections. Notice the differences between models. The 8B parameter model should be significantly cheaper per request than the 70B model, even for the same prompt.
What are visible vs. hidden costs?
Gartner's Practice 10 distinguishes between costs you can see on a bill (per-token charges, API fees) and costs that are harder to spot. Hidden costs include wasted tokens from verbose prompts, using an expensive model for a simple task, and redundant API calls that could have been cached. Your dashboard makes both types visible in one place.
You now have a complete cost governance layer: logging captures every dollar spent, budget guardrails prevent overruns, and your dashboard makes spending patterns visible. Next up, you'll add caching to your gateway and measure exactly how much money it saves.
Enable Caching and Measure Savings
In the previous step, you built cost tracking and budget guardrails to monitor and control your gateway spend. Now you are going to tackle one of the most impactful cost optimization techniques available: response caching. Gartner's Practice 7 specifically calls out caching as a key lever for reducing inference costs. The idea is simple. If someone asks the same question twice within a short window, why pay for a second LLM call when you already have the answer?
In This Step, Get Ready To:
- Configure in-memory caching in your LiteLLM gateway with a 300-second TTL.
- Run a cache effectiveness experiment comparing cached vs. uncached response times and costs.
- Test dynamic per-request cache controls to balance freshness vs. cost.
Configure In-Memory Caching
LiteLLM supports multiple cache backends, including Redis for production deployments. For this project, you will use the local in-memory cache type, which requires no external dependencies. The cache key is generated from the combination of model, messages, and parameters. Any identical request within the TTL window returns the cached response at zero inference cost.
- Open your config.yaml file.
- Add the following caching configuration under litellm_settings:
litellm_settings:
cache: True
cache_params:
type: local
ttl: 300What does this configuration do?
Setting cache: True enables LiteLLM's response caching layer. The type: local parameter tells it to store cached responses in memory (no Redis or external database needed). The ttl: 300 sets a 300-second (5-minute) time-to-live. Any identical request within that window returns the cached response instantly at zero cost.
- Stop your running LiteLLM proxy (press Ctrl+C in the terminal where it is running).
- Restart the proxy with the updated config:
litellm --config config.yaml- Verify caching is active by sending a test ping to the cache debug endpoint:
curl http://0.0.0.0:4000/cache/pingWhat should I see?
You should see a response confirming the cache is active. If you see an error, double-check that your litellm_settings section is properly indented in your YAML file.
Run a Cache Effectiveness Experiment
Now you will measure exactly how much caching saves. You will create a Python script that sends the same prompts twice. The first pass hits the live LLM API. The second pass should return cached responses at zero cost with significantly faster response times.
- Create a new file called cache_experiment.py.
- Add the following code:
import requests
import time
import json
# Gateway configuration
BASE_URL = "http://0.0.0.0:4000/chat/completions"
HEADERS = {
"Content-Type": "application/json",
"Authorization": "Bearer sk-1234" # Replace with your master key
}
# Five test prompts to send twice each
TEST_PROMPTS = [
"What is the capital of France?",
"Explain what an API gateway does in one sentence.",
"What is the difference between RAM and ROM?",
"Name three programming languages used for data science.",
"What does the HTTP status code 404 mean?"
]
def send_request(prompt, model="groq/llama-3.1-8b-instant"):
"""Send a chat completion request and capture timing and cost."""
payload = {
"model": model,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 100
}
start_time = time.time()
response = requests.post(BASE_URL, headers=HEADERS, json=payload)
elapsed = time.time() - start_time
if response.status_code != 200:
print(f"Error: {response.status_code} - {response.text}")
return None
# Extract cost from response header
cost = float(response.headers.get("x-litellm-response-cost", 0))
cache_hit = "x-litellm-cache-key" in response.headers
return {
"prompt": prompt[:50],
"time_seconds": round(elapsed, 3),
"cost": cost,
"cache_hit": cache_hit
}
def run_experiment():
"""Run prompts twice and compare cached vs uncached results."""
print("=" * 60)
print("CACHE EFFECTIVENESS EXPERIMENT")
print("=" * 60)
# First pass: uncached (hits the LLM API)
print("\n--- Pass 1: Uncached Requests ---")
uncached_results = []
for prompt in TEST_PROMPTS:
result = send_request(prompt)
if result:
uncached_results.append(result)
print(f" {result['prompt']:<50} | {result['time_seconds']:.3f}s | ${result['cost']:.6f}")
time.sleep(2) # Respect rate limits
# Brief pause before second pass
print("\nWaiting 3 seconds before cached pass...")
time.sleep(3)
# Second pass: should be cached
print("\n--- Pass 2: Cached Requests ---")
cached_results = []
for prompt in TEST_PROMPTS:
result = send_request(prompt)
if result:
cached_results.append(result)
hit_label = "CACHE HIT" if result['cache_hit'] else "MISS"
print(f" {result['prompt']:<50} | {result['time_seconds']:.3f}s | ${result['cost']:.6f} | {hit_label}")
time.sleep(1)
# Calculate savings
if uncached_results and cached_results:
total_uncached_cost = sum(r['cost'] for r in uncached_results)
total_cached_cost = sum(r['cost'] for r in cached_results)
total_uncached_time = sum(r['time_seconds'] for r in uncached_results)
total_cached_time = sum(r['time_seconds'] for r in cached_results)
cost_savings = total_uncached_cost - total_cached_cost
time_savings = total_uncached_time - total_cached_time
savings_pct = (cost_savings / total_uncached_cost * 100) if total_uncached_cost > 0 else 0
print("\n" + "=" * 60)
print("RESULTS SUMMARY")
print("=" * 60)
print(f" Uncached total cost: ${total_uncached_cost:.6f}")
print(f" Cached total cost: ${total_cached_cost:.6f}")
print(f" Cost savings: ${cost_savings:.6f} ({savings_pct:.1f}%)")
print(f" Uncached total time: {total_uncached_time:.3f}s")
print(f" Cached total time: {total_cached_time:.3f}s")
print(f" Time savings: {time_savings:.3f}s")
print("=" * 60)
# Save results to file for later analysis
report = {
"uncached": uncached_results,
"cached": cached_results,
"summary": {
"cost_savings_dollars": cost_savings,
"cost_savings_percent": savings_pct,
"time_savings_seconds": time_savings
}
}
with open("cache_results.json", "w") as f:
json.dump(report, f, indent=2)
print("\nResults saved to cache_results.json")
if __name__ == "__main__":
run_experiment()What does this script do?
The script sends 5 prompts to your gateway in two passes. The first pass forces each request through the live LLM API, recording response time and cost. The second pass sends the exact same prompts, which should return cached responses. It then compares the two passes and calculates your total cost and time savings.
The script checks for the x-litellm-cache-key response header to confirm whether each response was served from cache.
- Run the experiment:
python cache_experiment.pyWhat should I see?
Pass 1 should show response times of a few hundred milliseconds to a few seconds, with small cost values for each request. Pass 2 should show dramatically faster response times (often under 50ms) and $0.000000 costs. The results summary should show close to 100% cost savings on the cached pass.
Not seeing cache hits on Pass 2?
Make sure your config.yaml has the caching configuration properly indented under litellm_settings. Also confirm you restarted the proxy after adding the caching config. Help me debug my caching setup.
Test Dynamic Cache Controls
A blanket caching policy does not work for every use case. Some requests need fresh responses every time (like real-time data queries), while others are perfectly fine being cached for hours. LiteLLM supports per-request cache controls that let teams make this trade-off on a call-by-call basis.
- First, send a request that forces a fresh response by bypassing the cache. Run this curl command:
curl -s http://0.0.0.0:4000/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "groq/llama-3.1-8b-instant",
"messages": [{"role": "user", "content": "What is the capital of France?"}],
"max_tokens": 50,
"cache": {"no-cache": true}
}'What does this command do?
The "no-cache": true parameter tells LiteLLM to skip the cache entirely and send the request directly to the LLM provider. This is useful when you need a guaranteed fresh response, such as when querying about time-sensitive information.
- Next, send a request with a custom shorter TTL to demonstrate per-request cache windows:
curl -s http://0.0.0.0:4000/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "groq/llama-3.1-8b-instant",
"messages": [{"role": "user", "content": "Explain caching in one sentence."}],
"max_tokens": 50,
"cache": {"ttl": 60}
}'What does this command do?
The "ttl": 60 parameter overrides the default 300-second cache window for this specific request, setting it to just 60 seconds instead. This lets teams set shorter cache windows for data that changes more frequently, while keeping longer defaults for stable responses.
LiteLLM also supports no-store (do not store the response in cache at all), s-maxage (cache revalidation window), and namespace (isolate cache entries by team or use case). These dynamic controls are what make caching practical in production. Different teams and different request types can each have their own caching strategy, all managed through the same gateway.
Pro tip: cache strategy by use case
In production, you would set long TTLs (hours or days) for stable tasks like FAQ responses and document summarization. You would set short TTLs (seconds to minutes) for tasks that depend on recent data. And you would use no-cache for anything that must always be fresh, like real-time analysis or safety-critical decisions.
Your gateway now has intelligent caching that eliminates redundant inference costs. This is one of the highest-impact cost optimizations you can implement. Gartner's research shows that inference costs make up at least 70% of total model lifetime costs, and caching directly reduces that number. Next up, you will run a structured model cost-quality comparison experiment to implement Gartner's Practice 1.
Run a Model Cost-Quality Comparison Experiment
You have a gateway that tracks costs, enforces budgets, and caches responses. Now it is time to answer the most important question in enterprise AI: which model gives you the best results for the least money? Gartner's Practice 1 warns that organizations must normalize model prices for apples-to-apples comparison. A model that costs twice as much is not a problem if it delivers three times the quality. A model that costs half as much is not a bargain if it produces unusable output. In this step, you will build a structured benchmark that measures both cost and quality across all four of your configured models, then generate a report that maps your findings directly to Gartner's best practices.
In This Step, Get Ready To:
- Design and run a multi-model benchmark across five complexity levels.
- Score responses and calculate a normalized cost-per-quality-point metric.
- Generate a Gartner-aligned cost analysis report with model recommendations.
Design the Benchmark Script
A meaningful benchmark needs prompts that test different capabilities. A model that excels at simple factual questions might struggle with reasoning tasks. By testing across multiple complexity levels, you get a complete picture of each model's strengths and weaknesses relative to its cost.
- Create a new file called model_benchmark.py.
- Add the following code to define your test prompts and benchmark runner.
import requests
import json
import time
GATEWAY_URL = "http://0.0.0.0:4000/chat/completions"
API_KEY = "sk-1234" # Your master key from config.yaml
# Five test prompts spanning different complexity levels
TEST_PROMPTS = [
{
"category": "Factual Q&A",
"prompt": "What is the capital of Australia and when was it established?"
},
{
"category": "Summarization",
"prompt": "Summarize the key differences between SQL and NoSQL databases in 3 bullet points."
},
{
"category": "Reasoning",
"prompt": "A train leaves Station A at 9:00 AM traveling at 60 mph. Another train leaves Station B, 180 miles away, at 10:00 AM traveling at 90 mph toward Station A. At what time do they meet?"
},
{
"category": "Creative Writing",
"prompt": "Write a 4-line poem about a cloud engineer debugging at 3 AM."
},
{
"category": "Code Generation",
"prompt": "Write a Python function that takes a list of integers and returns the second largest unique value. Include error handling."
}
]
# The four models configured in your gateway
MODELS = [
"groq-llama-70b",
"groq-llama-8b",
"gemini-flash",
"gemini-flash-preview"
]
def benchmark_model(model, prompt_data):
"""Send a prompt to a model and capture cost, timing, and token data."""
headers = {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
}
payload = {
"model": model,
"messages": [{"role": "user", "content": prompt_data["prompt"]}],
"max_tokens": 300
}
start_time = time.time()
response = requests.post(GATEWAY_URL, headers=headers, json=payload)
elapsed = time.time() - start_time
if response.status_code != 200:
print(f" Error with {model}: {response.status_code} - {response.text[:100]}")
return None
data = response.json()
cost = float(response.headers.get("x-litellm-response-cost", 0))
usage = data.get("usage", {})
return {
"model": model,
"category": prompt_data["category"],
"response": data["choices"][0]["message"]["content"],
"response_time": round(elapsed, 2),
"input_tokens": usage.get("prompt_tokens", 0),
"output_tokens": usage.get("completion_tokens", 0),
"cost": cost
}
def run_benchmark():
"""Run all prompts against all models and collect results."""
results = []
for prompt_data in TEST_PROMPTS:
print(f"\nTesting category: {prompt_data['category']}")
for model in MODELS:
print(f" Sending to {model}...")
result = benchmark_model(model, prompt_data)
if result:
results.append(result)
# Pause between requests to respect free tier rate limits
time.sleep(3)
return results
if __name__ == "__main__":
print("Starting model benchmark...")
print("This will send 20 requests (5 prompts x 4 models).")
print("Estimated time: ~2 minutes (with rate limit pauses).\n")
all_results = run_benchmark()
# Save raw results for the next substep
with open("benchmark_results.json", "w") as f:
json.dump(all_results, f, indent=2)
print(f"\nBenchmark complete. {len(all_results)} results saved to benchmark_results.json")What does this script do?
The script sends each of the five test prompts to all four models in your gateway. For every request, it captures the response content, response time, token usage (input and output), and the cost from the x-litellm-response-cost header. A 3-second pause between requests keeps you within Groq's 30 RPM and Gemini's 10 RPM free tier limits.
- Make sure your LiteLLM proxy is still running with your config.yaml.
- Run the benchmark script.
python model_benchmark.pyThe script takes about 2 minutes to complete. You will see each model and category printed as it runs. When it finishes, a benchmark_results.json file appears in your project directory with all the raw data.
Seeing Rate Limit Errors?
Groq's free tier allows 30 requests per minute and 1,000 requests per day. Gemini's free tier allows 10 requests per minute for most models. If you hit rate limits, increase the time.sleep(3) value to time.sleep(7) in the script. Help me debug rate limit errors from my LiteLLM gateway.
Score and Normalize Results
Raw cost numbers alone do not tell you which model is the best choice. A model that costs $0.001 per request but produces a one-sentence answer is not cheaper than a model that costs $0.003 but delivers a complete, accurate response. Gartner's Practice 1 specifically recommends normalizing model prices so you can make apples-to-apples comparisons. You will now add a scoring function and calculate a "cost per quality point" metric for each model.
- Create a new file called score_results.py.
- Add the following scoring and analysis code.
import json
def load_results():
"""Load the raw benchmark results from the previous step."""
with open("benchmark_results.json", "r") as f:
return json.load(f)
def score_response(result):
"""Score a response on a 1-5 quality scale using heuristics.
Scoring criteria:
- Length: Does the response have enough substance? (1 point)
- Completeness: Does it address the full prompt? (1 point)
- Structure: Is it well-organized with clear formatting? (1 point)
- Detail: Does it include specific details or examples? (1 point)
- Coherence: Is the response logically consistent? (1 point)
"""
response = result["response"]
score = 0
# Length check: at least 50 characters indicates a substantive response
if len(response) > 50:
score += 1
# Completeness: at least 100 characters suggests the prompt was fully addressed
if len(response) > 100:
score += 1
# Structure: presence of newlines, bullet points, or numbered lists
if any(marker in response for marker in ["\n", "- ", "1.", "* "]):
score += 1
# Detail: contains specific examples, numbers, or code markers
if any(marker in response for marker in ["```", "def ", "example", "e.g.", "such as"]):
score += 1
# Coherence: response is long enough and does not repeat excessively
words = response.split()
if len(words) > 20:
unique_ratio = len(set(words)) / len(words)
if unique_ratio > 0.4: # At least 40% unique words
score += 1
return max(score, 1) # Minimum score of 1
def analyze_results(results):
"""Score all results and calculate cost-per-quality-point."""
scored_results = []
for result in results:
quality_score = score_response(result)
cost = result["cost"]
# Cost per quality point: lower is better
cost_per_quality = cost / quality_score if quality_score > 0 else cost
scored_results.append({
**result,
"quality_score": quality_score,
"cost_per_quality": cost_per_quality
})
return scored_results
def print_comparison_table(scored_results):
"""Print a formatted comparison table to the terminal."""
print("\n" + "=" * 90)
print("MODEL COST-QUALITY COMPARISON")
print("=" * 90)
print(f"{'Model':<25} {'Category':<18} {'Quality':>7} {'Cost ($)':>10} {'$/Quality':>12} {'Time (s)':>8}")
print("-" * 90)
for r in scored_results:
print(f"{r['model']:<25} {r['category']:<18} {r['quality_score']:>5}/5 {r['cost']:>10.6f} {r['cost_per_quality']:>12.6f} {r['response_time']:>8.2f}")
print("=" * 90)
def print_model_summary(scored_results):
"""Print per-model averages for quick comparison."""
models = set(r["model"] for r in scored_results)
print("\n" + "=" * 70)
print("PER-MODEL SUMMARY (AVERAGES)")
print("=" * 70)
print(f"{'Model':<25} {'Avg Quality':>11} {'Avg Cost':>10} {'Avg $/Qual':>12} {'Avg Time':>8}")
print("-" * 70)
for model in sorted(models):
model_results = [r for r in scored_results if r["model"] == model]
avg_quality = sum(r["quality_score"] for r in model_results) / len(model_results)
avg_cost = sum(r["cost"] for r in model_results) / len(model_results)
avg_cpq = sum(r["cost_per_quality"] for r in model_results) / len(model_results)
avg_time = sum(r["response_time"] for r in model_results) / len(model_results)
print(f"{model:<25} {avg_quality:>9.1f}/5 {avg_cost:>10.6f} {avg_cpq:>12.6f} {avg_time:>7.2f}s")
print("=" * 70)
if __name__ == "__main__":
results = load_results()
scored = analyze_results(results)
print_comparison_table(scored)
print_model_summary(scored)
# Save scored results for the report generator
with open("scored_results.json", "w") as f:
json.dump(scored, f, indent=2)
print(f"\nScored results saved to scored_results.json")What is cost per quality point?
This metric divides the cost of a request by its quality score. A model that costs $0.003 and scores 5/5 has a cost-per-quality of $0.0006. A model that costs $0.001 and scores 2/5 has a cost-per-quality of $0.0005. Lower is better. This is exactly the normalization Gartner recommends in Practice 1 to avoid comparing raw prices without accounting for output quality.
Pro tip: heuristic vs. human scoring
The scoring function uses automated heuristics like length, structure, and detail. In production, you would replace this with human evaluation or an LLM-as-judge approach. The heuristics here are good enough to demonstrate the cost-per-quality framework. Feel free to manually adjust scores in scored_results.json after reviewing the actual responses.
- Run the scoring script.
python score_results.pyYou should see two tables printed to your terminal. The first shows every model-prompt combination with its quality score, cost, cost-per-quality, and response time. The second shows per-model averages so you can quickly compare which model delivers the best value.
Generate the Gartner-Aligned Cost Analysis Report
The benchmark data is valuable on its own, but its real power comes from connecting your findings to Gartner's best practices framework. This final script generates a structured report that maps each insight to a specific Gartner practice. This is the kind of deliverable that demonstrates strategic thinking, not just technical skill.
- Create a new file called cost_report.py.
- Add the following report generation code.
import json
from datetime import datetime
def load_scored_results():
"""Load scored benchmark results."""
with open("scored_results.json", "r") as f:
return json.load(f)
def generate_cost_quality_matrix(scored_results):
"""Build a cost-vs-quality matrix showing each model's position."""
models = set(r["model"] for r in scored_results)
matrix = {}
for model in sorted(models):
model_data = [r for r in scored_results if r["model"] == model]
avg_quality = sum(r["quality_score"] for r in model_data) / len(model_data)
avg_cost = sum(r["cost"] for r in model_data) / len(model_data)
total_cost = sum(r["cost"] for r in model_data)
avg_time = sum(r["response_time"] for r in model_data) / len(model_data)
# Determine the best task category for this model
best_category = max(model_data, key=lambda r: r["quality_score"])
matrix[model] = {
"avg_quality": round(avg_quality, 2),
"avg_cost": round(avg_cost, 6),
"total_cost": round(total_cost, 6),
"avg_response_time": round(avg_time, 2),
"best_task": best_category["category"],
"cost_per_quality": round(avg_cost / avg_quality if avg_quality > 0 else avg_cost, 6)
}
return matrix
def generate_task_recommendations(scored_results):
"""For each task type, recommend the best model."""
categories = set(r["category"] for r in scored_results)
recommendations = {}
for category in sorted(categories):
cat_results = [r for r in scored_results if r["category"] == category]
# Best model = lowest cost-per-quality
best = min(cat_results, key=lambda r: r["cost_per_quality"])
recommendations[category] = {
"recommended_model": best["model"],
"quality_score": best["quality_score"],
"cost": best["cost"],
"cost_per_quality": round(best["cost_per_quality"], 6)
}
return recommendations
def print_report(matrix, recommendations, scored_results):
"""Print the full Gartner-aligned cost analysis report."""
total_cost = sum(r["cost"] for r in scored_results)
total_requests = len(scored_results)
# Estimate enterprise scale: 10,000 requests/day, 22 business days/month
monthly_requests = 10000 * 22
avg_cost_per_request = total_cost / total_requests if total_requests > 0 else 0
projected_monthly = avg_cost_per_request * monthly_requests
projected_annual = projected_monthly * 12
print("\n" + "#" * 80)
print(" GARTNER-ALIGNED AI COST OPTIMIZATION REPORT")
print(f" Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
print("#" * 80)
# Section 1: Cost-Quality Matrix
print("\n" + "=" * 80)
print(" SECTION 1: COST-QUALITY MATRIX")
print(" Gartner Practice 1: Be Objective About Model Trade-Offs")
print("=" * 80)
print(f"\n {'Model':<25} {'Quality':>8} {'Avg Cost':>10} {'$/Quality':>10} {'Best Task':<18}")
print(" " + "-" * 75)
for model, data in matrix.items():
print(f" {model:<25} {data['avg_quality']:>6}/5 ${data['avg_cost']:>9.6f} ${data['cost_per_quality']:>9.6f} {data['best_task']:<18}")
# Section 2: Task Recommendations
print("\n" + "=" * 80)
print(" SECTION 2: MODEL RECOMMENDATIONS BY TASK TYPE")
print(" Gartner Practice 3: Balance Upfront and Operational Costs")
print("=" * 80)
for task, rec in recommendations.items():
print(f"\n Task: {task}")
print(f" Recommended Model: {rec['recommended_model']}")
print(f" Quality: {rec['quality_score']}/5 | Cost: ${rec['cost']:.6f} | $/Quality: ${rec['cost_per_quality']:.6f}")
# Section 3: Caching Savings
print("\n" + "=" * 80)
print(" SECTION 3: CACHING SAVINGS SUMMARY")
print(" Gartner Practice 7: Automate Model Selection, Caching and Routing")
print("=" * 80)
print(f"\n Total benchmark cost (no caching): ${total_cost:.6f}")
print(f" If 50% of requests were cached: ${total_cost * 0.5:.6f} (50% savings)")
print(f" If 80% of requests were cached: ${total_cost * 0.2:.6f} (80% savings)")
print(f" Your cache experiment results from Step 3 show real savings data.")
# Section 4: Enterprise Projections
print("\n" + "=" * 80)
print(" SECTION 4: ENTERPRISE SCALE PROJECTIONS")
print(" Gartner Practice 10: Analyze Visible and Hidden Costs")
print("=" * 80)
print(f"\n Current experiment: {total_requests} requests = ${total_cost:.6f}")
print(f" Average cost per request: ${avg_cost_per_request:.6f}")
print(f" Projected monthly (10K req/day): ${projected_monthly:.2f}")
print(f" Projected annual: ${projected_annual:.2f}")
print(f" With 50% caching: ${projected_annual * 0.5:.2f}/year")
print(f" With model optimization: varies by task routing strategy")
# Section 5: Gartner Practice Mapping
print("\n" + "=" * 80)
print(" SECTION 5: GARTNER PRACTICE MAPPING")
print("=" * 80)
practices = [
("Practice 1", "Model Trade-Offs", "Cost-quality matrix above quantifies trade-offs"),
("Practice 2", "AI Model Sandbox", "LiteLLM gateway acts as your model sandbox"),
("Practice 3", "Balance Costs", "Task-specific model routing reduces operational costs"),
("Practice 5", "Manage SaaS Apps", "Budget guardrails prevent cost overruns"),
("Practice 7", "Caching and Routing", "In-memory caching eliminates redundant inference"),
("Practice 9", "Educate Users", "Prompt optimization reduces token waste"),
("Practice 10", "Analyze Costs", "Cost dashboard tracks visible and hidden costs")
]
for num, name, finding in practices:
print(f"\n {num}: {name}")
print(f" Finding: {finding}")
print("\n" + "#" * 80)
print(" END OF REPORT")
print("#" * 80)
if __name__ == "__main__":
scored = load_scored_results()
matrix = generate_cost_quality_matrix(scored)
recommendations = generate_task_recommendations(scored)
print_report(matrix, recommendations, scored)
# Save report data as JSON for further analysis
report_data = {
"generated_at": datetime.now().isoformat(),
"cost_quality_matrix": matrix,
"task_recommendations": recommendations,
"total_benchmark_cost": sum(r["cost"] for r in scored),
"total_requests": len(scored)
}
with open("cost_report.json", "w") as f:
json.dump(report_data, f, indent=2)
print("\nReport data also saved to cost_report.json")What does this report cover?
The report has five sections. Section 1 shows the cost-quality matrix mapping to Gartner Practice 1. Section 2 recommends the best model for each task type, mapping to Practice 3. Section 3 summarizes caching savings from Practice 7. Section 4 projects your experiment data to enterprise scale for Practice 10. Section 5 ties every finding back to the specific Gartner practice it implements.
- Run the report generator.
python cost_report.pyYour terminal now shows the complete Gartner-aligned cost analysis report. Review the cost-quality matrix to see which model delivers the best value. Check the task recommendations to see if different models win for different prompt types. The enterprise projections show what these costs would look like at scale.
Why does Gartner say inference costs dominate?
Gartner predicts that through 2028, the aggregated costs of model inference will be at least 70% of the total model lifetime costs. Training is a one-time expense. Inference happens every time a user sends a request. Your report demonstrates exactly why model selection, caching, and routing matter so much. Even small per-request savings compound dramatically at enterprise scale.
You now have hard data proving which models deliver the best value for each type of task. Next up, you will build a prompt optimization toolkit that demonstrates how rewriting prompts alone can cut token costs by 30% or more.
Build a Prompt Optimization Toolkit
Your model benchmark proved that choosing the right model matters. But there is another lever that most teams overlook: the prompts themselves. A bloated prompt with unnecessary context burns tokens on every single request, and those costs compound fast at scale. Gartner's Practice 9 specifically recommends educating users on cost-effective GenAI usage through prompt templates and prompt libraries. In this step, you will build a toolkit that quantifies exactly how much money prompt optimization saves.
In This Step, Get Ready To:
- Create wasteful vs. efficient prompt pairs that ask the same question with dramatically different token counts.
- Measure the token economics of each prompt pair through your gateway and calculate savings percentages.
- Generate a reusable prompt template library as a JSON deliverable your team can adopt.
Create Wasteful vs. Efficient Prompt Pairs
Token economics is straightforward: every token you send to an LLM costs money. A prompt stuffed with filler words, redundant instructions, and unnecessary context can easily be 3-5x longer than a lean version that produces the same quality output. The goal here is to create side-by-side comparisons that make this cost difference undeniable.
- Create a new file called prompt_optimizer.py in your project directory.
- Add the following script that defines 3 prompt pairs, each with a wasteful and an optimized version.
import requests
import json
import time
GATEWAY_URL = "http://0.0.0.0:4000/chat/completions"
HEADERS = {
"Content-Type": "application/json",
"Authorization": "Bearer sk-1234"
}
# Define 3 prompt pairs: wasteful vs. optimized
prompt_pairs = [
{
"name": "Capital City Lookup",
"wasteful": (
"I would really appreciate it if you could help me out with something. "
"I am trying to find out what the capital city of France is. I know this "
"might be a simple question, but I want to make absolutely sure I get the "
"right answer. Could you please tell me, in detail, what the capital city "
"of France is and maybe provide some additional context about why it is "
"the capital? Thank you so much for your help with this matter."
),
"optimized": "What is the capital of France? Answer in one sentence."
},
{
"name": "Python List Reversal",
"wasteful": (
"Hello, I am a software developer and I am currently working on a Python "
"project. I need to reverse a list in Python. I have tried a few different "
"approaches but I am not sure which one is the most Pythonic and efficient "
"way to do it. Could you please show me the best way to reverse a list in "
"Python? Please include multiple methods if possible, and explain the pros "
"and cons of each approach. I would also like to understand the time "
"complexity of each method. Thank you for your detailed explanation."
),
"optimized": "Show the most Pythonic way to reverse a list in Python. One method, one example."
},
{
"name": "Email Summary",
"wasteful": (
"I have received a very long and detailed email from my colleague and I "
"need your help to summarize it. The email is about a project update and "
"it contains a lot of information that I need to distill into a brief "
"summary. Please help me understand the key points. I want you to act as "
"a professional executive assistant who is skilled at summarizing complex "
"communications. Here is what the email generally discusses: the Q3 "
"revenue was $4.2M which exceeded the target by 12 percent, the new "
"product launch is delayed to March, and three team members are being "
"relocated to the London office. Please summarize these points concisely."
),
"optimized": (
"Summarize in 3 bullet points: Q3 revenue $4.2M (12% over target), "
"product launch delayed to March, 3 team members relocating to London."
)
}
]
def send_prompt(prompt, model="groq/llama-3.1-8b-instant"):
"""Send a prompt to the gateway and capture cost data."""
payload = {
"model": model,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 200,
# Force fresh responses, no caching
"cache": {"no-cache": True}
}
start = time.time()
resp = requests.post(GATEWAY_URL, headers=HEADERS, json=payload)
elapsed = time.time() - start
data = resp.json()
usage = data.get("usage", {})
cost = float(resp.headers.get("x-litellm-response-cost", 0))
return {
"input_tokens": usage.get("prompt_tokens", 0),
"output_tokens": usage.get("completion_tokens", 0),
"total_tokens": usage.get("total_tokens", 0),
"cost": cost,
"response_time": round(elapsed, 2),
"response": data["choices"][0]["message"]["content"]
}
print("Prompt Optimization Experiment")
print("=" * 60)
results = []
for pair in prompt_pairs:
print(f"\nTesting: {pair['name']}")
print("-" * 40)
# Send wasteful prompt
print(" Sending wasteful prompt...")
wasteful_result = send_prompt(pair["wasteful"])
time.sleep(2) # Respect rate limits
# Send optimized prompt
print(" Sending optimized prompt...")
optimized_result = send_prompt(pair["optimized"])
time.sleep(2)
# Calculate savings
token_savings = wasteful_result["input_tokens"] - optimized_result["input_tokens"]
token_savings_pct = (token_savings / wasteful_result["input_tokens"]) * 100 if wasteful_result["input_tokens"] > 0 else 0
cost_savings = wasteful_result["cost"] - optimized_result["cost"]
pair_result = {
"name": pair["name"],
"wasteful": wasteful_result,
"optimized": optimized_result,
"token_savings": token_savings,
"token_savings_pct": round(token_savings_pct, 1),
"cost_savings": cost_savings
}
results.append(pair_result)
print(f" Wasteful: {wasteful_result['input_tokens']} input tokens, ${wasteful_result['cost']:.6f}")
print(f" Optimized: {optimized_result['input_tokens']} input tokens, ${optimized_result['cost']:.6f}")
print(f" Savings: {token_savings} tokens ({token_savings_pct:.1f}%)")
print("\n" + "=" * 60)
print("EXPERIMENT COMPLETE")
print("=" * 60)What does this script do?
The script defines three prompt pairs where each pair asks the same question in two ways. The "wasteful" version is padded with filler words, unnecessary politeness, and redundant context. The "optimized" version delivers the same intent in a fraction of the tokens. The script sends both versions through your LiteLLM gateway and captures the token counts and costs from the response headers.
Notice the "cache": {"no-cache": True} parameter. This forces fresh LLM calls for every request so your cost comparison is accurate. Without this, cached responses would show $0 cost and skew your results.
- Make sure your LiteLLM proxy is still running with your config.yaml.
- Run the script.
python prompt_optimizer.pyYou should see a comparison table for each prompt pair showing the input token counts, costs, and savings percentage. The optimized prompts should use significantly fewer input tokens across all three pairs.
Getting rate limit errors?
The Groq free tier allows 30 requests per minute and 1,000 requests per day for most models. If you have been running many experiments, you may be approaching the daily limit. Try increasing the time.sleep(2) value to time.sleep(5) or switch to a Gemini model like gemini/gemini-2.5-flash-preview-09-2025. Help me troubleshoot rate limit errors with my LiteLLM gateway.
Measure the Token Economics
You now have raw data showing the token difference between wasteful and optimized prompts. The next step is to aggregate these results into a clear summary that quantifies the total savings. This is the evidence you would present to a team to justify investing in prompt optimization.
- Add the following summary section to the bottom of prompt_optimizer.py, after the experiment loop.
# Aggregate results and print summary report
total_wasteful_tokens = sum(r["wasteful"]["input_tokens"] for r in results)
total_optimized_tokens = sum(r["optimized"]["input_tokens"] for r in results)
total_wasteful_cost = sum(r["wasteful"]["cost"] for r in results)
total_optimized_cost = sum(r["optimized"]["cost"] for r in results)
overall_token_savings = total_wasteful_tokens - total_optimized_tokens
overall_token_savings_pct = (overall_token_savings / total_wasteful_tokens) * 100 if total_wasteful_tokens > 0 else 0
overall_cost_savings = total_wasteful_cost - total_optimized_cost
print("\nTOKEN ECONOMICS SUMMARY")
print("=" * 60)
print(f"{'Metric':<35} {'Wasteful':>10} {'Optimized':>10}")
print("-" * 60)
print(f"{'Total input tokens':<35} {total_wasteful_tokens:>10} {total_optimized_tokens:>10}")
print(f"{'Total cost':<35} ${total_wasteful_cost:>9.6f} ${total_optimized_cost:>9.6f}")
print("-" * 60)
print(f"{'Token savings':<35} {overall_token_savings:>10} ({overall_token_savings_pct:.1f}%)")
print(f"{'Cost savings':<35} ${overall_cost_savings:>9.6f}")
print()
# Enterprise scale projection
# Assume 10,000 prompts/day at an enterprise
daily_prompts = 10000
avg_savings_per_prompt = overall_cost_savings / len(results) if results else 0
daily_savings = avg_savings_per_prompt * daily_prompts
monthly_savings = daily_savings * 30
annual_savings = daily_savings * 365
print("ENTERPRISE SCALE PROJECTION (10,000 prompts/day)")
print("=" * 60)
print(f" Daily savings: ${daily_savings:.2f}")
print(f" Monthly savings: ${monthly_savings:.2f}")
print(f" Annual savings: ${annual_savings:.2f}")
print()
print("Gartner Practice 9: Educate Users on Cost-Effective Use of GenAI")
print(" -> Prompt templates and libraries are a key recommendation.")
print(" -> Your data proves prompt optimization alone can reduce")
print(f" input token costs by {overall_token_savings_pct:.1f}%.")
# Save results for the template library step
with open("prompt_optimization_results.json", "w") as f:
json.dump({
"pairs": [{"name": r["name"], "token_savings_pct": r["token_savings_pct"]} for r in results],
"overall_token_savings_pct": round(overall_token_savings_pct, 1),
"overall_cost_savings": round(overall_cost_savings, 8)
}, f, indent=2)
print("\nResults saved to prompt_optimization_results.json")What does this summary do?
The summary aggregates your three prompt pair experiments into totals. It then projects those savings to enterprise scale by assuming 10,000 prompts per day. Even small per-prompt savings become significant at that volume. The results are also saved to a JSON file so the next substep can reference them when building your template library.
- Run the updated script.
python prompt_optimizer.pyWhat should I see?
You should see the per-pair results followed by a Token Economics Summary table and an Enterprise Scale Projection. The overall token savings percentage should be somewhere in the range of 50-80%, depending on the model's tokenizer. The enterprise projection translates those tiny per-request savings into meaningful annual dollar amounts.
Why does prompt optimization matter so much at scale?
Gartner's research shows that inference costs account for at least 70% of total model lifetime costs. Every token you send is part of that 70%. Prompt optimization is one of the few cost levers that requires zero infrastructure changes. You do not need a different model, a different provider, or a different architecture. You just need better prompts.
Generate a Prompt Template Library
Individual prompt optimization is useful, but the real impact comes from making optimized prompts reusable across a team. A prompt template library gives every team member access to pre-optimized patterns. This is exactly what Gartner means by "prompt libraries" in Practice 9.
- Create a new file called build_template_library.py and add the following script.
import json
# Load experiment results
with open("prompt_optimization_results.json", "r") as f:
experiment_data = json.load(f)
# Build the prompt template library
templates = {
"version": "1.0",
"description": "Optimized prompt templates for cost-effective GenAI usage (Gartner Practice 9)",
"templates": [
{
"name": "Factual Lookup",
"category": "simple_qa",
"pattern": "{question} Answer in one sentence.",
"example": "What is the capital of France? Answer in one sentence.",
"anti_pattern": "I would really appreciate it if you could help me out with something. I am trying to find out {question}. Could you please tell me in detail...",
"expected_token_savings_pct": experiment_data["pairs"][0]["token_savings_pct"],
"recommended_model": "groq/llama-3.1-8b-instant",
"notes": "Use the smallest model for factual lookups. No reasoning needed."
},
{
"name": "Code Example",
"category": "code_generation",
"pattern": "Show the most {language_style} way to {task}. One method, one example.",
"example": "Show the most Pythonic way to reverse a list in Python. One method, one example.",
"anti_pattern": "Hello, I am a software developer working on a {language} project. I need to {task}. Please include multiple methods and explain the pros and cons of each approach...",
"expected_token_savings_pct": experiment_data["pairs"][1]["token_savings_pct"],
"recommended_model": "groq/llama-3.1-8b-instant",
"notes": "Constrain output scope explicitly. Asking for 'multiple methods' inflates both input and output tokens."
},
{
"name": "Structured Summary",
"category": "summarization",
"pattern": "Summarize in {n} bullet points: {key_facts}",
"example": "Summarize in 3 bullet points: Q3 revenue $4.2M (12% over target), product launch delayed to March, 3 team members relocating to London.",
"anti_pattern": "I have received a very long email and I need your help to summarize it. Please act as a professional executive assistant who is skilled at summarizing...",
"expected_token_savings_pct": experiment_data["pairs"][2]["token_savings_pct"],
"recommended_model": "gemini/gemini-2.5-flash-preview-09-2025",
"notes": "Pre-extract key facts before sending to the LLM. Do not ask the model to read and extract from verbose text when you already know the data points."
}
],
"optimization_principles": [
"State the output format explicitly (one sentence, 3 bullets, one example).",
"Remove all politeness filler (please, thank you, I would appreciate).",
"Pre-extract data points instead of asking the model to find them.",
"Constrain scope to prevent the model from generating unnecessary content.",
"Match model size to task complexity. Simple lookups do not need 70B parameter models."
],
"gartner_mapping": {
"practice": "Practice 9: Educate Users on Cost-Effective Use of GenAI",
"implementation": "This template library provides reusable, pre-optimized prompt patterns that reduce input token costs without sacrificing output quality.",
"measured_savings": f"{experiment_data['overall_token_savings_pct']}% average input token reduction"
}
}
# Save the template library
with open("prompt_templates.json", "w") as f:
json.dump(templates, f, indent=2)
print("Prompt Template Library generated: prompt_templates.json")
print(f"Templates: {len(templates['templates'])}")
print(f"Optimization principles: {len(templates['optimization_principles'])}")
print(f"Measured savings: {experiment_data['overall_token_savings_pct']}% input token reduction")
print()
print("Template summary:")
for t in templates["templates"]:
print(f" - {t['name']}: {t['expected_token_savings_pct']}% savings, recommended model: {t['recommended_model']}")What does this script do?
This script reads the experiment results from the previous substep and builds a structured JSON template library. Each template includes the optimized prompt pattern, an example of the anti-pattern to avoid, the measured savings percentage from your experiment, and a recommended model. The library also includes five optimization principles that any team member can follow when writing new prompts.
- Run the template library builder.
python build_template_library.py- Open prompt_templates.json to inspect the generated library.
cat prompt_templates.jsonYou should see a well-structured JSON file with three templates, each containing the optimized pattern, anti-pattern, savings data, and model recommendation. This file is a real deliverable. In an enterprise setting, you would check this into your team's repository and reference it in onboarding documentation.
Pro tip: extend your library
The three templates here are a starting point. In production, you would add templates for every common task your team performs: data extraction, classification, translation, document generation, and more. The key principle is the same for all of them. State the output format explicitly, remove filler, and pre-extract data points before sending to the LLM.
You now have a complete prompt optimization toolkit backed by real data. Your template library quantifies the cost impact of prompt engineering, which is exactly the kind of evidence teams need to adopt better practices. Next up, you will bring all of your findings together into a comprehensive Gartner-aligned cost analysis report.
Secret mission
The 97-Mission Challenge Catalog
You have built a working AI Cost Optimization Control Plane. Now it is time to push it further. Unlock progressively harder challenges by hitting cost optimization thresholds from your main project. Can you achieve 40%+ savings and unlock the Agentic AI Cost Simulator?
Before you go
Clean Up Your Resources
This project runs entirely on free tiers and local tools, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.
Resources you used:
- LiteLLM proxy server (running locally on http://0.0.0.0:4000).
- LiteLLM Python package (litellm[proxy] installed via pip).
- Project files: config.yaml, cost_dashboard.py, cache_experiment.py, model_benchmark.py, prompt_optimizer.py, prompt_templates.json.
- Cost log JSON file (local).
- Groq API key (free tier, no billing attached).
- Google Gemini API key (free tier, no billing attached).
- Environment variables: GROQ_API_KEY and GEMINI_API_KEY.
Keep everything running
No action needed. Choose this if you are still actively building or plan to continue with the secret missions.
- Your LiteLLM proxy will keep running until you stop it or close the terminal.
- Your API keys remain active on both Groq and Gemini free tiers at no cost.
- Your project files and cost logs stay on your local machine.
Pause - I'll come back to this later
Shut down the running proxy to free up your terminal and local memory. Your files and API keys stay intact for when you return.
- Stop the LiteLLM proxy by pressing Ctrl+C in the terminal where it is running.
- Keep your project folder with all Python scripts and config.yaml. You can restart the proxy anytime with litellm --config config.yaml.
- Your Groq and Gemini API keys remain valid. No need to regenerate them.
- Optionally, unset your environment variables by closing the terminal session or running unset GROQ_API_KEY and unset GEMINI_API_KEY.
Delete - I don't want to use this again
Remove all project resources and start fresh. This cleans up everything from your local machine and revokes API access.
- Stop the LiteLLM proxy by pressing Ctrl+C in the terminal where it is running.
- Delete your project folder containing config.yaml, all Python scripts, and log files.
- Uninstall LiteLLM by running pip uninstall litellm.
- Revoke your Groq API key by visiting the Groq console and deleting the key you created.
- Revoke your Gemini API key by visiting Google AI Studio and deleting the key you created.
- Unset your environment variables by running unset GROQ_API_KEY and unset GEMINI_API_KEY.
Mission accomplished
That's a Wrap!
Nice work! You just built an enterprise-grade AI Cost Optimization Control Plane that directly implements 6 of Gartner's 10 Best Practices for Optimizing Generative and Agentic AI Costs.
Gartner warns that at least 50% of GenAI projects will overrun their budgets by 2028. You now have the architectural toolkit to make sure yours doesn't.
Here's what you learned:
- Deployed a LiteLLM AI gateway that routes requests across Groq and Gemini free tiers, with multi-provider model routing, response caching, and budget guardrails all configured through a single config.yaml.
- Ran a structured model cost-quality benchmark across four LLM models, calculated normalized cost-per-quality-point metrics, and generated a Gartner-aligned cost analysis report mapping findings to all 10 best practices.
- Built a prompt optimization toolkit that quantifies token savings from efficient prompting, proving Gartner's Practice 9 recommendation with real cost data.
- Unlocked secret missions including Cache Commander, Budget Enforcer, Routing Strategist, Context Engineer, and the Agentic AI Cost Simulator, plus a full 97-mission challenge catalog with a scoring framework for continued learning.
Ready to quiz yourself?