Python Backend for AI Model Comparison

Build a FastAPI backend that benchmarks Claude, GPT, and Gemini live.

Introduction

30 Second Summary

When companies choose an AI model for their product, they don't just pick the most popular one. They benchmark multiple providers against each other, measuring real speed and cost to find the best fit for their use case.

In this project, you will build a Python backend that sends the same prompt to three major AI providers at once, measures real performance data, and computes which model gives you the best value at your usage scale.

What You'll Build

A FastAPI application deployed on Vercel that orchestrates live API calls to Claude, GPT, and Gemini, returning benchmark data including latency, token usage, actual cost, and cost-crossover analysis.

By the end of this project, you'll have:

  • A Python FastAPI backend deployed on Vercel with three production API endpoints.
  • A multi-provider comparison engine that calls Anthropic, OpenAI, and Google Gemini simultaneously and returns real benchmark data.
  • A cost-crossover analysis endpoint that computes break-even points between models at different usage volumes.
  • Secret Mission: An LLM-as-judge quality scoring endpoint that uses a cheap model to rate the output quality of more expensive models.

Are there any prerequisites?

You'll need basic Python knowledge (variables, functions, loops, dictionaries). You'll also need an existing Vercel account with a deployed frontend from a previous project.

Set Up Your Project

Before you can benchmark AI models, you need a Python backend that can talk to all three providers. FastAPI is the framework that turns your Python code into API endpoints that your frontend can call.

In this step, you will install Python, create your project structure, grab API keys from Anthropic, OpenAI, and Google, and confirm everything works locally. This sets the foundation for every endpoint you build in the next steps.

In this step, get ready to:
  • Install Python 3.12 and create a virtual environment.
  • Obtain API keys for Anthropic, OpenAI, and Google and store them in Vercel.
  • Initialize your project files and verify FastAPI runs locally.
Install Python 3.12 and create your environment

Your backend needs Python 3.12 or higher. macOS ships with an older version, so you will install the correct one via Homebrew.

  • Open your terminal by pressing Cmd+Space to open Spotlight, typing Terminal, and pressing Enter.
  • Check your current Python version by running:
python3 --version

✔️ I see version 3.12 or higher

Your Python is ready. You can use python3 in the commands that follow.

ⓧ I see a version lower than 3.12

macOS often ships with Python 3.9 or 3.10. You need 3.12 for this project's dependencies.

  • Install Python 3.12 via Homebrew by running:
brew install python@3.12

After installation, the binary is available as python3.12. Use python3.12 in all commands that follow instead of python3.

  • Verify the installation by running:
python3.12 --version

You should see Python 3.12.x in the output.

ⓧ Command not found

Python is not installed on your machine yet.

  • Install Python 3.12 via Homebrew by running:
brew install python@3.12

After installation, the binary is available as python3.12. Use python3.12 in all commands that follow.

  • Verify the installation by running:
python3.12 --version

You should see Python 3.12.x in the output.

Brew command not found?

If brew is not recognized, install Homebrew first by visiting brew.sh and following the one-line install command.

help me install Homebrew on my Mac

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir ai-model-comparator
cd ai-model-comparator
  • Create a virtual environment by running:
python3.12 -m venv venv

What is a virtual environment?

A virtual environment is an isolated Python installation inside your project folder. Packages you install here stay separate from your system Python, preventing version conflicts between projects.

  • Activate the virtual environment by running:
source venv/bin/activate

Your terminal prompt now shows (venv) at the beginning, confirming the environment is active.

  • Install the project dependencies by running:
pip install fastapi anthropic openai google-genai

What are these packages?

fastapi is your web framework for building API endpoints. anthropic, openai, and google-genai are the official SDKs for calling Claude, GPT, and Gemini.

You should see a success message listing all installed packages.

Seeing a permission error?

Make sure (venv) appears in your prompt. If it does not, run source venv/bin/activate again before retrying the install.

help me fix a pip permission error when installing packages

  • Open your code editor by pressing Cmd+Space, typing VS Code (or Cursor), and pressing Enter.
  • Click File in the top menu bar.
  • Click Open Folder.
  • Select the ai-model-comparator folder on your Desktop, then click Open.
Get API keys and store them in Vercel

Your backend calls three AI providers. Each requires an API key for authentication.

  • Open console.anthropic.com in your browser.
  • Sign up or sign in to your Anthropic account.
  • Navigate to Settings, then API Keys.
  • Click Create Key, give it a name like model-comparator, and copy the key immediately.
  • Paste the key somewhere safe (a notes app, password manager, etc.) as it will not be shown again.
  • Open platform.openai.com in your browser.
  • Sign up or sign in to your OpenAI account.
  • Navigate to the API Keys page.
  • Click Create new secret key, name it model-comparator, and copy it immediately.
  • Open aistudio.google.com in your browser.
  • Sign in with your Google account.
  • Click Get API key in the left sidebar.
  • Click Create API key and copy the generated key.

Why three separate keys?

Each provider maintains its own authentication system. Your backend will use these keys to prove it has permission to call each provider's API.

Now store all three keys as Vercel environment variables so they are available in both local dev and production.

  • Open vercel.com and sign in to your account.
  • Select your project from the dashboard (or create a new one for this backend).
  • Click Settings in the top navigation.
  • Click Environment Variables in the left sidebar.
  • Add a new variable with the key ANTHROPIC_API_KEY and paste your Anthropic key as the value.
  • Add another variable with the key OPENAI_API_KEY and paste your OpenAI key.
  • Add a third variable with the key GEMINI_API_KEY and paste your Google key.
  • Click Save after adding each variable.

Keep keys out of your code

Storing keys as environment variables means they never appear in your source files. This prevents accidentally committing secrets to version control.

Initialize the project structure and verify locally

Vercel detects your Python app by looking for specific files. You need a pyproject.toml that lists your dependencies and an app.py that defines a top-level app variable.

  • In your editor, create a new file called pyproject.toml in the root of your ai-model-comparator folder.
  • Paste the following configuration:
[project]
name = "ai-model-comparator"
version = "1.0.0"
requires-python = ">=3.12"
dependencies = [
    "fastapi>=0.117.1",
    "anthropic>=0.42.0",
    "openai>=1.82.0",
    "google-genai>=2.3.0",
]

What does pyproject.toml do?

This file tells Vercel which Python version and packages your app needs. When Vercel deploys your project, it reads this file and installs everything listed under dependencies.

  • Save pyproject.toml.
  • Create another new file called app.py in the same folder.
  • Paste the following code:
from fastapi import FastAPI

app = FastAPI()

@app.get("/api")
def root():
    return {"status": "ok", "service": "ai-model-comparator"}

What does this code do?

app = FastAPI() creates your application instance. Vercel looks for this top-level app variable to know where your API lives.

The @app.get("/api") decorator creates a health-check endpoint. When someone visits /api, they get back a JSON response confirming the service is running.

  • Save app.py.
  • Create the providers directory and its init file by running these commands in your terminal:
mkdir providers
touch providers/__init__.py
  • Confirm your project structure by running:
ls

You should see app.py, providers, pyproject.toml, and venv in the output.

Install the Vercel CLI if needed

The next command uses the Vercel CLI. If you do not have it installed, run:

npm i -g vercel

Then run vercel login to authenticate with your Vercel account.

  • Start the local development server by running:
vercel dev

What happens when you run this?

Vercel will ask you to link this directory to a project. Follow the prompts to either create a new Vercel project or link to an existing one. Once linked, Vercel starts a local server that mimics production.

  • Open your browser and navigate to http://localhost:3000/api.

You should see this JSON response:

{"status": "ok", "service": "ai-model-comparator"}

Not seeing the JSON response?

Make sure your app.py file is saved and that the app variable is defined at the top level (not inside a function or if-block).

Check your terminal for error messages. If Vercel says it cannot find a Python framework, ensure pyproject.toml exists in the same folder as app.py.

help me fix vercel dev not detecting my FastAPI app

✔️ Awesome, I've got everything!

Great. Double check you have saved both app.py and pyproject.toml.

ⓧ I'd like to double check the full code

Here is the complete code for each file at this point:

[project]
name = "ai-model-comparator"
version = "1.0.0"
requires-python = ">=3.12"
dependencies = [
    "fastapi>=0.117.1",
    "anthropic>=0.42.0",
    "openai>=1.82.0",
    "google-genai>=2.3.0",
]
from fastapi import FastAPI

app = FastAPI()

@app.get("/api")
def root():
    return {"status": "ok", "service": "ai-model-comparator"}

The providers/__init__.py file should be empty.

Your environment is ready and your backend is responding. Next up, you will build the multi-provider API client that sends prompts to Claude, GPT, and Gemini and captures their responses.

Build the Multi-Provider API Client

Your environment is ready and your API keys are stored. Now the backend needs to actually talk to three different AI providers: Anthropic's Claude, OpenAI's GPT, and Google's Gemini.

Each provider has its own Python SDK with different response formats. In this step, you'll write a standardized wrapper function for each one that returns the same data shape. You'll also add pricing data so the app can compute real costs per request.

In this step, get ready to:
  • Write provider functions for Anthropic, OpenAI, and Gemini that return standardized responses.
  • Add pricing data and a cost computation function to the main app.
  • Verify all three providers respond correctly.
Write the Anthropic provider function

Each provider client lives in its own file inside the providers/ directory. Every function accepts a prompt string and returns a dictionary with exactly three keys: text, input_tokens, and output_tokens. This standardized shape means the rest of the app never needs to know which SDK it's working with.

  • Create a new file called anthropic_client.py inside your providers/ directory and add this code:
import os
import anthropic


async def call_anthropic(prompt: str, model: str = "claude-haiku-4-5") -> dict:
    # Create a client using the API key from environment variables
    client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
    # Send the prompt to Claude and get a response
    message = client.messages.create(
        model=model,
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
    # Return standardized response with text and token counts
    return {
        "text": message.content[0].text,
        "input_tokens": message.usage.input_tokens,
        "output_tokens": message.usage.output_tokens,
    }

What does this code do?

This function wraps the Anthropic SDK's messages.create() method. It reads ANTHROPIC_API_KEY from the environment (which Vercel injects at runtime), sends the prompt to Claude, and extracts the response text plus token usage from the returned message object.

The default model is claude-haiku-4-5 because it's the cheapest option for benchmarking. The function is marked async so it can be gathered concurrently with other providers in the next step.

  • Save providers/anthropic_client.py.

Seeing a ModuleNotFoundError for anthropic?

Make sure your virtual environment is activated. Run pip install anthropic again if the module isn't found. Help me fix my anthropic import error

Write the OpenAI provider function
  • Create a new file called openai_client.py inside your providers/ directory and add this code:
import os
from openai import OpenAI


async def call_openai(prompt: str, model: str = "gpt-4o-mini") -> dict:
    # Create a client using the API key from environment variables
    client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
    # Send the prompt to GPT and get a response
    response = client.chat.completions.create(
        model=model,
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
    choice = response.choices[0]
    # Return standardized response with text and token counts
    return {
        "text": choice.message.content,
        "input_tokens": response.usage.prompt_tokens,
        "output_tokens": response.usage.completion_tokens,
    }

What does this code do?

The OpenAI SDK uses chat.completions.create() instead of messages.create(). The response structure is also different. OpenAI calls input tokens prompt_tokens and output tokens completion_tokens.

Your wrapper normalizes these into the same input_tokens and output_tokens keys so the rest of the app can treat all providers identically.

  • Save providers/openai_client.py.
Write the Gemini provider function
  • Create a new file called gemini_client.py inside your providers/ directory and add this code:
import os
from google import genai


async def call_gemini(prompt: str, model: str = "gemini-2.5-flash") -> dict:
    # Create a client using the API key from environment variables
    client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
    # Send the prompt to Gemini and get a response
    response = client.models.generate_content(
        model=model,
        contents=prompt,
    )
    usage = response.usage_metadata
    # Return standardized response with text and token counts
    return {
        "text": response.text,
        "input_tokens": usage.prompt_token_count,
        "output_tokens": usage.candidates_token_count,
    }

What does this code do?

Google's Gemini SDK is the most different of the three. It uses generate_content() instead of a chat interface. Token counts live inside response.usage_metadata with Google-specific names: prompt_token_count for input and candidates_token_count for output.

Your wrapper maps these to the same standardized keys. Downstream code never needs to know which naming convention each provider uses.

  • Save providers/gemini_client.py.

Import error for google.genai?

The package name is google-genai (not google-generativeai). Run pip install google-genai to install the correct package. Help me fix my Gemini import error

✔️ Awesome, I've got everything!

Great. Make sure all three files are saved before moving on.

ⓧ I'd like to double check the full code

Here are all three provider files for reference:

import os
import anthropic


async def call_anthropic(prompt: str, model: str = "claude-haiku-4-5") -> dict:
    client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
    message = client.messages.create(
        model=model,
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
    return {
        "text": message.content[0].text,
        "input_tokens": message.usage.input_tokens,
        "output_tokens": message.usage.output_tokens,
    }
import os
from openai import OpenAI


async def call_openai(prompt: str, model: str = "gpt-4o-mini") -> dict:
    client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
    response = client.chat.completions.create(
        model=model,
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
    choice = response.choices[0]
    return {
        "text": choice.message.content,
        "input_tokens": response.usage.prompt_tokens,
        "output_tokens": response.usage.completion_tokens,
    }
import os
from google import genai


async def call_gemini(prompt: str, model: str = "gemini-2.5-flash") -> dict:
    client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
    response = client.models.generate_content(
        model=model,
        contents=prompt,
    )
    usage = response.usage_metadata
    return {
        "text": response.text,
        "input_tokens": usage.prompt_token_count,
        "output_tokens": usage.candidates_token_count,
    }
Add pricing and cost infrastructure to app.py

The provider functions return raw token counts. To compute real costs, app.py needs a pricing table mapping each model to its per-token rate, plus a function that multiplies tokens by price.

  • Open app.py and replace its entire contents with the following imports, CORS setup, and app initialization:
import asyncio
import time
from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from providers.anthropic_client import call_anthropic
from providers.openai_client import call_openai
from providers.gemini_client import call_gemini

app = FastAPI()

# Allow cross-origin requests from your frontend
app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

What does this code do?

The imports bring in everything the app needs: asyncio for running providers concurrently (next step), time for latency measurement, and the three provider functions you just wrote.

CORS middleware allows your frontend (on a different domain) to call this API. The allow_origins=["*"] setting permits requests from any origin during development.

  • Below the CORS middleware block, add the pricing table, request model, and cost computation function:
# Per-model pricing in USD per million tokens
PRICING = {
    "claude-haiku-4-5": {"input": 1.00, "output": 5.00},
    "claude-sonnet-5": {"input": 2.00, "output": 10.00},
    "gpt-4o-mini": {"input": 0.15, "output": 0.60},
    "gpt-4o": {"input": 2.50, "output": 10.00},
    "gemini-2.5-flash": {"input": 0.15, "output": 3.50},
    "gemini-3.5-flash": {"input": 1.50, "output": 9.00},
}


class CompareRequest(BaseModel):
    prompt: str
    models: list[str] | None = None


def compute_cost(model: str, input_tokens: int, output_tokens: int) -> float:
    # Look up the model's pricing and calculate total cost
    if model not in PRICING:
        return 0.0
    prices = PRICING[model]
    input_cost = (input_tokens / 1_000_000) * prices["input"]
    output_cost = (output_tokens / 1_000_000) * prices["output"]
    return round(input_cost + output_cost, 6)

What does this code do?

The PRICING dictionary stores the cost per million tokens for each model's input and output. These are the official rates from each provider's pricing page.

The CompareRequest class uses Pydantic to validate incoming request bodies. The compute_cost() function takes a model name and token counts, then returns the actual USD cost of that request.

  • Finally, keep the existing root endpoint at the bottom of the file:
@app.get("/api")
def root():
    return {"status": "ok", "service": "ai-model-comparator"}
  • Save app.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved app.py before continuing.

ⓧ I'd like to double check the full code

Here is the complete app.py at this point:

import asyncio
import time
from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from providers.anthropic_client import call_anthropic
from providers.openai_client import call_openai
from providers.gemini_client import call_gemini

app = FastAPI()

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

PRICING = {
    "claude-haiku-4-5": {"input": 1.00, "output": 5.00},
    "claude-sonnet-5": {"input": 2.00, "output": 10.00},
    "gpt-4o-mini": {"input": 0.15, "output": 0.60},
    "gpt-4o": {"input": 2.50, "output": 10.00},
    "gemini-2.5-flash": {"input": 0.15, "output": 3.50},
    "gemini-3.5-flash": {"input": 1.50, "output": 9.00},
}


class CompareRequest(BaseModel):
    prompt: str
    models: list[str] | None = None


def compute_cost(model: str, input_tokens: int, output_tokens: int) -> float:
    if model not in PRICING:
        return 0.0
    prices = PRICING[model]
    input_cost = (input_tokens / 1_000_000) * prices["input"]
    output_cost = (output_tokens / 1_000_000) * prices["output"]
    return round(input_cost + output_cost, 6)


@app.get("/api")
def root():
    return {"status": "ok", "service": "ai-model-comparator"}
Verify the app loads correctly

With all four files saved, the app should start without import errors. This confirms the provider modules are wired up correctly.

  • Switch back to your terminal. If vercel dev is still running, stop it with Ctrl+C.
  • Restart the local dev server by running:
vercel dev
  • Watch the terminal output. The server should start without any ImportError or ModuleNotFoundError messages.
  • Confirm the app is serving by opening http://localhost:3000/api in your browser. You should see the JSON response:
{"status": "ok", "service": "ai-model-comparator"}

Seeing an ImportError on startup?

Check that your file names match exactly: providers/anthropic_client.py, providers/openai_client.py, and providers/gemini_client.py. The import statements in app.py must match these filenames exactly.

Also confirm providers/__init__.py exists (even if it's empty). Python needs this file to treat providers/ as a package. Help me debug my import error

Your multi-provider client is built. Next up, you'll wire these functions into a benchmarking endpoint that calls all three providers simultaneously and ranks them by cost and speed.

Build the Live Benchmarking Engine

You now have three provider functions that can call Claude, GPT, and Gemini individually. The next challenge is calling all three simultaneously and comparing their performance side by side.

In this step, you'll build the benchmarking engine. This is the core of your backend: it fires off concurrent API calls, measures real latency in milliseconds, computes actual cost per request, and returns structured comparison data sorted by price.

In this step, get ready to:
  • Create the /api/benchmark endpoint with concurrent API calls.
  • Write a provider caller that measures timing and computes cost per request.
  • Verify error handling so one failing provider doesn't crash the whole comparison.
Create the /api/benchmark endpoint

The benchmark endpoint needs to accept a user prompt, send it to all three providers at the same time, and return the results ranked by cost. Calling them concurrently instead of one-by-one means total response time equals the slowest provider, not the sum of all three.

First, you'll add two new Pydantic models that define the shape of incoming requests and outgoing results.

  • In app.py, find the existing CompareRequest class.
  • Add the following two new models directly below it (before the compute_cost function):
class CrossoverRequest(BaseModel):
    queries_per_day: int
    results: list[dict]


class BenchmarkResult(BaseModel):
    model: str
    provider: str
    response_text: str
    input_tokens: int
    output_tokens: int
    latency_ms: float
    cost_usd: float
    status: str

What do these models do?

CrossoverRequest defines the input shape for the cost-crossover endpoint you'll build in the next step. You're adding it now so you don't have to revisit the models section later.

BenchmarkResult defines the exact fields each provider result will return: model name, provider, response text, token counts, latency in milliseconds, cost in USD, and a status string indicating success or failure.

Now you'll write the endpoint itself. This function uses asyncio.gather() to fire off all three provider calls at once and waits for all of them to finish.

  • In app.py, scroll down to the existing root() endpoint.
  • Add the following endpoint below it:
@app.post("/api/benchmark")
async def benchmark(request: CompareRequest):
    prompt = request.prompt
    if not prompt:
        raise HTTPException(status_code=400, detail="Prompt is required")

    # Call all three providers concurrently
    tasks = [
        call_provider_safe("anthropic", "claude-haiku-4-5", prompt),
        call_provider_safe("openai", "gpt-4o-mini", prompt),
        call_provider_safe("gemini", "gemini-2.5-flash", prompt),
    ]
    results = await asyncio.gather(*tasks)
    # Separate successes from failures and sort by cost
    results_sorted = sorted(
        [r for r in results if r["status"] == "success"],
        key=lambda r: r["cost_usd"],
    )
    failed = [r for r in results if r["status"] != "success"]
    return {"results": results_sorted, "failed": failed}

Why asyncio.gather()?

Without asyncio.gather(), calling each provider sequentially would mean waiting for Claude to respond, then GPT, then Gemini. If each takes 2 seconds, that's 6 seconds total.

With asyncio.gather(), all three calls start at the same time. Total wait time equals the slowest provider (about 2 seconds). This is the standard pattern for fan-out requests in production backends.

Write the provider caller with timing and cost computation

Each provider call needs to be wrapped in a function that records timing, computes cost from token counts, and catches errors gracefully. The call_provider_safe function handles all three concerns in one place.

  • In app.py, add the following function below the benchmark endpoint you just wrote:
async def call_provider_safe(provider: str, model: str, prompt: str) -> dict:
    try:
        # Start the high-resolution timer
        start = time.perf_counter()
        if provider == "anthropic":
            response = await call_anthropic(prompt, model)
        elif provider == "openai":
            response = await call_openai(prompt, model)
        elif provider == "gemini":
            response = await call_gemini(prompt, model)
        else:
            raise ValueError(f"Unknown provider: {provider}")
        # Convert elapsed seconds to milliseconds
        elapsed_ms = (time.perf_counter() - start) * 1000

        # Compute real cost from token counts and pricing table
        cost = compute_cost(model, response["input_tokens"], response["output_tokens"])

        return {
            "model": model,
            "provider": provider,
            "response_text": response["text"],
            "input_tokens": response["input_tokens"],
            "output_tokens": response["output_tokens"],
            "latency_ms": round(elapsed_ms, 1),
            "cost_usd": cost,
            "status": "success",
        }
    except Exception as e:
        # If a provider fails, return a result with error status
        return {
            "model": model,
            "provider": provider,
            "response_text": "",
            "input_tokens": 0,
            "output_tokens": 0,
            "latency_ms": 0,
            "cost_usd": 0,
            "status": f"error: {str(e)}",
        }

How does the timing and cost work?

time.perf_counter() is Python's highest-resolution timer for measuring short durations. It returns seconds as a float, so multiplying by 1000 gives milliseconds. Recording it before and after the API call captures the real round-trip latency.

The compute_cost function you wrote in the previous step multiplies input tokens by the model's per-million input price and output tokens by the per-million output price. This gives you real cost-per-request in USD.

The try/except block ensures that if one provider times out or returns an error, the function returns a result dict with status: "error: ..." instead of crashing the entire gather. The benchmark endpoint then separates successes from failures and returns both.

  • Save app.py.

Seeing a NameError for call_provider_safe?

Make sure call_provider_safe is defined in app.py after the benchmark endpoint. Python resolves function names at call time, not definition time, so this ordering works fine.

If you placed it in a separate file, the import won't match. Keep it in app.py alongside the endpoint.

Help me debug my call_provider_safe function.

Test the benchmark endpoint end-to-end

Time to verify that all three providers respond, timing is captured, and cost is computed correctly.

  • Start the local dev server by running:
vercel dev
  • Open a new terminal tab and send a test request by running:
curl -X POST http://localhost:3000/api/benchmark \
  -H "Content-Type: application/json" \
  -d '{"prompt": "What is 2+2?"}'
  • You should see a JSON response with a results array containing three objects (one per provider), each with populated latency_ms, cost_usd, input_tokens, and output_tokens fields.
  • The results should be sorted by cost_usd from cheapest to most expensive.
  • The failed array should be empty if all three providers responded successfully.

What should the response look like?

You'll see something like this (values will differ based on real API responses):

{
  "results": [
    {"model": "gpt-4o-mini", "latency_ms": 892.3, "cost_usd": 0.000012, "status": "success", ...},
    {"model": "gemini-2.5-flash", "latency_ms": 1043.7, "cost_usd": 0.000045, "status": "success", ...},
    {"model": "claude-haiku-4-5", "latency_ms": 1521.2, "cost_usd": 0.000089, "status": "success", ...}
  ],
  "failed": []
}

The exact latency and cost depend on network conditions and response length. The key thing to verify is that all three fields are populated with non-zero values.

Seeing a provider in the failed array?

Check that your API keys are set correctly in Vercel environment variables. Run vercel env pull to sync them to your local .env.local file.

If only one provider fails, the other two should still appear in the results array. This is the error handling working as designed.

My benchmark endpoint is returning errors for one or more providers.

✔️ Awesome, I've got everything!

Great. Double check you've saved app.py and that all three providers are returning results.

ⓧ I'd like to double check the full code

Here's the complete app.py as it should look at the end of this step:

import asyncio
import time
from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from providers.anthropic_client import call_anthropic
from providers.openai_client import call_openai
from providers.gemini_client import call_gemini

app = FastAPI()

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

PRICING = {
    "claude-haiku-4-5": {"input": 1.00, "output": 5.00},
    "claude-sonnet-5": {"input": 2.00, "output": 10.00},
    "gpt-4o-mini": {"input": 0.15, "output": 0.60},
    "gpt-4o": {"input": 2.50, "output": 10.00},
    "gemini-2.5-flash": {"input": 0.15, "output": 3.50},
    "gemini-3.5-flash": {"input": 1.50, "output": 9.00},
}


class CompareRequest(BaseModel):
    prompt: str
    models: list[str] | None = None


class CrossoverRequest(BaseModel):
    queries_per_day: int
    results: list[dict]


class BenchmarkResult(BaseModel):
    model: str
    provider: str
    response_text: str
    input_tokens: int
    output_tokens: int
    latency_ms: float
    cost_usd: float
    status: str


def compute_cost(model: str, input_tokens: int, output_tokens: int) -> float:
    if model not in PRICING:
        return 0.0
    prices = PRICING[model]
    input_cost = (input_tokens / 1_000_000) * prices["input"]
    output_cost = (output_tokens / 1_000_000) * prices["output"]
    return round(input_cost + output_cost, 6)


@app.get("/api")
def root():
    return {"status": "ok", "service": "ai-model-comparator"}


@app.post("/api/benchmark")
async def benchmark(request: CompareRequest):
    prompt = request.prompt
    if not prompt:
        raise HTTPException(status_code=400, detail="Prompt is required")

    tasks = [
        call_provider_safe("anthropic", "claude-haiku-4-5", prompt),
        call_provider_safe("openai", "gpt-4o-mini", prompt),
        call_provider_safe("gemini", "gemini-2.5-flash", prompt),
    ]
    results = await asyncio.gather(*tasks)
    results_sorted = sorted(
        [r for r in results if r["status"] == "success"],
        key=lambda r: r["cost_usd"],
    )
    failed = [r for r in results if r["status"] != "success"]
    return {"results": results_sorted, "failed": failed}


async def call_provider_safe(provider: str, model: str, prompt: str) -> dict:
    try:
        start = time.perf_counter()
        if provider == "anthropic":
            response = await call_anthropic(prompt, model)
        elif provider == "openai":
            response = await call_openai(prompt, model)
        elif provider == "gemini":
            response = await call_gemini(prompt, model)
        else:
            raise ValueError(f"Unknown provider: {provider}")
        elapsed_ms = (time.perf_counter() - start) * 1000

        cost = compute_cost(model, response["input_tokens"], response["output_tokens"])

        return {
            "model": model,
            "provider": provider,
            "response_text": response["text"],
            "input_tokens": response["input_tokens"],
            "output_tokens": response["output_tokens"],
            "latency_ms": round(elapsed_ms, 1),
            "cost_usd": cost,
            "status": "success",
        }
    except Exception as e:
        return {
            "model": model,
            "provider": provider,
            "response_text": "",
            "input_tokens": 0,
            "output_tokens": 0,
            "latency_ms": 0,
            "cost_usd": 0,
            "status": f"error: {str(e)}",
        }

Your benchmarking engine is live. It calls three AI providers concurrently, measures real latency, and computes actual cost per request. Next up, you'll build the cost-crossover analysis that uses these results to answer the question: which model should I use at my scale?

Build the Cost-Crossover Analysis

Your benchmark engine is live and returning real cost data for every API call. But raw per-request costs don't answer the question that matters most when choosing a model for production: which one is cheapest at my scale?

In this step, you will build a cost-crossover endpoint that takes benchmark results and a daily query volume, then computes monthly costs, savings, and a recommendation string that accounts for latency trade-offs.

In this step, get ready to:
  • Create the /api/crossover endpoint that accepts query volume and benchmark results.
  • Compute monthly costs, savings, and a break-even comparison between models.
  • Return a recommendation string that factors in latency trade-offs.
Add the crossover endpoint to your app

The crossover analysis answers a practical question: if you run N queries per day, which model saves you the most money over a month?

  • In app.py, scroll to the bottom of the file (below the call_provider_safe function).
  • Add the /api/crossover endpoint by pasting this code at the end of the file:
@app.post("/api/crossover")
async def crossover(request: CrossoverRequest):
    # Require at least 2 models to compare
    if not request.results or len(request.results) < 2:
        raise HTTPException(
            status_code=400, detail="Need at least 2 model results to compare"
        )

    # Compute monthly cost for each model at the given query volume
    monthly_costs = []
    for result in request.results:
        monthly = result["cost_usd"] * request.queries_per_day * 30
        monthly_costs.append({
            "model": result["model"],
            "cost_per_request": result["cost_usd"],
            "monthly_cost": round(monthly, 2),
            "latency_ms": result["latency_ms"],
        })

    # Sort so the cheapest model is first
    monthly_costs.sort(key=lambda x: x["monthly_cost"])

    # Calculate how much the cheapest saves over the most expensive
    cheapest = monthly_costs[0]
    most_expensive = monthly_costs[-1]
    savings_monthly = round(most_expensive["monthly_cost"] - cheapest["monthly_cost"], 2)

    # Build a recommendation that includes latency trade-off
    recommendation = (
        f"At {request.queries_per_day} queries/day, {cheapest['model']} "
        f"saves ${savings_monthly}/month over {most_expensive['model']}. "
    )
    if cheapest["latency_ms"] < most_expensive["latency_ms"]:
        recommendation += "It is also faster."
    else:
        latency_diff = round(cheapest["latency_ms"] - most_expensive["latency_ms"], 0)
        recommendation += (
            f"Trade-off: {int(latency_diff)}ms slower per request."
        )

    return {
        "queries_per_day": request.queries_per_day,
        "monthly_costs": monthly_costs,
        "savings_monthly": savings_monthly,
        "recommendation": recommendation,
    }

What does this code do?

This endpoint takes two inputs: a queries_per_day number and an array of results from a previous benchmark. It uses the CrossoverRequest model you already defined in an earlier step.

  • The monthly cost formula is straightforward: cost_per_request * queries_per_day * 30. This projects a single benchmark data point into a real monthly bill.
  • After sorting by monthly cost, the endpoint identifies the cheapest and most expensive models and computes the dollar difference.
  • The recommendation string adds a latency check. If the cheapest model is also faster, it is a clear winner. If not, the string quantifies the latency penalty so the user can decide whether the savings justify the speed trade-off.
Test the crossover endpoint locally

The crossover endpoint does not call any external APIs. You can test it with mock data to verify the math and recommendation logic without spending API credits.

  • Make sure vercel dev is still running in your terminal from the previous step.
  • Open a new terminal tab and test the endpoint by running this command:
curl -X POST http://localhost:3000/api/crossover \
  -H "Content-Type: application/json" \
  -d '{"queries_per_day": 100, "results": [{"model": "gpt-4o-mini", "cost_usd": 0.000045, "latency_ms": 800}, {"model": "claude-haiku-4-5", "cost_usd": 0.000120, "latency_ms": 650}]}'

What does this command do?

This sends a POST request with a JSON body to your local crossover endpoint. The mock data simulates two models with different costs and latencies at 100 queries per day.

You do not need real benchmark results here. The endpoint only performs arithmetic on whatever numbers you send, so mock data works perfectly for validation.

  • Save app.py before running the curl command.
  • You should see a JSON response containing monthly_costs, savings_monthly, and a recommendation string that mentions the cheaper model and the dollar savings.

Getting a 404 or connection error?

  • Confirm vercel dev is running and serving at http://localhost:3000. If it stopped, restart it from your project directory.
  • Check that you saved app.py after adding the new endpoint. The dev server picks up file changes automatically, but unsaved edits are invisible.
  • If you see a 422 validation error, double-check that your JSON body includes both queries_per_day (an integer) and results (an array with at least 2 items).

Help me debug my crossover endpoint

✔️ Awesome, I've got everything!

Great. Double check you have saved app.py with the new crossover endpoint.

ⓧ I'd like to double check the full code

Here is the complete app.py file as it should look at the end of this step:

import asyncio
import time
from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from providers.anthropic_client import call_anthropic
from providers.openai_client import call_openai
from providers.gemini_client import call_gemini

app = FastAPI()

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

PRICING = {
    "claude-haiku-4-5": {"input": 1.00, "output": 5.00},
    "claude-sonnet-5": {"input": 2.00, "output": 10.00},
    "gpt-4o-mini": {"input": 0.15, "output": 0.60},
    "gpt-4o": {"input": 2.50, "output": 10.00},
    "gemini-2.5-flash": {"input": 0.15, "output": 3.50},
    "gemini-3.5-flash": {"input": 1.50, "output": 9.00},
}


class CompareRequest(BaseModel):
    prompt: str
    models: list[str] | None = None


class CrossoverRequest(BaseModel):
    queries_per_day: int
    results: list[dict]


class BenchmarkResult(BaseModel):
    model: str
    provider: str
    response_text: str
    input_tokens: int
    output_tokens: int
    latency_ms: float
    cost_usd: float
    status: str


def compute_cost(model: str, input_tokens: int, output_tokens: int) -> float:
    if model not in PRICING:
        return 0.0
    prices = PRICING[model]
    input_cost = (input_tokens / 1_000_000) * prices["input"]
    output_cost = (output_tokens / 1_000_000) * prices["output"]
    return round(input_cost + output_cost, 6)


@app.get("/api")
def root():
    return {"status": "ok", "service": "ai-model-comparator"}


@app.post("/api/benchmark")
async def benchmark(request: CompareRequest):
    prompt = request.prompt
    if not prompt:
        raise HTTPException(status_code=400, detail="Prompt is required")

    tasks = [
        call_provider_safe("anthropic", "claude-haiku-4-5", prompt),
        call_provider_safe("openai", "gpt-4o-mini", prompt),
        call_provider_safe("gemini", "gemini-2.5-flash", prompt),
    ]
    results = await asyncio.gather(*tasks)
    results_sorted = sorted(
        [r for r in results if r["status"] == "success"],
        key=lambda r: r["cost_usd"],
    )
    failed = [r for r in results if r["status"] != "success"]
    return {"results": results_sorted, "failed": failed}


async def call_provider_safe(provider: str, model: str, prompt: str) -> dict:
    try:
        start = time.perf_counter()
        if provider == "anthropic":
            response = await call_anthropic(prompt, model)
        elif provider == "openai":
            response = await call_openai(prompt, model)
        elif provider == "gemini":
            response = await call_gemini(prompt, model)
        else:
            raise ValueError(f"Unknown provider: {provider}")
        elapsed_ms = (time.perf_counter() - start) * 1000

        cost = compute_cost(model, response["input_tokens"], response["output_tokens"])

        return {
            "model": model,
            "provider": provider,
            "response_text": response["text"],
            "input_tokens": response["input_tokens"],
            "output_tokens": response["output_tokens"],
            "latency_ms": round(elapsed_ms, 1),
            "cost_usd": cost,
            "status": "success",
        }
    except Exception as e:
        return {
            "model": model,
            "provider": provider,
            "response_text": "",
            "input_tokens": 0,
            "output_tokens": 0,
            "latency_ms": 0,
            "cost_usd": 0,
            "status": f"error: {str(e)}",
        }


@app.post("/api/crossover")
async def crossover(request: CrossoverRequest):
    if not request.results or len(request.results) < 2:
        raise HTTPException(
            status_code=400, detail="Need at least 2 model results to compare"
        )

    monthly_costs = []
    for result in request.results:
        monthly = result["cost_usd"] * request.queries_per_day * 30
        monthly_costs.append({
            "model": result["model"],
            "cost_per_request": result["cost_usd"],
            "monthly_cost": round(monthly, 2),
            "latency_ms": result["latency_ms"],
        })

    monthly_costs.sort(key=lambda x: x["monthly_cost"])

    cheapest = monthly_costs[0]
    most_expensive = monthly_costs[-1]
    savings_monthly = round(most_expensive["monthly_cost"] - cheapest["monthly_cost"], 2)

    recommendation = (
        f"At {request.queries_per_day} queries/day, {cheapest['model']} "
        f"saves ${savings_monthly}/month over {most_expensive['model']}. "
    )
    if cheapest["latency_ms"] < most_expensive["latency_ms"]:
        recommendation += "It is also faster."
    else:
        latency_diff = round(cheapest["latency_ms"] - most_expensive["latency_ms"], 0)
        recommendation += (
            f"Trade-off: {int(latency_diff)}ms slower per request."
        )

    return {
        "queries_per_day": request.queries_per_day,
        "monthly_costs": monthly_costs,
        "savings_monthly": savings_monthly,
        "recommendation": recommendation,
    }

Your backend now answers the question every engineering team asks before choosing a model: which one fits our budget at scale? Next up, you will deploy this to Vercel and connect it to your frontend so users can see these comparisons live.

Deploy and Connect to Frontend

Your backend is feature-complete. You have a benchmarking engine, cost computation, and crossover analysis all wired up and tested locally.

Now it is time to ship it. In this step, you will deploy your Python backend to Vercel production, verify every endpoint responds at a live URL, and connect your existing frontend so users can trigger real comparisons from the browser.

In this step, get ready to:
  • Deploy your Python backend to Vercel production.
  • Verify all three endpoints respond correctly at the live URL.
  • Connect your existing frontend to the new Python endpoints and test end-to-end.
Confirm your configuration and deploy to production

Vercel's Python runtime automatically detects your FastAPI app when it finds a pyproject.toml declaring dependencies and an app.py file exporting a top-level app variable. Your project already matches this structure.

  • Open pyproject.toml in your editor and confirm it matches this:
[project]
name = "ai-model-comparator"
version = "1.0.0"
requires-python = ">=3.12"
dependencies = [
    "fastapi>=0.117.1",
    "anthropic>=0.42.0",
    "openai>=1.82.0",
    "google-genai>=2.3.0",
]

Why does this file matter for deployment?

Vercel reads pyproject.toml to install your Python dependencies on the server. The requires-python = ">=3.12" line tells Vercel which Python version to use. Without this file, Vercel cannot resolve your imports and the deployment fails.

  • Deploy your backend to Vercel production by running this command in your project directory:
vercel deploy --prod

What does this command do?

The --prod flag tells Vercel to deploy to your production domain instead of creating a preview URL. Vercel detects the FastAPI framework from your dependencies, installs them, and packages app.py as a serverless function.

Once the deploy completes, Vercel prints your production URL in the terminal. Copy it for the next step.

  • Verify the health check endpoint responds by running:
curl [[VERCEL_URL="your-vercel-production-url"]]/api
  • You should see a JSON response containing {"status": "ok", "service": "ai-model-comparator"}.

What about cold starts?

The first request after a period of inactivity may take 1-2 seconds longer than usual. This is called a cold start. Vercel needs to spin up the Python runtime before it can handle the request. Subsequent requests will be much faster.

  • Test the benchmark endpoint by running:
curl -X POST [[VERCEL_URL="your-vercel-production-url"]]/api/benchmark \
  -H "Content-Type: application/json" \
  -d '{"prompt": "What is Python?"}'
  • You should see a JSON response with a results array containing benchmark data from all three providers, including model, latency_ms, cost_usd, and response_text fields.

Seeing an error response?

  • If you get a 500 error, check that your API keys are set in your Vercel project's environment variables. Open the Vercel dashboard, navigate to your project's **Settings**, then **Environment Variables**, and confirm ANTHROPIC_API_KEY, OPENAI_API_KEY, and GEMINI_API_KEY are present for the Production environment.
  • If only one or two providers fail (the failed array has entries), check that specific provider's API key is correct and has available credits.
  • If you see a 404, make sure your app.py is in the project root directory and defines a top-level app variable. Redeploy after fixing.

Why isn't my Vercel deployment returning data from all three providers?

Update your frontend to call the new Python endpoints

Your existing frontend currently calls old JavaScript serverless functions. You need to update its fetch calls to point at your new Python backend's /api/benchmark and /api/crossover endpoints.

  • Open your frontend's JavaScript file where fetch calls are made.
  • Find the existing fetch call that triggers model comparison.
  • Update the URL to point to your Vercel production URL's /api/benchmark endpoint.
  • Make sure the request body sends a JSON object with a prompt field. Here is an example of what the fetch call should look like:
// Send the user's prompt to the Python benchmark endpoint
const response = await fetch('[[VERCEL_URL="your-vercel-production-url"]]/api/benchmark', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ prompt: userInput })
});
const data = await response.json();
// data.results contains the sorted benchmark results
// data.failed contains any providers that errored

What does the response JSON look like?

The /api/benchmark endpoint returns an object with two arrays:

  • results contains objects sorted by cost, each with model, provider, response_text, input_tokens, output_tokens, latency_ms, cost_usd, and status.
  • failed contains any provider results that returned an error instead of a successful response.
  • If your frontend also displays cost-crossover analysis, update that fetch call to point to /api/crossover with a body containing queries_per_day (integer) and results (the array from the benchmark response).
  • Save your frontend files.

Same-origin deployments skip CORS

If your frontend and backend are deployed under the same Vercel project, they share a domain. Requests between them are same-origin, so you do not need to worry about CORS headers. The allow_origins=["*"] in your app.py handles cross-origin cases when the frontend is on a different domain.

Test the full end-to-end flow

The final check confirms that a real user flow works from browser to backend and back.

  • Open your Vercel production URL in a browser.
  • Enter a test prompt in the input field (for example, "Explain recursion in one sentence").
  • Submit the comparison.
  • Verify the frontend displays results from all three providers with latency, token count, and cost data.

What should I see?

After submitting a prompt, the frontend should show a comparison table or card layout with each model's response text, latency in milliseconds, token usage, and cost in USD. The cheapest model appears first because your backend sorts results by cost_usd.

If you also call the /api/crossover endpoint, you should see a recommendation string explaining which model saves money at your specified usage volume.

Frontend loads but no results appear?

  • Open your browser's Developer Tools (Cmd+Option+I on macOS, Ctrl+Shift+I on Windows) and check the **Console** tab for errors.
  • If you see a CORS error, confirm your frontend is fetching from the correct production URL with the /api/ prefix.
  • If you see a 422 error, your request body format does not match what the endpoint expects. Confirm you are sending {"prompt": "..."} as JSON with the Content-Type: application/json header.

Why is my frontend not receiving data from the Python benchmark endpoint?

  • If your frontend is in a separate Vercel project, redeploy it after updating the fetch URLs by running:
vercel deploy --prod
  • Refresh your Vercel production URL in the browser and run one final comparison to confirm everything works.

✔️ Awesome, I've got everything!

Great. Make sure you have saved all your files and your production deployment is live.

ⓧ I'd like to double check the full code

Here is the complete pyproject.toml:

[project]
name = "ai-model-comparator"
version = "1.0.0"
requires-python = ">=3.12"
dependencies = [
    "fastapi>=0.117.1",
    "anthropic>=0.42.0",
    "openai>=1.82.0",
    "google-genai>=2.3.0",
]

Here is the complete app.py:

import asyncio
import time
from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from providers.anthropic_client import call_anthropic
from providers.openai_client import call_openai
from providers.gemini_client import call_gemini

app = FastAPI()

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

PRICING = {
    "claude-haiku-4-5": {"input": 1.00, "output": 5.00},
    "claude-sonnet-5": {"input": 2.00, "output": 10.00},
    "gpt-4o-mini": {"input": 0.15, "output": 0.60},
    "gpt-4o": {"input": 2.50, "output": 10.00},
    "gemini-2.5-flash": {"input": 0.15, "output": 3.50},
    "gemini-3.5-flash": {"input": 1.50, "output": 9.00},
}


class CompareRequest(BaseModel):
    prompt: str
    models: list[str] | None = None


class CrossoverRequest(BaseModel):
    queries_per_day: int
    results: list[dict]


class BenchmarkResult(BaseModel):
    model: str
    provider: str
    response_text: str
    input_tokens: int
    output_tokens: int
    latency_ms: float
    cost_usd: float
    status: str


def compute_cost(model: str, input_tokens: int, output_tokens: int) -> float:
    if model not in PRICING:
        return 0.0
    prices = PRICING[model]
    input_cost = (input_tokens / 1_000_000) * prices["input"]
    output_cost = (output_tokens / 1_000_000) * prices["output"]
    return round(input_cost + output_cost, 6)


@app.get("/api")
def root():
    return {"status": "ok", "service": "ai-model-comparator"}


@app.post("/api/benchmark")
async def benchmark(request: CompareRequest):
    prompt = request.prompt
    if not prompt:
        raise HTTPException(status_code=400, detail="Prompt is required")

    tasks = [
        call_provider_safe("anthropic", "claude-haiku-4-5", prompt),
        call_provider_safe("openai", "gpt-4o-mini", prompt),
        call_provider_safe("gemini", "gemini-2.5-flash", prompt),
    ]
    results = await asyncio.gather(*tasks)
    results_sorted = sorted(
        [r for r in results if r["status"] == "success"],
        key=lambda r: r["cost_usd"],
    )
    failed = [r for r in results if r["status"] != "success"]
    return {"results": results_sorted, "failed": failed}


async def call_provider_safe(provider: str, model: str, prompt: str) -> dict:
    try:
        start = time.perf_counter()
        if provider == "anthropic":
            response = await call_anthropic(prompt, model)
        elif provider == "openai":
            response = await call_openai(prompt, model)
        elif provider == "gemini":
            response = await call_gemini(prompt, model)
        else:
            raise ValueError(f"Unknown provider: {provider}")
        elapsed_ms = (time.perf_counter() - start) * 1000

        cost = compute_cost(model, response["input_tokens"], response["output_tokens"])

        return {
            "model": model,
            "provider": provider,
            "response_text": response["text"],
            "input_tokens": response["input_tokens"],
            "output_tokens": response["output_tokens"],
            "latency_ms": round(elapsed_ms, 1),
            "cost_usd": cost,
            "status": "success",
        }
    except Exception as e:
        return {
            "model": model,
            "provider": provider,
            "response_text": "",
            "input_tokens": 0,
            "output_tokens": 0,
            "latency_ms": 0,
            "cost_usd": 0,
            "status": f"error: {str(e)}",
        }


@app.post("/api/crossover")
async def crossover(request: CrossoverRequest):
    if not request.results or len(request.results) < 2:
        raise HTTPException(
            status_code=400, detail="Need at least 2 model results to compare"
        )

    monthly_costs = []
    for result in request.results:
        monthly = result["cost_usd"] * request.queries_per_day * 30
        monthly_costs.append({
            "model": result["model"],
            "cost_per_request": result["cost_usd"],
            "monthly_cost": round(monthly, 2),
            "latency_ms": result["latency_ms"],
        })

    monthly_costs.sort(key=lambda x: x["monthly_cost"])

    cheapest = monthly_costs[0]
    most_expensive = monthly_costs[-1]
    savings_monthly = round(most_expensive["monthly_cost"] - cheapest["monthly_cost"], 2)

    recommendation = (
        f"At {request.queries_per_day} queries/day, {cheapest['model']} "
        f"saves ${savings_monthly}/month over {most_expensive['model']}. "
    )
    if cheapest["latency_ms"] < most_expensive["latency_ms"]:
        recommendation += "It is also faster."
    else:
        latency_diff = round(cheapest["latency_ms"] - most_expensive["latency_ms"], 0)
        recommendation += (
            f"Trade-off: {int(latency_diff)}ms slower per request."
        )

    return {
        "queries_per_day": request.queries_per_day,
        "monthly_costs": monthly_costs,
        "savings_monthly": savings_monthly,
        "recommendation": recommendation,
    }

Here is providers/anthropic_client.py:

import os
import anthropic


async def call_anthropic(prompt: str, model: str = "claude-haiku-4-5") -> dict:
    client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
    message = client.messages.create(
        model=model,
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
    return {
        "text": message.content[0].text,
        "input_tokens": message.usage.input_tokens,
        "output_tokens": message.usage.output_tokens,
    }

Here is providers/openai_client.py:

import os
from openai import OpenAI


async def call_openai(prompt: str, model: str = "gpt-4o-mini") -> dict:
    client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
    response = client.chat.completions.create(
        model=model,
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
    choice = response.choices[0]
    return {
        "text": choice.message.content,
        "input_tokens": response.usage.prompt_tokens,
        "output_tokens": response.usage.completion_tokens,
    }

Here is providers/gemini_client.py:

import os
from google import genai


async def call_gemini(prompt: str, model: str = "gemini-2.5-flash") -> dict:
    client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
    response = client.models.generate_content(
        model=model,
        contents=prompt,
    )
    usage = response.usage_metadata
    return {
        "text": response.text,
        "input_tokens": usage.prompt_token_count,
        "output_tokens": usage.candidates_token_count,
    }

Secret mission

LLM-as-Judge Quality Scoring

Your benchmark tells you which model is cheapest and fastest. But which model gives the best answer? Add a meta-evaluation endpoint that uses a cheap model to rate the output quality of more expensive models on a 1-10 scale.

Clean Up Your Resources

Clean Up Your Resources

Your Vercel deployment runs on the free tier, so it costs nothing to keep. API costs only accrue when someone makes comparison requests to your endpoints.

Cost warning

Each comparison request calls three AI providers, which costs real money. If your app is publicly accessible and receives traffic, you could accumulate unexpected charges. Monitor usage in each provider's dashboard and consider removing API keys from Vercel if you are not actively using the app.

Resources you used:

  • Vercel project deployment (free tier).
  • Vercel environment variables (ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY).
  • Anthropic API key (console.anthropic.com).
  • OpenAI API key (platform.openai.com).
  • Google Gemini API key (aistudio.google.com).
  • Local project files on your Desktop.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Keep everything running

No action needed. Choose this if you are still actively building or want the app available for demos.

  • The Vercel deployment costs nothing on the free tier.
  • API costs only accrue when users make comparison requests.
  • Monitor usage in each provider's dashboard to catch unexpected charges early.

Pause - I'll come back to this later

Remove your API keys from Vercel to prevent any requests from reaching paid providers. Your code and deployment stay intact.

  • Open your Vercel project in the browser.
  • Go to Settings in the sidebar, then select Environment Variables.
  • Delete ANTHROPIC_API_KEY, OPENAI_API_KEY, and GEMINI_API_KEY from the environment variables list.
  • Your endpoints will return errors if called, but no API charges will accrue.
  • When you are ready to resume, re-add the keys and redeploy.

Delete - I don't want to use this again

Remove all project resources and revoke API keys to start fresh.

Delete the Vercel project:

  • Open your Vercel dashboard and select the ai-model-comparator project.
  • Click Settings in the sidebar.
  • Scroll to the bottom of the General page and find the Delete Project section.
  • Click Delete, enter the project name to confirm, then click Continue.

Revoke your API keys:

  • Anthropic: Sign in to console.anthropic.com, navigate to the API Keys section, locate your key, and disable or delete it.
  • OpenAI: Sign in to platform.openai.com, go to the API Keys page, find your key, and click Revoke.
  • Google: Sign in to aistudio.google.com/apikey, locate your key on the API Keys page, and delete it.

Delete local project files:

  • Remove the project folder from your Desktop by running this command:
cd ~/Desktop
rm -rf ai-model-comparator
  • On Windows, open PowerShell and run:
cd ~\Desktop
Remove-Item -Recurse -Force ai-model-comparator

That's a Wrap!

That's a Wrap!

Nice work! You just built a production Python backend from scratch that benchmarks AI models across three major providers and returns real performance data.

You've learned how to:

  • Build a FastAPI application that integrates three AI provider APIs (Anthropic, OpenAI, Google Gemini) with proper authentication, error handling, and response parsing.
  • Implement real-time benchmarking that measures latency with time.perf_counter() and computes actual cost-per-request from token counts and provider pricing.
  • Build a cost-crossover analysis endpoint that computes break-even points between models at different usage volumes, and deploy the full application to Vercel.
  • Secret Mission: Added an LLM-as-judge quality scoring endpoint that uses a cheap model to rate the output quality of more expensive models.

Ready to quiz yourself?