Build a Multi-Provider Streaming Chat

Stream live LLM responses from OpenAI, Anthropic, and Hugging Face in Python.

Introduction

30 Second Summary

When you chat with AI tools like ChatGPT or Claude, responses appear word by word in real time. That streaming effect isn't magic. It's a deliberate engineering pattern that every production AI app uses.

In this project, you will build a Python web server that streams responses from three different LLM providers (OpenAI, Anthropic, and Hugging Face) to a browser chat interface in real time.

What You'll Build

A FastAPI backend with three streaming endpoints that pipe LLM tokens to a browser chat UI using Server-Sent Events.

By the end of this project, you'll have:

  • A FastAPI server with /stream/openai, /stream/anthropic, and /stream/huggingface endpoints that stream LLM responses token by token.
  • A browser chat interface with a provider selector dropdown and real-time streaming display.
  • Hands-on experience with three different streaming API patterns and how to normalize them into a consistent SSE format.
  • Secret Mission: Add a "Race Mode" that streams all three providers simultaneously and displays them side-by-side in real time.

Are there any prerequisites?

You should be comfortable with basic Python (variables, functions, loops) and have a general understanding of how HTTP requests work. No prior experience with FastAPI, streaming, or LLM APIs is required.

Before We Start

Before you can stream responses from any LLM provider, you need a Python environment with the right libraries and valid API keys.

In this step, you will create a PyCharm project, install packages for three different providers, and store your credentials in a safe place.

In this step, get ready to:
  • Create a new PyCharm project with a virtual environment.
  • Install all required Python packages.
  • Get API keys from OpenAI, Anthropic, and Hugging Face and store them in a .env file.
Create a new PyCharm project

PyCharm manages a dedicated virtual environment for each project you create. This keeps your packages isolated from other Python work on your machine.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type PyCharm and press Enter to open it.
  • On the Welcome screen, click New Project.
  • In the Location field, change the last folder name to streaming-chat.
  • Leave the interpreter set to Project venv (the default). PyCharm will create a virtual environment automatically.
  • Uncheck Create a main.py welcome script if it is selected.
  • Click Create.

What is a virtual environment?

A virtual environment is an isolated Python installation that belongs to a single project. Packages you install inside it do not affect any other project on your machine.

PyCharm creates one automatically inside a venv folder when you start a new project. You never need to activate it manually because PyCharm handles that for you.

  • Open the built-in terminal by pressing ⌥F12 (macOS) or Alt+F12 (Windows).

You should see a terminal panel at the bottom of PyCharm with your virtual environment already activated (the prompt starts with (streaming-chat) or similar).

Don't see the virtual environment name in your prompt?

Close the terminal tab and reopen it with ⌥F12 / Alt+F12. PyCharm sometimes needs a fresh terminal to pick up the newly created environment.

Still stuck? Ask for help.

Install all dependencies

Your project needs six packages: a web framework, three LLM provider libraries, an SSE helper, and a dotenv loader.

  • Install every package in one go by running this command in the terminal:
pip install "fastapi[standard]" openai anthropic huggingface-hub sse-starlette python-dotenv
  • Wait for the installation to finish. You should see a success message ending with Successfully installed followed by a list of package names.

What are these packages?

  • fastapi[standard] is the web framework that will serve your streaming endpoints. The [standard] extra includes uvicorn (the server) and other useful defaults.
  • openai and anthropic are the official Python SDKs for OpenAI and Anthropic.
  • huggingface-hub provides the inference client for Hugging Face models.
  • sse-starlette adds Server-Sent Events response support to FastAPI.
  • python-dotenv loads environment variables from a .env file so you never hard-code secrets.

Seeing a permission error or install failure?

Make sure your terminal shows the virtual environment prefix (e.g. (streaming-chat)) before the prompt. If it does not, close and reopen the terminal so PyCharm activates the environment.

If you see a network timeout, check your internet connection and try the command again.

Help me troubleshoot my pip install failure.

Get your API keys and create the .env file

Each LLM provider requires an API key to authenticate your requests. You will store all three keys in a single .env file that your server reads at startup.

  • Get your OpenAI key by visiting platform.openai.com. Sign in (or create an account), navigate to the API keys section, and generate a new key. Copy it somewhere safe.
  • Get your Anthropic key by visiting console.anthropic.com. Sign in (or create an account), navigate to the API keys section, and generate a new key. Copy it somewhere safe.
  • Get your Hugging Face token by visiting huggingface.co/settings/tokens. Sign in (or create a free account), click New token, give it a name, and copy the generated token.

How much do these cost?

OpenAI and Anthropic charge per token (fractions of a cent per response). A full run-through of this project costs well under $0.50 total.

Hugging Face Inference Providers are free for open-source models, so the Hugging Face endpoint costs nothing.

  • Create a new file called .env in the root of your streaming-chat project. Right-click the project root in PyCharm's Project tool window, select New, then File, and type .env.
  • Paste the following into your .env file, replacing each placeholder with your actual key:
OPENAI_API_KEY=your-openai-key-here
ANTHROPIC_API_KEY=your-anthropic-key-here
HF_TOKEN=your-hf-token-here

What does this file do?

The python-dotenv library reads this file when your server starts and loads each line as an environment variable. Your code can then access the keys with os.getenv() without ever hard-coding secrets in source files.

Never commit a .env file to version control. If you add Git later, put .env in your .gitignore.

  • Save .env by pressing Cmd+S (macOS) or Ctrl+S (Windows).

✔️ Awesome, I've got everything!

Great. Double check you have replaced each placeholder with your real key before moving on.

ⓧ I'd like to double check the full code

Your .env file should look exactly like this (with your own keys in place of the placeholders):

OPENAI_API_KEY=your-openai-key-here
ANTHROPIC_API_KEY=your-anthropic-key-here
HF_TOKEN=your-hf-token-here

Your environment is ready with all three provider libraries installed and your API keys securely stored. Next up, you will write your first streaming endpoint with OpenAI.

Build the OpenAI Streaming Endpoint

When you chat with an AI assistant like ChatGPT, tokens appear one by one instead of making you wait for the entire response.

This streaming pattern is powered by Server-Sent Events (SSE). In this step, you'll create a FastAPI server that connects to OpenAI's Responses API and pipes tokens back to the browser as they're generated.

In this step, get ready to:
  • Set up the FastAPI application with CORS middleware and provider clients.
  • Write an async streaming endpoint for OpenAI.
  • Test the endpoint by running the development server.
Set up the FastAPI application

Every streaming chat app needs a backend server to sit between the browser and the LLM provider.

Your server will load API keys from the .env file, initialize clients for all three providers, and define the shape of incoming requests using Pydantic.

  • In PyCharm, right-click the streaming-chat folder in the Project panel on the left.
  • Select New, then Python File.
  • Name it main and press Enter.
  • Paste the following app setup code into main.py:
import os
from dotenv import load_dotenv
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from fastapi.staticfiles import StaticFiles
from pydantic import BaseModel
from sse_starlette import EventSourceResponse
from openai import AsyncOpenAI
from anthropic import AsyncAnthropic
from huggingface_hub import AsyncInferenceClient

# Load API keys from the .env file
load_dotenv()

app = FastAPI()

# Allow the browser frontend to make requests to this server
app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

# Initialize async clients for each LLM provider
openai_client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))
anthropic_client = AsyncAnthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
hf_client = AsyncInferenceClient(token=os.getenv("HF_TOKEN"))


# Define the expected shape of incoming chat requests
class ChatRequest(BaseModel):
    message: str

What does this code do?

  • load_dotenv() reads the .env file and makes your API keys available via os.getenv().
  • CORS middleware allows your browser frontend to call the backend without being blocked by browser security rules.
  • The three client objects (openai_client, anthropic_client, hf_client) are async-ready connections to each provider, initialized once and reused for every request.
  • ChatRequest is a Pydantic model that ensures every incoming request has a message field. FastAPI uses it to validate requests automatically.

Why initialize all three clients now?

You won't use the Anthropic or Hugging Face clients until later steps. Initializing them now means you only touch the imports and setup code once, and each future step can jump straight to building its endpoint.

  • Don't save the file yet. You'll add the endpoint next.

Seeing red underlines on your imports?

PyCharm may show import errors if it hasn't detected your virtual environment. Click the Python interpreter selector in the bottom-right corner and make sure it points to the .venv inside your project.

If the underlines persist, confirm you installed all packages in Step 1 by checking the terminal output for errors.

Help me fix import errors in PyCharm

Write the OpenAI streaming endpoint

The OpenAI Responses API sends streaming data as a series of typed events. Your endpoint needs to filter for the response.output_text.delta event type, which carries each new token of generated text.

  • Add the following endpoint below the ChatRequest class in main.py:
@app.post("/stream/openai")
async def stream_openai(request: ChatRequest):
    async def generate():
        # Call the OpenAI Responses API with streaming enabled
        stream = await openai_client.responses.create(
            model="gpt-4o-mini",
            input=request.message,
            stream=True,
        )
        # Filter for text delta events and yield each token
        async for event in stream:
            if event.type == "response.output_text.delta":
                yield {"data": event.delta}
        # Signal to the frontend that the stream is complete
        yield {"data": "[DONE]"}

    return EventSourceResponse(generate())

What does this endpoint do?

  • The @app.post("/stream/openai") decorator tells FastAPI to handle POST requests at that path.
  • generate() is an async generator. Instead of building the full response in memory, it yields one token at a time as they arrive from OpenAI.
  • The if event.type == "response.output_text.delta" filter skips non-text events (like metadata) and only passes actual generated tokens.
  • EventSourceResponse wraps the generator and formats each yielded dict as a proper SSE message that the browser can parse.
  • Save main.py.

IndentationError when you save?

The generate() function is nested inside stream_openai(). Make sure generate() is indented one level deeper than async def stream_openai, and the return statement is at the same level as async def generate().

Help me fix my indentation in the streaming endpoint

Test the streaming endpoint

FastAPI includes a development server that auto-reloads when you change your code. You'll start it now and send a test request to confirm streaming works.

  • Start the development server by running this command in your PyCharm terminal:
fastapi dev main.py

You should see output ending with a line that says Uvicorn running on http://127.0.0.1:8000.

What does fastapi dev do?

fastapi dev starts a local development server powered by Uvicorn. It watches your files and automatically restarts when you make changes, so you never have to manually stop and restart during development.

  • Open a second terminal tab in PyCharm by clicking the + icon in the terminal panel.
  • Test the streaming endpoint by running this command:
curl -X POST http://127.0.0.1:8000/stream/openai -H "Content-Type: application/json" -d "{\"message\": \"Hello\"}"

You should see lines starting with data: appearing one at a time in your terminal, each containing a word or part of a word from OpenAI's response. The final line will say data: [DONE].

Seeing a 422 or connection refused error?

  • If you see Connection refused, check that the server is still running in your first terminal tab.
  • A 422 error means the request body is malformed. Make sure you include -H "Content-Type: application/json" and that the JSON has a message field.
  • If you see an authentication error from OpenAI, double-check that your OPENAI_API_KEY in .env is correct and has credits available.

Help me debug my streaming endpoint

✔️ Awesome, I've got everything!

Great. Make sure you've saved main.py and that the server is still running.

ⓧ I'd like to double check the full code

Here is the complete main.py at this point:

import os
from dotenv import load_dotenv
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from fastapi.staticfiles import StaticFiles
from pydantic import BaseModel
from sse_starlette import EventSourceResponse
from openai import AsyncOpenAI
from anthropic import AsyncAnthropic
from huggingface_hub import AsyncInferenceClient

load_dotenv()

app = FastAPI()

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

openai_client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))
anthropic_client = AsyncAnthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
hf_client = AsyncInferenceClient(token=os.getenv("HF_TOKEN"))


class ChatRequest(BaseModel):
    message: str


@app.post("/stream/openai")
async def stream_openai(request: ChatRequest):
    async def generate():
        stream = await openai_client.responses.create(
            model="gpt-4o-mini",
            input=request.message,
            stream=True,
        )
        async for event in stream:
            if event.type == "response.output_text.delta":
                yield {"data": event.delta}
        yield {"data": "[DONE]"}

    return EventSourceResponse(generate())

Your first streaming endpoint is live. Next up, you'll add the Anthropic endpoint and see how a different provider handles the same streaming pattern.

Add the Anthropic Streaming Endpoint

Your OpenAI endpoint is streaming tokens to the terminal. Now you'll add a second provider.

Anthropic's streaming model differs from OpenAI's. Instead of filtering a single event type from a generic event stream, Anthropic provides a dedicated text_stream iterator that yields only the text deltas you care about. In this step, you'll add a second endpoint that streams Claude's responses.

In this step, get ready to:
  • Add a /stream/anthropic POST endpoint with an async streaming generator.
  • Test the endpoint and confirm Claude's response streams token by token.
Add the /stream/anthropic endpoint

Anthropic's Python SDK uses an async context manager pattern for streaming. You open the stream with async with, iterate over its text_stream property, and each iteration gives you a plain text string.

  • In main.py, find the closing line of the stream_openai function:
    return EventSourceResponse(generate())
  • Add the Anthropic endpoint below it by copying this code after a blank line:
@app.post("/stream/anthropic")
async def stream_anthropic(request: ChatRequest):
    async def generate():
        # Open a streaming connection to Claude
        async with anthropic_client.messages.stream(
            model="claude-haiku-4-5",
            max_tokens=1024,
            messages=[{"role": "user", "content": request.message}],
        ) as stream:
            # text_stream yields only the text deltas
            async for text in stream.text_stream:
                yield {"data": text}
        yield {"data": "[DONE]"}

    return EventSourceResponse(generate())

What does this code do?

This endpoint follows the same pattern as the OpenAI one: an async generator wrapped in an EventSourceResponse. The key differences are in how Anthropic handles streaming.

  • anthropic_client.messages.stream() opens the connection as an async context manager. The stream stays open until the async with block exits.
  • stream.text_stream is a dedicated iterator that yields only text strings. No event-type filtering needed.
  • model="claude-haiku-4-5" selects Claude Haiku 4.5, Anthropic's fastest and cheapest model at $1/1M input tokens.
  • max_tokens=1024 caps the response length. Anthropic requires this parameter for all requests.

Why is Anthropic's pattern different from OpenAI's?

OpenAI's Responses API returns a flat event stream where you filter by event type. Anthropic wraps streaming in a context manager so the SDK can guarantee the connection is properly closed, even if your code raises an exception mid-stream.

The text_stream convenience iterator hides the raw Server-Sent Events protocol entirely. Each provider solves the same problem (deliver tokens incrementally) with a different developer experience.

  • Save main.py.

Your dev server auto-reloads when you save, so the new endpoint is already live.

Server crashed after saving?

Check for indentation errors. The async with block must be indented inside generate(), and yield {"data": "[DONE]"} must be at the same level as async with (inside generate() but outside the context manager).

Also confirm the @app.post decorator is at the module level (no indentation), not nested inside the previous function.

help me fix an indentation error in my Anthropic streaming endpoint

Test the Anthropic endpoint

With the server running, you can send a test request to confirm Claude streams its response.

  • Open a second terminal in PyCharm by clicking the + icon in the Terminal panel.
  • Send a test request by running this command:
curl -X POST http://127.0.0.1:8000/stream/anthropic -H "Content-Type: application/json" -d '{"message": "Hello"}'
  • Watch the terminal output. You should see Claude's response appear as a series of data: lines, each containing a small chunk of text arriving incrementally.

What should I see?

The output looks similar to the OpenAI endpoint, but each data: line contains a text fragment from Claude. The chunks may be smaller or larger than OpenAI's depending on how Anthropic buffers its output. The final line will be data: [DONE].

Getting a 401 or authentication error?

Double-check your ANTHROPIC_API_KEY value in the .env file. Make sure there are no extra spaces or quotes around the key.

If you just created your Anthropic account, confirm you have API credits available at console.anthropic.com.

help me debug an authentication error with the Anthropic API

✔️ Awesome, I've got everything!

Great. Double check you've saved main.py and that your server is still running.

ⓧ I'd like to double check the full code

Here is the complete main.py at this point:

import os
from dotenv import load_dotenv
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from fastapi.staticfiles import StaticFiles
from pydantic import BaseModel
from sse_starlette import EventSourceResponse
from openai import AsyncOpenAI
from anthropic import AsyncAnthropic
from huggingface_hub import AsyncInferenceClient

load_dotenv()

app = FastAPI()

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

openai_client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))
anthropic_client = AsyncAnthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
hf_client = AsyncInferenceClient(token=os.getenv("HF_TOKEN"))


class ChatRequest(BaseModel):
    message: str


@app.post("/stream/openai")
async def stream_openai(request: ChatRequest):
    async def generate():
        stream = await openai_client.responses.create(
            model="gpt-4o-mini",
            input=request.message,
            stream=True,
        )
        async for event in stream:
            if event.type == "response.output_text.delta":
                yield {"data": event.delta}
        yield {"data": "[DONE]"}

    return EventSourceResponse(generate())


@app.post("/stream/anthropic")
async def stream_anthropic(request: ChatRequest):
    async def generate():
        async with anthropic_client.messages.stream(
            model="claude-haiku-4-5",
            max_tokens=1024,
            messages=[{"role": "user", "content": request.message}],
        ) as stream:
            async for text in stream.text_stream:
                yield {"data": text}
        yield {"data": "[DONE]"}

    return EventSourceResponse(generate())

Two providers down, one to go. Next up, you'll add the Hugging Face endpoint and connect to a free open-source model.

Add the Hugging Face Streaming Endpoint

You now have two streaming endpoints working. OpenAI uses the Responses API with event filtering. Anthropic uses a context-managed stream with text_stream.

The third provider, Hugging Face, gives you access to open-source models for free. Its streaming interface follows the OpenAI-compatible chat.completions pattern, so the code will feel familiar.

In this step, get ready to:
  • Add a streaming endpoint for Hugging Face's open-source models.
  • Test that all three providers now stream responses.
Add the Hugging Face endpoint to your server

The hf_client you set up in Step 2 already connects to Hugging Face's Inference Providers. Now you need an endpoint that uses it to stream responses from an open-source LLM.

  • In main.py, find the stream_anthropic function. Add the following code below it:
@app.post("/stream/huggingface")
async def stream_huggingface(request: ChatRequest):
    async def generate():
        # Open a streaming connection to a free open-source model
        stream = await hf_client.chat.completions.create(
            model="meta-llama/Meta-Llama-3-8B-Instruct",
            messages=[{"role": "user", "content": request.message}],
            max_tokens=1024,
            stream=True,
        )
        # Yield each token as it arrives
        async for chunk in stream:
            content = chunk.choices[0].delta.content
            if content:
                yield {"data": content}
        yield {"data": "[DONE]"}

    return EventSourceResponse(generate())

What does this code do?

This endpoint follows the same pattern as OpenAI and Anthropic: define an async generator, stream from the provider, yield each token, signal completion.

  • hf_client.chat.completions.create(stream=True) opens a streaming connection to the meta-llama/Meta-Llama-3-8B-Instruct model. This is a free open-source model hosted by Hugging Face.
  • chunk.choices[0].delta.content extracts the text token from each chunk. The if content: check skips empty chunks that arrive between tokens.
  • The [DONE] sentinel tells the frontend the stream is complete.

Why does this look like the OpenAI pattern?

Hugging Face's Inference Providers use an OpenAI-compatible interface. The chat.completions.create method, the choices[0].delta.content structure, and the stream=True flag all mirror OpenAI's Chat Completions API.

This is different from how we called OpenAI in this project (we used the Responses API). But the chunk structure is the same pattern many providers adopt.

  • Save main.py.
  • In the PyCharm terminal, stop your server with Ctrl+C and restart it by running:
fastapi dev main.py
  • Test the new endpoint by running this in a separate terminal:
curl -X POST http://127.0.0.1:8000/stream/huggingface -H "Content-Type: application/json" -d '{"message": "Hello"}'

You should see a stream of data: lines appearing incrementally in your terminal, ending with data: [DONE].

Not seeing a streamed response?

  • Check that your HF_TOKEN in .env is a valid Hugging Face access token with the Inference Providers permission enabled.
  • Make sure you restarted the server after saving your changes.
  • If you see a 429 rate limit error, wait a moment and try again. Free-tier models can have brief rate limits.

Help me debug why my Hugging Face streaming endpoint is not returning tokens.

✔️ Awesome, I've got everything!

Great. Double check you have saved main.py and all three endpoints are responding.

ⓧ I'd like to double check the full code

Here is the complete main.py after this step:

import os
from dotenv import load_dotenv
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from fastapi.staticfiles import StaticFiles
from pydantic import BaseModel
from sse_starlette import EventSourceResponse
from openai import AsyncOpenAI
from anthropic import AsyncAnthropic
from huggingface_hub import AsyncInferenceClient

load_dotenv()

app = FastAPI()

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

openai_client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))
anthropic_client = AsyncAnthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
hf_client = AsyncInferenceClient(token=os.getenv("HF_TOKEN"))


class ChatRequest(BaseModel):
    message: str


@app.post("/stream/openai")
async def stream_openai(request: ChatRequest):
    async def generate():
        stream = await openai_client.responses.create(
            model="gpt-4o-mini",
            input=request.message,
            stream=True,
        )
        async for event in stream:
            if event.type == "response.output_text.delta":
                yield {"data": event.delta}
        yield {"data": "[DONE]"}

    return EventSourceResponse(generate())


@app.post("/stream/anthropic")
async def stream_anthropic(request: ChatRequest):
    async def generate():
        async with anthropic_client.messages.stream(
            model="claude-haiku-4-5",
            max_tokens=1024,
            messages=[{"role": "user", "content": request.message}],
        ) as stream:
            async for text in stream.text_stream:
                yield {"data": text}
        yield {"data": "[DONE]"}

    return EventSourceResponse(generate())


@app.post("/stream/huggingface")
async def stream_huggingface(request: ChatRequest):
    async def generate():
        stream = await hf_client.chat.completions.create(
            model="meta-llama/Meta-Llama-3-8B-Instruct",
            messages=[{"role": "user", "content": request.message}],
            max_tokens=1024,
            stream=True,
        )
        async for chunk in stream:
            content = chunk.choices[0].delta.content
            if content:
                yield {"data": content}
        yield {"data": "[DONE]"}

    return EventSourceResponse(generate())

All three streaming endpoints are live. Next up, you will build the browser chat UI that connects to these endpoints and renders tokens in real time.

Build the Chat Frontend

Your backend streams tokens from three different LLM providers. But right now, the only way to see those tokens is through curl or an HTTP client.

In this step, you'll build a browser-based chat interface that lets you pick a provider, type a message, and watch the response stream in token by token. The frontend uses the EventSource browser pattern to read Server-Sent Events from your backend in real time.

In this step, get ready to:
  • Create the HTML chat interface with a provider selector and styling.
  • Write JavaScript that streams tokens from your backend and displays them live.
  • Mount the static files in FastAPI so the chat UI loads at your server's root URL.
Create the chat interface

Your chat UI lives in a single HTML file. It includes a provider dropdown, a text input, a send button, a chat container, and all the CSS and JavaScript needed to stream responses from your backend.

  • In the PyCharm terminal, create the static directory by running this command:
mkdir static
  • Confirm the directory exists by running:
ls

You should see static listed alongside main.py and .env.

  • Create a new file called index.html inside the static folder. In PyCharm, right-click the static folder in the project sidebar, select New, then HTML File.
  • Name it index (PyCharm adds the .html extension automatically).
  • Replace everything in the file with the following code:
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Multi-Provider Streaming Chat</title>
    <style>
        * { margin: 0; padding: 0; box-sizing: border-box; }
        body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif; background: #1a1a2e; color: #eee; height: 100vh; display: flex; flex-direction: column; }
        header { padding: 1rem; background: #16213e; text-align: center; }
        header h1 { font-size: 1.2rem; color: #a8d8ea; }
        #chat { flex: 1; overflow-y: auto; padding: 1rem; display: flex; flex-direction: column; gap: 0.75rem; }
        .message { max-width: 80%; padding: 0.75rem 1rem; border-radius: 1rem; line-height: 1.5; white-space: pre-wrap; }
        .user { align-self: flex-end; background: #0f3460; }
        .assistant { align-self: flex-start; background: #1a1a3e; border: 1px solid #333; }
        .provider-tag { font-size: 0.7rem; color: #a8d8ea; margin-bottom: 0.25rem; text-transform: uppercase; letter-spacing: 0.05em; }
        #input-area { padding: 1rem; background: #16213e; display: flex; gap: 0.5rem; }
        #input-area select { padding: 0.5rem; border-radius: 0.5rem; border: none; background: #0f3460; color: #eee; }
        #input-area input { flex: 1; padding: 0.75rem; border-radius: 0.5rem; border: none; background: #0f3460; color: #eee; font-size: 1rem; }
        #input-area input::placeholder { color: #666; }
        #input-area button { padding: 0.75rem 1.5rem; border-radius: 0.5rem; border: none; background: #a8d8ea; color: #1a1a2e; font-weight: bold; cursor: pointer; }
        #input-area button:hover { background: #86c5d8; }
        #input-area button:disabled { opacity: 0.5; cursor: not-allowed; }
    </style>
</head>
<body>
    <header>
        <h1>Multi-Provider Streaming Chat</h1>
    </header>
    <div id="chat"></div>
    <form id="input-area">
        <select id="provider">
            <option value="openai">OpenAI</option>
            <option value="anthropic">Anthropic</option>
            <option value="huggingface">Hugging Face</option>
        </select>
        <input type="text" id="message" placeholder="Type a message..." autocomplete="off" />
        <button type="submit">Send</button>
    </form>
    <script>
        const chat = document.getElementById('chat');
        const form = document.getElementById('input-area');
        const messageInput = document.getElementById('message');
        const providerSelect = document.getElementById('provider');
        const sendButton = form.querySelector('button');

        form.addEventListener('submit', async (e) => {
            e.preventDefault();
            const message = messageInput.value.trim();
            if (!message) return;

            const provider = providerSelect.value;
            addMessage(message, 'user');
            messageInput.value = '';
            sendButton.disabled = true;

            const assistantBubble = addMessage('', 'assistant', provider);

            try {
                const response = await fetch(`/stream/${provider}`, {
                    method: 'POST',
                    headers: { 'Content-Type': 'application/json' },
                    body: JSON.stringify({ message }),
                });

                const reader = response.body.getReader();
                const decoder = new TextDecoder();
                let buffer = '';

                while (true) {
                    const { done, value } = await reader.read();
                    if (done) break;

                    buffer += decoder.decode(value, { stream: true });
                    const lines = buffer.split('\n');
                    buffer = lines.pop();

                    for (const line of lines) {
                        if (line.startsWith('data:')) {
                            const data = line.slice(5).trim();
                            if (data === '[DONE]') break;
                            assistantBubble.querySelector('.text').textContent += data;
                            chat.scrollTop = chat.scrollHeight;
                        }
                    }
                }
            } catch (error) {
                assistantBubble.querySelector('.text').textContent = 'Error: ' + error.message;
            }

            sendButton.disabled = false;
        });

        function addMessage(text, role, provider) {
            const div = document.createElement('div');
            div.className = `message ${role}`;
            if (role === 'assistant') {
                div.innerHTML = `<div class="provider-tag">${provider || ''}</div><span class="text">${text}</span>`;
            } else {
                div.innerHTML = `<span class="text">${text}</span>`;
            }
            chat.appendChild(div);
            chat.scrollTop = chat.scrollHeight;
            return div;
        }
    </script>
</body>
</html>

What does this code do?

This file is a self-contained chat application. Here's what each section handles:

  • The <style> block creates a dark-themed full-height layout with user messages aligned right (blue) and assistant messages aligned left (dark with a border).
  • The <form> at the bottom holds a provider dropdown, a text input, and a send button.
  • The <div id="chat"> container is where message bubbles appear as the conversation progresses.
  • The <script> section contains the JavaScript that handles form submission, streams the response, and renders tokens live.
  • Save static/index.html.

You won't see a change in the browser yet because FastAPI isn't serving this file. You'll wire that up in the final substep.

Understand the streaming JavaScript

The JavaScript you just added is the core of the real-time experience. It sends the user's message to your backend and reads the streamed response token by token. This is the same pattern used by ChatGPT, Claude.ai, and every production LLM frontend.

How does the streaming logic work?

Standard SSE via the browser's built-in EventSource API only supports GET requests. Since your endpoints need a POST body (the user's message), the JavaScript uses fetch instead and manually parses the SSE stream.

Here's the flow: fetch sends the POST request, then response.body.getReader() opens a stream reader. A while (true) loop calls reader.read() repeatedly, accumulating chunks in a buffer.

Each chunk is decoded with TextDecoder, split by newlines, and checked for data: prefixes. Each data payload is a token that gets appended to the assistant bubble's .text span. The sentinel value [DONE] signals the stream is finished.

What does addMessage do?

The addMessage function creates a new message bubble and appends it to the chat container. For assistant messages, it includes a .provider-tag label showing which LLM is responding.

It returns the DOM element so the streaming loop can keep appending tokens to the same bubble as they arrive.

Mount the static files and test the app

FastAPI can serve static files directly using the StaticFiles class, which is already imported in your main.py. You need to tell FastAPI to serve your static folder at the root URL so the chat page loads when you visit the server.

Why mount at the root?

Mounting with html=True tells FastAPI to serve index.html automatically when someone visits /. The mount MUST be placed AFTER all your API route definitions. Otherwise it would intercept requests meant for /stream/openai and the other endpoints.

  • In main.py, scroll to the very bottom of the file (after the stream_huggingface function).
  • Find this line (the last line of the function):
    return EventSourceResponse(generate())
  • Add the following line below it, at the module level (no indentation):
app.mount("/", StaticFiles(directory="static", html=True), name="static")

What does this line do?

This tells FastAPI to serve any file inside the static/ directory when no API route matches. The html=True flag makes it automatically serve index.html for directory paths.

  • Save main.py.
  • Start the server by running:
fastapi dev main.py
  • Open your browser and go to http://127.0.0.1:8000.

You should see a dark-themed chat interface with a provider dropdown on the left, a text input in the center, and a blue **Send** button on the right.

  • Select a provider from the dropdown (start with OpenAI).
  • Type Hello in the message field and click Send.

You should see your message appear as a blue bubble on the right, then an assistant bubble on the left that fills in character by character with the LLM's response. A small provider tag above the response shows which model is replying.

Chat page not loading?

  • Make sure the app.mount line is the LAST line in main.py. If it appears before your route definitions, it will intercept API requests.
  • Check that static/index.html exists at that exact path relative to main.py. The static folder must be a direct sibling of main.py, not nested inside another folder.
  • If the page loads but streaming fails, check the browser console for errors. A 422 response usually means the request body format is wrong.

Help me debug why my chat page isn't loading or streaming isn't working.

✔️ Awesome, I've got everything!

Great. Double check you've saved both main.py and static/index.html.

ⓧ I'd like to double check the full code

Here is the complete main.py at the end of this step:

import os
from dotenv import load_dotenv
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from fastapi.staticfiles import StaticFiles
from pydantic import BaseModel
from sse_starlette import EventSourceResponse
from openai import AsyncOpenAI
from anthropic import AsyncAnthropic
from huggingface_hub import AsyncInferenceClient

load_dotenv()

app = FastAPI()

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

openai_client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))
anthropic_client = AsyncAnthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
hf_client = AsyncInferenceClient(token=os.getenv("HF_TOKEN"))


class ChatRequest(BaseModel):
    message: str


@app.post("/stream/openai")
async def stream_openai(request: ChatRequest):
    async def generate():
        stream = await openai_client.responses.create(
            model="gpt-4o-mini",
            input=request.message,
            stream=True,
        )
        async for event in stream:
            if event.type == "response.output_text.delta":
                yield {"data": event.delta}
        yield {"data": "[DONE]"}

    return EventSourceResponse(generate())


@app.post("/stream/anthropic")
async def stream_anthropic(request: ChatRequest):
    async def generate():
        async with anthropic_client.messages.stream(
            model="claude-haiku-4-5",
            max_tokens=1024,
            messages=[{"role": "user", "content": request.message}],
        ) as stream:
            async for text in stream.text_stream:
                yield {"data": text}
        yield {"data": "[DONE]"}

    return EventSourceResponse(generate())


@app.post("/stream/huggingface")
async def stream_huggingface(request: ChatRequest):
    async def generate():
        stream = await hf_client.chat.completions.create(
            model="meta-llama/Meta-Llama-3-8B-Instruct",
            messages=[{"role": "user", "content": request.message}],
            max_tokens=1024,
            stream=True,
        )
        async for chunk in stream:
            content = chunk.choices[0].delta.content
            if content:
                yield {"data": content}
        yield {"data": "[DONE]"}

    return EventSourceResponse(generate())


app.mount("/", StaticFiles(directory="static", html=True), name="static")

Here is the complete static/index.html:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Multi-Provider Streaming Chat</title>
    <style>
        * { margin: 0; padding: 0; box-sizing: border-box; }
        body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif; background: #1a1a2e; color: #eee; height: 100vh; display: flex; flex-direction: column; }
        header { padding: 1rem; background: #16213e; text-align: center; }
        header h1 { font-size: 1.2rem; color: #a8d8ea; }
        #chat { flex: 1; overflow-y: auto; padding: 1rem; display: flex; flex-direction: column; gap: 0.75rem; }
        .message { max-width: 80%; padding: 0.75rem 1rem; border-radius: 1rem; line-height: 1.5; white-space: pre-wrap; }
        .user { align-self: flex-end; background: #0f3460; }
        .assistant { align-self: flex-start; background: #1a1a3e; border: 1px solid #333; }
        .provider-tag { font-size: 0.7rem; color: #a8d8ea; margin-bottom: 0.25rem; text-transform: uppercase; letter-spacing: 0.05em; }
        #input-area { padding: 1rem; background: #16213e; display: flex; gap: 0.5rem; }
        #input-area select { padding: 0.5rem; border-radius: 0.5rem; border: none; background: #0f3460; color: #eee; }
        #input-area input { flex: 1; padding: 0.75rem; border-radius: 0.5rem; border: none; background: #0f3460; color: #eee; font-size: 1rem; }
        #input-area input::placeholder { color: #666; }
        #input-area button { padding: 0.75rem 1.5rem; border-radius: 0.5rem; border: none; background: #a8d8ea; color: #1a1a2e; font-weight: bold; cursor: pointer; }
        #input-area button:hover { background: #86c5d8; }
        #input-area button:disabled { opacity: 0.5; cursor: not-allowed; }
    </style>
</head>
<body>
    <header>
        <h1>Multi-Provider Streaming Chat</h1>
    </header>
    <div id="chat"></div>
    <form id="input-area">
        <select id="provider">
            <option value="openai">OpenAI</option>
            <option value="anthropic">Anthropic</option>
            <option value="huggingface">Hugging Face</option>
        </select>
        <input type="text" id="message" placeholder="Type a message..." autocomplete="off" />
        <button type="submit">Send</button>
    </form>
    <script>
        const chat = document.getElementById('chat');
        const form = document.getElementById('input-area');
        const messageInput = document.getElementById('message');
        const providerSelect = document.getElementById('provider');
        const sendButton = form.querySelector('button');

        form.addEventListener('submit', async (e) => {
            e.preventDefault();
            const message = messageInput.value.trim();
            if (!message) return;

            const provider = providerSelect.value;
            addMessage(message, 'user');
            messageInput.value = '';
            sendButton.disabled = true;

            const assistantBubble = addMessage('', 'assistant', provider);

            try {
                const response = await fetch(`/stream/${provider}`, {
                    method: 'POST',
                    headers: { 'Content-Type': 'application/json' },
                    body: JSON.stringify({ message }),
                });

                const reader = response.body.getReader();
                const decoder = new TextDecoder();
                let buffer = '';

                while (true) {
                    const { done, value } = await reader.read();
                    if (done) break;

                    buffer += decoder.decode(value, { stream: true });
                    const lines = buffer.split('\n');
                    buffer = lines.pop();

                    for (const line of lines) {
                        if (line.startsWith('data:')) {
                            const data = line.slice(5).trim();
                            if (data === '[DONE]') break;
                            assistantBubble.querySelector('.text').textContent += data;
                            chat.scrollTop = chat.scrollHeight;
                        }
                    }
                }
            } catch (error) {
                assistantBubble.querySelector('.text').textContent = 'Error: ' + error.message;
            }

            sendButton.disabled = false;
        });

        function addMessage(text, role, provider) {
            const div = document.createElement('div');
            div.className = `message ${role}`;
            if (role === 'assistant') {
                div.innerHTML = `<div class="provider-tag">${provider || ''}</div><span class="text">${text}</span>`;
            } else {
                div.innerHTML = `<span class="text">${text}</span>`;
            }
            chat.appendChild(div);
            chat.scrollTop = chat.scrollHeight;
            return div;
        }
    </script>
</body>
</html>

Your multi-provider streaming chat app is fully functional. You can switch between OpenAI, Anthropic, and Hugging Face from the dropdown and watch each model's response style in real time. Next up, you'll take on the Secret Mission and race all three providers head-to-head.

Secret mission

Race Mode

What if you could fire the same prompt to all three providers at once and watch them race? Build a Race Mode that streams OpenAI, Anthropic, and Hugging Face side-by-side so you can compare speed and quality in real time.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

This project runs entirely locally. The only costs come from API calls to OpenAI and Anthropic, which are charged per token and only happen when you actively send messages.

Resources you used:

  • streaming-chat project folder (includes virtual environment, source files, and .env file).
  • FastAPI dev server (running process in your terminal).
  • OpenAI API key (from platform.openai.com).
  • Anthropic API key (from console.anthropic.com).
  • Hugging Face access token (from huggingface.co/settings/tokens).

Keep everything running

No action needed. Choose this if you're still actively building or want to experiment with new prompts and providers.

  • Your project folder stays on your machine with all code intact.
  • Your API keys remain active. OpenAI and Anthropic only charge when you send requests, so there is no idle cost.
  • Your Hugging Face access token remains active. Inference Providers for open-source models is free.
  • If your dev server is still running, you can keep chatting at http://127.0.0.1:8000.

Pause - I'll come back to this later

Stop the running server to free up your terminal. Your code and keys stay in place for next time.

  • Switch to the terminal running the FastAPI server in PyCharm.
  • Press Ctrl+C to stop the server.
  • Your project folder, virtual environment, and .env file all remain on disk.
  • To resume later, open the streaming-chat project in PyCharm and run fastapi dev main.py again.

Delete - I don't want to use this again

Remove all project files and revoke your API keys so nothing lingers on your machine or accounts.

  • If the FastAPI server is still running, press Ctrl+C in the terminal to stop it.
  • Delete the project folder by running this command from your Desktop:

macOS

cd ~/Desktop
rm -rf streaming-chat

Windows

cd ~\Desktop
Remove-Item -Recurse -Force streaming-chat
  • Revoke your OpenAI API key: go to platform.openai.com, open the API keys page, find your key, and click the delete (trash) icon.
  • Revoke your Anthropic API key: go to console.anthropic.com, open API Keys in settings, find your key, and click Delete.
  • Revoke your Hugging Face token: go to huggingface.co/settings/tokens, find your token, and click Delete.

That's a Wrap!

That's a Wrap!

Nice work! You just built a multi-provider streaming chat application that connects to three different LLM APIs and pipes their responses to a browser in real time.

You've learned how to:

  • Built a FastAPI backend with three streaming endpoints, each using a different provider's streaming model: OpenAI's Responses API, Anthropic's Messages API, and Hugging Face's OpenAI-compatible Inference Providers.
  • Created a browser chat UI that consumes Server-Sent Events using fetch and ReadableStream, rendering tokens character by character in real time.
  • Implemented the production pattern used by ChatGPT and Claude.ai: a backend that proxies LLM streams to the frontend via SSE, keeping API keys safe on the server.
  • Secret Mission: Built a Race Mode that fires the same prompt to all three providers simultaneously using asyncio and displays their responses streaming side-by-side for real-time comparison.

Ready to quiz yourself?