Build a Model Inference API

Serve a real NLP model through a production-style FastAPI endpoint.

Introduction

30 Second Summary

Every AI model you interact with online has a service sitting in front of it. That service accepts your input, runs it through the model, and sends back the result.

In this project, you will build a model inference API from scratch using FastAPI. Your API will serve a real sentiment analysis model through structured endpoints with health monitoring and request tracking.

What You'll Build

A locally running HTTP API that wraps a real NLP sentiment analysis model and exposes predictions through validated endpoints with health checks and metrics.

By the end of this project, you'll have:

  • A FastAPI application with /predict, /health, and /metrics endpoints serving real sentiment predictions.
  • Pydantic request/response models that automatically validate incoming data and return clear errors.
  • A metrics collector tracking request count, average latency, and error rate for production monitoring.
  • Secret Mission: Add request rate limiting to protect your inference endpoint from overload.

Are there any prerequisites?

None. This guide includes setup instructions for all required tools, including VS Code, Python, and FastAPI. You will install everything from scratch in Step 1.

Set Up Your Development Environment

Before you can build an API, you need the right tools on your machine.

In this step, you will install VS Code as your code editor, create an isolated Python environment for your project, and install the libraries that power your API.

In this step, get ready to:
  • Install VS Code and set up the terminal command.
  • Create a project folder with a Python virtual environment.
  • Install FastAPI and TextBlob inside your virtual environment.
Install VS Code

VS Code is a lightweight code editor that gives you syntax highlighting, a built-in terminal, and extensions for Python development.

  • Open https://code.visualstudio.com/Download in your browser.
  • Click the macOS download button (choose the Universal build).
  • Open the downloaded .dmg file.
  • Drag Visual Studio Code.app into the Applications folder.
  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.

Now you need to set up the code terminal command so you can open projects from the command line.

  • Open the Command Palette by pressing Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows).
  • Type shell command into the search box.
  • Select Shell Command: Install 'code' command in PATH.

What does this do?

This command adds the code binary to your system PATH. After this, you can type code . in any terminal folder to open it in VS Code.

  • Close and reopen your Terminal app so the new PATH takes effect.
  • Verify the setup by running this command:
code --version

✔️ I see a version number

VS Code is installed and the terminal command is working. You are ready to move on.

ⓧ Command not found

Your terminal cannot find the code command. Try these fixes:

  • Make sure you fully closed and reopened Terminal (not just a new tab).
  • Go back to VS Code, open the Command Palette (Cmd+Shift+P), and run Shell Command: Install 'code' command in PATH again.
  • If it still does not work, manually add VS Code to your PATH by adding this line to your ~/.zprofile file:
export PATH="$PATH:/Applications/Visual Studio Code.app/Contents/Resources/app/bin"

Then restart your terminal and try code --version again.

Finally, install the Python extension so VS Code can understand your Python code.

  • In VS Code, click the Extensions icon in the left sidebar (or press Cmd+Shift+X on macOS / Ctrl+Shift+X on Windows).
  • Search for Python.
  • Install the one published by Microsoft (it has a blue checkmark icon).
Create your project with a virtual environment

A virtual environment is an isolated copy of Python for your project. Packages you install inside it stay separate from your system Python, preventing version conflicts between projects.

  • Open Terminal by pressing Cmd+Space (macOS) or the Windows key (Windows), typing Terminal, and pressing Enter.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder, move into it, create the virtual environment, and activate it by running these commands:
mkdir model-api
cd model-api
python3 -m venv .venv
source .venv/bin/activate

What do these commands do?

  • mkdir model-api creates a new folder called model-api on your Desktop.
  • python3 -m venv .venv creates a virtual environment in a hidden .venv folder inside your project.
  • source .venv/bin/activate activates the environment so pip installs packages into this isolated copy of Python instead of your system.
  • Check your terminal prompt. You should see (.venv) at the beginning of the line, confirming the virtual environment is active.

Don't see (.venv) in your prompt?

  • Make sure you ran source .venv/bin/activate (not just python3 -m venv .venv). The first command creates the environment. The second activates it.
  • Confirm you are inside the model-api folder by running pwd. It should end with Desktop/model-api.

Why isn't my virtual environment activating?

  • Open your project folder in VS Code by running this command:
code .

VS Code opens with your model-api folder loaded in the Explorer sidebar.

  • VS Code may show a notification asking you to select a Python interpreter. Click Select Interpreter and choose the one with .venv in its path.
  • If no notification appears, open the Command Palette (Cmd+Shift+P / Ctrl+Shift+P), type Python: Select Interpreter, and choose the .venv interpreter.

Why select the interpreter?

This tells VS Code to use the Python inside your virtual environment for linting, autocomplete, and running code. Without this, VS Code might use your system Python which does not have your project's packages installed.

Install dependencies

Your project needs two libraries: FastAPI (the web framework for your API) and TextBlob (an NLP library for sentiment analysis).

  • Make sure your virtual environment is still active ((.venv) shows in your prompt). Install both libraries by running this command:
pip install "fastapi[standard]" textblob

What does fastapi[standard] include?

The [standard] extra installs everything you need to run a FastAPI server: the fastapi-cli tool (which gives you the fastapi dev command), uvicorn (the async server), and utilities like httpx for testing.

  • Download the NLP data that TextBlob needs for sentiment analysis by running:
python -m textblob.download_corpora

What are corpora?

TextBlob uses pre-trained language data (called corpora) to analyze text. This one-time download gives TextBlob the word lists and models it needs to calculate sentiment scores.

  • Verify that FastAPI installed correctly by running:
python -c "import fastapi; print(fastapi.__version__)"
  • You should see a version number printed (such as 0.138.0 or similar).

Seeing ModuleNotFoundError?

  • Check that (.venv) still appears in your terminal prompt. If it does not, reactivate by running source .venv/bin/activate and then try the pip install again.
  • Make sure you used pip install "fastapi[standard]" with quotes around the package name. Without quotes, your shell may misinterpret the square brackets.

Help me fix my FastAPI installation error.

Your development environment is fully set up. Next up, you will write your first API endpoint that serves real sentiment predictions.

Build the Prediction Endpoint

Your development environment is ready. Python, FastAPI, and TextBlob are installed inside an isolated virtual environment.

In this step, you'll build the core serving pattern: accept text, run it through a model, return structured predictions. This is the same pattern that powers every production ML API.

In this step, get ready to:
  • Define Pydantic request and response models for structured input and output.
  • Create a /predict endpoint that runs TextBlob sentiment analysis.
  • Start the development server and test the endpoint.
Create the application file and define request/response models

Every FastAPI application starts with an app instance and data models. Pydantic models define the exact shape of incoming requests and outgoing responses.

  • In VS Code's Explorer sidebar, right-click inside the file list and select New File.
  • Name the file main.py and press Enter.
  • Add the following imports and application setup to main.py:
import time
from datetime import datetime, timezone

from fastapi import FastAPI, HTTPException, Request
from pydantic import BaseModel, field_validator
from textblob import TextBlob

# Create the FastAPI application instance
app = FastAPI(
    title="Model Inference API",
    description="A production-style sentiment analysis serving endpoint",
    version="1.0.0",
)

What does this code do?

The import block pulls in everything the API needs: FastAPI for the web framework, Pydantic for data validation, and TextBlob for sentiment analysis.

The app = FastAPI(...) call creates your application instance with a title, description, and version. These metadata fields appear in the auto-generated documentation page.

  • Save main.py.
  • You won't see a visible change yet. The next chunk defines the data models your endpoint will use.
  • Below the app instance, add the Pydantic models that define your request and response shapes:
# --- Pydantic Models ---


class PredictionRequest(BaseModel):
    text: str

    # Reject empty or whitespace-only strings
    @field_validator("text")
    @classmethod
    def text_must_not_be_empty(cls, v: str) -> str:
        if not v.strip():
            raise ValueError("text must not be empty")
        return v


class PredictionResponse(BaseModel):
    text: str
    polarity: float
    subjectivity: float
    sentiment: str

What do these models do?

PredictionRequest defines what the client must send: a text field that cannot be empty. The @field_validator decorator adds custom validation. If someone sends an empty string, FastAPI returns a clear error automatically.

PredictionResponse defines what the API returns: the original text, a polarity score (-1 to 1), a subjectivity score (0 to 1), and a human-readable sentiment label.

  • Save main.py.
  • You won't see a visible change yet. The next chunk creates the endpoint that uses these models.
Create the /predict endpoint

The endpoint is where the model serving pattern lives: receive a request, run the model, return a structured prediction.

  • Below the Pydantic models, add the prediction endpoint:
# --- Endpoints ---


@app.post("/predict", response_model=PredictionResponse)
async def predict(request: PredictionRequest) -> PredictionResponse:
    start = time.time()
    try:
        # Run TextBlob sentiment analysis on the input text
        blob = TextBlob(request.text)
        polarity = blob.sentiment.polarity
        subjectivity = blob.sentiment.subjectivity

        # Map polarity to a human-readable label
        if polarity > 0.1:
            sentiment = "positive"
        elif polarity < -0.1:
            sentiment = "negative"
        else:
            sentiment = "neutral"

        return PredictionResponse(
            text=request.text,
            polarity=round(polarity, 4),
            subjectivity=round(subjectivity, 4),
            sentiment=sentiment,
        )
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

What does this endpoint do?

The @app.post("/predict") decorator registers a POST endpoint at the /predict path. FastAPI automatically validates incoming JSON against PredictionRequest before your function runs.

Inside the function, TextBlob analyzes the text and returns polarity and subjectivity scores. The if/elif/else block converts the raw polarity number into a label: positive, negative, or neutral.

The start = time.time() line records when the request begins. You will use this for latency tracking in a later step.

Why this pattern?

This is the model serving pattern: load the model once (TextBlob loads its model on import), then serve predictions on every request. Production platforms like vLLM, Triton, and TorchServe work identically at a larger scale.

  • Save main.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved main.py and you're ready to run the server.

ⓧ I'd like to double check the full code

import time
from datetime import datetime, timezone

from fastapi import FastAPI, HTTPException, Request
from pydantic import BaseModel, field_validator
from textblob import TextBlob

app = FastAPI(
    title="Model Inference API",
    description="A production-style sentiment analysis serving endpoint",
    version="1.0.0",
)

# --- Pydantic Models ---


class PredictionRequest(BaseModel):
    text: str

    @field_validator("text")
    @classmethod
    def text_must_not_be_empty(cls, v: str) -> str:
        if not v.strip():
            raise ValueError("text must not be empty")
        return v


class PredictionResponse(BaseModel):
    text: str
    polarity: float
    subjectivity: float
    sentiment: str


# --- Endpoints ---


@app.post("/predict", response_model=PredictionResponse)
async def predict(request: PredictionRequest) -> PredictionResponse:
    start = time.time()
    try:
        blob = TextBlob(request.text)
        polarity = blob.sentiment.polarity
        subjectivity = blob.sentiment.subjectivity

        if polarity > 0.1:
            sentiment = "positive"
        elif polarity < -0.1:
            sentiment = "negative"
        else:
            sentiment = "neutral"

        return PredictionResponse(
            text=request.text,
            polarity=round(polarity, 4),
            subjectivity=round(subjectivity, 4),
            sentiment=sentiment,
        )
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))
Run and test your prediction endpoint

With the code in place, you can start the development server and send your first prediction request.

  • In your terminal (with the virtual environment still active), start the FastAPI development server by running:
fastapi dev main.py

What does this command do?

The fastapi dev command starts a local development server with auto-reload enabled. Every time you save main.py, the server restarts automatically with your latest changes.

The server runs at http://127.0.0.1:8000 by default.

  • Open your browser and go to http://127.0.0.1:8000/docs.
  • You should see the Swagger UI page showing your /predict endpoint with its request and response schema.

What is Swagger UI?

FastAPI auto-generates interactive documentation for every endpoint you create. Swagger UI lets you test your API directly in the browser without writing any client code.

  • Click the POST /predict row to expand it.
  • Click Try it out.
  • Replace the example JSON in the request body with:
{
  "text": "FastAPI is amazing for building APIs"
}
  • Click Execute.
  • You should see a 200 response with a JSON body containing polarity, subjectivity, and sentiment fields.

You can also test from the command line. Open a new terminal tab (keep the server running in the first tab) and run:

curl -X POST http://127.0.0.1:8000/predict -H "Content-Type: application/json" -d '{"text": "FastAPI is amazing for building APIs"}'
  • You should see a JSON response like:
{
  "text": "FastAPI is amazing for building APIs",
  "polarity": 0.6,
  "subjectivity": 0.9,
  "sentiment": "positive"
}

Server not starting?

  • Check that your terminal prompt shows (.venv). If not, activate the virtual environment again with source .venv/bin/activate.
  • Make sure you are in the model-api folder. Run ls and confirm you see main.py in the output.
  • If you see a syntax error, double check your code against the full file tab above. Mismatched parentheses or indentation are the most common culprits.

Help me debug my FastAPI server startup error.

Your prediction endpoint is serving real NLP results. Next up, you'll add a health check endpoint so orchestration tools can monitor whether your API is alive and ready.

Add Health Check and Startup Tracking

Your prediction endpoint is live, but in a production environment, other services need a way to ask your API a simple question: are you healthy and ready to handle requests?

In this step, you will add a health check endpoint that reports whether your model is loaded and how long the service has been running. This is the exact pattern that Kubernetes orchestrators use to decide whether to route traffic to your instance.

In this step, get ready to:
  • Track when your application starts up.
  • Build a health check endpoint that confirms model readiness.
  • Test that uptime increases on each call.
Add application state tracking

For a health endpoint to report uptime, the application needs to know when it started. You will record the startup timestamp as a module-level variable that persists for the lifetime of the process.

  • In main.py, find the # --- Pydantic Models --- comment.
  • Add the following application state section directly above it:
# --- Application State ---

startup_time = datetime.now(timezone.utc)

What does this code do?

This line runs once when Python first imports the module. It captures the exact moment your server started.

Because it is a module-level variable (not inside a function), it keeps its value for as long as the process is alive. Every request to the health endpoint will compare the current time against this timestamp to calculate uptime.

  • Save main.py.

You will not see a change yet. The next chunk uses this timestamp in the health endpoint.

Create the /health endpoint

The health endpoint needs to return structured data, so you will define a Pydantic response model first, then write the endpoint that uses it.

  • In main.py, find the PredictionResponse class (it ends with sentiment: str).
  • Add the HealthResponse model directly below it, before the # --- Endpoints --- comment:
class HealthResponse(BaseModel):
    status: str
    uptime_seconds: float
    model_loaded: bool

What does this model define?

  • status reports whether the service is healthy.
  • uptime_seconds shows how long the process has been running.
  • model_loaded confirms that TextBlob can run predictions. If the model fails to load, this field will be False.
  • Save main.py.

You will not see a change yet. The next chunk creates the endpoint that returns this model.

  • Now scroll to the bottom of main.py, after the entire predict function.
  • Add the /health endpoint by pasting this code at the end of the file:
@app.get("/health", response_model=HealthResponse)
async def health() -> HealthResponse:
    # Calculate how long the server has been running
    now = datetime.now(timezone.utc)
    uptime = (now - startup_time).total_seconds()

    # Verify the model can still run predictions
    model_loaded = True
    try:
        TextBlob("test")
    except Exception:
        model_loaded = False

    return HealthResponse(
        status="healthy",
        uptime_seconds=round(uptime, 2),
        model_loaded=model_loaded,
    )

What does this endpoint do?

  • It subtracts startup_time from the current time to calculate uptime in seconds.
  • It runs a quick TextBlob call to confirm the NLP model can still process text. If something is wrong with the model files, model_loaded flips to False.
  • It always returns HTTP 200 with the health payload. An orchestrator like Kubernetes calls this endpoint repeatedly. If it ever stops responding, traffic gets routed away from your instance.

Why a health endpoint instead of just checking if the server responds?

A web server can be alive but not ready. For example, your process could be running but the model files could be corrupted.

The health endpoint proves the model works by actually calling it. This is the readiness probe pattern. Kubernetes uses it to decide whether your instance should receive traffic.

  • Save main.py.

Seeing a syntax error when you save?

  • Make sure the @app.get("/health") decorator is at the same indentation level as @app.post("/predict") above it.
  • Check that the HealthResponse class is placed before the # --- Endpoints --- comment, not after it.
  • Confirm you have a blank line between the end of the predict function and the @app.get decorator.

Help me debug a syntax error in my FastAPI health endpoint

Test the health endpoint
  • Start the development server (if it is not already running) by running this command in your terminal:
fastapi dev main.py
  • Open http://127.0.0.1:8000/health in your browser.

You should see a JSON response with three fields: status showing "healthy", uptime_seconds showing a number, and model_loaded showing true.

  • Refresh the page a few seconds later.

The uptime_seconds value should be larger than before. This confirms your startup timestamp is working correctly.

Seeing a 404 Not Found response?

  • Check that the URL is exactly /health (no trailing slash, no typos).
  • If you started the server before saving, the hot-reload should have picked up your changes. Check the terminal for error messages.
  • Make sure your endpoint decorator says @app.get("/health") and the function is not accidentally indented inside the predict function.

Help me figure out why my /health endpoint returns 404

✔️ Awesome, I've got everything!

Great. Double check you have saved main.py before moving on.

ⓧ I'd like to double check the full code

Here is the full main.py as it should look at the end of this step:

import time
from datetime import datetime, timezone

from fastapi import FastAPI, HTTPException, Request
from pydantic import BaseModel, field_validator
from textblob import TextBlob

app = FastAPI(
    title="Model Inference API",
    description="A production-style sentiment analysis serving endpoint",
    version="1.0.0",
)

# --- Application State ---

startup_time = datetime.now(timezone.utc)


# --- Pydantic Models ---


class PredictionRequest(BaseModel):
    text: str

    @field_validator("text")
    @classmethod
    def text_must_not_be_empty(cls, v: str) -> str:
        if not v.strip():
            raise ValueError("text must not be empty")
        return v


class PredictionResponse(BaseModel):
    text: str
    polarity: float
    subjectivity: float
    sentiment: str


class HealthResponse(BaseModel):
    status: str
    uptime_seconds: float
    model_loaded: bool


# --- Endpoints ---


@app.post("/predict", response_model=PredictionResponse)
async def predict(request: PredictionRequest) -> PredictionResponse:
    start = time.time()
    try:
        blob = TextBlob(request.text)
        polarity = blob.sentiment.polarity
        subjectivity = blob.sentiment.subjectivity

        if polarity > 0.1:
            sentiment = "positive"
        elif polarity < -0.1:
            sentiment = "negative"
        else:
            sentiment = "neutral"

        return PredictionResponse(
            text=request.text,
            polarity=round(polarity, 4),
            subjectivity=round(subjectivity, 4),
            sentiment=sentiment,
        )
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))


@app.get("/health", response_model=HealthResponse)
async def health() -> HealthResponse:
    now = datetime.now(timezone.utc)
    uptime = (now - startup_time).total_seconds()

    model_loaded = True
    try:
        TextBlob("test")
    except Exception:
        model_loaded = False

    return HealthResponse(
        status="healthy",
        uptime_seconds=round(uptime, 2),
        model_loaded=model_loaded,
    )

Your API can now prove it is alive and ready. Next up, you will add a metrics endpoint that tracks how many requests your service handles and how fast it responds.

Add Request Metrics

Your API now reports its health on demand. But knowing the service is up is only half the picture.

In this step, you'll build a metrics collector that tracks every prediction request. It records total requests, processing latency, and error rates. These are the same signals production AI teams monitor in Prometheus and Grafana dashboards to detect model degradation.

In this step, get ready to:
  • Build a MetricsCollector class that tracks request count, latency, and errors.
  • Wire metrics recording into the prediction endpoint.
  • Create a /metrics endpoint that exposes the collected data.
Build a metrics collector

Production inference services need to answer questions like "how many requests have we served?" and "how fast are responses?" A dedicated metrics class keeps this tracking logic separate from your endpoint code.

  • In main.py, find the # --- Pydantic Models --- comment.
  • Add the following MetricsCollector class and its instance directly above that comment by copying this code:
# --- Metrics Collector ---


class MetricsCollector:
    def __init__(self):
        self.total_requests: int = 0
        self.total_errors: int = 0
        self.total_latency_ms: float = 0.0

    def record_request(self, latency_ms: float) -> None:
        self.total_requests += 1
        self.total_latency_ms += latency_ms

    def record_error(self) -> None:
        self.total_errors += 1

    def get_metrics(self) -> dict:
        avg_latency = (
            self.total_latency_ms / self.total_requests
            if self.total_requests > 0
            else 0.0
        )
        error_rate = (
            self.total_errors / self.total_requests
            if self.total_requests > 0
            else 0.0
        )
        return {
            "total_requests": self.total_requests,
            "total_errors": self.total_errors,
            "error_rate": round(error_rate, 4),
            "average_latency_ms": round(avg_latency, 2),
        }


metrics = MetricsCollector()

What does this code do?

The MetricsCollector class tracks three values: total_requests (how many predictions have been served), total_errors (how many failed), and total_latency_ms (cumulative processing time).

The get_metrics() method computes derived stats like average latency and error rate on demand. The guard if self.total_requests > 0 prevents division-by-zero when no requests have been served yet.

The metrics = MetricsCollector() line creates a single module-level instance that all endpoints share. This mirrors how production services use singleton collectors.

  • Save main.py.
  • You won't see a change yet. The next edits wire this collector into the prediction endpoint.

Seeing a syntax error on save?

Make sure the # --- Metrics Collector --- comment and the class sit between startup_time = datetime.now(timezone.utc) and # --- Pydantic Models ---. Check that each method is indented with four spaces inside the class body.

help me fix a syntax error in my MetricsCollector class

Now you need to connect this collector to the /predict endpoint so it records timing data on every successful request and counts errors.

  • In main.py, scroll down to the predict function.
  • Inside the try block, find the return PredictionResponse( line.
  • Find these lines directly above the return:
        else:
            sentiment = "neutral"

        return PredictionResponse(
  • Add two lines between the sentiment assignment and the return, so the section now looks like this:
        else:
            sentiment = "neutral"

        latency_ms = (time.time() - start) * 1000
        metrics.record_request(latency_ms)

        return PredictionResponse(
  • Next, find the except block at the bottom of the same function. It currently looks like this:
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))
  • Add metrics.record_error() above the raise, so the except block now looks like this:
    except Exception as e:
        metrics.record_error()
        raise HTTPException(status_code=500, detail=str(e))

What do these changes do?

The start = time.time() at the top of the function was already there from the previous step. Now latency_ms = (time.time() - start) * 1000 calculates how many milliseconds elapsed during processing.

On success, metrics.record_request(latency_ms) logs the request and its latency. On failure, metrics.record_error() increments the error counter before re-raising the exception.

  • Save main.py.
Create the /metrics endpoint

The collector is recording data, but you need an endpoint to expose it. You'll add a response model and a GET endpoint that returns the current metrics.

  • In main.py, find the HealthResponse class in the Pydantic Models section.
  • Add the following MetricsResponse class directly below HealthResponse:
class MetricsResponse(BaseModel):
    total_requests: int
    total_errors: int
    error_rate: float
    average_latency_ms: float

What does this model do?

MetricsResponse defines the exact JSON shape your metrics endpoint returns. FastAPI uses this model to validate outgoing data and generate the schema in Swagger UI automatically.

  • Save main.py.
  • Now scroll to the very bottom of main.py, below the health endpoint.
  • Add the following /metrics endpoint:
@app.get("/metrics", response_model=MetricsResponse)
async def get_metrics() -> MetricsResponse:
    data = metrics.get_metrics()
    return MetricsResponse(**data)

What does this endpoint do?

The /metrics endpoint calls metrics.get_metrics() to compute the current stats, then unpacks the dictionary into a MetricsResponse object. This is the same pattern production services use to feed monitoring dashboards.

  • Save main.py.
  • If your development server is still running, it auto-reloads on save. You should see a reload message in the terminal.

Server didn't reload?

If you stopped the server earlier, start it again by running fastapi dev main.py in your terminal (make sure the virtual environment is active). Look for the (.venv) prefix in your prompt.

help me restart my FastAPI development server

✔️ Awesome, I've got everything!

Great. Double check you've saved main.py and that the server reloaded without errors.

ⓧ I'd like to double check the full code

Here is the complete main.py at this point. Compare yours against it:

import time
from datetime import datetime, timezone

from fastapi import FastAPI, HTTPException, Request
from pydantic import BaseModel, field_validator
from textblob import TextBlob

app = FastAPI(
    title="Model Inference API",
    description="A production-style sentiment analysis serving endpoint",
    version="1.0.0",
)

# --- Application State ---

startup_time = datetime.now(timezone.utc)


# --- Metrics Collector ---


class MetricsCollector:
    def __init__(self):
        self.total_requests: int = 0
        self.total_errors: int = 0
        self.total_latency_ms: float = 0.0

    def record_request(self, latency_ms: float) -> None:
        self.total_requests += 1
        self.total_latency_ms += latency_ms

    def record_error(self) -> None:
        self.total_errors += 1

    def get_metrics(self) -> dict:
        avg_latency = (
            self.total_latency_ms / self.total_requests
            if self.total_requests > 0
            else 0.0
        )
        error_rate = (
            self.total_errors / self.total_requests
            if self.total_requests > 0
            else 0.0
        )
        return {
            "total_requests": self.total_requests,
            "total_errors": self.total_errors,
            "error_rate": round(error_rate, 4),
            "average_latency_ms": round(avg_latency, 2),
        }


metrics = MetricsCollector()


# --- Pydantic Models ---


class PredictionRequest(BaseModel):
    text: str

    @field_validator("text")
    @classmethod
    def text_must_not_be_empty(cls, v: str) -> str:
        if not v.strip():
            raise ValueError("text must not be empty")
        return v


class PredictionResponse(BaseModel):
    text: str
    polarity: float
    subjectivity: float
    sentiment: str


class HealthResponse(BaseModel):
    status: str
    uptime_seconds: float
    model_loaded: bool


class MetricsResponse(BaseModel):
    total_requests: int
    total_errors: int
    error_rate: float
    average_latency_ms: float


# --- Endpoints ---


@app.post("/predict", response_model=PredictionResponse)
async def predict(request: PredictionRequest) -> PredictionResponse:
    start = time.time()
    try:
        blob = TextBlob(request.text)
        polarity = blob.sentiment.polarity
        subjectivity = blob.sentiment.subjectivity

        if polarity > 0.1:
            sentiment = "positive"
        elif polarity < -0.1:
            sentiment = "negative"
        else:
            sentiment = "neutral"

        latency_ms = (time.time() - start) * 1000
        metrics.record_request(latency_ms)

        return PredictionResponse(
            text=request.text,
            polarity=round(polarity, 4),
            subjectivity=round(subjectivity, 4),
            sentiment=sentiment,
        )
    except Exception as e:
        metrics.record_error()
        raise HTTPException(status_code=500, detail=str(e))


@app.get("/health", response_model=HealthResponse)
async def health() -> HealthResponse:
    now = datetime.now(timezone.utc)
    uptime = (now - startup_time).total_seconds()

    model_loaded = True
    try:
        TextBlob("test")
    except Exception:
        model_loaded = False

    return HealthResponse(
        status="healthy",
        uptime_seconds=round(uptime, 2),
        model_loaded=model_loaded,
    )


@app.get("/metrics", response_model=MetricsResponse)
async def get_metrics() -> MetricsResponse:
    data = metrics.get_metrics()
    return MetricsResponse(**data)
Test the full API flow

Time to confirm your metrics are tracking accurately. You'll send a few prediction requests and then inspect the counters.

  • If your server is not running, start it by running:
fastapi dev main.py
  • Open a new terminal tab (keep the server running in the first one).
  • Send a prediction request by running:
curl -X POST http://127.0.0.1:8000/predict -H "Content-Type: application/json" -d '{"text": "FastAPI is amazing for building APIs"}'
  • Send two more requests to build up the counter. Try different text each time by running:
curl -X POST http://127.0.0.1:8000/predict -H "Content-Type: application/json" -d '{"text": "This project is going well"}'
curl -X POST http://127.0.0.1:8000/predict -H "Content-Type: application/json" -d '{"text": "I am not sure about this"}'
  • Check the metrics by opening http://127.0.0.1:8000/metrics in your browser.
  • You should see total_requests showing 3, average_latency_ms showing a small number, and error_rate at 0.0.
  • Now test validation by sending a request with only whitespace by running:
curl -X POST http://127.0.0.1:8000/predict -H "Content-Type: application/json" -d '{"text": "   "}'

Why doesn't the error counter change?

You'll see a 422 response. This is Pydantic catching the invalid input before it reaches the model. Validation errors are blocked at the door and never enter the try/except block where metrics.record_error() lives.

The error counter tracks runtime failures during model inference (500 errors). This gives you two layers of protection: Pydantic rejects bad input, and the try/except catches unexpected model failures.

  • Finally, visit http://127.0.0.1:8000/health in your browser to confirm the service is still healthy.
  • You should see status as healthy, model_loaded as true, and uptime_seconds showing how long the server has been running.

Metrics showing 0 after sending requests?

Make sure you saved main.py after adding latency_ms and metrics.record_request(latency_ms) inside the try block. If the server reloaded, any previous request counts reset to zero because the module-level metrics instance is re-created on reload.

Send your curl requests after confirming the server has finished reloading (look for the Application startup complete message in the server terminal).

help me debug why my /metrics endpoint shows zeros

Secret mission

Add Rate Limiting

Rate limiting is how production LLM serving platforms (vLLM, TGI) protect GPU resources from being overwhelmed. In this mission, you'll implement a sliding-window rate limiter that rejects requests when a client exceeds their budget.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • The model-api project folder on your Desktop (includes the virtual environment and all installed packages).
  • TextBlob NLP corpora downloaded to ~/nltk_data.
  • VS Code application and the Python extension.
  • FastAPI development server (running process).

Keep everything running

No action needed. Choose this if you're still actively building or want to revisit the project soon.

  • Your development server will stop when you close the Terminal window or press Ctrl+C.
  • The model-api folder stays on your Desktop ready to use.
  • VS Code and the Python extension remain installed for future projects.

Pause - I'll come back to this later

Stop the running process to free up system resources. Your project files stay on disk for when you return.

  • Switch to the Terminal where your FastAPI server is running.
  • Press Ctrl+C to stop the development server.
  • When you come back, reactivate the virtual environment and restart the server by running:
cd ~/Desktop/model-api
source .venv/bin/activate
fastapi dev main.py

Everything else stays in place.

Delete - I don't want to use this again

Remove all project files, downloaded data, and optionally VS Code itself.

  • Stop the development server by pressing Ctrl+C in the Terminal.
  • Remove the project folder (includes the virtual environment and all installed packages) by running:
rm -rf ~/Desktop/model-api
  • Remove the TextBlob NLP corpora by running:
rm -rf ~/nltk_data
  • To uninstall VS Code, open Finder, go to Applications, right-click Visual Studio Code, and select Move to Trash.
  • To remove VS Code settings and extensions, run:
rm -rf ~/Library/Application\ Support/Code
rm -rf ~/.vscode

Nice Work!

Nice Work!

You just built a production-style model inference API from scratch using FastAPI and Python. This is the exact serving pattern that AI infrastructure teams use to deploy ML models behind HTTP endpoints.

You've learned how to:

  • Build a complete model inference API with FastAPI serving real NLP predictions through structured endpoints with automatic input validation via Pydantic.
  • Implement production health checks matching Kubernetes readiness probe patterns, and a request metrics system tracking latency, throughput, and errors.
  • Use Python virtual environments (venv) for isolated dependency management, keeping your project packages separate from your system Python.
  • Secret Mission: Implemented rate limiting using FastAPI's dependency injection system to protect inference endpoints from overload.

Ready to quiz yourself?