Deploy an ML Prediction API to Kubernetes

Containerize a scikit-learn API and deploy it to a local Kubernetes cluster.

Introduction

30 Second Summary

Every time you ask an app to predict something, whether that's a spam filter, a product recommendation, or a medical risk score, a machine learning model is running behind the scenes. That model needs to handle thousands of users at once without going down.

In this project, you will train a machine learning model, package it into a Docker container, and deploy it to a local Kubernetes cluster. You'll scale it to multiple replicas and watch Kubernetes balance traffic across them automatically.

What You'll Build

A machine learning prediction API that classifies iris flowers, containerized with Docker and deployed to a Kubernetes cluster running on your machine.

By the end of this project, you'll have:

  • A trained scikit-learn model wrapped in a FastAPI web service with prediction and health check endpoints.
  • A Docker container image built from a custom Dockerfile that packages your entire application.
  • Real Kubernetes manifests (Deployment + Service) deployed to a local cluster using kubectl, and scaled to multiple replicas with load balancing.
  • Secret Mission: Add a Horizontal Pod Autoscaler that automatically scales your replicas based on CPU usage.

Are there any prerequisites?

You need Docker Desktop installed on macOS. No Python or Kubernetes experience is required. This project will walk you through everything from scratch.

Before We Start

To deploy a machine learning API to Kubernetes, you need the right tools in place. Your local machine needs Python to train the model, and a Kubernetes cluster to run the final containerized service.

In this step, you will install Python, enable a local Kubernetes cluster through Docker Desktop, and verify everything is connected. Once these foundations are solid, building and deploying will be smooth.

In this step, get ready to:
  • Install Python 3.12 and create a virtual environment.
  • Enable Kubernetes in Docker Desktop.
  • Verify kubectl can reach your local cluster.
Install Python 3.12 via Homebrew and verify

This project uses Python 3.12 to train the ML model and run the API locally. The Dockerfile also uses python:3.12-slim as its base image, so matching versions avoids compatibility surprises.

  • Open your terminal by pressing Cmd+Space to open Spotlight, typing Terminal, and pressing Enter.
  • Check your current Python version by running this command:
python3 --version

✔️ I see Python 3.12

Python 3.12 is ready. Skip ahead to creating the project directory below.

ⓧ I see a version lower than 3.12

macOS often ships with an older Python that the python3 command points to. You need to install Python 3.12 alongside it.

  • Install Python 3.12 with Homebrew by running:
brew install python@3.12

After it finishes, the python3 command may still point to your older version. Use python3.12 explicitly for all commands in this project.

  • Confirm the install by running:
python3.12 --version

You should see a version number starting with 3.12.

ⓧ Command not found

Python is not installed yet. Install it with Homebrew by running:

  • Install Python 3.12 by running:
brew install python@3.12

This may take a minute or two. When it finishes, verify the install by running:

python3.12 --version

You should see a version number starting with 3.12.

brew command not found?

If brew itself is not recognized, you need to install Homebrew first. Visit brew.sh and follow the install instructions, then retry the command above.

help me install Homebrew on macOS

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir iris-k8s
cd iris-k8s
  • Confirm the folder was created by running:
ls

You should see an empty listing (no files yet), confirming you are inside the new iris-k8s folder.

  • Create a Python virtual environment by running:
python3.12 -m venv venv
  • Activate the virtual environment by running:
source venv/bin/activate
  • Confirm pip is available inside the virtual environment by running:
pip --version

You should see a path containing iris-k8s/venv in the output. This confirms pip is running inside your virtual environment.

What is a virtual environment?

A virtual environment is an isolated Python installation inside your project folder. Packages you install with pip stay contained here, so they do not conflict with other projects on your machine.

Seeing an error about venv?

Make sure you ran python3.12 -m venv venv (not python3). The python3 command may point to an older system Python that does not match the version this project requires.

help me create a Python 3.12 virtual environment

Enable Kubernetes in Docker Desktop

Docker Desktop includes a built-in single-node Kubernetes cluster. Enabling it gives you a real cluster to deploy to without needing a cloud account or remote server.

  • Open Docker Desktop by pressing Cmd+Space, typing Docker, and pressing Enter.
  • Click the gear icon in the top-right corner to open Settings.
  • Select the Kubernetes tab in the left sidebar.
  • Check the box labeled Enable Kubernetes.
  • Click Apply & Restart.

Docker Desktop will download the Kubernetes components and start the cluster. This can take a few minutes the first time.

  • Wait until you see a green light beside Kubernetes running in the bottom-left of the Docker Desktop window.

What is Kubernetes?

Kubernetes is a container orchestration platform. It takes your containerized application and manages running it across one or more machines. It handles restarts, scaling, and networking automatically.

Docker Desktop's built-in cluster gives you a single-node version of this. It runs the exact same kubectl commands and YAML manifests you would use on a production cloud cluster.

Kubernetes not starting?

If the green indicator does not appear after 5 minutes, try clicking Reset Kubernetes Cluster in the Kubernetes settings tab, then wait again. Make sure Docker Desktop itself is running (the whale icon in your menu bar should not be animating).

help me troubleshoot Kubernetes not starting in Docker Desktop

Verify kubectl and cluster access

Docker Desktop automatically installs kubectl (the Kubernetes command-line tool) on your Mac. You need to confirm it can talk to the local cluster.

  • Switch back to your terminal and confirm your cluster is reachable by running:
kubectl cluster-info

You should see output mentioning Kubernetes control plane running at a local address.

  • Check the cluster node by running:
kubectl get nodes

You should see one node named docker-desktop with a Ready status.

kubectl: command not found?

This usually means Docker Desktop's Kubernetes has not fully started yet. Confirm the green Kubernetes running indicator is showing in Docker Desktop. If it is showing but the command still fails, try closing and reopening your terminal window.

help me get kubectl working with Docker Desktop

Your environment is locked and loaded. Next up, you will train an ML model and build the FastAPI prediction service that will run inside your cluster.

Set Up Python and Enable Kubernetes

Your Python environment and Kubernetes cluster are ready. Now you need an application to deploy into that cluster.

In this step, you'll train a simple machine learning model and wrap it in a web API. The model classifies iris flowers based on their measurements. The real goal is having a working API to containerize and deploy later.

In this step, get ready to:
  • Train a machine learning model on the Iris dataset.
  • Build a FastAPI prediction API with health check endpoints.
  • Test the API locally before containerizing it.
Install Python dependencies and train the model

Before you can build the API, you need three Python packages. scikit-learn trains the model, FastAPI serves predictions over HTTP, and joblib saves and loads the trained model to disk.

  • Install all three packages into your virtual environment by running:
pip install scikit-learn==1.8.0 "fastapi[standard]==0.136.1" joblib==1.4.2

Why pin exact versions?

Pinning versions ensures the same packages install inside your Docker container later. Without pins, a newer version could introduce breaking changes between your local environment and the container.

  • Verify the install succeeded by running:
pip show scikit-learn

You should see version 1.8.0 in the output.

Seeing an install error?

Make sure your virtual environment is active. You should see the environment name in parentheses at the start of your terminal prompt. Help me fix my pip install error.

Now you'll train the model. The Iris dataset is built into scikit-learn, so no file downloads are needed.

  • Create the app/ directory where the model file will be saved:
mkdir app
  • Create a file called train.py in your project directory and add the following code:
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
import joblib

# Load the classic Iris dataset
iris = load_iris()
X, y = iris.data, iris.target

# Create a pipeline: scale features, then classify
pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=200))
])

# Train the model
pipeline.fit(X, y)

# Save to a file that the API will load
joblib.dump(pipeline, "app/model.joblib")
print("Model trained and saved to app/model.joblib")

What does this code do?

The Pipeline combines two steps: StandardScaler normalizes the flower measurements to a common scale, and LogisticRegression classifies each flower into one of three species.

joblib.dump saves the entire trained pipeline to a file. This means the API can load it at startup without retraining every time.

  • Save train.py, then run it:
python train.py

You should see: Model trained and saved to app/model.joblib.

  • Confirm the model file was created by running:
ls app/

You should see model.joblib in the output.

Seeing a ModuleNotFoundError?

This means scikit-learn isn't installed in the active environment. Run which python to confirm it points to your virtual environment's Python, not the system Python. Help me fix my ModuleNotFoundError when running train.py.

Write the FastAPI prediction service

The trained model is just a file on disk. To make predictions accessible over HTTP, you'll wrap it in a FastAPI web service with three endpoints.

  • Create an empty app/__init__.py file to make app/ a Python package:
touch app/__init__.py
  • Create app/main.py and add the following imports, app setup, and data models:
import os
import joblib
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI(title="Iris Prediction API")

# Load the trained model at startup
model = joblib.load("app/model.joblib")

# Class names for the Iris dataset
CLASS_NAMES = ["setosa", "versicolor", "virginica"]


class IrisFeatures(BaseModel):
    sepal_length: float
    sepal_width: float
    petal_length: float
    petal_width: float


class Prediction(BaseModel):
    predicted_class: str
    class_id: int
    probabilities: dict

What does this code do?

The model loads once at startup so every prediction request reuses the same trained pipeline without reloading from disk.

IrisFeatures defines the 4 flower measurements the API accepts as input. Prediction defines the response shape: the species name, numeric class ID, and confidence probabilities. Pydantic validates incoming requests automatically.

  • Below the Prediction class, add the root and health endpoints:
@app.get("/")
def root():
    """Returns the hostname, which equals the Pod name in Kubernetes."""
    return {"hostname": os.getenv("HOSTNAME", "unknown")}


@app.get("/health")
def health():
    """Health check endpoint for Kubernetes probes."""
    return {"status": "ok"}

Why do you need these endpoints?

The / endpoint returns the machine's hostname. When running inside Kubernetes, this becomes the Pod name, which helps you see which replica handled your request.

The /health endpoint returns a simple status that Kubernetes probes will check to verify the app is alive and ready to serve traffic.

  • Below the health endpoint, add the prediction endpoint:
@app.post("/predict", response_model=Prediction)
def predict(features: IrisFeatures):
    """Predict the iris species from flower measurements."""
    data = [[
        features.sepal_length,
        features.sepal_width,
        features.petal_length,
        features.petal_width,
    ]]
    prediction = model.predict(data)[0]
    probabilities = model.predict_proba(data)[0]

    return Prediction(
        predicted_class=CLASS_NAMES[prediction],
        class_id=int(prediction),
        probabilities={
            name: round(float(prob), 4)
            for name, prob in zip(CLASS_NAMES, probabilities)
        },
    )

What does the predict endpoint do?

The function extracts the 4 feature values from the request body and passes them to the model as a 2D array (the format scikit-learn expects).

model.predict() returns the class ID, and model.predict_proba() returns confidence scores for each of the three species. The response maps these numeric values back to human-readable species names.

  • Save app/main.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file.

ⓧ I'd like to double check the full code

import os
import joblib
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI(title="Iris Prediction API")

# Load the trained model at startup
model = joblib.load("app/model.joblib")

# Class names for the Iris dataset
CLASS_NAMES = ["setosa", "versicolor", "virginica"]


class IrisFeatures(BaseModel):
    sepal_length: float
    sepal_width: float
    petal_length: float
    petal_width: float


class Prediction(BaseModel):
    predicted_class: str
    class_id: int
    probabilities: dict


@app.get("/")
def root():
    """Returns the hostname, which equals the Pod name in Kubernetes."""
    return {"hostname": os.getenv("HOSTNAME", "unknown")}


@app.get("/health")
def health():
    """Health check endpoint for Kubernetes probes."""
    return {"status": "ok"}


@app.post("/predict", response_model=Prediction)
def predict(features: IrisFeatures):
    """Predict the iris species from flower measurements."""
    data = [[
        features.sepal_length,
        features.sepal_width,
        features.petal_length,
        features.petal_width,
    ]]
    prediction = model.predict(data)[0]
    probabilities = model.predict_proba(data)[0]

    return Prediction(
        predicted_class=CLASS_NAMES[prediction],
        class_id=int(prediction),
        probabilities={
            name: round(float(prob), 4)
            for name, prob in zip(CLASS_NAMES, probabilities)
        },
    )
Test the API locally

Before packaging anything into a container, you should confirm the API works on its own.

  • Start the FastAPI development server by running:
fastapi dev app/main.py

You should see output indicating the server is running on http://127.0.0.1:8000.

What is fastapi dev?

The fastapi dev command starts a development server with auto-reload enabled. It comes bundled with the fastapi[standard] package you installed earlier.

  • Open a second terminal window in the same project directory with your virtual environment active.
  • Send a test prediction request by running:
curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"sepal_length": 5.1, "sepal_width": 3.5, "petal_length": 1.4, "petal_width": 0.2}'

You should see a JSON response with predicted_class set to setosa and probabilities for all three species.

Getting a connection refused error?

Make sure the FastAPI server is still running in your first terminal. If it crashed, check for syntax errors in app/main.py and restart it. Help me fix my FastAPI connection error.

  • Return to the terminal running the server and press Ctrl+C to stop it.

Your ML model is trained and the prediction API is working locally. Next up, you'll package everything into a Docker container so it can run anywhere.

Train the ML Model and Build the API

Your prediction API works perfectly on your local machine. The next step is packaging it into a Docker container so that Kubernetes can run identical copies of it across multiple pods.

A container bundles your application code, the trained model, and all Python dependencies into a single portable image. In this step, you'll write a Dockerfile, build a container image, and verify the containerized API responds to prediction requests.

In this step, get ready to:
  • Write a Dockerfile to package the prediction API.
  • Create a requirements file and build the Docker image.
  • Test the container locally with a prediction request.
Write the Dockerfile

A Dockerfile is a recipe that tells Docker how to build your container image layer by layer. Each instruction adds something: a base operating system, dependencies, your source code, and a startup command.

  • Create a new file called Dockerfile in the root of your project directory (the same folder that contains train.py) and paste the following code:
FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app/ ./app/

EXPOSE 8000

CMD ["fastapi", "run", "app/main.py", "--host", "0.0.0.0", "--port", "8000"]

What does this Dockerfile do?

  • FROM python:3.12-slim starts from a lightweight official Python image. The slim variant keeps the image small by excluding development tools.
  • WORKDIR /app sets the working directory inside the container.
  • COPY requirements.txt . and RUN pip install install dependencies first. Docker caches this layer so rebuilds are fast when only your code changes.
  • COPY app/ ./app/ copies your FastAPI application and the trained model into the image.
  • CMD tells Docker to start the FastAPI production server on port 8000 when the container runs.
  • Save Dockerfile.

Why fastapi run instead of fastapi dev?

The fastapi run command starts FastAPI in production mode. It disables auto-reload and listens on 0.0.0.0 (all interfaces) so the container is accessible from outside. The fastapi dev command you used earlier is for local development only.

✔️ Awesome, I've got everything!

Great. Make sure Dockerfile is saved in the same folder as train.py and the app/ directory.

ⓧ I'd like to double check the full code

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app/ ./app/

EXPOSE 8000

CMD ["fastapi", "run", "app/main.py", "--host", "0.0.0.0", "--port", "8000"]
Create requirements.txt and build the image

The Dockerfile expects a requirements.txt file listing the exact package versions to install inside the container. Pinning versions ensures the container always builds the same way.

  • Create a new file called requirements.txt in the root of your project directory and paste the following:
scikit-learn==1.8.0
fastapi[standard]==0.136.1
joblib==1.4.2

What does this file do?

Each line pins a package to the exact version you used locally. This guarantees the container uses the same versions of scikit-learn, fastapi, and joblib so predictions inside the container match what you tested on your machine.

  • Save requirements.txt.

✔️ Awesome, I've got everything!

Great. Double check you saved the file in the project root alongside Dockerfile.

ⓧ I'd like to double check the full code

scikit-learn==1.8.0
fastapi[standard]==0.136.1
joblib==1.4.2

Now you have both files Docker needs. Time to build the image.

  • Build the Docker image by running this command in your terminal:
docker build -t iris-api:1.0.0 .

What does this command do?

  • docker build reads the Dockerfile and executes each instruction to create an image.
  • -t iris-api:1.0.0 tags the image with the name iris-api and version 1.0.0.
  • . tells Docker to use the current directory as the build context (where to find the Dockerfile and files to copy).

The build takes around 30-60 seconds as Docker downloads the Python base image and installs your dependencies.

  • Verify the image exists by running:
docker images

You should see iris-api with tag 1.0.0 in the list.

Build failed?

  • If you see an error about requirements.txt not found, make sure the file is in the same directory where you ran the build command.
  • If the build fails during pip install, check that your requirements.txt matches the versions above exactly.
  • Make sure Docker Desktop is running. You can confirm by checking for the Docker icon in your menu bar.

Why did my Docker build fail?

Test the container locally

Before deploying to Kubernetes, you should confirm the container works on its own. This catches any issues with file paths or missing dependencies inside the image.

  • Start the container by running:
docker run --rm -p 8000:8000 iris-api:1.0.0

What does this command do?

  • --rm automatically removes the container when it stops, keeping things clean.
  • -p 8000:8000 maps port 8000 on your machine to port 8000 inside the container so you can reach the API.
  • iris-api:1.0.0 is the image you just built.

You should see FastAPI starting up with a message indicating the server is running.

  • Open a new terminal tab and send a test prediction request by running:
curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"sepal_length": 5.1, "sepal_width": 3.5, "petal_length": 1.4, "petal_width": 0.2}'

You should see a JSON response with "predicted_class": "setosa" and probability values. This confirms the model is loaded and predictions work from inside the container.

  • Switch back to the terminal running the container and press Ctrl+C to stop it.

Connection refused on port 8000?

  • Make sure the container is still running in the other terminal tab. If it exited immediately, check the terminal output for error messages.
  • If port 8000 is already in use (perhaps from the earlier fastapi dev session), stop that process first with Ctrl+C in its terminal.

Why can't I connect to my Docker container?

Your API is now proven to work inside a container. Next up, you'll write Kubernetes manifests to deploy this container image to your local cluster with multiple replicas.

Containerize the Application

Your container image is built, tested, and ready to go. Now you need to tell Kubernetes how to run it.

Kubernetes doesn't understand Dockerfiles. It reads YAML manifests that declare what you want running and how to expose it. In this step, you'll write a Deployment manifest (to create multiple copies of your API) and a Service manifest (to route traffic to them).

In this step, get ready to:
  • Write a Kubernetes Deployment manifest with 2 replicas and health checks.
  • Write a Kubernetes Service manifest that exposes the API on a local port.
Create the Deployment manifest

A Deployment tells Kubernetes how many copies (replicas) of your container to run, what image to use, and how to check that each copy is healthy.

  • Create a k8s/ directory to hold your manifests by running this command:
mkdir k8s
  • Confirm the directory was created by running:
ls

You should see k8s listed alongside your other project files.

  • Create k8s/deployment.yaml in your text editor and paste the following manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: iris-api
  labels:
    app: iris-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: iris-api
  template:
    metadata:
      labels:
        app: iris-api
    spec:
      containers:
        - name: iris-api
          image: iris-api:1.0.0
          imagePullPolicy: Never
          ports:
            - containerPort: 8000
          resources:
            requests:
              memory: "128Mi"
              cpu: "100m"
            limits:
              memory: "256Mi"
              cpu: "500m"
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 10
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 15
            periodSeconds: 10

What does this manifest do?

This YAML tells Kubernetes exactly how to run your iris-api container:

  • replicas: 2 creates two identical copies (Pods) of your API for redundancy.
  • imagePullPolicy: Never tells Kubernetes to use the local Docker image instead of trying to pull from a remote registry.
  • resources sets minimum (requests) and maximum (limits) CPU and memory each Pod can use.
  • readinessProbe checks /health to decide when a Pod is ready to receive traffic.
  • livenessProbe checks /health to decide if a stuck Pod should be restarted.

Why imagePullPolicy: Never?

Docker Desktop shares its Docker daemon with the built-in Kubernetes cluster. Images you build with docker build are already available to Kubernetes without pushing to a registry.

Setting imagePullPolicy: Never prevents Kubernetes from looking for iris-api:1.0.0 on Docker Hub (where it doesn't exist).

  • Save k8s/deployment.yaml.

File not saving?

Make sure the k8s/ directory exists before creating the file. If you skipped the mkdir k8s command, your editor may show an error about a missing directory.

Help me debug my k8s/deployment.yaml file save issue

Create the Service manifest

A Service exposes your Pods to network traffic. Without it, your Pods are running but unreachable from outside the cluster.

  • Create k8s/service.yaml in your text editor and paste the following manifest:
apiVersion: v1
kind: Service
metadata:
  name: iris-api-service
spec:
  type: NodePort
  selector:
    app: iris-api
  ports:
    - port: 80
      targetPort: 8000
      nodePort: 30080

What does this Service do?

This creates a NodePort Service that routes external traffic into your cluster:

  • nodePort: 30080 is the port you'll use to access the API from your browser (http://localhost:30080).
  • targetPort: 8000 is the port your FastAPI container listens on inside the Pod.
  • selector: app: iris-api matches any Pod with the label app: iris-api (set in your Deployment).
  • Save k8s/service.yaml.
  • Confirm both files exist by running:
ls k8s/

You should see deployment.yaml and service.yaml listed.

Only seeing one file?

Double check that both files are saved inside the k8s/ directory, not in the project root. If a file ended up in the wrong place, move it with mv deployment.yaml k8s/ or recreate it in the correct location.

Help me find my missing YAML file

✔️ Awesome, I've got everything!

Great. Double check you've saved both files.

ⓧ I'd like to double check the full code

Here's k8s/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: iris-api
  labels:
    app: iris-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: iris-api
  template:
    metadata:
      labels:
        app: iris-api
    spec:
      containers:
        - name: iris-api
          image: iris-api:1.0.0
          imagePullPolicy: Never
          ports:
            - containerPort: 8000
          resources:
            requests:
              memory: "128Mi"
              cpu: "100m"
            limits:
              memory: "256Mi"
              cpu: "500m"
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 10
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 15
            periodSeconds: 10

And here's k8s/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: iris-api-service
spec:
  type: NodePort
  selector:
    app: iris-api
  ports:
    - port: 80
      targetPort: 8000
      nodePort: 30080

Your Kubernetes manifests are ready. Next up, you'll deploy them to your cluster and watch your Pods come to life.

Write Kubernetes Manifests

Your container image is built and your Kubernetes manifests are written. The Deployment manifest tells Kubernetes how many copies of your container to run. The Service manifest tells Kubernetes how to route external traffic to those containers.

In this step, you'll deploy both manifests to your local cluster with kubectl, verify your pods are running, and test that requests get load-balanced across replicas.

In this step, get ready to:
  • Apply the Kubernetes manifests and verify pods are running.
  • Test the live prediction API through the Kubernetes Service.
  • Inspect the cluster with kubectl commands.
Apply the manifests and verify

The kubectl apply command reads your YAML manifests and creates the corresponding resources in the cluster. Pointing it at a directory applies every manifest file inside.

  • Deploy your Deployment and Service to Kubernetes by running this command:
kubectl apply -f k8s/

What does this command do?

kubectl apply -f k8s/ reads every YAML file in the k8s/ directory and creates the resources they describe. In your case, that means one Deployment (which spawns 2 Pods) and one Service (which routes traffic to those Pods).

  • Verify the Deployment is registered by running:
kubectl get deployments
  • You should see iris-api listed with 2/2 under the READY column.
  • Check the individual Pods by running:
kubectl get pods

Wait until both pods show Running in the STATUS column and 1/1 under READY. This may take 10-15 seconds while the readiness probe passes.

What do READY and STATUS mean?

The READY column shows how many containers inside the Pod have passed their readiness probe. 1/1 means 1 out of 1 containers is healthy. STATUS Running means the Pod is alive and its containers are executing.

If you see 0/1 under READY with STATUS Running, the container started but the readiness probe on /health has not passed yet. Wait a few more seconds and run kubectl get pods again.

Pods stuck in ImagePullBackOff?

This means Kubernetes is trying to pull the image from a remote registry instead of using your local Docker image. Check that your k8s/deployment.yaml has imagePullPolicy: Never set on the container spec.

Also confirm the image name matches exactly. Run docker images and verify you see iris-api with tag 1.0.0.

help me fix my pods stuck in ImagePullBackOff

Test the live service

Your NodePort Service maps port 30080 on your machine to port 8000 inside the containers. This means you can hit the API directly at localhost:30080.

  • Open http://localhost:30080/ in your browser.
  • You should see a JSON response with a hostname field showing one of your Pod names.
  • Send a prediction request by running this curl command:
curl -X POST http://localhost:30080/predict \
  -H "Content-Type: application/json" \
  -d '{"sepal_length": 5.1, "sepal_width": 3.5, "petal_length": 1.4, "petal_width": 0.2}'

What should I see?

The response should be a JSON object with predicted_class set to setosa, a class_id of 0, and a probabilities dictionary showing confidence scores for each species. This proves your ML model is serving predictions from inside the Kubernetes cluster.

  • Refresh http://localhost:30080/ in your browser several times.
  • Watch the hostname value change between two different Pod names.

Each refresh hits a different Pod. This is Kubernetes load balancing in action, distributing requests across your 2 replicas.

Not seeing different hostnames?

Browsers sometimes cache responses. Try opening a private/incognito window, or test with curl instead:

curl http://localhost:30080/

Run it 5-6 times. You should see two different Pod names appear in the hostname field across those requests.

help me understand why I'm only seeing one hostname in my Kubernetes responses

Inspect and troubleshoot

Kubernetes gives you several commands to peek inside your cluster and understand what your Pods are doing.

  • View the request logs from one of your Pods by running:
kubectl logs [[POD_NAME="your pod name from kubectl get pods, e.g. iris-api-abc12-xyz34"]]

Where do I get my pod name?

Copy a pod name from the NAME column in your kubectl get pods output. It will look something like iris-api-6f7b8c9d5-abc12. Each pod gets a unique random suffix.

  • Inspect the detailed state of a Pod (events, probe status, resource allocations) by running:
kubectl describe pod [[POD_NAME="your pod name from kubectl get pods"]]
  • You should see the Events section at the bottom showing the Pod being scheduled, the container starting, and the probes passing.
  • View cluster-wide activity by running:
kubectl get events

When are these commands useful?

kubectl logs shows you the same output you would see running the app locally. Use it to debug application errors like failed predictions or import issues.

kubectl describe pod is for infrastructure issues. If a Pod is stuck in Pending or CrashLoopBackOff, the Events section tells you why. kubectl get events gives you a timeline of everything happening across the cluster.

Your ML prediction API is live on Kubernetes, load-balancing across multiple replicas. Next up, you'll scale the deployment up and down and watch Kubernetes handle the changes automatically.

Deploy to Kubernetes

Your API is deployed and handling predictions across two Pods. In the real world, traffic doesn't stay constant. A sudden spike in requests could overwhelm your replicas, and quiet periods mean wasted resources.

Kubernetes makes scaling as simple as a single command. In this step, you'll scale your deployment up to handle more load, scale it back down to conserve resources, and simulate a rolling update to see how Kubernetes replaces pods without downtime.

In this step, get ready to:
  • Scale the iris-api deployment to 4 replicas and verify load balancing across all pods.
  • Scale back down to 2 replicas and observe graceful pod termination.
  • Simulate a rolling update by deleting and re-applying the deployment.
Scale the deployment up

Right now your deployment runs 2 replicas. Imagine traffic just doubled. You need more pods to handle the load.

  • Scale your deployment from 2 to 4 replicas by running:
kubectl scale deployment iris-api --replicas=4

You should see deployment.apps/iris-api scaled confirming the change.

  • Watch the new pods come online by running:
kubectl get pods -w

What should I see?

You'll see your original 2 pods plus 2 new pods appearing. The new pods transition from Pending to ContainerCreating to Running with 1/1 ready.

Once all 4 pods show Running, press Ctrl+C to stop watching.

  • Send multiple requests to the root endpoint to see all 4 pods responding by running:
for i in $(seq 1 8); do curl -s http://localhost:30080/ ; echo; done

You should see 4 different hostnames in the output. Each hostname corresponds to a different pod. Kubernetes distributes requests across all available replicas automatically.

Only seeing 2 hostnames?

The new pods might still be starting up. Run kubectl get pods and confirm all 4 show Running with 1/1 ready before retrying the curl loop.

If pods are stuck in Pending, your Docker Desktop may not have enough resources allocated. Open Docker Desktop, go to Settings > Resources, and increase the memory allocation.

Help me troubleshoot why my pods aren't scaling up in Docker Desktop Kubernetes

Scale down and perform a rolling update

Traffic has calmed down. You no longer need 4 replicas burning resources. Kubernetes can gracefully shut down excess pods without dropping active requests.

  • Scale back down to 2 replicas by running:
kubectl scale deployment iris-api --replicas=2
  • Watch Kubernetes gracefully terminate the excess pods by running:
kubectl get pods -w

What should I see?

Two of your pods will show a Terminating status, then disappear. Kubernetes finishes handling any in-flight requests before removing them.

Once only 2 pods remain in Running status, press Ctrl+C to stop watching.

Now let's simulate a rolling update. In production, you'd push a new image version and Kubernetes would swap pods one at a time. Here, you'll delete and re-apply the deployment to see Kubernetes recreate pods from scratch.

  • Delete the existing deployment by running:
kubectl delete deployment iris-api

You should see deployment.apps "iris-api" deleted confirming the removal. All pods are now gone.

  • Re-apply the deployment manifest to bring everything back by running:
kubectl apply -f k8s/deployment.yaml
  • Confirm the new pods are running by checking:
kubectl get pods

You should see 2 fresh pods with new names, both in Running status with 1/1 ready. The pod names are different from before because Kubernetes created brand new pods.

Why delete and re-apply?

In a real deployment, you'd update the image tag in your manifest (e.g., from iris-api:1.0.0 to iris-api:2.0.0) and run kubectl apply. Kubernetes would then perform a true rolling update, replacing pods one at a time so the service stays available throughout.

Deleting first simulates the same outcome (fresh pods) without requiring a new image build.

Pods stuck in Pending after re-apply?

Run kubectl describe pod followed by one of the pending pod names to see what's blocking it. The Events section at the bottom usually reveals the cause.

Make sure imagePullPolicy: Never is still set in your k8s/deployment.yaml. Without it, Kubernetes tries to pull the image from Docker Hub and fails.

Help me troubleshoot my pods not starting after re-applying the deployment manifest

  • Send a final test request to confirm the service is still working:
curl -s http://localhost:30080/health

You should see {"status":"ok"} confirming your API is healthy and ready for requests.

Scale and Observe

Your API is deployed and responding to requests. Now it's time to see one of Kubernetes's most powerful features in action: scaling.

In production, traffic spikes happen. Kubernetes lets you add or remove replicas of your application with a single command. In this step, you'll scale your deployment up, watch the load balancer spread requests across pods, then scale back down and simulate a rolling update.

In this step, get ready to:
  • Scale the deployment up to 4 replicas and observe load balancing across pods.
  • Scale back down to 2 replicas and simulate a rolling update.
Scale the deployment up

Right now your deployment has 2 replicas. Scaling up means telling Kubernetes to create more identical pods to handle additional traffic.

  • Scale your deployment to 4 replicas by running this command:
kubectl scale deployment iris-api --replicas=4

You should see the output deployment.apps/iris-api scaled confirming the change.

  • Watch the new pods come online by running:
kubectl get pods -w

What does the -w flag do?

The -w flag stands for "watch." It keeps the command running and prints updates in real time as pod statuses change. You'll see pods transition from Pending to ContainerCreating to Running.

Press Ctrl+C to stop watching once all 4 pods show Running with 1/1 ready.

Now that 4 pods are running, Kubernetes will distribute incoming requests across all of them. The / endpoint returns the hostname of whichever pod handles the request, so you can see load balancing in action.

  • Send multiple requests to the root endpoint by running this loop:
for i in $(seq 1 8); do curl -s http://localhost:30080/; echo; done

You should see JSON responses with different hostname values. Each hostname is a pod name. With 4 replicas, you should see up to 4 distinct names in the output.

Seeing the same hostname every time?

If all 8 responses show the same pod name, wait a few seconds and try again. Kubernetes needs a moment to register all pods as ready before it routes traffic to them.

Run kubectl get pods and confirm all 4 pods show 1/1 in the READY column before retrying the curl loop.

Help me troubleshoot why my Kubernetes service isn't load balancing across pods.

Scale down and perform a rolling update

Scaling down is just as simple. Kubernetes gracefully terminates the extra pods, finishing any in-flight requests before shutting them down.

  • Scale back to 2 replicas by running:
kubectl scale deployment iris-api --replicas=2
  • Watch Kubernetes gracefully terminate the excess pods by running:
kubectl get pods -w

You should see 2 pods transition to Terminating status and disappear. Press Ctrl+C once only 2 pods remain in Running status.

What is graceful termination?

When Kubernetes terminates a pod, it sends a shutdown signal and gives the pod time to finish processing active requests. This prevents dropped connections. In production, this means you can scale down without impacting users.

In production, deploying a new version of your application triggers a rolling update. Kubernetes gradually replaces old pods with new ones so the service never goes down. You can simulate this by deleting and re-applying the deployment.

  • Delete the current deployment by running:
kubectl delete -f k8s/deployment.yaml
  • Re-apply the deployment to bring the pods back by running:
kubectl apply -f k8s/deployment.yaml
  • Verify the new pods are running by running:
kubectl get pods

You should see 2 fresh pods with new names and a recent AGE value. The old pod names are gone. Kubernetes created entirely new pods from your deployment manifest.

How is this different from a real rolling update?

A true rolling update happens when you change the container image tag in your deployment YAML (for example, updating from iris-api:1.0.0 to iris-api:2.0.0) and re-apply. Kubernetes would gradually replace old pods with new ones, keeping the service available the entire time.

The delete-and-reapply approach causes a brief outage because all pods disappear before new ones start. It's fine for learning, but in production you'd change the image tag instead.

Pods stuck in Pending or CrashLoopBackOff?

If pods don't reach Running status after re-applying, check that Docker Desktop is still running and that the iris-api:1.0.0 image still exists locally.

Run docker images | grep iris-api to confirm the image is listed. If it's missing, return to your project directory and rebuild it with docker build -t iris-api:1.0.0 . before re-applying.

Help me debug why my pods won't start after re-applying my Kubernetes deployment.

Secret mission

Add a Horizontal Pod Autoscaler

You've scaled your deployment manually. But in production, traffic spikes don't wait for a human to run kubectl scale. Can you make Kubernetes scale your pods automatically based on CPU usage?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally on Docker Desktop, so there are no ongoing costs.

Resources you used:

  • Kubernetes Deployment (iris-api) with 2+ running Pods.
  • Kubernetes Service (iris-api-service) on NodePort 30080.
  • Docker image (iris-api:1.0.0) stored locally.
  • Python virtual environment and project files on your Desktop.

Keep everything running

No action needed. Choose this if you're still actively building or experimenting with your Kubernetes deployment.

  • Your Kubernetes Pods use minimal CPU and memory when idle.
  • The Docker image takes up a small amount of disk space (a few hundred MB).
  • You can continue sending requests to http://localhost:30080 whenever Docker Desktop is running with Kubernetes enabled.

Pause - I'll come back to this later

Remove the Kubernetes resources to free up memory, but keep your project files and Docker image so you can re-deploy quickly.

  • Delete the Kubernetes Deployment and Service by running:
  • Your project files and Docker image remain intact. Re-deploy anytime with kubectl apply -f k8s/.
kubectl delete -f k8s/

Delete - I don't want to use this again

Remove all project resources completely and start fresh.

  • Delete all Kubernetes resources (Deployment, Service, and HPA if you completed the Secret Mission) by running:
kubectl delete -f k8s/
  • Remove the Docker image by running:
docker rmi iris-api:1.0.0
  • Delete your project directory from the Desktop by running:
cd ~/Desktop
rm -rf iris-k8s
  • Optionally, disable Kubernetes in Docker Desktop to reclaim memory. Open Docker Desktop, go to Settings then Kubernetes, and uncheck Enable Kubernetes. Click Apply & Restart.

Nice Work!

Nice Work!

You just deployed a machine learning prediction service to a local Kubernetes cluster. This is the exact workflow DevOps engineers use every day to ship applications to production.

You've learned how to:

  • Trained an ML model and wrapped it in a production-ready FastAPI service with /predict and /health endpoints.
  • Written a Dockerfile from scratch and built a container image ready for deployment.
  • Deployed to a real Kubernetes cluster using kubectl apply, scaled replicas up and down, and observed load balancing across Pods.
  • Added a Horizontal Pod Autoscaler for automatic CPU-based scaling that responds to real traffic.

Ready to quiz yourself?