Deploy an ML Inference Stack to EC2

Build, push, and run a 4-container ML stack on AWS EC2 using Docker and ECR.

Introduction

30 Second Summary

Machine learning models are only useful when people can actually use them. A model running on your laptop cannot serve predictions to users, handle traffic, or stay online around the clock.

In this project, you will package a text classifier and its frontend into Docker containers, push them to Amazon ECR, and deploy a four-container stack on a live EC2 instance. By the end, you will have a publicly accessible ML application with built-in monitoring.

What You'll Build

A complete ML inference stack consisting of a FastAPI API serving a DistilBERT text classifier, a Streamlit frontend, Prometheus for metrics collection, and Grafana for monitoring.

By the end of this project, you'll have:

  • Two custom Docker images built locally and pushed to Amazon ECR, the AWS-native container registry.
  • A t3.medium EC2 instance running Amazon Linux 2023 with Docker and Docker Compose installed from scratch.
  • A four-container stack (FastAPI API, Streamlit UI, Prometheus, Grafana) deployed and publicly accessible via the EC2 public IP.
  • Secret Mission: Configure a Grafana monitoring dashboard that visualizes live inference metrics from Prometheus.

Are there any prerequisites?

This project assumes you already have an AWS account with credits, the AWS CLI installed and configured, and Docker Desktop installed. Step 1 covers installing VS Code and verifying your existing tools.

Set Up Your Local Environment

Before you can deploy anything to the cloud, you need a solid local setup. This project involves building custom Docker images, pushing them to a registry, and running them on a remote server.

In this step, you will install VS Code, verify your existing tools, and create all the project files for the ML inference stack.

In this step, get ready to:
  • Install VS Code and the Docker extension.
  • Verify that AWS CLI and Docker Desktop are ready to go.
  • Create all project files for the ML inference stack.
Install VS Code and the Docker extension

VS Code is the code editor you will use to create and manage all the project files. It also has extensions that add helpful tools for working with Docker.

  • Check if you already have VS Code installed by searching for it:
  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and check if it appears.

✔️ I already have VS Code

You are all set. Press Enter to open VS Code and move on to installing the Docker extension below.

ⓧ I need to install VS Code

  • Open your browser and navigate to https://code.visualstudio.com/.
  • Click the download button for your operating system and run the installer.
  • Once the installation finishes, open VS Code by searching for it again with Cmd+Space (macOS) or the Windows key (Windows), typing Visual Studio Code, and pressing Enter.

VS Code not appearing after install?

Try closing and reopening your search bar, then searching again. On Windows, make sure you checked Add to PATH during installation. If it still does not appear, restart your computer and try again.

help me get VS Code installed and opening

Now install the Docker extension so VS Code can help you work with Dockerfiles and Compose files.

  • In VS Code, click the Extensions icon in the left sidebar (it looks like four small squares).
  • Type ms-azuretools.vscode-docker in the search bar.
  • Click Install on the Docker extension by Microsoft.

What does the Docker extension do?

The Docker extension adds syntax highlighting for Dockerfiles and Compose files. It also gives you a sidebar panel where you can view running containers, images, and registries directly from VS Code.

Verify AWS CLI and Docker Desktop

You already have the AWS CLI and Docker Desktop installed. Before moving on, confirm they are both configured and running correctly.

  • Open the integrated terminal in VS Code by pressing Ctrl+` (backtick key, next to the 1 key on your keyboard).
  • Verify your AWS CLI configuration by running this command:
aws sts get-caller-identity
  • You should see a JSON response containing your Account ID, Arn, and UserId. Note your Account number. You will need it in a later step.

Seeing an error about credentials?

Make sure your AWS CLI is configured with valid credentials. Run aws configure and enter your Access Key ID, Secret Access Key, region (us-east-1), and output format (json).

help me configure the AWS CLI so I can run aws sts get-caller-identity

  • Make sure Docker Desktop is running on your machine. Look for its icon in your system tray (macOS menu bar or Windows taskbar). If you do not see it, open it by searching for Docker Desktop using Cmd+Space (macOS) or the Windows key (Windows).
  • Verify Docker is running by running this command in your terminal:
docker info
  • You should see detailed output about your Docker installation, including the server version, number of containers, and storage driver.

Docker info showing an error?

Make sure Docker Desktop is fully started. It takes a moment to initialize after opening. Look for the Docker whale icon in your system tray and wait until it shows a steady state (not animating).

help me get Docker Desktop running so docker info works

Create the project directory and all files

The ML inference stack has multiple services, each with its own source code, dependencies, and Dockerfile. You will create the full directory structure and all files now so everything is ready for building.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create the project folder with its subdirectories and move into it by running these commands:
mkdir -p ml-deploy-ec2/fastapi-app ml-deploy-ec2/streamlit-app ml-deploy-ec2/prometheus
cd ml-deploy-ec2
  • Confirm the folders exist by running:
ls
  • You should see fastapi-app, prometheus, and streamlit-app in the output.
  • Open the ml-deploy-ec2 folder in VS Code by running:
code .

What does code . do?

The code command opens VS Code from the terminal. The . tells it to open the current directory as a project folder. You should see the fastapi-app, prometheus, and streamlit-app folders in the Explorer sidebar on the left.

code command not found?

On macOS, open VS Code manually, press Cmd+Shift+P, type Shell Command: Install 'code' command in PATH, and press Enter. Then try code . again in your terminal.

On Windows, you may need to restart your terminal after installing VS Code. If the command still does not work, open VS Code manually, click File, then Open Folder, and select the ml-deploy-ec2 folder on your Desktop.

Now create the FastAPI service files. The FastAPI app serves a DistilBERT text classifier as a REST API.

  • In the Explorer sidebar, right-click the fastapi-app folder and select New File.
  • Name the file requirements.txt and paste the following content:
fastapi==0.115.12
uvicorn==0.34.3
transformers==4.52.4
torch==2.7.1
prometheus-fastapi-instrumentator==8.0.2

What are these dependencies?

FastAPI is the web framework for building the API. Uvicorn is the server that runs it.

Transformers and torch are the Hugging Face and PyTorch libraries that power the DistilBERT model. prometheus-fastapi-instrumentator automatically exposes metrics from your API for monitoring.

  • Save the file with Cmd+S (macOS) or Ctrl+S (Windows).
  • Right-click the fastapi-app folder again and select New File.
  • Name the file app.py and paste the following content:
from fastapi import FastAPI
from pydantic import BaseModel
from transformers import pipeline
from prometheus_fastapi_instrumentator import Instrumentator

app = FastAPI(title="BERT Binary Classifier API")

# Load the DistilBERT sentiment classification model
classifier = pipeline(
    task="text-classification",
    model="distilbert-base-uncased-finetuned-sst-2-english",
)

# Automatically expose Prometheus metrics at /metrics
Instrumentator().instrument(app).expose(app)


# Define the shape of incoming requests
class TextRequest(BaseModel):
    text: str


# Define the shape of outgoing responses
class PredictionResponse(BaseModel):
    label: str
    score: float


# Accept text, classify it, and return the result
@app.post("/predict", response_model=PredictionResponse)
async def predict(request: TextRequest):
    result = classifier(request.text)[0]
    return PredictionResponse(label=result["label"], score=result["score"])


# Simple health check endpoint for Docker healthchecks
@app.get("/health")
async def health():
    return {"status": "healthy"}

What does this code do?

  • classifier = pipeline(...) loads the DistilBERT model that classifies text as POSITIVE or NEGATIVE. This is a pre-trained sentiment analysis model from Hugging Face.
  • Instrumentator().instrument(app).expose(app) hooks into every request to automatically collect metrics like request count and latency. Prometheus scrapes these metrics later.
  • The /predict endpoint receives text, runs it through the classifier, and returns the label and confidence score.
  • The /health endpoint returns a simple status response so Docker can check whether the container is healthy.
  • Save app.py.
  • Right-click fastapi-app once more and create a new file called Dockerfile (no file extension). Paste the following content:
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

RUN python -c "from transformers import pipeline; pipeline('text-classification', model='distilbert-base-uncased-finetuned-sst-2-english')"

EXPOSE 8000

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

What does this Dockerfile do?

  • FROM python:3.11-slim starts from a lightweight Python image to keep the final image size manageable.
  • The RUN python -c "from transformers import pipeline; ..." line downloads the DistilBERT model at build time (~260MB). This way the model is baked into the image and the container does not need to download it every time it starts.
  • The CMD starts the Uvicorn server on port 8000.
  • Save Dockerfile.

Now create the Streamlit frontend files. Streamlit provides a web UI where users can type text and see classification results.

  • In the Explorer sidebar, right-click the streamlit-app folder and select New File.
  • Name the file requirements.txt and paste the following content:
streamlit==1.45.1
requests==2.32.3
  • Save the file.
  • Right-click streamlit-app again and create a new file called app.py. Paste the following content:
import streamlit as st
import requests
import os

# Read the API URL from an environment variable, defaulting to the Docker service name
API_URL = os.getenv("API_URL", "http://fastapi-bert:8000")

st.title("BERT Binary Text Classifier")
st.write("Enter text below to classify it as POSITIVE or NEGATIVE.")

text_input = st.text_area("Enter text to classify:", height=100)

if st.button("Classify"):
    if text_input.strip():
        try:
            # Send the text to the FastAPI backend for classification
            response = requests.post(
                f"{API_URL}/predict",
                json={"text": text_input},
                timeout=30,
            )
            response.raise_for_status()
            result = response.json()
            label = result["label"]
            score = result["score"]
            if label == "POSITIVE":
                st.success(f"**{label}** (confidence: {score:.4f})")
            else:
                st.error(f"**{label}** (confidence: {score:.4f})")
        except requests.exceptions.RequestException as e:
            st.error(f"Error connecting to API: {e}")
    else:
        st.warning("Please enter some text.")

What does this code do?

  • API_URL reads the backend address from an environment variable. Inside Docker Compose, this resolves to the FastAPI container's service name.
  • When the user clicks Classify, the app sends their text to the /predict endpoint and displays the result with a green (POSITIVE) or red (NEGATIVE) banner.
  • Save app.py.
  • Right-click streamlit-app once more and create a new file called Dockerfile. Paste the following content:
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

EXPOSE 8501

CMD ["streamlit", "run", "app.py", "--server.port=8501", "--server.address=0.0.0.0", "--server.headless=true"]

What does this Dockerfile do?

It follows the same pattern as the FastAPI Dockerfile. The CMD starts Streamlit on port 8501 in headless mode, which means it runs without trying to open a browser inside the container.

  • Save Dockerfile.

Now create the configuration files. These tell Prometheus where to scrape metrics and tell Docker Compose how to wire everything together.

  • In the Explorer sidebar, right-click the prometheus folder and select New File.
  • Name the file prometheus.yml and paste the following content:
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "fastapi"
    static_configs:
      - targets: ["fastapi-bert:8000"]

What does this config do?

This tells Prometheus to scrape metrics from the FastAPI container every 15 seconds. The target fastapi-bert:8000 uses the Docker Compose service name so Prometheus can find the FastAPI container on the internal network.

  • Save prometheus.yml.
  • Now create the local Docker Compose file. Right-click the root ml-deploy-ec2 folder in the Explorer sidebar (above the three subfolders) and select New File.
  • Name the file docker-compose.yml and paste the following content:
services:
  fastapi-bert:
    build: ./fastapi-app
    ports:
      - "8000:8000"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 60s

  streamlit:
    build: ./streamlit-app
    ports:
      - "8501:8501"
    environment:
      - API_URL=http://fastapi-bert:8000
    depends_on:
      fastapi-bert:
        condition: service_started

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    depends_on:
      fastapi-bert:
        condition: service_started

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    depends_on:
      prometheus:
        condition: service_started

What does this Compose file do?

  • It defines four services: fastapi-bert (the API), streamlit (the frontend), prometheus (metrics collection), and grafana (metrics dashboards).
  • The build fields point to the local Dockerfiles for the two custom services. Prometheus and Grafana use official pre-built images.
  • depends_on ensures services start in the right order. The Streamlit UI waits for the FastAPI backend. Grafana waits for Prometheus.
  • The healthcheck on fastapi-bert calls the /health endpoint every 30 seconds so Docker knows whether the container is running correctly.
  • Save docker-compose.yml.
  • Finally, create the EC2 variant of the Compose file. Right-click the root ml-deploy-ec2 folder again and select New File.
  • Name the file docker-compose.ec2.yml and paste the following content:
services:
  fastapi-bert:
    image: YOUR_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/ml-deploy/fastapi-bert:latest
    ports:
      - "8000:8000"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 60s

  streamlit:
    image: YOUR_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/ml-deploy/streamlit-app:latest
    ports:
      - "8501:8501"
    environment:
      - API_URL=http://fastapi-bert:8000
    depends_on:
      fastapi-bert:
        condition: service_started

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    depends_on:
      fastapi-bert:
        condition: service_started

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    depends_on:
      prometheus:
        condition: service_started

How is this different from docker-compose.yml?

The local Compose file uses build to build images from your Dockerfiles. This EC2 version uses image to pull pre-built images from Amazon ECR instead. On EC2, you do not have the source code, so you pull the images you pushed to the registry.

The YOUR_ACCOUNT_ID placeholder will be replaced with your actual AWS account ID in a later step when you deploy to EC2.

  • Save docker-compose.ec2.yml.

Your Explorer sidebar should now show all nine files across the three subdirectories.

✔️ Awesome, I've got everything!

Great. Double check that you have saved every file.

ⓧ I'd like to double check the full code

Here are all nine project files. Compare each one against your VS Code files to make sure they match.

fastapi==0.115.12
uvicorn==0.34.3
transformers==4.52.4
torch==2.7.1
prometheus-fastapi-instrumentator==8.0.2
from fastapi import FastAPI
from pydantic import BaseModel
from transformers import pipeline
from prometheus_fastapi_instrumentator import Instrumentator

app = FastAPI(title="BERT Binary Classifier API")

classifier = pipeline(
    task="text-classification",
    model="distilbert-base-uncased-finetuned-sst-2-english",
)

Instrumentator().instrument(app).expose(app)


class TextRequest(BaseModel):
    text: str


class PredictionResponse(BaseModel):
    label: str
    score: float


@app.post("/predict", response_model=PredictionResponse)
async def predict(request: TextRequest):
    result = classifier(request.text)[0]
    return PredictionResponse(label=result["label"], score=result["score"])


@app.get("/health")
async def health():
    return {"status": "healthy"}
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

RUN python -c "from transformers import pipeline; pipeline('text-classification', model='distilbert-base-uncased-finetuned-sst-2-english')"

EXPOSE 8000

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
streamlit==1.45.1
requests==2.32.3
import streamlit as st
import requests
import os

API_URL = os.getenv("API_URL", "http://fastapi-bert:8000")

st.title("BERT Binary Text Classifier")
st.write("Enter text below to classify it as POSITIVE or NEGATIVE.")

text_input = st.text_area("Enter text to classify:", height=100)

if st.button("Classify"):
    if text_input.strip():
        try:
            response = requests.post(
                f"{API_URL}/predict",
                json={"text": text_input},
                timeout=30,
            )
            response.raise_for_status()
            result = response.json()
            label = result["label"]
            score = result["score"]
            if label == "POSITIVE":
                st.success(f"**{label}** (confidence: {score:.4f})")
            else:
                st.error(f"**{label}** (confidence: {score:.4f})")
        except requests.exceptions.RequestException as e:
            st.error(f"Error connecting to API: {e}")
    else:
        st.warning("Please enter some text.")
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

EXPOSE 8501

CMD ["streamlit", "run", "app.py", "--server.port=8501", "--server.address=0.0.0.0", "--server.headless=true"]
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "fastapi"
    static_configs:
      - targets: ["fastapi-bert:8000"]
services:
  fastapi-bert:
    build: ./fastapi-app
    ports:
      - "8000:8000"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 60s

  streamlit:
    build: ./streamlit-app
    ports:
      - "8501:8501"
    environment:
      - API_URL=http://fastapi-bert:8000
    depends_on:
      fastapi-bert:
        condition: service_started

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    depends_on:
      fastapi-bert:
        condition: service_started

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    depends_on:
      prometheus:
        condition: service_started
services:
  fastapi-bert:
    image: YOUR_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/ml-deploy/fastapi-bert:latest
    ports:
      - "8000:8000"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 60s

  streamlit:
    image: YOUR_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/ml-deploy/streamlit-app:latest
    ports:
      - "8501:8501"
    environment:
      - API_URL=http://fastapi-bert:8000
    depends_on:
      fastapi-bert:
        condition: service_started

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    depends_on:
      fastapi-bert:
        condition: service_started

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    depends_on:
      prometheus:
        condition: service_started

Files not matching or missing?

Make sure each file is inside the correct subfolder. The Dockerfile files have no extension. The docker-compose.yml and docker-compose.ec2.yml files sit at the root of ml-deploy-ec2, not inside any subfolder.

help me check that all my project files are in the right place

Your local environment is locked and loaded. Next up, you will build these Docker images and test the entire stack on your machine before pushing anything to the cloud.

Build Docker Images and Test Locally

Your project directory is ready with all the source files, Dockerfiles, and Compose configurations. The next step in the deployment pipeline is to build your code into portable Docker images and verify everything works locally before touching the cloud.

Building locally is your dress rehearsal. It catches dependency issues, configuration mistakes, and broken code before you push anything to a remote server.

In this step, get ready to:
  • Build Docker images for the FastAPI API and Streamlit frontend.
  • Launch the full 4-container stack locally with Docker Compose.
  • Verify the ML classifier returns predictions.
Build the FastAPI and Streamlit Docker images

Docker turns each application's code, dependencies, and runtime into a self-contained image. You have two custom applications to build: the FastAPI inference API and the Streamlit frontend.

  • Make sure your terminal is in the ml-deploy-ec2 folder, then build the FastAPI image by running this command:
docker build -t fastapi-bert ./fastapi-app

What happens during this build?

Docker pulls the python:3.11-slim base image and installs the Python packages from requirements.txt (including PyTorch, which is large).

A special build step then pre-downloads the DistilBERT model (~260MB) so it's baked directly into the image. This makes the final image larger (~2GB) but means containers start faster without needing to download the model at runtime.

This build takes several minutes because of the PyTorch installation and model download. When it finishes, you should see a line that starts with naming to followed by your image name.

Build failing?

  • Make sure Docker Desktop is running. You can verify by running docker info in your terminal.
  • Check that you're in the ml-deploy-ec2 folder. The build context ./fastapi-app must point to the correct subdirectory.
  • If you see a network error while downloading packages, check your internet connection and try again.

Help me debug my Docker build for the FastAPI image.

  • Build the Streamlit image by running this command:
docker build -t streamlit-app ./streamlit-app

This build is much faster since there's no ML model to download. It should complete within a minute or two.

  • Confirm both images were built by running:
docker images

You should see both fastapi-bert and streamlit-app listed in the output with the latest tag.

Don't see both images?

  • If one image is missing, scroll up in your terminal to find the build error. A typo in a filename or a missing file in the build context is the most common cause.
  • Make sure both fastapi-app/ and streamlit-app/ directories contain a Dockerfile, requirements.txt, and app.py.

Help me figure out why my Docker images didn't build.

Launch the full stack with Docker Compose

Your docker-compose.yml file defines all four services: FastAPI, Streamlit, Prometheus, and Grafana. Docker Compose starts them all in the right order with a single command.

  • Launch all four containers in detached mode by running:
docker compose up -d

What does this command do?

The -d flag runs containers in the background (detached mode). Without it, your terminal would be locked showing container logs.

Docker Compose reads docker-compose.yml, pulls or uses each image, and starts the containers in dependency order. The FastAPI service starts first, then Streamlit and Prometheus, then Grafana.

Wait about 30 seconds for the DistilBERT model to load into memory inside the FastAPI container. Then test each service in your browser.

  • Open http://localhost:8000/docs in your browser. You should see the FastAPI interactive documentation page with a /predict endpoint listed.
  • Open http://localhost:8501. You should see the Streamlit UI with a text input area and a Classify button.
  • Open http://localhost:9090. You should see the Prometheus web interface.
  • Open http://localhost:3000. You should see the Grafana login page.
  • Enter admin for both the username and password.
  • Grafana will prompt you to change the default password. Click Skip for now.

Not seeing one of the services?

  • The FastAPI service takes 30-60 seconds to load the model. If http://localhost:8000/docs shows a connection error, wait a moment and refresh.
  • Check if all 4 containers are running by running docker compose ps in your terminal. All four should show a status of "Up".
  • If a container exited, check its logs. For example, run docker compose logs fastapi-bert to see the FastAPI container's output.

Help me debug why one of my Docker Compose services isn't starting.

Verify the classifier and stop the stack

The services are running, but you should send a real prediction request to confirm the ML pipeline works end to end.

  • Send a test prediction request to the FastAPI endpoint by running this command:
curl -X POST http://localhost:8000/predict -H "Content-Type: application/json" -d '{"text": "This product is amazing"}'

What should I see?

You should get a JSON response with a label field (either POSITIVE or NEGATIVE) and a score field showing the model's confidence. For "This product is amazing", the label should be POSITIVE with a high confidence score.

Using Windows PowerShell?

PowerShell aliases curl to a different command. Use curl.exe instead, or run the command in Git Bash or WSL.

  • Confirm you see a JSON response with a label and score field in the terminal output.

Seeing a connection refused error?

  • The model takes 30-60 seconds to load after the container starts. Wait a moment and try the curl command again.
  • Run docker compose ps to make sure the fastapi-bert container shows "Up" status and not "Exited".
  • Check the container logs with docker compose logs fastapi-bert for any Python errors or missing dependency messages.

Help me troubleshoot my curl request to the FastAPI predict endpoint.

The local stack is verified. Now stop all the containers so Docker Desktop frees up those resources.

  • Stop and remove all containers by running:
docker compose down

All 4 containers stop and are removed. Your images remain saved locally for the next step.

Your ML stack is verified and working locally. Next up, you'll push these images to Amazon ECR so they can be pulled onto a cloud server.

Push Images to Amazon ECR

Your Docker images are built, tested, and ready to go. In the last step, you confirmed that all four containers work together on your local machine.

To deploy these containers on a remote server, you need a way to transfer them. Amazon ECR (Elastic Container Registry) is AWS's container registry. It stores your Docker images so that any EC2 instance can pull and run them.

In this step, get ready to:
  • Create two ECR repositories for your Docker images.
  • Authenticate Docker to your ECR registry and tag your images.
  • Push both images to ECR.
Create two ECR repositories

Each Docker image needs its own ECR repository. A repository is a named storage location that holds different versions of the same image.

  • Create the repository for your FastAPI image by running this command:
aws ecr create-repository --repository-name ml-deploy/fastapi-bert --region us-east-1
  • Create the repository for your Streamlit image by running this command:
aws ecr create-repository --repository-name ml-deploy/streamlit-app --region us-east-1

What do these commands do?

Each command creates a new repository in your AWS account's ECR service. The --repository-name flag sets the name. The ml-deploy/ prefix groups both repos under a common namespace so they're easy to find.

The --region us-east-1 flag creates the repository in the US East (N. Virginia) region. This is where you'll also launch your EC2 instance later.

  • Each command outputs a JSON response. Look for the repositoryUri field. This is the full address you'll use to tag and push your images.

Seeing an access denied error?

  • Make sure your AWS CLI credentials have permissions for ECR operations. The IAM user or role you configured in Step 1 needs the AmazonEC2ContainerRegistryFullAccess managed policy (or equivalent permissions).
  • Verify your credentials are working by running aws sts get-caller-identity again. If it fails, reconfigure with aws configure.
  • Still stuck? Help me troubleshoot my ECR access denied error.
Authenticate Docker to ECR and tag your images

ECR is a private registry. Before you can push images to it, Docker needs to log in with your AWS credentials.

Finding your AWS account ID

Your 12-digit account ID was displayed when you ran aws sts get-caller-identity in Step 1. You can run that command again to see it. You'll need it for every command in this substep.

  • Authenticate Docker to your ECR registry by running this command:
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com

What does this command do?

This command has two parts joined by a pipe (|).

  • aws ecr get-login-password fetches a temporary authentication token from AWS.
  • docker login --username AWS --password-stdin takes that token and logs Docker into your ECR registry. The token is valid for 12 hours.
  • You should see Login Succeeded in your terminal output.

Login didn't succeed?

  • Double-check that you replaced the account ID placeholder with your actual 12-digit AWS account ID. No spaces or extra characters.
  • Make sure Docker Desktop is still running. If it stopped, restart it and try again.
  • Need more help? Help me fix my ECR Docker login.

Docker is now logged in, but your images are still tagged with local names like fastapi-bert:latest. Docker can only push an image to a registry when the image tag includes the full registry address.

  • Tag the FastAPI image with your ECR repository URI by running this command:
docker tag fastapi-bert:latest [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/ml-deploy/fastapi-bert:latest
  • Tag the Streamlit image with your ECR repository URI by running this command:
docker tag streamlit-app:latest [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/ml-deploy/streamlit-app:latest

What does tagging do?

Tagging doesn't create a copy of the image. It adds a second name that points to the same image on disk. The new name includes the full ECR registry address, which tells Docker where to push the image.

  • Confirm both tags exist by running:
docker images | grep ml-deploy
  • You should see two rows with repository names containing ml-deploy/fastapi-bert and ml-deploy/streamlit-app.
Push both images to ECR

Your images are tagged and Docker is authenticated. Time to push them up to ECR so your EC2 instance can pull them later.

  • Push the FastAPI image by running this command:
docker push [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/ml-deploy/fastapi-bert:latest

This push will take a few minutes

The FastAPI image is about 2GB because it contains the DistilBERT model baked in. You'll see Docker uploading multiple layers. Each layer shows a progress bar.

  • Once the first push completes, push the Streamlit image by running this command:
docker push [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/ml-deploy/streamlit-app:latest
  • This push is much faster since the Streamlit image is smaller.
  • Confirm the images arrived in ECR by running this command:
aws ecr describe-images --repository-name ml-deploy/fastapi-bert --region us-east-1
  • You should see JSON output with an imageDetails array containing your image tagged as latest.
  • You can also verify in the AWS Console. Navigate to the Amazon ECR service, then click Repositories. You should see both ml-deploy/fastapi-bert and ml-deploy/streamlit-app listed.

Push failed or timed out?

  • If you see an authentication error, your Docker login token may have expired (tokens last 12 hours). Re-run the authentication command from the previous substep.
  • If the push seems stuck, check your internet connection. The FastAPI image is about 2GB and needs a stable connection to upload.
  • If you see a "name unknown" or "repository does not exist" error, make sure the repository name in your tag matches the repository you created. The names are case-sensitive.
  • Still stuck? Help me troubleshoot my ECR push.

Why use a container registry instead of copying files directly?

You could copy your source code to EC2 and build the images there, but that would mean installing all build dependencies on the server and waiting through a long build process on a less powerful machine.

The build-push-pull workflow through a registry is the standard approach in production. You build once locally (or in CI/CD), push the finished image, and any server can pull and run it instantly.

Both of your custom Docker images are now stored in ECR, ready to be pulled by any server with the right credentials. Next up, you'll launch an EC2 instance and deploy the full stack on it.

Launch and Configure EC2

Your Docker images are sitting safely in Amazon ECR, ready to be pulled onto a cloud server. The next piece of the puzzle is the server itself.

In this step, you will launch an EC2 instance, configure its firewall rules to expose the right ports, connect to it over SSH, and install Docker so the instance is ready to run containers.

In this step, get ready to:
  • Launch a t3.medium EC2 instance with a security group that exposes your application ports.
  • Connect to the instance via SSH from your local terminal.
  • Install Docker and Docker Compose on the remote instance.
Launch an EC2 instance

Your ML stack needs a server with enough memory to run the DistilBERT model alongside three other containers. A t3.medium instance provides 2 vCPUs and 4 GiB of RAM, which is the minimum for this workload.

Why t3.medium instead of a free-tier instance?

Free-tier instances like t2.micro and t3.micro only have 1 GiB of RAM. Loading the DistilBERT model plus PyTorch, a web server, and three other containers would immediately run out of memory.

The t3.medium costs approximately $0.0418 per hour. Your AWS credits will cover this cost during the project.

  • In the AWS Management Console, confirm your region is set to US East (N. Virginia) (us-east-1) in the top navigation bar.
  • Search for EC2 in the search bar and open the EC2 dashboard.
  • Click Launch instance.

The launch instance wizard opens. Fill in the settings below.

  • Under Name and tags, enter ml-inference-stack in the Name field.
  • Under Application and OS Images (Amazon Machine Image), click Quick Start.
  • Select Amazon Linux. The Amazon Linux 2023 AMI should be selected by default.
  • Under Instance type, search for and select t3.medium.

Next, create a key pair so you can connect to your instance via SSH.

  • Under Key pair (login), click Create new key pair.
  • Enter ml-deploy-key as the key pair name.
  • Set the key pair type to RSA.
  • Set the private key file format to .pem.
  • Click Create key pair. Your browser downloads a file called ml-deploy-key.pem. Note where it saves (usually your Downloads folder).

Keep your .pem file safe

The ml-deploy-key.pem file is your private key. AWS does not store a copy, so if you lose it, you cannot connect to your instance. Do not delete or move this file until the project is complete.

Now configure the firewall to control which traffic can reach your instance.

  • Under Network settings, click Edit.
  • In the Security group name field, enter ml-stack-sg.
  • The first inbound rule is pre-filled for SSH (port 22). Set its Source type to My IP.

You need four more inbound rules so that your application services are reachable.

  • Click Add security group rule.
  • Set Type to Custom TCP.
  • Enter 8000 in the Port range field.
  • Set Source type to Anywhere. This opens port 8000 for the FastAPI service.
  • Click Add security group rule again.
  • Set Type to Custom TCP.
  • Enter 8501 in the Port range field.
  • Set Source type to Anywhere. This opens port 8501 for the Streamlit UI.
  • Click Add security group rule again.
  • Set Type to Custom TCP.
  • Enter 9090 in the Port range field.
  • Set Source type to My IP. This restricts Prometheus to your IP only.
  • Click Add security group rule one more time.
  • Set Type to Custom TCP.
  • Enter 3000 in the Port range field.
  • Set Source type to My IP. This restricts Grafana to your IP only.

Why are some ports open to everyone and others restricted?

FastAPI (port 8000) and Streamlit (port 8501) are your application endpoints. Opening them to 0.0.0.0/0 means anyone can demo the app.

Prometheus (port 9090) and Grafana (port 3000) expose operational metrics. Restricting them to your IP keeps your monitoring data private.

  • Under Configure storage, keep the default 8 GiB gp3 root volume.
  • Review your settings in the Summary panel on the right side.
  • Click Launch instance.

A success banner confirms the launch. The instance state starts as Pending and changes to Running within a minute or two.

  • Click the instance ID link in the success banner to open the Instances page.
  • Wait for the Instance state column to show Running.
  • Select the checkbox next to your instance.
  • In the Details tab below, find and copy the Public IPv4 address. You will need this to SSH in and to access your services later.

Instance stuck in Pending?

Wait up to two minutes. If the state changes to Terminated instead of Running, check that you selected t3.medium and that your account has sufficient credits.

Help me troubleshoot my EC2 instance that went to Terminated state instead of Running

Connect to your instance via SSH

Your instance is running, but you need a way to get inside it and run commands. SSH lets you open a secure terminal session on the remote server from your local machine.

  • Open a terminal on your local computer. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.
  • Navigate to the folder where your ml-deploy-key.pem file was downloaded by running:
cd ~/Downloads

Before connecting, you need to set the correct permissions on the key file. SSH refuses to use a key that other users can read.

macOS

  • Set the key file permissions by running:
chmod 400 ml-deploy-key.pem

Windows

  • Set the key file permissions by running:
icacls ml-deploy-key.pem /inheritance:r /grant:r "%USERNAME%:R"
  • Connect to your EC2 instance by running:
ssh -i ml-deploy-key.pem ec2-user@[[EC2_PUBLIC_IP="your EC2 public IPv4 address"]]

What does this command do?

The -i ml-deploy-key.pem flag tells SSH which private key to use for authentication. ec2-user is the default username on Amazon Linux 2023 instances.

SSH will ask if you want to continue connecting to an unknown host. Type yes and press Enter.

You should see the Amazon Linux 2023 welcome banner and a command prompt ending in $. You are now running commands on your remote EC2 instance.

Seeing Permission denied or Unprotected private key file?

Make sure you ran the chmod 400 (macOS) or icacls (Windows) command on ml-deploy-key.pem before connecting. Also verify the .pem file is in the same folder your terminal is currently in.

If you see a connection timeout, double-check that your security group has an SSH rule on port 22 with source set to My IP. Your IP may have changed since you created the rule.

Help me fix my SSH connection to EC2

Install Docker and Docker Compose on EC2

Your EC2 instance is a fresh Amazon Linux 2023 server with no container runtime installed. You need to install Docker and Docker Compose before you can deploy your stack.

  • Update the system packages and install Docker by running these commands on the EC2 instance:
# Update all system packages
sudo dnf update -y

# Install Docker from the AL2023 default repository
sudo dnf install -y docker

# Start the Docker daemon
sudo systemctl start docker

# Enable Docker to start on boot
sudo systemctl enable docker

What do these commands do?

The dnf command is the package manager on Amazon Linux 2023 (similar to apt on Ubuntu). systemctl start launches the Docker service immediately.

The systemctl enable command ensures Docker starts automatically if the instance reboots.

  • Add your user to the docker group so you can run Docker commands without sudo:
# Add ec2-user to the docker group
sudo usermod -aG docker $USER

# Apply the group change immediately
newgrp docker

What does newgrp do?

Normally, adding a user to a group requires you to log out and back in for the change to take effect. The newgrp docker command applies the group change to your current session immediately, so you can start using Docker right away.

Amazon Linux 2023 does not include Docker Compose by default. You need to install it as a CLI plugin.

  • Install the Docker Compose plugin by running these commands:
# Create the CLI plugins directory
sudo mkdir -p /usr/libexec/docker/cli-plugins

# Download the Docker Compose binary
sudo curl -SL https://github.com/docker/compose/releases/latest/download/docker-compose-linux-$(uname -m) -o /usr/libexec/docker/cli-plugins/docker-compose

# Make it executable
sudo chmod +x /usr/libexec/docker/cli-plugins/docker-compose

What does this install?

This downloads the latest Docker Compose binary from GitHub and places it in the Docker CLI plugins directory at /usr/libexec/docker/cli-plugins/. Once installed, you use the docker compose command (with a space, not a hyphen).

  • Verify Docker is working by running:
docker info
  • Verify Docker Compose is working by running:
docker compose version

The docker info command should show details about the Docker installation, including the server version. The docker compose version command should return a version number.

docker: permission denied?

If docker info returns a permission error, the group change may not have taken effect. Try running newgrp docker again. If the issue persists, disconnect from SSH and reconnect. The group change always takes effect on a fresh login.

Help me fix Docker permission denied on my EC2 instance

Your EC2 instance is fully set up with Docker and Docker Compose. Next up, you will pull your images from ECR and deploy the full ML stack on this server.

Deploy the Stack on EC2

Your EC2 instance has Docker and Docker Compose ready to go. Now it's time to pull your container images from ECR and launch the full ML inference stack on your cloud server.

There's one gap to close first. Your EC2 instance doesn't have AWS credentials yet, so it can't pull images from your private ECR repositories.

In this step, get ready to:
  • Authenticate Docker on EC2 to your ECR registry.
  • Create the Prometheus and Docker Compose configuration files on EC2.
  • Pull and launch all 4 containers, then test each service via the public IP.
Authenticate Docker on EC2 to your ECR registry

Before Docker on your EC2 instance can pull images from your private ECR repositories, it needs valid AWS credentials. You'll configure these with the AWS CLI and then log Docker into your registry.

  • In your SSH session on the EC2 instance, configure the AWS CLI by running this command. It will prompt you for 4 values one at a time.
aws configure
  • When prompted for AWS Access Key ID, paste your access key ID.
  • When prompted for AWS Secret Access Key, paste your secret access key.
  • When prompted for Default region name, enter us-east-1.
  • When prompted for Default output format, enter json.

Where do I find my access keys?

Use the same credentials you used to configure the AWS CLI on your local machine. If you need new ones, go to the AWS Console and navigate to IAM, then Users, select your user, and open the Security credentials tab.

In production, you'd attach an IAM instance profile to the EC2 instance instead. This avoids storing credentials on disk entirely. For this project, aws configure is simpler since it skips the extra IAM role creation steps.

The command finishes silently after you enter the fourth value. No confirmation message is shown.

  • Authenticate Docker to your ECR registry by running:
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com
  • You should see Login Succeeded in the terminal output.

Login failed?

Make sure you replaced the account ID placeholder with your actual 12-digit AWS account ID. You can find it in the AWS Console by clicking your username in the top-right corner.

If you see an error about invalid credentials, run aws configure again and double-check your access key and secret key values.

Why is my Docker ECR login failing on my EC2 instance?

Create the project files on EC2

Your local Docker Compose file uses build: directives to build images from source code. On EC2, you don't have the source code. Instead, the EC2 Compose file references your ECR image URIs directly so Docker pulls pre-built images from the registry.

  • Create the project directory and move into it by running:
mkdir -p ml-deploy-ec2/prometheus && cd ml-deploy-ec2
  • Create the Prometheus configuration file by copying and pasting this entire block into your terminal:
cat > prometheus/prometheus.yml << 'EOF'
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "fastapi"
    static_configs:
      - targets: ["fastapi-bert:8000"]
EOF

What is this << 'EOF' syntax?

This is called a heredoc. It lets you create a file directly from the terminal without opening a text editor. Everything between << 'EOF' and the closing EOF gets written to the file.

The Prometheus config tells Prometheus to scrape metrics from the fastapi-bert container on port 8000 every 15 seconds.

  • Confirm the file was created by running:
ls prometheus/

You should see prometheus.yml listed.

  • Create the EC2 Docker Compose file by copying and pasting this entire block into your terminal:
cat > docker-compose.ec2.yml << 'EOF'
services:
  fastapi-bert:
    image: [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/ml-deploy/fastapi-bert:latest
    ports:
      - "8000:8000"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 60s

  streamlit:
    image: [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/ml-deploy/streamlit-app:latest
    ports:
      - "8501:8501"
    environment:
      - API_URL=http://fastapi-bert:8000
    depends_on:
      fastapi-bert:
        condition: service_started

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    depends_on:
      fastapi-bert:
        condition: service_started

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    depends_on:
      prometheus:
        condition: service_started
EOF

How does this differ from the local Compose file?

The local docker-compose.yml uses build: ./fastapi-app to build images from source. The EC2 version uses image: to pull pre-built images from ECR instead.

This is the standard production pattern. You build and test images locally, push them to a registry, and pull them on the deployment server. The server never needs your source code.

  • Confirm both files are in place by running:
ls

You should see docker-compose.ec2.yml and a prometheus folder in the listing.

Files not showing up?

Make sure you're inside the ml-deploy-ec2 directory. Run pwd to check your current location. If you're not there, run cd ~/ml-deploy-ec2 and try ls again.

If a file has the wrong content, you can overwrite it by running the same heredoc command again.

My project files aren't appearing on my EC2 instance after running the heredoc commands.

✔️ Awesome, I've got everything!

Great. Make sure you're still inside the ml-deploy-ec2 directory before moving on to the next substep.

ⓧ I'd like to double check the full code

Here's what each file should contain. You can verify by running cat prometheus/prometheus.yml and cat docker-compose.ec2.yml on your EC2 instance.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "fastapi"
    static_configs:
      - targets: ["fastapi-bert:8000"]
services:
  fastapi-bert:
    image: [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/ml-deploy/fastapi-bert:latest
    ports:
      - "8000:8000"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 60s

  streamlit:
    image: [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/ml-deploy/streamlit-app:latest
    ports:
      - "8501:8501"
    environment:
      - API_URL=http://fastapi-bert:8000
    depends_on:
      fastapi-bert:
        condition: service_started

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    depends_on:
      fastapi-bert:
        condition: service_started

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    depends_on:
      prometheus:
        condition: service_started
Pull and launch the stack

Everything is in place. Docker is authenticated to ECR, and the Compose file knows which images to pull. Time to bring the stack online.

  • Launch all 4 containers by running:
docker compose -f docker-compose.ec2.yml up -d

What's happening during startup?

Docker pulls each image from ECR and Docker Hub. The FastAPI image is about 2GB (it contains the DistilBERT model), so the pull takes several minutes depending on your network speed.

After the images are pulled, the FastAPI container needs about 60 seconds to load the DistilBERT model into memory. The -d flag runs the containers in detached mode, so your terminal stays free while they start up.

Wait about 60 seconds after the command finishes for all containers to fully start.

  • Verify all 4 containers are running by running:
docker compose -f docker-compose.ec2.yml ps

You should see 4 containers listed, each with a status of Up: fastapi-bert on port 8000, streamlit on port 8501, prometheus on port 9090, and grafana on port 3000.

Containers not showing as Up?

If a container shows a status like Restarting or Exited, check its logs by running docker compose -f docker-compose.ec2.yml logs <service-name> (replace <service-name> with the failing container's name).

If the FastAPI container keeps restarting, it may be running out of memory loading the model. Confirm your instance is t3.medium (4 GiB RAM) in the EC2 console.

My containers aren't starting properly on EC2 after running docker compose up.

  • Open your browser and test each service using your EC2 instance's public IPv4 address. Open the following URL to test the FastAPI API documentation:
http://[[EC2_PUBLIC_IP="your EC2 public IPv4 address"]]:8000/docs

You should see the FastAPI interactive documentation page with the /predict and /health endpoints listed.

  • Open the Streamlit UI to test the text classifier:
http://[[EC2_PUBLIC_IP="your EC2 public IPv4 address"]]:8501
  • Type a sentence like This product is amazing into the text area.
  • Click Classify.

You should see a result showing POSITIVE with a high confidence score. Try a negative sentence like This is terrible and you should see NEGATIVE.

  • Open the Prometheus query interface:
http://[[EC2_PUBLIC_IP="your EC2 public IPv4 address"]]:9090

You should see the Prometheus query page. Click Status then Targets to confirm the fastapi target shows a state of UP.

  • Open the Grafana login page:
http://[[EC2_PUBLIC_IP="your EC2 public IPv4 address"]]:3000
  • Log in with username admin and password admin.
  • Grafana will prompt you to change the password. You can click Skip for now.

You should see the Grafana home page. All 4 services are now running on your EC2 instance and accessible from your browser.

Can't reach a service in the browser?

Make sure you're using http:// (not https://). These services are not configured for HTTPS.

Double-check that the security group ml-stack-sg has inbound rules for the port you're trying to reach. Ports 9090 and 3000 are restricted to your IP only, so you can't access them from a different network.

If the FastAPI docs page doesn't load, the model may still be loading. Wait another 30 seconds and try again.

I can't reach my services on EC2 through the browser even though the containers are running.

Secret mission

Build a Grafana Monitoring Dashboard

Your ML stack is live, but you have no visibility into how it performs under load. In this secret mission, you will wire Grafana to Prometheus and build a real-time monitoring dashboard that tracks request rate and P95 latency for your FastAPI inference endpoint.

Clean Up Your Resources

Clean Up Your Resources

Your t3.medium EC2 instance costs approximately $0.0418 per hour, which is not covered by the AWS free tier. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Cost warning

The t3.medium instance costs roughly $1 per day. Your AWS credits will cover this, but do not leave the instance running indefinitely.

If you are done with this project, choose Pause or Delete below to stop charges.

Resources you used:

  • 1 x t3.medium EC2 instance (ml-inference-stack).
  • 2 x ECR repositories (ml-deploy/fastapi-bert and ml-deploy/streamlit-app).
  • 1 x EBS volume (8 GiB gp3, attached to the EC2 instance).
  • 1 x key pair (ml-deploy-key in AWS and local .pem file).
  • 1 x security group (ml-stack-sg).
  • Local project folder (ml-deploy-ec2 on your Desktop).
  • Local Docker images (fastapi-bert and streamlit-app).

Keep everything running

No action needed. Choose this if you are still actively experimenting with your ML inference stack.

  • Monitor your credit usage in the AWS Billing console to make sure your credits cover ongoing costs.
  • Your t3.medium instance will continue running at approximately $0.0418 per hour.
  • Local Docker images (fastapi-bert at roughly 2 GB, plus streamlit-app) will remain on your machine.

Pause - I'll come back to this later

Stop the EC2 instance to halt compute charges while keeping your files and configuration intact.

  • In the EC2 console, select the ml-inference-stack instance.
  • Choose Instance state, then Stop instance.
  • You will not be charged for compute while stopped. EBS storage charges (~$0.08/month for 8 GiB gp3) continue.
  • To resume later, select the instance and choose Instance state, then Start instance. Note that the public IP address may change when you restart.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow each step in order.

Terminate the EC2 instance:

  • In the EC2 console, select the ml-inference-stack instance.
  • Choose Instance state, then Terminate instance.
  • Confirm the termination. The attached 8 GiB EBS volume is deleted automatically with the instance.
  • Wait for the instance state to show Terminated before continuing to the security group step below.

Delete the ECR repositories:

  • Delete the FastAPI image repository by running this command in your local terminal:
aws ecr delete-repository --repository-name ml-deploy/fastapi-bert --region us-east-1 --force
  • Delete the Streamlit image repository by running:
aws ecr delete-repository --repository-name ml-deploy/streamlit-app --region us-east-1 --force

Delete the key pair:

  • In the EC2 console, choose Key Pairs in the navigation pane.
  • Select ml-deploy-key and choose Actions, Delete.
  • Enter Delete in the confirmation field and choose Delete.
  • Also delete the local ml-deploy-key.pem file from wherever you saved it on your computer.

Delete the security group:

  • The security group cannot be deleted until the instance is fully terminated. Make sure the instance state shows Terminated first.
  • In the EC2 console, choose Security Groups in the navigation pane.
  • Select ml-stack-sg and choose Actions, Delete security groups.
  • Choose Delete to confirm.

Remove local Docker images:

  • Remove the project images from your local machine by running:
docker rmi fastapi-bert:latest streamlit-app:latest

Delete the local project folder:

macOS

  • Delete the project folder by running:
rm -rf ~/Desktop/ml-deploy-ec2

Windows

  • Delete the project folder by running:
rmdir /s /q %USERPROFILE%\Desktop\ml-deploy-ec2

That's a Wrap!

That's a Wrap!

Nice work! You just deployed a full ML inference stack to the cloud using the same build-push-pull pipeline that production teams rely on every day.

You've learned how to:

  • Build custom Docker images for a FastAPI + DistilBERT inference API and a Streamlit frontend, then push them to Amazon ECR using the authenticate-tag-push workflow.
  • Launch a t3.medium EC2 instance with Amazon Linux 2023, install Docker and Docker Compose from scratch, and configure security groups to expose exactly the ports your application needs.
  • Deploy a 4-container stack (FastAPI, Streamlit, Prometheus, Grafana) on EC2 using Docker Compose with images pulled from ECR.
  • Secret Mission: Configured a Grafana monitoring dashboard with live Prometheus metrics for your ML inference endpoint.

Ready to quiz yourself?