Deploy an AI Inference API to Azure

Containerize and deploy a semantic similarity API to Azure Container Apps.

Introduction

30 Second Summary

Machine learning models can do incredible things, but they're useless sitting in a notebook on your laptop. The real skill is getting them into a production API that anyone can call over the internet.

In this project, you will build a semantic similarity API powered by a sentence embedding model, containerize it with Docker, and deploy it live to Azure Container Apps with a public HTTPS URL.

What You'll Build

You send two sentences to your live API, and it instantly tells you how related they are with a similarity score between 0 and 1.

By the end of this project, you'll have:

  • A live HTTPS endpoint where you POST two sentences and instantly get back a cosine similarity score powered by a 22M-parameter embedding model.
  • A FastAPI application that loads the ML model once at startup using the lifespan pattern, keeping every response fast without reloading weights per request.
  • A Docker image with model weights baked in, so cold starts take seconds instead of minutes.
  • Secret Mission: Add structured JSON logging with correlation IDs, prepping the API for production observability.

Are there any prerequisites?

You should have Python 3.11, Docker, the Azure CLI, and VS Code installed on your machine. You also need an Azure account with an active subscription.

Before We Start

Taking an ML model from a local Python script to a live cloud API is a real platform engineering skill. In this project, you will build a semantic similarity API, containerize it with Docker, and deploy it to Azure Container Apps so anyone can reach it over HTTPS.

Before you write any application code, you need the right CLI extensions and a clean project structure. This step gets your local environment ready so the rest of the project flows smoothly.

In this step, get ready to:
  • Install the Azure Container Apps CLI extension.
  • Register required Azure resource providers.
  • Create a project directory with a Python virtual environment and dependencies.
Install the Azure Container Apps CLI extension

Azure Container Apps is the serverless container platform you will deploy to. Its CLI commands are delivered as an extension to the Azure CLI you already have installed.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.
  • Install (or upgrade) the Container Apps extension by running this command:
az extension add --name containerapp --upgrade

What does this command do?

The az extension add command installs an extension into your Azure CLI. The --upgrade flag ensures you get the latest version if you already had an older one.

Before you verify, what do you think the output will show when you ask the CLI about the extension you just installed?

  • Confirm the extension is installed by running:
az extension show --name containerapp

You should see JSON output including the extension name and a version number. This confirms the Container Apps extension is ready to use.

Seeing 'extension not found'?

Make sure you are signed in to the Azure CLI. Run az login and complete the browser sign-in flow, then retry the install command.

help me troubleshoot why az extension add failed for the containerapp extension

Register Azure resource providers

Azure subscriptions need specific resource providers registered before you can create resources of that type. Container Apps requires two providers that may not be active on your subscription yet.

  • Register the Container Apps provider by running:
az provider register --namespace Microsoft.App
  • Register the Log Analytics provider by running:
az provider register --namespace Microsoft.OperationalInsights

What do these commands do?

Each command tells Azure to enable a resource type on your subscription. Microsoft.App covers Container Apps themselves. Microsoft.OperationalInsights covers the Log Analytics workspace that stores your container logs.

These commands produce no output on success. Registration happens in the background and may take a minute to complete.

Create the project directory and install dependencies

You need a dedicated folder with its own Python virtual environment so your project dependencies stay isolated from the rest of your system.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir ai-inference-api
cd ai-inference-api
  • Create a Python virtual environment and activate it by running:
python3 -m venv .venv
source .venv/bin/activate

Your terminal prompt now shows (.venv) at the beginning, confirming the virtual environment is active.

Don't see (.venv) in your prompt?

Make sure you ran source .venv/bin/activate (not just the python3 -m venv line). The first command creates the environment. The second command activates it.

help me activate my Python virtual environment on macOS

  • Create the requirements.txt file that lists your project dependencies by running:
cat > requirements.txt << 'EOF'
fastapi==0.139.2
uvicorn==0.51.0
sentence-transformers==5.6.1
EOF

What are these dependencies?

FastAPI is the web framework you will use to build your API endpoints. Uvicorn is an ASGI server that runs your FastAPI app. sentence-transformers is the ML library that provides the text embedding model for similarity scoring.

  • Install all dependencies by running:
pip install -r requirements.txt

This downloads and installs FastAPI, Uvicorn, sentence-transformers, and all their sub-dependencies. The install takes 1-2 minutes because sentence-transformers pulls in PyTorch.

Once the install finishes, you should see a Successfully installed line listing all packages near the bottom of the output.

Install failing with a permissions error?

Make sure your virtual environment is active (you see (.venv) in your prompt). If you accidentally run pip install without the venv active, it tries to install system-wide and may fail.

help me fix pip install permissions errors when installing Python packages

✔️ Awesome, I've got everything!

Great. Double check that requirements.txt is saved in your ai-inference-api folder.

ⓧ I'd like to double check the full code

Here is the complete requirements.txt file:

fastapi==0.139.2
uvicorn==0.51.0
sentence-transformers==5.6.1

Your environment is locked and loaded. Next up, you will build the actual inference API that loads an ML model and serves similarity predictions.

Build the Inference API Locally

Your environment is set up with all the dependencies installed. Now you need the actual application code that loads the ML model and serves predictions.

In this step, you will build a FastAPI application that loads the all-MiniLM-L6-v2 sentence embedding model once at startup and exposes two endpoints. One computes semantic similarity between two sentences. The other confirms the model is healthy.

In this step, get ready to:
  • Create the FastAPI application with lifespan-managed model loading.
  • Test the similarity endpoint locally with a pair of sentences.
  • Verify the health endpoint confirms the model is loaded and ready.
Create the application foundation

An ML model is expensive to load. You do not want to reload it on every request. FastAPI's lifespan pattern solves this by running setup code once when the server starts and cleanup code when it shuts down.

  • Create a file called main.py in your project folder and add the following code:
import time
from contextlib import asynccontextmanager

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from sentence_transformers import SentenceTransformer

# Shared dictionary to hold the loaded model across requests
model_store = {}


@asynccontextmanager
async def lifespan(app: FastAPI):
    # Load the embedding model once at startup
    model_store["model"] = SentenceTransformer(
        "sentence-transformers/all-MiniLM-L6-v2"
    )
    yield
    model_store.clear()


app = FastAPI(title="AI Inference API", lifespan=lifespan)


# Request and response schemas for the similarity endpoint
class SimilarityRequest(BaseModel):
    sentence1: str
    sentence2: str


class SimilarityResponse(BaseModel):
    similarity: float
    model: str
    duration_ms: float

What does this code do?

The lifespan function is an async context manager. Code before yield runs at server startup and loads the model into model_store. Code after yield runs at shutdown and clears the model from memory.

The SimilarityRequest and SimilarityResponse classes are Pydantic models that define what the API accepts and returns. FastAPI uses these to validate incoming data and document the API automatically.

  • Save main.py.

Seeing an import error?

Make sure your virtual environment is activated. You should see (.venv) at the start of your terminal prompt. If not, activate it again from your project folder.

help me fix an import error in my FastAPI app

Add the API endpoints

The foundation is ready. Now add the two endpoints that handle requests.

  • In main.py, add the following code below the SimilarityResponse class:
@app.get("/health")
async def health():
    # Return 503 if model hasn't finished loading
    if "model" not in model_store:
        raise HTTPException(status_code=503, detail="Model not loaded")
    return {"status": "healthy", "model": "loaded"}


@app.post("/similarity", response_model=SimilarityResponse)
async def similarity(request: SimilarityRequest):
    if "model" not in model_store:
        raise HTTPException(status_code=503, detail="Model not loaded")

    start = time.time()
    model = model_store["model"]
    # Encode both sentences into embeddings and compute cosine similarity
    embeddings = model.encode([request.sentence1, request.sentence2])
    score = float(model.similarity(embeddings[0:1], embeddings[1:2])[0][0])
    duration_ms = (time.time() - start) * 1000

    return SimilarityResponse(
        similarity=score,
        model="all-MiniLM-L6-v2",
        duration_ms=round(duration_ms, 2),
    )

What does this code do?

  • The /health endpoint checks whether the model is loaded and returns a simple status object. If the model is missing, it returns a 503 error.
  • The /similarity endpoint encodes both sentences into vector embeddings, then computes the cosine similarity between them. The score ranges from 0 (unrelated) to 1 (identical meaning).
  • The duration_ms field measures how long the inference took, which is useful for monitoring performance later.
  • Save main.py.

Not sure where to add the code?

The two endpoint functions go at the very bottom of main.py, after the SimilarityResponse class definition. Make sure there is a blank line between the class and the first @app.get decorator.

help me figure out where to place my endpoint code in main.py

✔️ Awesome, I've got everything!

Great. Double check that you have saved main.py with both the foundation code and the two endpoints.

ⓧ I'd like to double check the full code

Scroll up and confirm your main.py contains these sections in order:

  • Imports (time, asynccontextmanager, FastAPI, HTTPException, BaseModel, SentenceTransformer).
  • The model_store dictionary.
  • The lifespan async context manager loading sentence-transformers/all-MiniLM-L6-v2.
  • The app = FastAPI(...) line.
  • The SimilarityRequest and SimilarityResponse Pydantic models.
  • The /health GET endpoint.
  • The /similarity POST endpoint.
Run the API locally and test the similarity endpoint

The code is complete. Time to start the server and send it a real request.

  • Start the API server by running this command:
uvicorn main:app --host 0.0.0.0 --port 8000

What does this command do?

This starts the uvicorn ASGI server, pointing it at the app object inside main.py. The --host 0.0.0.0 flag makes it accessible on all network interfaces, and --port 8000 sets the port number.

On first run, the model will download from Hugging Face (around 80MB). This takes 20-60 seconds depending on your internet speed. You will see download progress bars in the terminal.

  • Wait until you see a line containing Uvicorn running on http://0.0.0.0:8000 in your terminal. This means the model finished loading and the server is ready.
  • Open a second terminal window. Press Cmd+Space (macOS) or the Windows key (Windows), type Terminal, and press Enter.

Before you send the request, what do you think the API will return for two sentences with similar meanings?

  • Test the similarity endpoint by running this command in your second terminal:
curl -X POST http://localhost:8000/similarity \
  -H "Content-Type: application/json" \
  -d '{"sentence1": "The cat sat on the mat", "sentence2": "A kitten is sitting on a rug"}'

You should see a JSON response with a similarity score between 0 and 1. Because these two sentences have very similar meanings, expect a score around 0.6 to 0.8.

Getting a connection refused error?

Make sure the first terminal is still running uvicorn. Look for the Uvicorn running on message. If uvicorn crashed during model loading, check for a ModuleNotFoundError and confirm your virtual environment is active.

help me fix a connection refused error when testing my FastAPI server

Test the health endpoint

The similarity endpoint works. Now confirm the health check returns the expected status so monitoring tools can verify the API is ready.

  • Test the health endpoint by running this command in your second terminal:
curl http://localhost:8000/health

You should see:

{"status": "healthy", "model": "loaded"}
  • Return to your first terminal and press Ctrl+C to stop the uvicorn server.

Health endpoint returning 503?

A 503 response means the model has not finished loading yet. Wait another 30 seconds for the model download to complete, then try the health endpoint again. If the server log shows an error during loading, ensure you have enough disk space for the model files.

help me debug why my health endpoint returns 503

Why the lifespan pattern instead of loading per-request?

Loading a model takes seconds. If you loaded it on every request, each API call would take 2-3 seconds just for setup before doing any actual work. The lifespan pattern loads the model once at startup, then every request reuses the already-loaded model from model_store.

This is the standard pattern for serving ML models in production. You will see this same approach in frameworks like TensorFlow Serving and Triton Inference Server.

Your inference API is working locally. Next up, you will containerize it with Docker so it can run anywhere, not just on your machine.

Containerize and Run Locally

Your API runs perfectly when you launch it with uvicorn on your machine. Azure Container Apps runs containers, not raw Python processes. You need to package your app as a Docker image before you can deploy it.

In this step, you'll write a Dockerfile, build your first image, and discover a real-world problem that plagues naive ML container deployments.

In this step, get ready to:
  • Create a Dockerfile and .dockerignore for the FastAPI app.
  • Build and run the Docker container locally.
  • Observe the cold-start problem caused by model downloads on every container start.
Create the Dockerfile and .dockerignore

Docker needs to know which files to ignore when building an image. A .dockerignore file keeps unnecessary files out of the build context, reducing image size and build time.

  • Create a new file called .dockerignore in your ai-inference-api folder with the following content:
__pycache__
*.pyc
.venv
.git
.env

What does this file do?

Each line tells Docker to exclude a file pattern from the build context. __pycache__ and *.pyc are compiled Python bytecode that will be regenerated inside the container.

.venv is your local virtual environment (the container installs its own). .git and .env are development files that have no place in a production image.

Now for the main event. A Dockerfile is a set of instructions that tells Docker how to build your image layer by layer.

  • Create a file called Dockerfile (no extension) in your ai-inference-api folder with this content:
# Use a slim Python 3.11 base image
FROM python:3.11-slim

# Set the working directory inside the container
WORKDIR /app

# Copy and install dependencies first (for layer caching)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy the application code
COPY main.py .

# Document the port the app listens on
EXPOSE 8000

# Start the FastAPI app with uvicorn
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

What does this Dockerfile do?

  • FROM python:3.11-slim starts from a minimal Python 3.11 image (about 150MB instead of 900MB for the full image).
  • COPY requirements.txt then RUN pip install installs dependencies in a separate layer. Docker caches this layer, so rebuilds are fast unless dependencies change.
  • COPY main.py . adds your application code after dependencies are installed.
  • CMD defines the command that runs when the container starts.
  • Save both files.
  • Confirm they exist by running:
ls Dockerfile .dockerignore

You should see both files listed in the output.

Don't see both files?

  • Make sure Dockerfile has no file extension. It should be Dockerfile, not Dockerfile.txt.
  • The .dockerignore file starts with a dot, so it may be hidden in your file browser. The ls command shows it regardless.
  • Confirm you are in the ai-inference-api directory when you run the command.

Help me troubleshoot my Dockerfile setup.

✔️ Awesome, I've got everything!

Great. Double check you have saved both files in your ai-inference-api folder.

ⓧ I'd like to double check the full code

Here is the complete Dockerfile:

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY main.py .

EXPOSE 8000

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

And the complete .dockerignore:

__pycache__
*.pyc
.venv
.git
.env
Build and run the container locally

Time to build your Docker image and run it. This is the moment of truth for your containerized API.

  • Build the Docker image by running this command:
docker build -t ai-inference-api .

What does this command do?

docker build reads your Dockerfile and executes each instruction to create a layered image. -t ai-inference-api tags the resulting image with the name ai-inference-api so you can reference it later. The . tells Docker to use the current directory as the build context.

The build will take a few minutes as it installs PyTorch and sentence-transformers inside the image. You will see each Dockerfile instruction execute in sequence.

  • Once the build completes, confirm the image was created by running:
docker images ai-inference-api

You should see one row showing ai-inference-api with a latest tag and its size.

Build failed?

  • Make sure Docker Desktop is running. You can check with docker info in your terminal.
  • Verify requirements.txt exists in the same folder as your Dockerfile. The build context starts from where you run the command.
  • If you see a network error during pip install, check your internet connection. The build needs to download packages.

Help me debug my Docker build failure.

Now run the container and test the API through Docker.

  • Start the container by running:
docker run -p 8000:8000 ai-inference-api

What does this command do?

-p 8000:8000 maps port 8000 on your machine to port 8000 inside the container. This lets you reach the API at http://localhost:8000 just like before, except now it is running inside a container.

Watch the terminal output. You will notice the container takes a long time before the API becomes ready.

  • Open a second terminal and test the similarity endpoint by running:
curl -X POST http://localhost:8000/similarity \
  -H "Content-Type: application/json" \
  -d '{"sentence1": "The cat sat on the mat", "sentence2": "A kitten rested on the rug"}'

You should get back a JSON response with a similarity score, just like when you tested locally.

Getting a connection refused error?

  • The container may still be starting up. Wait 30-60 seconds and try again. The model needs to download before the API can serve requests.
  • Check the first terminal where the container is running. If you see download progress bars, the model is still loading.
  • Make sure port 8000 is not still in use from the previous step. Stop any lingering uvicorn process first.

Help me connect to my Docker container.

Observe the cold-start problem

Before you continue, think about what just happened. How long did it take from running the container to the API being ready to serve requests?

  • Stop the running container with Ctrl+C in the first terminal.
  • Start it again to confirm the problem repeats:
docker run -p 8000:8000 ai-inference-api

Before you test the health endpoint, do you think it will respond immediately or take a while? Why?

  • In your second terminal, immediately try the health endpoint:
curl http://localhost:8000/health

You will get a connection refused error. The container is still downloading the model from the internet. Wait 30 seconds or more, then try again.

Every single time this container restarts, it re-downloads approximately 80MB of model weights from Hugging Face. That is the cold-start problem for ML containers.

Why is this a problem in production?

Azure Container Apps scales to zero when your API has no traffic. When a new request arrives, it spins up a fresh container. If that container needs 30+ seconds to download a model before it can serve a request, your users wait 30+ seconds for a response.

The fix is to bake the model weights directly into the Docker image so they load from the local filesystem in seconds. You will do that in the next step.

  • Stop the container with Ctrl+C.

You have proven the API works inside a container. You have also uncovered a serious performance problem. Next up, you will bake the model weights into the image and eliminate that slow cold start entirely.

Bake the Model into the Docker Image

Your container runs, but that 30+ second startup is a deal-breaker for production. Every time the container restarts, it re-downloads 80MB of model weights from the internet.

The fix is to bake the model weights directly into the Docker image at build time. The download happens once during docker build, and every container started from that image already has the weights on its filesystem.

In this step, get ready to:
  • Create a download script that saves model weights during the Docker build.
  • Update the Dockerfile to run the download script at build time.
  • Rebuild and confirm the container starts in under 10 seconds.
Create the model download script

You need a script that downloads the model from Hugging Face and saves it to a specific path inside the image. This path is exactly where your app already looks for the model at startup.

  • Create a new file called download_model.py in your project folder with the following code:
from sentence_transformers import SentenceTransformer

# Download model from Hugging Face and save to local path
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
model.save("/app/models/all-MiniLM-L6-v2")
print("Model downloaded and saved to /app/models/all-MiniLM-L6-v2")

What does this script do?

  • SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") downloads the 80MB model weights from the Hugging Face Hub.
  • model.save("/app/models/all-MiniLM-L6-v2") writes the weights to the exact filesystem path your main.py loads from at startup.
  • This script only runs during docker build. The model becomes a permanent layer in your Docker image.
  • Save download_model.py.
  • Confirm the file exists by running:
ls download_model.py

You should see download_model.py listed in the output.

File not found?

  • Make sure you saved the file in your ai-inference-api project folder, not a subfolder.
  • Check the filename is exactly download_model.py with no typos or extra extensions.

Why isn't my download_model.py file showing up?

✔️ Awesome, I've got everything!

Great. Make sure download_model.py is saved and move on to updating the Dockerfile.

ⓧ I'd like to double check the full code

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
model.save("/app/models/all-MiniLM-L6-v2")
print("Model downloaded and saved to /app/models/all-MiniLM-L6-v2")
Update the Dockerfile to run the script at build time

The download script needs to execute during docker build so the model weights become a baked-in layer of your image.

  • In your Dockerfile, find the line that says:
COPY main.py .
  • Add these two lines directly above COPY main.py .:
COPY download_model.py .
RUN python download_model.py

What do these lines do?

  • COPY download_model.py . brings your script into the build context inside the container.
  • RUN python download_model.py executes the script during build, downloading the model and saving it as a permanent image layer.
  • Placing these lines after the pip install step means Docker can cache the dependency layer separately from the model layer.
  • Save your Dockerfile.

Not sure where to place the lines?

The two new lines go between RUN pip install --no-cache-dir -r requirements.txt and COPY main.py .. Make sure there is a blank line before and after the new block for readability.

Help me figure out where to put the download lines in my Dockerfile.

✔️ Awesome, I've got everything!

Great. Double check you saved the Dockerfile before rebuilding.

ⓧ I'd like to double check the full code

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY download_model.py .
RUN python download_model.py

COPY main.py .

EXPOSE 8000

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
Rebuild and confirm fast startup

Time to rebuild the image with the model baked in and see the difference.

  • Rebuild your Docker image by running:
docker build -t ai-inference-api .

Why does this build take longer?

The build now downloads 80MB of model weights during the RUN python download_model.py step. This takes a minute or two depending on your connection.

This is a one-time cost. Every container started from this image already has the model on disk. No more runtime downloads.

Once the build completes, you should see a success message at the end of the output.

Before you run the container, how quickly do you expect it to become ready now that the model is already baked into the image?

  • Run your newly built container:
docker run -p 8000:8000 ai-inference-api
  • Open a new terminal tab and test the health endpoint immediately:
curl http://localhost:8000/health

You should see the response within seconds of the container starting:

{"status": "healthy", "model": "loaded"}

The container becomes ready in under 10 seconds instead of 30+. The model loads straight from the filesystem, not the network.

Why is this so much faster?

In Step 3, every container start triggered an 80MB download from Hugging Face. Now the weights are already on disk as part of the image.

Loading model files from the local filesystem takes a few seconds. Downloading them over the internet took 30+ seconds. This is why production ML containers bake model weights into the image.

Still seeing slow startup or connection refused?

  • Make sure you rebuilt the image after updating the Dockerfile. Run docker build -t ai-inference-api . again.
  • If you get "connection refused" on the curl, wait a few seconds for the model to finish loading from disk. It should take under 10 seconds.
  • If port 8000 is already in use, stop the old container first with docker stop on the previous container ID or press Ctrl+C in the terminal running the old container.

My container still takes a long time to start. Help me debug the issue.

Your optimized container is ready for production. Next up, you will deploy it to Azure Container Apps and get a live public URL.

Deploy to Azure Container Apps

Your optimized Docker image is ready. The model weights are baked in, cold starts are fast, and the API serves predictions locally.

Now you need a public HTTPS endpoint so anyone can hit your API from anywhere. Azure Container Apps gives you exactly that. It provides automatic TLS, scale-to-zero when idle, and fast cold starts thanks to the baked-in model.

In this step, get ready to:
  • Deploy your container to Azure Container Apps with a single command.
  • Test the live health and similarity endpoints from your terminal.
  • View container logs to confirm the model loaded in the cloud.
Deploy using az containerapp up

The az containerapp up command is a one-shot deployment tool. It builds your image in the cloud using your Dockerfile, pushes it to a registry, and creates all the infrastructure in one go.

  • Confirm you are signed into the Azure CLI by running:
az account show

What does this check?

This prints your current Azure subscription details. If you see your subscription name and ID, you are logged in.

If you get an error, run az login and follow the browser prompts before continuing.

  • Make sure your terminal is in the ai-inference-api project directory (where your Dockerfile lives).
  • Deploy your container app to Azure by running this command:
az containerapp up \
  --resource-group ai-inference-rg \
  --name ai-inference-api \
  --ingress external \
  --target-port 8000 \
  --source .

What does this command do?

This single command handles six things at once:

  • Creates a resource group called ai-inference-rg.
  • Creates an Azure Container Registry to store your Docker image.
  • Builds the image from your local Dockerfile and pushes it to the registry.
  • Creates a Container Apps Environment with a Log Analytics Workspace.
  • Deploys the container app with external ingress on port 8000.
  • Returns the public URL for your live app.

This takes a few minutes

The first deployment builds your Docker image in the cloud, including downloading all Python packages and the model weights (~80MB). Expect 3-5 minutes.

You will see progress output in your terminal as the image builds and deploys.

Once the command completes, you will see output that includes the app's URL. Look for a line containing .azurecontainerapps.io in the output.

Seeing an authentication error?

If you see a message about sign-in or permissions, run az login to re-authenticate, then retry the az containerapp up command.

If you see a message about the containerapp extension not being found, run az extension add --name containerapp --upgrade to install it.

Help me troubleshoot my deployment

Test your live deployment

Your API is now live on the internet. Let's retrieve its URL and verify both endpoints work against the cloud deployment.

  • Get your app's public URL by running:
az containerapp show \
  --name ai-inference-api \
  --resource-group ai-inference-rg \
  --query "properties.configuration.ingress.fqdn" \
  -o tsv

This prints just the domain name (e.g. ai-inference-api.somehash.region.azurecontainerapps.io). Copy this value.

  • Before you test, what do you expect the /health endpoint to return? Think about what the model loading code does at startup.
  • Test the health endpoint by running (replace the URL with your actual domain):
curl https://[[APP_URL="your-app-name.somehash.region.azurecontainerapps.io"]]/health

You should see:

{"status": "healthy", "model": "loaded"}

What does this prove?

The health response confirms the model loaded successfully from the baked-in image weights inside the cloud container. No network download happened at startup.

  • Now test the similarity endpoint with a pair of sentences:
curl -X POST https://[[APP_URL="your-app-name.somehash.region.azurecontainerapps.io"]]/similarity \
  -H "Content-Type: application/json" \
  -d '{"sentence1": "The cat sat on the mat", "sentence2": "A kitten rested on the rug"}'

You should see a JSON response with a similarity score between 0 and 1, a model name, and a duration in milliseconds.

Getting a connection refused or timeout?

The container may still be starting up if you test immediately after deployment. Wait 30 seconds and try again.

Make sure you are using https:// not http:// in your URL. Azure Container Apps enforces HTTPS.

Help me debug my connection to the deployed API

View the container logs

The final confirmation is checking the container's own logs in the cloud to see that the model loaded from the filesystem at startup.

  • View the console logs from your deployed container by running:
az containerapp logs show \
  -n ai-inference-api \
  -g ai-inference-rg

What am I looking for?

In the log output, look for startup messages from uvicorn and sentence-transformers.

You should see the model being loaded from /app/models/all-MiniLM-L6-v2 (local path, not a download URL). This confirms the baked-in model is working in production.

Seeing empty logs or a scaled-to-zero message?

If the app scaled to zero between your last request and the log check, the logs from the previous instance may not appear. Send another request to /health to wake the container, wait a few seconds, then run the logs command again.

Help me troubleshoot my container logs

Your AI inference API is live on the internet with a public HTTPS endpoint. Anyone with the URL can send two sentences and get a semantic similarity score back.

Next up, the Secret Mission. You will add structured JSON logging with correlation IDs so every request is traceable through your production logs.

Secret mission

Add Structured JSON Logging with Correlation IDs

Production APIs need more than print statements. In this extension, you will add structured JSON logging with unique correlation IDs per request. This prepares your API for real-world observability where you can trace any request through your system.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses Azure Container Apps with the Consumption plan, which scales to zero automatically when idle.

Cost warning

With the Consumption plan free tier (180,000 vCPU-seconds/month), your app at scale-to-zero costs nothing while idle. If you set min-replicas to 1, expect minimal charges against your Azure credits.

Resources you used:

  • Azure Resource Group (contains all Azure resources below).
  • Azure Container Registry (stores your Docker image).
  • Azure Container Apps Environment (networking and logging infrastructure).
  • Azure Container App (your running inference API).
  • Log Analytics Workspace (stores container logs).
  • Local project folder with source code, Dockerfile, and virtual environment.

Keep everything running

No action needed. Choose this if you are still actively building or plan to extend the API soon.

  • Your Container App scales to zero automatically when not receiving requests. Cost is effectively $0 while idle.
  • The Azure Container Apps CLI extension remains installed in your Azure CLI. It uses negligible disk space and is useful for future container projects.
  • Your local project folder stays on your Desktop for further development.

Pause - I'll come back to this later

Scale-to-zero handles this for you. Your Container App already stops running when it receives no traffic, so there is nothing to manually pause.

  • Your app automatically scales down to zero replicas after a period of inactivity. No charges accrue while scaled to zero.
  • When you send a request again later, the app scales back up automatically. The baked-in model means startup takes seconds, not minutes.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes your Azure resources and local files permanently.

  • Delete the entire Azure Resource Group (this cascades to the Container Registry, Container Apps Environment, Container App, and Log Analytics Workspace) by running:
  • Note: replace the resource group name with the one created during your deployment.
az group delete --name [[RESOURCE_GROUP="your resource group name"]] --yes --no-wait

Find your resource group name

If you are unsure of the name, run this command to list your resource groups:

az group list --output table

Look for the group that was created when you ran az containerapp up. It typically starts with your app name.

  • Remove the Azure Container Apps CLI extension by running:
az extension remove --name containerapp
  • Delete your local project folder by running:
cd ~/Desktop
rm -rf [[PROJECT_DIR="your project folder name"]]

On Windows, use this instead:

cd ~\Desktop
rmdir /s /q [[PROJECT_DIR="your project folder name"]]
  • Confirm the folder is gone by running:
ls

That's a Wrap!

That's a Wrap!

Nice work! You just took an ML model from a local Python script to a live, production-ready API running on Azure Container Apps with a public HTTPS endpoint anyone can hit.

You've learned how to:

  • Build a FastAPI inference API using the lifespan pattern for proper ML model lifecycle management.
  • Containerize an application with Docker and optimize cold starts by baking model weights directly into the image.
  • Deploy to Azure Container Apps with a public HTTPS endpoint and scale-to-zero, solving the cold-start model download problem that plagues naive ML deployments.
  • Secret Mission: Add structured JSON logging with correlation IDs for production observability.

Ready to quiz yourself?