Resilient Microservices on Azure

Deploy 3 microservices on Azure Container Apps with circuit breakers and retries.

Introduction

30 Second Summary

When Netflix streams movies to 260M+ subscribers, individual services fail all the time. The system stays up because it's designed to self-heal automatically using resilience patterns like circuit breakers and retries.

In this project, you will build a three-service microservices system on Azure Container Apps, configure platform-level resiliency policies, and prove they work by intentionally breaking your services and watching them recover.

What You'll Build

A "Streaming Catalog" microservices system with three Node.js services that communicate internally, protected by Azure Container Apps resiliency policies that implement the same patterns Netflix uses in production.

By the end of this project, you'll have:

  • 3 Node.js Express microservices (API Gateway, Catalog Service, Recommendations Service) deployed on Azure Container Apps with internal service discovery.
  • Platform-level resiliency policies implementing circuit breakers, retries with exponential backoff, and timeouts to protect against cascading failures.
  • A fault injection endpoint that lets you simulate failures on demand and observe automatic recovery as the circuit breaker trips and resets.
  • Secret Mission: Configure auto-scaling rules and load-test your services to watch horizontal scaling happen in real time.

Are there any prerequisites?

You'll need an active Azure subscription, Azure CLI installed, Docker installed, and Node.js 24 installed on your machine.

Step 1 walks you through installing the Azure Container Apps CLI extension and creating your cloud environment.

Set Up the Azure Environment

With the Container Apps extension installed, you now have a toolbox that speaks the right language. But you still need somewhere to deploy your services.

In this substep, you will create the Azure infrastructure that hosts your microservices. This includes a resource group (a logical container for related resources) and a Container Apps environment (the shared network boundary where your three services will discover each other).

In this step, get ready to:
  • Sign in to Azure from the CLI.
  • Create a resource group to hold all your project resources.
  • Create a Container Apps environment for service-to-service communication.
Sign in to Azure and create the resource group

Before you can create any Azure resources, you need an active session with the CLI. After signing in, you will create a resource group that acts as a folder for all the resources in this project.

  • Sign in to your Azure account by running this command:
az login

A browser window opens for you to authenticate. After signing in, your terminal displays a list of your Azure subscriptions.

What does this command do?

The az login command opens a browser-based authentication flow. Once you confirm your identity, the CLI stores a token so every subsequent command runs under your account.

  • Create a resource group named netflix-resilience-rg by running this command:
az group create --name netflix-resilience-rg --location eastus

What does this command do?

A resource group is a logical container in Azure that holds related resources together. Everything you deploy in this project lives inside netflix-resilience-rg.

The --location eastus flag sets the region where your resources are physically hosted. You can substitute a region closer to you if you prefer.

  • Confirm the resource group was created successfully. You should see JSON output containing "provisioningState": "Succeeded" in your terminal.

Seeing a subscription error?

If you see an error about no subscriptions found, make sure you signed in with the correct Microsoft account that has an active Azure subscription.

If you have multiple subscriptions, set the correct one by running az account set --subscription followed by your subscription ID.

help me fix my Azure subscription error

Create the Container Apps environment

A Container Apps environment is a shared boundary where your microservices can discover and communicate with each other over an internal network. All three services you build later will live inside this environment.

  • Create the Container Apps environment by running this command:
az containerapp env create --name netflix-resilience-env --resource-group netflix-resilience-rg --location eastus

This command takes 1-2 minutes to complete. Wait for it to finish before moving on.

What does this command do?

This creates a managed environment on the consumption plan. The consumption plan means your services scale to zero when idle and you only pay for actual compute usage.

Services deployed to the same environment can reach each other via http://<app-name> without exposing anything to the public internet. This is how your API Gateway will talk to the Catalog and Recommendations services later.

Why a Container Apps environment instead of Kubernetes?

Azure Container Apps provides built-in resiliency policies (circuit breakers, retries, timeouts) as a platform feature. With AKS (Azure Kubernetes Service), you would need to configure a service mesh like Istio yourself. Container Apps lets you focus on resilience patterns instead of infrastructure plumbing.

Verify the environment is provisioned and ready
  • Check the environment's provisioning state by running this command:
az containerapp env show --name netflix-resilience-env --resource-group netflix-resilience-rg --query properties.provisioningState
  • You should see "Succeeded" in the output. This confirms your environment is ready for container app deployments.

Seeing "InProgress" or "Waiting"?

The environment can take a couple of minutes to provision. Wait 30 seconds and run the show command again.

If the state remains stuck after 5 minutes, try deleting the environment and recreating it with the same command above.

help me troubleshoot my Container Apps environment provisioning

Your Azure infrastructure is ready. You have a resource group and a Container Apps environment waiting to host your microservices. Next up, you will build the three services that make up your streaming catalog system.

Build the Three Microservices Locally

Your Azure Container Apps environment is provisioned and ready to host containers. Now it's time to build the three microservices that will run inside it.

A real streaming platform doesn't run as one giant application. It splits responsibilities across small, focused services that communicate over HTTP. If one service goes down, the others need to keep working. You'll build exactly this architecture with three Node.js Express services.

In this step, get ready to:
  • Create a Catalog Service that serves movie data.
  • Create a Recommendations Service with a failure-simulation endpoint.
  • Create an API Gateway with built-in fallback logic.
Create the Catalog Service

The Catalog Service is the simplest of the three. It holds a static list of movies and exposes endpoints that other services can call to retrieve movie data.

  • Move to your Desktop and create the project folder by running these commands:
cd ~/Desktop
mkdir netflix-resilience
cd netflix-resilience
  • Create the catalog-service directory and move into it:
mkdir catalog-service
cd catalog-service
  • Create a file called package.json inside catalog-service and add the following content:
{
  "name": "catalog-service",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "start": "node index.js"
  },
  "dependencies": {
    "express": "5.2.1"
  }
}

What does this file do?

The package.json file defines your service's metadata and dependencies.

  • "type": "module" enables ES module syntax (import instead of require).
  • "express": "5.2.1" pins the Express version so all three services use the same framework.
  • Install the dependencies by running:
npm install

You should see a summary line like "added X packages" confirming Express was installed.

Seeing permission errors or network failures?

  • Make sure you are inside the catalog-service folder (run pwd to check).
  • Check your internet connection. npm install downloads packages from the npm registry.

Still stuck? Help me troubleshoot npm install failing in my catalog-service directory.

  • Create a file called index.js inside catalog-service and add the following code:
import express from "express";

const app = express();
const PORT = 3000;

// Static movie data the Catalog Service serves
const movies = [
  { id: 1, title: "The Matrix", genre: "Sci-Fi", year: 1999 },
  { id: 2, title: "Inception", genre: "Sci-Fi", year: 2010 },
  { id: 3, title: "The Dark Knight", genre: "Action", year: 2008 },
  { id: 4, title: "Interstellar", genre: "Sci-Fi", year: 2014 },
  { id: 5, title: "Parasite", genre: "Thriller", year: 2019 },
];

// Health check endpoint for container orchestration
app.get("/health", (req, res) => {
  res.json({ status: "healthy", service: "catalog-service" });
});

// Return the full movie list
app.get("/movies", (req, res) => {
  res.json(movies);
});

// Return a single movie by ID
app.get("/movies/:id", (req, res) => {
  const movie = movies.find((m) => m.id === parseInt(req.params.id));
  if (!movie) return res.status(404).json({ error: "Movie not found" });
  res.json(movie);
});

app.listen(PORT, () => {
  console.log(`Catalog Service running on port ${PORT}`);
});

What does this code do?

  • /health returns a JSON status object. Azure Container Apps will use this to know if the service is alive.
  • /movies returns the full array of five movies. The API Gateway will call this endpoint.
  • /movies/:id returns a single movie by its numeric ID, or a 404 if the ID doesn't exist.
  • Save index.js.
  • Start the service by running:
node index.js

You should see Catalog Service running on port 3000 printed in your terminal.

  • Open a new terminal tab by pressing Cmd+T (macOS) or Ctrl+Shift+T (Windows/Linux).
  • Test the movies endpoint by running:
curl http://localhost:3000/movies

You should see a JSON array of five movies including The Matrix and Inception.

  • Switch back to the terminal running the service and press Ctrl+C to stop it.

Don't see the movie data?

  • If you see "Cannot find module", make sure you ran npm install before starting the server.
  • If port 3000 is already in use, stop whatever process is using it, then try again.

Need more help? Help me debug why my Catalog Service isn't responding on port 3000.

✔️ Awesome, I've got everything!

Great. Make sure you've saved index.js and stopped the server before moving on.

ⓧ I'd like to double check the full code

import express from "express";

const app = express();
const PORT = 3000;

const movies = [
  { id: 1, title: "The Matrix", genre: "Sci-Fi", year: 1999 },
  { id: 2, title: "Inception", genre: "Sci-Fi", year: 2010 },
  { id: 3, title: "The Dark Knight", genre: "Action", year: 2008 },
  { id: 4, title: "Interstellar", genre: "Sci-Fi", year: 2014 },
  { id: 5, title: "Parasite", genre: "Thriller", year: 2019 },
];

app.get("/health", (req, res) => {
  res.json({ status: "healthy", service: "catalog-service" });
});

app.get("/movies", (req, res) => {
  res.json(movies);
});

app.get("/movies/:id", (req, res) => {
  const movie = movies.find((m) => m.id === parseInt(req.params.id));
  if (!movie) return res.status(404).json({ error: "Movie not found" });
  res.json(movie);
});

app.listen(PORT, () => {
  console.log(`Catalog Service running on port ${PORT}`);
});
Create the Recommendations Service

The Recommendations Service suggests related movies based on a given movie ID. It also has a special feature: a /fail endpoint that toggles simulated failures on and off. You'll use this later to trigger the circuit breaker.

  • Navigate back to the netflix-resilience folder by running:
cd ~/Desktop/netflix-resilience
  • Create the recommendations-service directory and move into it:
mkdir recommendations-service
cd recommendations-service
  • Create package.json with the following content:
{
  "name": "recommendations-service",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "start": "node index.js"
  },
  "dependencies": {
    "express": "5.2.1"
  }
}
  • Install the dependencies by running:
npm install
  • Create index.js and add the following code:
import express from "express";

const app = express();
const PORT = 3000;

// Toggle that controls whether the service returns errors
let failMode = false;

// Mapping of movie IDs to recommended movie IDs
const recommendations = {
  1: [2, 4],
  2: [1, 4],
  3: [5],
  4: [1, 2],
  5: [3],
};

// Health check returns unhealthy when fail mode is active
app.get("/health", (req, res) => {
  if (failMode) {
    return res.status(503).json({ status: "unhealthy", failMode: true });
  }
  res.json({ status: "healthy", service: "recommendations-service" });
});

// Return recommendations for a given movie, or 500 if fail mode is on
app.get("/recommendations/:movieId", (req, res) => {
  if (failMode) {
    return res.status(500).json({ error: "Service unavailable (simulated failure)" });
  }
  const movieId = parseInt(req.params.movieId);
  const recs = recommendations[movieId] || [];
  res.json({ movieId, recommendedIds: recs });
});

// Toggle fail mode on or off with a POST request
app.post("/fail", (req, res) => {
  failMode = !failMode;
  res.json({ failMode, message: `Fail mode ${failMode ? "ENABLED" : "DISABLED"}` });
});

app.listen(PORT, () => {
  console.log(`Recommendations Service running on port ${PORT}`);
});

What does this code do?

  • failMode is a boolean toggle. When true, every request to /recommendations/:movieId returns a 500 error instead of real data.
  • POST /fail flips the toggle. This is your "chaos button" for testing resilience without redeploying.
  • The /health endpoint also reflects fail mode by returning 503, so Azure can detect the unhealthy state.
  • Save index.js and start the service:
node index.js

You should see Recommendations Service running on port 3000 in your terminal.

  • In your other terminal tab, test the recommendations endpoint:
curl http://localhost:3000/recommendations/1

You should see {"movieId":1,"recommendedIds":[2,4]} confirming that movie 1 (The Matrix) recommends movies 2 and 4.

  • Stop the service with Ctrl+C in the first terminal.

Getting connection refused?

  • Make sure you stopped the Catalog Service before starting this one. Both use port 3000.
  • Confirm you are running curl in a separate terminal from the running server.

Still stuck? Help me debug why my Recommendations Service isn't responding.

✔️ Awesome, I've got everything!

Great. Make sure the service is stopped before moving on to the API Gateway.

ⓧ I'd like to double check the full code

import express from "express";

const app = express();
const PORT = 3000;

let failMode = false;

const recommendations = {
  1: [2, 4],
  2: [1, 4],
  3: [5],
  4: [1, 2],
  5: [3],
};

app.get("/health", (req, res) => {
  if (failMode) {
    return res.status(503).json({ status: "unhealthy", failMode: true });
  }
  res.json({ status: "healthy", service: "recommendations-service" });
});

app.get("/recommendations/:movieId", (req, res) => {
  if (failMode) {
    return res.status(500).json({ error: "Service unavailable (simulated failure)" });
  }
  const movieId = parseInt(req.params.movieId);
  const recs = recommendations[movieId] || [];
  res.json({ movieId, recommendedIds: recs });
});

app.post("/fail", (req, res) => {
  failMode = !failMode;
  res.json({ failMode, message: `Fail mode ${failMode ? "ENABLED" : "DISABLED"}` });
});

app.listen(PORT, () => {
  console.log(`Recommendations Service running on port ${PORT}`);
});
Create the API Gateway

The API Gateway is the only service that faces the public internet. It receives external requests, calls the Catalog and Recommendations services internally, and assembles a combined response.

The key design pattern here is graceful degradation. If a downstream service is down, the Gateway returns a fallback response instead of crashing.

  • Navigate back to the netflix-resilience folder:
cd ~/Desktop/netflix-resilience
  • Create the api-gateway directory and move into it:
mkdir api-gateway
cd api-gateway
  • Create package.json with the following content:
{
  "name": "api-gateway",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "start": "node index.js"
  },
  "dependencies": {
    "express": "5.2.1"
  }
}
  • Install the dependencies by running:
npm install

Now you'll build the Gateway's application code in two parts: first the utility function that handles failures, then the route handlers.

  • Create index.js and add the imports, configuration, and the fetchWithFallback function:
import express from "express";

const app = express();
const PORT = 3000;

// URLs default to internal Container Apps service names
const CATALOG_URL = process.env.CATALOG_URL || "http://catalog-service";
const RECOMMENDATIONS_URL = process.env.RECOMMENDATIONS_URL || "http://recommendations-service";

// Calls an upstream service and returns fallback data if it fails
async function fetchWithFallback(url, fallback) {
  try {
    const response = await fetch(url, { signal: AbortSignal.timeout(8000) });
    if (!response.ok) {
      console.log(`Upstream returned ${response.status} for ${url}`);
      return fallback;
    }
    return await response.json();
  } catch (error) {
    console.log(`Failed to reach ${url}: ${error.message}`);
    return fallback;
  }
}

What does fetchWithFallback do?

This function is the heart of the Gateway's resilience strategy.

  • It calls an upstream URL with an 8-second timeout using AbortSignal.timeout(8000).
  • If the upstream returns a non-200 status or throws a network error, the function returns the fallback value you pass in.
  • The caller never sees an exception. It either gets real data or safe fallback data.
  • Add the route handlers and the server listen call below the fetchWithFallback function in the same file:
app.get("/health", (req, res) => {
  res.json({ status: "healthy", service: "api-gateway" });
});

// Fetch all movies from the Catalog Service
app.get("/catalog", async (req, res) => {
  const movies = await fetchWithFallback(`${CATALOG_URL}/movies`, []);
  res.json({ source: movies.length > 0 ? "live" : "fallback", movies });
});

// Fetch a single movie with its recommendations
app.get("/catalog/:id/full", async (req, res) => {
  const movie = await fetchWithFallback(
    `${CATALOG_URL}/movies/${req.params.id}`,
    null
  );

  if (!movie) {
    return res.status(404).json({ error: "Movie not found" });
  }

  const recs = await fetchWithFallback(
    `${RECOMMENDATIONS_URL}/recommendations/${req.params.id}`,
    { movieId: parseInt(req.params.id), recommendedIds: [], fallback: true }
  );

  res.json({
    movie,
    recommendations: recs,
    recommendationsSource: recs.fallback ? "fallback" : "live",
  });
});

app.listen(PORT, () => {
  console.log(`API Gateway running on port ${PORT}`);
});

What do these routes do?

  • /catalog calls the Catalog Service for the full movie list. If the Catalog is down, it returns an empty array with "source": "fallback".
  • /catalog/:id/full fetches a single movie AND its recommendations. If recommendations fail, the response still includes the movie data but flags "recommendationsSource": "fallback".
  • The CATALOG_URL and RECOMMENDATIONS_URL environment variables default to internal Container Apps service names. When deployed, services reach each other via http://app-name automatically.

Why environment variables for service URLs?

Locally, these URLs won't resolve because there's no container network. On Azure Container Apps, services in the same environment reach each other via http://app-name (no port needed). Using environment variables lets you override the URLs for local testing without changing code.

  • Save index.js and start the Gateway:
node index.js

You should see API Gateway running on port 3000 in your terminal.

  • In your other terminal tab, test the health endpoint:
curl http://localhost:3000/health

You should see {"status":"healthy","service":"api-gateway"}.

  • Test the catalog endpoint to see the fallback behavior:
curl http://localhost:3000/catalog

You should see {"source":"fallback","movies":[]} because the Catalog Service isn't running locally right now. This is graceful degradation in action.

  • Stop the Gateway with Ctrl+C.

Seeing an error instead of the fallback JSON?

  • Make sure both code chunks are in the same index.js file. The routes chunk goes directly below the fetchWithFallback function.
  • Check for missing closing braces or parentheses. A syntax error will prevent the server from starting at all.

Need help? Help me fix my API Gateway code so it starts without errors.

✔️ Awesome, I've got everything!

Great. Double check you've saved api-gateway/index.js and stopped the server.

ⓧ I'd like to double check the full code

import express from "express";

const app = express();
const PORT = 3000;

const CATALOG_URL = process.env.CATALOG_URL || "http://catalog-service";
const RECOMMENDATIONS_URL = process.env.RECOMMENDATIONS_URL || "http://recommendations-service";

async function fetchWithFallback(url, fallback) {
  try {
    const response = await fetch(url, { signal: AbortSignal.timeout(8000) });
    if (!response.ok) {
      console.log(`Upstream returned ${response.status} for ${url}`);
      return fallback;
    }
    return await response.json();
  } catch (error) {
    console.log(`Failed to reach ${url}: ${error.message}`);
    return fallback;
  }
}

app.get("/health", (req, res) => {
  res.json({ status: "healthy", service: "api-gateway" });
});

app.get("/catalog", async (req, res) => {
  const movies = await fetchWithFallback(`${CATALOG_URL}/movies`, []);
  res.json({ source: movies.length > 0 ? "live" : "fallback", movies });
});

app.get("/catalog/:id/full", async (req, res) => {
  const movie = await fetchWithFallback(
    `${CATALOG_URL}/movies/${req.params.id}`,
    null
  );

  if (!movie) {
    return res.status(404).json({ error: "Movie not found" });
  }

  const recs = await fetchWithFallback(
    `${RECOMMENDATIONS_URL}/recommendations/${req.params.id}`,
    { movieId: parseInt(req.params.id), recommendedIds: [], fallback: true }
  );

  res.json({
    movie,
    recommendations: recs,
    recommendationsSource: recs.fallback ? "fallback" : "live",
  });
});

app.listen(PORT, () => {
  console.log(`API Gateway running on port ${PORT}`);
});

All three services are built and tested locally. Next up, you'll containerize them with Docker and deploy them to your Azure Container Apps environment where they can communicate with each other over the internal network.

Containerize and Deploy to Azure Container Apps

Your three services work locally, but right now they only exist on your machine. To run them on Azure, each service needs to be packaged into a Docker container.

A container bundles your code, its dependencies, and runtime instructions into a single portable unit that runs identically on any machine. In this step, you will write Dockerfiles for each service, test them locally, then deploy all three to Azure Container Apps with one command per service.

In this step, get ready to:
  • Write Dockerfiles for each service and test them locally with Docker.
  • Deploy all three services to Azure Container Apps with the correct ingress settings.
  • Verify end-to-end service-to-service communication through the API Gateway's public URL.
Write Dockerfiles for each service and build locally

A Dockerfile is a recipe that tells Docker how to build your application into an image. All three services share the same structure, so you will write one Dockerfile and replicate it across each service folder.

  • Create a new file called Dockerfile inside your catalog-service folder, then paste the following code:
FROM node:22-slim
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY index.js .
EXPOSE 3000
CMD ["node", "index.js"]

What does this Dockerfile do?

  • FROM node:22-slim starts from a minimal Node.js 22 image, keeping the final container small.
  • COPY package*.json then RUN npm ci --omit=dev installs only production dependencies. Copying package files first lets Docker cache this layer.
  • EXPOSE 3000 declares the port the container listens on. Azure Container Apps reads this to configure routing automatically.
  • CMD ["node", "index.js"] starts your Express server when the container launches.
  • Create a Dockerfile inside recommendations-service with the exact same content:
FROM node:22-slim
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY index.js .
EXPOSE 3000
CMD ["node", "index.js"]
  • Create a Dockerfile inside api-gateway with the same content:
FROM node:22-slim
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY index.js .
EXPOSE 3000
CMD ["node", "index.js"]
  • Save all three Dockerfiles.
  • Build and test the Catalog Service container by running these commands from inside the catalog-service folder:
cd catalog-service
docker build -t catalog-service:dev .
docker run -d -p 3001:3000 --name catalog-test catalog-service:dev
  • Verify the container is serving data by running:
curl http://localhost:3001/movies

You should see a JSON array of five movies returned from the container.

  • Stop and remove the test container by running:
docker stop catalog-test && docker rm catalog-test
  • Return to the project root by running:
cd ..

Docker build failing?

  • Make sure Docker Desktop is running. The docker CLI cannot connect to the daemon if the app is closed.
  • Check that you are inside the catalog-service folder when running the build. The Dockerfile must be in the current directory.
  • If you see a package-lock.json error, run npm install once in the service folder to generate the lockfile, then re-run the build.

Help me troubleshoot my Docker build

Deploy all three services to Azure Container Apps

Azure Container Apps can build your Docker image in the cloud and deploy it in a single command. The az containerapp up command detects the Dockerfile, builds the image, pushes it to an auto-created Azure Container Registry, and deploys the container app.

The Catalog and Recommendations services use internal ingress so they are only reachable by other apps in the same environment. The API Gateway uses external ingress so it has a public URL.

  • Deploy the Catalog Service by running this command from the project root:
az containerapp up \
  --name catalog-service \
  --resource-group netflix-resilience-rg \
  --environment netflix-resilience-env \
  --source ./catalog-service \
  --ingress internal \
  --target-port 3000

What does this command do?

  • --source ./catalog-service tells Azure to build from the local source directory containing the Dockerfile.
  • --ingress internal makes the service reachable only by other container apps in the same environment. External traffic gets rejected with a 404.
  • --target-port 3000 routes incoming requests to the port your Express server listens on.
  • Azure automatically creates a Container Registry if one doesn't exist yet. It stores the built Docker image there.

This command takes 2-3 minutes to complete. Wait for it to finish before proceeding.

  • Deploy the Recommendations Service by running:
az containerapp up \
  --name recommendations-service \
  --resource-group netflix-resilience-rg \
  --environment netflix-resilience-env \
  --source ./recommendations-service \
  --ingress internal \
  --target-port 3000
  • Deploy the API Gateway with external ingress by running:
az containerapp up \
  --name api-gateway \
  --resource-group netflix-resilience-rg \
  --environment netflix-resilience-env \
  --source ./api-gateway \
  --ingress external \
  --target-port 3000

When this command completes, it prints the public URL for the API Gateway. Copy this URL. You will use it in the next substep.

Why internal vs external ingress?

In a microservices architecture, only the front door (the API Gateway) should be accessible from the internet. Backend services like Catalog and Recommendations are internal only. This reduces the attack surface and matches how production systems like Netflix operate.

Internal services can still communicate with each other using short URLs like http://catalog-service because Azure Container Apps provides built-in service discovery within the same environment.

Deployment taking too long or failing?

  • The first deployment creates a Container Registry, which adds 1-2 minutes. Subsequent deploys are faster.
  • If you see a permissions error, confirm you are signed in with az login and your subscription has the Microsoft.App provider registered (this was done in Step 1).
  • If the build fails, check that the Dockerfile and package.json are both present in the service folder you passed to --source.

Help me troubleshoot my Azure Container Apps deployment

Verify service-to-service communication

All three services are now deployed. The API Gateway calls the Catalog and Recommendations services internally. Hitting the API Gateway's public URL will prove that the full chain works end-to-end.

  • Get the API Gateway's public URL by running:
az containerapp show \
  --name api-gateway \
  --resource-group netflix-resilience-rg \
  --query properties.configuration.ingress.fqdn \
  --output tsv
  • Test the catalog endpoint by running this command, replacing the URL with your actual value:
curl https://[[GATEWAY_URL="your api-gateway FQDN from the previous command"]]/catalog

You should see a JSON response with "source": "live" and the five movies listed. This confirms the API Gateway successfully called the Catalog Service internally.

  • Test the full endpoint (catalog + recommendations) by running:
curl https://[[GATEWAY_URL="your api-gateway FQDN from the previous command"]]/catalog/1/full

You should see a JSON response containing the movie details for The Matrix, along with "recommendationsSource": "live" and a list of recommended movie IDs. This proves both internal services are responding through the API Gateway.

Getting empty movies or fallback responses?

  • If "source": "fallback" appears, the API Gateway could not reach the Catalog Service. Verify the catalog-service container app is running by checking az containerapp show --name catalog-service --resource-group netflix-resilience-rg --query properties.runningStatus.
  • Make sure all three services are in the same environment (netflix-resilience-env). Internal service discovery only works within a single environment.
  • Give the services 30-60 seconds after deployment to fully start. The first request after deployment may time out while the container initializes.

Help me debug my service-to-service communication

Your three-service system is now live on Azure. The API Gateway accepts public traffic and routes it to internal services that are invisible to the outside world. Next up, you will add resiliency policies that protect these services from cascading failures.

Configure Resiliency Policies

Your three services are deployed and communicating. But what happens when the Recommendations Service slows down or starts returning errors?

Without protection, one failing service can drag down the entire system. In this step, you will add platform-level resiliency policies that implement the same patterns Netflix uses to keep streaming for 260M+ subscribers.

In this step, get ready to:
  • Apply a resiliency policy with circuit breakers, retries, and timeouts to the Recommendations Service.
  • Apply a resiliency policy with retries and timeouts to the Catalog Service.
  • Verify both policies are active and protecting your services.
Apply a resiliency policy to the Recommendations Service

The Recommendations Service is the most failure-prone part of your system. It has a built-in failure toggle that you will use to simulate outages in the next step.

You will protect it with all three resilience patterns: a circuit breaker to stop sending traffic to failing replicas, retries with exponential backoff to handle transient errors, and a timeout to prevent requests from hanging forever.

  • Create the resiliency policy for the Recommendations Service by running this command:
az containerapp resiliency create \
  --resource-group netflix-resilience-rg \
  --container-app-name recommendations-service \
  --name recommendations-resiliency \
  --cb-sequential-errors 5 \
  --cb-interval 30 \
  --cb-max-ejection 100 \
  --http-retries 3 \
  --http-delay 1000 \
  --http-interval 10000 \
  --timeout 10

What does this command do?

This single command configures three layers of protection on the Recommendations Service:

  • Circuit breaker: --cb-sequential-errors 5 means after 5 consecutive 5xx errors, the failing replica gets ejected from the load balancing pool. --cb-interval 30 sets a 30-second evaluation window between ejection sweeps. --cb-max-ejection 100 allows all replicas to be ejected if they are all failing.
  • Retry: --http-retries 3 retries failed requests up to 3 times. --http-delay 1000 sets the initial delay to 1000 milliseconds (1 second). --http-interval 10000 caps the maximum delay between retries at 10000 milliseconds (10 seconds). The backoff grows exponentially between these two bounds.
  • Timeout: --timeout 10 cuts off any request that takes longer than 10 seconds, including all retries.

Why apply resiliency to inbound requests?

Azure Container Apps resiliency policies apply to inbound requests to the container app, not outbound requests from it. When you attach a policy to the Recommendations Service, you are protecting callers (like the API Gateway) from waiting on a misbehaving Recommendations Service.

This is similar to how Netflix ejects unhealthy instances from its load balancer pool at the platform level, rather than requiring each client to implement its own circuit breaker logic.

  • You should see a JSON response confirming the policy was created, including the circuitBreakerPolicy, httpRetryPolicy, and timeoutPolicy sections with your configured values.

Seeing a 'not found' or 'extension' error?

  • Make sure you have the containerapp extension installed and up to date. Run az extension add --name containerapp --upgrade to fix this.
  • Confirm the container app name matches exactly: recommendations-service. Run az containerapp list --resource-group netflix-resilience-rg --query "[].name" -o tsv to check.

Why isn't my resiliency policy being created?

Apply a resiliency policy to the Catalog Service

The Catalog Service is more stable, but it still benefits from retry and timeout protection. If the service has a brief network hiccup, retries give it a second chance. The timeout prevents slow queries from blocking the API Gateway.

  • Create the resiliency policy for the Catalog Service by running this command:
az containerapp resiliency create \
  --resource-group netflix-resilience-rg \
  --container-app-name catalog-service \
  --name catalog-resiliency \
  --http-retries 3 \
  --http-delay 1000 \
  --http-interval 10000 \
  --timeout 10

What does this command do?

This configures retry and timeout protection for the Catalog Service. Retries with exponential backoff handle transient failures (network blips, brief overloads). The 10-second timeout prevents slow responses from cascading upstream.

No circuit breaker is needed here because the Catalog Service serves static data and is unlikely to experience sustained failures. In a real production system, you would tune each service's resiliency settings based on its failure profile.

  • You should see a JSON response confirming the policy with httpRetryPolicy and timeoutPolicy sections.

Policy creation failed?

  • Each container app can only have one resiliency policy. If you already created one and need to change it, use az containerapp resiliency update instead of create.
  • Verify the container app name matches: catalog-service.

Why is my catalog resiliency policy failing to create?

Verify policies are active

Before moving on, confirm that both policies are attached and running.

  • Check the Recommendations Service policy by running:
az containerapp resiliency show \
  --resource-group netflix-resilience-rg \
  --container-app-name recommendations-service \
  --name recommendations-resiliency
  • You should see the full JSON output showing circuitBreakerPolicy, httpRetryPolicy, and timeoutPolicy with your configured values.
  • Check the Catalog Service policy by running:
az containerapp resiliency show \
  --resource-group netflix-resilience-rg \
  --container-app-name catalog-service \
  --name catalog-resiliency
  • You should see httpRetryPolicy and timeoutPolicy in the output with your configured values.

What is the circuit breaker actually doing?

The circuit breaker in Azure Container Apps works by ejecting failing replicas from the load balancing pool. When a replica returns 5 consecutive errors, the platform removes it from the pool so new requests go to healthy replicas instead.

Every 30 seconds (your --cb-interval setting), the platform re-evaluates ejected replicas. If a replica starts responding successfully, it gets added back to the pool. This is the half-open to closed transition you will observe in the next step.

Show command returns empty or error?

  • Double-check that the --name flag matches exactly what you used during creation: recommendations-resiliency or catalog-resiliency.
  • Run az containerapp resiliency list --resource-group netflix-resilience-rg --container-app-name recommendations-service to see all policies attached to the app.

Why can't I see my resiliency policy?

Both services are now protected by platform-level resiliency policies. Next up, you will intentionally break the Recommendations Service and watch these patterns kick in to save your system.

Inject Failures and Prove Resilience

Your microservices system is deployed and protected by resiliency policies. Now comes the moment of truth.

In this step, you will deliberately break the Recommendations Service and watch the entire resilience cycle play out. The circuit breaker will detect the failures, eject the unhealthy service from the pool, and the API Gateway will seamlessly serve fallback responses. Then you will restore the service and watch it self-heal.

In this step, get ready to:
  • Trigger simulated failures on the Recommendations Service.
  • Observe the circuit breaker tripping and fallback responses being served.
  • Prove the system self-heals when the service recovers.
Trigger the Recommendations Service failure mode

The Recommendations Service has an internal-only ingress, so you cannot reach it directly from your browser or terminal. You need to connect to a container inside the same environment and make the request from there.

  • Open an interactive shell inside the api-gateway container by running this command:
az containerapp exec \
  --name api-gateway \
  --resource-group netflix-resilience-rg

What does this command do?

The az containerapp exec command opens a shell session inside a running container, similar to docker exec. Because the api-gateway container lives in the same Container Apps environment as the Recommendations Service, it can reach any internal service via the http://<app-name> address.

You should see a shell prompt appear (something like # or $). You are now inside the running container.

  • From inside the container shell, toggle failure mode on the Recommendations Service by running:
node -e "fetch('http://recommendations-service/fail', {method: 'POST'}).then(r => r.json()).then(d => { console.log(JSON.stringify(d)); process.exit() })"

What does this command do?

This uses Node.js (already available in the container) to send a POST request to the Recommendations Service's /fail endpoint. The endpoint toggles failMode to true, causing all subsequent requests to /recommendations/:movieId to return 500 errors.

  • You should see output confirming fail mode is enabled:
{"failMode":true,"message":"Fail mode ENABLED"}
  • Exit the container shell by pressing Ctrl+D.

Not seeing the expected output?

If the node -e command hangs without output, the Recommendations Service replica may have scaled to zero. Wait 10-20 seconds for it to wake up, then try again.

If you see a connection error, double-check that you typed http://recommendations-service exactly (no port number needed for internal services). Help me troubleshoot the exec session

Observe the circuit breaker in action

Now that the Recommendations Service is returning 500 errors, every request through the API Gateway will trigger the resiliency policy. The retry mechanism will attempt 3 retries per request. After 5 consecutive errors, the circuit breaker will trip and eject the failing service from the pool.

  • Send several requests to the API Gateway's full catalog endpoint to trigger the circuit breaker. Run this loop in your local terminal:
for i in $(seq 1 10); do
  echo "--- Request $i ---"
  curl -s [[GATEWAY_URL="your API Gateway URL"]]/catalog/1/full | python3 -m json.tool
  echo ""
  sleep 2
done

What is happening with each request?

Each request to /catalog/1/full makes the API Gateway call both the Catalog Service and the Recommendations Service. The Catalog Service responds normally. The Recommendations Service returns 500 errors.

The resiliency policy retries each failed request 3 times with exponential backoff. After 5 consecutive errors are detected, the circuit breaker trips and ejects the Recommendations Service replica from the load balancing pool.

Once the circuit breaker is open, the API Gateway's fetchWithFallback function catches the error and returns the hardcoded fallback response instead.

  • Watch the responses as the loop runs. In the first few requests, you should see the recommendations field taking longer (retries happening). After several requests, you should see "recommendationsSource": "fallback" appear in the JSON output.
{
  "movie": {
    "id": 1,
    "title": "The Matrix",
    "genre": "Sci-Fi",
    "year": 1999
  },
  "recommendations": {
    "movieId": 1,
    "recommendedIds": [],
    "fallback": true
  },
  "recommendationsSource": "fallback"
}

The movie data still comes through perfectly from the Catalog Service. Only the broken Recommendations Service is gracefully degraded.

Why does the system keep working?

This is graceful degradation in action. The API Gateway serves partial data (movies without recommendations) instead of failing entirely. Users still get a useful response. This is the same pattern Netflix uses. If their recommendation engine goes down, you can still browse the catalog.

Still seeing live recommendations?

If recommendationsSource still shows "live", the fail mode toggle may not have registered. Re-run the exec command from the previous substep to confirm failMode is true.

If responses are timing out entirely, the 2-second sleep between requests may be too short for the retries to complete. Increase the sleep value to 5 seconds and try again. Help me debug why I'm not seeing fallback responses

Disable failures and watch the circuit breaker recover

The circuit breaker has a 30-second evaluation interval. After that window passes with no new errors, it moves from open to half-open state. It will let a test request through. If that succeeds, it closes fully and live traffic resumes.

  • Open another interactive shell in the api-gateway container by running:
az containerapp exec \
  --name api-gateway \
  --resource-group netflix-resilience-rg
  • Toggle fail mode off on the Recommendations Service by running:
node -e "fetch('http://recommendations-service/fail', {method: 'POST'}).then(r => r.json()).then(d => { console.log(JSON.stringify(d)); process.exit() })"
  • You should see output confirming fail mode is disabled:
{"failMode":false,"message":"Fail mode DISABLED"}
  • Exit the container shell by pressing Ctrl+D.
  • Wait about 30-40 seconds for the circuit breaker's evaluation interval to pass. Then send another batch of requests from your local terminal:
for i in $(seq 1 5); do
  echo "--- Recovery request $i ---"
  curl -s [[GATEWAY_URL="your API Gateway URL"]]/catalog/1/full | python3 -m json.tool
  echo ""
  sleep 3
done
  • Watch the responses transition. The first request or two may still show "recommendationsSource": "fallback" (circuit breaker still open). Then you should see "recommendationsSource": "live" return with actual recommendation IDs:
{
  "movie": {
    "id": 1,
    "title": "The Matrix",
    "genre": "Sci-Fi",
    "year": 1999
  },
  "recommendations": {
    "movieId": 1,
    "recommendedIds": [2, 4]
  },
  "recommendationsSource": "live"
}

What just happened?

You just witnessed the full resilience cycle that production systems like Netflix rely on. The circuit breaker detected failures, ejected the unhealthy service, served degraded responses, then automatically recovered once the downstream service became healthy again.

No human intervention was needed. No restarts. No redeployments. The platform handled everything automatically based on the policies you configured.

Recommendations not coming back to live?

The circuit breaker needs time to move from open to half-open. If you are still seeing fallback responses after 30 seconds, wait another 30 seconds and try again. The evaluation interval resets with each failed request, so make sure you stopped sending requests while waiting.

If live recommendations never return, confirm the fail mode was actually toggled off by running the exec and POST command again. You should see failMode: false in the response. Help me debug circuit breaker recovery

You have just proven that your microservices system can detect failures, protect itself from cascading outages, and self-heal automatically. These are the exact patterns that keep services like Netflix streaming for 260M+ subscribers.

Secret mission

Add Auto-Scaling and Load Test

Your services are resilient. But can they handle a traffic surge? In this secret mission, you'll configure HTTP-based auto-scaling on the Catalog Service so it scales from 0 to 5 replicas based on concurrent request count. Then you'll unleash a load test and watch replicas spin up in real time.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. Your Container Apps are on the consumption plan and scale to zero when idle, so no usage charges apply while the services are not receiving traffic.

Resources you used:

  • Azure Resource Group netflix-resilience-rg (contains: Container Apps Environment, 3 Container Apps, 1 Container Registry).
  • Azure Container Apps CLI extension (containerapp).
  • Local project files (api-gateway, catalog-service, recommendations-service directories).

Keep everything running

No action needed. Choose this if you're still actively experimenting with the services or plan to build on them.

  • Your Container Apps scale to zero automatically when idle. No charges apply while scaled to zero.
  • The free tier includes 180,000 vCPU-seconds and 2,000,000 requests per month. Casual testing stays well within these limits.
  • The containerapp CLI extension remains installed on your machine for future projects.

Pause - I'll come back to this later

Your services already scale to zero when not receiving traffic. There is no separate pause action needed for Azure Container Apps on the consumption plan.

  • Leave the Azure resources as-is. They cost nothing while idle.
  • Keep your local project files so you can redeploy or modify the code later.

Delete - I don't want to use this again

Remove all project resources and start fresh. Deleting the resource group removes everything inside it in one command.

  • Delete the entire Azure Resource Group (this removes the Container Apps Environment, all three Container Apps, and the auto-created Container Registry) by running:
az group delete --name netflix-resilience-rg --yes --no-wait

What does this command do?

The --yes flag skips the confirmation prompt. The --no-wait flag returns control to your terminal immediately while Azure deletes resources in the background. Full deletion takes a few minutes.

  • Remove the Azure Container Apps CLI extension by running:
az extension remove --name containerapp
  • Delete your local project files. On macOS or Linux, run:
cd ~/Desktop
rm -rf api-gateway catalog-service recommendations-service
  • On Windows, run:
cd ~\Desktop
rmdir /s /q api-gateway catalog-service recommendations-service
  • Confirm the deletion worked by running:
az group list --query "[?name=='netflix-resilience-rg']" --output table

If the group is still deleting, you may see it listed with a state of Deleting. Once fully removed, the command returns an empty table.

Nice Work!

Nice Work!

You just built a production-style microservices system that self-heals under failure. These are the same resilience patterns that keep Netflix streaming for 260M+ subscribers.

You've learned how to:

  • Built 3 Node.js microservices that communicate via internal service discovery and deployed them to Azure Container Apps with automatic containerization from source.
  • Configured platform-level circuit breakers, retries with exponential backoff, and timeouts as resiliency policies protecting your services from cascading failures.
  • Proved resilience by injecting failures into the Recommendations Service and watching the circuit breaker trip, fallback responses serve, and the system self-heal when the service recovered.
  • Secret Mission: Configured HTTP-based auto-scaling rules and load-tested the Catalog Service to observe horizontal scaling from 0 to 5 replicas in real time.

Ready to quiz yourself?