Monitor a Multi-Service App with Datadog

Instrument a Flask and Redis stack with Datadog APM, logs, and custom metrics.

Introduction

30 Second Summary

Imagine your app goes down at 2 AM and you don't find out until angry users flood your inbox. In the real world, DevOps engineers set up monitoring systems that catch problems before users even notice.

In this project, you will deploy a containerized Python application and instrument it with Datadog to collect metrics, traces, and logs. You'll build a custom dashboard and define reliability targets that mirror how production teams track application health.

What You'll Build

A multi-container Docker Compose stack with a Flask API and Redis backend, fully monitored by a Datadog Agent collecting infrastructure metrics, APM traces, container logs, and custom business metrics.

By the end of this project, you'll have:

  • A Docker Compose stack with three services (Flask app, Redis, Datadog Agent) sending infrastructure metrics, APM traces, and logs to Datadog.
  • A custom Datadog dashboard correlating metrics, traces, logs, and custom DogStatsD metrics in a single view.
  • A Service Level Objective with an error budget tracking your application's latency target.
  • Secret Mission: Add a Synthetic API Test to monitor your /health endpoint and trigger an alert when it fails.

Are there any prerequisites?

This project assumes you have Docker Desktop, Python, and VS Code already installed on your machine. You should also be comfortable with basic Docker concepts (images, containers) and basic Python syntax.

Before We Start

When a containerized application breaks in production, teams need more than gut instinct to find the problem. They need metrics showing what happened, traces showing where it happened, and logs showing why it happened.

In this step, you'll create a free Datadog account and set up a project directory with the credentials your monitoring stack will need.

In this step, get ready to:
  • Sign up for a Datadog 14-day free trial.
  • Copy your API key from Datadog.
  • Create a project directory with a .env file storing your credentials.
Sign up for a Datadog free trial

Datadog offers a 14-day free trial with full access to infrastructure monitoring, APM, logs, and more. No credit card is required.

  • Open your browser and navigate to:
  • https://www.datadoghq.com/free-datadog-trial/
  • Click Get Started Free.
  • Fill out the sign-up form with your name, email, and a password.
  • When asked to choose a stack or region, accept the defaults (US region is fine for this project).
  • Complete the sign-up and wait for the Datadog dashboard to load.

What is Datadog?

Datadog is an observability platform that unifies metrics, traces, and logs into a single view. It collects data from your infrastructure and applications so you can monitor performance, detect issues, and troubleshoot faster.

Copy your API key

The Datadog Agent uses an API key to authenticate and send data to your Datadog account. Every piece of monitoring data flows through this key.

  • In the Datadog dashboard, click your account avatar in the bottom-left corner of the navigation.
  • Select Organization Settings.
  • Click the API Keys tab.
  • You should see a default API key already created for your organization. Click it to reveal the full key value.
  • Click the copy icon to copy the key to your clipboard.

Why do I need an API key?

The API key identifies your organization. The Datadog Agent running in your Docker stack sends all metrics, traces, and logs to Datadog using this key. Without it, the Agent cannot authenticate.

Keep this key private

Never commit your API key to version control. In this project you'll store it in a .env file that stays on your local machine.

Create your project directory and configuration

Your project needs a dedicated folder to hold the Flask app, Docker configuration, and environment variables.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir datadog-monitoring
cd datadog-monitoring
  • Confirm the folder was created by running:
ls

You should see the folder is empty, which is expected. Next, you'll add the environment file.

  • Create the .env file by running:
touch .env
  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Visual Studio Code and press Enter to open it.
  • In VS Code, click File in the top menu bar.
  • Click Open Folder.
  • Navigate to your Desktop, select the datadog-monitoring folder, and click Open.
  • In the Explorer sidebar on the left, click .env to open it.
  • Paste the following into .env:
DD_API_KEY=your-api-key-here
DD_SITE=datadoghq.com
  • Replace your-api-key-here with the API key you copied from Datadog earlier.
  • Save the file with Cmd+S (macOS) or Ctrl+S (Windows).

What does this file do?

The .env file stores environment variables that Docker Compose injects into your containers at runtime.

DD_API_KEY authenticates the Datadog Agent. DD_SITE tells the Agent which Datadog region to send data to. The default datadoghq.com is the US1 region.

✔️ Awesome, I've got everything!

Great. Double check you've saved your .env file with your real API key in place of the placeholder.

ⓧ I'd like to double check the full code

Here is what your .env file should look like (with your own API key replacing the placeholder):

DD_API_KEY=your-api-key-here
DD_SITE=datadoghq.com

Can't find your API key?

Make sure you navigated to Organization Settings (not Personal Settings). The API Keys tab is under the Access section on the Organization Settings page.

If you don't see any keys listed, click New Key to create one. Name it something like datadog-monitoring-project and click Create API key.

Help me find my Datadog API key

Your Datadog account is ready and your project directory is set up. Next up, you'll build the Flask API that generates the metrics, traces, and logs for Datadog to collect.

Build the Flask API with Datadog Instrumentation

In the last step, you created your Datadog account and stored your API key. Now it's time to build the application you're going to monitor.

You'll create a Python Flask API with a Redis backend, instrumented from the start with Datadog's tracing library and custom business metrics. Every request your app handles will generate observable data that flows straight into Datadog.

In this step, get ready to:
  • Pin the Python dependencies for Flask, Redis, ddtrace, and DogStatsD.
  • Configure trace-correlated logging so your logs link directly to APM traces.
  • Build four API endpoints instrumented with custom metrics and distributed tracing.
Pin your project dependencies

Your Flask app relies on four key libraries. Pinning exact versions ensures a reproducible build when you containerize the app in the next step.

  • In VS Code, create a new file called requirements.txt in your project folder.
  • Add the following dependency list:
flask==3.1.3
redis==7.3.0
ddtrace==4.8.2
datadog==0.52.1

What do these libraries do?

  • flask is the web framework for your API.
  • redis is the Python client for talking to your Redis database.
  • ddtrace is Datadog's APM tracing library. It automatically instruments Flask and Redis calls to produce distributed traces.
  • datadog provides the DogStatsD client for sending custom business metrics (counts, gauges, histograms) to the Datadog Agent over UDP.
  • Save requirements.txt.
  • Confirm the file exists by running:
ls

You should see requirements.txt alongside your existing .env file in the output.

Don't see requirements.txt?

  • Make sure you saved the file (Ctrl+S or Cmd+S).
  • Confirm the file is saved in the same folder where your .env lives. Check the VS Code Explorer sidebar to see the file tree.

Why isn't my requirements.txt showing up in the terminal?

Set up the app foundation with tracing and logging

Before writing any endpoints, you need to set up the core plumbing. This includes initializing the DogStatsD client, configuring log formatting with trace correlation fields, and connecting to Redis.

  • Create a new file called app.py in your project folder.
  • Add the following setup code:
import logging
import time
import random

from flask import Flask, jsonify, request
from ddtrace import tracer
from datadog import initialize, statsd
import redis

# Initialize DogStatsD client
initialize(statsd_host="datadog-agent", statsd_port=8125)

# Configure logging with Datadog trace correlation
FORMAT = (
    "%(asctime)s %(levelname)s [%(name)s] [%(filename)s:%(lineno)d] "
    "[dd.service=%(dd.service)s dd.env=%(dd.env)s "
    "dd.version=%(dd.version)s dd.trace_id=%(dd.trace_id)s "
    "dd.span_id=%(dd.span_id)s] - %(message)s"
)
logging.basicConfig(format=FORMAT, level=logging.INFO)
log = logging.getLogger(__name__)

app = Flask(__name__)

# Connect to Redis
redis_client = redis.Redis(host="redis", port=6379, decode_responses=True)

What does this code do?

  • initialize(statsd_host="datadog-agent", statsd_port=8125) tells the DogStatsD client to send custom metrics to the Datadog Agent container over UDP port 8125.
  • The FORMAT string includes dd.trace_id and dd.span_id fields. These let Datadog correlate each log line with the exact trace that produced it.
  • The Redis client uses host="redis" because Docker Compose networking resolves container names as hostnames. This will connect once the full stack is running.

The hostnames datadog-agent and redis only resolve inside Docker Compose. That's expected. You'll launch the containers in a later step.

You won't see a change yet. The next section adds the endpoints that make this app do something.

Add the instrumented API endpoints

Each endpoint records a custom metric via DogStatsD, so you can track exactly how your API is used. The /slow endpoint also creates a custom APM span so you can trace simulated latency.

  • In app.py, add the following route definitions below the redis_client line:

@app.route("/")
def home():
    statsd.increment("flask_app.request.count", tags=["endpoint:home"])
    log.info("Home endpoint called")
    return jsonify({"message": "Flask App with Datadog Observability", "status": "ok"})


@app.route("/health")
def health():
    statsd.increment("flask_app.request.count", tags=["endpoint:health"])
    try:
        redis_client.ping()
        return jsonify({"status": "healthy", "redis": "connected"}), 200
    except redis.ConnectionError:
        log.error("Redis connection failed")
        return jsonify({"status": "unhealthy", "redis": "disconnected"}), 503


@app.route("/items", methods=["GET"])
def get_items():
    statsd.increment("flask_app.request.count", tags=["endpoint:items", "method:get"])
    log.info("Fetching all items from Redis")
    keys = redis_client.keys("item:*")
    items = {}
    for key in keys:
        items[key] = redis_client.get(key)
    statsd.gauge("flask_app.items.total", len(items))
    return jsonify({"items": items, "count": len(items)})


@app.route("/items", methods=["POST"])
def create_item():
    statsd.increment("flask_app.request.count", tags=["endpoint:items", "method:post"])
    data = request.get_json()
    item_id = f"item:{random.randint(1000, 9999)}"
    redis_client.set(item_id, data.get("value", "default"))
    log.info(f"Created item {item_id}")
    statsd.increment("flask_app.items.created")
    return jsonify({"id": item_id, "value": data.get("value", "default")}), 201


@app.route("/slow")
def slow_endpoint():
    statsd.increment("flask_app.request.count", tags=["endpoint:slow"])
    with tracer.trace("slow_operation", service="flask-app", resource="simulate_delay") as span:
        delay = random.uniform(0.5, 2.0)
        span.set_tag("delay_seconds", delay)
        log.warning(f"Slow endpoint called, simulating {delay:.2f}s delay")
        time.sleep(delay)
    statsd.histogram("flask_app.request.duration", delay, tags=["endpoint:slow"])
    return jsonify({"message": "slow response", "delay": delay})


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

What do these endpoints do?

  • / is a simple home route. It increments a request counter each time it's called.
  • /health pings Redis to check connectivity. This is the endpoint you'd use for container health checks.
  • /items (GET) fetches all stored items and reports the total count as a DogStatsD gauge metric.
  • /items (POST) creates a new item in Redis and increments the flask_app.items.created counter.
  • /slow simulates variable latency (0.5 to 2 seconds). It uses tracer.trace() to create a custom APM span and records the delay as a histogram metric.

Every endpoint calls statsd.increment() with an endpoint tag. This lets you filter request counts per route in your Datadog dashboard later.

Why custom metrics on top of auto-instrumentation?

The ddtrace library auto-instruments Flask to produce APM traces. But custom DogStatsD metrics let you track business-specific signals that auto-instrumentation can't infer.

For example, knowing how many items are stored in Redis (flask_app.items.total) is a business metric. Knowing the p90 latency of your slow endpoint (flask_app.request.duration) is an operational metric. Both are invisible without explicit instrumentation.

  • Save app.py.

You won't be able to run this app locally because it depends on the redis and datadog-agent containers. In the next step, you'll containerize this app and launch the full stack with Docker Compose.

Seeing a syntax error in VS Code?

  • Make sure there's an empty blank line between the redis_client line and the first @app.route decorator.
  • Check that f-strings use the f"...{variable}" syntax with a lowercase f and matching quotes.
  • Verify you didn't accidentally paste the endpoint code inside the FORMAT string. The routes should start at the top indentation level.

Why is VS Code showing an error in my app.py?

✔️ Awesome, I've got everything!

Great. Double check you've saved both requirements.txt and app.py.

ⓧ I'd like to double check the full code

Here's what your complete app.py should look like:

import logging
import time
import random

from flask import Flask, jsonify, request
from ddtrace import tracer
from datadog import initialize, statsd
import redis

# Initialize DogStatsD client
initialize(statsd_host="datadog-agent", statsd_port=8125)

# Configure logging with Datadog trace correlation
FORMAT = (
    "%(asctime)s %(levelname)s [%(name)s] [%(filename)s:%(lineno)d] "
    "[dd.service=%(dd.service)s dd.env=%(dd.env)s "
    "dd.version=%(dd.version)s dd.trace_id=%(dd.trace_id)s "
    "dd.span_id=%(dd.span_id)s] - %(message)s"
)
logging.basicConfig(format=FORMAT, level=logging.INFO)
log = logging.getLogger(__name__)

app = Flask(__name__)

# Connect to Redis
redis_client = redis.Redis(host="redis", port=6379, decode_responses=True)


@app.route("/")
def home():
    statsd.increment("flask_app.request.count", tags=["endpoint:home"])
    log.info("Home endpoint called")
    return jsonify({"message": "Flask App with Datadog Observability", "status": "ok"})


@app.route("/health")
def health():
    statsd.increment("flask_app.request.count", tags=["endpoint:health"])
    try:
        redis_client.ping()
        return jsonify({"status": "healthy", "redis": "connected"}), 200
    except redis.ConnectionError:
        log.error("Redis connection failed")
        return jsonify({"status": "unhealthy", "redis": "disconnected"}), 503


@app.route("/items", methods=["GET"])
def get_items():
    statsd.increment("flask_app.request.count", tags=["endpoint:items", "method:get"])
    log.info("Fetching all items from Redis")
    keys = redis_client.keys("item:*")
    items = {}
    for key in keys:
        items[key] = redis_client.get(key)
    statsd.gauge("flask_app.items.total", len(items))
    return jsonify({"items": items, "count": len(items)})


@app.route("/items", methods=["POST"])
def create_item():
    statsd.increment("flask_app.request.count", tags=["endpoint:items", "method:post"])
    data = request.get_json()
    item_id = f"item:{random.randint(1000, 9999)}"
    redis_client.set(item_id, data.get("value", "default"))
    log.info(f"Created item {item_id}")
    statsd.increment("flask_app.items.created")
    return jsonify({"id": item_id, "value": data.get("value", "default")}), 201


@app.route("/slow")
def slow_endpoint():
    statsd.increment("flask_app.request.count", tags=["endpoint:slow"])
    with tracer.trace("slow_operation", service="flask-app", resource="simulate_delay") as span:
        delay = random.uniform(0.5, 2.0)
        span.set_tag("delay_seconds", delay)
        log.warning(f"Slow endpoint called, simulating {delay:.2f}s delay")
        time.sleep(delay)
    statsd.histogram("flask_app.request.duration", delay, tags=["endpoint:slow"])
    return jsonify({"message": "slow response", "delay": delay})


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

Your application code is ready. Next up, you'll wrap it in a Dockerfile and wire up the full three-container stack with Docker Compose.

Create the Dockerfile and Docker Compose Stack

Your Flask app is fully instrumented with APM traces, custom DogStatsD metrics, and log correlation. Now you need to package it into a container and orchestrate all the services together.

In this step, you'll create a Dockerfile that bakes Unified Service Tagging directly into your image. Then you'll write a Docker Compose file that spins up the Flask app, Redis, and the Datadog Agent as a single coordinated stack.

In this step, get ready to:
  • Create a Dockerfile with Unified Service Tags and Datadog labels.
  • Write a Docker Compose file defining the Flask app, Redis, and Datadog Agent services.
  • Configure the Datadog Agent to collect APM traces, logs, and custom metrics from your containers.
Create the Dockerfile with Unified Service Tags

Unified Service Tagging is a Datadog best practice that ties your metrics, traces, and logs together through three tags: DD_SERVICE, DD_ENV, and DD_VERSION. You set these as both environment variables (for the tracing library inside the container) and Docker labels (for the Agent reading container metadata from the outside).

  • In VS Code, create a new file called Dockerfile in your project directory (no file extension).
  • Paste the following code into it:
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

ENV DD_SERVICE="flask-app"
ENV DD_ENV="dev"
ENV DD_VERSION="1.0.0"

LABEL com.datadoghq.tags.service="flask-app"
LABEL com.datadoghq.tags.env="dev"
LABEL com.datadoghq.tags.version="1.0.0"

EXPOSE 5000

CMD ["ddtrace-run", "python", "app.py"]

What does this code do?

  • FROM python:3.11-slim uses a lightweight Python base image.
  • COPY requirements.txt and RUN pip install install dependencies first so Docker caches this layer separately from your app code.
  • The three ENV lines set Unified Service Tags so ddtrace knows which service, environment, and version to attach to all telemetry.
  • The three LABEL lines duplicate those tags as Docker labels so the Datadog Agent can read them from the container metadata externally.
  • CMD ["ddtrace-run", "python", "app.py"] wraps your app startup with the tracing library, enabling automatic instrumentation of Flask and Redis.

Why both ENV and LABEL?

The ENV variables are read by ddtrace inside the running process. The LABEL values are read by the Datadog Agent from the Docker API, outside the container. Setting both ensures metrics, traces, and logs all carry the same service, environment, and version tags regardless of where they originate.

  • Save Dockerfile.

✔️ Awesome, I've got everything!

Great. Make sure you've saved the file before moving on.

ⓧ I'd like to double check the full code

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

ENV DD_SERVICE="flask-app"
ENV DD_ENV="dev"
ENV DD_VERSION="1.0.0"

LABEL com.datadoghq.tags.service="flask-app"
LABEL com.datadoghq.tags.env="dev"
LABEL com.datadoghq.tags.version="1.0.0"

EXPOSE 5000

CMD ["ddtrace-run", "python", "app.py"]
Write docker-compose.yml with the Flask app and Redis services

Docker Compose lets you define and run multiple containers as a single stack. You'll define the Flask app and Redis first, then add the Datadog Agent in the next substep.

  • Create a new file called docker-compose.yml in your project directory.
  • Paste the following code into it:
services:
  flask-app:
    build: .
    container_name: flask-app
    ports:
      - "5000:5000"
    environment:
      - DD_AGENT_HOST=datadog-agent
      - DD_TRACE_AGENT_PORT=8126
      - DD_SERVICE=flask-app
      - DD_ENV=dev
      - DD_VERSION=1.0.0
    labels:
      com.datadoghq.tags.service: "flask-app"
      com.datadoghq.tags.env: "dev"
      com.datadoghq.tags.version: "1.0.0"
      com.datadoghq.ad.logs: '[{"source": "python", "service": "flask-app"}]'
    depends_on:
      - redis
      - datadog-agent
    networks:
      - monitoring

  redis:
    image: redis:7-alpine
    container_name: redis
    ports:
      - "6379:6379"
    labels:
      com.datadoghq.tags.service: "redis"
      com.datadoghq.tags.env: "dev"
      com.datadoghq.ad.logs: '[{"source": "redis", "service": "redis"}]'
    networks:
      - monitoring

What does this code do?

  • build: . tells Docker Compose to build the Flask app image from the Dockerfile in the current directory.
  • DD_AGENT_HOST=datadog-agent points the ddtrace library to the Agent container by its Docker network hostname.
  • The labels section repeats Unified Service Tags for the Agent to read, plus com.datadoghq.ad.logs tells the Agent how to tag logs from this container. This is Autodiscovery in action.
  • depends_on ensures Redis and the Agent start before the Flask app.
  • Redis uses the redis:7-alpine image and gets its own Datadog labels for log collection.

You won't see a change yet. The next substep adds the Datadog Agent service to complete this file.

Configure the Datadog Agent service

The Datadog Agent is the central collector that receives traces, metrics, and logs from your other containers and forwards them to the Datadog platform. It runs as a sidecar container in your stack.

  • In docker-compose.yml, add the following code below the redis service (at the same indentation level as flask-app and redis):
  datadog-agent:
    image: gcr.io/datadoghq/agent:7
    container_name: datadog-agent
    pid: host
    environment:
      - DD_API_KEY=${DD_API_KEY}
      - DD_SITE=${DD_SITE}
      - DD_APM_ENABLED=true
      - DD_APM_NON_LOCAL_TRAFFIC=true
      - DD_LOGS_ENABLED=true
      - DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true
      - DD_PROCESS_AGENT_ENABLED=true
      - DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true
      - DD_CONTAINER_EXCLUDE="name:datadog-agent"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    ports:
      - "8126:8126"
      - "8125:8125/udp"
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

What does this configuration do?

  • gcr.io/datadoghq/agent:7 is the official Datadog Agent Docker image.
  • pid: host gives the Agent visibility into host processes for live process monitoring.
  • DD_API_KEY=${DD_API_KEY} reads your API key from the .env file. Docker Compose automatically loads .env variables.
  • DD_APM_ENABLED and DD_APM_NON_LOCAL_TRAFFIC enable trace collection from other containers in the stack.
  • DD_LOGS_ENABLED and DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL enable log collection from every container.
  • DD_DOGSTATSD_NON_LOCAL_TRAFFIC allows the Agent to receive custom metrics from the Flask app over UDP port 8125.
  • DD_CONTAINER_EXCLUDE prevents the Agent from monitoring its own container.
  • The volume mounts give the Agent read-only access to the Docker socket, process info, cgroup data, and container log files.
  • Port 8126 receives APM traces. Port 8125/udp receives DogStatsD custom metrics.
  • The networks section at the bottom creates a shared bridge network so all three containers can communicate by hostname.
  • Save docker-compose.yml.
  • Validate the file has no YAML errors by running this command in your terminal:
docker compose config -q
  • If no output appears and you return to the prompt, your file is valid.

Seeing a YAML error?

  • YAML is whitespace-sensitive. Make sure you used spaces (not tabs) and each service is indented exactly 2 spaces under services:.
  • Check that the networks: block at the bottom is at the root level (no leading spaces), not nested inside a service.
  • Confirm your .env file exists in the same directory as docker-compose.yml.

Help me fix a YAML syntax error in my docker-compose.yml file.

✔️ Awesome, I've got everything!

Great. Double check you've saved docker-compose.yml before moving on.

ⓧ I'd like to double check the full code

services:
  flask-app:
    build: .
    container_name: flask-app
    ports:
      - "5000:5000"
    environment:
      - DD_AGENT_HOST=datadog-agent
      - DD_TRACE_AGENT_PORT=8126
      - DD_SERVICE=flask-app
      - DD_ENV=dev
      - DD_VERSION=1.0.0
    labels:
      com.datadoghq.tags.service: "flask-app"
      com.datadoghq.tags.env: "dev"
      com.datadoghq.tags.version: "1.0.0"
      com.datadoghq.ad.logs: '[{"source": "python", "service": "flask-app"}]'
    depends_on:
      - redis
      - datadog-agent
    networks:
      - monitoring

  redis:
    image: redis:7-alpine
    container_name: redis
    ports:
      - "6379:6379"
    labels:
      com.datadoghq.tags.service: "redis"
      com.datadoghq.tags.env: "dev"
      com.datadoghq.ad.logs: '[{"source": "redis", "service": "redis"}]'
    networks:
      - monitoring

  datadog-agent:
    image: gcr.io/datadoghq/agent:7
    container_name: datadog-agent
    pid: host
    environment:
      - DD_API_KEY=${DD_API_KEY}
      - DD_SITE=${DD_SITE}
      - DD_APM_ENABLED=true
      - DD_APM_NON_LOCAL_TRAFFIC=true
      - DD_LOGS_ENABLED=true
      - DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true
      - DD_PROCESS_AGENT_ENABLED=true
      - DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true
      - DD_CONTAINER_EXCLUDE="name:datadog-agent"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    ports:
      - "8126:8126"
      - "8125:8125/udp"
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

Your entire stack is defined and ready to launch. Next up, you'll spin up all three containers and watch the telemetry flow into Datadog.

Launch the Stack and Explore Datadog

Your code is written, your Dockerfile is built, and your Compose stack is configured. Now it's time to bring everything to life and see the three pillars of observability in action.

In this step, you'll launch all three containers, generate some traffic to produce traces and metrics, and then explore Datadog's UI to verify that infrastructure metrics, APM traces, and correlated logs are all flowing in.

In this step, get ready to:
  • Launch the Docker Compose stack and verify all three containers are healthy.
  • Generate traffic and explore APM traces and the Service Map in Datadog.
  • Confirm log entries are correlated with trace IDs.
Launch the stack and verify containers

Docker Compose will build the Flask image from your Dockerfile, pull the Redis and Datadog Agent images, and start all three containers on the same network.

  • Switch back to your terminal in the project directory and launch the stack by running this command:
docker compose up -d --build

What does this command do?

The --build flag forces Docker to rebuild the Flask app image so any code changes are included. The -d flag runs all containers in detached mode so your terminal stays free.

Docker Compose creates a monitoring network and starts redis and datadog-agent first (because flask-app depends on them), then builds and starts the Flask container.

  • The first build takes 1-2 minutes while Docker pulls base images and installs Python dependencies. Verify all containers are running by checking their status:
docker compose ps

You should see three containers listed with a status of Up: flask-app, redis, and datadog-agent.

Seeing a container exit or error?

  • If datadog-agent exits, check your .env file. The DD_API_KEY must be your actual Datadog API key, not the placeholder text.
  • If flask-app fails to start, run docker compose logs flask-app to see the error output. A common cause is a typo in app.py or requirements.txt.
  • Make sure Docker Desktop is running. Without the Docker daemon active, Compose cannot start containers.

Having trouble getting all containers running?

  • Now generate some traffic so Datadog has traces, metrics, and logs to display. Send a few requests to each endpoint by running these commands:
curl http://localhost:5000/
curl http://localhost:5000/health
curl http://localhost:5000/items
curl -X POST http://localhost:5000/items -H "Content-Type: application/json" -d '{"value": "test-item"}'
curl http://localhost:5000/slow

Why send requests before checking Datadog?

The Datadog Agent takes a minute or two to register with the platform after starting. These requests ensure traces, custom metrics, and log entries exist in the pipeline by the time you open the Datadog UI.

The /slow endpoint introduces a random delay between 0.5 and 2 seconds. This simulated latency gives you interesting trace data to explore later.

You should see JSON responses from each endpoint. The /health endpoint returns {"status": "healthy", "redis": "connected"} confirming Redis connectivity.

Wait 2-3 minutes before exploring Datadog

The Agent needs a short time to forward metrics, traces, and logs to the Datadog platform. Give it 2-3 minutes after starting the stack before navigating the UI.

Explore infrastructure metrics and APM traces

With traffic flowing, your Datadog account is now receiving data from three sources: infrastructure metrics from the Agent, APM traces from ddtrace, and custom metrics from DogStatsD.

  • Open your browser and navigate to your Datadog account at app.datadoghq.com.
  • In the left sidebar, click Infrastructure.
  • Click Hosts to open the Infrastructure List.
  • You should see your host listed with container metrics (CPU, memory, network). Click on the host to expand details and confirm you see three containers attached.

What is the Infrastructure List?

The Infrastructure List shows every host reporting to Datadog via the Agent. Because your Agent container has access to the Docker socket, it automatically discovers and reports metrics for all containers in the Compose stack.

This is Autodiscovery in action. The Agent detects new containers and begins collecting their metrics without any manual configuration.

  • Now navigate to APM. In the left sidebar, click APM.
  • Click Services to view the service catalog.
  • Look for a service named flask-app in the list. Click on it to see performance metrics like latency, throughput, and error rates.

What is Unified Service Tagging?

The flask-app service name appears because you set DD_SERVICE, DD_ENV, and DD_VERSION in your Dockerfile and Compose file. These three tags are what Datadog calls Unified Service Tagging.

They connect metrics, traces, and logs together so you can pivot between them using the same service identity. This is a Datadog best practice that interviewers specifically ask about for DevOps roles.

  • Next, view the Service Map. In the left sidebar under APM, click Service Map.
  • You should see a visual graph showing flask-app connected to redis with an arrow indicating the dependency direction.

How does the Service Map know about Redis?

The ddtrace library automatically instruments common libraries, including the redis Python package. Every time your Flask app calls Redis, ddtrace records a span for that call. Datadog uses these spans to build the dependency graph automatically.

No additional configuration was needed. This is the power of automatic instrumentation combined with Unified Service Tagging.

Verify log correlation with trace IDs

The final piece of the observability puzzle is verifying that your application logs contain trace IDs. This lets you jump from a trace directly to its related logs, and vice versa.

  • In the left sidebar of Datadog, click Logs to open the Log Explorer.
  • In the search bar, type service:flask-app and press Enter to filter logs to your Flask application.
  • You should see log entries from your recent requests. Click on any log entry to expand its side panel.
  • In the expanded log, look for the dd.trace_id and dd.span_id fields. These prove that your logs are correlated with APM traces.

What is log-trace correlation?

When ddtrace processes a request, it injects the active trace ID and span ID into your Python log records. Your log format string includes dd.trace_id and dd.span_id placeholders, so every log line carries the ID of the trace that produced it.

This means you can click a slow trace in APM and immediately see every log line that happened during that request. In production, this saves hours of debugging time.

  • To see correlation in action, go back to APM and click Traces.
  • Click on any flask-app trace to open the trace detail view.
  • In the trace detail panel, click the Logs tab. You should see the log entries that were emitted during that specific request, linked by trace ID.

Not seeing logs in the trace panel?

  • Wait 2-3 more minutes. Logs can take slightly longer than traces to appear in the Datadog UI.
  • Make sure the com.datadoghq.ad.logs label is present on your flask-app service in docker-compose.yml. This tells the Agent to collect logs from that container.
  • Verify your .env file has DD_SITE=datadoghq.com set correctly for your Datadog region.

Why aren't my correlated logs showing up in the trace panel?

Your observability stack is fully operational. All three pillars are flowing into Datadog: infrastructure metrics from the Agent, distributed traces from ddtrace, and correlated logs tied together by trace IDs. Next up, you'll build a custom dashboard to visualize all of this data in a single view.

Build a Custom Dashboard

In the previous step, you explored the individual pillars of observability across Datadog's platform. Infrastructure metrics, APM traces, and logs each live in their own section.

The real power of a monitoring platform comes from seeing all of these signals together on a single screen. In this step, you will build a custom Datadog dashboard that combines container metrics, APM data, custom DogStatsD metrics, and a live log stream into one unified view.

In this step, get ready to:
  • Create a new dashboard named "Flask App Observability".
  • Add widgets for infrastructure, APM, and custom metrics.
  • Configure template variables to filter the entire dashboard by environment and service tags.
Create the dashboard

Every observability setup needs a single pane of glass. Your dashboard will pull data from multiple Datadog sources into one view.

  • In the Datadog sidebar, hover over Dashboards and click Dashboard List.
  • Click + New Dashboard in the top-right corner.
  • Enter Flask App Observability as the dashboard name.
  • Select Dashboard as the layout type (the grid-based option).
  • Click New Dashboard to confirm.

Your empty dashboard opens in edit mode, ready for widgets.

Add infrastructure and APM widgets

You will add six widgets covering container health, request throughput, latency, errors, and your custom business metrics. Each widget pulls from a different Datadog data source.

  • Click Add Widget to open the widget tray.
  • Drag a Timeseries widget onto the dashboard canvas.
  • In the widget editor, set the metric to docker.cpu.usage and filter with container_name:flask-app in the from field.
  • Title the widget Container CPU Usage and click Done.
  • Add a second Timeseries widget. Set the metric to docker.mem.rss, filter by container_name:flask-app, title it Container Memory Usage, and click Done.
  • Add another Timeseries widget. Change the data source to APM. Select the flask-app service, set the measure to Request count, and group by resource_name. Title it APM Request Count by Endpoint and click Done.
  • Add another Timeseries APM widget. Select the flask-app service, set the measure to p90 latency. Title it p90 Latency and click Done.
  • Add another Timeseries APM widget. Select the flask-app service, set the measure to Error rate. Title it Error Rate and click Done.

What do these widgets show?

The CPU and Memory timeseries show container-level infrastructure health. If your Flask app starts consuming unexpected resources, you will spot it here first.

The APM widgets show request-level performance. Request count tells you throughput, p90 latency shows how slow the worst 10% of requests are, and error rate reveals how often requests fail.

  • Add a Timeseries widget. Set the data source to Metrics and search for flask_app.items.total. Title it Items in Redis (Gauge) and click Done.
  • Add another Timeseries widget. Search for the metric flask_app.request.duration and filter by endpoint:slow. Title it Request Duration Histogram (Slow Endpoint) and click Done.
  • Add a Log Stream widget from the widget tray. In the filter field, type service:flask-app. Title it Flask App Logs and click Done.

Why custom DogStatsD metrics on your dashboard?

The flask_app.items.total gauge and flask_app.request.duration histogram are the business metrics you defined in app.py using DogStatsD. They go beyond what auto-instrumentation provides.

Having infrastructure metrics, APM traces, custom metrics, and logs on one dashboard lets you correlate symptoms across layers. A spike in p90 latency alongside a jump in Redis item count tells a very different story than latency alone.

  • Verify that all widgets display data. If a custom metric widget shows "No data", generate some traffic by hitting your endpoints.

Generate traffic quickly

Open a new terminal and run a few curl commands against your Flask app to populate the dashboard widgets with fresh data.

curl http://localhost:5000/
curl http://localhost:5000/items
curl -X POST http://localhost:5000/items -H "Content-Type: application/json" -d '{"value": "test-item"}'
curl http://localhost:5000/slow

Widget showing No Data?

Make sure your Docker Compose stack is still running. Custom DogStatsD metrics only appear after the app has received at least one request to the relevant endpoint.

If APM widgets are empty, confirm the flask-app service is visible under APM > Services. It can take 1-2 minutes for new data to propagate to dashboard widgets.

Need more help? Why is my dashboard widget showing No Data even though my containers are running?

Add template variables for filtering

Template variables let anyone viewing your dashboard filter every widget at once using dropdown menus. This is where Unified Service Tagging pays off. Because you set DD_ENV, DD_SERVICE, and DD_VERSION on your containers, these tags are available as filter dimensions across all your data.

  • At the top of your dashboard, click Add Variable.
  • Select the tag env.
  • Leave the default value as * (shows all environments).
  • Click Done.
  • Click Add Variable again.
  • Select the tag service.
  • Leave the default value as *.
  • Click Done.

You should now see two dropdown menus in the dashboard header labeled $env and $service.

  • Test the $service variable by selecting flask-app from the dropdown. All widgets should scope down to only show data tagged with service:flask-app.
  • Set both variables back to * to see all data again.
  • Click Save in the top-right corner to save your dashboard.

Why template variables matter in production

In a real environment, you might have dev, staging, and prod environments sharing the same dashboard layout. Template variables let anyone on the team switch views instantly without editing widget queries.

This is exactly why you configured Unified Service Tagging earlier. The env and service tags flow consistently through metrics, traces, and logs. The dashboard can filter all three data types at once.

Your dashboard is complete. Next up, you will create a monitor that alerts you when your app's performance degrades.

Create a Latency Monitor

Your custom dashboard gives you a real-time view of your application's health. But you can't stare at a dashboard 24/7 waiting for something to go wrong.

Datadog Monitors solve this problem by automatically watching your metrics and alerting you when something crosses a threshold. In this step, you'll create an APM monitor that fires when your Flask app's p90 latency gets too high.

In this step, get ready to:
  • Create an APM monitor in Datadog.
  • Configure a p90 latency threshold of 500ms over a 5-minute window.
  • Write a notification message with service context and a dashboard link.
Navigate to the APM monitor creation page

Datadog offers different monitor types for different data sources. The APM monitor type watches trace-level metrics like latency and error rate directly from your instrumented services.

  • In the Datadog left sidebar, hover over Monitors.
  • Click New Monitor.
  • Select APM from the list of monitor types.
  • Select the APM Metrics tab at the top.

Why APM Metrics and not Trace Analytics?

APM Metrics monitors use pre-aggregated service-level data (latency percentiles, error rates, request counts) that Datadog calculates automatically from your traces. Trace Analytics monitors let you query individual spans with custom filters.

For a service-wide latency threshold, APM Metrics is the right choice because it evaluates the overall service health rather than individual trace events.

Configure the p90 latency threshold

The p90 latency represents the response time that 90% of requests complete within. If this number climbs above 500ms, it means a significant portion of your users are experiencing slow responses.

  • Under Select monitor scope, set Primary Tags to env:dev.
  • Set Service to flask-app.
  • Leave Resource set to * (all endpoints).
  • Under Set alert conditions, select Threshold Alert.
  • In the Alert when dropdown, select p90 latency.
  • Set the direction to above.
  • Enter 0.5 in the Alert threshold field (this represents 500ms, since latency is measured in seconds).
  • Set the evaluation window to 5 minutes using the time dropdown.

Why 0.5 and not 500?

Datadog APM reports latency in seconds, not milliseconds. So 500ms is entered as 0.5. The preview graph above the threshold section shows a marker at your cutoff point to confirm you've set the right value.

Write the notification message and save

A good alert notification includes enough context for the on-call engineer to start investigating immediately. You'll include the service name, environment, and a direct link to your dashboard.

  • Scroll down to the Configure notifications and automations section.
  • In the monitor name field at the top of this section, enter a descriptive name like High p90 Latency on flask-app.
  • In the notification message body, enter the following context:
P90 latency on flask-app (env: dev) has exceeded 500ms.

Dashboard: Flask App Observability
Service: flask-app
Environment: dev

Investigate the /slow endpoint and check Redis connectivity.

What does this message do?

When this monitor triggers, anyone receiving the notification gets immediate context about which service is affected, which environment it's in, and where to look first. In a real production setup, you would also include a clickable link to your dashboard URL.

  • Leave the remaining settings at their defaults.
  • Click Create at the bottom of the page to save your monitor.

You should see your new monitor appear on the Monitors list page with a green OK status (since your p90 latency is likely below 500ms right now).

Trigger your monitor to test it

Hit the /slow endpoint repeatedly to push your p90 latency above 500ms. Run curl http://localhost:5000/slow several times in quick succession. After 5 minutes, the monitor should transition to an alerting state.

Monitor stuck in No Data?

If your monitor shows No Data instead of OK, your Flask app may not be sending traces. Confirm your containers are running with docker compose ps and hit any endpoint (like http://localhost:5000/) to generate trace data.

Also verify that the service name in the monitor scope matches flask-app exactly (case-sensitive).

help me debug why my Datadog APM monitor shows No Data

Your alerting is now in place. Next up, you'll define a Service Level Objective to track your app's reliability over time with an error budget.

Define a Service Level Objective

Your monitor fires when latency spikes. That tells you something is wrong right now.

A Service Level Objective answers a different question: over the past week, what percentage of time was your service actually meeting its latency target? This is how SRE teams track reliability and communicate it to stakeholders using a concrete number.

In this step, get ready to:
  • Create a Time Slice SLO for the flask-app service.
  • Set a 99% uptime target over a 7-day rolling window.
  • Observe the error budget calculation that tells you how much downtime you can afford.
Navigate to the SLO creation page

Time Slice SLOs let you define reliability using a simple condition over a metric. They do not require an existing Datadog monitor as a prerequisite.

  • In the Datadog left sidebar, click Service Mgmt.
  • Click SLOs.
  • Click + New SLO to open the Create SLO page.
  • Select By Time Slices to define your SLO measurement.

Why Time Slices instead of Monitor-based?

A Monitor-based SLO ties its calculation to an existing monitor's uptime. A Time Slice SLO lets you define a custom uptime condition directly with a metric query. You can experiment with different thresholds and instantly see how they affect your reliability score during creation.

Since you want to track latency as a percentage of time (not event counts), Time Slices are the right fit.

Define the uptime condition

The uptime condition tells Datadog what "good" looks like. Each time slice (a 5-minute interval by default) is evaluated against your condition. If the condition is met, that slice counts as uptime.

  • In the Define your SLI section, build a metric query that targets the p90 latency of your flask-app service.
  • Use the metric query builder to select the APM latency distribution metric for the flask-app service. Filter by service:flask-app and select p90 as the aggregation.
  • Set the comparator to < (less than).
  • Set the threshold to 1 (representing 1 second).
  • Leave the time slice interval at the default 5 minutes.

What does this uptime condition mean?

You are defining uptime as: the p90 latency of the flask-app service is less than 1 second. Every 5 minutes, Datadog evaluates this condition. If the p90 latency is below 1 second during that slice, it counts as uptime. If it exceeds 1 second, that slice counts as downtime.

Set the target, name, and save the SLO
  • Under Set your target & time window, set the target to 99%.
  • Select 7 days as the rolling time window.
  • In the Name field, enter Flask App Latency SLO.
  • Click Create.

After creating the SLO, Datadog displays the details panel showing your uptime percentage and the remaining error budget.

What is an error budget?

With a 99% target over 7 days, you are allowing 1% of your time slices to fail the latency condition. In a 7-day period (168 hours), 1% equals approximately 1 hour and 41 minutes of allowed downtime.

This is your error budget. As long as total downtime stays within this budget, your service is meeting its reliability commitment. If you burn through the budget too fast, it signals that something needs attention before users are impacted.

  • Observe the error budget displayed in the SLO details panel. It shows how much downtime remains before breaching your 99% target.

SLO showing no data?

If the SLO displays no status data, your flask-app service may not have generated enough trace data yet. Send a few requests to your endpoints to generate fresh latency data.

Also confirm that the metric query is scoped to service:flask-app and that your containers are still running.

Having trouble? Help me debug why my SLO shows no data.

You now have a production-grade reliability target for your service. In the next section, you will extend this setup with a Synthetic API Test to proactively monitor your endpoints.

Secret mission

Add a Synthetic API Test

Your monitoring stack is reactive. It catches problems after users report them. What if you could proactively test your endpoints every 5 minutes and get alerted the moment something breaks? Set up Datadog Synthetic Monitoring to do exactly that.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses Docker containers running locally and a Datadog free trial that auto-downgrades after 14 days with no charges.

Resources you used:

  • Docker Compose stack (flask-app, redis, datadog-agent containers).
  • Local project files (app.py, Dockerfile, docker-compose.yml, requirements.txt, .env).
  • Datadog free trial account with dashboard, monitor, SLO, and Synthetic test.

Keep everything running

No action needed. Choose this if you're still actively exploring Datadog features during your 14-day trial.

  • Your Docker Compose stack continues running and sending data to Datadog.
  • Your Datadog dashboard, monitor, SLO, and Synthetic test remain active.
  • The Datadog trial auto-downgrades after 14 days with no charges. No credit card is required.

Pause - I'll come back to this later

Shut down running containers to free up system resources while keeping your files and Datadog configuration intact.

  • Stop all containers without removing them by running:
  • Your Datadog dashboard, monitor, SLO, and Synthetic test stay saved in your Datadog account.
  • When you're ready to resume, run docker compose start from the same directory.
docker compose stop

If you also set up the Private Location container during the Secret Mission, stop it separately by running:

docker stop dd-private-location

Delete - I don't want to use this again

Remove all project containers, volumes, and local files to start completely fresh.

  • Stop and remove all containers and volumes by running:
docker compose down -v
  • If you set up the Private Location container during the Secret Mission, remove it by running:
docker stop dd-private-location && docker rm dd-private-location
  • Delete your local project files by navigating to the parent folder and running:
cd ..
rm -rf datadog-monitoring
  • Your Datadog free trial auto-downgrades after 14 days with no charges. If you want to disable your account sooner, contact Datadog support through the help widget inside your Datadog organization.
  • Any dashboards, monitors, SLOs, and Synthetic tests you created are tied to your Datadog account. They will become inaccessible once the trial period ends.

Nice Work!

Nice Work!

You just built a production-grade observability setup from scratch. Your multi-service application stack is now fully monitored with Datadog, demonstrating the three pillars of observability (metrics, traces, logs) plus custom business metrics and SLOs unified in a single platform.

You've learned how to:

  • Deploy a multi-service Docker Compose stack instrumented with Datadog APM using ddtrace, Unified Service Tagging, and custom DogStatsD metrics.
  • Build a custom Datadog dashboard correlating infrastructure metrics, APM traces, container logs, and custom business metrics in a single view.
  • Define a Service Level Objective with an error budget to track application reliability targets and create an alerting monitor that detects performance degradation.
  • Secret Mission: Set up Synthetic Monitoring with a Private Location to proactively test endpoint availability against your local environment.

Ready to quiz yourself?