Observability Stack for a 3-Tier App

Add metrics, logs, and traces to your ALB + EC2 + RDS app with CloudWatch and X-Ray.

Introduction

30 Second Summary

When your app breaks at 2am, the first question is always the same: what went wrong? Without observability, you are left guessing. With it, you know in seconds.

In this project, you will build a full observability stack on top of your existing 3-tier AWS application. You will set up real-time metrics, searchable logs, and end-to-end request traces that pinpoint problems the moment they happen.

What You'll Build

When something breaks, you will open a single dashboard and answer three questions in under 60 seconds: Is something wrong? What happened? Where is the bottleneck?

By the end of this project, you'll have:

  • A CloudWatch dashboard showing real-time CPU, memory, and error rate metrics for your running app, plus an alarm that fires when errors spike.
  • Diagnostic Logs Insights queries that search thousands of log entries and surface errors, status counts, and latency breakdowns in seconds.
  • An X-Ray service map tracing every request hop-by-hop from your Flask app to RDS, showing exactly where time is spent in each segment.
  • Secret Mission: Connect Grafana Cloud as an alternative dashboard for richer visualizations pulled directly from CloudWatch.

Are there any prerequisites?

Yes. This project builds on a running 3-tier application (ALB + EC2 + RDS with Python/Flask). You need that app deployed and accessible before starting here.

You also need an AWS account with console access and SSH access to your EC2 instance using your key pair.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Your 3-tier app is currently a black box. You're about to turn it into a glass box by layering on the same observability toolkit that production engineering teams rely on.

Set Up IAM Permissions and SSH Access

Your 3-tier app is running, but it is a black box. The EC2 instance has no permission to send telemetry data to AWS observability services.

In this step, you will establish an SSH connection to your instance and grant it the IAM permissions it needs to ship metrics, logs, and traces to CloudWatch and X-Ray.

In this step, get ready to:
  • SSH into the EC2 instance and confirm your Flask app is running.
  • Attach CloudWatch and X-Ray IAM policies to the instance role.
  • Verify AWS CLI access works from the instance.
SSH into your EC2 instance

You need a terminal session on your EC2 instance to install and configure observability tools in the coming steps. You will also confirm your Flask app is responding to requests.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.
  • Connect to your EC2 instance by running this command:
ssh -i [[PEMPATH="path/to/your-key.pem"]] ec2-user@[[EC2_PUBLIC_DNS="your EC2 public IPv4 DNS"]]

What are these values?

Replace the two placeholders with the values from your Project 1 setup. Your .pem file is the key pair you downloaded when you created the instance. Your public DNS is visible on the EC2 instance summary page in the AWS Console.

You should see a login banner and a command prompt ending in ec2-user. This confirms you are connected.

  • Confirm your Flask app is running by sending a request to it:
curl http://localhost:5000/

You should see a JSON response:

{"status": "healthy"}

Seeing 'Permission denied' or 'Connection refused'?

  • If you see a permissions error about your key file, run chmod 400 on your .pem file and try again.
  • If the connection is refused, confirm your EC2 instance is in the Running state in the AWS Console and that its security group allows SSH on port 22.
  • If curl returns an error, your Flask app may not be running. Try python3 app.py & from the app directory to start it.

Help me troubleshoot my SSH connection or Flask app

Attach IAM policies to your EC2 instance role

Your EC2 instance uses an IAM role to authenticate with AWS services. You need to add two managed policies that grant permission to send metrics, logs, and trace data.

  • Sign into the AWS Management Console.
  • In the search bar at the top, type IAM and select IAM from the results.
  • In the left navigation pane, click Roles.
  • Search for and click the name of the IAM role attached to your EC2 instance (the role you created in Project 1).
  • Click the Add permissions button.
  • Select Attach policies.
  • In the search box, type CloudWatchAgentServerPolicy and select the checkbox next to it.
  • Clear the search, then type AWSXRayDaemonWriteAccess and select the checkbox next to it.
  • Click Attach policies.

Both policies now appear in the Permissions tab of your role.

What do these policies allow?

CloudWatchAgentServerPolicy grants permission to write custom metrics and log data to CloudWatch. AWSXRayDaemonWriteAccess grants permission to upload trace segments to X-Ray.

Without these, any observability agent you install will be denied when it tries to ship data.

Verify AWS CLI access from the instance

The IAM policies are attached, but you should confirm the instance can actually authenticate with AWS. The AWS CLI is pre-installed on Amazon Linux instances and automatically uses the instance role credentials.

Before you run the next command, what do you expect to see? Think about what identity the instance will present to AWS.

  • In your SSH session, verify the instance role is working by running:
aws sts get-caller-identity

You should see JSON output containing your AWS account ID and an ARN that includes your instance role name:

{
    "Account": "123456789012",
    "UserId": "AROAEXAMPLE:i-0abc123def456",
    "Arn": "arn:aws:sts::123456789012:assumed-role/YourInstanceRole/i-0abc123def456"
}

What does this output mean?

The Arn field shows the instance is authenticating as your IAM role (not an IAM user). This means all AWS SDK and CLI calls from this instance inherit the permissions you just attached.

Seeing 'Unable to locate credentials'?

  • Make sure your EC2 instance has an IAM role attached. In the EC2 console, select your instance and click Actions.
  • Select Security, then Modify IAM role and confirm a role is selected.
  • If you just attached the policies, wait 30 seconds for the credentials to propagate and try again.

Help me fix my IAM role configuration

Your instance is connected and authorized. Next up, you will install the CloudWatch Agent to start shipping real metrics and logs from your app.

Ship Logs and Metrics with the CloudWatch Agent

In the previous step you confirmed your EC2 instance can authenticate to AWS services. Now it is time to put that access to work.

Your Flask app writes logs to a local file that disappears the moment the instance terminates. Without shipping those logs to a durable store, you lose all evidence of what went wrong. The CloudWatch Agent solves this by collecting both system metrics (CPU, memory) and application logs from your instance and streaming them to CloudWatch in near-real-time.

In this step, get ready to:
  • Install the CloudWatch Agent and enable detailed monitoring on your EC2 instance.
  • Create a configuration file that defines which metrics and logs to collect.
  • Verify metrics and logs appear in the CloudWatch console.
Install the CloudWatch Agent

The unified CloudWatch Agent handles metrics, logs, and traces in a single daemon. You will install it from the Amazon Linux package manager and then enable detailed monitoring so EC2 reports metrics at 1-minute intervals instead of the default 5.

  • In your SSH session on the EC2 instance, install the CloudWatch Agent by running this command:
sudo yum install amazon-cloudwatch-agent -y
  • Confirm the package installed successfully by running:
rpm -qa amazon-cloudwatch-agent

You should see a line showing the installed package name and version.

No output from the rpm command?

If you see no output, the install did not complete. Re-run the sudo yum install amazon-cloudwatch-agent -y command and check for permission errors. Confirm your instance has internet access or a VPC endpoint for the Amazon Linux repositories.

Help me troubleshoot why the CloudWatch Agent did not install on my EC2 instance.

Next, enable detailed monitoring on the EC2 instance so built-in EC2 metrics report at 1-minute granularity.

  • Switch to your browser and open the EC2 console.
  • In the left navigation pane, click Instances.
  • Select your instance (click its checkbox).
  • Click Actions, then Monitor and troubleshoot, then Manage detailed monitoring.
  • On the Detailed monitoring page, select Enable.
  • Click Confirm.

Why enable detailed monitoring?

Basic monitoring sends EC2 metrics every 5 minutes. Detailed monitoring sends them every 1 minute. When diagnosing an incident, 5-minute gaps can hide the exact moment a spike occurred.

Create and apply the agent configuration

The CloudWatch Agent needs a JSON configuration file telling it what to collect. You will create one that ships CPU, memory, and disk metrics plus your Flask app's log file.

  • Create the configuration file by running this command to open nano:
sudo nano /opt/aws/amazon-cloudwatch-agent/etc/cloudwatch-agent-config.json
  • Paste the following metrics configuration into the empty file:
{
  "agent": {
    "metrics_collection_interval": 60,
    "run_as_user": "root"
  },
  "metrics": {
    "namespace": "FlaskApp",
    "metrics_collected": {
      "cpu": {
        "resources": ["*"],
        "measurement": ["usage_active"],
        "totalcpu": true
      },
      "mem": {
        "measurement": ["used_percent"]
      },
      "disk": {
        "resources": ["/"],
        "measurement": ["used_percent"]
      }
    },
    "append_dimensions": {
      "InstanceId": "${aws:InstanceId}"
    }
  },

What does this section do?

The agent block sets a 60-second collection interval and runs as root so it can read all system metrics.

The metrics block tells the agent to publish CPU usage, memory percentage, and disk usage to a custom namespace called FlaskApp. The append_dimensions line tags every metric with your instance ID so you can filter by instance.

  • Directly below the metrics section (no blank line between), paste the logs configuration to complete the file:
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/home/ec2-user/flask-app/app.log",
            "log_group_name": "flask-app-logs",
            "log_stream_name": "{instance_id}-app",
            "timestamp_format": "%Y-%m-%d %H:%M:%S"
          }
        ]
      }
    }
  }
}

What does the logs section do?

The file_path tells the agent which file to tail. Every new line your Flask app writes to /home/ec2-user/flask-app/app.log gets shipped to a CloudWatch Log Group named flask-app-logs.

The timestamp_format tells CloudWatch how to parse the datetime from each log line so that Logs Insights queries can filter by time accurately.

  • Save the file by pressing Ctrl+O, then Enter, then exit nano with Ctrl+X.

✔️ Awesome, I've got everything!

Great. Make sure you saved the file before continuing.

ⓧ I'd like to double check the full code

Your file has two main sections that you pasted above, in this order:

  • The agent + metrics block (collection interval, namespace, CPU/mem/disk, append_dimensions).
  • The logs block (file_path pointing to your Flask app log, log group flask-app-logs, timestamp format).

Scroll up and confirm both chunks are present in the file as one valid JSON object (opening { at the top, closing } at the bottom). Make sure there are no trailing commas after the last section.

Now start the agent with your configuration file.

  • Start the CloudWatch Agent by running this command:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/cloudwatch-agent-config.json

What does this command do?

  • -a fetch-config tells the agent to load the config file.
  • -m ec2 tells the agent it is running on an EC2 instance (so it uses the instance role for credentials).
  • -s starts the agent immediately after loading the config.
  • -c file:... points to your configuration file path.
  • Verify the agent is running by checking its status:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status

You should see JSON output with "status": "running" confirming the agent is active.

Status shows stopped?

If the status field says stopped, check for JSON syntax errors in your config file. A trailing comma or missing brace will prevent the agent from starting. Review the agent log at /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log for details.

Help me debug why my CloudWatch Agent status shows stopped.

Verify logs and metrics appear in CloudWatch

The agent is running, but you need to confirm that data is actually arriving in CloudWatch. It takes about 1-2 minutes for the first data points to appear after the agent starts.

  • Generate some log data by hitting your Flask app endpoint from the EC2 instance:
curl http://localhost:5000/

You should see a JSON response like {"status": "healthy"} confirming the app is running. This request also writes a log line to app.log for the agent to pick up.

  • Wait about 60 seconds for the agent to ship its first batch.
  • Switch to your browser and open the CloudWatch console.
  • In the left navigation pane, click Metrics, then All metrics.
  • Look for the FlaskApp custom namespace in the list. Click it, then click InstanceId.
  • Confirm you see cpu_usage_active and mem_used_percent listed as metrics.

Before you check the logs, predict what you expect to see. Do you think a log group already exists, or does the agent need to create it?

  • In the CloudWatch left navigation pane, click Logs, then Log groups.
  • Find the log group named flask-app-logs in the list. Click it.
  • Click the log stream (named with your instance ID followed by -app). You should see your recent health check log event inside.

What should I see?

The log stream contains timestamped entries from your Flask app. You should see at least one INFO event for the health check request you triggered with curl. The agent created the log group and stream automatically on first data delivery.

Don't see the FlaskApp namespace or flask-app-logs?

It takes 1-2 minutes after starting the agent for data to appear. Refresh the CloudWatch console page after waiting.

If nothing appears after 3 minutes, verify the agent status is running. Check that your IAM role has CloudWatchAgentServerPolicy attached (you did this in Step 1). Also confirm that the file /home/ec2-user/flask-app/app.log exists on your instance.

Help me troubleshoot why my CloudWatch Agent data is not appearing in the console.

Your app is no longer a black box. Metrics and logs are streaming to CloudWatch where they persist beyond the life of any single instance. Next up, you will write Logs Insights queries that slice through those logs in seconds to find errors and patterns.

Query Logs with CloudWatch Logs Insights

Your Flask app's logs are streaming into CloudWatch, but raw log text is like reading a novel to find a single typo. You need a way to ask precise questions of your logs and get answers in seconds, not minutes of scrolling.

CloudWatch Logs Insights is a query language purpose-built for searching log data at scale. In this step, you'll write queries that filter errors, count them by time window, and attempt to diagnose latency by endpoint.

In this step, get ready to:
  • Run your first Logs Insights query to see recent log events and filter for errors.
  • Write diagnostic queries that count errors per hour and parse log patterns.
  • Hit the diagnostic wall where logs alone cannot pinpoint the bottleneck.
Run your first Logs Insights query

Logs Insights lets you query one or more log groups using a pipe-delimited query language. You select the log group, write a query, and get structured results back instantly.

  • In the AWS Management Console, navigate to CloudWatch.
  • In the left sidebar, expand Logs and click Logs Insights.
  • In the Select log group(s) dropdown, search for and select flask-app-logs.

The query editor pre-fills with the default query. This shows the 25 most recent log events from your app.

  • Click Run query with the default query:
fields @timestamp, @message
| sort @timestamp desc
| limit 25

What does this query do?

  • fields @timestamp, @message selects which fields to display in the results table.
  • sort @timestamp desc orders results newest-first.
  • limit 25 caps the output at 25 log events.

You should see a results table with your Flask app's log entries, showing timestamps and messages like Health check requested or Fetching items from database.

Now filter for just the errors. Replace the query in the editor with:

  • Clear the query editor and paste this query, then click Run query:
fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 25

What does the filter command do?

The filter command narrows results to only log events matching a condition. Here, /ERROR/ is a regex pattern that matches any message containing the word ERROR.

If you see no results, that means your app has not logged any errors yet. You will generate some in the final substep.

Not seeing any log events?

  • Make sure you selected the flask-app-logs log group in the dropdown at the top of the page.
  • Check that the time range selector (top right) covers the last 1 hour. If your app has only been running briefly, use Last 30 minutes.
  • help me troubleshoot why my Logs Insights query returns no results for flask-app-logs
Write diagnostic queries

A single filter is useful, but real diagnostics require aggregation. You need queries that count errors over time and reveal patterns.

Your first diagnostic query counts errors grouped by hour. This tells you whether errors are spiking or steady.

  • Clear the query editor and paste this query, then click Run query:
filter @message like /ERROR/
| stats count(*) as exceptionCount by bin(1h)
| sort exceptionCount desc

What does this query do?

  • filter @message like /ERROR/ narrows to error-level entries only.
  • stats count(*) as exceptionCount by bin(1h) groups the matching events into 1-hour buckets and counts how many errors fell in each bucket.
  • sort exceptionCount desc puts the busiest hours at the top so you spot spikes immediately.

If your app has logged errors, you see a table with columns for the time bin and the count. If no errors exist yet, you will generate them shortly.

Your second diagnostic query finds the 10 most recent 500-generating errors. In your app, any log line at ERROR level corresponds to a 500 response.

  • Clear the query editor and paste this query, then click Run query:
fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 10

Why does this find 500 errors?

Your Flask app logs at the ERROR level whenever it catches an exception and returns a 500 status code. Filtering for /ERROR/ in the message captures every one of those failed requests.

Your third diagnostic query uses the parse command to extract structure from your log messages and group by activity type.

  • Clear the query editor and paste this query, then click Run query:
parse @message "* * *" as timestamp, level, msg
| stats count(*) as requests by msg
| sort requests desc
| limit 10

What does parse do here?

The parse command splits each log message on spaces using wildcards. It extracts the timestamp, level (INFO or ERROR), and the remaining message text into separate fields.

The stats count(*) by msg then counts how many times each unique message appears. This reveals which log lines are most common, giving you a frequency view of your app's behaviour.

Save queries you use often

Click Save above the query editor to save any query for reuse. Saved queries appear under the Saved queries tab for quick access during future investigations.

Hit the diagnostic wall

Your queries work, but they need data to query. You need to generate real traffic including requests that hit the database, then use your queries to investigate.

  • Switch back to your SSH terminal session on the EC2 instance.
  • Generate traffic to your app by running these commands:
# Hit the health endpoint
curl http://localhost:5000/

# Hit the database-backed endpoint multiple times
curl http://localhost:5000/items
curl http://localhost:5000/items
curl http://localhost:5000/items

# Create a new item (POST request)
curl -X POST http://localhost:5000/items \
  -H "Content-Type: application/json" \
  -d '{"name": "test-item", "description": "observability test"}'

Why generate traffic?

Each request generates log entries in your Flask app. The GET requests to /items query the database, which may succeed or fail depending on your RDS state. Either way, the log entries give your Logs Insights queries real data to search.

  • Wait about 30 seconds for the CloudWatch Agent to ship the new log entries.
  • Switch back to the CloudWatch Logs Insights console in your browser.

Now run the combined error-count query to see what your traffic produced. Before you run it, predict: will you be able to tell from the query results whether a slow request was caused by your Flask code, the network hop, or the RDS query itself?

  • Paste this query and click Run query:
filter @message like /ERROR/
| stats count(*) as exceptionCount by bin(1h)
| sort exceptionCount desc

You see a table showing error counts grouped by hour. You can tell WHAT went wrong (a database error, a connection timeout, an application exception). But look at the results closely.

Can you tell WHERE in the request path the time was spent? Was the delay in your Flask code processing the request? Was it the network between EC2 and RDS? Was it the database query itself running slowly?

This is the diagnostic wall

Logs tell you WHAT happened but not WHERE. You can see "Database error: Connection timed out" in the log, but you cannot determine which segment of the request consumed the time.

Was it 50ms in Flask and 2000ms in RDS? Or 1800ms in Flask and 200ms in RDS? The logs contain no per-segment timing breakdown. This gap is exactly what distributed tracing solves.

Still not seeing any ERROR entries?

  • If your RDS instance is healthy, the /items requests succeed with INFO-level logs only. Try sending a malformed POST to force an error:
  • Run curl -X POST http://localhost:5000/items -H "Content-Type: application/json" -d '{}' to trigger a KeyError that logs at ERROR level.
  • help me generate ERROR-level log entries in my Flask app so I can practice Logs Insights queries

You have built a powerful query toolkit that finds errors in seconds. But you have also hit the limit of what logs alone can reveal. Next up, you will close this gap by adding distributed tracing with X-Ray, which times every segment of a request so you can pinpoint exactly where latency lives.

Trace Requests End-to-End with X-Ray

Your Logs Insights queries found errors and slow responses. You know what happened. But you still cannot answer where in the request path the time was spent.

AWS X-Ray solves this by tracing each request hop-by-hop, from your Flask code through the network to your RDS database. In this step, you will install the X-Ray daemon, instrument your app with the X-Ray SDK, and view a service map that pinpoints exactly which segment is slow.

In this step, get ready to:
  • Install the X-Ray daemon on your EC2 instance.
  • Instrument your Flask app with the X-Ray SDK to create trace segments.
  • View the X-Ray service map showing per-segment latency between your app and RDS.
Install the X-Ray daemon

The X-Ray daemon is a background process that listens on UDP port 2000. Your instrumented app sends trace data to the daemon, and the daemon batches and ships it to the X-Ray service. The AWSXRayDaemonWriteAccess policy you attached in Step 1 already grants it permission to do this.

  • In your SSH session, download the X-Ray daemon RPM by running this command:
curl https://s3.us-east-2.amazonaws.com/aws-xray-assets.us-east-2/xray-daemon/aws-xray-daemon-3.x.rpm -o /home/ec2-user/xray.rpm
  • Install the RPM by running:
sudo yum install -y /home/ec2-user/xray.rpm

What does this install?

The RPM installs the X-Ray daemon binary and registers it as a systemd service. It auto-starts immediately after installation, so you do not need to run a separate start command.

  • Confirm the daemon is running by checking its service status:
systemctl status xray

You should see the service listed as active (running). This confirms the daemon is listening and ready to receive trace data from your app.

Seeing inactive or failed?

If the status shows inactive, start it manually with sudo systemctl start xray and check the status again.

If you see a permissions error, confirm your EC2 instance role has the AWSXRayDaemonWriteAccess policy attached from Step 1.

Help me troubleshoot why the X-Ray daemon won't start on my Amazon Linux EC2 instance.

Instrument your Flask app with the X-Ray SDK

The daemon is running, but your Flask app does not know to send it trace data yet. You need to install the X-Ray SDK and add middleware that wraps every incoming request in a trace segment.

  • Install the X-Ray SDK for Python by running:
pip install aws-xray-sdk

Now you need to update app.py to import and configure X-Ray. The changes go in the top section of the file, before your route definitions.

  • Open app.py in your editor on the EC2 instance:
nano /home/ec2-user/flask-app/app.py
  • Replace the entire top section of the file (from the first import line down through app = Flask(__name__)) with this updated code that adds X-Ray instrumentation:
from flask import Flask, jsonify, request
from aws_xray_sdk.core import xray_recorder, patch_all
from aws_xray_sdk.ext.flask.middleware import XRayMiddleware
import logging
import mysql.connector
import os

# Configure logging to file
logging.basicConfig(
    filename='/home/ec2-user/flask-app/app.log',
    level=logging.INFO,
    format='%(asctime)s %(levelname)s %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S'
)
logger = logging.getLogger(__name__)

# Configure X-Ray
plugins = ('EC2Plugin',)
xray_recorder.configure(
    service='flask-app',
    plugins=plugins
)

# Patch all supported libraries (including mysql-connector)
patch_all()

app = Flask(__name__)

# Apply X-Ray middleware to Flask
XRayMiddleware(app, xray_recorder)

What does this code do?

xray_recorder.configure() sets the service name that appears in the X-Ray console and loads the EC2Plugin to tag traces with your instance ID and availability zone.

patch_all() automatically instruments supported libraries like mysql-connector. Every database call now creates a subsegment in your traces without any extra code.

XRayMiddleware(app, xray_recorder) wraps every incoming Flask request in a trace segment. Each request gets its own trace ID that follows it through your entire code path.

  • Leave the rest of the file (all your route functions and the if __name__ block) exactly as they are.
  • Save the file and exit the editor.
  • Restart your Flask app so the new X-Ray instrumentation takes effect:
cd /home/ec2-user/flask-app
python app.py &

Stop the old process first

If your Flask app is already running, stop it first with pkill -f 'python app.py' before starting the new version. Otherwise port 5000 will already be in use.

  • Test that the app still responds by running:
curl http://localhost:5000/

You should see {"status": "healthy"} confirming the app is running with X-Ray middleware active.

App won't start or shows an import error?

If you see ModuleNotFoundError: No module named 'aws_xray_sdk', the SDK installed to a different Python environment. Run which python to confirm which Python you are using and reinstall with that interpreter's pip.

If the app crashes with a different error, check your app.py imports match the code above exactly. A missing comma in the plugins tuple is a common typo.

Help me fix an import or startup error after adding the X-Ray SDK to my Flask app.

✔️ Awesome, I've got everything!

Great. Double check you have saved app.py and that the health check returned a 200 response.

ⓧ I'd like to double check the full code

Your app.py is longer than 30 lines in total, so here is what to confirm:

  • The top section (imports through XRayMiddleware(app, xray_recorder)) matches the code block above exactly.
  • The get_db_connection() function is unchanged from your previous version.
  • The three route functions (/, /items GET, /items POST) are unchanged.
  • The if __name__ == '__main__': block at the bottom is unchanged.
Generate traffic and view the X-Ray service map

Your app is now sending trace data to the X-Ray daemon with every request. To see the service map, you need to generate some traffic that exercises both your Flask code and the database connection.

  • Hit the health endpoint several times by running:
curl http://localhost:5000/
curl http://localhost:5000/
curl http://localhost:5000/
  • Hit the database-backed items endpoint to generate traces that include RDS subsegments:
curl http://localhost:5000/items
curl -X POST http://localhost:5000/items -H "Content-Type: application/json" -d '{"name": "test-item", "description": "trace test"}'
curl http://localhost:5000/items

Why multiple requests?

X-Ray needs a few traces before it can draw the service map. Each request creates one trace. Hitting both the health endpoint and the database-backed endpoint ensures the service map shows all the nodes in your architecture.

  • Wait about 30 seconds for the daemon to batch and ship the traces to AWS.
  • Switch to the AWS Management Console in your browser.
  • In the search bar at the top, type X-Ray and select CloudWatch (X-Ray traces are accessed through the CloudWatch console).
  • In the left navigation, expand X-Ray traces and click Service map.

Before you look at the map, predict what you will see. How many nodes do you expect? What do they represent?

You should see at least two nodes connected by lines. One node represents your Flask app (labeled flask-app). The other represents the downstream RDS database call. The lines show request flow, and each node displays its average latency.

  • Click the flask-app node to see recent traces.
  • Click any individual trace to open its timeline view.

The timeline shows millisecond-by-millisecond breakdown of where time was spent. You can see the Flask segment (your application code) and the database subsegment (the RDS query) as separate bars with exact durations.

What does this solve?

In the previous step, your Logs Insights query showed a database timeout occurred. Now you can see that the database subsegment took 2,100ms while the Flask segment only took 15ms. The bottleneck is the RDS query, not your app code. This is the diagnostic gap that distributed tracing closes.

Service map is empty?

Traces can take up to 60 seconds to appear. Adjust the time range in the top right of the console to Last 5 minutes and refresh.

If nothing appears after a minute, confirm the daemon is running with systemctl status xray. Then check that your IAM role has the AWSXRayDaemonWriteAccess policy and that you are viewing the same region where your EC2 instance runs.

Help me figure out why my X-Ray service map is empty even though I generated traffic.

You now have all three observability pillars working. Metrics tell you something is wrong. Logs tell you what happened. Traces tell you where it happened. Next up, you will bring them together into a single CloudWatch dashboard with an alarm that wakes you up when things go sideways.

Build a Dashboard, Alarm, and Runbook

You now have all three observability pillars working: metrics show resource health, logs reveal error details, and traces pinpoint where latency lives. But these signals are scattered across different console pages.

When something breaks at 2am, you need a single screen that answers "is something wrong?" in seconds. A dashboard gives your team that single pane of glass. An alarm wakes someone up. A runbook tells them exactly what to do next.

In this step, get ready to:
  • Create a CloudWatch dashboard with metrics, logs, and response time widgets.
  • Set up a CloudWatch alarm that fires when error count spikes.
  • Write an incident runbook documenting your full diagnostic workflow.
Create a CloudWatch dashboard

A CloudWatch dashboard combines metrics, log queries, and trace data into a single view. You will create one with three widgets covering the three observability pillars.

  • In the AWS Console, navigate to CloudWatch in the search bar.
  • In the left navigation pane, click Dashboards.
  • Click Create dashboard.
  • Enter FlaskApp-Dashboard as the name and click Create dashboard.

A dialog appears asking you to add your first widget.

  • Select Line and click Configure.
  • In the Browse tab, search for your custom namespace FlaskApp.
  • Select cpu_usage_active and mem_used_percent metrics.
  • Click Create widget.

Your first widget now shows a line graph of CPU and memory usage over time.

  • Click the + symbol to add another widget.
  • Select Line and click Configure.
  • Switch to the Logs tab instead of the Metrics tab.
  • Select the flask-app-logs log group.
  • Replace the default query with the following:
filter @message like /ERROR/
| stats count(*) as errorCount by bin(1h)
  • Click Run query to confirm results appear.
  • Click Create widget.

What does this Logs Insights widget do?

This widget runs a Logs Insights query on every dashboard refresh. It counts lines containing "ERROR" in your Flask app logs and groups them by hour. A spike in the graph tells you immediately that something went wrong.

  • Click the + symbol to add the third widget.
  • Select Number and click Configure.
  • In the Browse tab, search for the AWS/ApplicationELB namespace.
  • Select the TargetResponseTime metric for your load balancer.
  • Set the statistic to Average.
  • Click Create widget.
  • Click Save dashboard in the top right corner.

All three widgets now display live data on a single screen. The line graph shows system resource trends, the Logs Insights widget shows error frequency, and the number widget shows average response time in seconds.

Don't see data in a widget?

  • Check the time range in the top right of the dashboard. Set it to Last 1 hour or Last 3 hours to match when your agent started shipping data.
  • For the FlaskApp namespace, confirm the CloudWatch Agent is still running on EC2.
  • For the AWS/ApplicationELB namespace, your ALB must have received recent traffic. Hit your app endpoint a few times to generate data points.

Help me troubleshoot my CloudWatch dashboard widgets not showing data.

Create a CloudWatch alarm

A dashboard shows current state, but nobody stares at a dashboard all day. Alarms watch your metrics continuously and change state when a threshold is breached.

  • In the CloudWatch left navigation, click Alarms, then All alarms.
  • Click Create alarm.
  • Click Select metric.
  • Navigate to Logs > Log Group Metrics.
  • Select the IncomingLogEvents metric for the flask-app-logs log group.
  • Click Select metric.

Why IncomingLogEvents?

Your Flask app only logs errors with the "ERROR" prefix. A sudden spike in incoming log events correlates directly with application errors. For production, you would create a custom metric filter that counts only ERROR lines, but for this project the log event count gives you a working alarm quickly.

  • Set the Statistic to Sum.
  • Set the Period to 5 minutes.
  • Under Conditions, select Greater than.
  • Enter 5 as the threshold value.
  • Click Next.
  • On the Configure actions page, select Remove next to the default SNS notification. You do not need an SNS action for this project.
  • Click Next.
  • Enter FlaskApp-HighErrorCount as the alarm name.
  • Click Next, review your settings, and click Create alarm.

Before you check the alarm state, what do you think it will show? Your app has low traffic right now, so the error count should be well below 5.

Your alarm now appears in the All alarms list with a green OK state. This confirms the threshold is set correctly and your app is currently healthy.

Where would you add notifications?

In the Configure actions step you skipped, you would create an SNS topic and subscribe an email address or PagerDuty endpoint. When the alarm enters ALARM state, SNS delivers the notification to that endpoint. For this project, the alarm state change itself is sufficient proof the system works.

Alarm shows Insufficient data?

  • This means CloudWatch has not received enough data points for the selected metric and period. Wait 5-10 minutes for the metric to populate.
  • Generate some traffic to your app to produce log events. Hit your health endpoint a few times.

Help me figure out why my CloudWatch alarm shows Insufficient data instead of OK.

Write your incident runbook

A dashboard and alarm tell you something is wrong. But at 2am, you need a documented plan that tells you exactly what to check and in what order. That document is called a runbook.

  • Switch back to your SSH terminal connected to the EC2 instance.
  • Create the runbook file by running this command:
cat > /home/ec2-user/flask-app/runbook.md << 'EOF'
# Runbook: High Response Time / App Slow

## Symptom

CloudWatch alarm fires: error count exceeds 5 in a 5-minute period.

## Diagnosis Steps

1. Open CloudWatch dashboard - check CPU, memory, and error rate widgets.
   - If CPU > 80%: likely compute bottleneck.
   - If memory > 90%: possible memory leak.
   - If error rate spiked: proceed to step 2.

2. Query CloudWatch Logs Insights for recent errors:
   ```
   filter @message like /ERROR/
   | sort @timestamp desc
   | limit 20
   ```
   - Look for patterns: database timeouts? connection refused? application exceptions?
EOF

What does the diagnosis section cover?

  • Step 1 uses the dashboard you just built to narrow down whether the issue is CPU, memory, or error-rate driven.
  • Step 2 uses the Logs Insights queries from earlier to find the specific error pattern.
  • Each step has clear decision criteria so an on-call engineer can follow it without prior context.
  • Append the rest of the runbook by running:
cat >> /home/ec2-user/flask-app/runbook.md << 'EOF'

3. Check X-Ray service map and recent traces:
   - Open X-Ray > Service Map.
   - Look for red/orange nodes indicating errors or high latency.
   - Click the slowest trace to see segment breakdown.
   - Identify which segment (Flask app or RDS) has the longest duration.

## Resolution

- If RDS is the bottleneck (segment > 2s):
  - Check for slow queries using RDS Performance Insights.
  - Consider adding an index or read replica.
  - Verify RDS instance is not at connection limit.

- If EC2/Flask is the bottleneck (segment > 2s, RDS is fast):
  - Check for memory leaks or blocking I/O in application code.
  - Consider scaling up instance size or adding instances behind ALB.

- If network is the bottleneck (gap between segments):
  - Verify security groups allow traffic.
  - Check if RDS is in the same AZ as EC2.

## Escalation

If root cause unclear after 15 minutes, escalate to senior engineer with:
- Screenshot of CloudWatch dashboard at time of incident
- Logs Insights query results showing the error pattern
- X-Ray trace ID of a representative slow/failed request
EOF

What does the resolution section cover?

The resolution section maps each possible bottleneck (RDS, EC2, or network) to concrete actions. The escalation section ensures that if you cannot resolve within 15 minutes, you hand off with all the evidence someone else needs to pick up immediately.

  • Verify the runbook was saved correctly by running:
cat /home/ec2-user/flask-app/runbook.md

You should see the full runbook printed to your terminal, starting with the header and ending with the escalation criteria.

File is empty or missing?

  • Make sure you ran both cat > (first part) and cat >> (second part) commands. The first creates the file, the second appends to it.
  • Check you are in the correct directory. Run ls /home/ec2-user/flask-app/ to confirm the file exists.

Help me fix my runbook file that is empty or not saving correctly on my EC2 instance.

Now let's confirm the full picture. Before you check your dashboard, predict what you expect to see across all three widgets given your current low-traffic state.

  • Return to the CloudWatch console and open your FlaskApp-Dashboard.
  • Confirm all three widgets display data.
  • Navigate to Alarms > All alarms and confirm FlaskApp-HighErrorCount shows OK (green).

Your dashboard loads live data across all three widgets. Your alarm shows OK state. Your runbook documents the exact steps from alarm to resolution.

You now have a complete observability stack: metrics for the big picture, logs for the details, traces for the path, a dashboard to see it all at once, an alarm to wake you up, and a runbook to guide you through. Next up, the Secret Mission takes this further with a Grafana Cloud dashboard.

Secret mission

Connect Grafana Cloud for a Custom Dashboard

Your CloudWatch dashboard works, but production teams often want richer visualizations and the ability to mix data from multiple sources on a single screen. In this secret mission, you connect Grafana Cloud (free tier) to your CloudWatch data and build a custom multi-panel dashboard with more flexible graphing options.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. All services used in this project fall within AWS Free Tier limits for a single low-traffic application, so there are no ongoing costs while the resources remain active.

Resources you used:

  • CloudWatch Log Group (flask-app-logs) streaming your Flask app logs.
  • CloudWatch Dashboard (FlaskApp-Dashboard) with three widgets.
  • CloudWatch Alarm monitoring error count.
  • CloudWatch Agent running on EC2 (ships metrics and logs).
  • X-Ray daemon running on EC2 (ships trace data).
  • X-Ray traces stored for 30 days automatically.
  • IAM policies (CloudWatchAgentServerPolicy and AWSXRayDaemonWriteAccess) attached to your EC2 instance role.
  • IAM user grafana-cloudwatch-reader with access key (Secret Mission).
  • Grafana Cloud account and dashboard (Secret Mission).

Keep everything running

No action needed. Choose this if you are still actively building on your observability stack or plan to use it as a reference.

  • All CloudWatch services stay within Free Tier for a single low-traffic app (5 GB logs, 3 dashboards, 10 alarms).
  • X-Ray retains traces for 30 days. The first 100,000 traces recorded per month are free.
  • The CloudWatch Agent and X-Ray daemon continue running on your EC2 instance with negligible resource usage.
  • Your Grafana Cloud free tier account remains active with no action required.

Pause - I'll come back to this later

Stop running processes to free up EC2 resources while keeping your configuration intact. CloudWatch retains historical data.

  • Stop the CloudWatch Agent on your EC2 instance by running:
  • Stop the X-Ray daemon on your EC2 instance by running:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a stop
sudo systemctl stop xray
  • Stop your EC2 instance from the EC2 console to pause all compute costs.
  • Your CloudWatch dashboard, alarm, and log group remain accessible. X-Ray traces are retained for 30 days automatically.
  • Your Grafana Cloud dashboard remains intact. No action needed there.

Delete - I don't want to use this again

Remove all observability resources and start fresh. Run these commands from your EC2 SSH session.

  • Stop and remove the CloudWatch Agent:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a stop
sudo yum remove -y amazon-cloudwatch-agent
  • Stop and remove the X-Ray daemon:
sudo systemctl stop xray
sudo yum remove -y xray
  • Delete the CloudWatch Log Group:
aws logs delete-log-group --log-group-name flask-app-logs
  • Delete the CloudWatch Dashboard:
aws cloudwatch delete-dashboards --dashboard-names FlaskApp-Dashboard
  • Delete the CloudWatch Alarm. Replace the alarm name with the name you chose when creating it:
aws cloudwatch delete-alarms --alarm-names [[ALARM_NAME="your alarm name"]]
  • Detach the IAM policies from your EC2 instance role. In the IAM console, navigate to Roles, select your EC2 instance role, and remove the CloudWatchAgentServerPolicy and AWSXRayDaemonWriteAccess policies.
  • Delete the Grafana IAM user. In the IAM console, navigate to Users, select grafana-cloudwatch-reader, click Delete, and confirm.
  • Delete your Grafana Cloud dashboard. In Grafana Cloud, open your dashboard, click the gear icon for Dashboard settings, scroll to the bottom, and click Delete dashboard.
  • Remove the runbook.md file from your EC2 instance if you no longer need it:
rm /home/ec2-user/flask-app/runbook.md

Nice Work!

Nice Work!

Nice work! You just built a full observability stack on top of your 3-tier application. Your app is no longer a black box. When something breaks, you can now answer Is something wrong?, What happened?, and Where is the bottleneck? in under 60 seconds.

You've learned how to:

  • Instrument a Flask app with the X-Ray SDK to trace requests end-to-end from EC2 to RDS, revealing exactly where latency lives in each request segment.
  • Write CloudWatch Logs Insights queries that filter errors, count exceptions by time window, and parse response latency across your application logs.
  • Build a CloudWatch dashboard with metrics, logs, and trace widgets, a CloudWatch alarm that fires on error spikes, and an incident runbook documenting the full diagnostic workflow from alert to resolution.
  • Secret Mission: Connected Grafana Cloud to CloudWatch as a data source and built a custom multi-panel dashboard with richer visualizations for CPU/memory, error rate, and X-Ray trace durations.

Ready to quiz yourself?