EC2 Monitoring Dashboard with Grafana
Monitor EC2 health in real time using CloudWatch metrics and Grafana Cloud.
Introduction
30 Second Summary
Think about how a car dashboard gives you speed, fuel, and engine temperature at a glance. Production servers need the same kind of visibility, but most teams discover problems only after users start complaining.
In this project, you will build a real-time monitoring dashboard that pulls live metrics from an AWS server and alerts you the moment something goes wrong.
What You'll Build
You'll open a single Grafana dashboard URL, see your EC2 instance's CPU, memory, disk, and network health updating in real time, and watch an alert fire the instant you deliberately overload the server.
By the end of this project, you'll have:
- A live monitoring dashboard with multiple panels showing CPU, memory, disk, and network metrics side by side, exactly like a production ops team would use.
- A working alert rule that detects high CPU usage and transitions to a firing state, proving your monitoring catches real problems before users notice.
- A stress-tested server where you intentionally spike CPU to 100% and watch the dashboard respond in real time.
- Secret Mission: Add a CloudWatch Logs panel that streams your instance's system logs directly into the same dashboard for complete observability.
Are there any prerequisites?
You'll need an AWS account (Free Tier eligible), the AWS CLI installed and configured on your machine, and an SSH key pair for connecting to your EC2 instance.
Before We Start
Before we dive in, let's lock in what you're about to build and why it matters. Understanding the goal upfront helps you connect each step to the bigger picture of production monitoring.
Launch an EC2 Instance with the Right IAM Role
For your monitoring dashboard to display real-time data, you need a running server that can publish metrics to CloudWatch. AWS doesn't grant this permission automatically.
Without the correct IAM role attached, the CloudWatch Agent you install later won't be able to send data. In this step, you'll create the role, launch your instance, and discover which metrics CloudWatch collects on its own.
In this step, get ready to:
- Create an IAM role that grants CloudWatch publishing permissions.
- Launch a t2.micro EC2 instance with the role attached.
- SSH in and discover what CloudWatch monitors by default.
Create an IAM role for CloudWatch
IAM roles let AWS services act on your behalf. Your EC2 instance needs permission to send metrics to CloudWatch, and a role is how you grant that access without embedding credentials on the server.
- Sign in to the AWS Management Console at https://console.aws.amazon.com.
- Type IAM in the top search bar and select IAM from the results.
- In the left sidebar, click Roles.
- Click Create role.
- Under Trusted entity type, select AWS service.
- Under Use case, select EC2.
- Click Next.
Now you'll attach the managed policy that grants CloudWatch publishing permissions.
- In the permissions search bar, type CloudWatchAgentServerPolicy.
- Check the box next to CloudWatchAgentServerPolicy.
- Click Next.
- In the Role name field, type CloudWatchAgentRole.
- Click Create role.
You should see a success banner confirming the role was created.
What is CloudWatchAgentServerPolicy?
This is an AWS-managed policy that grants permission to publish custom metrics and logs to CloudWatch. By attaching it to a role, any EC2 instance using that role can send data without needing hardcoded credentials.
Don't see the policy in search results?
Make sure you typed the full name CloudWatchAgentServerPolicy with no spaces. If the search returns no results, try clearing the search bar and typing CloudWatch to see all CloudWatch-related policies, then scroll to find it.
help me find the CloudWatchAgentServerPolicy in IAM
Launch a t2.micro EC2 instance
Now you'll launch the server that will generate the metrics your dashboard monitors. You'll attach the IAM role you just created so the instance has permission to publish to CloudWatch from the moment it starts.
- Type EC2 in the top search bar and select EC2 from the results.
- Click Launch instances.
- In the Name field, type MonitoringTarget.
- Under Application and OS Images, confirm Amazon Linux 2023 AMI is selected (it is the default).
- Under Instance type, select t2.micro (Free Tier eligible).
- Under Key pair, select your existing SSH key pair from the dropdown.
Why t2.micro?
For AWS accounts created before July 15, 2025, t2.micro is the Free Tier eligible instance type. You get 750 hours per month at no charge for 12 months.
- Under Network settings, click Edit.
- Check Allow SSH traffic from and select My IP from the dropdown.
- Expand the Advanced details section at the bottom of the page.
- In the IAM instance profile dropdown, select CloudWatchAgentRole.
- Click Launch instance.
Wait for the instance to reach Running state. Click on the instance ID link to open its details page and confirm the IAM role shows CloudWatchAgentRole in the Security tab.
Don't see CloudWatchAgentRole in the IAM instance profile dropdown?
The role may take a minute to propagate after creation. Try refreshing the page by clicking the circular refresh icon next to the IAM instance profile dropdown. If it still doesn't appear, go back to IAM and verify the role's trusted entity is set to EC2 (not another service).
help me attach my IAM role to an EC2 instance
SSH in and check default CloudWatch metrics
Your instance is live. Now you'll connect to it over SSH and then check what CloudWatch already knows about your server.
- On the instance details page, copy the Public IPv4 address.
- Open your terminal by pressing Cmd+Space and typing Terminal.
- Connect to your instance by running this command:
ssh -i [[PEMPATH="path/to/your-key.pem"]] ec2-user@[[EC2_PUBLIC_IP="your EC2 public IPv4 address"]]
What does this command do?
ssh opens a secure connection to your instance. The -i flag points to your private key file, and ec2-user is the default username on Amazon Linux instances.
If prompted to confirm the host fingerprint, type yes and press Enter. You should see a command prompt starting with [ec2-user@ confirming you are connected.
Getting a permission denied or timeout error?
If you see Permission denied: check that your key file path is correct and that you ran chmod 400 on it previously. If the connection times out, confirm your security group allows SSH from your current IP (your IP may have changed since you created the rule).
help me fix my SSH connection to EC2
Now check what CloudWatch is already collecting for your instance. Switch back to the AWS Console in your browser.
- Type CloudWatch in the top search bar and select CloudWatch.
- In the left sidebar, click All metrics (under Metrics).
- Click the AWS/EC2 namespace.
- Click Per-Instance Metrics.
Before you look at the available metrics, take a guess. What do you think CloudWatch is tracking for your instance? Do you expect to see memory usage and disk space?
Scroll through the list. You'll see metrics like CPUUtilization, NetworkIn, NetworkOut, and StatusCheckFailed. But there's a gap. Memory usage and disk space are nowhere to be found.
This is not a mistake. CloudWatch only collects basic hypervisor-level metrics by default. It cannot see inside your instance to measure memory or disk usage. That gap is exactly what you'll close in the next step by installing the CloudWatch Agent.
Why are memory and disk missing?
CloudWatch collects metrics from the hypervisor layer (the host machine running your instance). CPU and network data are visible from the outside, but memory and disk usage live inside the operating system. An agent running inside the instance is required to capture and publish those metrics.
Your instance is running and your IAM role is attached. You've discovered the monitoring gap firsthand. Next up, you'll install the CloudWatch Agent to fill that gap and start streaming memory and disk metrics.
Install and Configure the CloudWatch Agent
In Step 1, you discovered the gap: CloudWatch only shows basic metrics like CPUUtilization and network traffic by default. Memory usage and disk space are completely invisible.
The CloudWatch Agent fills that gap. You'll install it on your EC2 instance, tell it which metrics to collect via a configuration file, and start it running.
In this step, get ready to:
- Install the CloudWatch Agent on your EC2 instance.
- Create a configuration file that collects CPU, memory, and disk metrics.
- Start the agent and confirm your new metrics appear in CloudWatch.
Install the CloudWatch Agent package
The CloudWatch Agent is a lightweight daemon that runs on your instance and publishes metrics to CloudWatch on a schedule. It is available as a standard package in the Amazon Linux 2023 repositories.
- In the SSH session connected to your EC2 instance from the previous step, install the agent by running this command:
sudo yum install -y amazon-cloudwatch-agent
You should see download progress followed by a Complete! message confirming the package installed successfully.
Seeing a 'No match for argument' error?
Make sure you typed the package name exactly as amazon-cloudwatch-agent with hyphens, not underscores or spaces.
If the error persists, confirm you are running Amazon Linux 2023 by running cat /etc/os-release. help me troubleshoot the CloudWatch Agent install on my EC2 instance
Create the agent configuration file
The agent needs a JSON configuration file to know which metrics to collect and how often. You'll create one that captures CPU usage, memory utilization, and disk space.
- Create the configuration file by running this command:
sudo tee /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json << 'EOF'
{
"metrics": {
"metrics_collected": {
"cpu": {
"measurement": ["cpu_usage_idle", "cpu_usage_user", "cpu_usage_system"],
"metrics_collection_interval": 60
},
"mem": {
"measurement": ["mem_used_percent"],
"metrics_collection_interval": 60
},
"disk": {
"measurement": ["disk_used_percent"],
"metrics_collection_interval": 60,
"resources": ["/"]
}
}
}
}
EOF
What does this configuration do?
- cpu collects three CPU metrics: idle time, user-space usage, and system usage. Together these show how hard your instance is working.
- mem collects mem_used_percent which tells you how much RAM is consumed. This is one of the metrics CloudWatch cannot see without the agent.
- disk collects disk_used_percent for the root filesystem /. This is the other invisible metric from Step 1.
- metrics_collection_interval of 60 means the agent reports each metric once every 60 seconds.
- Confirm the file was written correctly by running:
cat /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
You should see the JSON configuration printed back to the terminal, matching what you pasted above.
Seeing an empty output or 'No such file' error?
Make sure you included sudo before tee. The directory requires root permissions to write into.
Also confirm the directory exists by running ls /opt/aws/amazon-cloudwatch-agent/etc/. If the directory is missing, the agent package did not install correctly. Re-run the install command from the previous substep. help me fix my CloudWatch Agent configuration file
Start the agent and verify metrics appear in CloudWatch
With the configuration in place, you can start the agent. It will immediately begin reading metrics and publishing them to CloudWatch under the CWAgent namespace.
- Start the CloudWatch Agent by running this command:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -s
What does this command do?
- -a fetch-config tells the agent to load a configuration file.
- -m ec2 tells the agent it is running on an EC2 instance (so it can discover metadata automatically).
- -c file:/opt/aws/... points to the JSON configuration you just created.
- -s starts the agent after loading the configuration.
- Verify the agent is running by checking its status:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status
You should see JSON output containing "status": "running" along with the start time and version number.
Status shows 'stopped' instead of 'running'?
This usually means the configuration file has a syntax error. Run cat /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json and check for missing commas, brackets, or quotation marks.
Also confirm your EC2 instance has the IAM role CloudWatchAgentRole attached. Without it, the agent cannot publish metrics and will stop itself. help me get my CloudWatch Agent running
Now you need to wait 2-3 minutes for the agent to collect and publish its first data points. Before you check the console, think about this: where do you expect the new metrics to appear?
- Open the AWS Console and navigate to CloudWatch.
- In the left sidebar, click All metrics under Metrics.
- Look under Custom namespaces for a namespace called CWAgent.
- Click into CWAgent and you should see mem_used_percent and disk_used_percent metrics listed for your instance.
Why a separate namespace?
Default EC2 metrics (like CPUUtilization) live under the AWS/EC2 namespace. Metrics published by the CloudWatch Agent appear under CWAgent by default.
This separation makes it clear which metrics come from AWS natively and which ones your agent is collecting.
Don't see the CWAgent namespace after 3 minutes?
Make sure you are looking in the same AWS Region where your EC2 instance is running. Check the region selector in the top-right corner of the console.
Re-run the status command to confirm the agent is still running. If it stopped, check the agent logs at /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log for error details. help me find my CWAgent metrics in CloudWatch
Your instance is now publishing five custom metrics every 60 seconds. Next up, you'll connect this data to Grafana Cloud where you can build a proper monitoring dashboard with richer visualizations than what the CloudWatch console offers.
Connect Grafana Cloud to CloudWatch
CloudWatch is collecting your instance's metrics, but its built-in dashboards offer limited visualization options. For richer graphs, gauge panels, and professional alerting, you need a dedicated monitoring platform.
In this step, you'll sign up for Grafana Cloud (free tier, no credit card required) and connect it to CloudWatch using IAM credentials. Once connected, Grafana can pull all your EC2 metrics for visualization.
In this step, get ready to:
- Create a free Grafana Cloud account and access your Grafana instance.
- Create an IAM user with read-only CloudWatch permissions for Grafana.
- Configure CloudWatch as a data source in Grafana and verify the connection.
Sign up for Grafana Cloud Free
Grafana Cloud Free gives you a fully managed Grafana instance with built-in support for CloudWatch as a data source. No plugins to install, no servers to manage.
- Open your browser and navigate to https://grafana.com/products/cloud/.
- Click Create free account.
- Sign up using your email, Google, or GitHub account. No credit card is required.
- Once your account is created, you'll land on the Grafana Cloud Portal.
- Click Launch on your Grafana instance card to open your Grafana dashboard.
What is Grafana Cloud?
Grafana Cloud is a managed observability platform. The free tier includes 10,000 metrics series, 14-day retention, and up to 3 users.
Because CloudWatch is a built-in data source, you can connect it without installing any plugins. You just need credentials with the right permissions.
Bookmark your Grafana URL
Your Grafana instance URL looks something like your-org.grafana.net. Bookmark it now so you can return to it easily in later steps.
Create an IAM user for Grafana
Grafana needs read-only access to your CloudWatch metrics. You'll create a dedicated IAM user with just enough permissions to query metrics, without any write access.
- Switch back to the AWS Management Console.
- Search for IAM in the top search bar and open the IAM service.
- In the left sidebar, click Users.
- Click Create user.
- Enter grafana-cloudwatch-user as the user name.
- Click Next.
On the permissions page, you'll attach a custom policy that grants only the CloudWatch read permissions Grafana needs.
- Select Attach policies directly.
- Click Create policy (this opens a new tab).
In the new tab, you'll define the policy using JSON.
- Click the JSON tab in the policy editor.
- Clear the default content and paste the following policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"cloudwatch:ListMetrics",
"cloudwatch:GetMetricData",
"cloudwatch:DescribeAlarms",
"ec2:DescribeInstances",
"ec2:DescribeRegions",
"tag:GetResources"
],
"Resource": "*"
}
]
}
What does this policy allow?
This policy grants read-only access to CloudWatch metrics and EC2 instance metadata. Grafana uses these permissions to discover your instances, list available metrics, and pull metric data for dashboards.
The tag:GetResources permission lets Grafana filter resources by tags, which is useful when you have many instances.
- Click Next.
- Enter GrafanaCloudWatchReadOnly as the policy name.
- Click Create policy.
Now return to the tab where you were creating the IAM user.
- Click the refresh button next to the policy search field.
- Search for GrafanaCloudWatchReadOnly and select the checkbox next to it.
- Click Next.
- Review the user details, then click Create user.
You should see a success banner confirming the user was created. Now you need to generate an access key for this user.
- Click on grafana-cloudwatch-user in the users list to open its details.
- Click the Security credentials tab.
- Scroll down to Access keys and click Create access key.
- On the best practices page, select Other as the use case.
- Click Next.
- Click Create access key.
Save your credentials now
You will see your Access key ID and Secret access key on this page. Copy both values and store them somewhere safe (a password manager or secure note). The secret key is only shown once.
You will paste these into Grafana in the next substep.
- Click Done after saving your credentials.
Don't see 'Create access key'?
Make sure you clicked into the user's detail page first (not just the users list). The Security credentials tab with the access key option is only visible on the individual user's page.
Help me find where to create an access key for my IAM user.
Add CloudWatch as a data source in Grafana
With your IAM credentials ready, you can now point Grafana at your CloudWatch data.
- Switch back to your Grafana instance in the browser.
- In the left sidebar, click Connections.
- Click Add new connection.
- Search for CloudWatch and select it from the results.
- Click Add new data source in the top right.
You'll now see the CloudWatch data source configuration form.
- Under Authentication, confirm the Authentication Provider is set to Access & secret key (this is the default).
- Paste your Access Key ID into the corresponding field.
- Paste your Secret Access Key into the corresponding field.
- Set the Default Region to the region where your EC2 instance is running (the same region you used when launching it, such as us-east-1).
Why Access & secret key?
Grafana Cloud supports multiple authentication methods for AWS. Access key authentication is the simplest for a learning project because it works immediately without cross-account trust policies or role assumption configuration.
In production, you would typically use IAM role assumption for tighter security. The monitoring patterns you learn here transfer directly regardless of auth method.
Before you click Save & Test, what do you think will happen? If your credentials and region are correct, the connection should succeed. If something is wrong (typo in the key, wrong region), you'll see an error.
- Click Save & Test.
You should see a green banner with the message Data source is working. This confirms Grafana can read your CloudWatch metrics.
Seeing an error instead?
- Double-check that you pasted the full access key ID and secret access key with no trailing spaces.
- Verify the Default Region matches the region your EC2 instance is in. Open the EC2 console to confirm.
- Confirm the GrafanaCloudWatchReadOnly policy is attached to grafana-cloudwatch-user in IAM.
Help me debug my Grafana CloudWatch connection error.
Grafana is now connected to your CloudWatch data. Next up, you'll build a multi-panel dashboard that visualizes your EC2 metrics in real time.
Build Your Monitoring Dashboard
In the previous step, you connected Grafana Cloud to your CloudWatch data source and confirmed the connection works.
Now it is time to turn that raw data into something visual. Production teams rely on a single dashboard to monitor all critical metrics at a glance, so they can correlate issues immediately. High CPU combined with high memory usage, for example, signals an overloaded instance.
In this step, get ready to:
- Create a Grafana dashboard with a CPU Utilization time series panel.
- Add memory, disk, and network panels to the dashboard.
- Arrange panels in a 2x2 grid and enable auto-refresh.
Create the CPU utilization panel
Every dashboard starts with a single panel. You will create a new dashboard and add a Time Series line graph that pulls CPU utilization directly from CloudWatch.
- In your Grafana Cloud instance, click Dashboards in the main menu on the left.
- Click New and select New Dashboard.
- Click the Add new element icon (blue plus sign) and click Panel to add a new panel to the dashboard.
- On the new panel, click Configure visualization to open the panel editor.
The panel editor opens with a default data source pre-selected. You need to point it at CloudWatch and query the CPU metric.
- In the Data source drop-down at the top of the query editor, select your CloudWatch data source.
- Set Namespace to AWS/EC2.
- Set Metric name to CPUUtilization.
- Under Dimensions, add InstanceId and select your EC2 instance's ID from the drop-down.
- Click Run queries to preview the data in the panel.
You should see a line graph plotting your instance's CPU usage over time. Now set the visualization type and title.
- In the visualization picker on the right, confirm Time series is selected. If not, click All visualizations and choose Time series.
- Enter CPU Utilization as the panel title.
- Click Save in the top-right corner of the panel editor.
- When prompted, enter EC2 Health Monitor as the dashboard title.
- Click Save.
Your dashboard now has one panel showing a live CPU utilization line graph.
What is a Time Series panel?
A Time Series panel plots data points over time as a line graph. It is ideal for metrics like CPU utilization where you want to see trends, spikes, and patterns across minutes or hours.
Seeing 'No data' in the panel?
Make sure you selected the correct Namespace (AWS/EC2) and that the InstanceId dimension matches your running EC2 instance. Also verify your data source's region matches the region where your instance is running.
help me troubleshoot why my Grafana CloudWatch panel shows no data
Add memory, disk, and network panels
One metric is useful, but real monitoring requires visibility across CPU, memory, disk, and network together. You will add three more panels to your dashboard.
Start with the Memory Used Percent gauge.
- Return to the dashboard view by clicking Back to dashboard if you are still in the panel editor.
- Click Edit in the top-right corner of the dashboard.
- Click the Add new element icon (blue plus sign) and click Panel.
- On the new panel, click Configure visualization.
- Set the Data source to CloudWatch.
- Set Namespace to CWAgent.
- Set Metric name to mem_used_percent.
- Under Dimensions, add InstanceId and select your instance.
- Click Run queries.
Now change the visualization type to a Gauge and configure thresholds.
- Click All visualizations in the panel editor sidebar and select Gauge.
- Scroll down to the Thresholds section in the panel options.
- Click + Add threshold.
- Enter 70 as the value. Click the colored circle to open the color picker and select yellow.
- Click + Add threshold again.
- Enter 90 as the value and set its color to red.
- Enter Memory Used Percent as the panel title.
- Click Save in the top-right corner, then click Save on the confirmation dialog.
What do the thresholds do?
Thresholds color the gauge based on the metric's current value. Below 70% the gauge stays green (healthy). Between 70% and 90% it turns yellow (warning). Above 90% it turns red (critical). This gives you an instant visual signal about memory pressure.
Next, add the Disk Used Percent gauge.
- Click Back to dashboard, then click the Add new element icon and add another Panel.
- Click Configure visualization.
- Set Data source to CloudWatch, Namespace to CWAgent, Metric name to disk_used_percent.
- Add the InstanceId dimension and select your instance.
- Click Run queries.
- Select Gauge from All visualizations.
- Enter Disk Used Percent as the panel title.
- Click Save, then Save again on the dialog.
Finally, add the Network In/Out time series panel with two queries on the same graph.
- Click Back to dashboard, click the Add new element icon, and add another Panel.
- Click Configure visualization.
- Set Data source to CloudWatch, Namespace to AWS/EC2, Metric name to NetworkIn.
- Add the InstanceId dimension and select your instance.
- In the sidebar, click the blue plus sign next to Queries & Expressions and select Add query.
- On the second query, set Namespace to AWS/EC2 and Metric name to NetworkOut.
- Add the InstanceId dimension on the second query as well.
- Click Run queries. You should see two lines on the same graph.
- Confirm Time series is selected as the visualization.
- Enter Network In/Out as the panel title.
- Click Save, then Save on the dialog.
Why two queries on one panel?
Placing NetworkIn and NetworkOut on the same Time Series panel lets you visually compare inbound and outbound traffic side by side. If one spikes without the other, you can quickly tell whether the instance is receiving a burst of requests or sending a large response.
Second query showing no data?
Make sure the second query also has the InstanceId dimension set. Each query is independent. If the dimension is missing, Grafana cannot filter to your specific instance.
help me fix my second NetworkOut query in Grafana
Arrange the panels and enable auto-refresh
Your four panels are on the dashboard but may not be arranged neatly. Arranging them in a 2x2 grid and enabling auto-refresh turns this into a live monitoring display.
- Click Back to dashboard if you are still in the panel editor.
- Click Edit in the top-right corner of the dashboard.
- Drag the panels to arrange them in a 2x2 grid: CPU Utilization (top-left), Memory Used Percent (top-right), Disk Used Percent (bottom-left), Network In/Out (bottom-right).
- Resize each panel by dragging its bottom-right corner so all four panels are roughly equal in size.
- Click Save to save the layout.
Now enable auto-refresh so the dashboard updates with live data every minute.
- In the dashboard header, find the Refresh dashboard icon (circular arrow) near the top-right.
- Click the small down arrow next to the refresh icon to open the interval picker.
- Select 1m from the list.
- Click Save to persist the auto-refresh setting.
Before you check the final result, what do you expect to see on the dashboard now? Think about what each panel type looks like with live data.
Look at your dashboard. You should see 4 panels updating with live data: a CPU Utilization line graph, a Memory Used Percent gauge, a Disk Used Percent gauge, and a Network In/Out line graph with two lines. The dashboard auto-refreshes every minute.
Why a 1-minute refresh?
Your CloudWatch Agent collects metrics every 60 seconds (the metrics_collection_interval you configured earlier). Refreshing the dashboard more frequently than that would just re-fetch the same data. A 1-minute refresh aligns with your data collection cadence.
Panels not updating automatically?
Make sure you selected 1m from the refresh interval dropdown (not just clicked the refresh icon once). The interval picker is the small arrow beside the refresh button, not the button itself. After saving, you should see a countdown timer next to the refresh icon.
help me enable auto-refresh on my Grafana dashboard
Your monitoring dashboard is live. You can now see CPU, memory, disk, and network health for your EC2 instance at a glance, updating every minute. Next up, you will create an alert rule and stress test the instance to prove your monitoring actually catches problems in real time.
Create an Alert and Stress Test
Your dashboard gives you a live view of your EC2 instance's health. But a dashboard only works if someone is watching it.
Real monitoring alerts you proactively when something goes wrong. In this step, you'll create an alert rule in Grafana and then deliberately trigger it to prove the full alerting loop works.
In this step, get ready to:
- Create an alert rule that fires when CPU usage exceeds 80%.
- Stress test your EC2 instance to spike CPU to 100%.
- Verify the alert transitions from Normal to Firing and back.
Create a high CPU alert rule
Grafana's alerting system evaluates a metric query on a schedule. When the metric breaches a threshold for a specified duration, the alert transitions from "Normal" to "Firing" state.
- In Grafana, click Alerts & IRM in the left sidebar.
- Click Alert rules.
- Click + New alert rule.
- Enter High CPU Alert as the alert rule name at the top of the page.
- In the query section, select your CloudWatch data source.
- Set Namespace to AWS/EC2.
- Set Metric name to CPUUtilization.
- Set Statistic to Average.
What are these query settings doing?
You're telling Grafana to pull the average CPU utilization from CloudWatch every evaluation cycle. This is the same metric you already see on your dashboard's CPU panel, but now Grafana will automatically check it against a threshold.
- Set Period to 60s.
- In the alert condition, set the threshold to Is above 80.
- Scroll down to the evaluation behavior section. Create a new evaluation group or select an existing one, and set the Evaluation interval to 1m.
- Set the pending period to 1m so the alert must breach for 1 minute before firing.
- Click Save rule and exit in the top right corner.
You should see your new High CPU Alert listed in the alert rules page with a "Normal" state.
Why 80% as the threshold?
In production, sustained CPU above 80% often indicates an overloaded instance that could degrade response times. The 80% threshold is a common starting point for infrastructure alerts because it gives your team time to respond before the instance hits 100% and drops requests.
Don't see your alert rule listed?
Make sure you clicked Save rule and exit rather than just closing the browser tab. Check the folder you saved it to by expanding folders in the alert rules list.
If the rule shows an error state, verify that the CloudWatch data source is still working by navigating back to Connections and clicking Save & Test on your CloudWatch data source.
Help me troubleshoot my Grafana alert rule
Stress test your EC2 instance
Now you need to prove the alert actually fires. You'll install a stress-testing tool on your instance and deliberately peg the CPU at 100%.
- SSH into your EC2 instance by running this command:
ssh -i [[PEMPATH="path to your .pem key file"]] ec2-user@[[EC2_PUBLIC_IP="your instance public IPv4 address"]]
- Install the stress tool by running:
sudo yum install -y stress
You should see output ending with "Complete!" confirming the package installed.
Package not found?
The stress package is included in Amazon Linux 2023 repositories by default. If you see an error, try running sudo yum clean all first, then retry the install command.
Help me install stress on Amazon Linux 2023
Before you run the stress test, think about what you expect to happen. Do you think the alert will fire immediately, or will there be a delay?
- Run a CPU stress test for 3 minutes by running:
stress --cpu 1 --timeout 180
What does this command do?
The --cpu 1 flag spawns one worker that continuously calculates square roots, pegging the CPU at 100%. The --timeout 180 flag stops the test automatically after 180 seconds (3 minutes).
Your terminal will appear to hang while the stress test runs. This is normal. The command will print a completion message after 3 minutes.
Watch the alert fire
While the stress test runs on your instance, switch back to your Grafana dashboard to watch what happens in real time.
- Open your EC2 Health Monitor dashboard in Grafana.
- Watch the CPU Utilization panel. Within 1-2 minutes, you should see the line spike toward 100%.
- Navigate to Alerts & IRM in the left sidebar, then click Alert rules.
- Find your High CPU Alert rule. Within 2-3 minutes of the stress test starting, its state should change from Normal to Firing.
Why does it take 2-3 minutes?
CloudWatch collects the CPUUtilization metric at 5-minute intervals for basic monitoring (or 1-minute with detailed monitoring). Then Grafana evaluates the rule every 1 minute and requires the condition to persist for the 1-minute pending period before transitioning to Firing.
- Wait for the stress test to finish (3 minutes total). Your terminal will print a message like stress: info: [xxxx] successful run completed.
- Return to the alert rules page in Grafana. After another evaluation cycle (1-2 minutes after the test ends), the alert should transition back to Normal.
You've just proven the full monitoring loop works. Your dashboard detected the spike, your alert fired, and it recovered automatically once the problem resolved.
Alert stuck in Normal during the stress test?
CloudWatch basic monitoring reports CPU every 5 minutes. If you launched your instance with basic monitoring, the metric may not update fast enough. Wait the full 5 minutes and check again.
Also verify that the InstanceId dimension in your alert query matches your running instance. You can find your instance ID in the EC2 console.
Help me troubleshoot why my Grafana alert didn't fire
Secret mission
Add CloudWatch Logs to Your Dashboard
Your dashboard shows metrics, but metrics only tell you something is wrong. Logs tell you why. In this secret mission, you'll configure the CloudWatch Agent to ship system logs, grant Grafana the right permissions, and add a live logs panel so you can correlate metric spikes with log events in a single view.
Clean Up Your Resources
Clean Up Your Resources
Decide whether to keep your resources running, pause them to come back later, or delete them entirely. The EC2 instance is covered by AWS Free Tier (750 hours/month for 12 months), so it costs nothing while your account remains eligible.
Cost warning
If you leave your EC2 instance running beyond your Free Tier allowance, it costs approximately $0.0116/hour. If you stop the instance (Pause option), compute charges stop but EBS storage charges of approximately $0.10/GB/month still apply to the root volume.
Resources you used:
- EC2 instance (t2.micro, Amazon Linux 2023).
- IAM Role: CloudWatchAgentRole.
- IAM User: grafana-cloudwatch-user.
- Security Group (SSH access on port 22).
- CloudWatch Log Group: /ec2/system-logs.
- CloudWatch custom metrics (CWAgent namespace).
- Grafana Cloud Free account.
Keep everything running
No action needed. Choose this if you are still actively building or experimenting with your monitoring dashboard.
- Monitor your Free Tier usage in the AWS Billing console to avoid surprise charges.
- Your Grafana Cloud Free account has no ongoing charges. The dashboard and alert rule remain accessible anytime you log in.
- Custom metrics in the CWAgent namespace are free for the first 10 metrics.
Pause - I'll come back to this later
Shut down the running instance to stop compute charges while keeping all your files and configuration intact.
- In the EC2 console, select your instance, click Instance state, then Stop instance.
- Compute charges stop immediately. EBS storage charges of approximately $0.10/GB/month continue on the root volume.
- The CloudWatch Agent stops collecting metrics while the instance is stopped. Your Grafana dashboard will show a gap in data until you restart.
- To resume later, select the instance and click Instance state then Start instance. The agent resumes automatically.
Delete - I don't want to use this again
Remove all project resources and start fresh. Follow these steps in order.
- Terminate the EC2 instance: in the EC2 console, select your instance, click Instance state, then Terminate instance. Confirm the termination. This permanently deletes the instance and its root volume.
- Delete the IAM Role: open the IAM console, choose Roles, search for CloudWatchAgentRole, select it, and click Delete. Enter the role name to confirm. The console automatically detaches the managed policy and deletes the instance profile.
- Delete the IAM User: in the IAM console, choose Users, select grafana-cloudwatch-user, and click Delete. Confirm the deletion.
- Delete the Security Group: in the EC2 console, go to Security Groups, select the security group you created for SSH access, click Actions, then Delete security groups. Confirm the deletion.
- Delete the CloudWatch Log Group: in the CloudWatch console, choose Log groups in the left navigation, select /ec2/system-logs, click Actions, then Delete log group(s). Confirm.
- Delete the Grafana dashboard and alert: in your Grafana instance, go to Dashboards, find EC2 Health Monitor, click the three-dot menu, and select Delete. Then go to Alerting and delete the High CPU Alert rule.
- Your Grafana Cloud Free account has no ongoing costs. You can leave it active for future projects or log out and ignore it.
Nice Work!
Nice Work!
You just built a production-grade monitoring dashboard that gives you real-time visibility into your EC2 instance's health. From launching the instance to watching an alert fire during a live stress test, you've proven you can set up the same observability stack that ops teams rely on every day.
Here's what you learned:
- Installed and configured the CloudWatch Agent to collect memory, CPU, and disk metrics that AWS doesn't provide by default.
- Connected Grafana Cloud to AWS CloudWatch as a data source and built a multi-panel dashboard with time series and gauge visualizations.
- Created an alert rule and validated it fires by stress-testing your instance, proving the full monitoring loop from spike to notification.
- Secret Mission: Added CloudWatch Logs streaming to create a complete observability dashboard with metrics and logs in one view.
Ready to quiz yourself?