AIOps Anomaly Detection Pipeline

Build an ML-powered monitoring pipeline with CloudWatch and Grafana Cloud.

Introduction

30 Second Summary

When a website goes down or an app starts running slow, someone has to notice before customers do. In the real world, engineering teams rely on intelligent monitoring systems that use machine learning to spot unusual behavior automatically and send alerts the moment something goes wrong.

In this project, you will build an AIOps anomaly detection pipeline that monitors a cloud server using AWS 's ML-powered anomaly detection, visualizes everything in a professional Cloud dashboard, and sends you email alerts when something unusual happens. You will even simulate a real incident and watch your pipeline detect it in real time.

What You'll Build

A complete observability pipeline that collects infrastructure metrics from an instance, applies machine learning-based anomaly detection via CloudWatch, triggers automated email alerts through and visualizes everything in a Grafana Cloud dashboard.

By the end of this project, you'll have:

  • A running EC2 instance pushing metrics to CloudWatch with an ML-powered anomaly detection model actively monitoring CPU behavior.
  • A Grafana Cloud dashboard visualizing EC2 health metrics with anomaly detection bands, connected live to your AWS account.
  • A working alert pipeline where a simulated CPU spike triggers CloudWatch anomaly detection, fires an SNS email notification, and appears in your Grafana dashboard in real time.
  • An AIOps architecture diagram mapping how enterprise tools like BigPanda and ServiceNow ITOM extend this pipeline for production use.
  • Secret Mission: add a custom application-level metric with its own anomaly detector and Grafana panel, demonstrating full-stack observability.

Are there any prerequisites?

You need an AWS account with the configured and basic familiarity with Linux command line. You will also create a free Grafana Cloud account during the project. No credit card is required for Grafana Cloud.

Before We Start

Before We Start...

Launch and Connect to Your EC2 Instance

Before you can monitor anything, you need something to monitor. In this project, that something is an EC2 instance, a virtual server running in the AWS cloud.

In this step, you will launch an EC2 instance, connect to it via SSH, and give it the permissions it needs to send metrics to CloudWatch.

In this step, get ready to:
  • Launch a free-tier EC2 instance running Amazon Linux 2023.
  • SSH into the instance and verify the AWS CLI is available.
  • Attach an IAM role so the instance can push metrics to CloudWatch.
Launch a free-tier EC2 instance

Your EC2 instance is the infrastructure your AIOps pipeline will monitor. You will launch a small, free-tier eligible instance and tag it so it is easy to find later.

  • Open the AWS Management Console at https://console.aws.amazon.com and sign in.
  • In the top navigation bar, confirm you are in your preferred AWS Region (for example, US East (N. Virginia)).
  • Search for EC2 in the top search bar and select EC2 from the results.
  • Click Launch instance on the EC2 Dashboard.

What is an EC2 instance?

An EC2 instance is a virtual server you rent from AWS. You choose the operating system, the size (CPU and memory), and the region it runs in. For this project, you will use a tiny instance that fits within the AWS Free Tier.

  • Under Name and tags, enter AIOps-Lab as the instance name.
  • Under Application and OS Images (Amazon Machine Image), click Quick Start and select Amazon Linux.
  • Make sure the Amazon Linux 2023 AMI is selected. It should be the default option.
  • Under Instance type, select t2.micro.

Don't see t2.micro?

If t2.micro is not available in your region, select t3.micro instead. Both are free-tier eligible. If your AWS account was created on or after July 15, 2025, your free-tier eligible options may include t3.micro, t3.small, t4g.micro, or t4g.small. Look for the "Free tier eligible" label next to the instance type.

  • Under Key pair (login), select an existing key pair from the dropdown.

I have a key pair

Select your existing key pair from the dropdown and continue to the next step.

I need to create one

  • Click Create new key pair.
  • Enter AIOps-Lab-Key as the key pair name.
  • Leave the key pair type as RSA and the file format as .pem.
  • Click Create key pair.
  • Your browser downloads a .pem file. Save it somewhere safe. You will need it to connect via SSH.

Do not lose your key pair file

The .pem file is your only way to SSH into the instance. If you lose it, you will not be able to connect. Save it to a known location on your computer.

  • Under Network settings, confirm that Allow SSH traffic from is checked.
  • Change the SSH source dropdown to My IP. This restricts SSH access to your current IP address for security.
  • Leave all other settings as their defaults.
  • Click Launch instance in the Summary panel on the right.

You should see a success message confirming your instance is launching.

  • Click View all instances to go to the Instances page.
  • Wait until the Instance state column shows Running and the Status check column shows 2/2 checks passed. This can take 1-2 minutes.

Instance stuck in Pending?

Give it a minute. If it moves to Terminated instead of Running, check that you selected a free-tier eligible instance type and that your region supports it.

Help me troubleshoot my EC2 instance not launching.

SSH into the instance

Now that your instance is running, you need to connect to it. SSH gives you a terminal session directly on the instance, just like opening Terminal on your own computer.

  • On the Instances page, click on your AIOps-Lab instance to open its details.
  • In the Instance summary section, find the Public IPv4 address field and copy the IP address.
  • Also note the Instance ID (it starts with i-). You will need this later for CloudWatch.

Open a terminal on your local computer.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.

Before connecting, you need to set the correct permissions on your key pair file. This prevents SSH from rejecting the connection.

macOS

  • Set the file permissions on your key pair by running this command:
chmod 400 [[PEMPATH="path/to/your-key.pem"]]
  • Connect to your instance by running this command:
ssh -i [[PEMPATH="path/to/your-key.pem"]] ec2-user@[[EC2_PUBLIC_IP="your EC2 public IPv4 address"]]

Windows

  • Connect to your instance by running this command in PowerShell:
ssh -i [[PEMPATH="pathtoyour-key.pem"]] ec2-user@[[EC2_PUBLIC_IP="your EC2 public IPv4 address"]]

Pro tip: key permissions on Windows

If you get a permissions error, right-click the .pem file, go to Properties > Security > Advanced, disable inheritance, remove all users except yourself, and try again.

What is ec2-user?

Amazon Linux 2023 creates a default user called ec2-user when the instance launches. This is the user you SSH in as. It has sudo privileges, so you can run administrative commands.

SSH will ask you to confirm the host fingerprint the first time you connect. Type yes and press Enter.

You should see a welcome banner and a command prompt that looks like [ec2-user@ip-... ~]$.

  • Verify you are connected by running this command:
whoami

You should see ec2-user printed in the terminal.

  • Verify the AWS CLI is available by running:
aws --version

You should see output starting with aws-cli/2. followed by a version number. Amazon Linux 2023 comes with the AWS CLI pre-installed.

Connection refused or timed out?

  • Double-check that you copied the correct Public IPv4 address from the EC2 console (not the private IP).
  • Make sure your security group allows SSH (port 22) from your IP. If your IP changed since you launched the instance, update the inbound rule.
  • Confirm the instance state is Running and status checks show 2/2 checks passed.
  • Help me troubleshoot my SSH connection to EC2.
Configure IAM permissions for the instance

Your EC2 instance needs permission to send metrics to CloudWatch and interact with SNS. Instead of storing credentials on the instance, you will attach an IAM role. The instance automatically receives temporary credentials through this role.

What is an IAM role?

An IAM role is a set of permissions that AWS services can assume. When you attach a role to an EC2 instance, any application running on that instance can use the role's permissions without needing hardcoded access keys. The credentials rotate automatically, making this the secure way to grant AWS access.

  • Open the IAM console by searching for IAM in the top search bar and selecting IAM.
  • In the left navigation pane, click Roles.
  • Click Create role.
  • Under Trusted entity type, select AWS service.
  • Under Service or use case, select EC2.
  • Click Next.
  • In the Permissions policies search box, type CloudWatchFullAccess.
  • Select the checkbox next to CloudWatchFullAccess.
  • Click Next.
  • Under Role name, enter AIOps-EC2-CloudWatch-Role.
  • Click Create role.

You should see a success banner confirming the role was created. Now you need to attach it to your running instance.

  • Go back to the EC2 console by searching for EC2 in the top search bar.
  • In the left navigation pane, click Instances.
  • Select your AIOps-Lab instance by clicking its checkbox.
  • Click Actions in the top menu bar.
  • Select Security, then Modify IAM role.
  • In the IAM role dropdown, select AIOps-EC2-CloudWatch-Role.
  • Click Update IAM role.

You should see a success message confirming the IAM role was attached to your instance.

  • Switch back to your SSH terminal and verify the role works by running this command:
aws cloudwatch list-metrics --namespace AWS/EC2 --region [[AWS_REGION="your AWS region, e.g. us-east-1"]]

What does this command do?

  • aws cloudwatch list-metrics asks CloudWatch to list all available metrics.
  • --namespace AWS/EC2 filters to only EC2-related metrics.
  • --region tells the CLI which AWS region to query. Use the same region where you launched your instance.

You should see JSON output listing metrics like CPUUtilization, NetworkIn, and DiskReadOps. If you see this output without any errors, your IAM role is working correctly.

Why not just use access keys?

You could paste AWS access keys directly onto the instance, but that is a security risk. If someone gains access to the instance, they get your permanent credentials. IAM roles provide temporary, automatically rotating credentials instead. This is the AWS-recommended approach for granting permissions to EC2 instances.

🙋‍♀️ Seeing an access denied error?

  • Make sure you selected the correct instance when attaching the role. Go back to Actions > Security > Modify IAM role and confirm AIOps-EC2-CloudWatch-Role is selected.
  • It can take a minute for the role to propagate. Wait 30 seconds and try the command again.
  • Verify the --region flag matches the region where your instance is running. You can see your region in the top-right corner of the AWS Console.
  • Help me fix my IAM role permissions on EC2.

Your EC2 instance is live, connected, and has the permissions it needs. Next up, you will enable CloudWatch's ML-powered anomaly detection to start learning what "normal" looks like for your instance.

Enable CloudWatch Anomaly Detection on EC2 Metrics

Your EC2 instance is running and pushing metrics to CloudWatch automatically. Now you need to teach CloudWatch what "normal" looks like for your instance's CPU behavior.

CloudWatch Anomaly Detection uses machine learning to analyze your metric's historical patterns. It builds a model of expected values and displays them as a gray band on the metric graph. Any value that falls outside this band is flagged as anomalous. This is the "AI" in AIOps.

In this step, get ready to:
  • Explore your EC2 instance's CPU metrics in the CloudWatch console.
  • Enable ML-powered anomaly detection on the CPUUtilization metric.
  • Verify the anomaly detector using the AWS CLI from your instance.
Explore EC2 metrics in CloudWatch

CloudWatch automatically collects CPU, network, and disk metrics from every EC2 instance every 5 minutes. Before enabling anomaly detection, take a look at what CloudWatch is already tracking.

  • In the AWS Console, type CloudWatch in the top search bar and select CloudWatch.
  • In the left navigation pane, click Metrics, then All metrics.
  • In the Browse tab, select the EC2 namespace.
  • Select Per-Instance Metrics.
  • In the search field, enter your instance ID (the one from Step 1) and press Enter.
  • Select the checkbox next to the CPUUtilization metric row for your instance.

You should see a graph appear showing the CPU usage of your instance over time. The line will be mostly flat and low since your instance is idle.

What is CPUUtilization?

CPUUtilization measures the percentage of allocated compute units that your instance is currently using. CloudWatch collects this metric automatically every 5 minutes with basic monitoring. An idle instance typically shows values near 0-2%.

Enable anomaly detection on CPUUtilization

With static thresholds, you would need to manually decide what CPU percentage counts as "too high." Anomaly detection removes that guesswork. CloudWatch's ML model learns the metric's hourly, daily, and weekly patterns and creates a dynamic baseline that adapts over time.

Why anomaly detection instead of a static threshold?

A static threshold like "alert when CPU exceeds 80%" ignores context. A batch job that runs every night at 90% CPU is normal. A web server that suddenly jumps to 50% at 3 AM is suspicious. Anomaly detection captures these patterns automatically, which is the core principle behind AIOps.

  • With the CPUUtilization metric still graphed, click the Graphed metrics tab below the graph.
  • Find the Actions column in the metrics table. Click the anomaly detection icon (it looks like a small pulse/heartbeat) next to your CPUUtilization metric.

A new row labeled ANOMALY_DETECTION_BAND appears under your metric. A gray band appears on the graph representing the range of values CloudWatch considers "normal."

  • In the Details column for the ANOMALY_DETECTION_BAND row, check that the threshold value at the end of the formula is 2. This is the default.

What does the threshold number mean?

The threshold controls the width of the "normal" band. A higher number creates a wider band, meaning CloudWatch tolerates more variation before flagging something as anomalous. A lower number creates a narrower band that triggers on smaller deviations. The default value of 2 is a good starting point for this project.

💡 Don't see the gray band yet?

For new models, the anomaly detection band can take up to 3 hours to fully appear on the graph. You may see a preview band immediately, which is an approximation. The model trains on up to two weeks of historical data. Since your instance is new, the band will be narrow at first and become more accurate over time. You can continue with the project while it trains.

Verify the anomaly detector using the CLI

Observability engineers often manage anomaly detectors programmatically using the AWS CLI. This is how you would automate anomaly detection setup across dozens of instances in a production environment.

  • SSH into your EC2 instance (if you are not already connected).
  • Create the anomaly detector programmatically (or verify the one you just created in the console) by running this command:
aws cloudwatch put-anomaly-detector \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=[[EC2_INSTANCE_ID="your EC2 instance ID"]] \
  --stat Average \
  --region [[AWS_REGION="your AWS region e.g. us-east-1"]]

What does this command do?

  • put-anomaly-detector creates (or updates) an anomaly detection model for a specific CloudWatch metric.
  • --namespace AWS/EC2 tells CloudWatch to look in the EC2 metrics namespace.
  • --dimensions scopes the detector to your specific instance using its ID.
  • --stat Average tells the model to analyze the average CPU value for each data point.

This command produces no output on success. If you already created the detector via the console, running this command is safe. It will update the existing detector.

  • Verify the anomaly detector was created by running this command:
aws cloudwatch describe-anomaly-detectors \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --region [[AWS_REGION="your AWS region e.g. us-east-1"]]

What should I see?

You should see JSON output containing your anomaly detector with a StateValue field. The value will be one of these:

  • PENDING_TRAINING means the model is still being built.
  • TRAINED_INSUFFICIENT_DATA means the model is trained but needs more historical data for full accuracy.
  • TRAINED means the model is fully trained.

For a brand new instance, you will most likely see PENDING_TRAINING or TRAINED_INSUFFICIENT_DATA. This is expected.

🙋‍♀️ Seeing an access denied error?

  • Make sure the IAM role you attached to your instance in Step 1 has the CloudWatchFullAccess policy. Go to IAM > Roles, find your role, and check its attached policies.
  • Confirm the IAM role is actually attached to your instance. In EC2, select your instance and check the IAM role field in the instance details.
  • Double check that the --region flag matches the region where your EC2 instance is running.

Still stuck? Help me troubleshoot the access denied error when running CloudWatch CLI commands from my EC2 instance.

Your anomaly detection model is now training on your instance's CPU behavior. Next up, you will create an SNS email alert and a CloudWatch alarm that fires when the CPU breaks out of the anomaly band.

Set Up SNS Email Alerts and a CloudWatch Anomaly Alarm

Your anomaly detection model is now learning what "normal" looks like for your EC2 instance's CPU. But detection alone is not enough.

In a real production environment, you need to be notified the moment something goes wrong. In this step, you will wire up Amazon SNS to send email alerts and create a CloudWatch alarm that fires whenever CPU behavior leaves the anomaly detection band.

In this step, get ready to:
  • Create an SNS topic and subscribe your email to receive anomaly alerts.
  • Create a CloudWatch alarm that uses anomaly detection to trigger notifications.
  • Verify the alarm is active and ready to detect your simulated incident.
Create an SNS topic and email subscription

An SNS topic is a messaging channel. You publish a message to the topic, and SNS delivers it to every subscriber. You will create a topic for anomaly alerts and subscribe your email address to it.

  • In the AWS Console search bar, type SNS and select Simple Notification Service.
  • In the left navigation pane, click Topics.
  • Click Create topic.
  • For Type, select Standard.
  • For Name, enter AIOps-Anomaly-Alerts.
  • Scroll down and click Create topic.

You should see the topic details page with the topic's ARN displayed at the top.

What is an SNS topic?

Think of an SNS topic as a broadcast channel. When CloudWatch detects an anomaly, it publishes a message to this topic. SNS then delivers that message to every subscriber, whether that is an email address, an SMS number, or even a Lambda function. One topic can fan out to many destinations at once.

Now you need to subscribe your email so you actually receive the alerts.

  • On the AIOps-Anomaly-Alerts topic details page, click Create subscription.
  • For Protocol, select Email.
  • For Endpoint, enter your email address.
  • Click Create subscription.

Confirm your email subscription now

Check your email inbox for a message from AWS Notifications. Open it and click the Confirm subscription link. SNS cannot deliver alerts to your email until you confirm. If you do not see the email, check your spam folder.

  • After confirming, go back to the SNS console and refresh the page.
  • Your subscription's Status should now show Confirmed.

Subscription still showing Pending confirmation?

Make sure you check the spam or junk folder of the email address you entered. The confirmation email comes from no-reply@sns.amazonaws.com. If you still do not see it after a few minutes, delete the subscription and create a new one with a different email address.

Help me troubleshoot my SNS email subscription

Create a CloudWatch alarm based on anomaly detection

Now you will connect the anomaly detection model from Step 2 to the SNS topic you just created. The alarm watches your CPU metric and fires a notification whenever the value leaves the expected band.

  • In the AWS Console search bar, type CloudWatch and select CloudWatch.
  • In the left navigation pane, click Alarms, then All alarms.
  • Click Create alarm.
  • Click Select metric.

A metric browser opens. You need to find the CPUUtilization metric for your specific EC2 instance.

  • Click EC2 under the AWS namespaces section.
  • Click Per-Instance Metrics.
  • In the search box, type your EC2 instance ID to filter the list.
  • Select the checkbox next to CPUUtilization for your instance.
  • Click Select metric.

The "Specify metric and conditions" page appears with a graph of your CPU metric.

  • Under Conditions, select Anomaly detection as the threshold type.
  • For the "Whenever CPUUtilization is" dropdown, select Outside of the band.
  • For Anomaly detection threshold, enter 2.

What does the anomaly detection threshold control?

The threshold number controls how wide the "normal" band is. A higher number creates a thicker band that tolerates more variation. A lower number creates a thinner band that triggers on smaller deviations. The value 2 is the default and works well for this demo.

  • Click Additional configuration to expand it.
  • For Datapoints to alarm, set both values to 1 out of 1. This makes the alarm trigger immediately for demo purposes.
  • Click Next.

You are now on the "Configure actions" page. This is where you connect the alarm to your SNS topic.

  • Under Notification, make sure In alarm is selected as the alarm state trigger.
  • For the SNS topic, select Select an existing SNS topic.
  • Choose AIOps-Anomaly-Alerts from the dropdown.
  • Click Next.

You are now on the "Add name and description" page.

  • For Alarm name, enter AIOps-CPU-Anomaly-Alarm.
  • Click Next.

You are now on the "Preview and create" page.

  • Review the alarm configuration to confirm the metric, conditions, and notification settings are correct.
  • Click Create alarm.

Why use anomaly detection instead of a static threshold?

A static threshold alarm fires when CPU exceeds a fixed number like 80%. The problem is that "normal" changes over time. A batch job might push CPU to 70% every night, which is expected. An anomaly detection alarm learns these patterns and only fires when behavior is truly unusual. This is the core "AI" in AIOps.

🙋‍♀️ Don't see your SNS topic in the dropdown?

Make sure you created the SNS topic in the same AWS region as your EC2 instance. SNS topics are region-specific. If the regions do not match, the topic will not appear in the CloudWatch alarm configuration.

Help me find my SNS topic when creating a CloudWatch alarm

Verify the alarm is active
  • In the CloudWatch left navigation pane, click Alarms, then All alarms.
  • Find AIOps-CPU-Anomaly-Alarm in the list.

Your alarm should show a state of OK or INSUFFICIENT_DATA.

Why does my alarm show INSUFFICIENT_DATA?

This is expected. The anomaly detection model needs time to train on your metric's historical data. For new models, it can take up to 3 hours for the anomaly detection band to appear. The alarm will transition to OK once enough data is available, and to ALARM when you simulate an incident in a later step.

Your alert pipeline is now fully wired. When CPU behavior breaks out of the learned "normal" band, CloudWatch triggers the alarm and SNS delivers an email straight to your inbox. Next up, you will connect Grafana Cloud to CloudWatch so you can visualize everything in a professional dashboard.

Connect Grafana Cloud to AWS CloudWatch

In the previous step, you set up an automated alert pipeline that sends you an email when CloudWatch detects an anomaly. That covers the alerting side of observability.

Now you need a way to actually see your metrics in real time. Grafana Cloud is a managed observability platform that connects to data sources like CloudWatch and displays your metrics in professional dashboards. In this step, you will create a free Grafana Cloud account, set up a dedicated IAM user for Grafana to read your AWS data, and wire everything together.

In this step, get ready to:
  • Sign up for a free Grafana Cloud account and open the Grafana UI.
  • Create an IAM user with read-only CloudWatch permissions and generate access keys.
  • Add CloudWatch as a data source in Grafana and verify the connection.
Sign up for Grafana Cloud

Grafana Cloud has a free-forever plan that includes 10k metric series, 14-day retention, and 3 active users. No credit card required.

  • Open your browser and navigate to grafana.com.
  • Click Create free account in the top right corner.
  • Sign up using your email, Google, GitHub, Microsoft, or Amazon account.
  • Follow the prompts to finish creating your account.

After your account is created, you land on the Grafana Cloud portal. This is the management page for your Grafana stack.

  • Click Launch on your Grafana stack to open the Grafana UI.

You should see the Grafana home screen with a left sidebar containing options like Dashboards, Explore, and Connections.

Don't see the Launch button?

Your Grafana stack may take a minute to provision after account creation. Refresh the portal page and look for a card labeled with your stack name. The Launch button appears on that card once provisioning is complete.

Help me find the Launch button on my Grafana Cloud portal.

Create an IAM user for Grafana with read-only access

Grafana Cloud needs credentials to read your CloudWatch metrics. You will create a dedicated IAM user with read-only permissions so Grafana can query your data without being able to modify anything in your AWS account.

  • In the AWS Console, type IAM in the top search bar and select IAM.
  • Click Users in the left navigation pane.
  • Click Create user.
  • Enter grafana-cloudwatch-reader as the user name.
  • Click Next.
  • On the Set permissions page, select Attach policies directly.
  • Search for CloudWatchReadOnlyAccess and check the box next to it.
  • Search for CloudWatchLogsReadOnlyAccess and check the box next to it.
  • Click Next.
  • Review the user details and click Create user.

Why a separate IAM user for Grafana?

The IAM role you attached to your EC2 instance in Step 1 has full CloudWatch access because the instance needs to write metrics. Grafana only needs to read metrics, so you create a separate user with read-only permissions. This follows the principle of least privilege.

Now you need to generate access keys for this user so Grafana can authenticate.

  • Click on the user name grafana-cloudwatch-reader to open the user details page.
  • Select the Security credentials tab.
  • In the Access keys section, click Create access key.
  • On the Access key best practices & alternatives page, select Third-party service as the use case.
  • Check the confirmation checkbox at the bottom of the page.
  • Click Next.
  • Click Create access key.

You will see your Access Key ID and Secret Access Key on the screen. This is the only time you can view the secret access key.

  • Click Show to reveal the Secret Access Key.
  • Copy both the Access Key ID and Secret Access Key and paste them into a secure text file on your computer.
  • Click Done.

Save your keys before closing this page

AWS will never show the Secret Access Key again after you close this page. If you lose it, you will need to delete the access key and create a new one.

Add CloudWatch as a data source in Grafana

With your IAM user and access keys ready, you can now connect Grafana to your AWS account. Grafana includes a built-in CloudWatch plugin, so there is nothing to install.

  • In the Grafana UI, click Connections in the left sidebar.
  • Click Add new connection.
  • Type CloudWatch in the search bar.
  • Select the CloudWatch data source from the results.
  • Click Add new data source in the upper right.

Grafana opens the Settings tab for your new CloudWatch data source.

  • Under Authentication Provider, select Access & secret key.
  • Paste your Access Key ID into the Access Key ID field.
  • Paste your Secret Access Key into the Secret Access Key field.
  • Set the Default Region to match the region where your EC2 instance is running (e.g., us-east-1).
  • Scroll to the bottom and click Save & test.

You should see a green confirmation message indicating the connection is successful.

What if I see a logs permission error?

Grafana tests both metrics and logs connections. If you see a warning about logs permissions, that is fine. The CloudWatchLogsReadOnlyAccess policy you attached should cover logs, but some accounts may show a partial warning. The metrics connection is what matters for this project.

🙋‍♀️ Seeing an authentication error?

Double-check that you pasted the Access Key ID and Secret Access Key correctly. Extra spaces or missing characters will cause authentication to fail.

Make sure the Default Region matches the region where your EC2 instance is running. A mismatched region will not show your instance's metrics.

Verify that the CloudWatchReadOnlyAccess policy is attached to the grafana-cloudwatch-reader user in the IAM console.

Help me troubleshoot my Grafana CloudWatch connection error.

Your Grafana instance is now connected to your AWS account and can read CloudWatch metrics in real time. Next up, you will build a professional dashboard that visualizes your EC2 instance's health metrics.

Build Your AIOps Observability Dashboard in Grafana

In the previous step, you connected Grafana Cloud to your AWS account through the CloudWatch data source. Your dashboard tool can now pull live metrics from AWS.

A data source connection on its own does not show you anything useful. You need panels that query specific metrics, display them visually, and give you an at-a-glance view of your instance's health. In this step, you will build a custom dashboard with multiple metric panels, then compare it against a professionally designed pre-built dashboard.

In this step, get ready to:
  • Create a Grafana dashboard with a CPUUtilization panel connected to your EC2 instance.
  • Add NetworkIn and DiskReadOps panels for multi-signal observability.
  • Import a pre-built EC2 dashboard and compare it with your custom dashboard.
Create a new dashboard and add a CPUUtilization panel

Each Grafana panel is a single visualization powered by a query to a data source. You will start by creating a panel that queries the CPUUtilization metric from CloudWatch for your specific EC2 instance.

  • Click Dashboards in the left sidebar of Grafana.
  • Click New and select New Dashboard.
  • In the Add drawer, click the Panel card to add a new visualization to your dashboard.
  • On the new panel, click Configure.

The panel editor opens with a query builder at the bottom.

  • In the Queries tab, click the Data source drop-down and select your CloudWatch data source.
  • Set the Namespace to AWS/EC2.
  • Set the Metric name to CPUUtilization.
  • Set the Statistic to Average.
  • Under Dimensions, add a dimension with Name set to InstanceId and Value set to your EC2 instance ID.
  • Leave the Period set to auto. Grafana calculates the best interval based on your selected time range.
  • Click Refresh to query the data source.

You should see a line graph populate with CPU utilization data from your instance.

What is a Grafana panel query?

A valid CloudWatch metric query requires three things: a namespace (the AWS service), a metric name, and at least one statistic. Dimensions are optional but let you filter down to a specific resource like your EC2 instance.

The Period controls how often data points appear. When set to auto, Grafana picks a period based on your time range so the graph stays readable.

  • Scroll up to the Panel options section in the panel editor sidebar.
  • Enter EC2 CPU Utilization in the Title field.
  • Click Save in the top right corner of the panel editor.

Your first panel now appears on the dashboard with a CPU utilization graph.

Not seeing any data in the panel?

  • Make sure your CloudWatch data source passed the Save & test check in the previous step. If it failed, double-check your IAM access keys and region.
  • Confirm the InstanceId dimension value matches your actual EC2 instance ID exactly. You can find it in the AWS Console under EC2, then Instances.
  • Check that the time range in the top right of Grafana covers a period when your instance was running. Try setting it to Last 3 hours.
  • Help me troubleshoot my empty Grafana panel.
Add NetworkIn and DiskReadOps panels

A single metric rarely tells the full story of an incident. Observability engineers monitor multiple signals so they can correlate events across CPU, network, and disk activity.

  • Click the Add element icon (blue plus sign) in the toolbar to open the sidebar.
  • Drag a new Panel onto the dashboard canvas.
  • Click Configure on the new panel.
  • Select your CloudWatch data source in the Data source drop-down.
  • Set the Namespace to AWS/EC2.
  • Set the Metric name to NetworkIn.
  • Set the Statistic to Sum.
  • Add the same InstanceId dimension with your EC2 instance ID as the value.
  • Click Refresh to see data populate.
  • Title this panel Network Traffic In (Bytes).
  • Click Save.
  • Repeat the same process to add a third panel.
  • Set the Metric name to DiskReadOps.
  • Set the Statistic to Sum.
  • Add the same InstanceId dimension.
  • Title this panel Disk Read Operations.
  • Click Save.

Your dashboard now has three panels showing CPU, network, and disk metrics side by side.

Why monitor multiple signals?

A CPU spike on its own could mean many things. When you see a CPU spike alongside a network traffic surge, you can narrow the cause to something like a traffic flood or a DDoS event. If the CPU spike coincides with heavy disk reads, it might point to a runaway log scan or database query.

This multi-signal approach is a core principle of observability engineering. The more correlated signals you have, the faster you can identify root cause.

🙋‍♀️ Seeing 'No data' on NetworkIn or DiskReadOps?

Some EC2 metrics like DiskReadOps only report data when there is actual disk activity. A freshly launched instance with no workload may show zero or no data for these metrics. This is normal. The panels will populate once you generate load in the next step.

Why is my panel showing no data?

Import the pre-built EC2 dashboard and save your custom dashboard

Grafana ships with curated dashboards for popular AWS services. Importing one gives you a professionally designed reference point to compare against your custom dashboard.

  • Click Connections in the left sidebar.
  • Click Data sources.
  • Click on your CloudWatch data source to open its configuration page.
  • Click the Dashboards tab at the top of the configuration page.

You will see a list of curated dashboards including Amazon EC2, Amazon EBS, AWS Lambda, and others.

  • Click Import next to Amazon EC2.

What are curated CloudWatch dashboards?

Grafana includes pre-configured dashboards for five popular AWS services: Amazon EC2, Amazon EBS, AWS Lambda, Amazon CloudWatch Logs, and Amazon RDS. These dashboards are built by the Grafana team and include panels for the most commonly monitored metrics.

If you customize an imported dashboard, save it under a different name. Grafana overwrites imported dashboards when new versions are released.

  • Navigate back to Dashboards in the left sidebar.
  • Open the imported Amazon EC2 dashboard and explore its panels.

Compare the pre-built dashboard with your custom dashboard. Notice how the professional version includes additional metrics and formatting that you can learn from for future dashboards.

  • Navigate back to Dashboards in the left sidebar.
  • Find your untitled custom dashboard and click on it.
  • Click the dashboard settings icon (gear icon) in the top toolbar.
  • Enter AIOps Anomaly Detection Lab as the dashboard name.
  • Click Save dashboard.
  • Take a screenshot of your custom dashboard showing all three panels.

Can't find your custom dashboard?

If you did not save the dashboard before navigating away, it may have been lost. Click Dashboards in the left sidebar and check for any unsaved or recently created dashboards. If it is gone, repeat the panel creation steps above. Each panel only takes a minute to recreate.

Help me find my missing dashboard.

Your observability dashboard is live and pulling real metrics from AWS. Next up, you will simulate a real-world incident by spiking the CPU on your EC2 instance and watch the entire pipeline react in real time.

Simulate an Incident and Watch the Pipeline React

In the previous step, you built a professional Grafana dashboard with live panels pulling EC2 metrics from CloudWatch.

Now it is time to put your entire pipeline to the test. You will simulate a real-world incident by spiking the CPU on your instance, then watch the anomaly detection model flag it, the alarm fire, and the email notification arrive. This is the moment where every piece you have built connects into a single end-to-end flow.

In this step, get ready to:
  • Generate a CPU spike on your EC2 instance using the stress utility.
  • Observe the anomaly in CloudWatch and your Grafana dashboard.
  • Document the full incident lifecycle from detection to notification.
Install the stress utility and generate a CPU spike

To test your anomaly detection pipeline, you need a way to push the CPU to 100% on demand. A Linux utility called stress does exactly this. It spawns worker processes that consume all available CPU cycles for a set duration.

  • SSH into your EC2 instance using the same terminal session from earlier steps.
  • Install the stress utility by running this command:
sudo yum install -y stress

What does this command do?

This installs the stress package from the Amazon Linux 2023 package repository. The -y flag automatically confirms the installation prompt so you do not have to type "yes" manually.

  • Confirm the installation succeeded by checking the output for Complete! at the end.

Seeing "No match for argument: stress"?

If the package is not found, try running sudo dnf install -y stress instead. Amazon Linux 2023 uses dnf as its default package manager, and yum is a wrapper for it. Both should work, but dnf is the canonical command.

Help me install the stress utility on Amazon Linux 2023

Before you start the stress test, open your Grafana dashboard and the CloudWatch console in separate browser tabs. You want to watch the metrics update in near real-time while the test runs.

  • In your browser, open a tab with your Grafana AIOps Anomaly Detection Lab dashboard.
  • Open another tab with the CloudWatch console, navigated to Metrics > All metrics > AWS/EC2 > Per-Instance Metrics > your CPUUtilization metric with anomaly detection enabled.
  • Back in your SSH terminal, start the CPU stress test by running this command:
stress --cpu 1 --timeout 300

What does this command do?

  • --cpu 1 spawns one worker process that continuously calculates square roots, consuming 100% of a CPU core.
  • --timeout 300 stops the test automatically after 300 seconds (5 minutes).
  • On a t2.micro or t3.micro instance with 1 vCPU, this pegs the CPU at 100% utilization for the full 5 minutes.

Your terminal will appear to hang while the stress test runs. This is normal. The command will print stress: info: [xxxx] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd and then block until the timeout completes.

Why 5 minutes?

CloudWatch collects EC2 metrics every 5 minutes with basic monitoring. Running the stress test for 5 minutes ensures at least one full data point captures the spike at 100% CPU. A shorter test might fall between collection intervals and not appear in your metrics.

🙋‍♀️ Terminal frozen or unresponsive?

The stress command blocks your terminal for the full 300 seconds. If you need to stop it early, press Ctrl+C. If your SSH session disconnects, the stress process may continue running on the instance until the timeout expires.

My SSH session disconnected during the stress test, what should I do?

Observe the anomaly in CloudWatch and Grafana

Once the stress test finishes (or while it is still running), switch to your browser tabs to see the impact across your pipeline.

  • In the CloudWatch console, click the refresh icon on your CPUUtilization metric graph.
  • Look for a sharp spike in the metric line. If the anomaly detection band (the gray shaded area) is visible, the spike should extend well above it.

What is the anomaly detection band?

The gray band on the graph represents what CloudWatch's ML model considers "normal" CPU behavior. The model learns from your instance's historical patterns. Any metric value that falls outside this band is flagged as anomalous.

For a new instance, the band may be narrow or may not appear at all yet. CloudWatch anomaly detection models can take up to 3 hours to generate the initial band. If you do not see the band, that is expected. The spike will still be visible in the raw metric data.

  • Navigate to CloudWatch > Alarms and check the state of your AIOps-CPU-Anomaly-Alarm alarm.

It shows ALARM

The anomaly detection model had enough training data to recognize the CPU spike as abnormal. Check your email inbox for a notification from SNS. The email subject line will contain the alarm name AIOps-CPU-Anomaly-Alarm and the body will include details about the metric value and threshold.

It still shows INSUFFICIENT_DATA or OK

This is normal for a newly created anomaly detection model. The model needs time to train on your instance's metric history. The alarm may not transition to ALARM state until the model has accumulated enough data points to define a "normal" baseline.

You can still see the CPU spike in the raw CPUUtilization metric graph. The anomaly detection band and alarm behavior will improve as the model trains over the next few hours.

  • Switch to your Grafana dashboard tab.
  • Click the refresh icon in the top right corner of the dashboard (or set the auto-refresh interval to 5m using the dropdown next to the refresh icon).
  • Look at the EC2 CPU Utilization panel. You should see the same spike reflected here.

You should see the CPU utilization jump to near 100% on both the CloudWatch graph and the Grafana panel. The spike will appear as a sharp vertical line followed by a plateau, then a drop back to near 0% after the stress test ended.

Don't see the spike in Grafana?

Grafana queries CloudWatch on a schedule, so there is a delay. Make sure the time range in the top right corner of your dashboard covers the last 30 minutes or 1 hour. Click the refresh icon to force a new query.

If you still do not see data, confirm that the InstanceId dimension in your panel query matches your EC2 instance ID. You can find your instance ID in the EC2 console.

Help me troubleshoot why my Grafana dashboard is not showing the CPU spike

Document the incident lifecycle

In production, observability engineers do not just detect incidents. They document them. Recording what happened, when it was detected, and how the team was notified creates a trail that improves future response times.

  • Take a screenshot of the CloudWatch metric graph showing the CPU spike outside the anomaly detection band (or above the normal baseline if the band has not appeared yet).
  • Take a screenshot of the CloudWatch alarm in ALARM state (or INSUFFICIENT_DATA if the model is still training).
  • If you received an SNS email notification, take a screenshot of it.
  • Take a screenshot of your Grafana dashboard showing the spike in the EC2 CPU Utilization panel.

What would a real incident summary look like?

In production, you would write a brief incident report covering these points:

  • What happened: CPU utilization spiked to 100% on the AIOps-Lab instance.
  • When it was detected: CloudWatch anomaly detection flagged the spike within one data collection interval (5 minutes with basic monitoring).
  • How the team was notified: SNS sent an email alert to the on-call engineer.
  • What action was taken: The root cause was identified as a stress test. In a real incident, next steps would include scaling the instance, investigating the process consuming CPU, or rolling back a deployment.

You just watched your entire AIOps pipeline react to a simulated incident. Next up, you will map out the enterprise architecture showing where tools like BigPanda and ServiceNow ITOM extend this pipeline for production use.

Map the Enterprise AIOps Architecture

In the previous step, you simulated a real-world CPU spike and watched your pipeline detect, alert, and visualize the anomaly.

Your pipeline works, but it only covers part of the picture. In a production environment, monitoring tools like CloudWatch and Grafana generate thousands of alerts across dozens of services. Without a way to correlate those alerts and route them into a ticketing system, teams drown in noise. In this step, you will learn how enterprise tools like BigPanda and ServiceNow ITOM extend the pipeline you built into a full enterprise observability stack.

In this step, get ready to:
  • Learn how BigPanda fits into the AIOps stack as an event correlation layer.
  • Learn how ServiceNow ITOM handles incident management and service dependency mapping.
  • Create an architecture diagram mapping the full enterprise observability pipeline.
Understand where BigPanda fits

Right now, your pipeline sends SNS email alerts directly to your inbox when CloudWatch detects an anomaly. That works for a single EC2 instance. In production, a company might have hundreds of instances, Lambda functions, RDS databases, and third-party services all firing alerts at the same time.

BigPanda is an AIOps event correlation platform. It sits between your monitoring tools (CloudWatch, Grafana, Datadog, Splunk) and your ITSM platform (ServiceNow). Instead of SNS sending alerts directly to email, CloudWatch alarms would send notifications to BigPanda via webhook or one of its 50+ built-in integrations.

BigPanda then performs three critical functions:

  1. Alert noise reduction. BigPanda filters out false positives and deduplicates recurring events. This can reduce alert noise by up to 95%.
  2. Intelligent event correlation. BigPanda groups related alerts from multiple monitoring sources into a single actionable incident. Teams focus on one incident instead of chasing dozens of seemingly different alerts.
  3. Root cause analysis. BigPanda uses ML-driven pattern recognition to surface the probable root cause of an incident.

How would BigPanda help with your CPU spike?

Imagine your EC2 CPU spike coincided with a Lambda timeout and an RDS connection pool exhaustion. Without BigPanda, you would receive three separate alerts and investigate them independently. BigPanda would correlate all three into a single incident and surface the CPU spike as the probable root cause.

Understand where ServiceNow ITOM fits

Once BigPanda identifies and correlates an incident, it needs somewhere to go. That is where ServiceNow IT Operations Management (ITOM) comes in.

ServiceNow ITOM is the incident management and service dependency layer. BigPanda automatically creates a ServiceNow incident ticket with enriched context: the root cause, affected services, historical similar incidents, and suggested remediation steps.

ServiceNow ITOM provides three key capabilities:

  1. Configuration Management Database (CMDB). The CMDB maps every service, server, database, and application in your infrastructure along with the dependencies between them. When an incident occurs, the CMDB shows the blast radius of the problem.
  2. Event Management. ServiceNow ingests and correlates events from monitoring tools. It aggregates thousands of alerts, suppresses duplicates, and highlights only actionable alerts.
  3. IT workflows. ServiceNow orchestrates the full incident lifecycle: escalation, change management, and problem management. It tracks who is working on what, what changed, and whether the fix worked.

How does this connect to what you built?

The SNS email notification you received in the previous step simulates the "create a ticket" step. In production, BigPanda would auto-create a ServiceNow incident with all the alert context attached. No manual email triage needed.

💡 Why are BigPanda and ServiceNow ITOM conceptual only?

BigPanda requires enterprise licensing with no free tier available. ServiceNow has Developer Instances, but the ITOM capabilities are limited and require additional configuration that exceeds this project's scope. Understanding where these tools fit in the architecture is the skill employers look for.

Create your AIOps architecture diagram

Now you will put everything together into a visual architecture diagram. This is the kind of diagram you would present in a portfolio or job interview to show you understand the full AIOps stack.

  • Open any diagramming tool you prefer. draw.io (free, browser-based) and Lucidchart (free tier available) both work well. Pen and paper is also fine.

Your diagram should have six layers, stacked from bottom to top:

  1. Infrastructure Layer. Your EC2 instance emitting metrics. Label it with your instance ID.
  2. Telemetry Layer. CloudWatch collecting and storing metrics, logs, and traces.
  3. Intelligence Layer. CloudWatch Anomaly Detection applying ML models to detect deviations.
  4. Visualization Layer. Grafana Cloud dashboards displaying real-time metrics and anomaly bands.
  5. Correlation Layer. BigPanda ingesting alerts from CloudWatch (and other monitoring tools), performing noise reduction and event correlation.
  6. Incident Management Layer. ServiceNow ITOM receiving correlated incidents, managing the CMDB, and orchestrating remediation workflows.
  • Draw arrows between each layer and label the connections. For example, label the arrow from Intelligence to Correlation as "CloudWatch Alarm via SNS".
  • Label the arrow from Correlation to Incident Management as "REST API/Webhook".
  • Label the arrow from Infrastructure to Telemetry as "CloudWatch Agent / EC2 metrics".

Pro tip: color code your layers

Use different colors for the layers you built hands-on (Infrastructure through Visualization) versus the layers that are conceptual (Correlation and Incident Management). This makes it clear which parts of the stack you have direct experience with.

  • Save your completed diagram as part of your project documentation.

Why does this diagram matter for your career?

Observability engineering interviews often ask candidates to whiteboard a monitoring architecture. Knowing where each tool fits and how data flows between layers shows you understand the full incident lifecycle. Most candidates can name the tools. Fewer can explain how they connect.

Secret mission

Add Custom Application-Level Telemetry

Your pipeline monitors infrastructure metrics, but real-world observability engineers also instrument application-level telemetry. Can you push a custom CloudWatch metric that simulates API request latency, build an anomaly detector for it, and visualize it alongside your infrastructure data in Grafana?

Before you go

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. The EC2 instance is free tier eligible, but CloudWatch alarms cost $0.10 per alarm per month, so leaving them running will incur small charges.

Cost warning

CloudWatch anomaly detection alarms cost $0.10 per alarm per month. If you created two alarms (one for CPU, one for the secret mission custom metric), that is $0.20/month. The EC2 instance is free tier eligible for 12 months (or 6 months for newer accounts), but will cost money if your free tier has expired. Delete your alarms and terminate your instance if you are done with this project.

Resources you used:

  • 1 EC2 instance (t2.micro or t3.micro, free tier eligible).
  • 1 IAM role with CloudWatchFullAccess attached to the EC2 instance.
  • 1 IAM user (grafana-cloudwatch-reader) with access keys.
  • 1 SNS topic (AIOps-Anomaly-Alerts) with 1 email subscription.
  • 1-2 CloudWatch anomaly detection alarms.
  • 1-2 CloudWatch anomaly detection models.
  • 1 Grafana Cloud account (free, no charges).

Keep everything running

No action needed. Choose this if you are still actively building or want to continue experimenting with your AIOps pipeline.

  • Your EC2 instance will continue consuming free tier hours. If you are within your free tier period, there is no compute cost.
  • CloudWatch alarms will continue running at $0.10 per alarm per month.
  • Monitor your AWS billing dashboard to track any charges.
  • Your Grafana Cloud free account has no ongoing costs.

Pause - I'll come back to this later

Stop your EC2 instance to pause compute charges while keeping your setup intact. Your CloudWatch alarms and anomaly models will remain but show INSUFFICIENT_DATA while the instance is stopped.

  • In the EC2 Console, select your instance, then choose Instance state > Stop instance.
  • Your CloudWatch alarms will remain active but will transition to INSUFFICIENT_DATA state. They still cost $0.10 per alarm per month.
  • Your Grafana Cloud dashboard and data source configuration will persist.
  • To resume, start the instance again from the EC2 Console. Note that the public IP address may change.

Delete - I don't want to use this again

Remove all project resources completely. Follow these steps in order.

Delete alarms before anomaly models

You must delete your CloudWatch alarms before deleting the anomaly detection models. If you delete the model first, the alarm will automatically recreate it.

  • Delete CloudWatch alarms: Go to CloudWatch > Alarms, select your alarms, then choose Actions > Delete.
  • Delete CloudWatch anomaly detection models: In CloudWatch, select and delete the anomaly detectors associated with your metrics.
  • Delete the SNS topic: Go to SNS > Topics, select AIOps-Anomaly-Alerts, then choose Delete.
  • Terminate the EC2 instance: Go to EC2, select your instance, then choose Instance state > Terminate instance.
  • Delete the IAM user: Go to IAM > Users, select grafana-cloudwatch-reader, then choose Delete.
  • Delete the IAM role: Go to IAM > Roles, find the role you created for the EC2 instance, then choose Delete.
  • Your Grafana Cloud free account can remain. It does not cost anything.

Mission accomplished

Nice work! You just built a production-style AIOps observability pipeline from scratch. You launched an EC2 instance, applied ML-powered anomaly detection with CloudWatch, visualized everything in a professional Grafana Cloud dashboard, and watched the full detection-to-notification pipeline fire in real time.

Here's what you learned:

  • Enabled CloudWatch Anomaly Detection to learn normal metric behavior using ML and flag deviations automatically, replacing brittle static thresholds with intelligent baselines.
  • Built an automated alert pipeline using SNS and CloudWatch alarms that notifies you the moment an anomaly is detected.
  • Connected Grafana Cloud to AWS CloudWatch as a live data source and built a multi-signal observability dashboard to monitor EC2 health in real time.
  • Mapped the enterprise AIOps architecture showing how BigPanda (event correlation) and ServiceNow ITOM (incident management) extend the pipeline for production use.
  • Added custom application-level telemetry with its own anomaly detector and Grafana panel, demonstrating full-stack observability across both infrastructure and application layers.

Ready to quiz yourself?