Centralized Log Pipeline with Firehose

Stream CloudTrail and VPC Flow Logs through Firehose into a central S3 archive.

Introduction

30 Second Summary

When something breaks in a cloud environment, you need to trace what happened across API calls, network traffic, and server metrics. If those logs are scattered across separate services, tracking down the root cause means jumping between screens instead of getting answers fast.

In this project, you will build a centralized log pipeline on AWS. It captures API calls via CloudTrail and network traffic via VPC Flow Logs, streams everything through Amazon Data Firehose into a single S3 archive, and displays it all on a unified CloudWatch dashboard.

What You'll Build

You will open a single dashboard and immediately see who called what API in your account, what network connections were attempted across your VPC, and whether your EC2 instance is healthy.

By the end of this project, you'll have:

  • A live feed of every API call and network connection in your account, all flowing into CloudWatch Logs within minutes of occurring.
  • An automated archive pipeline where Amazon Data Firehose batches, compresses, and delivers all logs to a single S3 bucket without you writing any code.
  • A unified CloudWatch dashboard where you query across API activity, network traffic, and EC2 health from one screen using Logs Insights.
  • Secret Mission: Build a metric filter and alarm that detects rejected SSH connections and emails you the moment they happen.

Do I need any prerequisites?

You need an AWS account with admin access and the AWS CLI installed and configured on your machine. Step 1 will walk you through confirming everything is ready.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters. What problem does centralized logging solve that having logs scattered across individual services does not?

Set Up the Foundation

A centralized log pipeline needs three things before it can do anything useful: a destination where logs will land, resources that actually generate log data, and permissions that let AWS services talk to each other.

Without an EC2 instance in a VPC, there is no network traffic to capture and no meaningful CloudTrail activity to observe. In this step, you will create the S3 bucket for your log archive, launch an EC2 instance that generates real traffic, and set up the IAM roles that authorize the pipeline.

In this step, get ready to:
  • Create an S3 bucket as your centralized Log Archive.
  • Launch a t2.micro EC2 instance in your default VPC.
  • Create IAM roles for Firehose and VPC Flow Logs.
Create the S3 Log Archive bucket

Every log your pipeline collects will ultimately land in a single Amazon S3 bucket. Creating this bucket first gives you a destination to point everything at later.

  • First, note your AWS account ID. Retrieve it by running this command in your terminal:
aws sts get-caller-identity --query Account --output text
  • Copy the 12-digit number that appears. You will use it in the next command to create a globally unique bucket name.
  • Create your Log Archive bucket by running this command (replace the account ID placeholder with your actual account ID):
aws s3 mb s3://log-archive-bucket-[[ACCOUNT_ID="your-12-digit-account-id"]]

What does this command do?

The aws s3 mb command creates ("makes") a new S3 bucket. The bucket name includes your account ID to ensure it is globally unique across all of AWS.

The bucket is created in your CLI's default region. All resources in this project should be in the same region.

  • Confirm the bucket exists by running:
aws s3 ls | findstr log-archive

You should see your new bucket listed with today's date.

Don't see your bucket?

  • Make sure you replaced the placeholder with your actual 12-digit account ID (no spaces, no dashes).
  • If you see a "BucketAlreadyExists" error, someone else may have used that name. Try appending a short suffix like -logs to your bucket name.
  • Check that your AWS CLI is configured to the correct region by running aws configure get region.

Help me troubleshoot my S3 bucket creation error.

Launch an EC2 instance

This instance will generate both CloudTrail events (from console actions you take) and network traffic that VPC Flow Logs can capture. You do not need to connect to it, so no key pair is required.

  • In the AWS Management Console, type EC2 in the search bar at the top and select the EC2 service.
  • Click Launch instance.
  • Under Name and tags, enter log-generator for the name.
  • Under Application and OS Images, keep the default Amazon Linux AMI selected.
  • Under Instance type, select t2.micro (Free tier eligible).
  • Under Key pair (login), select Proceed without a key pair (Not recommended). You will not need SSH access for this project.
  • Leave the Network settings at their defaults. This uses your default VPC and creates a security group allowing SSH inbound.
  • Click Launch instance.

You should see a success message confirming your instance is launching.

Why no key pair?

This instance exists solely to generate log data. You will never SSH into it. Skipping the key pair keeps the setup simple and avoids managing unnecessary credentials.

  • Confirm the instance is running by executing this command in your terminal:
aws ec2 describe-instances --query "Reservations[*].Instances[*].[InstanceId,State.Name]" --output table

You should see a table with your instance ID and the state running. It may take a minute to transition from pending to running.

Instance stuck in pending?

  • Wait 30-60 seconds and run the command again. EC2 instances take a moment to start.
  • If the state shows terminated, check the EC2 console for an error message. A common cause is exceeding your region's instance limit.

Help me troubleshoot my EC2 instance not starting.

Create the IAM roles

Your pipeline needs two IAM roles. One lets Amazon Data Firehose write log data into your S3 bucket. The other lets VPC Flow Logs publish network traffic records to CloudWatch Logs.

  • Move to your Desktop and create a project folder for your policy files by running these commands:
cd ~/Desktop
mkdir log-pipeline
cd log-pipeline

Now you will create the FirehoseToS3Role. This role allows the Firehose service to assume it and write objects to your S3 bucket.

  • In the AWS Management Console, type IAM in the search bar and select the IAM service.
  • In the left navigation pane, click Roles, then click Create role.
  • For Trusted entity type, choose Custom trust policy.
  • Delete the placeholder JSON in the policy editor and paste the following trust policy:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "firehose.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

What does this trust policy do?

A trust policy defines WHO can assume this role. Here, you are telling AWS that the Firehose service (firehose.amazonaws.com) is allowed to take on this role's permissions. Without this trust relationship, Firehose cannot act on your behalf.

  • Click Next.
  • On the Add permissions page, skip adding any managed policies for now. Click Next.
  • For Role name, enter FirehoseToS3Role.
  • Click Create role.

You should see a success banner confirming the role was created. Now you need to add the S3 permissions as an inline policy.

  • In the Roles list, click FirehoseToS3Role to open it.
  • Select the Permissions tab.
  • Click Add permissions, then Create inline policy.
  • Select the JSON tab in the policy editor and replace the contents with:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetBucketLocation",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::log-archive-bucket-ACCOUNT_ID",
        "arn:aws:s3:::log-archive-bucket-ACCOUNT_ID/*"
      ]
    }
  ]
}

What does this permissions policy do?

This grants Firehose three specific abilities on your log archive bucket: s3:PutObject to write log files, s3:GetBucketLocation to determine the bucket's region, and s3:ListBucket to verify the bucket exists.

Replace ACCOUNT_ID in both Resource lines with your actual 12-digit account ID so the ARN matches your real bucket name.

  • Replace ACCOUNT_ID in both Resource lines with your actual 12-digit account ID.
  • Click Next.
  • For the policy name, enter FirehoseS3Access.
  • Click Create policy.

Your FirehoseToS3Role now has the permissions it needs. Confirm it by checking the Permissions tab. You should see FirehoseS3Access listed as an inline policy.

Next, create the VPCFlowLogsRole. This role allows the VPC Flow Logs service to publish log data to CloudWatch Logs.

  • In the left navigation pane, click Roles, then click Create role.
  • For Trusted entity type, choose Custom trust policy.
  • Delete the placeholder JSON and paste the following trust policy:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "vpc-flow-logs.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

What does this trust policy do?

This tells AWS that the VPC Flow Logs service (vpc-flow-logs.amazonaws.com) is allowed to assume this role. When you enable flow logs later, the service uses this role to publish records to CloudWatch Logs on your behalf.

  • Click Next.
  • Skip the Add permissions page and click Next.
  • For Role name, enter VPCFlowLogsRole.
  • Click Create role.

Now add the CloudWatch Logs permissions as an inline policy.

  • In the Roles list, click VPCFlowLogsRole to open it.
  • Select the Permissions tab.
  • Click Add permissions, then Create inline policy.
  • Select the JSON tab and replace the contents with:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams"
      ],
      "Resource": "*"
    }
  ]
}

What does this permissions policy do?

This grants the VPC Flow Logs service full access to create and write to CloudWatch Logs log groups and log streams. The five actions cover creating the log group, creating streams within it, writing log events, and querying existing groups and streams.

  • Click Next.
  • For the policy name, enter VPCFlowLogsCWLAccess.
  • Click Create policy.

Check the Permissions tab for VPCFlowLogsRole. You should see VPCFlowLogsCWLAccess listed as an inline policy.

Why two separate roles?

Each AWS service in your pipeline needs its own role with the minimum permissions for its specific job. Firehose only needs to write to S3. VPC Flow Logs only needs to publish to CloudWatch Logs. Keeping roles narrow follows the principle of least privilege and makes it easy to audit who has access to what.

  • Verify both roles exist by running this command in your terminal:
aws iam get-role --role-name FirehoseToS3Role --query "Role.RoleName" --output text
aws iam get-role --role-name VPCFlowLogsRole --query "Role.RoleName" --output text

You should see both role names printed, one per line.

Seeing "NoSuchEntity" error?

  • Double-check the role name spelling. It must be exactly FirehoseToS3Role and VPCFlowLogsRole (case-sensitive).
  • If you accidentally used a different name, go back to the IAM console and check the Roles list to find the exact name you entered.

Help me troubleshoot my IAM role creation.

Before you move on, predict what you would see if you ran the EC2 describe-instances command again right now. Has anything changed since you last checked?

  • Run the final verification to confirm your EC2 instance is still healthy:
aws ec2 describe-instances --query "Reservations[*].Instances[*].[InstanceId,State.Name]" --output table

You should see your instance ID with the state running. All three foundation pieces are in place: your S3 archive bucket, your log-generating EC2 instance, and both IAM roles.

Your foundation is locked in. You have a destination bucket, a traffic-generating instance, and the permissions to wire everything together. Next up, you will turn on the recorders and watch real log data start flowing into CloudWatch.

Enable CloudTrail and VPC Flow Logs

Your AWS account has the foundation in place. An EC2 instance is running, IAM roles are ready, and your S3 Log Archive bucket is waiting for data.

The problem is that your account is generating API activity and network traffic right now, but none of it is being captured anywhere you can see it. This step turns on the recorders so the activity becomes visible in CloudWatch Logs.

In this step, get ready to:
  • Create a CloudTrail trail delivering management events to S3 and CloudWatch Logs.
  • Enable VPC Flow Logs on your default VPC publishing to CloudWatch Logs.
  • Generate traffic to produce log data in both log groups.
Create a CloudTrail trail

AWS CloudTrail records every API call made in your account. By creating a trail, you tell CloudTrail to capture these management events and deliver them to both your S3 bucket and a CloudWatch Logs log group.

  • In the AWS Management Console, search for CloudTrail and open the service.
  • Click Create trail.
  • In Trail name, enter management-events.
  • For Storage location, select Use existing S3 bucket.
  • Enter your Log Archive bucket name (log-archive-bucket- followed by your account ID).
  • Clear the checkbox for Log file SSE-KMS encryption to keep things simple with default SSE-S3 encryption.

Now enable CloudWatch Logs delivery on this same page:

  • In the CloudWatch Logs section, choose Enabled.
  • For the log group, select New and replace the suggested name with /aws/cloudtrail.
  • For Role name, select New to let the console create an IAM role for CloudTrail to write to CloudWatch Logs.
  • Click Next.

On the Choose log events page:

  • Keep the default selection of Management events with both Read and Write selected.
  • Leave Data events, Insights events, and Network activity events at their defaults.
  • Click Next.

On the Review and create page:

  • Review your settings to confirm the trail delivers to both S3 and CloudWatch Logs.
  • Click Create trail.

Your new trail appears on the Trails page with logging turned on.

What does CloudTrail capture?

Management events cover API calls that create, modify, or delete AWS resources. Every time you launch an instance, create a bucket, or attach a policy, CloudTrail records it.

The first copy of management events delivered to S3 is free per region. Sending them to CloudWatch Logs incurs a small ingestion charge.

Seeing a permissions error on trail creation?

Make sure your IAM user or role has the AWSCloudTrail_FullAccess managed policy attached, or equivalent admin permissions. The console needs permission to create the S3 bucket policy and the CloudWatch Logs role.

Help me troubleshoot my CloudTrail trail creation permissions error.

Enable VPC Flow Logs on your default VPC

VPC Flow Logs capture information about the IP traffic going to and from network interfaces in your VPC. You already have a VPCFlowLogsRole from the previous step, so you just need to create the log group and enable flow logging.

  • Retrieve your default VPC ID by running this command:
aws ec2 describe-vpcs --filters "Name=is-default,Values=true" --query "Vpcs[0].VpcId" --output text

You should see a VPC ID like vpc-0abc1234def56789. Copy this value.

  • Create the CloudWatch Logs log group for your flow logs by running:
aws logs create-log-group --log-group-name /aws/vpc/flowlogs
  • Enable VPC Flow Logs on your default VPC by running this command, replacing the VPC ID and account ID with your own values:
aws ec2 create-flow-logs --resource-type VPC --resource-ids [[VPC_ID="your-default-vpc-id"]] --traffic-type ALL --log-group-name /aws/vpc/flowlogs --deliver-logs-permission-arn arn:aws:iam::[[AWS_ACCOUNT_ID="your-12-digit-account-id"]]:role/VPCFlowLogsRole

You should see a response containing a FlowLogIds array with your new flow log ID.

Why capture ALL traffic?

The --traffic-type ALL flag captures both accepted and rejected connections. Rejected traffic is especially useful for security monitoring because it shows blocked connection attempts.

Seeing an error about the log group or role?

If you see an error mentioning the log group, confirm it was created by running aws logs describe-log-groups --log-group-name-prefix /aws/vpc. If the log group is missing, re-run the create-log-group command.

If the error mentions the IAM role, double-check that the role name VPCFlowLogsRole matches exactly what you created in Step 1.

Help me troubleshoot my VPC Flow Logs creation error.

Generate traffic and verify logs appear

Both recorders are now active, but they need traffic to capture. An SSH connection attempt to your EC2 instance generates both a CloudTrail event (from the console actions you have been performing) and a rejected flow log entry (since the default security group does not allow inbound SSH from the internet).

  • Get your EC2 instance's public IP address by running:
aws ec2 describe-instances --query "Reservations[*].Instances[*].PublicIpAddress" --output text
  • Attempt an SSH connection to your instance by running the following command (replace the IP with your instance's public IP):
ssh ec2-user@[[EC2_PUBLIC_IP="your EC2 instance public IP address"]]
  • Wait a few seconds for the connection to hang or be refused, then press Ctrl+C to cancel it.

Why does the connection fail?

The default security group does not allow inbound SSH from the internet. The connection gets rejected at the network level, which is exactly what you want. This rejected traffic shows up as a REJECT entry in your VPC Flow Logs.

Before you check for logs, what do you think you will see in CloudWatch Logs right now? Will both log groups have data immediately?

  • Open the CloudWatch console by searching for CloudWatch in the AWS console search bar.
  • In the left navigation pane, under Logs, click Log groups.
  • Confirm you see both /aws/cloudtrail and /aws/vpc/flowlogs listed.

CloudTrail events typically appear within 5 minutes of an API call. VPC Flow Logs may take 5-10 minutes to publish their first records. If you do not see log streams yet, wait a few minutes and refresh.

  • Click on the /aws/cloudtrail log group and verify you see at least one log stream with events.
  • Navigate back to Log groups, click on /aws/vpc/flowlogs, and confirm a log stream appears with flow records.

Notice the fragmentation problem?

You now have logs in two separate places. To see API activity you check one log group. To see network traffic you check a different one. In a real environment with dozens of services, this scattering makes searching and correlating events painful.

The next step introduces Amazon Data Firehose to solve exactly this problem by streaming everything into a single archive.

Both recorders are now streaming into their own CloudWatch Logs log groups. Next up, you will use Amazon Data Firehose to pipe all of this data into a single S3 archive automatically.

Stream Logs to S3 with Amazon Data Firehose

You now have logs landing in two separate CloudWatch Log Groups. One captures API activity, the other captures network traffic.

In a real environment with dozens of accounts and services, this fragmentation makes searching, archiving, and compliance painful. Amazon Data Firehose solves this by acting as a managed streaming pipe that automatically batches, compresses, and delivers log data to a central destination without you writing any code.

In this step, get ready to:
  • Create an Amazon Data Firehose delivery stream that delivers logs to your S3 archive bucket.
  • Connect both CloudWatch Log Groups to Firehose using subscription filters.
  • Verify compressed log files appear automatically in S3.
Create the Firehose delivery stream

Amazon Data Firehose is a fully managed service that captures, transforms, and delivers streaming data to destinations like S3. You configure a delivery stream once, and Firehose handles buffering, compression, and delivery automatically.

  • Create the delivery stream named log-archive-stream by running this command in your terminal:
aws firehose create-delivery-stream \
  --delivery-stream-name "log-archive-stream" \
  --s3-destination-configuration '{"RoleARN": "arn:aws:iam::[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]:role/FirehoseToS3Role", "BucketARN": "arn:aws:s3:::[[LOG_BUCKET="your log archive bucket name"]]", "BufferingHints": {"IntervalInSeconds": 60, "SizeInMBs": 1}}'

What does this command do?

This creates a Firehose delivery stream that buffers incoming data for 60 seconds (or until 1 MB accumulates) before delivering it to your S3 bucket. The FirehoseToS3Role you created earlier gives Firehose permission to write objects to the bucket.

The default buffer interval is 300 seconds (5 minutes), but setting it to 60 seconds means you will see data arrive in S3 faster during testing.

You should see a JSON response containing a DeliveryStreamARN value. Copy and save this ARN. You will need it in the next substep.

  • Confirm the stream is active by running:
aws firehose describe-delivery-stream \
  --delivery-stream-name "log-archive-stream" \
  --query "DeliveryStreamDescription.DeliveryStreamStatus" \
  --output text

You should see ACTIVE in the output. It may take a minute or two after creation for the stream to reach this state.

Seeing CREATING instead of ACTIVE?

Wait 1-2 minutes and run the describe command again. If the status shows CREATING_FAILED, double-check that your FirehoseToS3Role ARN and bucket ARN are correct. You would need to delete the failed stream and recreate it.

Help me troubleshoot my Firehose delivery stream

Create the CWLtoFirehoseRole and attach the subscription filter for CloudTrail

Subscription filters tell CloudWatch Logs to forward matching log events to a destination in real time. Before you can create a subscription filter that targets Firehose, you need an IAM role that CloudWatch Logs can assume to write to your delivery stream.

  • Create a trust policy file for the new role by running:
cat > cwl-trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Principal": {
      "Service": "logs.[[AWS_REGION="your AWS region, e.g. us-east-1"]].amazonaws.com"
    },
    "Action": "sts:AssumeRole"
  }
}
EOF

Why is the region in the trust policy?

CloudWatch Logs uses a region-specific service principal (like logs.us-east-1.amazonaws.com) when assuming IAM roles. You must match your actual region here, or the subscription filter creation will fail with an access denied error.

  • Create the IAM role by running:
aws iam create-role \
  --role-name CWLtoFirehoseRole \
  --assume-role-policy-document file://cwl-trust-policy.json

You should see a JSON response with the new role's ARN and creation details.

  • Attach a permissions policy that allows this role to put data into the Firehose stream by running:
cat > cwl-firehose-permissions.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "firehose:*",
      "Resource": "arn:aws:firehose:[[AWS_REGION="your AWS region, e.g. us-east-1"]]:[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]:deliverystream/log-archive-stream"
    }
  ]
}
EOF

aws iam put-role-policy \
  --role-name CWLtoFirehoseRole \
  --policy-name FirehoseAccess \
  --policy-document file://cwl-firehose-permissions.json

What does firehose:* grant?

This gives CloudWatch Logs full Firehose permissions on the log-archive-stream delivery stream only. The wildcard is scoped to a single stream ARN, so it cannot affect other delivery streams in your account.

IAM role propagation can take a few seconds. Wait about 10 seconds before the next command.

  • Create the subscription filter on the CloudTrail log group by running:
aws logs put-subscription-filter \
  --log-group-name "/aws/cloudtrail" \
  --filter-name "CloudTrailToFirehose" \
  --filter-pattern "" \
  --destination-arn "arn:aws:firehose:[[AWS_REGION="your AWS region, e.g. us-east-1"]]:[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]:deliverystream/log-archive-stream" \
  --role-arn "arn:aws:iam::[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]:role/CWLtoFirehoseRole"

What is a subscription filter?

A subscription filter is a real-time forwarding rule on a CloudWatch Log Group. The empty --filter-pattern "" means all log events are forwarded. You could narrow this to specific event patterns, but for centralized archiving you want everything.

Each log group supports up to two subscription filters. You are using one of those two slots here.

If the command returns with no output, it succeeded.

Getting an InvalidParameterException?

This usually means the CWLtoFirehoseRole trust policy region does not match your actual region, or the role has not propagated yet. Verify the region in your trust policy file matches the region in your delivery stream ARN. Wait 10-15 seconds and try again.

Help me debug my subscription filter

Attach the subscription filter for VPC Flow Logs and verify delivery

With the CWLtoFirehoseRole already in place, you can reuse it for the second log group. Both subscription filters route to the same Firehose stream, which means all your logs converge into a single S3 destination.

  • Create the second subscription filter on the VPC Flow Logs log group by running:
aws logs put-subscription-filter \
  --log-group-name "/aws/vpc/flowlogs" \
  --filter-name "FlowLogsToFirehose" \
  --filter-pattern "" \
  --destination-arn "arn:aws:firehose:[[AWS_REGION="your AWS region, e.g. us-east-1"]]:[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]:deliverystream/log-archive-stream" \
  --role-arn "arn:aws:iam::[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]:role/CWLtoFirehoseRole"

Why the same role for both filters?

The CWLtoFirehoseRole grants permission to write to the delivery stream. Both log groups need the same permission (put data into the same Firehose stream), so one role serves both subscription filters.

Data sent from CloudWatch Logs to Amazon Data Firehose is already compressed with gzip level 6 compression. The files landing in S3 will be .gz files without any extra configuration on your part.

Both subscription filters are now active. Firehose buffers incoming data for 60 seconds before writing a batch to S3.

Before you check S3, what do you expect to find in the bucket after waiting a few minutes?

  • Wait about 5 minutes, then check your S3 bucket for delivered log files by running:
aws s3 ls s3://[[LOG_BUCKET="your log archive bucket name"]]/ --recursive

You should see compressed .gz files organized in a YYYY/MM/DD/HH folder structure. These files were delivered automatically by Firehose from both your CloudTrail and VPC Flow Logs data.

No files appearing after 5 minutes?

Confirm both subscription filters exist by running aws logs describe-subscription-filters --log-group-name "/aws/cloudtrail" and checking the output shows your filter. Do the same for /aws/vpc/flowlogs.

Also verify the Firehose stream is still ACTIVE with the describe-delivery-stream command from earlier. If the stream shows errors, check that the FirehoseToS3Role has s3:PutObject permission on your bucket.

Help me troubleshoot my log delivery pipeline

Your centralized log pipeline is now live. CloudTrail events and VPC Flow Logs both stream through a single Firehose delivery stream into your S3 archive. Next up, you will build a CloudWatch dashboard that queries across all of this data in one place.

Build a Unified CloudWatch Dashboard

Your logs are now flowing to S3 through Amazon Data Firehose. That gives you a durable archive for compliance and long-term analysis.

But archives don't help when something is happening right now. You need real-time visibility. A CloudWatch dashboard combines live metrics, log queries, and alarms into a single view. In this step, you will build one that pulls together EC2 health, API activity, and network traffic patterns.

In this step, get ready to:
  • Create a CloudWatch dashboard with an EC2 CPUUtilization metric widget.
  • Add a Logs Insights query widget showing the top 10 API calls from CloudTrail.
  • Add a Logs Insights query widget showing accepted vs rejected VPC traffic.
Create a CloudWatch dashboard and add a CPU metric widget

Your EC2 instance has been running and publishing CPU metrics since Step 1. A line graph widget is the quickest way to confirm the instance is healthy at a glance.

  • Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
  • In the navigation pane, choose Dashboards.
  • Choose Create dashboard.
  • In the Create new dashboard dialog box, enter CentralizedLogDashboard as the name.
  • Choose Create dashboard.

An Add to this dashboard dialog appears automatically, prompting you to add your first widget.

  • Choose Line as the widget type.
  • Choose Configure.
  • In the Browse tab, choose EC2.
  • Choose Per-Instance Metrics.
  • Find your instance and select the checkbox next to CPUUtilization.
  • Choose Create widget.
  • Choose Save dashboard.

You should see a line graph showing your instance's CPU utilization over the selected time range.

What does this widget show?

CPUUtilization is the percentage of allocated compute capacity that your EC2 instance is actively using. A t2.micro sitting idle typically shows very low usage (under 5%). Any spikes you see are from API calls and instance activity you triggered in earlier steps.

Don't see CPUUtilization listed?

Make sure you selected EC2 first, then drilled into Per-Instance Metrics. If your instance was launched very recently (under 5 minutes ago), the metric may not have published yet. Wait a few minutes and refresh.

I can't find my EC2 instance's CPUUtilization metric in the CloudWatch Browse tab. Help me troubleshoot.

Add a Logs Insights widget for CloudTrail API activity

Your first widget shows instance health. Now you need to see what API calls are happening in your account. CloudWatch Logs Insights lets you run SQL-like queries against your log groups and pin the results to a dashboard as a live widget.

  • In the CloudWatch navigation pane, choose Logs.
  • Choose Logs Insights.
  • In the Select log group(s) dropdown, search for and select /aws/cloudtrail.
  • Clear the default query in the query editor and paste the following query:
stats count(*) by eventName | sort count desc | limit 10
  • Choose Run.

You should see a table of the top 10 most frequent API calls in your account, sorted by count. This confirms your CloudTrail log group has data and the query works.

  • Choose Add to dashboard.
  • In the dialog, select CentralizedLogDashboard from the dashboard list.
  • Select Bar as the widget type.
  • Enter Top 10 API Calls as the widget name.
  • Choose Add to dashboard.

What does this query do?

The stats count(*) by eventName command groups every CloudTrail event by its API call name and counts how many times each one occurred. The sort count desc | limit 10 part sorts them from most frequent to least, and shows only the top 10.

This gives you an instant picture of what is happening in your account. You will typically see calls like DescribeInstances, AssumeRole, and GetObject at the top.

Query returns no results?

CloudTrail events take 5-10 minutes to appear in CloudWatch Logs after the trail is created. If your log group was only created recently, wait a few minutes and run the query again.

Also confirm you selected /aws/cloudtrail (not the VPC Flow Logs group) in the log group dropdown.

Help me figure out why CloudTrail events aren't appearing in my log group.

Add a Logs Insights widget for VPC Flow Log traffic

Your second query widget targets network traffic. This shows you the breakdown of accepted vs rejected connections across your VPC.

  • In the CloudWatch navigation pane, choose Logs.
  • Choose Logs Insights.
  • In the Select log group(s) dropdown, clear the previous selection and select /aws/vpc/flowlogs.
  • Clear the query editor and paste the following query:
stats count(*) by action | sort count desc
  • Choose Run.

You should see results showing counts for ACCEPT and REJECT actions. This confirms your VPC Flow Logs are actively capturing network traffic.

  • Choose Add to dashboard.
  • Select CentralizedLogDashboard from the dashboard list.
  • Select Bar as the widget type.
  • Enter VPC Traffic by Action as the widget name.
  • Choose Add to dashboard.

What does this query reveal?

VPC Flow Logs record every network connection attempt in your VPC. The action field is either ACCEPT (traffic allowed by security groups/NACLs) or REJECT (traffic blocked).

A healthy VPC usually shows mostly accepted traffic. If you see a high number of rejected connections, especially on port 22, that could indicate unauthorized SSH attempts. The Secret Mission explores how to alert on exactly this pattern.

Before you check your finished dashboard, what do you expect to see when you open it? Think about the three data sources you just connected.

  • In the navigation pane, choose Dashboards.
  • Choose CentralizedLogDashboard.

You should see three widgets displaying live data: a CPU utilization line graph for your EC2 instance, a bar chart of the top 10 API calls from CloudTrail, and a bar chart showing accepted vs rejected VPC traffic.

Why does this matter?

You started this project with logs scattered across separate services. Now you have a single dashboard where you can see who called what API, what network connections were attempted, and whether your instance is healthy. This is the foundation of unified observability.

In production, teams add dozens of widgets across multiple accounts. The pattern you just built (log groups with subscription filters feeding Firehose for archive, plus Logs Insights queries for real-time analysis) scales to enterprise environments.

Widgets showing no data or empty results?

Logs Insights widgets re-run their query every time the dashboard loads. If the time range selector at the top of the dashboard is set too narrowly (for example, 1 hour) and your logs were generated earlier, the queries may return empty. Expand the time range to 3 hours or 12 hours.

For the EC2 metric widget, confirm your instance is still in a running state. A stopped instance does not publish CPU metrics.

Help me troubleshoot why my dashboard widgets aren't displaying results.

Your unified dashboard is live. You can now see API activity, network traffic, and instance health in one place. Next up, the Secret Mission challenges you to go beyond passive monitoring and create an alarm that actively alerts you when suspicious SSH traffic is detected.

Secret mission

Alert on Rejected SSH Traffic

Your dashboard shows what already happened. But what if something bad is happening right now? In this bonus mission, you'll build a real-time alert that detects rejected SSH connections to your instance and sends you an email notification within minutes.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses several AWS services that incur ongoing charges, including Amazon Data Firehose ingestion fees and CloudWatch Logs storage costs.

Cost warning

Amazon Data Firehose has no free tier and charges $0.029 per GB ingested. CloudTrail delivery to CloudWatch Logs incurs ingestion charges. If left running with even minimal traffic, expect approximately $1-2 per day in combined charges.

If you are not planning to extend this project immediately, choose Pause or Delete below to avoid unexpected costs.

Resources you used:

  • EC2 instance (t2.micro, free-tier eligible for 12 months).
  • CloudTrail trail delivering management events to S3 and CloudWatch Logs.
  • Amazon Data Firehose delivery stream (log-archive-stream).
  • S3 Log Archive bucket with compressed log files.
  • CloudWatch Logs log groups (/aws/cloudtrail and /aws/vpc/flowlogs).
  • VPC Flow Logs on your default VPC.
  • IAM roles (FirehoseToS3Role, CWLtoFirehoseRole, VPCFlowLogsRole).
  • CloudWatch dashboard (CentralizedLogDashboard).
  • SNS topic (RejectedSSHAlerts) and CloudWatch Alarm (Secret Mission).

Keep everything running

No action needed. Choose this if you are still actively building or plan to extend this pipeline to cross-account monitoring.

  • Your Firehose stream continues delivering logs to S3 (charges ~$0.029/GB ingested).
  • CloudTrail keeps recording management events (first trail to S3 is free, but CloudWatch Logs ingestion is charged).
  • Your EC2 instance remains running (free-tier eligible for 12 months).
  • Expect approximately $1-2/day in combined charges with minimal traffic.

Pause - I'll come back to this later

Shut down the EC2 instance to eliminate compute charges. Firehose and CloudTrail continue generating small charges while active.

  • Stop your EC2 instance by running:
  • Note the instance ID from your earlier work, then stop it with the AWS CLI.
aws ec2 stop-instances --instance-ids <your-instance-id>
  • This eliminates EC2 compute charges. Firehose and CloudTrail remain active but generate minimal cost without significant traffic.
  • When you return, start the instance again with:
aws ec2 start-instances --instance-ids <your-instance-id>

Delete - I don't want to use this again

Remove all project resources and start fresh. Run each of the following commands in order from your terminal.

  • Delete the subscription filters first (they must be removed before the delivery stream):
aws logs delete-subscription-filter --log-group-name /aws/cloudtrail --filter-name <your-cloudtrail-filter-name>
aws logs delete-subscription-filter --log-group-name /aws/vpc/flowlogs --filter-name <your-flowlogs-filter-name>
  • Delete the Firehose delivery stream:
aws firehose delete-delivery-stream --delivery-stream-name log-archive-stream
  • Delete the CloudTrail trail:
aws cloudtrail delete-trail --name <your-trail-name>
  • Delete VPC Flow Logs (find your flow log ID with aws ec2 describe-flow-logs first):
aws ec2 describe-flow-logs --query "FlowLogs[*].FlowLogId" --output text
aws ec2 delete-flow-logs --flow-log-ids <your-flow-log-id>
  • Terminate the EC2 instance:
aws ec2 terminate-instances --instance-ids <your-instance-id>
  • Delete the CloudWatch Logs log groups:
aws logs delete-log-group --log-group-name /aws/cloudtrail
aws logs delete-log-group --log-group-name /aws/vpc/flowlogs
  • Delete the CloudWatch dashboard:
aws cloudwatch delete-dashboards --dashboard-names CentralizedLogDashboard
  • Delete the CloudWatch Alarm and SNS topic (from the Secret Mission):
aws cloudwatch delete-alarms --alarm-names RejectedSSHAlarm
aws sns delete-topic --topic-arn <your-sns-topic-arn>
  • Empty and delete the S3 Log Archive bucket:
aws s3 rm s3://<your-log-archive-bucket> --recursive
aws s3 rb s3://<your-log-archive-bucket>
  • Delete the IAM roles (remove inline policies first, then the roles):
aws iam delete-role-policy --role-name FirehoseToS3Role --policy-name <your-policy-name>
aws iam delete-role --role-name FirehoseToS3Role
aws iam delete-role-policy --role-name CWLtoFirehoseRole --policy-name <your-policy-name>
aws iam delete-role --role-name CWLtoFirehoseRole
aws iam delete-role-policy --role-name VPCFlowLogsRole --policy-name <your-policy-name>
aws iam delete-role --role-name VPCFlowLogsRole

Nice Work!

Nice Work!

You just built a centralized log pipeline that captures API activity and network traffic across your AWS environment, streams it through Amazon Data Firehose into a single archive, and visualizes it all on a unified dashboard.

Here's what you learned:

  • Built a centralized logging pipeline using CloudTrail for API management events and VPC Flow Logs for network traffic, both feeding into CloudWatch Logs.
  • Created an Amazon Data Firehose delivery stream with subscription filters that automatically batches, compresses, and delivers log data from multiple sources to a single S3 archive bucket.
  • Designed a unified CloudWatch dashboard with Logs Insights queries that let you search across EC2 metrics, API call patterns, and network traffic from a single pane of glass.
  • Secret Mission: Created a metric filter and CloudWatch Alarm that detects rejected SSH traffic and sends an alert via SNS.

Ready to quiz yourself?