Build a Security Monitoring System

Create alerts for when your sensitive information is accessed using AWS CloudTrail, CloudWatch, and SNS.

Introduction

⚡️ 30 second Summary

Welcome to this project on setting up a monitoring system with AWS CloudTrail, CloudWatch and SNS! 🔔

Ever wondered how to keep tabs on who’s accessing your most sensitive data in AWS? Whether it’s API keys, database credentials, or other critical secrets, it's a security risk every time someone accesses your confidential information. That's why companies invest heavily in robust monitoring systems to track and alert on any unusual activity.

In this project, you'll build your own powerful monitoring system using AWS. Protecting data is an essential skill for roles like Security Engineer, DevOps Engineer, and Systems or Cloud Administrator.

In this project, get ready to...
  • 🏞️ Set up AWS CloudTrail to track secret access events.
  • 🔎 Use AWS CloudWatch to log access attempts and trigger notifications.
  • 🔔 Create SNS alerts to get notified when your secrets are accessed.
  • 💎 Build a second notification system and compare which approach delivers better security alerts.

If you're up for a bit of a challenge, quiz yourself on the key concepts up ahead in this project.

Before we start Step #1...

By the end of this project, you'll learn how to securely store secrets, track who's accessing them, and get notified instantly when something suspicious happens.

We’ll break down today's project into two stages:

  • Stage 1: Set up the secret & logging
    • In 🔑 Step #1, you'll create a secret in AWS Secrets Manager.
    • In 🏞️ Step #2, you'll enable CloudTrail to record logs of your AWS account's activity.
    • In 😈 Step #3, you'll test your CloudTrail set up - let's check whether it records a log when you access your secret!
  • Stage 2: Set up the monitoring & alert system
    • In 🔎 Step #4, you'll set up a CloudWatch filter that looks for logs about accessing your secret.
    • In 🔔 Step #5, you'll configure a CloudWatch Alarm and SNS to send you an email when a new event passes the filters (i.e. when your secret's been accessed).
    • In 💌 Step #6, you'll test and troubleshoot your entire monitoring system!

Create a Secret

Let's start by creating a secret in AWS Secrets Manager. This secret will be something we want to monitor access to.

In this step, you're going to:

  • Create a new Secrets Manager secret.

Create a New Secret

  • Log in to the AWS Management Console as your IAM Admin user.
  • In the AWS Management Console search bar at the top, search for Secrets Manager and select Secrets Manager from the results.

What is AWS Secrets Manager?

AWS Secrets Manager helps you protect secrets, which are passwords, API keys, credentials and sensitive information. Instead of storing important credentials in your code (yikes!) or sharing them via email (double yikes!), you can tuck them safely away in Secrets Manager.

In our project, we're just storing a dummy secret, but in real life, this is where you'd keep database passwords, API keys, and other sensitive information that would cause a major headache if they leaked.

  • Select Store a new secret to begin creating your secret.
  • Under Choose secret type, select Other type of secret.

What are the different secret types?

Secrets Manager gives you a few different options depending on what you're trying to protect...

  • Credentials for RDS database: Perfect when you need to store and automatically rotate passwords for your MySQL, PostgreSQL, or SQL Server databases.
  • Credentials for DocumentDB database: Similar to RDS, but specifically for DocumentDB - one of AWS's document database services.
  • Credentials for other databases: When you need to store credentials for other database types that don't fit the first two options.
  • API keys and other secrets: This is the catch-all category we're using today! It's for anything that isn't a database credential - API keys, OAuth tokens, encryption keys, or just plain text secrets like our demo value.

Enter Your Secret

  • In the Key/value tab, enter The Secret is as the Key.
  • Enter a random secret or hot take that you have as the Value! For example, I need 3 coffees a day to function, or rice is the best carb

Extra for Experts: Why does Secrets Manager use key-value pairs?

A key-value pair is a simple way to store data by associating a name or label (the key) with a specific value. In AWS Secrets Manager, key-value pairs help you structure secrets in a way that makes them easy to retrieve. The key acts as an identifier e.g. "The Secret is", and the value is the actual data you want to protect ("I need 3 coffees a day to function").

A key-value pair structure makes it much easier for your applications to grab exactly what they need when you have different pieces of information (such as a username, password, and website) in a single secret. Rather than parsing through a blob of text to find the password, your app can just ask for the value of the specific key it wants.

Some common real-world examples include:

  • Database credentials where keys might be "username", "password", "host", "port", and "database_name"
  • API authentication where keys could be "api_key", "client_id", "client_secret", and "endpoint_url"
  • Encryption keys with identifiers like "primary_key", "rotation_date", and "algorithm"
  • We'll keep the default Encryption key setting.

Extra for Experts: What is Encryption?

Encryption is like scrambling a message so that only someone with the right key can unscramble it and read it. In our case, Secrets Manager uses an encryption service called AWS KMS (Key Management Service) to encrypt your secrets, so your secret can't be read by anyone without the right access.

You can also learn more about encryption in our project on AWS KMS!

  • Select Next.
  • Welcome to the Configure secret page!
  • Under Secret name, enter TopSecretInfo
  • Under Description - optional, add a description like Secret created for NextWork's project on Building a Monitoring System

What are the other settings we're skipping?

In AWS Secrets Manager, there are several additional settings we're not configuring in this basic project:

Tags are like digital sticky notes that help you organize and categorize your AWS resources. They consist of a key and value (e.g., "Environment: Production" or "Project: Security-Training") and are useful for tracking costs, filtering resources, or enforcing security policies across large AWS environments.

Resource permissions allow you to control which users, roles, or AWS services can access your secret. This is powerful for implementing the principle of least privilege, where you grant only the minimum necessary permissions to each entity that needs to use the secret.

Secret replication lets you copy your secret to multiple AWS regions, ensuring applications running in different geographic locations can access the secret quickly without having to make cross-region calls. This improves performance and provides redundancy if a region experiences issues.

These advanced settings are extremely valuable in production environments but aren't essential for our initial learning project.

  • Click Next.
  • Click Next again to skip the Configure rotation - optional section.

Extra for Experts: What does configure rotation mean in Secrets Manager?

When you enable rotation, Secrets Manager will periodically automatically replace your credentials (like passwords or API keys) with new values, typically every 30, 60, or 90 days. This limits how long a compromised secret would be useful to an attacker.

Let's review your secret set up:

  • Secret type: Other type of secret
  • Encryption key: aws/secretsmanager
  • Secret name: TopSecretInfo
  • Description: Secret created for NextWork's project on Building a Monitoring System
  • Secret replication: Disabled
  • Automatic rotation: Disabled
  • Click Store at the bottom of the review page.

Extra for Experts: What's the sample code in the Secrets Manager setup? Why is it there?

AWS provides these ready-to-use code snippets in multiple programming languages (like Python, Java, JavaScript, .NET) showing exactly how to securely fetch your secret using code.

Instead of developers having to figure out the right approach themselves, they can simply copy this pre-written code into their application, perhaps modify it slightly, and immediately start using the secret securely. A pretty convenient way to adopt good security practices rather than cutting corners with hardcoded credentials or insecure storage methods!

  • You should see a green banner at the top.
  • In the green banner, select View details.
  • You can now see your newly created secret TopSecretInfo

Awesome! You've created your first secret in Secrets Manager. It's looking great. Ready to set up CloudTrail to track access to this secret?

Configure CloudTrail

Now, let's configure CloudTrail to track access to our secret. CloudTrail is a monitoring service - it records events that happened in your AWS account, like creating resources, updating a name or setting... and accessing secrets in Secrets Manager 👀

In this step, you're going to:

  • Create a new CloudTrail trail to record your account's activity.
  • Configure the trail to store logs in an S3 bucket.

Create a New Trail

  • In the AWS Management Console, head to the CloudTrail console.

What is AWS CloudTrail?

AWS CloudTrail is a monitoring service - think of it as an activity recorder throughout your AWS account. It documents every action taken, like who did what, when they did it, and where they did it from.

This continuous recording is super valuable for security (spotting unusual activity), troubleshooting (figuring out what changed when something breaks), and meeting compliance requirements (proving you're following the rules).

In our project, we're using it to keep an eye on who's accessing our secret 👀

  • From the left hand navigation panel, select Trails.
  • Select Create trail to start setting up a new trail.
  • Welcome to the Choose trail attributes page! Let's create a trail!

What is a trail?

A trail tells CloudTrail exactly what activity to record and where to save those recordings. When you create a trail, you're essentially saying "Hey CloudTrail, please keep track of all xyz activities and store the data in this specific location."

You can have multiple trails for different purposes - you'll learn about the different types of trails in the next page!

  • Under Trail name, enter secrets-manager-trail.
  • In the Storage location section, select Create new S3 bucket.

Why use S3 for CloudTrail logs?

CloudTrail logs can grow a lot over time - you're recording every action on your account! S3 gives you practically unlimited storage that's both super durable (your logs won't get lost) and cost-effective (you only pay for what you use).

Plus, S3 works seamlessly with other AWS services, so when you want to analyze those logs later with tools such as Athena or Lambda, you can integrate them easily.

  • Under Trail log bucket and folder, enter a unique bucket name:
nextwork-secrets-manager-trail-[[AWS_SECURITY_MONITORING_INITIALS="yourinitials"]]
  • 🚨 Make sure to uncheck Log file SSE-KMS encryption - otherwise, you'll get charged for creating a new customer managed KMS key!
  • Keep the other default settings.

Extra for experts: What are these other settings?

These additional settings give you more control over your CloudTrail configuration:

  • Log file validation lets you verify that your log files haven't been tampered with after CloudTrail delivered them. When enabled, CloudTrail creates a digital signature alongside each log file, which you can use later to confirm the logs are authentic and unchanged.
  • SNS notification delivery sends you a notification whenever a new log file is delivered to your S3 bucket. This helps you know immediately when new activity has been logged, rather than having to check manually.
  • CloudWatch Logs integration forwards your CloudTrail logs to CloudWatch Logs in addition to S3. This is super useful because it lets you set up real-time monitoring, create metric filters, and trigger alarms based on specific activities in your logs - exactly what we're doing in this project! We'll enable this in Step #4 and focus on CloudTrail for now.
  • Scroll down and select Next.

Configure Log Events

  • On the Choose log events page, ensure Management events is selected under Event type.

What are the different types of CloudTrail events?

CloudTrail captures different types of events, each showing you a different view into what's happening in your AWS account:

  • Management events: These show you admin actions that configure your AWS resources - creating an EC2 instance, updating a security group, or in our case, accessing a secret. We're focusing on these in our project since secret access gets recorded here.
  • Data events: These track high-volume actions that operate ON your resources rather than creating or configuring them - like uploading a file to S3 or running a Lambda function.
  • Insights events: These detect unusual patterns in your management events, like someone suddenly creating 100x more IAM users than normal.
  • Network activity events: These track network-related activities, like changes to your VPC configuration or traffic to a subnet.

Extra for Experts: Why is retrieving a secret a management event, not a data event?

This is a really good question! When you retrieve a secret value using the GetSecretValue API, you're not just reading raw data - you're using a control-plane (i.e. management) action to decrypt and access protected configuration information. The secret itself is considered a configuration resource for your applications.

The distinction matters for practical reasons too - management events are enabled by default in CloudTrail and generally included in your basic CloudTrail costs, while data events require additional configuration and can increase your CloudTrail charges because of their volume. By classifying secret retrieval as a management event, AWS makes it easier to monitor this security-critical operation without incurring the higher costs associated with data event logging.

  • Under API activity, keep both Read and Write checked.

What are Read vs Write activity?

  • Read API activity happens when someone views but doesn't change anything. For example, listing your S3 buckets, describing your EC2 instances, or in our project, viewing (but not changing) metadata about a secret.
  • Write API activity occurs when changes happen - creating, deleting, modifying resources, or even retrieving the value of a secret (which is what we want to monitor).

💡 Extra for Experts: Why is retrieving a secret value considered a "Write" API activity?

This is quite helpful for security monitoring! By treating secret retrieval as a "Write" operation, it ensures these critical security events are captured in your CloudTrail logs even if someone configures CloudTrail to only log write events (which is a common cost-saving measure).

As you come across more and more Secrets Manger API calls, you'll notice a pattern - operations that only access metadata about secrets (like ListSecrets or DescribeSecret) are classified as "Read" operations, while those that access the actual sensitive values or modify secrets are classified as "Write" operations.

  • Check Exclude AWS KMS events.
  • Check Exclude Amazon RDS Data API events.

Extra for Experts: Why are we excluding these events?

KMS, AWS's encryption service, encrypts most resources in AWS and works behind the scenes whenever you access and open an AWS resource you've created. Because of this, KMS events can make up more than 99% of all your CloudTrail events - that's a lot of noise when you're trying to spot important activities!

Similarly, we're excluding Amazon RDS Data API events because they can generate lots of routine database access logs that aren't relevant to our secrets monitoring.

  • Select Next.
  • Review your trail setup on the Review and create page:

Step 1: Choose trail attributes

  • Trail name: secrets-manager-trail
  • Multi-region trail: Yes
  • Apply trail to my organization: Not enabled
  • Trail log location: nextwork-secrets-manager-trail-yourinitials/AWSLogs
  • Log file SSE-KMS encryption: Not enabled
  • Log file validation: Enabled
  • SNS notification delivery: Disabled

Step 2: Choose log events

  • API activity: All
  • Exclude AWS KMS events: Yes
  • Exclude Amazon RDS Data API events: Yes
  • Click Create trail.
  • You should see a green banner at the top - congrats! You've created your trail.

Fantastic! You've set up CloudTrail to monitor API calls.

It's all set to track access to your secret. Let's generate some secret access events in the next step!

Generate Secret Access Events

Now that CloudTrail is set up, let's generate some secret access events. We'll access our secret in a couple of ways to make sure CloudTrail logs these events.

In this step, you're going to:

  • Expose your own secret (aka open and see the secret you recorded)!
  • Check that CloudTrail recorded what you did 👀

Access Your Secret

  • Navigate back to the Secrets Manager console.
  • Pick your TopSecretInfo secret.
  • On the secret details page, scroll down to the Overview section.
  • Select Retrieve secret value.

What does Retrieve secret value do?

When you click Retrieve secret value, you're opening the actual content of your secret. Behind the scenes, this triggers what's called a GetSecretValue API call to AWS. It's this specific API call that we want to monitor with CloudTrail, because it represents someone reading your secret!

  • You should now see the secret value displayed!
  • Select Close to close the secret value display.

Access Your Secret Over AWS CLI

Turns out, the console is not the only way to access a secret.

If you'd like to be a little 💅 extra 💅, here's your chance at accessing your secret in a second way - the AWS CLI!

  • Open AWS CloudShell - click on the CloudShell icon in the AWS Management Console's top navigation bar.

What is AWS CloudShell?

AWS CloudShell gives you a command-line terminal that's already logged into your AWS account and ready to go. This saves you the time you'd usually have to spend on the AWS CLI on your computer, remembering to configure credentials, and keeping everything updated.

💡 What is AWS CLI?

The AWS Command Line Interface (CLI) is like a text-based remote control for all your AWS services. Instead of clicking around in the web console, you can type commands to make things happen. For example, rather than clicking through several screens to create a trail or secret, you can just type a single command as we did. Developers love the CLI because it's fast, can be scripted for automation, and gives you more precise control.

  • The CloudShell terminal should now be opened and ready (it can take up to 30 seconds).
  • In the CloudShell terminal, run the following command.
  • Make sure to replace your-region-code at the end of the command. Use the region code you see when you select your Region dropdown (e.g. us-east-2 for the Ohio region):
aws secretsmanager get-secret-value --secret-id "TopSecretInfo" --region [[AWS_SECURITY_MONITORING_REGION="your-region-code"]]
  • The command should run successfully and give you the secret's value in JSON format.

Oooo... You've generated secret access events through the console and the AWS CLI. Consider this our secret accessed. Do you think CloudTrail knows what you just did?

Let's see whether CloudTrail captured the events 👀

Analyse Your CloudTrail Events

  • Head to the CloudTrail console again.
  • You should now be on the CloudTrail dashboard.
  • In the left navigation pane, select Event history.

What is CloudTrail Event history?

Event history is where you can find all of your account's management events from the last 90 days. Let's use it to quickly confirm whether CloudTrail captured our secret access events, without having to dig through the raw log files in S3.

  • Welcome to the Event history page!
  • Under Lookup attributes, select the dropdown and choose Event source.
  • In the search bar next to Event source, enter secretsmanager.amazonaws.com

What are lookup attributes and event source?

Lookup attributes are like search filters that help you find specific events in your CloudTrail logs:

  • Event source - the AWS service that generated the event (like secretsmanager.amazonaws.com)
  • Event name - the specific action that was performed (like GetSecretValue)
  • User name - who performed the action
  • Resource type - what kind of resource was involved

We're using Event source to filter for only Secrets Manager events, which helps us quickly spot if and when our secret was accessed without wading through tons of unrelated events!

  • You should now see events related to secretsmanager.amazonaws.com.
  • Scroll through the event list - check for an event caled GetSecretValue. This events means your secret's value was retrieved or used. Woah!

Interesting! Now you've confirmed that CloudTrail records an event when someone accesses your secret.

What's next? Imagine if you want to be alerted too... Let's add a notification system so that AWS automatically emails you whenever your secret's being viewed by someone.

Track Secrets Access Using CloudWatch Metrics

Alright! We know CloudTrail can track events for us, next up is to figure out how we can get alerts when your secret does get accessed.

We'll start by sending all of CloudTrail's logs to CloudWatch. Once it's in CloudWatch, we can set up helpful tools like CloudWatch metrics and alarms that can trigger notifications based on events!

In this step, you're going to:

  • Enable CloudWatch Logs for your CloudTrail trail.
  • Define a CloudWatch Metric to track secret access.

Analyse Your CloudWatch Logs

Let's start this step by telling CloudTrail that we want a copy of all logs sent to another service - CloudWatch!

  • Still in your CloudTrail console, select Trails in the left navigation pane.
  • Select your trail secrets-manager-trail
  • You should now be seeing the details of your trail.
  • Scroll down to the CloudWatch Logs section.
  • Select Edit.
  • Check the Enabled checkbox for CloudWatch Logs.

What are Amazon CloudWatch Logs?

Amazon CloudWatch Logs is a service that helps you bring together your logs from different AWS services, including CloudTrail, for visibility, troubleshooting, and analysis.

It's especially powerful because once your logs are in CloudWatch, you can create alerts based on specific patterns (such as someone accessing your secret), visualize trends, or trigger automated responses.

For this project, we're sending our CloudTrail logs to CloudWatch Logs so we can set up alerts when someone accesses our secret.

Extra for Experts: What other logs can CloudWatch Logs store?

CloudWatch Logs can store and monitor logs from a wide variety of sources, not just CloudTrail! Here's what else you can send to CloudWatch Logs:

  • Application logs from your applications running on EC2, ECS, or Kubernetes
  • On-premises server logs using the CloudWatch agent
  • Network logs like VPC Flow Logs and Route 53 DNS query logs
  • IoT device logs for monitoring your connected devices
  • CloudFront access logs to track viewer requests to your CDN content
  • Select New log group.
  • Under Log group name, enter nextwork-secretsmanager-loggroup.

What's a log group?

A log group represents a collection of logs from a specific application or service. We're creating a new log group to store CloudTrail logs that came from our CloudTrail trail.

  • Under IAM Role, select New.
  • Under Role name, enter CloudTrailRoleForCloudWatchLogs_secrets-manager-trail

Extra for Experts: Why create an IAM Role for CloudWatch Logs?

Instead of giving CloudTrail unlimited access to write anywhere it wants, AWS is creating a role that specifically allows it to write logs to CloudWatch Logs - and nothing else!

This follows a security principle called "least privilege" - only giving services exactly the permissions they need, nothing more. This way, even if something unexpected happens, the potential impact remains contained to just what that role can do.

  • Select Save changes to save the new CloudWatch Logs setup.

Verify Your CloudWatch Logs

  • Head to the CloudWatch console. Let's verify that CloudTrail is really passing the logs to a new log group.
  • In the left navigation pane, expand Logs and select Log groups.
  • Welcome to the Log groups page!
  • In the Log groups page, search for and select nextwork-secretsmanager-loggroup
  • You might see multiple Log streams (i.e. subfolders of log groups). Pick any one of them. If you only see one, that's fine too!
  • You should now see heaps of logs inside the stream. If you don't see rows and rows of logs straight away, you might need to wait a few minutes and refresh your page first.
  • Amazing! You can even expand one of the logs to see all the spicy details inside.

Okay... so what? Couldn't I see events in CloudTrail anyway?

Great question! Yes, you could see the same events in CloudTrail Event History, which is great for quick investigations on recent events. Sending logs to CloudWatch Logs is big milestone for us, because:

  1. CloudWatch Logs is where we can set up alerts and automated responses when specific events happen, which we'll start setting up in this step.
  2. CloudTrail Event History only keeps events for 90 days, while CloudWatch Logs can store them for as long as you'd like.
  3. CloudWatch Logs has powerful filtering tools that let us focus on exactly the events we care about.

Create Metric Filter

  • Head back to your log group.
  • At the top of your log group, select Actions and then Create metric filter from the dropdown menu.
  • Welcome to the Define pattern page! This is where we tell CloudWatch about the kind of actions we're looking for.
  • In the Filter pattern field, enter "GetSecretValue"

What are CloudWatch metric filters?

Metric filters automatically scan through your logs looking for specific patterns. Instead of you manually reading thousands of log entries to find mentions of "GetSecretValue", a metric filter automatically detects these patterns and keeps count for you.

We're creating a filter to specifically detect when someone retrieves our secret's value, which is defined by the action "GetSecretValue".

  • We'll get to know the Test pattern section in Step #6 - for now, let's focus on creating the metric filter!
  • Select Next.
  • For the Filter name, let's use GetSecretsValue
  • Under Metric details, name the Metric namespace: SecurityMetrics

What is a metric namespace?

A metric namespace is a like a folder for your CloudWatch metrics. Namespaces help you organize your metrics and prevent naming conflicts with metrics from other AWS services or applications. We're creating a namespace called SecurityMetrics to group all our security-related metrics.

  • Metric name: Enter Secret is accessed.
  • Metric value: Enter 1.
  • Default value: Enter 0.

What are metric and default values?

  • Metric value is what gets recorded when our filter spots a match in the logs. We're setting it to 1 so that each time someone accesses our secret, the counter increases by exactly one.
  • Default value is what gets recorded when our filter doesn't find any matches during a given time period. We're setting it to 0 so that time periods with no secret access show up as zero on our charts, rather than not showing up at all. This gives us a complete picture - we can see both when access happened AND when it didn't.
  • Select Next - let's review our work!
  • Welcome to the Review and create page for the metric filter.
  • Filter pattern: GetSecretValue
  • Filter name: GetSecretValue
  • Metric name: Secret is accessed
  • Metric namespace: SecurityMetrics
  • Applied on transformed logs: -
  • Metric value: 1
  • Default value: 0
  • Unit: -
  • Select Create metric filter to finalize and create your metric filter.
  • You should see a green banner at the top confirming metric filter creation.

Fantastic! You've created a metric filter and metric to track secret access. Next, we'll create a CloudWatch Alarm to notify us when the secret is accessed.

Create CloudWatch Alarm and SNS Topic

Now, let's create a CloudWatch Alarm that triggers when our SecretIsAccessed metric exceeds a threshold. We'll also set up an SNS topic to receive email notifications when the alarm is triggered.

What is Amazon SNS?

Amazon Simple Notification Service (SNS) is AWS's built-in messaging system. It lets your AWS resources send notifications to people (via email, SMS, or mobile push) or even to other applications.

In our project, we're using SNS to send an email alert when CloudWatch detects that someone accessed our secret, so you can respond quickly to potential security issues.

In this step, you're going to:

  • Create a CloudWatch Alarm based on the SecretIsAccessed metric.
  • Create an SNS topic to send email notifications.
  • Subscribe your email address to the SNS topic.

Create CloudWatch Alarm

  • Still in your CloudWatch Alarm's page, select the Metric filters tab.
  • Scroll down and check the box next to the GetSecretValue metric filter.
  • Select Create alarm.
  • Now, welcome to the CloudWatch alarm setup!
  • Under Metric, let's use the following values:
  • Namespace: SecurityMetrics
  • Metric name: Secret is accessed
  • Statistic: Average
  • Period: 5 minutes

What do Statistic and Period mean?

  • Statistic tells CloudWatch how to analyze the CloudWatch metric. In this case, we're saying that the we're looking for the average number of times the secret is accessed.
  • Period is how often CloudWatch checks in on your metric - we've chosen 5 minutes. You can think of it as checking every 5 minutes to see if anyone accessed the secret during that time.
  • Under Conditions, set Threshold type to Static.

Extra for Experts: What does the other setting (anomaly detection) mean?

Anomaly detection is where CloudWatch studies your metric over time (usually two weeks) to understand its typical patterns, including daily and weekly cycles. It then creates a band of "expected values" that automatically adjusts based on your workload.

While Static thresholds are better for clear security boundaries like "alert whenever my secret is accessed" (which is why we're using Static for this project), Anomaly Detection are great for metrics where "normal" changes over time or follows patterns. It reduces false alarm while still catching true anomalies that might indicate problems.

  • Set Whenever SecretIsAccessed is... to Greater/Equal.
  • Set than... to 1.

What are alarm thresholds?

The alarm threshold is when the alarm should trigger. We're setting up a static threshold so that your alarm goes off when the SecretIsAccessed metric is greater than or equal to 1 in a 5-minute period.

This is a very sensitive setting (i.e. it's easy for this alarm to go off) - which is great for high-security scenarios! In a real production environment, you might adjust this depending on how sensitive your secret is. For instance, if it's normal for your application to access a secret 10 times every 5 minutes, you might set a higher threshold to only alert on unusual access patterns.

  • Select Next.
  • We're now on the Configure actions page! This is how we tell CloudWatch what to do when we want to be alerted.
  • Under Notification, keep the default setting In alarm
  • Under the heading Select a notification to the following SNS topic, Select Create new topic.
  • Under Topic name, enter SecurityAlarms.
  • Under Email endpoints, enter your email address that you can access. In the next step, you'll check this inbox for emails (when the alarm goes off)!

What is an SNS topic?

An SNS (Simple Notification Service) topic is like a broadcast channel for your notifications. First, you create the channel (topic), then you invite subscribers (such as your email), and finally, you send messages to the topic. SNS automatically delivers that message to all subscribers.

The beauty of this approach is flexibility - you could start by just emailing yourself about security alerts, but later add a text message notification, trigger an automated response via Lambda, or even integrate with your team's Slack channel - all without changing how the alert is generated, just by adding more subscribers to your topic.

  • Select Create topic.
  • The SNS topic should now be created!
  • Select Next - we'll head to add name and description for the alarm.
  • Under Alarm name, enter Secret is accessed
  • Under Alarm description, enter a description like This alarm goes off whenever a secret in Secrets Manager is accessed.
  • Select Next to review and create the alarm.

Let's review your alarm set up:

  • Namespace: SecurityMetrics
  • Metric name: Secret is accessed
  • Statistic: Average
  • Period: 5 minutes
  • Threshold type: Static
  • Whenever Secret is accessed is: Greater/Equal (>=)
  • than...: 1
  • Notification: When In alarm, send a notification to "SecurityAlarms"
  • Name: Secret is accessed
  • Description: This alarm goes off whenever a secret is Secrets Manager is accessed.
  • Select Create alarm.
  • You should see a green banner at the top confirming that your alarm was created!
  • There's also blue banner below it, telling us that a subscription is pending confirmation.

A subscription? What's that?

Think of a subscription as an email address that will receive the messages you publish to your SNS topic.

Confirm SNS Subscription

  • Ooo check your created alarm's row too - there's a Warning sign under the Actions heading.
  • Select the warning sign.

What does this warning say?

Just like the blue popup, this warning tells us that your alarm is allllllmost working. When you set up your email with SNS, AWS doesn't just start sending you emails right away - that would be a bit intrusive! Instead, they want to make sure it's really you asking for these alerts. That's why they've sent a confirmation email to your inbox.

You'll need to head into your email inbox to confirm your subscription to this alarm.

  • Check your email inbox for an email from AWS Notifications with the subject AWS notification - Subscription Confirmation.
  • This email is to confirm your subscription to the SNS topic you created!
  • Select Confirm subscription.
  • You should now see a Subscription confirmed! page in your browser.

Excellent! You've set up a CloudWatch Alarm and SNS topic. Now, let's test the email notification in the next step.

Test Email Notification

Let's test if our email notification system works as expected. We'll trigger the alarm by accessing the secret again and check if we receive an email notification.

In this step, you're going to:

  • Retrieve your secret value again to trigger the alarm.
  • Troubleshoot your monitoring system - why aren't you getting notified?

Trigger Alarm

  • Head back to the Secrets Manager console. Let's try to trigger our own alarm by retrieving the secret value again!

Note

If you'd like, you could choose to retrieve the secret value over CloudShell instead - do you remember how to do it? Give it a go!

  • Head to your TopSecretInfo secret again.
  • Select Retrieve secret value.
  • Aha! Secret exposed again!
  • What do you think will happen now - are you going to get the alert email about your secret getting accessed?
  • Check your email inbox after a few minutes (it might take up to 5 minutes for the alarm to trigger and the email to arrive).

Ummmm, I didn't get an email

Huh, neither did we... Looks like we have some troubleshooting to do (yay)!

You're super close to the finish line - this is the final thing to do before you see a working monitoring and notification system. You've GOT THIS 🔥

Looks like we're not receiving the email after we trigger the alarm! This is actually a common scenario in real-world cloud environments.

Setting up a monitoring system is one thing, but making sure all the parts work together correctly often requires some troubleshooting.

Troubleshoot Your Notification Setup

There are a few places in our monitoring system where things could be breaking:

  • CloudTrail didn't record the GetSecretValue event.
  • CloudTrail isn't sending logs to CloudWatch.
  • CloudWatch's metric filter isn't filtering logs correctly.
  • CloudWatch's Alarm isn't triggering an action.
  • SNS isn't delivering emails to you.

CloudTrail

Do we think CloudTrail actually recorded the event? What if CloudTrail missed it, and didn't know you've retrieved your secret's value?

You've done this before in 😈 Step #3 and we'll do it again! Do you remember how to check your CloudTrail logs?

  • Head to the CloudTrail console again.
  • You should now be on the CloudTrail dashboard.
  • In the left navigation pane, select Event history.
  • Under Lookup attributes, select the dropdown and choose Event source.
  • In the search bar next to Event source, enter secretsmanager.amazonaws.com
  • You should see at least one GetSecretValue event at the top of the list, which matches the time you viewed your secret. There should be more than one row, since it's now your second time retrieving a secret.

Nice - that should mean CloudTrail did capture the event!

Note

Don't forget to jump back up to the top of these tabs!

Let's check off all the troubleshooting steps.

Log Delivery

Hmmm... could it be that CloudTrail isn't sending Logs to CloudWatch?

  • Stay in the CloudTrail console.
  • Select Trails in the left navigation pane.
  • Click on your secrets-manager-trail.
  • Check the Last log file delivered timestamp - it should be recent if logs are being delivered properly!

I don't see a last delivery timestamp!

If CloudTrail isn't delivering logs to CloudWatch:

  • Click Edit next to CloudWatch Logs.
  • Ensure Enabled is checked.
  • Verify the log group name is correct.
  • Make sure the IAM role has permissions to write to CloudWatch Logs.
  • Save your changes.
  • Scroll down to the CloudWatch Logs section.
  • Make sure it shows the correct log group name i.e. nextwork-secretsmanager-loggroup.

Nice - that should mean CloudTrail is sending logs over to CloudWatch!

Note

Don't forget to jump back up to the top of these tabs!

Let's check off all the troubleshooting steps.

Metric Filter

What if something went wrong with the CloudWatch metric filter? Maybe the filter is rejecting the events we're actually trying to capture...

  • Head to the CloudWatch console.
  • In the left navigation pane, select Log groups.
  • Find and click on your nextwork-secretsmanager-loggroup.
  • Select the Metric filters tab.
  • Check the checkbox for your filter GetSecretsValue and select Edit.
  • Let's put your filter to the test! Here's a bunch of test log events. Enter them in the Log event messages panel.
{"Records":[
{"eventVersion":"1.11","userIdentity":{"type":"IAMUser","principalId":"AIDAXXXXXXXXXXXXXXXX","arn":"arn:aws:iam::00000000000:user/IAM-Admin","accountId":"00000000000","accessKeyId":"ASIAXXXXXXXXXXXXXXXX","userName":"IAM-Admin","sessionContext":{"attributes":{"creationDate":"2025-03-19T09:17:54Z","mfaAuthenticated":"false"}}},"eventTime":"2025-03-19T18:44:29Z","eventSource":"secretsmanager.amazonaws.com","eventName":"GetSecretValue","awsRegion":"us-west-2","sourceIPAddress":"000.00.00.00","userAgent":"Some user agent info here","requestParameters":{"secretId":"arn:aws:secretsmanager:us-west-2:00000000000:secret:TopSecretInfo-lvRO68"},"responseElements":null,"requestID":"1471f5f6-aee8-4c56-92c8-eadd8ea4b3a2","eventID":"06e83132-44aa-412c-8e1d-da427a6dc6b1","readOnly":true,"eventType":"AwsApiCall","managementEvent":true,"recipientAccountId":"00000000000","eventCategory":"Management","tlsDetails":{"tlsVersion":"TLSv1.3","cipherSuite":"TLS_AES_128_GCM_SHA256","clientProvidedHostHeader":"secretsmanager.us-west-2.amazonaws.com"},"sessionCredentialFromConsole":"true"},
{"eventVersion":"1.11","userIdentity":{"type":"IAMUser","principalId":"AIDAXXXXXXXXXXXXXXXX","arn":"arn:aws:iam::00000000000:user/IAM-Admin","accountId":"00000000000","accessKeyId":"ASIAXXXXXXXXXXXXXXXX","userName":"IAM-Admin","sessionContext":{"attributes":{"creationDate":"2025-03-19T09:17:54Z","mfaAuthenticated":"false"}}},"eventTime":"2025-03-19T18:44:27Z","eventSource":"secretsmanager.amazonaws.com","eventName":"GetResourcePolicy","awsRegion":"us-west-2","sourceIPAddress":"000.00.00.00","userAgent":"Some user agent info here","requestParameters":{"secretId":"arn:aws:secretsmanager:us-west-2:00000000000:secret:TopSecretInfo-lvRO68"},"responseElements":null,"requestID":"6ca2b8cb-1da9-4f0b-8e2d-244aa0ef95d0","eventID":"88a52c53-19ce-4341-a49f-e08a07fc1c32","readOnly":true,"eventType":"AwsApiCall","managementEvent":true,"recipientAccountId":"00000000000","eventCategory":"Management","tlsDetails":{"tlsVersion":"TLSv1.3","cipherSuite":"TLS_AES_128_GCM_SHA256","clientProvidedHostHeader":"secretsmanager.us-west-2.amazonaws.com"},"sessionCredentialFromConsole":"true"},
{"eventVersion":"1.11","userIdentity":{"type":"IAMUser","principalId":"AIDAXXXXXXXXXXXXXXXX","arn":"arn:aws:iam::00000000000:user/IAM-Admin","accountId":"00000000000","accessKeyId":"ASIAXXXXXXXXXXXXXXXX","userName":"IAM-Admin","sessionContext":{"attributes":{"creationDate":"2025-03-19T09:17:54Z","mfaAuthenticated":"false"}}},"eventTime":"2025-03-19T10:18:06Z","eventSource":"secretsmanager.amazonaws.com","eventName":"ListSecretVersionIds","awsRegion":"us-west-2","sourceIPAddress":"000.00.00.00","userAgent":"Some user agent info here","requestParameters":{"secretId":"arn:aws:secretsmanager:us-west-2:00000000000:secret:TopSecretInfo-lvRO68","maxResults":100},"responseElements":null,"requestID":"9541d6b3-10c7-4356-aedf-9da4a0bebb67","eventID":"ac01a292-7781-408c-a04c-c41e9dfb241f","readOnly":true,"eventType":"AwsApiCall","managementEvent":true,"recipientAccountId":"00000000000","eventCategory":"Management","tlsDetails":{"tlsVersion":"TLSv1.3","cipherSuite":"TLS_AES_128_GCM_SHA256","clientProvidedHostHeader":"secretsmanager.us-west-2.amazonaws.com"},"sessionCredentialFromConsole":"true"},
{"eventVersion":"1.11","userIdentity":{"type":"IAMUser","principalId":"AIDAXXXXXXXXXXXXXXXX","arn":"arn:aws:iam::00000000000:user/IAM-Admin","accountId":"00000000000","accessKeyId":"ASIAXXXXXXXXXXXXXXXX","userName":"IAM-Admin","sessionContext":{"attributes":{"creationDate":"2025-03-19T09:17:54Z","mfaAuthenticated":"false"}}},"eventTime":"2025-03-19T10:18:01Z","eventSource":"secretsmanager.amazonaws.com","eventName":"GetResourcePolicy","awsRegion":"us-west-2","sourceIPAddress":"000.00.00.00","userAgent":"Some user agent info here","requestParameters":{"secretId":"arn:aws:secretsmanager:us-west-2:00000000000:secret:TopSecretInfo-lvRO68"},"responseElements":null,"requestID":"5c396be9-cf7e-4815-9dc9-043625394106","eventID":"f1b6f551-66d9-46c3-a6b4-3adfc6948fcd","readOnly":true,"eventType":"AwsApiCall","managementEvent":true,"recipientAccountId":"00000000000","eventCategory":"Management","tlsDetails":{"tlsVersion":"TLSv1.3","cipherSuite":"TLS_AES_128_GCM_SHA256","clientProvidedHostHeader":"secretsmanager.us-west-2.amazonaws.com"},"sessionCredentialFromConsole":"true"},
{"eventVersion":"1.11","userIdentity":{"type":"IAMUser","principalId":"AIDAXXXXXXXXXXXXXXXX","arn":"arn:aws:iam::00000000000:user/IAM-Admin","accountId":"00000000000","accessKeyId":"ASIAXXXXXXXXXXXXXXXX","userName":"IAM-Admin","sessionContext":{"attributes":{"creationDate":"2025-03-19T09:17:54Z","mfaAuthenticated":"false"}}},"eventTime":"2025-03-19T10:17:54Z","eventSource":"secretsmanager.amazonaws.com","eventName":"ListSecretVersionIds","awsRegion":"us-west-2","sourceIPAddress":"000.00.00.00","userAgent":"Some user agent info here","requestParameters":{"secretId":"arn:aws:secretsmanager:us-west-2:00000000000:secret:TopSecretInfo-lvRO68","maxResults":100},"responseElements":null,"requestID":"a2e29de0-0f66-48e0-948e-777c821c3f56","eventID":"32a3cc8f-12d2-41f9-8ead-7f499ef87980","readOnly":true,"eventType":"AwsApiCall","managementEvent":true,"recipientAccountId":"00000000000","eventCategory":"Management","tlsDetails":{"tlsVersion":"TLSv1.3","cipherSuite":"TLS_AES_128_GCM_SHA256","clientProvidedHostHeader":"secretsmanager.us-west-2.amazonaws.com"},"sessionCredentialFromConsole":"true"},
{"eventVersion":"1.11","userIdentity":{"type":"IAMUser","principalId":"AIDAXXXXXXXXXXXXXXXX","arn":"arn:aws:iam::00000000000:user/IAM-Admin","accountId":"00000000000","accessKeyId":"ASIAXXXXXXXXXXXXXXXX","userName":"IAM-Admin","sessionContext":{"attributes":{"creationDate":"2025-03-19T09:17:54Z","mfaAuthenticated":"false"}}},"eventTime":"2025-03-19T10:17:50Z","eventSource":"secretsmanager.amazonaws.com","eventName":"DescribeSecret","awsRegion":"us-west-2","sourceIPAddress":"000.00.00.00","userAgent":"Some user agent info here","requestParameters":{"secretId":"TopSecretInfo"},"responseElements":null,"requestID":"b6504118-1361-46b0-a1af-229f5701924e","eventID":"a559b1cb-6633-451e-914c-993f1eb77bc6","readOnly":true,"eventType":"AwsApiCall","managementEvent":true,"recipientAccountId":"00000000000","eventCategory":"Management","tlsDetails":{"tlsVersion":"TLSv1.3","cipherSuite":"TLS_AES_128_GCM_SHA256","clientProvidedHostHeader":"secretsmanager.us-west-2.amazonaws.com"},"sessionCredentialFromConsole":"true"},
{"eventVersion":"1.11","userIdentity":{"type":"IAMUser","principalId":"AIDAXXXXXXXXXXXXXXXX","arn":"arn:aws:iam::00000000000:user/IAM-Admin","accountId":"00000000000","accessKeyId":"ASIAXXXXXXXXXXXXXXXX","userName":"IAM-Admin","sessionContext":{"attributes":{"creationDate":"2025-03-19T09:17:54Z","mfaAuthenticated":"false"}}},"eventTime":"2025-03-19T10:17:50Z","eventSource":"secretsmanager.amazonaws.com","eventName":"GetResourcePolicy","awsRegion":"us-west-2","sourceIPAddress":"000.00.00.00","userAgent":"Some user agent info here","requestParameters":{"secretId":"arn:aws:secretsmanager:us-west-2:00000000000:secret:TopSecretInfo-lvRO68"},"responseElements":null,"requestID":"cca46a10-d504-4d42-b8a0-e3df56907c13","eventID":"c97eb70f-6eaa-46c7-830c-118672429622","readOnly":true,"eventType":"AwsApiCall","managementEvent":true,"recipientAccountId":"00000000000","eventCategory":"Management","tlsDetails":{"tlsVersion":"TLSv1.3","cipherSuite":"TLS_AES_128_GCM_SHA256","clientProvidedHostHeader":"secretsmanager.us-west-2.amazonaws.com"},"sessionCredentialFromConsole":"true"},
{"eventVersion":"1.11","userIdentity":{"type":"IAMUser","principalId":"AIDAXXXXXXXXXXXXXXXX","arn":"arn:aws:iam::00000000000:user/IAM-Admin","accountId":"00000000000","accessKeyId":"ASIAW3MEFRAF2VAB6VVO","userName":"IAM-Admin","sessionContext":{"attributes":{"creationDate":"2025-03-19T09:17:54Z","mfaAuthenticated":"false"}}},"eventTime":"2025-03-19T10:16:39Z","eventSource":"secretsmanager.amazonaws.com","eventName":"GetSecretValue","awsRegion":"us-west-2","sourceIPAddress":"000.00.00.00","userAgent":"Some user agent info here","requestParameters":{"secretId":"arn:aws:secretsmanager:us-west-2:00000000000:secret:TopSecretInfo-lvRO68"},"responseElements":null,"requestID":"3cc2aa25-f1cd-429f-9ede-7bd245fb8263","eventID":"8948ce7e-42bc-4f01-848d-09439980b454","readOnly":true,"eventType":"AwsApiCall","managementEvent":true,"recipientAccountId":"00000000000","eventCategory":"Management","tlsDetails":{"tlsVersion":"TLSv1.3","cipherSuite":"TLS_AES_128_GCM_SHA256","clientProvidedHostHeader":"secretsmanager.us-west-2.amazonaws.com"},"sessionCredentialFromConsole":"true"},
{"eventVersion":"1.11","userIdentity":{"type":"IAMUser","principalId":"AIDAXXXXXXXXXXXXXXXX","arn":"arn:aws:iam::00000000000:user/IAM-Admin","accountId":"00000000000","accessKeyId":"ASIAW3MEFRAF2VAB6VVO","userName":"IAM-Admin","sessionContext":{"attributes":{"creationDate":"2025-03-19T09:17:54Z","mfaAuthenticated":"false"}}},"eventTime":"2025-03-19T10:14:26Z","eventSource":"secretsmanager.amazonaws.com","eventName":"DescribeSecret","awsRegion":"us-west-2","sourceIPAddress":"000.00.00.00","userAgent":"Some user agent info here","requestParameters":{"secretId":"TopSecretInfo"},"responseElements":null,"requestID":"ab02eb00-acfa-4dcf-802b-8feb329c1534","eventID":"2517d82f-b182-4bcd-9b60-f14545addf00","readOnly":true,"eventType":"AwsApiCall","managementEvent":true,"recipientAccountId":"00000000000","eventCategory":"Management","tlsDetails":{"tlsVersion":"TLSv1.3","cipherSuite":"TLS_AES_128_GCM_SHA256","clientProvidedHostHeader":"secretsmanager.us-west-2.amazonaws.com"},"sessionCredentialFromConsole":"true"},
{"eventVersion":"1.11","userIdentity":{"type":"IAMUser","principalId":"AIDAXXXXXXXXXXXXXXXX","arn":"arn:aws:iam::00000000000:user/IAM-Admin","accountId":"00000000000","accessKeyId":"ASIAW3MEFRAF2VAB6VVO","userName":"IAM-Admin","sessionContext":{"attributes":{"creationDate":"2025-03-19T09:17:54Z","mfaAuthenticated":"false"}}},"eventTime":"2025-03-19T10:14:26Z","eventSource":"secretsmanager.amazonaws.com","eventName":"GetResourcePolicy","awsRegion":"us-west-2","sourceIPAddress":"000.00.00.00","userAgent":"Some user agent info here","requestParameters":{"secretId":"arn:aws:secretsmanager:us-west-2:00000000000:secret:TopSecretInfo-lvRO68"},"responseElements":null,"requestID":"92d2f0c3-30ca-4d1c-a858-1bd691e97a83","eventID":"b7913e5c-dabf-464b-89fd-760185a82902","readOnly":true,"eventType":"AwsApiCall","managementEvent":true,"recipientAccountId":"00000000000","eventCategory":"Management","tlsDetails":{"tlsVersion":"TLSv1.3","cipherSuite":"TLS_AES_128_GCM_SHA256","clientProvidedHostHeader":"secretsmanager.us-west-2.amazonaws.com"},"sessionCredentialFromConsole":"true"},
{"eventVersion":"1.11","userIdentity":{"type":"IAMUser","principalId":"AIDAXXXXXXXXXXXXXXXX","arn":"arn:aws:iam::00000000000:user/IAM-Admin","accountId":"00000000000","accessKeyId":"ASIAW3MEFRAF3UQQZW7R","userName":"IAM-Admin","sessionContext":{"attributes":{"creationDate":"2025-03-19T09:17:54Z","mfaAuthenticated":"false"}}},"eventTime":"2025-03-19T10:11:42Z","eventSource":"secretsmanager.amazonaws.com","eventName":"GetResourcePolicy","awsRegion":"us-west-2","sourceIPAddress":"000.00.00.00","userAgent":"Some user agent info here","requestParameters":{"secretId":"arn:aws:secretsmanager:us-west-2:00000000000:secret:TopSecretInfo-lvRO68"},"responseElements":null,"requestID":"5a820410-fa2e-41be-89b4-a56a9b60c310","eventID":"58daeab8-96b4-42f7-9f8d-a3ced4246480","readOnly":true,"eventType":"AwsApiCall","managementEvent":true,"recipientAccountId":"00000000000","eventCategory":"Management","tlsDetails":{"tlsVersion":"TLSv1.3","cipherSuite":"TLS_AES_128_GCM_SHA256","clientProvidedHostHeader":"secretsmanager.us-west-2.amazonaws.com"},"sessionCredentialFromConsole":"true"},
{"eventVersion":"1.11","userIdentity":{"type":"IAMUser","principalId":"AIDAXXXXXXXXXXXXXXXX","arn":"arn:aws:iam::00000000000:user/IAM-Admin","accountId":"00000000000","accessKeyId":"ASIAW3MEFRAF3UQQZW7R","userName":"IAM-Admin","sessionContext":{"attributes":{"creationDate":"2025-03-19T09:17:54Z","mfaAuthenticated":"false"}}},"eventTime":"2025-03-19T10:11:42Z","eventSource":"secretsmanager.amazonaws.com","eventName":"DescribeSecret","awsRegion":"us-west-2","sourceIPAddress":"000.00.00.00","userAgent":"Some user agent info here","requestParameters":{"secretId":"TopSecretInfo"},"responseElements":null,"requestID":"de945614-69f1-462f-8ca0-2b45fd8bd85f","eventID":"a63ef733-892d-429e-8afb-2dd2b18f969d","readOnly":true,"eventType":"AwsApiCall","managementEvent":true,"recipientAccountId":"00000000000","eventCategory":"Management","tlsDetails":{"tlsVersion":"TLSv1.3","cipherSuite":"TLS_AES_128_GCM_SHA256","clientProvidedHostHeader":"secretsmanager.us-west-2.amazonaws.com"},"sessionCredentialFromConsole":"true"},
{"eventVersion":"1.11","userIdentity":{"type":"IAMUser","principalId":"AIDAXXXXXXXXXXXXXXXX","arn":"arn:aws:iam::00000000000:user/IAM-Admin","accountId":"00000000000","accessKeyId":"ASIAW3MEFRAFYGPUHL43","userName":"IAM-Admin","sessionContext":{"attributes":{"creationDate":"2025-03-19T09:17:54Z","mfaAuthenticated":"false"}}},"eventTime":"2025-03-19T09:34:11Z","eventSource":"secretsmanager.amazonaws.com","eventName":"GetSecretValue","awsRegion":"us-west-2","sourceIPAddress":"000.00.00.00","userAgent":"Some user agent info here","requestParameters":{"secretId":"arn:aws:secretsmanager:us-west-2:00000000000:secret:TopSecretInfo-lvRO68"},"responseElements":null,"requestID":"048367e2-093a-411c-a23f-7c2784a2ce3b","eventID":"98012fdf-147c-49a1-903a-6cb7ce80d24e","readOnly":true,"eventType":"AwsApiCall","managementEvent":true,"recipientAccountId":"00000000000","eventCategory":"Management","tlsDetails":{"tlsVersion":"TLSv1.3","cipherSuite":"TLS_AES_128_GCM_SHA256","clientProvidedHostHeader":"secretsmanager.us-west-2.amazonaws.com"},"sessionCredentialFromConsole":"true"},
{"eventVersion":"1.11","userIdentity":{"type":"IAMUser","principalId":"AIDAXXXXXXXXXXXXXXXX","arn":"arn:aws:iam::00000000000:user/IAM-Admin","accountId":"00000000000","accessKeyId":"ASIAW3MEFRAFYGPUHL43","userName":"IAM-Admin","sessionContext":{"attributes":{"creationDate":"2025-03-19T09:17:54Z","mfaAuthenticated":"false"}}},"eventTime":"2025-03-19T09:34:08Z","eventSource":"secretsmanager.amazonaws.com","eventName":"GetResourcePolicy","awsRegion":"us-west-2","sourceIPAddress":"000.00.00.00","userAgent":"Some user agent info here","requestParameters":{"secretId":"arn:aws:secretsmanager:us-west-2:00000000000:secret:TopSecretInfo-lvRO68"},"responseElements":null,"requestID":"b4e9ad41-1e3b-4d0e-a5f2-705ebb8bbf1a","eventID":"0f34638c-c399-4e81-80e2-b349e1e5e0ee","readOnly":true,"eventType":"AwsApiCall","managementEvent":true,"recipientAccountId":"00000000000","eventCategory":"Management","tlsDetails":{"tlsVersion":"TLSv1.3","cipherSuite":"TLS_AES_128_GCM_SHA256","clientProvidedHostHeader":"secretsmanager.us-west-2.amazonaws.com"},"sessionCredentialFromConsole":"true"},
{"eventVersion":"1.11","userIdentity":{"type":"IAMUser","principalId":"AIDAXXXXXXXXXXXXXXXX","arn":"arn:aws:iam::00000000000:user/IAM-Admin","accountId":"00000000000","accessKeyId":"ASIAW3MEFRAFYGPUHL43","userName":"IAM-Admin","sessionContext":{"attributes":{"creationDate":"2025-03-19T09:17:54Z","mfaAuthenticated":"false"}}},"eventTime":"2025-03-19T09:34:08Z","eventSource":"secretsmanager.amazonaws.com","eventName":"DescribeSecret","awsRegion":"us-west-2","sourceIPAddress":"000.00.00.00","userAgent":"Some user agent info here","requestParameters":{"secretId":"TopSecretInfo"},"responseElements":null,"requestID":"b812abcb-f102-4c57-b99e-43702b73f377","eventID":"643a639f-a697-4990-875d-b716e60a2cbb","readOnly":true,"eventType":"AwsApiCall","managementEvent":true,"recipientAccountId":"00000000000","eventCategory":"Management","tlsDetails":{"tlsVersion":"TLSv1.3","cipherSuite":"TLS_AES_128_GCM_SHA256","clientProvidedHostHeader":"secretsmanager.us-west-2.amazonaws.com"},"sessionCredentialFromConsole":"true"},
{"eventVersion":"1.11","userIdentity":{"type":"IAMUser","principalId":"AIDAXXXXXXXXXXXXXXXX","arn":"arn:aws:iam::00000000000:user/IAM-Admin","accountId":"00000000000","accessKeyId":"ASIAW3MEFRAFXQ3ZA6E6","userName":"IAM-Admin","sessionContext":{"attributes":{"creationDate":"2025-03-17T21:06:59Z","mfaAuthenticated":"false"}}},"eventTime":"2025-03-17T23:21:57Z","eventSource":"secretsmanager.amazonaws.com","eventName":"GetSecretValue","awsRegion":"us-west-2","sourceIPAddress":"000.00.00.00","userAgent":"Some user agent info here","requestParameters":{"secretId":"arn:aws:secretsmanager:us-west-2:00000000000:secret:TopSecretInfo-lvRO68"},"responseElements":null,"requestID":"5462961f-9a07-4ac0-89ac-358bda01422d","eventID":"c4c54144-834e-4b61-afba-58107f102e82","readOnly":true,"eventType":"AwsApiCall","managementEvent":true,"recipientAccountId":"00000000000","eventCategory":"Management","tlsDetails":{"tlsVersion":"TLSv1.3","cipherSuite":"TLS_AES_128_GCM_SHA256","clientProvidedHostHeader":"secretsmanager.us-west-2.amazonaws.com"},"sessionCredentialFromConsole":"true"},
{"eventVersion":"1.11","userIdentity":{"type":"IAMUser","principalId":"AIDAXXXXXXXXXXXXXXXX","arn":"arn:aws:iam::00000000000:user/IAM-Admin","accountId":"00000000000","accessKeyId":"ASIAW3MEFRAFXQ3ZA6E6","userName":"IAM-Admin","sessionContext":{"attributes":{"creationDate":"2025-03-17T21:06:59Z","mfaAuthenticated":"false"}}},"eventTime":"2025-03-17T23:20:47Z","eventSource":"secretsmanager.amazonaws.com","eventName":"GetResourcePolicy","awsRegion":"us-west-2","sourceIPAddress":"000.00.00.00","userAgent":"Some user agent info here","requestParameters":{"secretId":"arn:aws:secretsmanager:us-west-2:00000000000:secret:TopSecretInfo-lvRO68"},"responseElements":null,"requestID":"e57086d3-6562-422e-9030-500cb09377a3","eventID":"41fe58d3-5279-4c91-82af-9a34ea1a3092","readOnly":true,"eventType":"AwsApiCall","managementEvent":true,"recipientAccountId":"00000000000","eventCategory":"Management","tlsDetails":{"tlsVersion":"TLSv1.3","cipherSuite":"TLS_AES_128_GCM_SHA256","clientProvidedHostHeader":"secretsmanager.us-west-2.amazonaws.com"},"sessionCredentialFromConsole":"true"},
{"eventVersion":"1.11","userIdentity":{"type":"IAMUser","principalId":"AIDAXXXXXXXXXXXXXXXX","arn":"arn:aws:iam::00000000000:user/IAM-Admin","accountId":"00000000000","accessKeyId":"ASIAW3MEFRAFXQ3ZA6E6","userName":"IAM-Admin","sessionContext":{"attributes":{"creationDate":"2025-03-17T21:06:59Z","mfaAuthenticated":"false"}}},"eventTime":"2025-03-17T23:20:47Z","eventSource":"secretsmanager.amazonaws.com","eventName":"DescribeSecret","awsRegion":"us-west-2","sourceIPAddress":"000.00.00.00","userAgent":"Some user agent info here","requestParameters":{"secretId":"TopSecretInfo"},"responseElements":null,"requestID":"c0d24117-034a-42dc-b296-cf54667f958a","eventID":"9230b624-5280-4f93-9304-e3eff8671113","readOnly":true,"eventType":"AwsApiCall","managementEvent":true,"recipientAccountId":"00000000000","eventCategory":"Management","tlsDetails":{"tlsVersion":"TLSv1.3","cipherSuite":"TLS_AES_128_GCM_SHA256","clientProvidedHostHeader":"secretsmanager.us-west-2.amazonaws.com"},"sessionCredentialFromConsole":"true"},
{"eventVersion":"1.11","userIdentity":{"type":"IAMUser","principalId":"AIDAXXXXXXXXXXXXXXXX","arn":"arn:aws:iam::00000000000:user/IAM-Admin","accountId":"00000000000","accessKeyId":"ASIAW3MEFRAF5BIR5VCI","userName":"IAM-Admin","sessionContext":{"attributes":{"creationDate":"2025-03-17T21:06:59Z","mfaAuthenticated":"false"}}},"eventTime":"2025-03-17T22:40:42Z","eventSource":"secretsmanager.amazonaws.com","eventName":"CreateSecret","awsRegion":"us-west-2","sourceIPAddress":"000.00.00.00","userAgent":"Some user agent info here","requestParameters":{"name":"TopSecretInfo","clientRequestToken":"6d546fbc-ebd0-45a7-8ddc-3617337d741f","description":"Secret for monitoring project","forceOverwriteReplicaSecret":false},"responseElements":{"arn":"arn:aws:secretsmanager:us-west-2:00000000000:secret:TopSecretInfo-lvRO68"},"requestID":"c206b241-000e-4bcc-88da-bb78208ea93e","eventID":"cb618040-e1ec-4102-a39d-f4670210f08f","readOnly":false,"eventType":"AwsApiCall","managementEvent":true,"recipientAccountId":"00000000000","eventCategory":"Management","tlsDetails":{"tlsVersion":"TLSv1.3","cipherSuite":"TLS_AES_128_GCM_SHA256","clientProvidedHostHeader":"secretsmanager.us-west-2.amazonaws.com"},"sessionCredentialFromConsole":"true"},
{"eventVersion":"1.11","userIdentity":{"type":"IAMUser","principalId":"AIDAXXXXXXXXXXXXXXXX","arn":"arn:aws:iam::00000000000:user/IAM-Admin","accountId":"00000000000","accessKeyId":"ASIAW3MEFRAF5BIR5VCI","userName":"IAM-Admin","sessionContext":{"attributes":{"creationDate":"2025-03-17T21:06:59Z","mfaAuthenticated":"false"}}},"eventTime":"2025-03-17T22:36:14Z","eventSource":"secretsmanager.amazonaws.com","eventName":"DescribeSecret","awsRegion":"us-west-2","sourceIPAddress":"000.00.00.00","userAgent":"Some user agent info here","errorCode":"ResourceNotFoundException","errorMessage":"Secrets Manager can't find the specified secret.","requestParameters":{"secretId":"TopSecretInfo"},"responseElements":null,"requestID":"8ac10643-61cf-462a-9cde-70a522f26670","eventID":"2b9df39e-9c8a-4bb2-a880-995800ebe35f","readOnly":true,"eventType":"AwsApiCall","managementEvent":true,"recipientAccountId":"00000000000","eventCategory":"Management","tlsDetails":{"tlsVersion":"TLSv1.3","cipherSuite":"TLS_AES_128_GCM_SHA256","clientProvidedHostHeader":"secretsmanager.us-west-2.amazonaws.com"},"sessionCredentialFromConsole":"true"}]}

What is this test log data?

This block of text contains sample CloudTrail log entries that simulate various activities related to AWS Secrets Manager. Each entry represents a different API action that might occur when interacting with your secrets.

The data is formatted as JSON and includes multiple event records showing different operations like GetSecretValue (accessing a secret), DescribeSecret (viewing secret metadata), CreateSecret (creating a new secret), and other common Secrets Manager actions.

We're providing this test data so you can easily validate your metric filter without having to generate many different types of events yourself. When you paste this into the test field, CloudWatch will analyze it using your filter pattern and highlight which events match your criteria (in this case, looking for GetSecretValue events).

This is a common technique used by AWS administrators to test their monitoring rules before deploying them to production environments.

  • Select Test pattern.
  • Aha! Some successfull results.
  • If you scroll further down the results, you'll notice that they record an event called GetSecretValue

This tells us the metric filter does correctly filter for events where your secret is accessed.

Note

Don't forget to jump back up to the top of these tabs!

Let's check off all the troubleshooting steps.

Alarm Configuration

Maybe your CloudWatch alarm isn't getting triggered properly?

Let's see if the alarm can be triggered at all.

Trigger the alarm manually

  • Open AWS CloudShell.
  • First, let's learn about how to write a CLI command for manually triggering our alarm:
aws cloudwatch set-alarm-state help
  • Nice! We get an introduction to the cloudwatch set-alarm-state command. Continuing scrolling down the terminal response by pressing the down arrow on your keyboard.
  • This will show you the exact syntax and required parameters, which are:
    • --alarm-name: The name of the CloudWatch alarm
    • --state-value: Possible values are ALARM, OK, or INSUFFICIENT_DATA
    • --state-reason: Some text explaining why you're changing the state
  • Gotcha! Let's manually set the alarm to ALARM state.
  • First, run q in the CloudWatch terminal to quit 'reading' mode.
  • Next, let's trigger our alarm manually.
aws cloudwatch set-alarm-state \
    --alarm-name "Secret is accessed" \
    --state-value ALARM \
    --state-reason "Manually triggered for testing"
  • Check your email inbox. There should be a notification email that your alarm is in alarm, and Manually triggered for testing as the resason!

I didn't get an email!

Ah interesting! Here are some troubleshooting tips:

  • Check your alarm - is it in in alarm state? If it's not - maybe the command didn't work properly.
  • Make sure you use the exact alarm name in your command as it appears in CloudWatch
  • Wait a few minutes for the email to land in your inbox, and check your spam.
  • Make sure your SNS subscription is in Confirmed state.

Adjust Your Alarm Settings

Hmm... so we know the alarm can trigger an email. Since it's not triggering an email at the moment, we're down to the final investigation. Maybe your CloudWatch alarm isn't being triggered when it should!

Let's check a few critical settings:

  • Head back to the CloudWatch console.
  • Select All alarms from the left hand navigation panel.
  • Check the checkbox for your alarm.
  • Select the Actions dropdown, and select Edit.
  • On the alarm details page, look at the Statistic field.
  • Aha - maybe the statistic should be set to Sum, not Average or any other statistic!
  • This is crucial because:
    • Sum adds up all occurrences of secret access in the period (what we want).
    • Average would calculate an average rate (i.e. the average number of times our secret was accessed per second over the 5 minute period), which might never cross our threshold.
  • Change the Statistic dropdown from Average to Sum.
  • You can also update the Period from 5 minutes to 1 minute, so we can trigger the alarm even faster when we see the Secret's value.
  • Check that the Threshold type is set to Static.
  • Confirm that the condition is Greater/Equal than 1.
  • Select Skip to Preview and create.
  • In the review page, make sure Statistic is now Sum.
  • Select Update alarm at the bottom of the page.

Note

Don't forget to jump back up to the top of these tabs!

Let's check off all the troubleshooting steps.

SNS Subscription

Note

Aha! If you're here because you skipped to the last one, we see you! Try again 😼

Since we've confirmed our email's subscription to SNS, surely that means you should be receiving emails?

Let's confirm this by going directly to SNS 🏃

  • Head to the SNS console.
  • In the left hand navigation panel, select Topics.
  • Select your topic i.e. SecurityAlarms
  • Select Publish message. Let's try to manually send a message to all the emails subscribed to this topic!
  • Add a message Subject like Testing
  • In the message body, enter a quick message like Wassup
  • That's it! Select Publish message.

Extras for Experts: What are the Message attributes we skipped?

Message attributes are extra bits of information about the SNS message itself - like the timestamps, location data, signatures, or IDs - that you can send along with your main message. Think of it as your message's metadata!

The cool thing is that whoever receives your message can check these attributes (or use automatic filters) to figure out what to do with it without having to open the message. For example, a banking app might use message attributes to mark certain notifications as "urgent" or "standard," allowing receiving systems to prioritize the processing of critical alerts.

For our project, we're keeping things simple and skipping these extras, but they're super handy when you need more sophisticated message handling!

  • You should see a confirmation message at the top of the page.
  • Let's dash back to your inbox - it's the moment of truth! Can SNS send emails to you?
  • Ooo, message received!

Wait, I didn't receive an email!

Hmmmm, looks like we have a side quest to do back in your SNS console:

  • Head back to the SNS service
  • In the left navigation pane, select Subscriptions.
  • Look for your email subscription in the list.
  • Check the Status column - it should say "Confirmed". If it says "Pending confirmation," that's our issue!
  • Check your email inbox (including spam/junk folders) for an email from AWS Notifications.
  • Open the confirmation email and click on the Confirm subscription link.
  • If you can't find the confirmation email, you can request a new one by selecting your subscription and choosing Request confirmation from the Actions menu.

You'll be back on track in no time! If you're still having trouble, don't hesitate to ask for help in the NextWork Community 👋

This means SNS is actually set up properly (no issues here either, haha)

Note

Don't forget to jump back up to the top of these tabs!

Let's check off all the troubleshooting steps.

All done?

🏁 Only continue below when you've completed all the troubleshooting steps!

Let's try accessing the secret again after you've made these fixes, and see if you receive a notification this time!

Access Your Secret Again

  • Now that we have direct CloudTrail SNS notifications set up, let's generate another secret access event.
  • Head back to the Secrets Manager console.
  • Navigate to your TopSecretInfo secret again.
  • Select Retrieve secret value to access the secret.
  • This will trigger another secret access event that should now be captured by both our CloudWatch Alarm and the direct CloudTrail notifications.

Verify Your Alarm

  • Head back to the CloudWatch console.
  • Refresh your Secret is accessed alarm.
  • It should be in the In alarm state!
  • You should receive notifications now.

My alarm isn't in alarm

Try retrieving your secrets value again, and wait 2-5 minutes. It can take up to 5 minutes for CloudTrail pick up on the event and for CloudWatch to sound the alarm!

If it's still not in alarm after 5 minutes (and you've accessed the secret multiple times), there's an issue with the metric or filter. Double check that you've ticked off everything in the troubleshooting tips (you can do this).

Is this a success?!

  • Now that you see your alarm in alarm state, head back to your inbox!
  • Look for an email from AWS Notifications with the subject ALARM: "SecretIsAccessedAlarm".
  • YESSSSSS - that's your monitoring system at work!

Congratulations!

You've successfully set up and tested your secret access monitoring and notification system. You should now receive email alerts whenever your secret is accessed.

Secret mission

Now that we've built out an entire monitoring system, you might be wondering why this solution has as many components as it does.

There's CloudTrail, CloudWatch, alarms, SNS and more! So if we took away CloudWatch and alarms from the solution... what do you think would happen?

Your secret mission, should you choose to accept it, is to configure direct CloudTrail notifications and compare that against using CloudWatch and alarms. This challenge will build your critical thinking skills around how architecture decisions (like using CloudWatch) are made when building cloud solutions (like this monitoring system).

In this secret mission, you're going to:

  • Configure direct SNS notifications from CloudTrail
  • Compare notification methods between CloudWatch and CloudTrail
  • Showcase advanced cloud architecture decision-making skills in your documentation!

Apply your Solutions Architect skills

Delete your resources

Delete your resources

It's important to clean up the resources we created in this project - it saves us from getting charged!

Resources to delete:

  • The CloudTrail trail.
  • The S3 bucket for CloudTrail logs.
  • The CloudWatch alarm and log group.
  • The Secrets Manager secret.
  • The SNS topic and subscription.

CloudTrail

  • Head to the CloudTrail console.
  • Select Trails in the left navigation pane.
  • Select the checkbox next to secrets-manager-trail.
  • Select Delete.
  • In the confirmation dialog box, type Delete to confirm deletion.
  • Select Delete.

S3

  • Head to the S3 console.
  • Select Buckets in the left navigation pane.
  • Select your CloudTrail S3 bucket (e.g., nextwork-secrets-manager-trail-yourinitials).
  • Select Empty to empty the bucket before deletion.
  • In the confirmation dialog box, type permanently delete to confirm.
  • Select Delete objects.
  • Wait until the bucket is empty.
  • Select your CloudTrail S3 bucket again.
  • Select Delete.
  • In the confirmation dialog box, enter your bucket name to confirm deletion.
  • Select Delete bucket.
  • The S3 bucket should now be deleted.

CloudWatch

Alarm

  • Head to the CloudWatch console.
  • Select Alarms in the left navigation pane.
  • Select the checkbox next to SecretIsAccessedAlarm.
  • Select Actions and then Delete from the dropdown menu.
  • Select Delete to confirm the deletion.

Log group

  • Select Log groups in the left navigation pane.
  • Search for and select the checkbox next to nextwork-secretsmanager-loggroup.
  • Select Actions and then Delete log group(s) from the dropdown menu.
  • In the confirmation dialog, select Delete to confirm the deletion.
  • The log group should now be deleted from the CloudWatch Log groups list.

Secrets Manager

  • Head to the Secrets Manager console.
  • Select Secrets in the left navigation pane.
  • Select the checkbox next to TopSecretInfo secret.
  • Select Actions and then Delete secret from the dropdown menu.
  • Welcome to the Schedule secret deletion page!
  • Under Schedule secret deletion, set the Waiting period to 7 days (default).
  • Select Schedule deletion.
  • The secret should now be scheduled for deletion.

SNS

Topic

  • Head to the SNS console.
  • Select Topics in the left navigation pane.
  • Select the checkbox next to SecurityAlarms topic.
  • Select Delete.
  • In the confirmation dialog box, type delete me to confirm deletion.
  • Select Delete.
  • The SNS topic should now be deleted.

Subscription

  • Select Subscriptions in the left navigation pane.
  • Select the checkbox next to the subscription you created.
  • Select Delete.
  • Select Delete to confirm the deletion.
  • The SNS subscription should now be deleted.

Note

Don't forget to jump back up to the top of these tabs until you've deleted everything!

Get your documentation!

Get your documentation!

Nice work! 🛡️ You've successfully set up an AWS security monitoring system to track and alert on secret access!

You've learned how to:

  • 🔑 Securely store and manage secrets using AWS Secrets Manager.
  • 📜 Monitor secret access by enabling AWS CloudTrail logging.
  • 🔎 Investigate security events in CloudTrail Event History and S3 logs.
  • 📈 Create CloudWatch Metric Filters to track secret access events.
  • 🔔 Set up CloudWatch Alarms and SNS notifications for real-time security alerts.

p.s. Does it say "Still tasks to complete!" at the bottom of the screen?

This means you still have screenshots left to upload, or questions left to answer!

  1. Press Ctrl+F (Windows) or Command+F (Mac) on your keyboard.
  2. Search for the text Return to later.
  3. Jump straight to your incomplete tasks!
  4. 🙋‍♀️ Still stuck? Ask the community!