Build a Serverless Heartbeat Monitor
Deploy a dead man's switch platform on AWS that alerts when pings stop.
Introduction
30 Second Summary
Every day, critical background processes like backups, cron jobs, and deployment pipelines run silently. When they stop working, nobody notices until the damage is done. Traditional monitoring watches for errors that happen. It cannot detect a job that simply never ran.
In this project, you will build PulseGuard, a serverless dead man's switch platform on AWS. External processes send heartbeat pings to prove they're alive. If a ping doesn't arrive on time, PulseGuard detects the silence and fires real alerts.
What You'll Build
A production-grade serverless heartbeat monitoring platform using Go, AWS Lambda, DynamoDB, SQS, SNS, EventBridge Scheduler, and Terraform.
By the end of this project, you'll have:
- A live serverless platform deployed on AWS that processes real heartbeat pings and fires real alerts when processes go silent.
- A complete Infrastructure-as-Code repository with Terraform managing 20+ AWS resources, CI/CD via GitHub Actions, and documented architecture decisions.
- A chaos engineering test suite that proves the system's reliability under failure conditions like DynamoDB throttling and SQS backpressure.
- Secret Mission: Add a webhook notification channel with exponential backoff retry and circuit breaker pattern.
Are there any prerequisites?
You'll need basic Go knowledge (or willingness to learn), an AWS account with free tier access, and comfort with the Linux terminal. Experience with Terraform or Docker is helpful but not required.
Before We Start
Before you can build PulseGuard, you need the right tools on your machine. Terraform manages your AWS infrastructure as code, the AWS CLI connects your terminal to your AWS account, and Go compiles the Lambda functions that power the platform.
In this step, you will install and verify each of these tools so you have a clean, working foundation for the rest of the project.
In this step, get ready to:
- Install Terraform and verify it runs.
- Install and configure AWS CLI v2 with a dedicated IAM user.
- Initialize your Go module and verify the Lambda build.
Install Terraform
Terraform lets you describe your entire cloud infrastructure in configuration files, then deploy it all with a single command. You will use it to provision every AWS resource PulseGuard needs.
- Add the HashiCorp GPG key and APT repository by running these commands:
wget -O - https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(grep -oP '(?<=UBUNTU_CODENAME=).*' /etc/os-release || lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list- Install Terraform by running:
sudo apt update && sudo apt install terraform- Verify the installation by running:
terraform version✔️ I see version 1.15 or higher
Terraform is installed and ready. Move on to the next substep.
ⓧ I see an older version
Your system has a stale package cached. Re-run the repository setup commands above to ensure the HashiCorp source is active, then run:
sudo apt update && sudo apt install --only-upgrade terraform- Run terraform version again to confirm you now see 1.15 or higher.
ⓧ Command not found
The APT repository was not added correctly. Retry the GPG key and repository steps above character for character, then run:
sudo apt update && sudo apt install terraform- Run terraform version to confirm you see a version number.
What is Terraform?
Terraform is an Infrastructure as Code (IaC) tool by HashiCorp. You describe the resources you want in .tf files, and Terraform figures out how to create, update, or destroy them in the correct order.
For PulseGuard, Terraform will manage over 20 AWS resources including Lambda functions, DynamoDB tables, SQS queues, and API Gateway endpoints.
Install and configure AWS CLI v2
The AWS CLI is the bridge between your terminal and your AWS account. Terraform uses the credentials you configure here to authenticate with AWS.
- Download and install AWS CLI v2 by running these commands:
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install- Verify the installation by running:
aws --version✔️ I see a version number starting with aws-cli/2
AWS CLI v2 is installed. Continue to the configuration step below.
ⓧ I see aws-cli/1 or an older version
You have AWS CLI version 1 installed. Uninstall it first, then re-run the installation commands above. See the official AWS CLI migration guide for details.
ⓧ Command not found
The install did not complete successfully. Confirm you have unzip installed by running sudo apt install unzip, then re-run the three commands above.
Now you need to create a dedicated IAM user for Terraform with programmatic access and a least-privilege policy.
- Sign in to the AWS Management Console at https://console.aws.amazon.com.
- Open the IAM service by typing IAM in the search bar and selecting it.
- Click Users in the left sidebar.
- Click Create user.
- Enter pulseguard-terraform as the user name.
- Click Next.
- Select Attach policies directly.
- Search for and attach AdministratorAccess.
- Click Next.
- Click Create user.
Why AdministratorAccess?
For learning purposes, AdministratorAccess simplifies the setup. In a production environment, you would create a custom policy granting only the permissions Terraform needs.
- Click on your new pulseguard-terraform user.
- Click the Security credentials tab.
- Scroll down to Access keys and click Create access key.
- Select Command Line Interface (CLI) as the use case.
- Acknowledge the recommendation and click Next.
- Click Create access key.
- Copy the Access key ID and Secret access key somewhere safe. You will not be able to see the secret key again after leaving this page.
Keep your credentials safe
Never commit your Access key ID or Secret access key to version control. Store them securely and treat them like a password.
- Configure the AWS CLI with your new credentials by running:
aws configureWhat does aws configure ask?
The command will prompt you for four values:
- AWS Access Key ID - paste the one you just copied.
- AWS Secret Access Key - paste the secret you just copied.
- Default region name - enter us-east-1 (or your preferred region).
- Default output format - enter json.
- Verify your credentials work by running:
aws sts get-caller-identityYou should see a JSON response with your account ID, user ARN, and user ID. This confirms the CLI can authenticate with AWS.
Seeing an authentication error?
Double-check that you pasted the correct Access Key ID and Secret Access Key. Run aws configure again to re-enter them.
If you see a permissions error, confirm that AdministratorAccess is attached to your pulseguard-terraform user in the IAM console.
help me fix my AWS CLI authentication error
Initialize the Go module and verify the Lambda build
Go is already on your machine. Now you need to create the project workspace and pull in the AWS Lambda SDK so Go can compile Lambda-compatible binaries.
- Move to your Desktop by running this command:
cd ~/Desktop- Create the project folder and move into it by running these commands:
mkdir pulseguard
cd pulseguard- Initialize the Go module by running:
go mod init github.com/[[GITHUB_USERNAME="your-github-username"]]/pulseguardWhat does go mod init do?
This creates a go.mod file that tracks your project's dependencies and module path. The module path is conventionally your GitHub repository URL so Go tooling can resolve imports.
- Install the AWS Lambda Go SDK by running:
go get github.com/aws/aws-lambda-go@v1.47.0- Create a minimal main.go to test the build by running:
cat << 'EOF' > main.go
package main
import (
"context"
"fmt"
"github.com/aws/aws-lambda-go/lambda"
)
func handler(ctx context.Context) (string, error) {
return fmt.Sprintf("PulseGuard is alive"), nil
}
func main() {
lambda.Start(handler)
}
EOF- Build the binary for the provided.al2023 Lambda runtime by running:
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -tags lambda.norpc -o bootstrap main.goWhat does this build command do?
AWS Lambda expects a Linux binary named bootstrap when using the provided.al2023 runtime.
- GOOS=linux GOARCH=amd64 cross-compiles for the Lambda execution environment.
- CGO_ENABLED=0 produces a fully static binary with no C library dependencies.
- -tags lambda.norpc excludes the deprecated RPC component, reducing binary size.
- Confirm the binary was created by running:
ls bootstrapYou should see bootstrap listed in the output. This proves Go can compile Lambda-ready binaries on your machine.
Build failed or no bootstrap file?
Make sure you are inside the pulseguard directory when running the build command.
If you see an import error about aws-lambda-go, run go mod tidy to resolve dependencies, then retry the build.
help me fix my Go Lambda build
- Clean up the test file since you will replace it with real handler code in a later step:
rm main.go bootstrapFinally, create a free Grafana Cloud account so you have a visualization dashboard ready for the observability step later.
- Open https://grafana.com/auth/sign-up/create-user in your browser.
- Sign up with your email or GitHub account.
- Select the Free plan when prompted.
- Note down your Grafana Cloud stack URL. You will use it in the observability step.
✔️ Awesome, I've got everything!
Great. Double check that your go.mod file is saved in the pulseguard folder. The go.sum file was generated automatically by go get.
ⓧ I'd like to double check the full code
Your go.mod should look like this:
module github.com/<you>/pulseguard
go 1.22
require (
github.com/aws/aws-lambda-go v1.47.0
)The go.sum file is auto-generated and contains checksums for your dependencies. You do not need to edit it manually.
Your environment is ready. Next up, you will design the architecture for PulseGuard and make key decisions about how the system's components fit together.
Design the Architecture
Your development environment is ready. Before writing any code, you need a clear picture of how all the pieces fit together.
PulseGuard is an event-driven system where multiple AWS services hand off work to each other in a chain. Designing the architecture upfront prevents costly rework later and gives you a reference document to consult as you build each Lambda function.
In this step, get ready to:
- Create an architecture diagram showing all AWS components and their data flow.
- Write Architecture Decision Records documenting why you chose each service.
- Design the DynamoDB single-table schema for storing monitor state.
Create the architecture diagram
A visual diagram makes the data flow through PulseGuard concrete. You will map every AWS service and show how a heartbeat ping travels from ingestion to alert dispatch.
- Create the docs directory in your project by running this command:
- In your terminal, make sure you are inside your pulseguard project folder from the previous step.
mkdir -p docs/adr- Open draw.io in your browser (or use Excalidraw if you prefer a sketchier style).
- Create a new blank diagram.
- Add the following components as labelled boxes, connected by arrows in this order:
- API Gateway receives the HTTP ping from the external process.
- Ingestion Lambda validates the token and writes to DynamoDB.
- EventBridge Scheduler triggers the Detector Lambda every 1 minute.
- Detector Lambda scans DynamoDB for overdue monitors and pushes messages to SQS.
- Alert Lambda reads from SQS and publishes notifications to SNS (email or SMS).
- Export your diagram as a PNG file.
- In draw.io, click File, then Export as, then PNG.
- Save it as docs/architecture.png in your project folder.
Why draw the diagram first?
When you can see the full data flow on a single page, you spot design gaps early. It also serves as a communication tool if someone else needs to review your architecture later.
- Confirm the file is in place by running:
ls docs/You should see architecture.png and the adr folder listed.
Write Architecture Decision Records
An Architecture Decision Record (ADR) captures the reasoning behind a key technical choice. When future-you (or a teammate) asks "why didn't we use RDS?", the ADR answers immediately.
- Create ADR-001-dynamodb-over-rds.md inside docs/adr/ by pasting the following content:
# ADR-001: Use DynamoDB over RDS
## Status: Accepted
## Context
PulseGuard needs a low-latency, serverless-compatible data store for monitor state.
## Decision
Use DynamoDB single-table design.
## Consequences
No VPC required, scales to zero, pay-per-request pricing, conditional writes for safe state transitions.Why DynamoDB over RDS?
Lambda functions spin up and down constantly. A traditional RDS database requires connection pooling, and Lambda can exhaust connection limits at scale.
DynamoDB has no connections to manage. It delivers single-digit millisecond reads and writes, which matches the sub-second latency PulseGuard needs for ping ingestion.
- Save docs/adr/ADR-001-dynamodb-over-rds.md.
- Create ADR-002-eventbridge-scheduler.md inside docs/adr/ by pasting this content:
# ADR-002: Use EventBridge Scheduler over CloudWatch Events
## Status: Accepted
## Context
Need a reliable 1-minute periodic trigger for the Detector Lambda.
## Decision
Use EventBridge Scheduler.
## Consequences
At-least-once delivery guarantee, flexible rate/cron expressions, dedicated scheduling service.Why EventBridge Scheduler over CloudWatch Events?
CloudWatch Events (now called EventBridge Rules) works for basic cron triggers. But EventBridge Scheduler is a dedicated scheduling service with an at-least-once delivery guarantee.
For a Dead Man's Switch, missed invocations are catastrophic. If the detector fails to run, alerts never fire. The scheduler's delivery guarantee makes it the safer choice.
- Save docs/adr/ADR-002-eventbridge-scheduler.md.
- Create ADR-003-sqs-between-detection-alerting.md inside docs/adr/ by pasting this content:
# ADR-003: Use SQS Between Detection and Alerting
## Status: Accepted
## Context
Decoupling detection from alert dispatch improves resilience.
## Decision
Use SQS queue with DLQ.
## Consequences
Retry logic, backpressure handling, dead-letter queue for failed deliveries.Why SQS between detection and alerting?
If the Alert Lambda fails (SNS throttling, permission error), you do not want to lose the alert. SQS holds the message and retries delivery automatically.
The dead-letter queue catches permanently failed messages. You can inspect them later and replay them after fixing the issue.
- Save docs/adr/ADR-003-sqs-between-detection-alerting.md.
- Verify all three ADRs exist by running:
ls docs/adr/You should see ADR-001-dynamodb-over-rds.md, ADR-002-eventbridge-scheduler.md, and ADR-003-sqs-between-detection-alerting.md listed.
Don't see all three files?
Check that you created each file inside docs/adr/ and not in the project root. If you accidentally saved them in the wrong directory, move them with mv filename docs/adr/.
Help me find where my ADR files ended up.
Design the DynamoDB single-table schema
DynamoDB single-table design stores multiple entity types in one table. PulseGuard uses it to keep all monitor state in a single, fast lookup surface.
- Create docs/schema.md by pasting the following schema definition:
# DynamoDB Single-Table Schema
## Table: pulseguard
- **Partition Key**: tenant_id (String)
- **Sort Key**: monitor_id (String)
- **Attributes**:
- interval (Number) — expected ping interval in seconds
- grace_period (Number) — grace window in seconds
- last_ping_at (String) — ISO8601 timestamp of last ping
- status (String) — healthy | grace | alerting | resolved
- alert_target (String) — SNS topic ARN or emailWhat does this schema do?
The tenant_id partition key enables multi-tenant isolation. Each tenant's monitors live in their own partition, so one tenant's queries never touch another's data.
The status field implements a state machine. A monitor moves through healthy to grace to alerting to resolved. This prevents duplicate alerts from firing.
- Save docs/schema.md.
- Confirm the full docs structure by running:
ls docs/You should see adr, architecture.png, and schema.md listed.
Missing schema.md?
Make sure you saved the file as docs/schema.md and not just schema.md in the project root.
Help me organize my docs folder structure.
✔️ Awesome, I've got everything!
Great. Double check that you have saved all your files in the docs/ directory.
ⓧ I'd like to double check the full code
Here is each file for comparison:
# ADR-001: Use DynamoDB over RDS
## Status: Accepted
## Context
PulseGuard needs a low-latency, serverless-compatible data store for monitor state.
## Decision
Use DynamoDB single-table design.
## Consequences
No VPC required, scales to zero, pay-per-request pricing, conditional writes for safe state transitions.# ADR-002: Use EventBridge Scheduler over CloudWatch Events
## Status: Accepted
## Context
Need a reliable 1-minute periodic trigger for the Detector Lambda.
## Decision
Use EventBridge Scheduler.
## Consequences
At-least-once delivery guarantee, flexible rate/cron expressions, dedicated scheduling service.# ADR-003: Use SQS Between Detection and Alerting
## Status: Accepted
## Context
Decoupling detection from alert dispatch improves resilience.
## Decision
Use SQS queue with DLQ.
## Consequences
Retry logic, backpressure handling, dead-letter queue for failed deliveries.# DynamoDB Single-Table Schema
## Table: pulseguard
- **Partition Key**: tenant_id (String)
- **Sort Key**: monitor_id (String)
- **Attributes**:
- interval (Number) — expected ping interval in seconds
- grace_period (Number) — grace window in seconds
- last_ping_at (String) — ISO8601 timestamp of last ping
- status (String) — healthy | grace | alerting | resolved
- alert_target (String) — SNS topic ARN or emailYour architecture is mapped, your design decisions are documented, and your data model is defined. Next up, you will write the first Go Lambda function that receives heartbeat pings and stores them in DynamoDB.
Build the Ping Ingestion Lambda
Your architecture is designed and documented. Now it's time to turn that design into running code.
The first function you'll build is the Ping Ingestion Lambda. This is the front door of PulseGuard. External processes hit it with an HTTP request to prove they're alive. The handler validates the request, records the timestamp, and emits a metric.
In this step, get ready to:
- Write the Go Lambda handler that validates tokens and updates DynamoDB.
- Implement URL-based token authentication for each monitor.
- Add structured JSON logging and emit a custom CloudWatch metric.
Write the Go Lambda handler
The ingestion handler is a single Go function that receives HTTP requests routed by API Gateway. It validates the monitor's token, updates the last_ping_at timestamp in DynamoDB with a conditional write, and emits a CloudWatch metric.
- Install the AWS SDK v2 packages for DynamoDB and CloudWatch by running this command:
go get github.com/aws/aws-sdk-go-v2/config github.com/aws/aws-sdk-go-v2/service/dynamodb github.com/aws/aws-sdk-go-v2/service/cloudwatchWhat does this command do?
This pulls in three AWS SDK v2 modules. The config package loads credentials and region from your environment. The dynamodb and cloudwatch packages give you typed clients for each service.
- Create the directory for the ingestion Lambda by running:
mkdir -p cmd/ingestion- Create a new file at cmd/ingestion/main.go and add the package declaration, imports, and AWS client initialization:
package main
import (
"context"
"encoding/json"
"fmt"
"log"
"os"
"time"
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/cloudwatch"
cwtypes "github.com/aws/aws-sdk-go-v2/service/cloudwatch/types"
"github.com/aws/aws-sdk-go-v2/service/dynamodb"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
)
var (
ddbClient *dynamodb.Client
cwClient *cloudwatch.Client
tableName = os.Getenv("DYNAMODB_TABLE")
)
func init() {
cfg, err := config.LoadDefaultConfig(context.Background())
if err != nil {
log.Fatalf("failed to load AWS config: %v", err)
}
ddbClient = dynamodb.NewFromConfig(cfg)
cwClient = cloudwatch.NewFromConfig(cfg)
}What does this code do?
The init() function runs once when the Lambda cold-starts. It creates shared DynamoDB and CloudWatch clients that persist across invocations. This avoids recreating clients on every request.
The tableName variable reads from the DYNAMODB_TABLE environment variable, which you'll set later through Terraform.
- Below the init() function, add the structured logging utility:
type LogEntry struct {
Level string `json:"level"`
Message string `json:"message"`
MonitorID string `json:"monitor_id,omitempty"`
TenantID string `json:"tenant_id,omitempty"`
Timestamp string `json:"timestamp"`
}
func logJSON(level, message, monitorID, tenantID string) {
entry := LogEntry{
Level: level,
Message: message,
MonitorID: monitorID,
TenantID: tenantID,
Timestamp: time.Now().UTC().Format(time.RFC3339),
}
b, _ := json.Marshal(entry)
log.Println(string(b))
}What does this code do?
The LogEntry struct defines the shape of every log line. Each log includes the severity level, a message, the monitor and tenant context, and a timestamp.
The logJSON function serializes this struct to JSON and writes it to stdout. CloudWatch Logs can parse these structured entries, making it easy to filter and search later.
- Below the logJSON function, add the main handler and entry point:
func handler(ctx context.Context, req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
monitorID := req.PathParameters["monitor_id"]
token := req.PathParameters["token"]
if monitorID == "" || token == "" {
return events.APIGatewayProxyResponse{StatusCode: 400, Body: "missing monitor_id or token"}, nil
}
// Fetch monitor to validate token
getOut, err := ddbClient.GetItem(ctx, &dynamodb.GetItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
"tenant_id": &types.AttributeValueMemberS{Value: "default"},
"monitor_id": &types.AttributeValueMemberS{Value: monitorID},
},
})
if err != nil || getOut.Item == nil {
logJSON("warn", "monitor not found", monitorID, "default")
return events.APIGatewayProxyResponse{StatusCode: 404, Body: "monitor not found"}, nil
}
storedToken := getOut.Item["token"].(*types.AttributeValueMemberS).Value
if storedToken != token {
logJSON("warn", "invalid token", monitorID, "default")
return events.APIGatewayProxyResponse{StatusCode: 403, Body: "forbidden"}, nil
}
now := time.Now().UTC().Format(time.RFC3339)
_, err = ddbClient.UpdateItem(ctx, &dynamodb.UpdateItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
"tenant_id": &types.AttributeValueMemberS{Value: "default"},
"monitor_id": &types.AttributeValueMemberS{Value: monitorID},
},
UpdateExpression: aws.String("SET last_ping_at = :t"),
ConditionExpression: aws.String("attribute_exists(monitor_id)"),
ExpressionAttributeValues: map[string]types.AttributeValue{
":t": &types.AttributeValueMemberS{Value: now},
},
})
if err != nil {
logJSON("error", fmt.Sprintf("failed to update last_ping_at: %v", err), monitorID, "default")
return events.APIGatewayProxyResponse{StatusCode: 500, Body: "internal error"}, nil
}
// Emit CloudWatch metric
_, _ = cwClient.PutMetricData(ctx, &cloudwatch.PutMetricDataInput{
Namespace: aws.String("PulseGuard"),
MetricData: []cwtypes.MetricDatum{
{
MetricName: aws.String("PingsReceived"),
Value: aws.Float64(1),
Unit: cwtypes.StandardUnitCount,
Dimensions: []cwtypes.Dimension{
{Name: aws.String("MonitorID"), Value: aws.String(monitorID)},
},
},
},
})
logJSON("info", "ping received", monitorID, "default")
return events.APIGatewayProxyResponse{StatusCode: 200, Body: "ok"}, nil
}
func main() {
lambda.Start(handler)
}What does this code do?
The handler function is the core of PulseGuard's ingestion path. It extracts monitor_id and token from the URL path parameters, then validates the token by fetching the monitor record from DynamoDB.
If the token matches, it performs a conditional write (ConditionExpression: attribute_exists(monitor_id)) to update last_ping_at. The conditional write ensures you only update an existing monitor. It never accidentally creates a phantom record.
After a successful write, the handler emits a PingsReceived metric to the PulseGuard CloudWatch namespace. This metric powers your dashboards and alarms in later steps.
- Save cmd/ingestion/main.go.
Seeing red squiggles in your editor?
If your editor highlights unresolved imports, that's expected. The go get command pulled the modules, but your editor may need a restart to pick them up. You can also run go mod tidy to clean up the module graph.
help me fix unresolved imports in my Go Lambda handler
✔️ Awesome, I've got everything!
Great. Double check you've saved cmd/ingestion/main.go before moving on.
ⓧ I'd like to double check the full code
package main
import (
"context"
"encoding/json"
"fmt"
"log"
"os"
"time"
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/cloudwatch"
cwtypes "github.com/aws/aws-sdk-go-v2/service/cloudwatch/types"
"github.com/aws/aws-sdk-go-v2/service/dynamodb"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
)
var (
ddbClient *dynamodb.Client
cwClient *cloudwatch.Client
tableName = os.Getenv("DYNAMODB_TABLE")
)
func init() {
cfg, err := config.LoadDefaultConfig(context.Background())
if err != nil {
log.Fatalf("failed to load AWS config: %v", err)
}
ddbClient = dynamodb.NewFromConfig(cfg)
cwClient = cloudwatch.NewFromConfig(cfg)
}
type LogEntry struct {
Level string `json:"level"`
Message string `json:"message"`
MonitorID string `json:"monitor_id,omitempty"`
TenantID string `json:"tenant_id,omitempty"`
Timestamp string `json:"timestamp"`
}
func logJSON(level, message, monitorID, tenantID string) {
entry := LogEntry{
Level: level,
Message: message,
MonitorID: monitorID,
TenantID: tenantID,
Timestamp: time.Now().UTC().Format(time.RFC3339),
}
b, _ := json.Marshal(entry)
log.Println(string(b))
}
func handler(ctx context.Context, req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
monitorID := req.PathParameters["monitor_id"]
token := req.PathParameters["token"]
if monitorID == "" || token == "" {
return events.APIGatewayProxyResponse{StatusCode: 400, Body: "missing monitor_id or token"}, nil
}
// Fetch monitor to validate token
getOut, err := ddbClient.GetItem(ctx, &dynamodb.GetItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
"tenant_id": &types.AttributeValueMemberS{Value: "default"},
"monitor_id": &types.AttributeValueMemberS{Value: monitorID},
},
})
if err != nil || getOut.Item == nil {
logJSON("warn", "monitor not found", monitorID, "default")
return events.APIGatewayProxyResponse{StatusCode: 404, Body: "monitor not found"}, nil
}
storedToken := getOut.Item["token"].(*types.AttributeValueMemberS).Value
if storedToken != token {
logJSON("warn", "invalid token", monitorID, "default")
return events.APIGatewayProxyResponse{StatusCode: 403, Body: "forbidden"}, nil
}
now := time.Now().UTC().Format(time.RFC3339)
_, err = ddbClient.UpdateItem(ctx, &dynamodb.UpdateItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
"tenant_id": &types.AttributeValueMemberS{Value: "default"},
"monitor_id": &types.AttributeValueMemberS{Value: monitorID},
},
UpdateExpression: aws.String("SET last_ping_at = :t"),
ConditionExpression: aws.String("attribute_exists(monitor_id)"),
ExpressionAttributeValues: map[string]types.AttributeValue{
":t": &types.AttributeValueMemberS{Value: now},
},
})
if err != nil {
logJSON("error", fmt.Sprintf("failed to update last_ping_at: %v", err), monitorID, "default")
return events.APIGatewayProxyResponse{StatusCode: 500, Body: "internal error"}, nil
}
// Emit CloudWatch metric
_, _ = cwClient.PutMetricData(ctx, &cloudwatch.PutMetricDataInput{
Namespace: aws.String("PulseGuard"),
MetricData: []cwtypes.MetricDatum{
{
MetricName: aws.String("PingsReceived"),
Value: aws.Float64(1),
Unit: cwtypes.StandardUnitCount,
Dimensions: []cwtypes.Dimension{
{Name: aws.String("MonitorID"), Value: aws.String(monitorID)},
},
},
},
})
logJSON("info", "ping received", monitorID, "default")
return events.APIGatewayProxyResponse{StatusCode: 200, Body: "ok"}, nil
}
func main() {
lambda.Start(handler)
}Understand the URL-based token authentication
Each monitor in PulseGuard gets a unique ping URL like /ping/{monitor_id}/{token}. The token is embedded directly in the URL path, not in a header.
Why embed tokens in the URL?
Heartbeat pings come from cron jobs, CI pipelines, and backup scripts. Many of these tools can fire a simple HTTP GET but don't support custom headers easily. Embedding the token in the URL means any tool that can hit a URL can ping PulseGuard.
The handler validates the token by fetching the monitor record from DynamoDB and comparing the stored token against the URL parameter. If they don't match, the request gets a 403 response. No session management, no cookies, no OAuth flows required.
Why use a conditional write?
The ConditionExpression: attribute_exists(monitor_id) ensures the update only succeeds if the monitor already exists in the table. Without it, DynamoDB would happily create a new item for a typo in the URL. The conditional write acts as a second safety net after token validation.
Build and verify the Lambda binary
AWS Lambda requires Go functions to be compiled as a Linux binary named bootstrap. The provided.al2023 runtime executes this binary directly.
- Resolve all module dependencies by running:
go mod tidy- Compile the ingestion Lambda into a Linux binary by running:
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -tags lambda.norpc -o bootstrap ./cmd/ingestionWhat does this command do?
GOOS=linux GOARCH=amd64 cross-compiles for the Lambda execution environment regardless of your local OS. CGO_ENABLED=0 produces a statically linked binary with no external C dependencies.
The -tags lambda.norpc flag excludes the deprecated RPC component, reducing the binary size. The output is named bootstrap because that's what the provided.al2023 runtime expects to execute.
- Confirm the binary was created by running:
ls -lh bootstrap- You should see a file named bootstrap with a size around 10-20 MB.
Build failed with missing module errors?
If you see an error like no required module provides package, run go mod tidy again. This resolves transitive dependencies that go get may have missed.
If the build succeeds but produces no file, check that the -o bootstrap flag is present. Without it, Go outputs to a default location.
help me fix my Go Lambda build errors
Your first Lambda is compiled and ready to deploy. Next up, you'll build the Deadline Detector that scans for monitors that have gone silent.
Build the Deadline Detector Lambda
The Ingestion Lambda records when each monitor last pinged. But recording pings is only half the equation.
PulseGuard's core value is detecting the absence of pings. In this step, you will build the Detector Lambda that runs every minute, checks each monitor's deadline, and pushes alerts onto an SQS queue when a monitor goes silent.
In this step, get ready to:
- Write the Go Lambda handler triggered by EventBridge Scheduler that scans DynamoDB for overdue monitors.
- Implement the state machine that transitions monitors through healthy, grace, alerting, and resolved.
- Push missed-deadline monitors onto the SQS alert queue with deduplication keys.
Write the detector Lambda handler that scans DynamoDB
Traditional monitoring triggers on errors. PulseGuard triggers on silence. The Detector Lambda runs on a schedule and compares the current time against each monitor's last ping plus its configured interval and grace period.
- Add the SQS SDK to your Go module by running this command from the pulseguard project root:
go get github.com/aws/aws-sdk-go-v2/service/sqsWhat does this command do?
This downloads the AWS SQS SDK package and adds it to your go.mod file automatically. The Detector Lambda needs SQS to push alert messages downstream.
- Create the detector directory by running:
mkdir -p cmd/detector- Confirm the directory exists by running:
ls cmd/You should see both detector and ingestion listed.
- Create cmd/detector/main.go and add the package declaration, imports, global variables, and initialization function by pasting the following code:
package main
import (
"context"
"encoding/json"
"log"
"os"
"strconv"
"time"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/dynamodb"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
"github.com/aws/aws-sdk-go-v2/service/sqs"
)
var (
ddbClient *dynamodb.Client
sqsClient *sqs.Client
tableName = os.Getenv("DYNAMODB_TABLE")
alertQueueURL = os.Getenv("ALERT_QUEUE_URL")
)
func init() {
cfg, _ := config.LoadDefaultConfig(context.Background())
ddbClient = dynamodb.NewFromConfig(cfg)
sqsClient = sqs.NewFromConfig(cfg)
}What does this code do?
This sets up the Lambda's dependencies. The init() function runs once on cold start and creates both a DynamoDB client (to read monitor state) and an SQS client (to push alerts).
The two environment variables DYNAMODB_TABLE and ALERT_QUEUE_URL will be injected by Terraform when you deploy later. For now, the code just reads them.
- Below the init() function, add the two struct definitions that model monitor state and alert payloads:
type Monitor struct {
TenantID string
MonitorID string
Interval int64
GracePeriod int64
LastPingAt time.Time
Status string
AlertTarget string
}
type AlertMessage struct {
TenantID string `json:"tenant_id"`
MonitorID string `json:"monitor_id"`
AlertTarget string `json:"alert_target"`
DetectionTimestamp string `json:"detection_timestamp"`
DeduplicationKey string `json:"deduplication_key"`
}What are these structs for?
Monitor represents a single monitor's state read from DynamoDB. It holds the interval, grace period, last ping time, and current status.
AlertMessage is the payload sent to SQS when a deadline is missed. The DeduplicationKey combines the monitor ID and detection timestamp to prevent duplicate alerts.
- Below the struct definitions, add the parseMonitor function that converts raw DynamoDB items into typed Go structs:
func parseMonitor(item map[string]types.AttributeValue) Monitor {
m := Monitor{}
if v, ok := item["tenant_id"].(*types.AttributeValueMemberS); ok {
m.TenantID = v.Value
}
if v, ok := item["monitor_id"].(*types.AttributeValueMemberS); ok {
m.MonitorID = v.Value
}
if v, ok := item["interval"].(*types.AttributeValueMemberN); ok {
m.Interval, _ = strconv.ParseInt(v.Value, 10, 64)
}
if v, ok := item["grace_period"].(*types.AttributeValueMemberN); ok {
m.GracePeriod, _ = strconv.ParseInt(v.Value, 10, 64)
}
if v, ok := item["last_ping_at"].(*types.AttributeValueMemberS); ok {
m.LastPingAt, _ = time.Parse(time.RFC3339, v.Value)
}
if v, ok := item["status"].(*types.AttributeValueMemberS); ok {
m.Status = v.Value
}
if v, ok := item["alert_target"].(*types.AttributeValueMemberS); ok {
m.AlertTarget = v.Value
}
return m
}Why type assertions for each field?
DynamoDB returns items as a map of generic AttributeValue interfaces. Each value could be a string (S), number (N), or other type.
The type assertions (*types.AttributeValueMemberS) safely extract the underlying value, and the ok check avoids panics if a field is missing.
- Save cmd/detector/main.go.
Seeing import errors in your editor?
Your editor may show red squiggles because the file is incomplete. The imports reference packages used in functions you have not written yet. These errors resolve once you add the remaining functions in the next substeps.
Help me resolve these import warnings.
Implement the state machine detection logic
A naive detector would fire an alert every single minute a monitor stays overdue. The state machine prevents this by tracking where each monitor sits in the lifecycle.
How does the state machine work?
Each monitor progresses through four states: healthy (pings arriving on time), grace (first missed deadline, one more chance), alerting (alert sent, no duplicates), and resolved (ping resumed after an alert).
The transition from grace to alerting is the only point where an alert fires. If a monitor is already alerting, the detector skips it. This prevents duplicate notifications.
- In cmd/detector/main.go, below parseMonitor, add the handler function that EventBridge Scheduler will invoke every minute:
func handler(ctx context.Context) error {
now := time.Now().UTC()
out, err := ddbClient.Scan(ctx, &dynamodb.ScanInput{
TableName: aws.String(tableName),
})
if err != nil {
log.Printf(`{"level":"error","message":"scan failed: %v"}`, err)
return err
}
for _, item := range out.Items {
monitor := parseMonitor(item)
deadline := monitor.LastPingAt.Add(time.Duration(monitor.Interval+monitor.GracePeriod) * time.Second)
if now.Before(deadline) {
// Still healthy
if monitor.Status != "healthy" {
updateStatus(ctx, monitor, "healthy")
}
continue
}
switch monitor.Status {
case "healthy":
updateStatus(ctx, monitor, "grace")
case "grace":
updateStatus(ctx, monitor, "alerting")
sendAlert(ctx, monitor, now)
case "alerting":
// Already alerting, no duplicate
case "resolved":
updateStatus(ctx, monitor, "alerting")
sendAlert(ctx, monitor, now)
}
}
return nil
}What does this handler do?
The handler scans all monitors from DynamoDB and computes each one's deadline: last_ping_at + interval + grace_period. If the current time is before the deadline, the monitor is healthy.
If the deadline has passed, the switch statement manages transitions. Moving from grace to alerting fires the alert. Staying in alerting does nothing. This is how the state machine prevents duplicate notifications.
- Below the handler function, add the updateStatus helper that writes state transitions to DynamoDB:
func updateStatus(ctx context.Context, m Monitor, newStatus string) {
_, err := ddbClient.UpdateItem(ctx, &dynamodb.UpdateItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
"tenant_id": &types.AttributeValueMemberS{Value: m.TenantID},
"monitor_id": &types.AttributeValueMemberS{Value: m.MonitorID},
},
UpdateExpression: aws.String("SET #s = :s"),
ExpressionAttributeNames: map[string]string{"#s": "status"},
ExpressionAttributeValues: map[string]types.AttributeValue{":s": &types.AttributeValueMemberS{Value: newStatus}},
})
if err != nil {
log.Printf(`{"level":"error","message":"updateStatus failed","monitor_id":%q,"error":%q}`, m.MonitorID, err.Error())
}
}Why use ExpressionAttributeNames here?
The word status is a DynamoDB reserved keyword. Using #s as a placeholder in the expression and mapping it to status in ExpressionAttributeNames avoids a validation error from DynamoDB.
- Save cmd/detector/main.go.
Editor complaining about sendAlert not defined?
The handler function calls sendAlert which you have not written yet. Go will report this at build time, but your editor may flag it now. You will add sendAlert in the next substep.
Is this expected, or did I miss something?
Push missed-deadline monitors onto the SQS alert queue
When the state machine transitions a monitor to alerting, it needs to push the alert downstream without blocking the detection loop. SQS decouples detection from dispatch so that throttled or failed alerts do not prevent other monitors from being checked.
- Below updateStatus, add the sendAlert function and the main entry point:
func sendAlert(ctx context.Context, m Monitor, detectedAt time.Time) {
detectionTS := detectedAt.Format(time.RFC3339)
msg := AlertMessage{
TenantID: m.TenantID,
MonitorID: m.MonitorID,
AlertTarget: m.AlertTarget,
DetectionTimestamp: detectionTS,
DeduplicationKey: m.MonitorID + "#" + detectionTS,
}
body, _ := json.Marshal(msg)
_, err := sqsClient.SendMessage(ctx, &sqs.SendMessageInput{
QueueUrl: aws.String(alertQueueURL),
MessageBody: aws.String(string(body)),
})
if err != nil {
log.Printf(`{"level":"error","message":"sendAlert failed","monitor_id":%q,"error":%q}`, m.MonitorID, err.Error())
}
}
func main() {
lambda.Start(handler)
}What does sendAlert do?
It serializes the monitor metadata into an AlertMessage JSON payload and sends it to the SQS alert queue. The DeduplicationKey combines the monitor ID with the detection timestamp so that downstream consumers can detect and discard duplicates.
The main() function registers the handler with the Lambda runtime, exactly like the Ingestion Lambda.
- Save cmd/detector/main.go.
- Build the detector binary to verify the code compiles by running:
cd cmd/detector && GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -tags lambda.norpc -o bootstrap .You should see no output if the build succeeds. The bootstrap binary now exists in cmd/detector/.
- Confirm the binary was created by running:
ls -la bootstrapYou should see the bootstrap file listed with its size.
Build failing with undefined references?
Make sure you saved cmd/detector/main.go with all the functions in order: parseMonitor, handler, updateStatus, sendAlert, and main. If the error mentions a missing SQS package, check that you ran go get github.com/aws/aws-sdk-go-v2/service/sqs from the project root.
Help me fix my build error.
✔️ Awesome, I've got everything!
Great. Double check you have saved cmd/detector/main.go and confirmed the bootstrap binary builds without errors.
ⓧ I'd like to double check the full code
Here is the complete cmd/detector/main.go file. Compare your version against this:
package main
import (
"context"
"encoding/json"
"log"
"os"
"strconv"
"time"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/dynamodb"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
"github.com/aws/aws-sdk-go-v2/service/sqs"
)
var (
ddbClient *dynamodb.Client
sqsClient *sqs.Client
tableName = os.Getenv("DYNAMODB_TABLE")
alertQueueURL = os.Getenv("ALERT_QUEUE_URL")
)
func init() {
cfg, _ := config.LoadDefaultConfig(context.Background())
ddbClient = dynamodb.NewFromConfig(cfg)
sqsClient = sqs.NewFromConfig(cfg)
}
type Monitor struct {
TenantID string
MonitorID string
Interval int64
GracePeriod int64
LastPingAt time.Time
Status string
AlertTarget string
}
type AlertMessage struct {
TenantID string `json:"tenant_id"`
MonitorID string `json:"monitor_id"`
AlertTarget string `json:"alert_target"`
DetectionTimestamp string `json:"detection_timestamp"`
DeduplicationKey string `json:"deduplication_key"`
}
func handler(ctx context.Context) error {
now := time.Now().UTC()
out, err := ddbClient.Scan(ctx, &dynamodb.ScanInput{
TableName: aws.String(tableName),
})
if err != nil {
log.Printf(`{"level":"error","message":"scan failed: %v"}`, err)
return err
}
for _, item := range out.Items {
monitor := parseMonitor(item)
deadline := monitor.LastPingAt.Add(time.Duration(monitor.Interval+monitor.GracePeriod) * time.Second)
if now.Before(deadline) {
// Still healthy
if monitor.Status != "healthy" {
updateStatus(ctx, monitor, "healthy")
}
continue
}
switch monitor.Status {
case "healthy":
updateStatus(ctx, monitor, "grace")
case "grace":
updateStatus(ctx, monitor, "alerting")
sendAlert(ctx, monitor, now)
case "alerting":
// Already alerting, no duplicate
case "resolved":
updateStatus(ctx, monitor, "alerting")
sendAlert(ctx, monitor, now)
}
}
return nil
}
func parseMonitor(item map[string]types.AttributeValue) Monitor {
m := Monitor{}
if v, ok := item["tenant_id"].(*types.AttributeValueMemberS); ok {
m.TenantID = v.Value
}
if v, ok := item["monitor_id"].(*types.AttributeValueMemberS); ok {
m.MonitorID = v.Value
}
if v, ok := item["interval"].(*types.AttributeValueMemberN); ok {
m.Interval, _ = strconv.ParseInt(v.Value, 10, 64)
}
if v, ok := item["grace_period"].(*types.AttributeValueMemberN); ok {
m.GracePeriod, _ = strconv.ParseInt(v.Value, 10, 64)
}
if v, ok := item["last_ping_at"].(*types.AttributeValueMemberS); ok {
m.LastPingAt, _ = time.Parse(time.RFC3339, v.Value)
}
if v, ok := item["status"].(*types.AttributeValueMemberS); ok {
m.Status = v.Value
}
if v, ok := item["alert_target"].(*types.AttributeValueMemberS); ok {
m.AlertTarget = v.Value
}
return m
}
func updateStatus(ctx context.Context, m Monitor, newStatus string) {
_, err := ddbClient.UpdateItem(ctx, &dynamodb.UpdateItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
"tenant_id": &types.AttributeValueMemberS{Value: m.TenantID},
"monitor_id": &types.AttributeValueMemberS{Value: m.MonitorID},
},
UpdateExpression: aws.String("SET #s = :s"),
ExpressionAttributeNames: map[string]string{"#s": "status"},
ExpressionAttributeValues: map[string]types.AttributeValue{":s": &types.AttributeValueMemberS{Value: newStatus}},
})
if err != nil {
log.Printf(`{"level":"error","message":"updateStatus failed","monitor_id":%q,"error":%q}`, m.MonitorID, err.Error())
}
}
func sendAlert(ctx context.Context, m Monitor, detectedAt time.Time) {
detectionTS := detectedAt.Format(time.RFC3339)
msg := AlertMessage{
TenantID: m.TenantID,
MonitorID: m.MonitorID,
AlertTarget: m.AlertTarget,
DetectionTimestamp: detectionTS,
DeduplicationKey: m.MonitorID + "#" + detectionTS,
}
body, _ := json.Marshal(msg)
_, err := sqsClient.SendMessage(ctx, &sqs.SendMessageInput{
QueueUrl: aws.String(alertQueueURL),
MessageBody: aws.String(string(body)),
})
if err != nil {
log.Printf(`{"level":"error","message":"sendAlert failed","monitor_id":%q,"error":%q}`, m.MonitorID, err.Error())
}
}
func main() {
lambda.Start(handler)
}Your Detector Lambda is ready. Next up, you will build the Alert Dispatch Pipeline that reads from the SQS queue and delivers notifications via SNS.
Build the Alert Dispatch Pipeline
Your Deadline Detector Lambda now identifies monitors that missed their heartbeat window and pushes alert messages onto an SQS queue. The next piece of the puzzle is the Alert Dispatch Lambda, which consumes those messages and actually notifies someone.
In this step, you will build the final Lambda in the pipeline. This function reads alert messages from SQS, publishes notifications via SNS (email or SMS), and uses a deduplication key to guarantee that the same alert is never sent twice.
In this step, get ready to:
- Write the Alert Dispatch Lambda in Go, triggered by SQS.
- Configure the SQS dead-letter queue for failed alert deliveries.
- Add idempotency to prevent duplicate notifications.
Write the SQS-triggered alert handler
The Alert Dispatch Lambda is the third and final function in PulseGuard's processing chain. SQS triggers it automatically whenever a new alert message arrives in the queue.
- Add the SNS SDK package to your Go module by running this command:
go get github.com/aws/aws-sdk-go-v2/service/sns- Create the cmd/alerter directory by running:
mkdir -p cmd/alerter- Create a new file at cmd/alerter/main.go and add the package declaration, imports, and initialization code:
package main
import (
"context"
"encoding/json"
"fmt"
"log"
"os"
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/dynamodb"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
"github.com/aws/aws-sdk-go-v2/service/sns"
)
var (
snsClient *sns.Client
ddbClient *dynamodb.Client
tableName = os.Getenv("DYNAMODB_TABLE")
)
func init() {
cfg, _ := config.LoadDefaultConfig(context.Background())
snsClient = sns.NewFromConfig(cfg)
ddbClient = dynamodb.NewFromConfig(cfg)
}
type AlertMessage struct {
TenantID string `json:"tenant_id"`
MonitorID string `json:"monitor_id"`
AlertTarget string `json:"alert_target"`
DetectionTimestamp string `json:"detection_timestamp"`
DeduplicationKey string `json:"deduplication_key"`
}What does this code do?
- The imports bring in the SQS event types (to receive messages), the SNS client (to publish notifications), and the DynamoDB client (for idempotency tracking).
- The init() function creates both an SNS client and a DynamoDB client from the default AWS config.
- The AlertMessage struct matches the exact JSON shape your Detector Lambda pushes onto the SQS queue.
- Below the AlertMessage struct, add the handler function and the main entrypoint:
func handler(ctx context.Context, sqsEvent events.SQSEvent) error {
for _, record := range sqsEvent.Records {
var msg AlertMessage
if err := json.Unmarshal([]byte(record.Body), &msg); err != nil {
log.Printf(`{"level":"error","message":"failed to parse SQS message: %v"}`, err)
continue
}
// Idempotency check
getOut, err := ddbClient.GetItem(ctx, &dynamodb.GetItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
"tenant_id": &types.AttributeValueMemberS{Value: msg.TenantID},
"monitor_id": &types.AttributeValueMemberS{Value: msg.MonitorID},
},
})
if err == nil && getOut.Item != nil {
if v, ok := getOut.Item["last_alert_key"].(*types.AttributeValueMemberS); ok && v.Value == msg.DeduplicationKey {
log.Printf(`{"level":"info","message":"duplicate alert skipped","dedup_key":%q}`, msg.DeduplicationKey)
continue
}
}
// Publish to SNS
_, err = snsClient.Publish(ctx, &sns.PublishInput{
TopicArn: aws.String(msg.AlertTarget),
Message: aws.String(fmt.Sprintf("ALERT: Monitor %s missed its deadline at %s", msg.MonitorID, msg.DetectionTimestamp)),
Subject: aws.String(fmt.Sprintf("PulseGuard Alert: %s", msg.MonitorID)),
})
if err != nil {
log.Printf(`{"level":"error","message":"SNS publish failed: %v"}`, err)
return err // triggers SQS retry
}
// Record deduplication key
_, _ = ddbClient.UpdateItem(ctx, &dynamodb.UpdateItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
"tenant_id": &types.AttributeValueMemberS{Value: msg.TenantID},
"monitor_id": &types.AttributeValueMemberS{Value: msg.MonitorID},
},
UpdateExpression: aws.String("SET last_alert_key = :k"),
ExpressionAttributeValues: map[string]types.AttributeValue{
":k": &types.AttributeValueMemberS{Value: msg.DeduplicationKey},
},
})
log.Printf(`{"level":"info","message":"alert dispatched","monitor_id":%q}`, msg.MonitorID)
}
return nil
}
func main() {
lambda.Start(handler)
}What does the handler do?
The handler receives an events.SQSEvent containing one or more alert messages from the queue. For each message, it:
- Unmarshals the JSON body into an AlertMessage struct.
- Checks DynamoDB for a matching last_alert_key to skip duplicates.
- Publishes a notification to the SNS topic ARN stored in AlertTarget.
- Records the deduplication key back to DynamoDB so future retries are skipped.
If the SNS publish fails, the function returns an error. This tells SQS the message was not processed successfully, triggering an automatic retry.
- Save cmd/alerter/main.go.
- Verify the file compiles by running:
cd cmd/alerter && go build -o bootstrap . && cd ../..You should see no output if the build succeeds. A bootstrap binary appears in cmd/alerter/.
Seeing import errors?
- Make sure you ran go get github.com/aws/aws-sdk-go-v2/service/sns from the project root before building.
- Check that your import paths match exactly. The SNS import is github.com/aws/aws-sdk-go-v2/service/sns (not service/sqs).
- Run go mod tidy from the project root to resolve any dependency mismatches.
help me debug a Go build error in my alerter Lambda
Configure the SQS dead-letter queue
When the Alert Lambda fails to publish a notification (SNS outage, throttling, permission error), the message needs to go somewhere safe instead of disappearing. This is where the dead-letter queue (DLQ) comes in.
How does the DLQ work with SQS?
When your Lambda returns an error, SQS considers the message unprocessed and makes it visible again after the visibility timeout expires. SQS retries delivery automatically.
After a configured number of failed attempts (you will set this to 3 in Terraform), SQS moves the message to a separate DLQ. Messages in the DLQ stay there until you manually inspect and replay them.
- In cmd/alerter/main.go, find the line:
return err // triggers SQS retryThis single return err is what activates the retry mechanism. By returning a non-nil error to the Lambda runtime, SQS knows this message failed and should be retried.
Why not just log and continue?
If you used continue after an SNS failure, the message would be removed from the queue permanently. The alert would be lost forever.
Returning an error preserves the message for retry. After 3 failures, the DLQ catches it so you can investigate and replay later. You will configure the DLQ infrastructure in Terraform during Step 6.
Add idempotency with the deduplication key
In distributed systems, messages can be delivered more than once. If the Detector Lambda runs twice for the same missed deadline, or SQS retries after a transient error, your Alert Lambda could send duplicate notifications.
How does the deduplication key prevent duplicates?
The Detector Lambda creates a DeduplicationKey by combining monitor_id and detection_timestamp (e.g. my-monitor#2026-05-21T10:00:00Z). This uniquely identifies a single detection event.
Before publishing, the Alert Lambda reads last_alert_key from DynamoDB. If it matches the current message's key, the alert was already sent and the message is skipped. After a successful publish, the new key is written back to DynamoDB.
- In cmd/alerter/main.go, find the idempotency check section:
// Idempotency check
getOut, err := ddbClient.GetItem(ctx, &dynamodb.GetItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
"tenant_id": &types.AttributeValueMemberS{Value: msg.TenantID},
"monitor_id": &types.AttributeValueMemberS{Value: msg.MonitorID},
},
})
if err == nil && getOut.Item != nil {
if v, ok := getOut.Item["last_alert_key"].(*types.AttributeValueMemberS); ok && v.Value == msg.DeduplicationKey {
log.Printf(`{"level":"info","message":"duplicate alert skipped","dedup_key":%q}`, msg.DeduplicationKey)
continue
}
}- And the deduplication key recording section after the SNS publish:
// Record deduplication key
_, _ = ddbClient.UpdateItem(ctx, &dynamodb.UpdateItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
"tenant_id": &types.AttributeValueMemberS{Value: msg.TenantID},
"monitor_id": &types.AttributeValueMemberS{Value: msg.MonitorID},
},
UpdateExpression: aws.String("SET last_alert_key = :k"),
ExpressionAttributeValues: map[string]types.AttributeValue{
":k": &types.AttributeValueMemberS{Value: msg.DeduplicationKey},
},
})Why record the key after publishing?
The order matters for correctness. If you recorded the key first and then the SNS publish failed, the message would be permanently marked as sent even though no notification went out.
By writing the key only after a successful publish, a retry will correctly re-attempt the notification. The worst case is a duplicate alert (key write fails after publish), which is far better than a missed alert.
- Return to the project root and verify all three Lambda binaries build successfully:
cd cmd/ingestion && go build -o bootstrap . && cd ../..
cd cmd/detector && go build -o bootstrap . && cd ../..
cd cmd/alerter && go build -o bootstrap . && cd ../..All three commands should produce no output, confirming a clean build for each Lambda.
Build fails for one of the Lambdas?
- Run go mod tidy from the project root to sync all dependencies across the module.
- Check that each main.go file starts with package main and has a func main() entrypoint.
- If you see version conflicts, delete go.sum and run go mod tidy again to regenerate it.
help me fix a Go build error across my PulseGuard Lambda functions
✔️ Awesome, I've got everything!
Great. Double check you have saved cmd/alerter/main.go and that all three binaries compile cleanly.
ⓧ I'd like to double check the full code
Here is the complete cmd/alerter/main.go file:
package main
import (
"context"
"encoding/json"
"fmt"
"log"
"os"
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/dynamodb"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
"github.com/aws/aws-sdk-go-v2/service/sns"
)
var (
snsClient *sns.Client
ddbClient *dynamodb.Client
tableName = os.Getenv("DYNAMODB_TABLE")
)
func init() {
cfg, _ := config.LoadDefaultConfig(context.Background())
snsClient = sns.NewFromConfig(cfg)
ddbClient = dynamodb.NewFromConfig(cfg)
}
type AlertMessage struct {
TenantID string `json:"tenant_id"`
MonitorID string `json:"monitor_id"`
AlertTarget string `json:"alert_target"`
DetectionTimestamp string `json:"detection_timestamp"`
DeduplicationKey string `json:"deduplication_key"`
}
func handler(ctx context.Context, sqsEvent events.SQSEvent) error {
for _, record := range sqsEvent.Records {
var msg AlertMessage
if err := json.Unmarshal([]byte(record.Body), &msg); err != nil {
log.Printf(`{"level":"error","message":"failed to parse SQS message: %v"}`, err)
continue
}
// Idempotency check
getOut, err := ddbClient.GetItem(ctx, &dynamodb.GetItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
"tenant_id": &types.AttributeValueMemberS{Value: msg.TenantID},
"monitor_id": &types.AttributeValueMemberS{Value: msg.MonitorID},
},
})
if err == nil && getOut.Item != nil {
if v, ok := getOut.Item["last_alert_key"].(*types.AttributeValueMemberS); ok && v.Value == msg.DeduplicationKey {
log.Printf(`{"level":"info","message":"duplicate alert skipped","dedup_key":%q}`, msg.DeduplicationKey)
continue
}
}
// Publish to SNS
_, err = snsClient.Publish(ctx, &sns.PublishInput{
TopicArn: aws.String(msg.AlertTarget),
Message: aws.String(fmt.Sprintf("ALERT: Monitor %s missed its deadline at %s", msg.MonitorID, msg.DetectionTimestamp)),
Subject: aws.String(fmt.Sprintf("PulseGuard Alert: %s", msg.MonitorID)),
})
if err != nil {
log.Printf(`{"level":"error","message":"SNS publish failed: %v"}`, err)
return err // triggers SQS retry
}
// Record deduplication key
_, _ = ddbClient.UpdateItem(ctx, &dynamodb.UpdateItemInput{
TableName: aws.String(tableName),
Key: map[string]types.AttributeValue{
"tenant_id": &types.AttributeValueMemberS{Value: msg.TenantID},
"monitor_id": &types.AttributeValueMemberS{Value: msg.MonitorID},
},
UpdateExpression: aws.String("SET last_alert_key = :k"),
ExpressionAttributeValues: map[string]types.AttributeValue{
":k": &types.AttributeValueMemberS{Value: msg.DeduplicationKey},
},
})
log.Printf(`{"level":"info","message":"alert dispatched","monitor_id":%q}`, msg.MonitorID)
}
return nil
}
func main() {
lambda.Start(handler)
}Your alert pipeline is now complete in code. Next up, you will deploy this entire stack to AWS using Terraform modules, wiring the SQS queue, DLQ, SNS topics, and Lambda triggers together as infrastructure.
Deploy Infrastructure with Terraform
You have three Lambda handlers, an architecture diagram, and detailed ADRs. Now it is time to turn those design decisions into real AWS resources.
In this step, you will write modularized Terraform configurations that declare every piece of PulseGuard's infrastructure. Then you will deploy the full stack with a single command and verify the pipeline end-to-end with a live ping.
In this step, get ready to:
- Write Terraform modules for IAM, DynamoDB, SQS, SNS, Lambda, API Gateway, and EventBridge.
- Configure an S3 backend for Terraform state with DynamoDB state locking.
- Run terraform plan and terraform apply, then verify the stack with a manual curl ping.
Write the Terraform modules
Terraform modules let you group related resources into reusable units. Each module handles one concern: IAM for permissions, DynamoDB for storage, SQS for queuing, and so on.
- Create the module directory structure inside your pulseguard project folder by running these commands:
mkdir -p terraform/modules/{iam,dynamodb,sqs,sns,lambda,api-gateway,eventbridge}
mkdir -p distWhat does this command do?
The mkdir -p command creates nested directories in one shot. The brace expansion creates all seven module folders under terraform/modules/. The dist folder will hold the compiled Lambda zip files that Terraform needs during deployment.
- Confirm the directories exist by running:
ls terraform/modulesYou should see api-gateway, dynamodb, eventbridge, iam, lambda, sns, and sqs listed.
Module interface files
Each Terraform module needs variables.tf (declaring its inputs) and outputs.tf (exposing values other modules consume). Create these for each module, declaring variables that match the var. references you see in the code below, and outputs matching the module.<name>.<output> references in the root main.tf.
- Create the IAM module by adding the following to terraform/modules/iam/main.tf:
resource "aws_iam_role" "lambda" {
name = "pulseguard-lambda-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Service = "lambda.amazonaws.com" }
Action = "sts:AssumeRole"
}]
})
}
resource "aws_iam_role_policy" "lambda" {
role = aws_iam_role.lambda.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{ Effect = "Allow", Action = ["dynamodb:*"], Resource = "*" },
{ Effect = "Allow", Action = ["sqs:*"], Resource = "*" },
{ Effect = "Allow", Action = ["sns:Publish"], Resource = "*" },
{ Effect = "Allow", Action = ["cloudwatch:PutMetricData"], Resource = "*" },
{ Effect = "Allow", Action = ["logs:*"], Resource = "*" }
]
})
}What does this code do?
The IAM module creates a single Lambda execution role that all three functions share. The assume_role_policy allows the Lambda service to assume this role.
The attached policy grants permissions for DynamoDB, SQS, SNS, CloudWatch metrics, and CloudWatch Logs. In production you would scope these to specific resource ARNs, but wildcard is fine for development.
- Save terraform/modules/iam/main.tf.
- Create the DynamoDB module by adding the following to terraform/modules/dynamodb/main.tf:
resource "aws_dynamodb_table" "pulseguard" {
name = var.table_name
billing_mode = "PAY_PER_REQUEST"
hash_key = "tenant_id"
range_key = "monitor_id"
attribute {
name = "tenant_id"
type = "S"
}
attribute {
name = "monitor_id"
type = "S"
}
}What does this code do?
This creates the PulseGuard DynamoDB table with pay-per-request billing. The composite key (tenant_id partition key + monitor_id sort key) matches the single-table schema you designed in Step 2.
Pay-per-request means you pay only for the reads and writes you actually use. This is ideal for bursty workloads like heartbeat pings.
- Save terraform/modules/dynamodb/main.tf.
- Create the SQS and SNS modules. Add the following to terraform/modules/sqs/main.tf:
resource "aws_sqs_queue" "dlq" {
name = var.dlq_name
}
resource "aws_sqs_queue" "alerts" {
name = var.queue_name
redrive_policy = jsonencode({
deadLetterTargetArn = aws_sqs_queue.dlq.arn
maxReceiveCount = 3
})
}- Add the following to terraform/modules/sns/main.tf:
resource "aws_sns_topic" "alerts" {
name = var.topic_name
}What does this code do?
The SQS module creates two queues: the main alert queue and a dead-letter queue (DLQ). The redrive_policy with maxReceiveCount = 3 means if the alerter Lambda fails to process a message three times, SQS moves it to the DLQ for manual inspection.
The SNS module creates a single topic that the alerter publishes to. Subscribers (email, SMS) are added after deployment.
- Save both terraform/modules/sqs/main.tf and terraform/modules/sns/main.tf.
- Create the Lambda module by adding the following to terraform/modules/lambda/main.tf:
resource "aws_lambda_function" "ingestion" {
function_name = "pulseguard-ingestion"
role = var.lambda_role_arn
handler = "bootstrap"
runtime = "provided.al2023"
filename = "${path.module}/../../dist/ingestion.zip"
environment {
variables = {
DYNAMODB_TABLE = var.table_name
}
}
}
resource "aws_lambda_function" "detector" {
function_name = "pulseguard-detector"
role = var.lambda_role_arn
handler = "bootstrap"
runtime = "provided.al2023"
filename = "${path.module}/../../dist/detector.zip"
environment {
variables = {
DYNAMODB_TABLE = var.table_name
ALERT_QUEUE_URL = var.alert_queue_url
}
}
}
resource "aws_lambda_function" "alerter" {
function_name = "pulseguard-alerter"
role = var.lambda_role_arn
handler = "bootstrap"
runtime = "provided.al2023"
filename = "${path.module}/../../dist/alerter.zip"
environment {
variables = {
DYNAMODB_TABLE = var.table_name
}
}
}
resource "aws_lambda_event_source_mapping" "alerter_sqs" {
event_source_arn = var.alert_queue_arn
function_name = aws_lambda_function.alerter.arn
batch_size = 10
}What does this code do?
This module declares all three Lambda functions using the provided.al2023 runtime. Each function's handler is bootstrap because Go compiles to a standalone binary with that name.
The aws_lambda_event_source_mapping resource wires the SQS alert queue directly to the alerter function. When messages arrive in the queue, Lambda polls and invokes the alerter automatically in batches of up to 10.
- Save terraform/modules/lambda/main.tf.
- Create the API Gateway module by adding the following to terraform/modules/api-gateway/main.tf:
resource "aws_apigatewayv2_api" "pulseguard" {
name = "pulseguard"
protocol_type = "HTTP"
}
resource "aws_apigatewayv2_integration" "ingestion" {
api_id = aws_apigatewayv2_api.pulseguard.id
integration_type = "AWS_PROXY"
integration_uri = var.ingestion_lambda_arn
payload_format_version = "2.0"
}
resource "aws_apigatewayv2_route" "ping" {
api_id = aws_apigatewayv2_api.pulseguard.id
route_key = "GET /ping/{monitor_id}/{token}"
target = "integrations/${aws_apigatewayv2_integration.ingestion.id}"
}
resource "aws_apigatewayv2_stage" "default" {
api_id = aws_apigatewayv2_api.pulseguard.id
name = "$default"
auto_deploy = true
}What does this code do?
This creates an HTTP API Gateway with a single route: GET /ping/{monitor_id}/{token}. When a heartbeat ping hits this URL, API Gateway proxies the request to the ingestion Lambda.
The $default stage with auto_deploy = true means every route change deploys automatically. No manual staging needed during development.
- Save terraform/modules/api-gateway/main.tf.
- Create the EventBridge module by adding the following to terraform/modules/eventbridge/main.tf:
resource "aws_scheduler_schedule" "detector" {
name = "pulseguard-detector"
flexible_time_window { mode = "OFF" }
schedule_expression = "rate(1 minute)"
target {
arn = var.detector_lambda_arn
role_arn = var.lambda_role_arn
input = "{}"
}
}What does this code do?
EventBridge Scheduler fires the detector Lambda every 1 minute. The flexible_time_window is OFF because you want exact 1-minute intervals for deadline detection. This is the heartbeat of PulseGuard itself.
- Save terraform/modules/eventbridge/main.tf.
Module won't validate?
If terraform validate complains about undeclared variables, make sure each module has a variables.tf declaring every var. reference used in its main.tf. Similarly, add outputs.tf exposing any value the root module references (like module.iam.lambda_role_arn).
help me set up variables.tf and outputs.tf for my Terraform modules
Configure the S3 backend for Terraform state
Terraform state tracks every resource it manages. Storing state locally is risky because losing the file means Terraform forgets your infrastructure exists. An S3 backend solves this by persisting state remotely with encryption and locking.
- Create the S3 bucket for state storage by running:
aws s3api create-bucket --bucket pulseguard-tfstate --region us-east-1- Create the DynamoDB table for state locking by running:
aws dynamodb create-table \
--table-name pulseguard-tflock \
--attribute-definitions AttributeName=LockID,AttributeType=S \
--key-schema AttributeName=LockID,KeyType=HASH \
--billing-mode PAY_PER_REQUEST \
--region us-east-1Why DynamoDB for state locking?
State locking prevents two people (or CI jobs) from running terraform apply at the same time and corrupting the state file. The lock table must have a partition key named LockID with type String. Terraform writes a lock entry before any state-modifying operation and deletes it when done.
- Verify both resources were created by running:
aws s3 ls | grep pulseguard-tfstate
aws dynamodb describe-table --table-name pulseguard-tflock --query 'Table.TableStatus'You should see the bucket name in the S3 list output and "ACTIVE" for the DynamoDB table status.
Bucket name already taken?
S3 bucket names are globally unique. If pulseguard-tfstate is taken, append your initials or a short random string (e.g. pulseguard-tfstate-abc123). Update the bucket field in main.tf to match.
help me troubleshoot my S3 bucket creation for Terraform state
- Create the root Terraform configuration by adding the following to terraform/main.tf:
terraform {
required_version = ">= 1.15"
backend "s3" {
bucket = "pulseguard-tfstate"
key = "pulseguard/terraform.tfstate"
region = "us-east-1"
dynamodb_table = "pulseguard-tflock"
encrypt = true
}
}
provider "aws" {
region = var.aws_region
}
module "iam" {
source = "./modules/iam"
}
module "dynamodb" {
source = "./modules/dynamodb"
table_name = var.table_name
}
module "sqs" {
source = "./modules/sqs"
queue_name = "pulseguard-alerts"
dlq_name = "pulseguard-alerts-dlq"
}
module "sns" {
source = "./modules/sns"
topic_name = "pulseguard-alerts"
}
module "lambda" {
source = "./modules/lambda"
table_name = var.table_name
alert_queue_url = module.sqs.queue_url
alert_topic_arn = module.sns.topic_arn
lambda_role_arn = module.iam.lambda_role_arn
}
module "api_gateway" {
source = "./modules/api-gateway"
ingestion_lambda_arn = module.lambda.ingestion_lambda_arn
}
module "eventbridge" {
source = "./modules/eventbridge"
detector_lambda_arn = module.lambda.detector_lambda_arn
lambda_role_arn = module.iam.lambda_role_arn
}- Create the variables file by adding the following to terraform/variables.tf:
variable "aws_region" {
default = "us-east-1"
}
variable "table_name" {
default = "pulseguard"
}What does the root configuration do?
The terraform block pins the minimum Terraform version to 1.15 and configures the S3 backend with encryption and DynamoDB locking. The provider block tells Terraform to deploy everything into the specified AWS region.
Each module block wires modules together by passing outputs from one module as inputs to another. This is the dependency graph: IAM feeds into Lambda, SQS and SNS feed into Lambda, Lambda feeds into API Gateway and EventBridge.
- Save both terraform/main.tf and terraform/variables.tf.
Deploy the stack and verify with a manual curl ping
Before Terraform can deploy Lambda functions, it needs the compiled Go binaries packaged as zip files. Then you will initialize Terraform, review the plan, and apply.
- Build all three Lambda binaries and package them by running:
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -tags lambda.norpc -o bootstrap cmd/ingestion/main.go && zip dist/ingestion.zip bootstrap && rm bootstrap
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -tags lambda.norpc -o bootstrap cmd/detector/main.go && zip dist/detector.zip bootstrap && rm bootstrap
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -tags lambda.norpc -o bootstrap cmd/alerter/main.go && zip dist/alerter.zip bootstrap && rm bootstrapWhat do these build commands do?
Each line compiles a Go binary for Linux (GOOS=linux GOARCH=amd64), names it bootstrap (required by the provided.al2023 runtime), zips it into the dist/ folder, then cleans up the raw binary. The -tags lambda.norpc flag excludes deprecated RPC components to reduce binary size.
- Confirm the zip files exist by running:
ls dist/You should see ingestion.zip, detector.zip, and alerter.zip.
- Initialize Terraform by running:
cd terraform
terraform initTerraform downloads the AWS provider, configures the S3 backend, and validates your module structure. You should see a success message ending with "Terraform has been successfully initialized."
Init fails with backend error?
Double-check that the S3 bucket name in your main.tf backend block matches the bucket you created earlier. Also verify your AWS CLI credentials have permission to access the bucket and DynamoDB table.
help me fix terraform init failing on the S3 backend configuration
- Preview what Terraform will create by running:
terraform planTerraform shows you every resource it plans to create. You should see 15-20+ resources including the DynamoDB table, Lambda functions, SQS queues, SNS topic, API Gateway, EventBridge schedule, and IAM role. Review the plan to confirm everything looks correct.
- Deploy all resources by running:
terraform apply- Terraform will ask you to confirm. Type yes and press Enter.
Wait for all resources to finish creating. This typically takes 1-2 minutes. You should see "Apply complete!" with a count of resources added.
- Verify the deployment end-to-end by sending a test ping. First, grab your API Gateway URL by running:
aws apigatewayv2 get-apis --query 'Items[?Name==`pulseguard`].ApiEndpoint' --output text- Send a test ping using the URL from the previous command:
curl -s -o /dev/null -w "%{http_code}" [[API_URL="your API Gateway endpoint"]]/ping/test-monitor/test-tokenYou should see 200 as the response code. This confirms that API Gateway, the ingestion Lambda, and DynamoDB are all working together.
Getting a 5xx error?
Check the Lambda logs for the ingestion function. The most common issue is a missing DYNAMODB_TABLE environment variable or an IAM permission that did not propagate yet. IAM propagation can take a few seconds after deployment.
help me debug why my PulseGuard ping endpoint is returning a 5xx error
✔️ Awesome, I've got everything!
Great. Make sure all your Terraform files are saved and your deployment is showing 200 OK on the ping endpoint.
ⓧ I'd like to double check the full code
Here is the complete terraform/main.tf:
terraform {
required_version = ">= 1.15"
backend "s3" {
bucket = "pulseguard-tfstate"
key = "pulseguard/terraform.tfstate"
region = "us-east-1"
dynamodb_table = "pulseguard-tflock"
encrypt = true
}
}
provider "aws" {
region = var.aws_region
}
module "iam" {
source = "./modules/iam"
}
module "dynamodb" {
source = "./modules/dynamodb"
table_name = var.table_name
}
module "sqs" {
source = "./modules/sqs"
queue_name = "pulseguard-alerts"
dlq_name = "pulseguard-alerts-dlq"
}
module "sns" {
source = "./modules/sns"
topic_name = "pulseguard-alerts"
}
module "lambda" {
source = "./modules/lambda"
table_name = var.table_name
alert_queue_url = module.sqs.queue_url
alert_topic_arn = module.sns.topic_arn
lambda_role_arn = module.iam.lambda_role_arn
}
module "api_gateway" {
source = "./modules/api-gateway"
ingestion_lambda_arn = module.lambda.ingestion_lambda_arn
}
module "eventbridge" {
source = "./modules/eventbridge"
detector_lambda_arn = module.lambda.detector_lambda_arn
lambda_role_arn = module.iam.lambda_role_arn
}Here is terraform/variables.tf:
variable "aws_region" {
default = "us-east-1"
}
variable "table_name" {
default = "pulseguard"
}Here is terraform/modules/iam/main.tf:
resource "aws_iam_role" "lambda" {
name = "pulseguard-lambda-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Service = "lambda.amazonaws.com" }
Action = "sts:AssumeRole"
}]
})
}
resource "aws_iam_role_policy" "lambda" {
role = aws_iam_role.lambda.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{ Effect = "Allow", Action = ["dynamodb:*"], Resource = "*" },
{ Effect = "Allow", Action = ["sqs:*"], Resource = "*" },
{ Effect = "Allow", Action = ["sns:Publish"], Resource = "*" },
{ Effect = "Allow", Action = ["cloudwatch:PutMetricData"], Resource = "*" },
{ Effect = "Allow", Action = ["logs:*"], Resource = "*" }
]
})
}Here is terraform/modules/dynamodb/main.tf:
resource "aws_dynamodb_table" "pulseguard" {
name = var.table_name
billing_mode = "PAY_PER_REQUEST"
hash_key = "tenant_id"
range_key = "monitor_id"
attribute {
name = "tenant_id"
type = "S"
}
attribute {
name = "monitor_id"
type = "S"
}
}Here is terraform/modules/sqs/main.tf:
resource "aws_sqs_queue" "dlq" {
name = var.dlq_name
}
resource "aws_sqs_queue" "alerts" {
name = var.queue_name
redrive_policy = jsonencode({
deadLetterTargetArn = aws_sqs_queue.dlq.arn
maxReceiveCount = 3
})
}Here is terraform/modules/sns/main.tf:
resource "aws_sns_topic" "alerts" {
name = var.topic_name
}Here is terraform/modules/lambda/main.tf:
resource "aws_lambda_function" "ingestion" {
function_name = "pulseguard-ingestion"
role = var.lambda_role_arn
handler = "bootstrap"
runtime = "provided.al2023"
filename = "${path.module}/../../dist/ingestion.zip"
environment {
variables = {
DYNAMODB_TABLE = var.table_name
}
}
}
resource "aws_lambda_function" "detector" {
function_name = "pulseguard-detector"
role = var.lambda_role_arn
handler = "bootstrap"
runtime = "provided.al2023"
filename = "${path.module}/../../dist/detector.zip"
environment {
variables = {
DYNAMODB_TABLE = var.table_name
ALERT_QUEUE_URL = var.alert_queue_url
}
}
}
resource "aws_lambda_function" "alerter" {
function_name = "pulseguard-alerter"
role = var.lambda_role_arn
handler = "bootstrap"
runtime = "provided.al2023"
filename = "${path.module}/../../dist/alerter.zip"
environment {
variables = {
DYNAMODB_TABLE = var.table_name
}
}
}
resource "aws_lambda_event_source_mapping" "alerter_sqs" {
event_source_arn = var.alert_queue_arn
function_name = aws_lambda_function.alerter.arn
batch_size = 10
}Here is terraform/modules/api-gateway/main.tf:
resource "aws_apigatewayv2_api" "pulseguard" {
name = "pulseguard"
protocol_type = "HTTP"
}
resource "aws_apigatewayv2_integration" "ingestion" {
api_id = aws_apigatewayv2_api.pulseguard.id
integration_type = "AWS_PROXY"
integration_uri = var.ingestion_lambda_arn
payload_format_version = "2.0"
}
resource "aws_apigatewayv2_route" "ping" {
api_id = aws_apigatewayv2_api.pulseguard.id
route_key = "GET /ping/{monitor_id}/{token}"
target = "integrations/${aws_apigatewayv2_integration.ingestion.id}"
}
resource "aws_apigatewayv2_stage" "default" {
api_id = aws_apigatewayv2_api.pulseguard.id
name = "$default"
auto_deploy = true
}Here is terraform/modules/eventbridge/main.tf:
resource "aws_scheduler_schedule" "detector" {
name = "pulseguard-detector"
flexible_time_window { mode = "OFF" }
schedule_expression = "rate(1 minute)"
target {
arn = var.detector_lambda_arn
role_arn = var.lambda_role_arn
input = "{}"
}
}Your entire PulseGuard infrastructure is now live on AWS. Next up, you will automate deployments with a CI/CD pipeline using GitHub Actions so that every code push triggers a build, plan, and deploy cycle automatically.
Set Up CI/CD with GitHub Actions
Your infrastructure is deployed and tested manually. Every time you update a Lambda handler or adjust Terraform, you run commands by hand.
Manual deployments break down fast. You forget a flag, skip a test, or deploy from a branch that was never reviewed. In this step, you will build a GitHub Actions pipeline that automates everything: run tests and plan on PRs, then build, upload, and deploy on merge to main.
In this step, get ready to:
- Create a GitHub Actions workflow with validate and deploy jobs.
- Add the Go build step that compiles Lambda binaries for Linux.
- Configure GitHub OIDC with AWS for keyless authentication.
Create the GitHub Actions workflow
A CI/CD pipeline catches problems before they reach production. Your workflow will have two jobs: a validate job that runs on every pull request, and a deploy job that only runs when code merges to main.
- Create the workflow directory and file by running these commands:
mkdir -p .github/workflows
touch .github/workflows/ci.yml- Confirm the file exists by running:
ls .github/workflows/You should see ci.yml in the output.
- Open .github/workflows/ci.yml and paste the validate job:
name: PulseGuard CI/CD
on:
pull_request:
branches: [main]
push:
branches: [main]
permissions:
id-token: write
contents: read
pull-requests: write
jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials (OIDC)
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/pulseguard-github-actions
aws-region: us-east-1
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: '1.22'
- name: Go tests
run: go test ./...
- name: Setup Terraform
uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.15.0
- name: Terraform fmt check
run: terraform fmt -check -recursive terraform/
- name: Terraform init
run: terraform -chdir=terraform init
- name: Terraform plan
id: plan
run: terraform -chdir=terraform plan -no-color
continue-on-error: true
- name: Post plan as PR comment
if: github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
script: |
const output = `#### Terraform Plan\n\`\`\`\n${{ steps.plan.outputs.stdout }}\n\`\`\``;
github.rest.issues.createComment({ issue_number: context.issue.number, owner: context.repo.owner, repo: context.repo.repo, body: output });What does this job do?
The validate job runs on every PR and push to main. It checks out your code, authenticates to AWS via OIDC, runs Go tests, checks Terraform formatting, and produces a Terraform plan.
The permissions block at the top grants id-token: write so the workflow can request a short-lived OIDC token from GitHub. This token gets exchanged for temporary AWS credentials without storing any long-lived keys.
The plan step posts the output as a PR comment so reviewers can see exactly what infrastructure changes a PR introduces before approving.
Add the Go build and deploy job
The deploy job only runs on merges to main. It compiles your three Lambda handlers for linux/amd64, uploads the zips to S3, updates each Lambda function, then applies Terraform.
- Add the deploy job below the validate job in the same ci.yml file:
deploy:
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
needs: validate
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials (OIDC)
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/pulseguard-github-actions
aws-region: us-east-1
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: '1.22'
- name: Build Lambda binaries
run: |
mkdir -p dist
for fn in ingestion detector alerter; do
GOOS=linux GOARCH=amd64 go build -tags lambda.norpc -o bootstrap ./cmd/$fn
zip dist/$fn.zip bootstrap
rm bootstrap
done
- name: Upload Lambda zips to S3
run: |
for fn in ingestion detector alerter; do
aws s3 cp dist/$fn.zip s3://pulseguard-lambda-artifacts/$fn.zip
done
- name: Update Lambda function code
run: |
for fn in ingestion detector alerter; do
aws lambda update-function-code --function-name pulseguard-$fn --s3-bucket pulseguard-lambda-artifacts --s3-key $fn.zip
done
- name: Setup Terraform
uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.15.0
- name: Terraform apply
run: |
terraform -chdir=terraform init
terraform -chdir=terraform apply -auto-approveWhat does the build step do?
The GOOS=linux GOARCH=amd64 go build -tags lambda.norpc -o bootstrap command cross-compiles each handler into a Linux binary named bootstrap. This is the required binary name for the provided.al2023 runtime.
The lambda.norpc build tag excludes the deprecated RPC component, reducing binary size. Each binary is zipped, uploaded to the pulseguard-lambda-artifacts S3 bucket, then deployed to its respective Lambda function.
Why SQS-style deploy instead of Terraform alone?
Terraform manages infrastructure definitions, but deploying new Lambda code through Terraform alone requires re-uploading the zip and waiting for a full plan/apply cycle. The aws lambda update-function-code command deploys new code instantly without touching the rest of the stack.
The Terraform apply at the end catches any infrastructure drift introduced in the PR.
- Save .github/workflows/ci.yml.
✔️ Awesome, I've got everything!
Great. Double check you have saved your file before moving on.
ⓧ I'd like to double check the full code
name: PulseGuard CI/CD
on:
pull_request:
branches: [main]
push:
branches: [main]
permissions:
id-token: write
contents: read
pull-requests: write
jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials (OIDC)
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/pulseguard-github-actions
aws-region: us-east-1
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: '1.22'
- name: Go tests
run: go test ./...
- name: Setup Terraform
uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.15.0
- name: Terraform fmt check
run: terraform fmt -check -recursive terraform/
- name: Terraform init
run: terraform -chdir=terraform init
- name: Terraform plan
id: plan
run: terraform -chdir=terraform plan -no-color
continue-on-error: true
- name: Post plan as PR comment
if: github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
script: |
const output = `#### Terraform Plan\n\`\`\`\n${{ steps.plan.outputs.stdout }}\n\`\`\``;
github.rest.issues.createComment({ issue_number: context.issue.number, owner: context.repo.owner, repo: context.repo.repo, body: output });
deploy:
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
needs: validate
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials (OIDC)
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/pulseguard-github-actions
aws-region: us-east-1
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: '1.22'
- name: Build Lambda binaries
run: |
mkdir -p dist
for fn in ingestion detector alerter; do
GOOS=linux GOARCH=amd64 go build -tags lambda.norpc -o bootstrap ./cmd/$fn
zip dist/$fn.zip bootstrap
rm bootstrap
done
- name: Upload Lambda zips to S3
run: |
for fn in ingestion detector alerter; do
aws s3 cp dist/$fn.zip s3://pulseguard-lambda-artifacts/$fn.zip
done
- name: Update Lambda function code
run: |
for fn in ingestion detector alerter; do
aws lambda update-function-code --function-name pulseguard-$fn --s3-bucket pulseguard-lambda-artifacts --s3-key $fn.zip
done
- name: Setup Terraform
uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.15.0
- name: Terraform apply
run: |
terraform -chdir=terraform init
terraform -chdir=terraform apply -auto-approveConfigure GitHub OIDC with AWS
The workflow references a role called pulseguard-github-actions. You need to create this role in AWS and establish OIDC trust so GitHub can assume it without stored credentials.
- Create the S3 bucket for Lambda artifacts by running:
aws s3 mb s3://pulseguard-lambda-artifactsWhat is OIDC federation?
OIDC federation lets GitHub Actions request a short-lived token from GitHub's OIDC provider at https://token.actions.githubusercontent.com. AWS validates that token and issues temporary credentials scoped to a specific IAM role. No long-lived AWS keys ever touch GitHub.
- Create the OIDC identity provider in your AWS account by running:
aws iam create-open-id-connect-provider \
--url https://token.actions.githubusercontent.com \
--client-id-list sts.amazonaws.comWhat does this command do?
This registers GitHub's token issuer (token.actions.githubusercontent.com) as a trusted identity provider in your AWS account. The --client-id-list sts.amazonaws.com sets the audience claim that the aws-actions/configure-aws-credentials action sends when requesting credentials.
AWS automatically retrieves the TLS certificate thumbprint for the provider. You do not need to supply one manually.
- Create a trust policy file for the IAM role by running:
cat > /tmp/trust-policy.json << 'EOF'
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::YOUR_ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
},
"StringLike": {
"token.actions.githubusercontent.com:sub": "repo:YOUR_GITHUB_ORG/pulseguard:*"
}
}
}
]
}
EOF- Replace YOUR_ACCOUNT_ID with your 12-digit AWS account ID.
- Replace YOUR_GITHUB_ORG/pulseguard with your GitHub username or organization and repository name.
Why the sub condition matters
The token.actions.githubusercontent.com:sub condition restricts which repositories can assume this role. Without it, any GitHub Actions workflow anywhere could potentially assume your role. The wildcard (*) after the repo name allows any branch or event type from your repository.
AWS enforces this condition for the GitHub OIDC provider. If you omit it, role creation fails with a MalformedPolicyDocument error.
- Create the IAM role using the trust policy by running:
aws iam create-role \
--role-name pulseguard-github-actions \
--assume-role-policy-document file:///tmp/trust-policy.json- Attach a permissions policy that grants the role access to the resources your pipeline needs by running:
aws iam put-role-policy \
--role-name pulseguard-github-actions \
--policy-name pulseguard-deploy \
--policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"lambda:UpdateFunctionCode",
"lambda:GetFunction",
"s3:PutObject",
"s3:GetObject",
"dynamodb:*",
"sqs:*",
"sns:*",
"iam:PassRole",
"logs:*",
"apigateway:*",
"scheduler:*",
"cloudwatch:*"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:ListBucket",
"dynamodb:GetItem",
"dynamodb:PutItem",
"dynamodb:DeleteItem"
],
"Resource": [
"arn:aws:s3:::pulseguard-tfstate/*",
"arn:aws:s3:::pulseguard-tfstate",
"arn:aws:dynamodb:*:*:table/pulseguard-tflock"
]
}
]
}'What permissions does this role need?
The first statement grants access to update Lambda functions, push artifacts to S3, and manage the PulseGuard infrastructure (DynamoDB, SQS, SNS, API Gateway, EventBridge, CloudWatch).
The second statement grants access to the Terraform state bucket and lock table so the pipeline can run terraform plan and terraform apply.
- Add your AWS account ID as a GitHub repository secret. In your repository on GitHub, navigate to **Settings**.
- Select **Secrets and variables**, then **Actions**.
- Click **New repository secret**.
- Set the name to AWS_ACCOUNT_ID and paste your 12-digit AWS account ID as the value.
- Click **Add secret**.
Finding your AWS account ID
Run aws sts get-caller-identity in your terminal. Your account ID is the 12-digit number in the Account field of the JSON output.
- Verify the OIDC setup by pushing a test branch and opening a PR against main:
git checkout -b test-ci
git add .github/workflows/ci.yml
git commit -m "Add CI/CD workflow"
git push origin test-ci- Open a pull request from test-ci to main on GitHub.
- Check the **Actions** tab. You should see the validate job running.
- Once it completes, look for a Terraform plan posted as a comment on your PR.
Seeing an OIDC authentication error?
If the workflow fails at the Configure AWS credentials (OIDC) step, check these common causes:
- Verify your AWS_ACCOUNT_ID secret matches your actual account ID (no spaces or dashes).
- Confirm the token.actions.githubusercontent.com:sub condition in your trust policy matches your GitHub org and repo name exactly.
- Make sure you created the OIDC provider with --url https://token.actions.githubusercontent.com (no trailing slash).
Help me debug my GitHub Actions OIDC authentication error with AWS
Your pipeline is live. Every PR now gets automatic tests, format checks, and a Terraform plan comment. Every merge to main triggers a full build-and-deploy cycle. Next up, you will build the observability stack to monitor PulseGuard in production.
Build the Observability Stack
Your CI/CD pipeline is deploying infrastructure and Lambda code automatically on every merge. The system is live, processing pings, detecting missed deadlines, and firing alerts.
But how do you know it's actually working? If the Detector Lambda silently stops running, nobody gets alerted. You need observability: dashboards that surface system health at a glance, and alarms that alert you when the monitor itself fails.
In this step, get ready to:
- Create a CloudWatch dashboard with key operational metrics.
- Add a meta-monitoring alarm that fires if the Detector Lambda stops running.
- Connect Grafana Cloud to CloudWatch for unified visualization.
Create the CloudWatch dashboard Terraform module
Your Lambda functions already emit custom metrics like PingsReceived and MissedDeadlines. A CloudWatch dashboard collects these alongside AWS-native metrics into a single operational view.
- Create the module directory by running this command:
mkdir -p terraform/modules/cloudwatch- Create the file terraform/modules/cloudwatch/main.tf and add the dashboard resource:
resource "aws_cloudwatch_dashboard" "pulseguard" {
dashboard_name = "PulseGuard-Overview"
dashboard_body = jsonencode({
widgets = [
{ type = "metric", properties = { title = "Pings Received/min", metrics = [["PulseGuard", "PingsReceived"]], period = 60, stat = "Sum" } },
{ type = "metric", properties = { title = "Missed Deadlines/min", metrics = [["PulseGuard", "MissedDeadlines"]], period = 60, stat = "Sum" } },
{ type = "metric", properties = { title = "Alerts Dispatched/min", metrics = [["PulseGuard", "AlertsDispatched"]], period = 60, stat = "Sum" } },
{ type = "metric", properties = { title = "DLQ Depth", metrics = [["AWS/SQS", "ApproximateNumberOfMessagesVisible", "QueueName", "pulseguard-alerts-dlq"]], period = 60, stat = "Maximum" } },
{ type = "metric", properties = { title = "Detector Lambda Duration P99", metrics = [["AWS/Lambda", "Duration", "FunctionName", "pulseguard-detector"]], period = 60, stat = "p99" } },
{ type = "metric", properties = { title = "DynamoDB Consumed RCU", metrics = [["AWS/DynamoDB", "ConsumedReadCapacityUnits", "TableName", "pulseguard"]], period = 60, stat = "Sum" } }
]
})
}What does this code do?
The aws_cloudwatch_dashboard resource uses jsonencode to build the dashboard body as structured HCL rather than raw JSON strings. Each widget pulls from a different metric source.
- The first three widgets track your custom PulseGuard namespace metrics that the Lambda functions emit.
- The DLQ Depth widget monitors how many failed alert messages are stuck in the dead-letter queue.
- The Duration P99 widget shows the 99th percentile execution time of the Detector Lambda.
- The DynamoDB widget tracks consumed read capacity to spot throttling before it causes missed detections.
- Save terraform/modules/cloudwatch/main.tf.
You won't see the dashboard yet. The next chunk adds the meta-monitoring alarm to the same file, and then you'll deploy both together.
Add the meta-monitoring alarm
A heartbeat monitor that silently stops checking is worse than no monitor at all. The concept of "monitoring the monitor" ensures you get alerted if the Detector Lambda itself stops running.
- In terraform/modules/cloudwatch/main.tf, add the following alarm resource below the dashboard resource:
resource "aws_cloudwatch_metric_alarm" "detector_not_running" {
alarm_name = "PulseGuard-DetectorNotRunning"
comparison_operator = "LessThanThreshold"
evaluation_periods = 2
metric_name = "Invocations"
namespace = "AWS/Lambda"
period = 60
statistic = "Sum"
threshold = 1
alarm_description = "Detector Lambda has not run in 2 minutes"
dimensions = {
FunctionName = "pulseguard-detector"
}
}What does this alarm do?
This alarm watches the Invocations metric for pulseguard-detector. If the sum of invocations drops below 1 for two consecutive 60-second periods, the alarm triggers. That means the EventBridge schedule stopped firing, and no deadline checks are running.
This is recursive reliability: the monitor watches your cron jobs, and this alarm watches the monitor.
- Save terraform/modules/cloudwatch/main.tf.
- Deploy both the dashboard and alarm by running:
terraform -chdir=terraform init
terraform -chdir=terraform apply- Type yes when prompted to confirm the apply.
After the apply completes, open the AWS Console and navigate to CloudWatch > Dashboards. You should see PulseGuard-Overview listed with six metric widgets.
Dashboard not appearing?
- Run terraform plan and check for errors. A missing module reference in your root main.tf will show zero resources to add.
- Confirm the cloudwatch module is referenced in terraform/main.tf with the correct source path.
- Check that your IAM role has cloudwatch:PutDashboard and cloudwatch:PutMetricAlarm permissions.
help me troubleshoot why my CloudWatch dashboard isn't deploying via Terraform
✔️ Awesome, I've got everything!
Great. Make sure you've saved terraform/modules/cloudwatch/main.tf and verified the dashboard in the AWS Console.
ⓧ I'd like to double check the full code
Here is the complete terraform/modules/cloudwatch/main.tf file:
resource "aws_cloudwatch_dashboard" "pulseguard" {
dashboard_name = "PulseGuard-Overview"
dashboard_body = jsonencode({
widgets = [
{ type = "metric", properties = { title = "Pings Received/min", metrics = [["PulseGuard", "PingsReceived"]], period = 60, stat = "Sum" } },
{ type = "metric", properties = { title = "Missed Deadlines/min", metrics = [["PulseGuard", "MissedDeadlines"]], period = 60, stat = "Sum" } },
{ type = "metric", properties = { title = "Alerts Dispatched/min", metrics = [["PulseGuard", "AlertsDispatched"]], period = 60, stat = "Sum" } },
{ type = "metric", properties = { title = "DLQ Depth", metrics = [["AWS/SQS", "ApproximateNumberOfMessagesVisible", "QueueName", "pulseguard-alerts-dlq"]], period = 60, stat = "Maximum" } },
{ type = "metric", properties = { title = "Detector Lambda Duration P99", metrics = [["AWS/Lambda", "Duration", "FunctionName", "pulseguard-detector"]], period = 60, stat = "p99" } },
{ type = "metric", properties = { title = "DynamoDB Consumed RCU", metrics = [["AWS/DynamoDB", "ConsumedReadCapacityUnits", "TableName", "pulseguard"]], period = 60, stat = "Sum" } }
]
})
}
resource "aws_cloudwatch_metric_alarm" "detector_not_running" {
alarm_name = "PulseGuard-DetectorNotRunning"
comparison_operator = "LessThanThreshold"
evaluation_periods = 2
metric_name = "Invocations"
namespace = "AWS/Lambda"
period = 60
statistic = "Sum"
threshold = 1
alarm_description = "Detector Lambda has not run in 2 minutes"
dimensions = {
FunctionName = "pulseguard-detector"
}
}Connect Grafana Cloud to CloudWatch
CloudWatch dashboards are useful inside AWS, but a dedicated visualization tool gives you richer panel options, better alerting, and a single pane of glass across services. Grafana Cloud has a free tier that includes a built-in CloudWatch data source plugin.
- Sign up for a free Grafana Cloud account at grafana.com if you don't already have one.
- Once logged in, open your Grafana instance (it will have a URL like your-org.grafana.net).
- Click Connections in the left-side menu.
- Click Add new connection.
- Type CloudWatch in the search bar and select the CloudWatch data source.
- Click Add new data source in the upper right.
Grafana takes you to the Settings tab for the new data source.
- Under Authentication, select Access & secret key (the default for Grafana Cloud).
- Enter the Access Key ID and Secret Access Key for an IAM user with cloudwatch:ListMetrics, cloudwatch:GetMetricData, and cloudwatch:GetMetricStatistics permissions.
- Set the Default Region to us-east-1 (or whichever region your PulseGuard stack is deployed to).
- In the Namespaces of Custom Metrics field, enter PulseGuard. Grafana cannot auto-discover custom metric namespaces.
- Click Save & test. You should see a green success message confirming the connection works.
Why add the custom namespace manually?
Grafana's CloudWatch plugin uses the ListMetrics API to populate the query editor. Custom namespaces (like PulseGuard) are not automatically loaded. You must list them explicitly in the data source settings so they appear in the metric dropdown.
- Create a new dashboard in Grafana by clicking the + icon in the sidebar, then New dashboard.
- Add panels for each of the key metrics: PingsReceived, MissedDeadlines, AlertsDispatched, DLQ depth, and Lambda duration P99.
- For each panel, select the CloudWatch data source, choose the appropriate namespace (PulseGuard for custom metrics, AWS/Lambda or AWS/SQS for AWS-native metrics), and configure the metric and statistic.
- Save the dashboard with the name PulseGuard Overview.
- Export the dashboard JSON by clicking the share icon on the dashboard, selecting Export, and saving the file as docs/grafana-dashboard.json in your project repository.
Version-control your Grafana dashboard
Exporting the dashboard JSON to your repository means anyone cloning the project can import it into their own Grafana instance. It also lets you track layout changes the same way you track code changes.
CloudWatch metrics not appearing in Grafana?
- Confirm the IAM user has cloudwatch:ListMetrics and cloudwatch:GetMetricData permissions attached.
- Verify the Default Region in the data source settings matches the region where your PulseGuard stack is deployed.
- Check that PulseGuard is listed in the Namespaces of Custom Metrics field. Grafana cannot auto-discover custom namespaces.
- Ensure your Lambda functions have actually emitted at least one data point. If no pings have been sent, the custom metrics will not exist yet.
help me troubleshoot why my CloudWatch custom metrics are not showing in Grafana Cloud
Your observability stack is live. You have a CloudWatch dashboard for quick AWS Console checks, a meta-monitoring alarm watching the watcher, and a Grafana dashboard for rich visualization. Next up, you'll put all of this to the test with chaos engineering experiments.
Run Chaos Engineering Tests
Your observability stack is live. You can see every ping, missed deadline, and alert dispatch flowing through CloudWatch and Grafana.
But dashboards only tell you what happened. They don't tell you what will happen when something breaks. Chaos engineering is the discipline of intentionally injecting failures into a system to observe how it recovers. In this step, you'll break PulseGuard on purpose and prove it handles real-world failure modes gracefully.
In this step, get ready to:
- Simulate DynamoDB throttling and observe how the Detector Lambda handles capacity exhaustion.
- Test SQS retry exhaustion and verify messages land safely in the dead-letter queue.
- Document all chaos test results and write a Failure Modes section in the README.
Write the DynamoDB throttling chaos test
Your Detector Lambda scans DynamoDB every minute looking for missed deadlines. In production, DynamoDB can throttle reads and writes when capacity is exhausted. You need to prove the detector handles this gracefully without losing data.
The test strategy is simple: temporarily reduce the table's provisioned capacity to the absolute minimum, let the detector run while throttled, then restore capacity and confirm no monitors were missed permanently.
- Create the chaos/ directory in your project root by running this command:
mkdir chaos- Confirm the directory exists by running:
lsYou should see chaos listed alongside your existing directories like cmd, terraform, and docs.
- Create the DynamoDB throttling test script by adding this code to chaos/test_dynamodb_throttle.py:
#!/usr/bin/env python3
"""Chaos test: DynamoDB throttling simulation."""
import boto3
import time
import json
ddb = boto3.client('dynamodb', region_name='us-east-1')
def throttle_table(table_name, rcu=1, wcu=1):
print(f"Reducing {table_name} capacity to {rcu} RCU / {wcu} WCU")
ddb.update_table(
TableName=table_name,
BillingMode='PROVISIONED',
ProvisionedThroughput={'ReadCapacityUnits': rcu, 'WriteCapacityUnits': wcu}
)
def restore_table(table_name):
print(f"Restoring {table_name} to PAY_PER_REQUEST")
ddb.update_table(TableName=table_name, BillingMode='PAY_PER_REQUEST')
if __name__ == '__main__':
table = 'pulseguard'
throttle_table(table)
print("Sleeping 90s to let detector run under throttling...")
time.sleep(90)
restore_table(table)
print("Done. Check CloudWatch logs for ConditionalCheckFailedException handling.")What does this script do?
The script has three phases that run automatically:
- throttle_table() switches the pulseguard table from on-demand to provisioned mode with just 1 RCU and 1 WCU. This starves the Detector Lambda of read/write capacity.
- The 90-second sleep gives the EventBridge Scheduler time to trigger at least one detector run while the table is throttled.
- restore_table() switches back to PAY_PER_REQUEST mode so the system recovers to normal operation.
- Save chaos/test_dynamodb_throttle.py.
- Run the chaos test by executing:
python3 chaos/test_dynamodb_throttle.pyThe script will print status messages as it throttles the table, sleeps for 90 seconds, then restores capacity. After it finishes, open your CloudWatch Logs for the pulseguard-detector function and look for error entries during the throttled window.
Script failing with permission errors?
Your AWS credentials must have dynamodb:UpdateTable permission. If you're using the same IAM user from Step 1, add this permission to the user's policy.
Also confirm boto3 is installed in your Python environment. Install it with pip3 install boto3 if needed.
Why is my chaos test failing with AccessDeniedException?
Write the SQS dead-letter queue retry test
The second chaos test targets the alert dispatch pipeline. Your SQS queue is configured with a dead-letter queue (DLQ) that catches messages after 3 failed delivery attempts. You need to prove this safety net actually works.
The strategy is to make the Alert Lambda impossible to invoke by setting its reserved concurrency to 0. This forces every SQS delivery attempt to fail, pushing messages into the DLQ after the retry limit is exhausted.
- Create the SQS DLQ test script by adding this code to chaos/test_sqs_dlq.py:
#!/usr/bin/env python3
"""Chaos test: SQS DLQ retry verification."""
import boto3
import json
import time
lambda_client = boto3.client('lambda', region_name='us-east-1')
sqs = boto3.client('sqs', region_name='us-east-1')
ALERTER_FN = 'pulseguard-alerter'
DLQ_URL = 'https://sqs.us-east-1.amazonaws.com/[[AWS_ACCOUNT_ID="your AWS account ID"]]/pulseguard-alerts-dlq'
def break_alerter():
print("Setting alerter reserved concurrency to 0 (simulating failure)")
lambda_client.put_function_concurrency(FunctionName=ALERTER_FN, ReservedConcurrentExecutions=0)
def fix_alerter():
print("Restoring alerter concurrency")
lambda_client.delete_function_concurrency(FunctionName=ALERTER_FN)
def replay_dlq():
print("Replaying messages from DLQ...")
while True:
resp = sqs.receive_message(QueueUrl=DLQ_URL, MaxNumberOfMessages=10)
messages = resp.get('Messages', [])
if not messages:
break
for msg in messages:
print(f"Replaying: {msg['Body']}")
sqs.delete_message(QueueUrl=DLQ_URL, ReceiptHandle=msg['ReceiptHandle'])
if __name__ == '__main__':
break_alerter()
print("Waiting 5 minutes for messages to exhaust retries and land in DLQ...")
time.sleep(300)
fix_alerter()
replay_dlq()
print("Chaos test complete.")What does this script do?
break_alerter() sets the Alert Lambda's reserved concurrency to 0. This prevents AWS from invoking the function at all, simulating a complete failure of the alerting layer.
The 5-minute wait gives SQS time to attempt delivery 3 times (the maxReceiveCount configured on your queue). After 3 failures, messages automatically move to the DLQ.
fix_alerter() removes the concurrency restriction, then replay_dlq() reads messages from the DLQ and logs them. In a real recovery scenario, you would re-publish these messages to the main queue for processing.
- Save chaos/test_sqs_dlq.py.
- Before running this test, make sure at least one monitor has missed its deadline so that messages are flowing into the alert queue. You can stop one of your test cron jobs or manually trigger a missed deadline.
- Run the SQS DLQ test by executing:
python3 chaos/test_sqs_dlq.pyThis script takes about 5 minutes to complete. Watch the output for confirmation that messages landed in the DLQ and were successfully replayed.
Messages not appearing in the DLQ?
Confirm that your SQS queue's maxReceiveCount is set to 3 in your Terraform configuration. If it's higher, messages will retry more times before moving to the DLQ.
Also ensure that messages are actually being sent to the alert queue. Check that your Detector Lambda has flagged at least one monitor as alerting in DynamoDB before running the test.
Why aren't messages moving to my dead-letter queue?
✔️ Awesome, I've got everything!
Great. Make sure both test scripts are saved and that you've captured screenshots of the CloudWatch logs showing the error handling behavior.
ⓧ I'd like to double check the full code
Here is chaos/test_dynamodb_throttle.py:
#!/usr/bin/env python3
"""Chaos test: DynamoDB throttling simulation."""
import boto3
import time
import json
ddb = boto3.client('dynamodb', region_name='us-east-1')
def throttle_table(table_name, rcu=1, wcu=1):
print(f"Reducing {table_name} capacity to {rcu} RCU / {wcu} WCU")
ddb.update_table(
TableName=table_name,
BillingMode='PROVISIONED',
ProvisionedThroughput={'ReadCapacityUnits': rcu, 'WriteCapacityUnits': wcu}
)
def restore_table(table_name):
print(f"Restoring {table_name} to PAY_PER_REQUEST")
ddb.update_table(TableName=table_name, BillingMode='PAY_PER_REQUEST')
if __name__ == '__main__':
table = 'pulseguard'
throttle_table(table)
print("Sleeping 90s to let detector run under throttling...")
time.sleep(90)
restore_table(table)
print("Done. Check CloudWatch logs for ConditionalCheckFailedException handling.")Here is chaos/test_sqs_dlq.py:
#!/usr/bin/env python3
"""Chaos test: SQS DLQ retry verification."""
import boto3
import json
import time
lambda_client = boto3.client('lambda', region_name='us-east-1')
sqs = boto3.client('sqs', region_name='us-east-1')
ALERTER_FN = 'pulseguard-alerter'
DLQ_URL = 'https://sqs.us-east-1.amazonaws.com/[[AWS_ACCOUNT_ID="your AWS account ID"]]/pulseguard-alerts-dlq'
def break_alerter():
print("Setting alerter reserved concurrency to 0 (simulating failure)")
lambda_client.put_function_concurrency(FunctionName=ALERTER_FN, ReservedConcurrentExecutions=0)
def fix_alerter():
print("Restoring alerter concurrency")
lambda_client.delete_function_concurrency(FunctionName=ALERTER_FN)
def replay_dlq():
print("Replaying messages from DLQ...")
while True:
resp = sqs.receive_message(QueueUrl=DLQ_URL, MaxNumberOfMessages=10)
messages = resp.get('Messages', [])
if not messages:
break
for msg in messages:
print(f"Replaying: {msg['Body']}")
sqs.delete_message(QueueUrl=DLQ_URL, ReceiptHandle=msg['ReceiptHandle'])
if __name__ == '__main__':
break_alerter()
print("Waiting 5 minutes for messages to exhaust retries and land in DLQ...")
time.sleep(300)
fix_alerter()
replay_dlq()
print("Chaos test complete.")Document chaos test results and update the README
Chaos test results are only valuable if they're documented. Future you (and anyone reviewing your project) needs to understand what was tested, what was observed, and what the system's limits are.
- Create the chaos results documentation by adding this content to docs/chaos-results.md:
# Chaos Engineering Test Results
## Test 1: DynamoDB Throttling
- **Scenario**: Reduced table to 1 RCU/WCU for 90 seconds
- **Observed**: Detector Lambda received ProvisionedThroughputExceededException, logged errors, retried on next EventBridge tick
- **Result**: No data loss. Monitors recovered on next successful scan.
## Test 2: SQS DLQ Retry
- **Scenario**: Set alerter Lambda concurrency to 0 for 5 minutes
- **Observed**: SQS retried 3 times (maxReceiveCount=3), messages moved to DLQ
- **Result**: DLQ replay after fix successfully dispatched all pending alerts.What should I put in this document?
The template above captures the standard chaos test report structure: scenario, observation, and result. Update the observations to match what you actually saw in your CloudWatch logs.
Add screenshots from your CloudWatch dashboard and logs alongside each test. Store them in your docs/ folder and reference them with relative links in the markdown.
- Save docs/chaos-results.md.
- Now update your project's README.md to include a Failure Modes section. Add the following content:
# PulseGuard
A serverless Dead Man's Switch platform.
## Failure Modes
### DynamoDB Throttling
The Detector Lambda handles ConditionalCheckFailedException gracefully and retries on the next EventBridge tick (1 minute). No data loss observed.
### Alert Lambda Failure
SQS retries failed messages 3 times before routing to the DLQ. Messages can be replayed from the DLQ using chaos/test_sqs_dlq.py after the underlying issue is resolved.Why document failure modes in the README?
Anyone operating or evaluating PulseGuard needs to understand its behavior under failure. The Failure Modes section communicates that you've tested the system's resilience, not just its happy path.
This is exactly what interviewers look for. It signals you think about production reliability, not just feature delivery.
- Save README.md.
- Commit all your chaos engineering work by running:
git add chaos/ docs/chaos-results.md README.md
git commit -m "Add chaos engineering tests and failure mode docs"Git commit failing?
Make sure you're in the project root directory when running git add. The paths chaos/ and docs/chaos-results.md are relative to your project root.
Why is my git commit failing?
Your chaos engineering suite is complete. You've proven that PulseGuard survives DynamoDB throttling without data loss and that the SQS dead-letter queue catches failed alerts for replay. Next up, you'll set the whole system live with real monitors and capture a demo of PulseGuard detecting a genuine failure.
Set Up the Live Demo
Your infrastructure is deployed, tested under chaos conditions, and proven reliable. Now it's time to prove PulseGuard works with real workloads over real time.
In this step, you'll register three real monitors in DynamoDB, point actual cron jobs and CI pipelines at your API, let the system run for over a week, and then intentionally break something to capture PulseGuard detecting a real failure.
In this step, get ready to:
- Create three real monitors with ping scripts and cron entries.
- Run the system for a week and intentionally trigger a missed-deadline alert.
- Record a demo showing the full detection-to-recovery cycle.
Create three real monitors
PulseGuard needs real external processes pinging it to prove the system works end-to-end. You'll register three monitors that cover common reliability patterns: a periodic cron job, an hourly backup script, and an event-driven CI pipeline.
Each monitor needs a unique token for authentication. Generate three tokens by running this command three times and noting each result:
- Generate three unique tokens by running this command three separate times:
uuidgenWhat are these tokens for?
Each monitor gets a unique ping URL with an embedded token, like /ping/monitor-cron-5m/abc123.... The ingestion Lambda validates this token before updating the heartbeat timestamp.
Save all three tokens somewhere safe. You'll use them in the DynamoDB items and in your ping scripts.
- Register the first monitor (5-minute cron job) by running:
aws dynamodb put-item \
--table-name pulseguard \
--item '{
"tenant_id": {"S": "default"},
"monitor_id": {"S": "monitor-cron-5m"},
"interval_seconds": {"N": "300"},
"grace_seconds": {"N": "60"},
"status": {"S": "healthy"},
"token": {"S": "your-first-generated-token"},
"alert_target": {"S": "arn:aws:sns:us-east-1:YOUR_ACCOUNT:pulseguard-alerts"},
"created_at": {"S": "'$(date -u +%Y-%m-%dT%H:%M:%SZ)'"}
}'- Register the second monitor (hourly backup) by running:
aws dynamodb put-item \
--table-name pulseguard \
--item '{
"tenant_id": {"S": "default"},
"monitor_id": {"S": "monitor-backup-1h"},
"interval_seconds": {"N": "3600"},
"grace_seconds": {"N": "300"},
"status": {"S": "healthy"},
"token": {"S": "your-second-generated-token"},
"alert_target": {"S": "arn:aws:sns:us-east-1:YOUR_ACCOUNT:pulseguard-alerts"},
"created_at": {"S": "'$(date -u +%Y-%m-%dT%H:%M:%SZ)'"}
}'- Register the third monitor (deploy pipeline) by running:
aws dynamodb put-item \
--table-name pulseguard \
--item '{
"tenant_id": {"S": "default"},
"monitor_id": {"S": "monitor-deploy-pipeline"},
"interval_seconds": {"N": "86400"},
"grace_seconds": {"N": "3600"},
"status": {"S": "healthy"},
"token": {"S": "your-third-generated-token"},
"alert_target": {"S": "arn:aws:sns:us-east-1:YOUR_ACCOUNT:pulseguard-alerts"},
"created_at": {"S": "'$(date -u +%Y-%m-%dT%H:%M:%SZ)'"}
}'Why 86400 seconds for the deploy pipeline?
The deploy pipeline monitor expects at least one CI run per day (86400 seconds). If no deployment happens within 24 hours plus the 1-hour grace period, PulseGuard treats it as a missed heartbeat and fires an alert.
- Verify all three monitors exist by running:
aws dynamodb scan --table-name pulseguard --select COUNTYou should see "Count": 3 in the response confirming all three monitors are registered.
Seeing Count: 0 or an error?
Make sure your AWS CLI region matches where you deployed the pulseguard table. Check your region with aws configure get region.
If a put-item command failed silently, re-run it. DynamoDB put-item is idempotent, so re-running is safe.
Help me debug my DynamoDB put-item commands
Now create the ping scripts that your cron job and backup process will execute.
- Create the scripts/ directory by running:
mkdir -p scripts- Create scripts/cron-ping.sh with the following content:
#!/bin/bash
# Run via crontab: */5 * * * * /path/to/cron-ping.sh
curl -sf "https://[[API_GW_URL="your-api-gateway-invoke-url"]]/ping/monitor-cron-5m/your-cron-monitor-token" || echo "ping failed"What does this script do?
The -s flag makes curl silent (no progress bar). The -f flag makes curl return a non-zero exit code on HTTP errors.
If the ping fails for any reason, the script echoes "ping failed" to stderr so your cron daemon logs the failure.
- Create scripts/backup-ping.sh with the following content:
#!/bin/bash
# Run after backup completes
curl -sf "https://[[API_GW_URL="your-api-gateway-invoke-url"]]/ping/monitor-backup-1h/your-backup-monitor-token" || echo "ping failed"- Make both scripts executable by running:
chmod +x scripts/cron-ping.sh scripts/backup-ping.sh- Replace your-cron-monitor-token and your-backup-monitor-token in the scripts with the actual tokens you generated earlier.
- Replace the API Gateway URL placeholder with your actual invoke URL from the Terraform output.
Next, schedule the cron ping to run every 5 minutes.
- Open your crontab for editing by running:
crontab -e- Add this line at the bottom of the file, replacing the path with the absolute path to your script:
*/5 * * * * /absolute/path/to/scripts/cron-ping.sh- Save and exit the editor.
- Verify the cron entry was saved by running:
crontab -lYou should see your */5 * * * * entry listed in the output.
Set up the backup ping too
Add a separate crontab entry for the hourly backup ping: 0 * * * * /absolute/path/to/scripts/backup-ping.sh. This runs at the top of every hour, simulating a backup script that reports success after completing.
Finally, add a ping step to your CI workflow so every deployment heartbeats to PulseGuard.
- In .github/workflows/ci.yml, add this step at the end of your deployment job:
- name: Ping PulseGuard
if: github.ref == 'refs/heads/main'
run: curl -sf "https://${{ secrets.PULSEGUARD_API_URL }}/ping/monitor-deploy-pipeline/${{ secrets.PULSEGUARD_DEPLOY_TOKEN }}" || trueWhat does this workflow step do?
The if: github.ref == 'refs/heads/main' condition ensures the ping only fires on merges to main, not on every PR.
The token and URL live in GitHub Actions secrets, keeping them out of your repository code. The || true prevents a failed ping from blocking your deployment.
- In your GitHub repository, navigate to Settings then Secrets and variables then Actions.
- Click New repository secret.
- Add PULSEGUARD_API_URL with your API Gateway invoke URL (without https://).
- Add PULSEGUARD_DEPLOY_TOKEN with the third token you generated earlier.
Now update your README to document the live monitors.
- In README.md, add this section between the project title and the ## Failure Modes heading:
## Live Monitors
| Monitor ID | Interval | Description |
|---|---|---|
| monitor-cron-5m | 5 min | Cron job on local machine |
| monitor-backup-1h | 1 hour | Hourly backup script |
| monitor-deploy-pipeline | on push | GitHub Actions CI pipeline |- Save README.md.
- Commit and push all changes by running:
git add scripts/ README.md .github/workflows/ci.yml
git commit -m "Add live monitors and ping scripts"
git push origin mainThe push to main triggers your CI pipeline, which now includes the PulseGuard ping step. Your first deploy-pipeline heartbeat should arrive within minutes.
- Wait 5 minutes, then verify pings are arriving by checking your CloudWatch dashboard for the PingsReceived metric.
Not seeing pings arrive?
Run the cron-ping script manually to check for errors: bash scripts/cron-ping.sh. If you see "ping failed", verify your API Gateway URL and token are correct.
Check CloudWatch Logs for the ingestion Lambda to see if requests are arriving but failing authentication.
Help me debug why my pings are not reaching PulseGuard
Run the system and trigger a missed-deadline alert
The real value of PulseGuard is proving it detects absence over time. Let the system collect real heartbeat data for at least one week before triggering a failure.
- Let all three monitors run for at least 7 days, collecting real heartbeat data.
- Check your CloudWatch dashboard daily to confirm all monitors show healthy status and pings are arriving on schedule.
Why wait a full week?
A week of continuous data proves your system handles real-world conditions: network hiccups, timezone transitions, daylight saving changes, and bursty traffic patterns. This data makes your demo credible to interviewers.
After a week of clean data, intentionally kill the 5-minute cron job to trigger a real missed-deadline alert.
- Remove the cron entry by running:
crontab -e- Delete or comment out the */5 * * * * line for cron-ping.sh, then save and exit.
- Wait 6 minutes (the 5-minute interval plus 1-minute grace period).
- Check your email or SNS subscription for the missed-deadline alert from PulseGuard.
You should receive an alert notification indicating that monitor-cron-5m has missed its heartbeat deadline.
No alert after 6 minutes?
Check the EventBridge Scheduler to confirm the detector Lambda is still running every minute. Look at the detector Lambda's CloudWatch Logs to see if it detected the missed deadline.
Verify the monitor's status changed from healthy to grace and then alerting by scanning DynamoDB:
aws dynamodb get-item \
--table-name pulseguard \
--key '{"tenant_id": {"S": "default"}, "monitor_id": {"S": "monitor-cron-5m"}}'Help me figure out why my missed-deadline alert did not fire
- After confirming the alert fired, restore the cron entry to resolve the monitor:
crontab -e- Re-add the */5 * * * * line for cron-ping.sh, then save and exit.
- Wait 5 minutes for the next successful ping to arrive.
- Confirm the monitor status transitions back to healthy in your CloudWatch dashboard.
Record the demo video
A recorded demo is the single most powerful artifact for interviews. It proves the system works end-to-end without requiring the interviewer to deploy anything.
- Create the demo output directory by running:
mkdir -p docs/demo- Open a screen recording tool (QuickTime on macOS, OBS on Windows/Linux, or a GIF recorder like peek or gifcap).
- Open your Grafana or CloudWatch dashboard showing monitor statuses and the PingsReceived metric.
Record the following sequence in a single take:
- Show the dashboard in its green/healthy state (all monitors receiving pings).
- Switch to your terminal and kill the cron job (delete the crontab entry).
- Wait for the alert to fire (approximately 6 minutes).
- Show the alert notification arriving (email, SMS, or dashboard alarm state).
- Restore the cron entry in your terminal.
- Show the dashboard recovering as pings resume and the monitor returns to healthy.
Speed up the recording
If recording a GIF, you can speed up the 6-minute wait to keep the final file under 30 seconds. Most GIF tools support playback speed adjustment in post-processing.
- Save the recording as docs/demo/demo.gif (or docs/demo/demo.mp4 if video).
- Commit the demo artifact by running:
git add docs/demo/
git commit -m "Add live demo recording"
git push origin mainYou should see docs/demo/demo.gif in your repository, ready to embed in your README or share with interviewers.
Demo file too large for Git?
GitHub has a 100MB file size limit. If your GIF or video exceeds this, compress it or use a shorter recording window. GIF files over 10MB often indicate you should crop the idle waiting time.
Help me reduce the size of my demo recording for Git
Your live demo proves PulseGuard works in the real world, not just in test conditions. This recording and the weeks of real heartbeat data are what separate your project from a tutorial clone.
Secret mission
Webhook Notification Channel with Circuit Breaker
Your SNS alerts work, but what about Slack webhooks, PagerDuty, or custom HTTP endpoints? Build an alternative notification path with exponential backoff retry and a circuit breaker that stops hammering a dead endpoint. This is the resilience pattern that separates production-grade systems from demos.
Clean Up Your Resources
Clean Up Your Resources
Decide whether to keep your resources running, pause them to come back later, or delete them entirely. Most of PulseGuard's AWS resources fall within Free Tier limits, but CloudWatch custom metrics cost approximately $0.30 per metric per month.
Cost warning
If you leave PulseGuard running, expect approximately $1-3/month beyond Free Tier, primarily from CloudWatch custom metrics. The EventBridge Scheduler, Lambda invocations, DynamoDB capacity, SQS requests, and API Gateway calls all remain within Free Tier for PulseGuard's typical usage.
If cost is a concern and you are not actively using PulseGuard for interviews, choose Pause or Delete below.
Resources you used:
- AWS Lambda functions (ingestion, detector, alerter).
- DynamoDB table (pulseguard).
- API Gateway REST API.
- SQS queues (alerts queue, alerts DLQ, pulseguard-webhook-dlq).
- SNS topics for alert notifications.
- EventBridge Scheduler (1-minute detector sweep).
- CloudWatch dashboards, alarms, and custom metrics.
- IAM roles and policies.
- S3 bucket for Terraform state.
- DynamoDB table for Terraform state locking.
- Local project files (pulseguard directory on your Desktop).
- Grafana Cloud account (free tier).
- GitHub repository with Actions workflow.
Keep everything running
No action needed. Choose this if you are still actively building or want to keep PulseGuard live for interview demos.
- Your system continues processing heartbeat pings and firing alerts.
- Expected cost is $1-3/month beyond Free Tier.
- The Grafana Cloud free tier dashboard stays available at no charge.
- Your GitHub Actions pipeline runs on every push to main.
Pause - I'll come back to this later
Stop the detection loop to eliminate ongoing Lambda invocations and custom metric writes. Your pings still land in DynamoDB, but no alerting occurs.
- Open terraform/modules/eventbridge/main.tf and comment out the scheduler resource.
- Navigate to your terraform/ directory and run terraform apply to remove the schedule.
- This stops the Detector Lambda from running every minute, halting custom metric emissions and alert dispatches.
- To resume, uncomment the EventBridge resource and run terraform apply again.
Delete - I don't want to use this again
Remove all project resources completely. Save your Terraform code locally or in GitHub if you want to redeploy later.
- Navigate to your terraform/ directory and destroy all managed AWS resources by running:
- When prompted, type yes to confirm the destroy operation.
terraform destroyThe Terraform state backend (S3 bucket and DynamoDB lock table) is not managed by the same Terraform configuration, so you need to remove them separately.
- Delete the S3 state bucket and all its contents by running:
aws s3 rb s3://your-pulseguard-state-bucket --force- Delete the DynamoDB state lock table by running:
aws dynamodb delete-table --table-name your-pulseguard-lock-table- Delete the local project files by running:
cd ~/Desktop
rm -rf pulseguard- Delete your GitHub repository: go to your repository page, click Settings, scroll to the bottom, and click Delete this repository.
- Delete your Grafana Cloud stack: go to grafana.com, sign in, navigate to My Account, find your stack, and click Delete Stack.
- Optionally remove the dedicated IAM user you created for Terraform in the AWS Console under IAM > Users > select the user > Delete.
That's a Wrap!
That's a Wrap!
You just built a production-grade reliability platform from scratch. Not a tutorial deployment. Not a clone. A system that solves a real problem that real engineers complain about every day.
Here's what you accomplished:
- Designed and deployed an event-driven serverless architecture on AWS that processes heartbeat pings with sub-second latency, using Go Lambda functions on the provided.al2023 runtime, a DynamoDB single-table design with conditional writes and state machine transitions, and the "absence detection" pattern via EventBridge Scheduler sweeps.
- Deployed 20+ AWS resources via modularized Terraform with S3/DynamoDB state management, a GitHub Actions CI/CD pipeline using OIDC authentication, and a full observability stack with CloudWatch custom metrics, dashboards, and meta-monitoring that monitors the monitor.
- Conducted chaos engineering experiments simulating DynamoDB throttling and SQS failures, documented failure modes, and operated a live system for weeks with real cron jobs proving reliability end to end.
- Built a webhook notification channel with exponential backoff retry, jitter, and a circuit breaker pattern that trips after 5 consecutive failures and probes for recovery every 5 minutes.
Ready to quiz yourself?