AI DevOps Copilot: Incident Analysis
Build an AI copilot that remembers incidents, scores severity, and emails weekly ops reports.
Introduction
30 Second Summary
The best on-call engineers don't just fix problems. They remember every past incident, spot patterns, and prevent future outages before they happen.
In this project, you will upgrade your AI DevOps Copilot with incident memory, automatic severity scoring, multi-mode AI analysis, and a weekly digest system that emails you operational reports every Monday.
What You'll Build
An intelligent incident analysis system powered by Amazon Bedrock, DynamoDB, EventBridge, and SNS that remembers every infrastructure incident, detects patterns, and proactively generates weekly operational reports.
By the end of this project, you'll have:
- A DynamoDB incident memory layer that stores structured records of every remediation event with automatic severity scoring.
- A multi-mode AI analysis engine powered by Amazon Bedrock that provides incident summaries, root cause analysis, preventive recommendations, and repeat offender detection.
- An automated weekly digest system using EventBridge and SNS that emails you AI-generated operational reports every Monday.
- An expanded REST API with /ask, /incidents, and /digest endpoints for multi-mode copilot interaction.
- Secret Mission: Build an enhanced ops console dashboard with a severity-coded incident timeline, mode selector, and on-demand digest generation.
Are there any prerequisites?
This is Part 2 of the AI DevOps Copilot series. You must complete Part 1 (Self-Healing Infrastructure) first, and all Part 1 infrastructure must be deployed and running in us-east-1.
Create the Incident Memory Layer
In Part 1, your copilot could answer questions about what's happening right now. But it had no memory of past incidents.
Without a record of previous events, the copilot can't spot patterns like the same alarm firing every Tuesday at 3 AM. In this step, you'll create a DynamoDB table that stores every incident as a structured record. This gives your copilot long-term memory it can query for trends and repeat offenders.
In this step, get ready to:
- Create the DevOpsIncidentLog DynamoDB table with a partition key and on-demand billing.
- Add a Global Secondary Index on the timestamp attribute for time-range queries.
- Verify the table and GSI are both active.
Create the DevOpsIncidentLog table
Your incident memory layer is a single DynamoDB table. Each row represents one remediation event, keyed by a unique incident ID.
- In the AWS Management Console, type DynamoDB in the top search bar and select DynamoDB.
- In the left navigation pane, click Tables.
- Click Create table.
You'll now configure the table details and capacity settings.
- For Table name, enter DevOpsIncidentLog.
- For Partition key, enter incident_id and leave the type as String.
- Leave Sort key empty. You do not need a sort key for this table.
- Under Table settings, select Customize settings.
Why customize settings?
The default setting uses provisioned capacity with auto scaling. For a learning project with unpredictable, low-volume traffic, on-demand billing is simpler and cheaper. You also need to add a Global Secondary Index, which requires custom settings.
- Under Read/write capacity settings, select On-demand.
Add a Global Secondary Index for time-range queries
A Global Secondary Index (GSI) lets you query the table using an attribute other than the partition key. Your copilot needs to pull incidents from the last 24 hours or 7 days, so a GSI on timestamp enables efficient time-based lookups.
- Scroll down to the Secondary indexes section.
- Click Create global index.
- For Partition key, enter timestamp and leave the type as String.
- Leave Sort key empty.
- For Index name, keep the auto-generated name timestamp-index.
- Leave Attribute projections set to All.
- Click Create index.
What is a Global Secondary Index?
Think of it as a second lookup table that DynamoDB maintains automatically. The base table is keyed by incident_id (great for finding one specific incident). The GSI is keyed by timestamp (great for finding all incidents in a time window).
Every time you write an item to the base table, DynamoDB copies the relevant attributes into the GSI automatically.
- Scroll to the bottom of the page and click Create table.
DynamoDB begins provisioning the table. This typically takes 10-20 seconds.
Verify the table is active and the GSI is ready
- Wait for the Status column in the tables list to show Active.
- Click the table name DevOpsIncidentLog to open its detail page.
- Click the Indexes tab.
- Confirm that timestamp-index shows a status of Active.
Table or GSI stuck in Creating?
If the status stays on Creating for more than a minute, refresh the page. DynamoDB tables with a single GSI typically become active within 30 seconds.
If it remains stuck, check you are in the us-east-1 region (visible in the top-right of the console).
ask me for help troubleshooting my DynamoDB table creation
You should now see both the table and its GSI marked as Active. Your incident memory layer is ready to receive records.
Your copilot now has a place to store every incident it encounters. Next up, you'll update the remediation Lambda to automatically log incidents with severity scoring every time it takes action.
Add Severity Scoring to the Remediation Lambda
Your DynamoDB incident memory layer is ready. Now it is time to teach the remediation Lambda to use it.
Right now, the SelfHealingRemediation Lambda from Part 1 fixes problems and forgets them instantly. In this step, you will add severity scoring logic and incident logging so that every remediation event gets classified (Critical, High, Medium, or Low) and permanently recorded in DynamoDB.
In this step, get ready to:
- Update the Lambda's IAM role to allow DynamoDB writes and queries.
- Replace the Lambda code with severity scoring and incident logging functions.
- Test with a simulated event and verify a record appears in DynamoDB.
Update the Lambda's IAM role
The remediation Lambda needs permission to write incident records to DynamoDB and query for recent incidents when calculating severity.
- Open the IAM console at https://console.aws.amazon.com/iam/.
- In the left navigation pane, choose Roles.
- Search for the execution role attached to your SelfHealingRemediation Lambda function and click its name.
- Choose the Permissions tab.
- Click Add permissions, then choose Create inline policy.
- Select the JSON tab in the Policy editor.
- Replace the contents with the following policy by pasting this JSON:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dynamodb:PutItem",
"dynamodb:Query",
"dynamodb:Scan"
],
"Resource": "arn:aws:dynamodb:us-east-1:*:table/DevOpsIncidentLog"
}
]
}
What does this policy do?
This grants the Lambda three specific DynamoDB actions on the DevOpsIncidentLog table only.
- PutItem allows writing new incident records.
- Query and Scan allow reading recent incidents to determine if the same alarm has fired before (used for severity scoring).
- Click Next.
- For Policy name, enter DynamoDBIncidentLogAccess.
- Click Create policy.
You should see the new inline policy listed under the role's permissions.
Don't see the role?
If you are unsure which role belongs to your Lambda, open the Lambda console, select SelfHealingRemediation, choose the Configuration tab, then Permissions. The role name is shown under Execution role. Click the role name to jump directly to it in IAM.
help me find my Lambda execution role in the AWS console
Replace the Lambda code with severity scoring and incident logging
Now you will replace the entire Lambda function code with an updated version that calculates severity and logs incidents to DynamoDB after every remediation.
- Open the Lambda console and select the SelfHealingRemediation function.
- Choose the Code tab.
- Select all the existing code in the editor and delete it.
- Paste the following imports and client setup at the top of the file:
import json
import boto3
import time
import uuid
from datetime import datetime, timezone, timedelta
ssm_client = boto3.client('ssm', region_name='us-east-1')
sns_client = boto3.client('sns', region_name='us-east-1')
dynamodb = boto3.resource('dynamodb', region_name='us-east-1')
ec2_client = boto3.client('ec2', region_name='us-east-1')
INCIDENT_TABLE_NAME = 'DevOpsIncidentLog'
SNS_TOPIC_ARN_ENV = 'SNS_TOPIC_ARN'
import os
SNS_TOPIC_ARN = os.environ.get(SNS_TOPIC_ARN_ENV, '')
What's new in these imports?
- uuid generates unique IDs for each incident record.
- datetime, timezone, timedelta handle timestamps and time-range comparisons for severity scoring.
- dynamodb is a Boto3 resource for table-level operations (PutItem, Scan).
- ec2_client checks instance state to detect stopped or terminated instances.
- INCIDENT_TABLE_NAME points to the DynamoDB table you created in Step 1.
- Below the imports, paste the two helper functions that handle severity scoring and incident logging:
def calculate_severity(outcome, alarm_name, instance_id):
"""
Calculate incident severity based on rules:
- Critical: Remediation failed OR instance stopped/terminated
- High: Remediation succeeded but same alarm fired 2+ times in last 2 hours
- Medium: Remediation succeeded, first occurrence in last 2 hours
- Low: Alarm resolved on its own (no action needed)
"""
if outcome == 'failure':
return 'Critical'
# Check instance state
try:
response = ec2_client.describe_instance_status(
InstanceIds=[instance_id],
IncludeAllInstances=True
)
if response['InstanceStatuses']:
state = response['InstanceStatuses'][0]['InstanceState']['Name']
if state in ['stopped', 'terminated']:
return 'Critical'
except Exception:
pass
# Check for repeat incidents in the last 2 hours
table = dynamodb.Table(INCIDENT_TABLE_NAME)
two_hours_ago = (datetime.now(timezone.utc) - timedelta(hours=2)).isoformat()
try:
response = table.scan(
FilterExpression='alarm_name = :alarm AND #ts > :time_threshold',
ExpressionAttributeNames={'#ts': 'timestamp'},
ExpressionAttributeValues={
':alarm': alarm_name,
':time_threshold': two_hours_ago
}
)
recent_count = response.get('Count', 0)
if recent_count >= 1:
return 'High'
except Exception:
pass
return 'Medium'
def log_incident(alarm_name, trigger_reason, remediation_action, outcome, instance_id, duration_seconds, raw_data):
"""Log a structured incident record to DynamoDB."""
table = dynamodb.Table(INCIDENT_TABLE_NAME)
severity = calculate_severity(outcome, alarm_name, instance_id)
incident_record = {
'incident_id': str(uuid.uuid4()),
'timestamp': datetime.now(timezone.utc).isoformat(),
'alarm_name': alarm_name,
'trigger_reason': trigger_reason,
'remediation_action': remediation_action,
'outcome': outcome,
'instance_id': instance_id,
'duration_seconds': int(duration_seconds),
'severity': severity,
'raw_data': json.dumps(raw_data)
}
table.put_item(Item=incident_record)
print(f"Incident logged: {incident_record['incident_id']} | Severity: {severity}")
return incident_record
How does severity scoring work?
The calculate_severity function applies a clear decision tree to each incident.
- Critical: The remediation itself failed, or the EC2 instance is stopped/terminated.
- High: Remediation succeeded, but the same alarm has already fired within the last 2 hours. This flags repeat offenders.
- Medium: Remediation succeeded and this is the first occurrence. A normal incident.
The log_incident function calls calculate_severity, builds a structured record with all incident metadata, and writes it to DynamoDB with put_item.
- Finally, paste the updated lambda_handler function below the helpers:
def lambda_handler(event, context):
print(f"Received event: {json.dumps(event)}")
start_time = time.time()
# Extract alarm details from the event
detail = event.get('detail', {})
alarm_name = detail.get('alarmName', 'Unknown')
instance_id = detail.get('configuration', {}).get('metrics', [{}])[0].get('metricStat', {}).get('metric', {}).get('dimensions', {}).get('InstanceId', '')
# If instance_id not in alarm detail, try environment variable
if not instance_id:
instance_id = os.environ.get('INSTANCE_ID', 'unknown')
trigger_reason = detail.get('state', {}).get('reason', 'CPU threshold exceeded')
remediation_action = 'systemctl restart nginx'
outcome = 'success'
try:
# Send SSM command to restart nginx
response = ssm_client.send_command(
InstanceIds=[instance_id],
DocumentName='AWS-RunShellScript',
Parameters={'commands': ['systemctl restart nginx']},
TimeoutSeconds=60
)
command_id = response['Command']['CommandId']
print(f"SSM Command sent: {command_id}")
# Wait briefly for command to complete
time.sleep(5)
# Check command status
result = ssm_client.get_command_invocation(
CommandId=command_id,
InstanceId=instance_id
)
if result['Status'] in ['Success']:
outcome = 'success'
print("Remediation successful: nginx restarted")
else:
outcome = 'failure'
print(f"Remediation status: {result['Status']}")
except Exception as e:
outcome = 'failure'
print(f"Remediation failed: {str(e)}")
duration_seconds = time.time() - start_time
# Log the incident to DynamoDB
incident_record = log_incident(
alarm_name=alarm_name,
trigger_reason=trigger_reason,
remediation_action=remediation_action,
outcome=outcome,
instance_id=instance_id,
duration_seconds=duration_seconds,
raw_data=detail
)
# Send SNS notification
if SNS_TOPIC_ARN:
try:
message = (
f"Remediation Complete\n"
f"Alarm: {alarm_name}\n"
f"Instance: {instance_id}\n"
f"Action: {remediation_action}\n"
f"Outcome: {outcome}\n"
f"Severity: {incident_record['severity']}\n"
f"Duration: {duration_seconds:.1f}s\n"
f"Incident ID: {incident_record['incident_id']}"
)
sns_client.publish(
TopicArn=SNS_TOPIC_ARN,
Subject=f"[{incident_record['severity']}] Remediation: {alarm_name}",
Message=message
)
except Exception as e:
print(f"SNS notification failed: {str(e)}")
return {
'statusCode': 200,
'body': json.dumps({
'message': 'Remediation complete',
'incident_id': incident_record['incident_id'],
'severity': incident_record['severity'],
'outcome': outcome
})
}
What changed in the handler?
The handler now calls log_incident() after the remediation attempt completes. This means every single remediation (success or failure) gets a permanent record in DynamoDB.
The SNS notification also now includes the severity level and incident ID, so your email alerts immediately tell you how critical an event is.
- Click Deploy to save and deploy the updated code.
You should see a green banner confirming the function was updated successfully.
Set the timeout to 60 seconds
The function waits for SSM command completion, which can take several seconds. Go to Configuration > General configuration > Edit and set the timeout to 60 seconds if it is not already configured from Part 1.
✔️ Awesome, I've got everything!
Great. Double check you have clicked Deploy and the green success banner appeared.
ⓧ I'd like to double check the full code
Here is the complete updated Lambda code. Make sure your editor matches this exactly:
import json
import boto3
import time
import uuid
from datetime import datetime, timezone, timedelta
ssm_client = boto3.client('ssm', region_name='us-east-1')
sns_client = boto3.client('sns', region_name='us-east-1')
dynamodb = boto3.resource('dynamodb', region_name='us-east-1')
ec2_client = boto3.client('ec2', region_name='us-east-1')
INCIDENT_TABLE_NAME = 'DevOpsIncidentLog'
SNS_TOPIC_ARN_ENV = 'SNS_TOPIC_ARN'
import os
SNS_TOPIC_ARN = os.environ.get(SNS_TOPIC_ARN_ENV, '')
def calculate_severity(outcome, alarm_name, instance_id):
"""
Calculate incident severity based on rules:
- Critical: Remediation failed OR instance stopped/terminated
- High: Remediation succeeded but same alarm fired 2+ times in last 2 hours
- Medium: Remediation succeeded, first occurrence in last 2 hours
- Low: Alarm resolved on its own (no action needed)
"""
if outcome == 'failure':
return 'Critical'
# Check instance state
try:
response = ec2_client.describe_instance_status(
InstanceIds=[instance_id],
IncludeAllInstances=True
)
if response['InstanceStatuses']:
state = response['InstanceStatuses'][0]['InstanceState']['Name']
if state in ['stopped', 'terminated']:
return 'Critical'
except Exception:
pass
# Check for repeat incidents in the last 2 hours
table = dynamodb.Table(INCIDENT_TABLE_NAME)
two_hours_ago = (datetime.now(timezone.utc) - timedelta(hours=2)).isoformat()
try:
response = table.scan(
FilterExpression='alarm_name = :alarm AND #ts > :time_threshold',
ExpressionAttributeNames={'#ts': 'timestamp'},
ExpressionAttributeValues={
':alarm': alarm_name,
':time_threshold': two_hours_ago
}
)
recent_count = response.get('Count', 0)
if recent_count >= 1:
return 'High'
except Exception:
pass
return 'Medium'
def log_incident(alarm_name, trigger_reason, remediation_action, outcome, instance_id, duration_seconds, raw_data):
"""Log a structured incident record to DynamoDB."""
table = dynamodb.Table(INCIDENT_TABLE_NAME)
severity = calculate_severity(outcome, alarm_name, instance_id)
incident_record = {
'incident_id': str(uuid.uuid4()),
'timestamp': datetime.now(timezone.utc).isoformat(),
'alarm_name': alarm_name,
'trigger_reason': trigger_reason,
'remediation_action': remediation_action,
'outcome': outcome,
'instance_id': instance_id,
'duration_seconds': int(duration_seconds),
'severity': severity,
'raw_data': json.dumps(raw_data)
}
table.put_item(Item=incident_record)
print(f"Incident logged: {incident_record['incident_id']} | Severity: {severity}")
return incident_record
def lambda_handler(event, context):
print(f"Received event: {json.dumps(event)}")
start_time = time.time()
# Extract alarm details from the event
detail = event.get('detail', {})
alarm_name = detail.get('alarmName', 'Unknown')
instance_id = detail.get('configuration', {}).get('metrics', [{}])[0].get('metricStat', {}).get('metric', {}).get('dimensions', {}).get('InstanceId', '')
# If instance_id not in alarm detail, try environment variable
if not instance_id:
instance_id = os.environ.get('INSTANCE_ID', 'unknown')
trigger_reason = detail.get('state', {}).get('reason', 'CPU threshold exceeded')
remediation_action = 'systemctl restart nginx'
outcome = 'success'
try:
# Send SSM command to restart nginx
response = ssm_client.send_command(
InstanceIds=[instance_id],
DocumentName='AWS-RunShellScript',
Parameters={'commands': ['systemctl restart nginx']},
TimeoutSeconds=60
)
command_id = response['Command']['CommandId']
print(f"SSM Command sent: {command_id}")
# Wait briefly for command to complete
time.sleep(5)
# Check command status
result = ssm_client.get_command_invocation(
CommandId=command_id,
InstanceId=instance_id
)
if result['Status'] in ['Success']:
outcome = 'success'
print("Remediation successful: nginx restarted")
else:
outcome = 'failure'
print(f"Remediation status: {result['Status']}")
except Exception as e:
outcome = 'failure'
print(f"Remediation failed: {str(e)}")
duration_seconds = time.time() - start_time
# Log the incident to DynamoDB
incident_record = log_incident(
alarm_name=alarm_name,
trigger_reason=trigger_reason,
remediation_action=remediation_action,
outcome=outcome,
instance_id=instance_id,
duration_seconds=duration_seconds,
raw_data=detail
)
# Send SNS notification
if SNS_TOPIC_ARN:
try:
message = (
f"Remediation Complete\n"
f"Alarm: {alarm_name}\n"
f"Instance: {instance_id}\n"
f"Action: {remediation_action}\n"
f"Outcome: {outcome}\n"
f"Severity: {incident_record['severity']}\n"
f"Duration: {duration_seconds:.1f}s\n"
f"Incident ID: {incident_record['incident_id']}"
)
sns_client.publish(
TopicArn=SNS_TOPIC_ARN,
Subject=f"[{incident_record['severity']}] Remediation: {alarm_name}",
Message=message
)
except Exception as e:
print(f"SNS notification failed: {str(e)}")
return {
'statusCode': 200,
'body': json.dumps({
'message': 'Remediation complete',
'incident_id': incident_record['incident_id'],
'severity': incident_record['severity'],
'outcome': outcome
})
}
Test with a simulated event
You will now invoke the Lambda with a test event that simulates a CloudWatch alarm, then verify the incident record lands in DynamoDB.
- In the Lambda console for SelfHealingRemediation, choose the Test tab.
- Choose Create new event.
- For Event name, enter SimulatedAlarm.
- Replace the default JSON with the following test event:
{
"detail": {
"alarmName": "HighCPU-TestAlarm",
"state": {
"reason": "Threshold crossed: CPU utilization > 80% for 5 minutes"
},
"configuration": {
"metrics": [
{
"metricStat": {
"metric": {
"dimensions": {
"InstanceId": ""
}
}
}
}
]
}
}
}
Why is InstanceId empty?
When the InstanceId field is empty in the event, the Lambda falls back to the INSTANCE_ID environment variable you configured in Part 1. This simulates a real alarm trigger without needing to generate actual CPU load.
- Click Save changes.
- Click Test.
The function will execute. Depending on whether your EC2 instance has nginx running, the outcome may be success or failure. Either way, an incident record gets written to DynamoDB.
- Under Execution result, expand Details. You should see a JSON response containing incident_id, severity, and outcome.
- Open the DynamoDB console and choose Tables in the left navigation.
- Click DevOpsIncidentLog.
- Choose Explore table items.
You should see at least one item with fields like incident_id, alarm_name, severity, and timestamp. This confirms the incident memory layer is working.
No items in the table?
- Check the Lambda execution result for errors. If you see an AccessDeniedException mentioning DynamoDB, the inline policy was not attached to the correct role.
- Verify the table name in your code matches exactly: DevOpsIncidentLog (case-sensitive).
- Make sure you clicked Deploy after pasting the new code. Unsaved code runs the old version.
help me debug why my Lambda isn't writing to DynamoDB
Your remediation Lambda now has memory. Every incident it handles gets scored, classified, and stored permanently. Next up, you will enhance the copilot Lambda to query this history and provide multi-mode AI analysis.
Enhance the Copilot Lambda with Multi-Mode Analysis
Your remediation Lambda now logs every incident with a severity score. The next piece of the puzzle is teaching your copilot Lambda to read that history and reason about it in different ways.
In this step, you will upgrade the DevOpsCopilotFunction to query DynamoDB for incident records and run them through four specialized Bedrock analysis modes. Each mode uses a tailored system prompt so the AI focuses on a specific angle: summarizing, finding root causes, recommending prevention, or detecting repeat offenders.
In this step, get ready to:
- Grant the copilot Lambda DynamoDB read access.
- Replace the copilot Lambda code with multi-mode analysis capabilities.
- Test each analysis mode in the Lambda test console.
Grant DynamoDB read access to the copilot Lambda
The copilot Lambda needs permission to read incident records from DynamoDB. You will add an inline policy to its execution role, similar to what you did for the remediation Lambda in the previous step.
- In the AWS Management Console, search for IAM and open the IAM service.
- Click **Roles** in the left sidebar.
- Search for the execution role attached to your DevOpsCopilotFunction Lambda.
- Click on the role name to open it.
- Click **Add permissions**, then select **Create inline policy**.
- Switch to the **JSON** tab and paste the following policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dynamodb:Query",
"dynamodb:Scan"
],
"Resource": "arn:aws:dynamodb:us-east-1:*:table/DevOpsIncidentLog"
}
]
}
- Click **Next**.
- Enter a policy name like DynamoDBReadIncidentLog.
- Click **Create policy**.
What does this policy do?
This inline policy allows the copilot Lambda to run Query and Scan operations on the DevOpsIncidentLog table. The copilot needs Scan because it uses a FilterExpression on the timestamp attribute to pull incidents within a time range.
You should see the new policy listed under the role's permissions. The copilot Lambda can now read from DynamoDB.
Don't see the new policy?
Make sure you selected the correct role for the DevOpsCopilotFunction, not the SelfHealingRemediation role. You can verify the role name by opening the Lambda function and checking the **Configuration** > **Permissions** section.
Help me find the correct IAM role for my DevOpsCopilotFunction Lambda.
Replace the copilot Lambda code with multi-mode analysis
Now you will replace the entire DevOpsCopilotFunction code with an enhanced version. The new code adds DynamoDB incident queries, four analysis modes with tailored system prompts, and path-based request routing for multiple API endpoints.
- In the AWS Management Console, search for Lambda and open the Lambda service.
- Click on **DevOpsCopilotFunction** to open it.
- In the **Code** tab, select all the existing code in the editor and delete it.
- Paste the following updated code into the editor. This is the complete replacement for the function.
The code is presented in logical sections below so you can understand each part. Paste each section in order, building up the full file from top to bottom.
Start with the imports, client setup, and existing data collection helpers:
import json
import boto3
import os
from datetime import datetime, timezone, timedelta
from boto3.dynamodb.conditions import Key, Attr
bedrock_client = boto3.client('bedrock-runtime', region_name='us-east-1')
cloudwatch_client = boto3.client('cloudwatch', region_name='us-east-1')
logs_client = boto3.client('logs', region_name='us-east-1')
ec2_client = boto3.client('ec2', region_name='us-east-1')
dynamodb = boto3.resource('dynamodb', region_name='us-east-1')
MODEL_ID = 'amazon.nova-micro-v1:0'
INCIDENT_TABLE_NAME = 'DevOpsIncidentLog'
def collect_alarm_history():
"""Collect CloudWatch alarm history from the last 24 hours."""
try:
end_time = datetime.now(timezone.utc)
start_time = end_time - timedelta(hours=24)
response = cloudwatch_client.describe_alarm_history(
AlarmTypes=['MetricAlarm'],
HistoryItemType='StateUpdate',
StartDate=start_time,
EndDate=end_time,
MaxRecords=50
)
return response.get('AlarmHistoryItems', [])
except Exception as e:
return [{'error': str(e)}]
def collect_remediation_logs():
"""Collect remediation Lambda execution logs from the last 24 hours."""
try:
end_time = int(datetime.now(timezone.utc).timestamp() * 1000)
start_time = end_time - (24 * 60 * 60 * 1000)
response = logs_client.filter_log_events(
logGroupName='/aws/lambda/SelfHealingRemediation',
startTime=start_time,
endTime=end_time,
limit=50
)
events = response.get('events', [])
return [{'timestamp': e['timestamp'], 'message': e['message'][:200]} for e in events]
except Exception as e:
return [{'error': str(e)}]
def collect_instance_status():
"""Collect EC2 instance status."""
try:
instance_id = os.environ.get('INSTANCE_ID', '')
if not instance_id:
return [{'status': 'No instance ID configured'}]
response = ec2_client.describe_instance_status(
InstanceIds=[instance_id],
IncludeAllInstances=True
)
statuses = response.get('InstanceStatuses', [])
return [{'instance_id': s['InstanceId'],
'state': s['InstanceState']['Name'],
'system_status': s.get('SystemStatus', {}).get('Status', 'unknown'),
'instance_status': s.get('InstanceStatus', {}).get('Status', 'unknown')}
for s in statuses]
except Exception as e:
return [{'error': str(e)}]
What changed from Part 1?
The imports now include from boto3.dynamodb.conditions import Key, Attr for DynamoDB filter expressions. A dynamodb resource client and two new constants (MODEL_ID and INCIDENT_TABLE_NAME) have been added at the top. The three existing helper functions remain the same.
Next, add the new collect_incident_history function that queries DynamoDB:
def collect_incident_history(time_range='24h'):
"""Query DynamoDB for incident history within the specified time range."""
table = dynamodb.Table(INCIDENT_TABLE_NAME)
hours_map = {'24h': 24, '7d': 168, '30d': 720}
hours = hours_map.get(time_range, 24)
threshold = (datetime.now(timezone.utc) - timedelta(hours=hours)).isoformat()
try:
response = table.scan(
FilterExpression=Attr('timestamp').gte(threshold)
)
items = response.get('Items', [])
# Handle pagination for large result sets
while 'LastEvaluatedKey' in response:
response = table.scan(
FilterExpression=Attr('timestamp').gte(threshold),
ExclusiveStartKey=response['LastEvaluatedKey']
)
items.extend(response.get('Items', []))
# Sort by timestamp descending so newest incidents appear first
items.sort(key=lambda x: x.get('timestamp', ''), reverse=True)
return items
except Exception as e:
return [{'error': str(e)}]
What does this function do?
This function accepts a time range string (24h, 7d, or 30d) and scans the DynamoDB table for all incidents newer than that threshold. The Attr('timestamp').gte(threshold) filter expression returns only records with a timestamp greater than or equal to the cutoff.
The pagination loop ensures all results are collected even if DynamoDB splits them across multiple pages. Results are sorted newest-first for easy consumption by the AI.
Now add the get_system_prompt function that returns tailored prompts for each analysis mode:
def get_system_prompt(mode):
"""Return a tailored system prompt based on analysis mode."""
prompts = {
'summary': (
"You are an AI DevOps analyst. Analyze the provided incident data and give a clear, "
"concise summary of what happened. Prioritize High and Critical severity incidents. "
"Include timestamps, outcomes, and severity levels. Format the response in plain English "
"that a DevOps engineer can quickly scan. Start with the most important findings."
),
'root_cause': (
"You are an AI DevOps root cause analyst. Analyze the provided incident data to identify "
"WHY incidents are happening. Look for correlations: time-of-day patterns, frequency, "
"sequences of events, common triggers, and severity distribution. Identify the underlying "
"cause, not just the symptom. Be specific about what evidence supports your conclusions. "
"Structure your response as: 1) Pattern identified, 2) Evidence, 3) Likely root cause, "
"4) Confidence level."
),
'prevent': (
"You are an AI DevOps prevention specialist. Based on the incident patterns found in the "
"data, suggest specific AWS configuration changes, scaling strategies, or monitoring "
"improvements to prevent recurrence. Be specific with your recommendations. For example: "
"'Add a memory utilization CloudWatch alarm with threshold 80%' or 'Consider upgrading "
"from t2.micro to t2.small based on 5 CPU-related incidents this week'. Each recommendation "
"should include: what to change, why it helps, and estimated effort to implement."
),
'repeat_offender': (
"You are an AI DevOps repeat offender detector. Analyze the incident history to identify "
"which incident type has occurred most frequently. For each repeat offender: "
"1) Identify the pattern (what, when, how often), "
"2) Analyze whether frequency is increasing, decreasing, or stable, "
"3) Estimate when the next occurrence is likely based on current patterns, "
"4) Flag any incident occurring 3+ times in a 7-day window as a 'REPEAT OFFENDER', "
"5) Provide a focused remediation plan for that specific issue. "
"Be data-driven and cite specific timestamps and counts."
)
}
return prompts.get(mode, prompts['summary'])
Why four different system prompts?
Each mode steers Nova Micro toward a specific analytical lens. The summary mode gives a quick overview. The root_cause mode digs into correlations. The prevent mode outputs actionable AWS recommendations. The repeat_offender mode identifies patterns recurring 3+ times in a week.
Using the Bedrock Converse API system parameter keeps these instructions separate from user input, giving the model a clear role for each request.
Finally, add the updated analyze_with_bedrock function and all the request handler functions:
def analyze_with_bedrock(data, question, mode='summary'):
"""Send collected data to Bedrock Nova Micro for AI analysis."""
system_prompt = get_system_prompt(mode)
# For deeper analysis modes, include 7-day incident history
if mode in ['root_cause', 'prevent', 'repeat_offender']:
incident_history = collect_incident_history('7d')
data['incident_history_7d'] = incident_history
user_message = f"""Here is the current infrastructure data:
Alarm History (last 24h): {json.dumps(data.get('alarm_history', []), default=str)[:3000]}
Remediation Logs (last 24h): {json.dumps(data.get('remediation_logs', []), default=str)[:2000]}
Instance Status: {json.dumps(data.get('instance_status', []), default=str)[:1000]}
Incident History (DynamoDB): {json.dumps(data.get('incident_history', data.get('incident_history_7d', [])), default=str)[:4000]}
User Question: {question}
Provide your analysis based on the '{mode}' mode."""
try:
response = bedrock_client.converse(
modelId=MODEL_ID,
messages=[{'role': 'user', 'content': [{'text': user_message}]}],
system=[{'text': system_prompt}],
inferenceConfig={'maxTokens': 2048, 'temperature': 0.3}
)
output_message = response['output']['message']
return output_message['content'][0]['text']
except Exception as e:
return f"Error analyzing with Bedrock: {str(e)}"
def handle_ask(body):
"""Handle POST /ask requests."""
question = body.get('question', 'Give me a summary of recent incidents')
mode = body.get('mode', 'summary')
time_range = '7d' if mode in ['root_cause', 'prevent', 'repeat_offender'] else '24h'
data = {
'alarm_history': collect_alarm_history(),
'remediation_logs': collect_remediation_logs(),
'instance_status': collect_instance_status(),
'incident_history': collect_incident_history(time_range)
}
analysis = analyze_with_bedrock(data, question, mode)
return {
'statusCode': 200,
'headers': {'Content-Type': 'application/json', 'Access-Control-Allow-Origin': '*'},
'body': json.dumps({
'answer': analysis,
'mode': mode,
'data_sources': {
'alarm_history_count': len(data['alarm_history']),
'remediation_log_count': len(data['remediation_logs']),
'incident_history_count': len(data['incident_history'])
}
})
}
def handle_incidents(params):
"""Handle GET /incidents requests."""
time_range = params.get('range', '24h') if params else '24h'
incidents = collect_incident_history(time_range)
return {
'statusCode': 200,
'headers': {'Content-Type': 'application/json', 'Access-Control-Allow-Origin': '*'},
'body': json.dumps({'incidents': incidents, 'count': len(incidents), 'time_range': time_range}, default=str)
}
def handle_digest():
"""Handle POST /digest requests (on-demand digest)."""
incidents = collect_incident_history('7d')
digest_prompt = (
"Generate a weekly ops review report. Include: "
"1) Total incidents this week, "
"2) Severity breakdown (Critical/High/Medium/Low counts), "
"3) Patterns detected, "
"4) Repeat offenders flagged (any incident 3+ times in 7 days), "
"5) Most critical issues, "
"6) Root causes identified, "
"7) Top 3 preventive recommendations, "
"8) Overall system health rating (Healthy / Needs Attention / Critical), "
"9) Single most important action the operator should take this week. "
"Format as a readable report."
)
data = {
'incident_history': incidents,
'alarm_history': collect_alarm_history(),
'remediation_logs': collect_remediation_logs(),
'instance_status': collect_instance_status()
}
analysis = analyze_with_bedrock(data, digest_prompt, 'summary')
return {
'statusCode': 200,
'headers': {'Content-Type': 'application/json', 'Access-Control-Allow-Origin': '*'},
'body': json.dumps({
'digest': analysis,
'incident_count': len(incidents),
'generated_at': datetime.now(timezone.utc).isoformat()
})
}
def lambda_handler(event, context):
"""Main handler - routes requests based on path and method."""
print(f"Received event: {json.dumps(event)}")
http_method = event.get('httpMethod', 'POST')
path = event.get('path', '/ask')
body = {}
if event.get('body'):
try:
body = json.loads(event['body'])
except (json.JSONDecodeError, TypeError):
body = {}
# Route based on path and HTTP method
if path == '/ask' and http_method == 'POST':
return handle_ask(body)
elif path == '/incidents' and http_method == 'GET':
params = event.get('queryStringParameters')
return handle_incidents(params)
elif path == '/digest' and http_method == 'POST':
return handle_digest()
elif http_method == 'OPTIONS':
return {
'statusCode': 200,
'headers': {
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Methods': 'GET,POST,OPTIONS',
'Access-Control-Allow-Headers': 'Content-Type'
},
'body': ''
}
else:
return handle_ask(body)
What do the handler functions do?
The lambda_handler inspects the request path and HTTP method to route traffic. handle_ask processes the /ask endpoint with the new mode parameter. handle_incidents returns raw incident records from DynamoDB. handle_digest generates an on-demand weekly ops report.
The OPTIONS handler supports CORS preflight requests so a browser-based dashboard can call the API directly.
- Click **Deploy** to save and deploy the updated function code.
You should see a green "Successfully updated the function" banner at the top of the editor.
Increase the function timeout
This Lambda calls Bedrock with large context payloads, which can take 15 to 30 seconds to process. Increase the timeout to avoid premature failures.
- Click the **Configuration** tab, then **General configuration**.
- Click **Edit**.
- Set **Timeout** to 1 minute 0 seconds.
- Click **Save**.
✔️ Awesome, I've got everything!
Great. Double check you clicked **Deploy** and saw the success banner.
ⓧ I'd like to double check the full code
Here is the complete lambda_copilot.py code for comparison:
import json
import boto3
import os
from datetime import datetime, timezone, timedelta
from boto3.dynamodb.conditions import Key, Attr
bedrock_client = boto3.client('bedrock-runtime', region_name='us-east-1')
cloudwatch_client = boto3.client('cloudwatch', region_name='us-east-1')
logs_client = boto3.client('logs', region_name='us-east-1')
ec2_client = boto3.client('ec2', region_name='us-east-1')
dynamodb = boto3.resource('dynamodb', region_name='us-east-1')
MODEL_ID = 'amazon.nova-micro-v1:0'
INCIDENT_TABLE_NAME = 'DevOpsIncidentLog'
def collect_alarm_history():
"""Collect CloudWatch alarm history from the last 24 hours."""
try:
end_time = datetime.now(timezone.utc)
start_time = end_time - timedelta(hours=24)
response = cloudwatch_client.describe_alarm_history(
AlarmTypes=['MetricAlarm'],
HistoryItemType='StateUpdate',
StartDate=start_time,
EndDate=end_time,
MaxRecords=50
)
return response.get('AlarmHistoryItems', [])
except Exception as e:
return [{'error': str(e)}]
def collect_remediation_logs():
"""Collect remediation Lambda execution logs from the last 24 hours."""
try:
end_time = int(datetime.now(timezone.utc).timestamp() * 1000)
start_time = end_time - (24 * 60 * 60 * 1000)
response = logs_client.filter_log_events(
logGroupName='/aws/lambda/SelfHealingRemediation',
startTime=start_time,
endTime=end_time,
limit=50
)
events = response.get('events', [])
return [{'timestamp': e['timestamp'], 'message': e['message'][:200]} for e in events]
except Exception as e:
return [{'error': str(e)}]
def collect_instance_status():
"""Collect EC2 instance status."""
try:
instance_id = os.environ.get('INSTANCE_ID', '')
if not instance_id:
return [{'status': 'No instance ID configured'}]
response = ec2_client.describe_instance_status(
InstanceIds=[instance_id],
IncludeAllInstances=True
)
statuses = response.get('InstanceStatuses', [])
return [{'instance_id': s['InstanceId'],
'state': s['InstanceState']['Name'],
'system_status': s.get('SystemStatus', {}).get('Status', 'unknown'),
'instance_status': s.get('InstanceStatus', {}).get('Status', 'unknown')}
for s in statuses]
except Exception as e:
return [{'error': str(e)}]
def collect_incident_history(time_range='24h'):
"""Query DynamoDB for incident history within the specified time range."""
table = dynamodb.Table(INCIDENT_TABLE_NAME)
hours_map = {'24h': 24, '7d': 168, '30d': 720}
hours = hours_map.get(time_range, 24)
threshold = (datetime.now(timezone.utc) - timedelta(hours=hours)).isoformat()
try:
response = table.scan(
FilterExpression=Attr('timestamp').gte(threshold)
)
items = response.get('Items', [])
while 'LastEvaluatedKey' in response:
response = table.scan(
FilterExpression=Attr('timestamp').gte(threshold),
ExclusiveStartKey=response['LastEvaluatedKey']
)
items.extend(response.get('Items', []))
items.sort(key=lambda x: x.get('timestamp', ''), reverse=True)
return items
except Exception as e:
return [{'error': str(e)}]
def get_system_prompt(mode):
"""Return a tailored system prompt based on analysis mode."""
prompts = {
'summary': (
"You are an AI DevOps analyst. Analyze the provided incident data and give a clear, "
"concise summary of what happened. Prioritize High and Critical severity incidents. "
"Include timestamps, outcomes, and severity levels. Format the response in plain English "
"that a DevOps engineer can quickly scan. Start with the most important findings."
),
'root_cause': (
"You are an AI DevOps root cause analyst. Analyze the provided incident data to identify "
"WHY incidents are happening. Look for correlations: time-of-day patterns, frequency, "
"sequences of events, common triggers, and severity distribution. Identify the underlying "
"cause, not just the symptom. Be specific about what evidence supports your conclusions. "
"Structure your response as: 1) Pattern identified, 2) Evidence, 3) Likely root cause, "
"4) Confidence level."
),
'prevent': (
"You are an AI DevOps prevention specialist. Based on the incident patterns found in the "
"data, suggest specific AWS configuration changes, scaling strategies, or monitoring "
"improvements to prevent recurrence. Be specific with your recommendations. For example: "
"'Add a memory utilization CloudWatch alarm with threshold 80%' or 'Consider upgrading "
"from t2.micro to t2.small based on 5 CPU-related incidents this week'. Each recommendation "
"should include: what to change, why it helps, and estimated effort to implement."
),
'repeat_offender': (
"You are an AI DevOps repeat offender detector. Analyze the incident history to identify "
"which incident type has occurred most frequently. For each repeat offender: "
"1) Identify the pattern (what, when, how often), "
"2) Analyze whether frequency is increasing, decreasing, or stable, "
"3) Estimate when the next occurrence is likely based on current patterns, "
"4) Flag any incident occurring 3+ times in a 7-day window as a 'REPEAT OFFENDER', "
"5) Provide a focused remediation plan for that specific issue. "
"Be data-driven and cite specific timestamps and counts."
)
}
return prompts.get(mode, prompts['summary'])
def analyze_with_bedrock(data, question, mode='summary'):
"""Send collected data to Bedrock Nova Micro for AI analysis."""
system_prompt = get_system_prompt(mode)
if mode in ['root_cause', 'prevent', 'repeat_offender']:
incident_history = collect_incident_history('7d')
data['incident_history_7d'] = incident_history
user_message = f"""Here is the current infrastructure data:
Alarm History (last 24h): {json.dumps(data.get('alarm_history', []), default=str)[:3000]}
Remediation Logs (last 24h): {json.dumps(data.get('remediation_logs', []), default=str)[:2000]}
Instance Status: {json.dumps(data.get('instance_status', []), default=str)[:1000]}
Incident History (DynamoDB): {json.dumps(data.get('incident_history', data.get('incident_history_7d', [])), default=str)[:4000]}
User Question: {question}
Provide your analysis based on the '{mode}' mode."""
try:
response = bedrock_client.converse(
modelId=MODEL_ID,
messages=[{'role': 'user', 'content': [{'text': user_message}]}],
system=[{'text': system_prompt}],
inferenceConfig={'maxTokens': 2048, 'temperature': 0.3}
)
output_message = response['output']['message']
return output_message['content'][0]['text']
except Exception as e:
return f"Error analyzing with Bedrock: {str(e)}"
def handle_ask(body):
"""Handle POST /ask requests."""
question = body.get('question', 'Give me a summary of recent incidents')
mode = body.get('mode', 'summary')
time_range = '7d' if mode in ['root_cause', 'prevent', 'repeat_offender'] else '24h'
data = {
'alarm_history': collect_alarm_history(),
'remediation_logs': collect_remediation_logs(),
'instance_status': collect_instance_status(),
'incident_history': collect_incident_history(time_range)
}
analysis = analyze_with_bedrock(data, question, mode)
return {
'statusCode': 200,
'headers': {'Content-Type': 'application/json', 'Access-Control-Allow-Origin': '*'},
'body': json.dumps({
'answer': analysis,
'mode': mode,
'data_sources': {
'alarm_history_count': len(data['alarm_history']),
'remediation_log_count': len(data['remediation_logs']),
'incident_history_count': len(data['incident_history'])
}
})
}
def handle_incidents(params):
"""Handle GET /incidents requests."""
time_range = params.get('range', '24h') if params else '24h'
incidents = collect_incident_history(time_range)
return {
'statusCode': 200,
'headers': {'Content-Type': 'application/json', 'Access-Control-Allow-Origin': '*'},
'body': json.dumps({'incidents': incidents, 'count': len(incidents), 'time_range': time_range}, default=str)
}
def handle_digest():
"""Handle POST /digest requests (on-demand digest)."""
incidents = collect_incident_history('7d')
digest_prompt = (
"Generate a weekly ops review report. Include: "
"1) Total incidents this week, "
"2) Severity breakdown (Critical/High/Medium/Low counts), "
"3) Patterns detected, "
"4) Repeat offenders flagged (any incident 3+ times in 7 days), "
"5) Most critical issues, "
"6) Root causes identified, "
"7) Top 3 preventive recommendations, "
"8) Overall system health rating (Healthy / Needs Attention / Critical), "
"9) Single most important action the operator should take this week. "
"Format as a readable report."
)
data = {
'incident_history': incidents,
'alarm_history': collect_alarm_history(),
'remediation_logs': collect_remediation_logs(),
'instance_status': collect_instance_status()
}
analysis = analyze_with_bedrock(data, digest_prompt, 'summary')
return {
'statusCode': 200,
'headers': {'Content-Type': 'application/json', 'Access-Control-Allow-Origin': '*'},
'body': json.dumps({
'digest': analysis,
'incident_count': len(incidents),
'generated_at': datetime.now(timezone.utc).isoformat()
})
}
def lambda_handler(event, context):
"""Main handler - routes requests based on path and method."""
print(f"Received event: {json.dumps(event)}")
http_method = event.get('httpMethod', 'POST')
path = event.get('path', '/ask')
body = {}
if event.get('body'):
try:
body = json.loads(event['body'])
except (json.JSONDecodeError, TypeError):
body = {}
if path == '/ask' and http_method == 'POST':
return handle_ask(body)
elif path == '/incidents' and http_method == 'GET':
params = event.get('queryStringParameters')
return handle_incidents(params)
elif path == '/digest' and http_method == 'POST':
return handle_digest()
elif http_method == 'OPTIONS':
return {
'statusCode': 200,
'headers': {
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Methods': 'GET,POST,OPTIONS',
'Access-Control-Allow-Headers': 'Content-Type'
},
'body': ''
}
else:
return handle_ask(body)
Seeing a deployment error?
If the deployment fails, check that the code has no extra indentation issues from pasting. The Lambda console editor sometimes introduces leading whitespace. Also confirm the runtime is set to python3.12 under **Runtime settings**.
Help me fix a deployment error in my DevOpsCopilotFunction Lambda.
Test each analysis mode with the Lambda test console
Now you will verify that all four modes work by creating test events that simulate API Gateway requests.
- In the **Code** tab of your DevOpsCopilotFunction, click the **Test** tab.
- Click **Create new event**.
- Enter testSummaryMode as the event name.
- Replace the default JSON with the following test event:
{
"httpMethod": "POST",
"path": "/ask",
"body": "{\"question\": \"Summarize recent incidents\", \"mode\": \"summary\"}"
}
- Click **Save**, then click **Test**.
What should I see?
After 10 to 30 seconds, you should see a green "Execution result: succeeded" banner. The response body will contain an answer field with an AI-generated summary of your incidents, a mode field showing summary, and a data_sources object showing how many records were collected.
- Test the other modes by creating new test events with mode set to root_cause, prevent, and repeat_offender.
- Verify each mode returns a successful response with an answer field containing analysis relevant to that mode.
Function timing out?
If you see a "Task timed out" error, make sure you increased the timeout to 60 seconds in the **Configuration** > **General configuration** settings. Bedrock calls with large context can take 15 to 30 seconds to respond.
If you see an AccessDeniedException mentioning DynamoDB, go back to IAM and confirm the inline policy was added to the correct role for this function.
Help me debug my DevOpsCopilotFunction test failures.
Your copilot Lambda now has full incident history awareness and four specialized analysis modes. Next up, you will expand the REST API to expose these capabilities through new endpoints.
Expand the REST API with New Endpoints
Your copilot Lambda can now analyze incidents in four different modes. It can also serve incident history and generate weekly digests on demand.
The problem is that the outside world can only reach your Lambda through the single /ask endpoint you set up in Part 1. In this step, you will add two new API Gateway resources (/incidents and /digest) so each capability has its own clean URL.
In this step, get ready to:
- Add /incidents and /digest resources to your REST API.
- Deploy the updated API to your existing stage.
- Test all three endpoints to confirm they work.
Add the /incidents and /digest resources
Your Lambda already routes requests based on the path (/ask, /incidents, /digest). Now you need matching resources in API Gateway so the requests can reach the Lambda in the first place.
- Return to the API Gateway console at https://console.aws.amazon.com/apigateway.
- Select your existing REST API (the one you created in Part 1).
- In the Resources pane, select the root resource (/).
- Choose Create resource.
- For Resource name, enter incidents.
- Choose Create resource.
You should see /incidents appear in the resource tree under the root.
- With /incidents selected, choose Create method.
- For Method type, select GET.
- For Integration type, select Lambda Function.
- Turn on Lambda proxy integration.
- For Lambda function, select DevOpsCopilotFunction.
- Choose Create method.
Why Lambda proxy integration?
With Lambda proxy integration, API Gateway passes the entire incoming request (path, method, headers, query strings, body) directly to your Lambda function. Your Lambda then inspects event['path'] and event['httpMethod'] to decide how to route it. This keeps your API Gateway configuration simple because you do not need mapping templates.
Now repeat the same process for the /digest resource.
- In the Resources pane, select the root resource (/) again.
- Choose Create resource.
- For Resource name, enter digest.
- Choose Create resource.
- With /digest selected, choose Create method.
- For Method type, select POST.
- For Integration type, select Lambda Function.
- Turn on Lambda proxy integration.
- For Lambda function, select DevOpsCopilotFunction.
- Choose Create method.
Your resource tree should now show three resources: /ask, /incidents, and /digest.
What about the /ask endpoint and the new mode parameter?
You do not need to change the existing /ask POST method in API Gateway. Because it uses Lambda proxy integration, the full request body (including the new mode field) is already passed to your Lambda. Your updated Lambda code reads body.get('mode', 'summary') from the body automatically.
Don't see DevOpsCopilotFunction in the dropdown?
Make sure you are in the us-east-1 region. The Lambda function dropdown only shows functions in the same region as the API. Also confirm the function name matches exactly. Help me find my Lambda function in the API Gateway integration dropdown.
Deploy the API to your existing stage
Adding resources and methods does not make them live automatically. You must redeploy the API to push changes to clients.
- In the Resources pane, choose Deploy API.
- For Stage, select your existing stage name from the dropdown (the stage you created in Part 1).
- Choose Deploy.
- From the main navigation pane, choose Stages.
- Under Stage details, choose the copy icon to copy your API's invoke URL.
- Save this URL somewhere accessible. You will use it for testing and again in Step 5.
What is the invoke URL?
The invoke URL is the base address for calling your API from outside the console. It follows the pattern https://{api-id}.execute-api.us-east-1.amazonaws.com/{stage}. You append resource paths like /ask, /incidents, or /digest to call each endpoint.
Test all three endpoints
You can use the API Gateway console's built-in test feature to verify each endpoint without leaving the browser. This sends a real invocation to your Lambda.
- In the Resources pane, select the POST method under /ask.
- Choose the Test tab. You might need to choose the right arrow button to show the tab.
- In the Request body field, enter the following JSON:
{"question": "What happened in the last 24 hours?", "mode": "summary"}
- Choose Test.
You should see a 200 status code in the response. The response body contains an answer field with the AI summary and a mode field showing summary.
Why does the first test take a while?
The DevOpsCopilotFunction Lambda calls Bedrock for its analysis. This can take 10-30 seconds on the first invocation due to cold start plus the AI inference time. Subsequent calls within the same warm Lambda are faster.
Now test the /incidents endpoint.
- In the Resources pane, select the GET method under /incidents.
- Choose the Test tab.
- In the Query strings field, enter range=24h.
- Choose Test.
You should see a 200 status code. The response body contains an incidents array and a count field. If you ran test events in Step 2, the incidents you logged should appear here.
Finally, test the /digest endpoint.
- In the Resources pane, select the POST method under /digest.
- Choose the Test tab.
- Leave the Request body empty.
- Choose Test.
You should see a 200 status code. The response body contains a digest field with the AI-generated weekly report and a generated_at timestamp.
Seeing a 500 status or timeout?
The most common cause is a Lambda timeout. The digest endpoint calls Bedrock with a large prompt, which can take over 30 seconds. Check that your DevOpsCopilotFunction has a timeout of at least 60 seconds in the Lambda console (under Configuration > General configuration).
If you see an access denied error in the logs, verify that the Lambda's IAM role has DynamoDB Query and Scan permissions on the DevOpsIncidentLog table (added in Step 3). Help me debug a 500 error on my /digest API Gateway endpoint.
Your REST API now has three working endpoints. Next, you will build an automated weekly digest system that generates AI-powered reports on a schedule and emails them to you.
Build the Weekly Digest Automation
Your copilot can now analyze incidents on demand. But a truly proactive system should surface patterns before anyone asks.
In this step, you will build a weekly digest system. A new Lambda function pulls seven days of incident history from DynamoDB, feeds it to Bedrock for AI-powered summarization, and emails the report to your SNS topic every Monday morning via EventBridge.
In this step, get ready to:
- Create the WeeklyDigestFunction Lambda with incident collection, AI digest generation, and SNS email publishing.
- Schedule weekly execution with an EventBridge cron rule and configure IAM permissions.
- Test the digest by manually invoking the function and checking your email.
Create the WeeklyDigestFunction Lambda
This Lambda will run once a week, scan the DevOpsIncidentLog table for the last seven days of incidents, ask Nova Micro to generate an operational digest, and publish the result to SNS as an email.
- In the AWS console search bar, type Lambda and select the Lambda service.
- Click Create function.
- Select Author from scratch.
- For Function name, enter WeeklyDigestFunction.
- For Runtime, select Python 3.12.
- Under Permissions, expand Change default execution role and select Create a new role with basic Lambda permissions.
- Click Create function.
After creation, you will see the function configuration page with a code editor.
- In the Code source editor, select all the default code and replace it with the following:
import json
import boto3
import os
from datetime import datetime, timezone, timedelta
from boto3.dynamodb.conditions import Attr
# Initialize AWS service clients
bedrock_client = boto3.client('bedrock-runtime', region_name='us-east-1')
sns_client = boto3.client('sns', region_name='us-east-1')
dynamodb = boto3.resource('dynamodb', region_name='us-east-1')
# Configuration constants
MODEL_ID = 'amazon.nova-micro-v1:0'
INCIDENT_TABLE_NAME = 'DevOpsIncidentLog'
SNS_TOPIC_ARN = os.environ.get('SNS_TOPIC_ARN', '')
def collect_incident_history(days=7):
"""Query DynamoDB for incident history within the specified number of days."""
table = dynamodb.Table(INCIDENT_TABLE_NAME)
threshold = (datetime.now(timezone.utc) - timedelta(days=days)).isoformat()
try:
response = table.scan(FilterExpression=Attr('timestamp').gte(threshold))
items = response.get('Items', [])
# Handle pagination for large result sets
while 'LastEvaluatedKey' in response:
response = table.scan(FilterExpression=Attr('timestamp').gte(threshold), ExclusiveStartKey=response['LastEvaluatedKey'])
items.extend(response.get('Items', []))
# Sort by timestamp descending (most recent first)
items.sort(key=lambda x: x.get('timestamp', ''), reverse=True)
return items
except Exception as e:
print(f"Error querying DynamoDB: {str(e)}")
return []
def generate_digest(incidents):
"""Use Bedrock Nova Micro to generate a weekly digest report."""
system_prompt = (
"You are an AI DevOps operations analyst generating a weekly ops review. "
"Your report should be clear, actionable, and formatted for email delivery. "
"Include: total incidents, severity breakdown, patterns detected, repeat offenders, "
"most critical issues, root causes identified, and top 3 preventive recommendations. "
"Compare to previous data if available. Identify escalating trends. "
"Rate overall system health (Healthy / Needs Attention / Critical). "
"Highlight the single most important action the operator should take this week."
)
# Calculate severity breakdown for context
severity_counts = {'Critical': 0, 'High': 0, 'Medium': 0, 'Low': 0}
for incident in incidents:
severity = incident.get('severity', 'Medium')
severity_counts[severity] = severity_counts.get(severity, 0) + 1
user_message = f"""Generate a weekly operations digest report based on the following data:
Total incidents this week: {len(incidents)}
Severity breakdown: {json.dumps(severity_counts)}
Incident details (last 7 days):
{json.dumps(incidents[:50], default=str)[:6000]}
Please provide a comprehensive weekly ops review report."""
try:
response = bedrock_client.converse(
modelId=MODEL_ID,
messages=[{'role': 'user', 'content': [{'text': user_message}]}],
system=[{'text': system_prompt}],
inferenceConfig={'maxTokens': 3000, 'temperature': 0.3}
)
output_message = response['output']['message']
return output_message['content'][0]['text']
except Exception as e:
return f"Error generating digest: {str(e)}"
def lambda_handler(event, context):
print("Weekly Digest Lambda triggered")
# Collect incidents from the last 7 days
incidents = collect_incident_history(days=7)
print(f"Found {len(incidents)} incidents in the last 7 days")
# Generate AI-powered digest
digest_report = generate_digest(incidents)
print("Digest report generated")
# Calculate severity counts for the email summary
severity_counts = {'Critical': 0, 'High': 0, 'Medium': 0, 'Low': 0}
for incident in incidents:
severity = incident.get('severity', 'Medium')
severity_counts[severity] = severity_counts.get(severity, 0) + 1
# Format the email body
email_body = f"""
=====================================
WEEKLY DEVOPS OPERATIONS DIGEST
Generated: {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M UTC')}
=====================================
SUMMARY
-------
Total Incidents: {len(incidents)}
Critical: {severity_counts['Critical']} | High: {severity_counts['High']} | Medium: {severity_counts['Medium']} | Low: {severity_counts['Low']}
AI ANALYSIS
-----------
{digest_report}
=====================================
This report was generated automatically by your AI DevOps Copilot.
"""
# Publish to SNS
if SNS_TOPIC_ARN:
try:
sns_client.publish(
TopicArn=SNS_TOPIC_ARN,
Subject=f"Weekly DevOps Digest - {datetime.now(timezone.utc).strftime('%Y-%m-%d')}",
Message=email_body
)
print("Digest published to SNS successfully")
except Exception as e:
print(f"Failed to publish to SNS: {str(e)}")
else:
print("No SNS_TOPIC_ARN configured, skipping email")
return {
'statusCode': 200,
'body': json.dumps({
'message': 'Weekly digest generated and sent',
'incident_count': len(incidents),
'severity_breakdown': severity_counts,
'digest_preview': digest_report[:500]
})
}
What does this code do?
This Lambda has three main responsibilities:
- collect_incident_history() scans the DynamoDB table for all incidents in the past 7 days, handling pagination for large result sets.
- generate_digest() sends the incident data to Bedrock Nova Micro with a system prompt tailored for weekly ops reviews. It calculates severity breakdowns as additional context for the AI.
- lambda_handler() orchestrates everything. It collects incidents, generates the AI analysis, formats a readable email body with severity counts, and publishes it to SNS.
- Click Deploy to save your code changes.
You should see a green banner confirming the function was updated successfully.
Now configure the environment variable and timeout for this function.
- Click the Configuration tab.
- Select Environment variables in the left sidebar.
- Click Edit.
- Click Add environment variable.
- For Key, enter SNS_TOPIC_ARN.
- For Value, paste your SNS topic ARN from Part 1.
- Click Save.
- Still in the Configuration tab, select General configuration in the left sidebar.
- Click Edit.
- Set Timeout to 2 minutes and 0 seconds.
- Click Save.
Why 2 minutes?
The digest Lambda processes more data than the copilot Lambda. It scans a full week of incidents, sends a larger context to Bedrock, and then publishes to SNS. Two minutes gives it plenty of headroom.
Don't see the Configuration tab?
Make sure you are on the WeeklyDigestFunction page itself, not the Lambda functions list. Click your function name in the list to open it.
help me find the Configuration tab for my Lambda function
✔️ Awesome, I've got everything!
Great. Make sure you saved both the code (Deploy) and the configuration changes.
ⓧ I'd like to double check the full code
import json
import boto3
import os
from datetime import datetime, timezone, timedelta
from boto3.dynamodb.conditions import Attr
bedrock_client = boto3.client('bedrock-runtime', region_name='us-east-1')
sns_client = boto3.client('sns', region_name='us-east-1')
dynamodb = boto3.resource('dynamodb', region_name='us-east-1')
MODEL_ID = 'amazon.nova-micro-v1:0'
INCIDENT_TABLE_NAME = 'DevOpsIncidentLog'
SNS_TOPIC_ARN = os.environ.get('SNS_TOPIC_ARN', '')
def collect_incident_history(days=7):
"""Query DynamoDB for incident history within the specified number of days."""
table = dynamodb.Table(INCIDENT_TABLE_NAME)
threshold = (datetime.now(timezone.utc) - timedelta(days=days)).isoformat()
try:
response = table.scan(FilterExpression=Attr('timestamp').gte(threshold))
items = response.get('Items', [])
while 'LastEvaluatedKey' in response:
response = table.scan(FilterExpression=Attr('timestamp').gte(threshold), ExclusiveStartKey=response['LastEvaluatedKey'])
items.extend(response.get('Items', []))
items.sort(key=lambda x: x.get('timestamp', ''), reverse=True)
return items
except Exception as e:
print(f"Error querying DynamoDB: {str(e)}")
return []
def generate_digest(incidents):
"""Use Bedrock Nova Micro to generate a weekly digest report."""
system_prompt = (
"You are an AI DevOps operations analyst generating a weekly ops review. "
"Your report should be clear, actionable, and formatted for email delivery. "
"Include: total incidents, severity breakdown, patterns detected, repeat offenders, "
"most critical issues, root causes identified, and top 3 preventive recommendations. "
"Compare to previous data if available. Identify escalating trends. "
"Rate overall system health (Healthy / Needs Attention / Critical). "
"Highlight the single most important action the operator should take this week."
)
severity_counts = {'Critical': 0, 'High': 0, 'Medium': 0, 'Low': 0}
for incident in incidents:
severity = incident.get('severity', 'Medium')
severity_counts[severity] = severity_counts.get(severity, 0) + 1
user_message = f"""Generate a weekly operations digest report based on the following data:
Total incidents this week: {len(incidents)}
Severity breakdown: {json.dumps(severity_counts)}
Incident details (last 7 days):
{json.dumps(incidents[:50], default=str)[:6000]}
Please provide a comprehensive weekly ops review report."""
try:
response = bedrock_client.converse(
modelId=MODEL_ID,
messages=[{'role': 'user', 'content': [{'text': user_message}]}],
system=[{'text': system_prompt}],
inferenceConfig={'maxTokens': 3000, 'temperature': 0.3}
)
output_message = response['output']['message']
return output_message['content'][0]['text']
except Exception as e:
return f"Error generating digest: {str(e)}"
def lambda_handler(event, context):
print("Weekly Digest Lambda triggered")
incidents = collect_incident_history(days=7)
print(f"Found {len(incidents)} incidents in the last 7 days")
digest_report = generate_digest(incidents)
print("Digest report generated")
severity_counts = {'Critical': 0, 'High': 0, 'Medium': 0, 'Low': 0}
for incident in incidents:
severity = incident.get('severity', 'Medium')
severity_counts[severity] = severity_counts.get(severity, 0) + 1
email_body = f"""
=====================================
WEEKLY DEVOPS OPERATIONS DIGEST
Generated: {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M UTC')}
=====================================
SUMMARY
-------
Total Incidents: {len(incidents)}
Critical: {severity_counts['Critical']} | High: {severity_counts['High']} | Medium: {severity_counts['Medium']} | Low: {severity_counts['Low']}
AI ANALYSIS
-----------
{digest_report}
=====================================
This report was generated automatically by your AI DevOps Copilot.
"""
if SNS_TOPIC_ARN:
try:
sns_client.publish(
TopicArn=SNS_TOPIC_ARN,
Subject=f"Weekly DevOps Digest - {datetime.now(timezone.utc).strftime('%Y-%m-%d')}",
Message=email_body
)
print("Digest published to SNS successfully")
except Exception as e:
print(f"Failed to publish to SNS: {str(e)}")
else:
print("No SNS_TOPIC_ARN configured, skipping email")
return {
'statusCode': 200,
'body': json.dumps({
'message': 'Weekly digest generated and sent',
'incident_count': len(incidents),
'severity_breakdown': severity_counts,
'digest_preview': digest_report[:500]
})
}
Schedule with EventBridge and configure permissions
Before the schedule can work, the Lambda's execution role needs permission to read DynamoDB, invoke Bedrock, and publish to SNS.
- Still on the WeeklyDigestFunction page, click the Configuration tab.
- Select Permissions in the left sidebar.
- Under Execution role, click the role name link. This opens the IAM console in a new tab.
- Click Add permissions, then Create inline policy.
- Switch to the JSON tab and paste this policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dynamodb:Scan"
],
"Resource": "arn:aws:dynamodb:us-east-1:*:table/DevOpsIncidentLog"
},
{
"Effect": "Allow",
"Action": [
"bedrock:InvokeModel"
],
"Resource": "arn:aws:bedrock:us-east-1::foundation-model/amazon.nova-micro-v1:0"
},
{
"Effect": "Allow",
"Action": [
"sns:Publish"
],
"Resource": "*"
}
]
}
What permissions does this grant?
- dynamodb:Scan lets the function scan the DevOpsIncidentLog table for incident history.
- bedrock:InvokeModel allows calling amazon.nova-micro-v1:0 specifically.
- sns:Publish lets the function send the digest email through your SNS topic.
- Click Next.
- For Policy name, enter WeeklyDigestPermissions.
- Click Create policy.
You should see the new policy listed under the role's permissions. Now create the EventBridge scheduled rule.
- In the AWS console search bar, type EventBridge and select the Amazon EventBridge service.
- In the left navigation pane under Scheduler, click Scheduled rule (legacy).
- Click Create scheduled rule.
- For Name, enter WeeklyDigestSchedule.
- Make sure Enable the scheduled rule is toggled on.
- Click Next.
- For Schedule pattern, choose A fine-grained schedule that runs at a specific time.
- For Cron expression, enter cron(0 9 ? * MON *). This triggers every Monday at 9:00 AM UTC.
- Click Next.
What does this cron expression mean?
The six fields are: minutes, hours, day-of-month, month, day-of-week, year. So cron(0 9 ? * MON *) means: minute 0, hour 9, any day-of-month (?), any month (*), Monday only (MON), any year (*).
- For Target type, select AWS service.
- For Select a target, choose Lambda function.
- For Function, select WeeklyDigestFunction.
- Click Next.
- Click Next again to skip tags.
- Review your settings and click Create rule.
EventBridge automatically creates a resource-based policy on your Lambda granting the events.amazonaws.com principal permission to invoke it.
Don't see your function in the target dropdown?
Make sure you are in the us-east-1 region. The function must be in the same region as the EventBridge rule. Also confirm the function was created and deployed successfully.
help me connect my EventBridge rule to my Lambda function
Test the weekly digest manually
The schedule runs every Monday, but you can test right now by manually invoking the function.
- Navigate back to the Lambda console and open your WeeklyDigestFunction.
- Click the Test tab.
- For Event name, enter ManualDigestTest.
- Leave the event JSON as the default empty object {}.
- Click Test.
The function may take 15-30 seconds to complete as it queries DynamoDB and calls Bedrock.
- Check the execution result. You should see a statusCode: 200 response with incident_count and severity_breakdown fields.
- Check your email inbox for a message with the subject Weekly DevOps Digest followed by today's date.
What should the email look like?
The email contains a plain-text report with a header banner, a summary section showing total incidents and severity breakdown, and an AI Analysis section with Bedrock's generated insights. If you have no incidents logged yet, the report will note zero incidents and suggest the system is healthy.
Function timed out or returned an error?
- If you see a timeout error, confirm you set the timeout to 2 minutes in General configuration.
- If you see an AccessDeniedException, double-check that the inline policy you attached includes dynamodb:Scan, bedrock:InvokeModel, and sns:Publish.
- If no email arrives, verify that the SNS_TOPIC_ARN environment variable is set correctly and that your email subscription is confirmed.
help me debug my WeeklyDigestFunction Lambda errors
Your AI copilot now has a complete automated reporting system. Every Monday, it will email you an AI-generated operational digest without any manual intervention. Next up, the Secret Mission challenges you to build a full ops console dashboard with an incident timeline, mode selector, and on-demand digest generation.
Secret mission
Enhanced Chat Dashboard
Your copilot backend is fully operational with incident memory, multi-mode analysis, and automated weekly digests. But a real ops engineer needs a visual command center. In this secret mission, you will build a full dashboard with a mode selector, severity-coded incident timeline, and on-demand digest generation.
Clean Up Your Resources
Clean Up Your Resources
Decide whether to keep your resources running, pause them to come back later, or delete them entirely. Most Part 2 resources incur negligible cost at rest. The DynamoDB table only charges when read or written, and Lambda functions remain within the free tier unless you exceed 1 million requests per month.
The main ongoing cost is the EventBridge rule that fires every Monday and invokes the digest Lambda. Each invocation calls Bedrock ($0.000035 per 1K input tokens, $0.00014 per 1K output tokens) and reads from DynamoDB. This amounts to fractions of a cent per week.
Cost warning
If you leave the EventBridge schedule enabled, the WeeklyDigestFunction Lambda will continue to invoke Bedrock every Monday. While each invocation costs less than a cent, the charges accumulate over time. Disable or delete the schedule if you are done experimenting.
Resources you used:
- DynamoDB table (DevOpsIncidentLog).
- Lambda function (WeeklyDigestFunction).
- EventBridge rule (WeeklyDigestSchedule).
- API Gateway resources (/incidents GET and /digest POST endpoints added in Part 2).
- Local file (dashboard.html).
- EC2 instance from Part 1 (continues running, t2.micro free tier eligible).
Keep everything running
No action needed. Choose this if you are still actively building on the copilot or plan to continue with future parts of the series.
- The EventBridge rule will fire every Monday at 9:00 AM UTC and email you a digest.
- The DynamoDB table will persist all incident records.
- API Gateway and Lambda functions remain available for dashboard use.
- The EC2 instance from Part 1 continues running (t2.micro free tier eligible for 12 months).
Pause - I'll come back to this later
Shut down the scheduled automation to stop weekly emails while keeping all resources intact for later use.
- Open the Amazon EventBridge console at https://console.aws.amazon.com/events/.
- In the navigation pane, choose Rules.
- Select the button next to WeeklyDigestSchedule.
- Choose Disable, then confirm by choosing Disable again.
The rule is now disabled. No weekly digest emails will be sent until you re-enable it. All other resources remain intact and can be used on-demand via the dashboard or API.
Delete - I don't want to use this again
Remove all Part 2 resources and start fresh. Part 1 resources (SelfHealingRemediation Lambda, DevOpsCopilotFunction Lambda, SNS topic, EC2 instance) can remain for other projects.
Delete the EventBridge rule
- Open the Amazon EventBridge console at https://console.aws.amazon.com/events/.
- In the navigation pane, choose Rules.
- Select the button next to WeeklyDigestSchedule and choose Actions, then Delete.
- Confirm by choosing Delete.
Delete the WeeklyDigestFunction Lambda
- Open the Functions page of the Lambda console.
- Select the checkbox next to WeeklyDigestFunction.
- Choose Actions, then Delete.
- Type confirm in the text input field and choose Delete.
Delete the DynamoDB table
- Open the DynamoDB console at https://console.aws.amazon.com/dynamodb/.
- In the left navigation pane, choose Tables.
- Select the checkbox next to DevOpsIncidentLog.
- Choose Delete.
- Confirm the deletion when prompted. This permanently removes the table and all incident records.
Remove the API Gateway resources added in Part 2
- Open the API Gateway console and select your REST API.
- In the Resources panel, select the /incidents resource.
- Choose Actions, then Delete Resource. Confirm the deletion.
- Repeat for the /digest resource.
- Deploy the API to your existing stage so the changes take effect.
Delete the local dashboard file
- Delete dashboard.html from your computer. You can move it to the trash or run:
macOS
rm dashboard.html
Windows
del dashboard.html
Delete the WeeklyDigestFunction execution role (optional)
- Open the IAM console Roles page.
- Search for the role associated with WeeklyDigestFunction.
- Select it, choose Delete, enter the role name to confirm, and choose Delete.
Nice Work!
Nice Work!
Congratulations. You've transformed a reactive Q&A chatbot into a proactive, pattern-aware incident analyst that remembers every remediation, scores severity automatically, and delivers AI-generated operational intelligence straight to your inbox.
You've learned how to:
- Built a DynamoDB incident memory layer with automatic severity scoring (Critical/High/Medium/Low) that gives your copilot long-term recall across days and weeks.
- Created four specialized AI analysis modes powered by Amazon Bedrock: incident summary, root cause analysis, preventive recommendations, and repeat offender detection.
- Built automated weekly digest reports triggered by an EventBridge scheduled rule every Monday and delivered via SNS email, with an expanded REST API exposing /ask, /incidents, and /digest endpoints.
- Secret Mission: Created an enhanced ops console dashboard with a severity-coded incident timeline, mode selector, and on-demand digest generation.
Ready to quiz yourself?