AI-Assisted Lambda Triage on AWS

Investigate and remediate a Lambda incident with CloudWatch AI.

Introduction

30 Second Summary

A broken checkout rarely arrives with a neat explanation. Useful clues are often scattered across separate screens.

In this project, you will use AWS CloudFormation to deploy a checkout-processing AWS Lambda function with Amazon CloudWatch monitoring. You will follow a controlled failure from detection through Amazon CloudWatch investigations to verified recovery.

What You'll Build

Picture showing someone a public incident timeline that runs from a broken checkout to a controlled 400 recovery.

By the end of this project, you'll have:

  • A reproducible incident demo where a malformed checkout event drives ai-devops-lambda-errors into ALARM.
  • An evidence-backed investigation that compares AI-generated hypotheses with validation_failed and ValueError log evidence.
  • A public recovery case study on GitHub that shows the same malformed event returning a controlled 400 after remediation.
  • Secret Mission: Run and save an Amazon CloudWatch Logs Insights query that isolates the validation failures for your incident evidence.

Are there any prerequisites?

You need an active AWS Free plan account with permission to deploy CloudFormation stacks in us-east-1.

Your Windows workspace also needs Visual Studio Code, AWS CLI v2, Python, and GitHub.

Step 1 checks your command-line session before any resources are deployed.

Before We Start

This step helps you commit to building a public AI-assisted incident-response case study before the hands-on work begins. Your finished evidence shows how you detect an incident, investigate its cause, apply a remediation, and verify recovery.

Verify Your AWS Workspace

A wrong AWS account can turn a controlled lab into a cleanup problem. A wrong Region can block the investigation feature you need later.

A predictable local workspace keeps each project artifact in a known place.

This step verifies AWS CLI v2 against your intended Free plan account. Every check explicitly targets us-east-1.

In this step, get ready to:
  • Create a local workspace with separate folders for events, artifacts, and queries.
  • Confirm AWS CLI v2 is available.
  • Verify the intended account without recording credentials or account identifiers.
Create the local workspace

Visual Studio Code treats an opened folder as a workspace. Its subfolders give each artifact type a predictable home.

  • Press the Windows key to open the search bar.
  • Type Visual Studio Code into the search bar.
  • Press Enter to open Visual Studio Code.
  • Click File in the menu bar.
  • Click Open Folder....

You now have the folder picker open. This is where you create the project workspace.

  • Choose your Desktop as the project location.
  • Click New Folder.
  • Enter ai-devops-incident-triage as the folder name.
  • Press Enter to create the folder.

The ai-devops-incident-triage folder now exists on your Desktop.

  • Select the ai-devops-incident-triage folder.
  • Click Select Folder.
  • Select Yes, I trust the authors if the Workspace Trust dialog appears.

You should see ai-devops-incident-triage at the top of the Explorer view.

  • Select ai-devops-incident-triage in the Explorer view.
  • Click New Folder in the Explorer toolbar.
  • Enter events as the folder name.
  • Press Enter to create the folder.

You should see events nested directly inside ai-devops-incident-triage.

  • Select ai-devops-incident-triage in the Explorer view.
  • Click New Folder in the Explorer toolbar.
  • Enter artifacts as the folder name.
  • Press Enter to create the folder.

You should see artifacts at the same level as events.

  • Select ai-devops-incident-triage in the Explorer view.
  • Click New Folder in the Explorer toolbar.
  • Enter queries as the folder name.
  • Press Enter to create the folder.

Your local evidence lanes are ready. Each later project output has a dedicated location.

Explorer now shows events beside artifacts. It also shows queries at the same level.

Confirm AWS CLI v2

AWS CLI v2 provides the command interface used throughout this project. Checking it now prevents later commands from failing because an older major version is installed.

  • Click View in the Visual Studio Code menu bar.
  • Click Terminal.
  • Select PowerShell from the terminal shell dropdown if another shell opens.
  • Check the installed AWS CLI version by running this command:
aws --version

What does this command check?

The command prints the installed AWS CLI release. The major-version prefix tells you whether this terminal is using AWS CLI v2.

You should see output that starts with aws-cli/2. That prefix confirms the required CLI generation is available.

Seeing a different version?

  • Follow the official AWS CLI installation guide to install or update AWS CLI v2 for Windows.
  • Return to the Visual Studio Code terminal after the installation completes.
  • Repeat the version check above.

Help me get AWS CLI v2 working on Windows.

Verify the AWS account and Region

The identity check tells you which account the current credentials represent. It only reads session identity data.

This check can feel sensitive because the response contains account identifiers. Your credential secrets stay hidden.

Before you run this check, which AWS account do you expect your current session to identify?

  • Check your current CLI identity in the target Region by running this command:
aws sts get-caller-identity --region us-east-1

What does this command do?

  • The Account field identifies the AWS account attached to the current credentials.
  • The Arn field identifies the current AWS identity.
  • The --region flag targets us-east-1.
  • The selected Region supports the investigation feature used later in this project.

You should see a JSON response containing Account. The same response includes Arn.

  • Compare the displayed Account value with your intended AWS Free plan account.
  • Keep the returned identifiers out of project files, screenshots, and repository content.

That match is your deployment guardrail. Your current CLI session now points to the intended account for commands targeting us-east-1.

Identity check failed?

  • Reconnect your existing AWS CLI session with the authentication method you normally use.
  • Repeat the identity check after the session is active.
  • Confirm that the returned account belongs to your intended AWS Free plan account.

Help me troubleshoot my AWS CLI identity check without exposing credentials or account identifiers.

Your verified workspace now targets the intended AWS account in us-east-1. Next, you'll deploy the healthy baseline.

Deploy the Healthy Baseline

Your AWS workspace is pointed at the intended account in us-east-1. The next risk is trusting an incident lab before its normal path has ever worked.

CloudFormation turns the workload into repeatable infrastructure as code.

You deploy the Lambda checkout function in temporary unsafe mode with CloudWatch monitoring. A healthy invocation proves the workload works before malformed input exposes the hidden validation flaw.

In this step, get ready to:
  • Create the CloudFormation template plus the JSON event files.
  • Deploy the monitored checkout workload in unsafe validation mode.
  • Invoke a healthy checkout event and inspect its successful response.
Create the baseline files

The template captures every cloud resource required by the incident exercise. Keeping the workload plus its monitoring in one file makes the baseline reproducible.

  • In the Visual Studio Code Explorer sidebar, select the ai-devops-incident-triage folder.
  • Click New File in the Explorer toolbar.
  • Type template.yaml into the file name field.
  • Press Enter to create the file.
  • Define the template parameter plus the log group by pasting this first section into template.yaml:
AWSTemplateFormatVersion: '2010-09-09'
Description: AI-assisted serverless incident triage lab.

Parameters:
  ValidationMode:
    Type: String
    Default: safe
    AllowedValues:
      - unsafe
      - safe
    Description: Controls whether malformed events raise an exception or return a controlled response.

Resources:
  IncidentLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: /aws/lambda/ai-devops-incident-triage
      RetentionInDays: 1

What does this section define?

  • YAML indentation places each property beneath the resource or parameter it configures.
  • ValidationMode accepts the two release modes used during the incident lifecycle.
  • safe remains the default value in the reusable template.
  • IncidentLogGroup creates /aws/lambda/ai-devops-incident-triage with one-day retention.
  • Save template.yaml by pressing Ctrl+S.
  • Confirm the editor shows ValidationMode above IncidentLogGroup.

Does the template structure look uneven?

Check that nested YAML lines use spaces consistently. Make sure Resources: starts at the far left.

Help me check the parameter and log group indentation in my CloudFormation YAML.

The function needs an IAM execution role before Lambda can run it. This role provides the basic permission needed to write application output to CloudWatch Logs.

  • Place your cursor on a blank line beneath RetentionInDays: 1.
  • Add the execution role by pasting this section:
  IncidentFunctionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

What does this role allow?

  • IncidentFunctionRole is the execution role attached to the function.
  • lambda.amazonaws.com identifies Lambda as the service allowed to assume the role.
  • AWSLambdaBasicExecutionRole grants the CloudWatch Logs write permissions used by the function.
  • Save template.yaml.
  • Confirm IncidentFunctionRole is aligned with IncidentLogGroup beneath Resources:.

Is the role nested under the log group?

Move IncidentFunctionRole left until it uses the same two-space indentation as IncidentLogGroup. A deeper indent makes it part of the previous resource.

Help me align IncidentFunctionRole correctly in template.yaml.

The function resource connects its runtime configuration to inline Python code. Its logging configuration sends structured application output to the log group you just defined.

  • Place your cursor on a blank line beneath the managed policy ARN.
  • Add the function configuration plus its logging helper by pasting this section:
  IncidentFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: ai-devops-incident-triage
      Runtime: python3.14
      Handler: index.lambda_handler
      Role: !GetAtt IncidentFunctionRole.Arn
      MemorySize: 128
      Timeout: 5
      Environment:
        Variables:
          VALIDATION_MODE: !Ref ValidationMode
      LoggingConfig:
        ApplicationLogLevel: INFO
        LogFormat: JSON
        LogGroup: !Ref IncidentLogGroup
        SystemLogLevel: INFO
      Code:
        ZipFile: |
          import json
          import os

          def write_log(level, event_name, **fields):
              payload = {"level": level, "event": event_name, **fields}
              print(json.dumps(payload))

How is the function configured?

  • IncidentFunction creates the function named ai-devops-incident-triage.
  • python3.14 supplies the runtime for the inline handler.
  • VALIDATION_MODE receives the stack's ValidationMode parameter.
  • LoggingConfig sends application logs to the dedicated log group in JSON format.
  • write_log() converts each application event into a searchable JSON string.
  • Save template.yaml.
  • Confirm write_log() remains indented beneath ZipFile: |.

Does the Python code align with the imports?

Keep import json plus def write_log at the same indentation inside the YAML block. Keep the helper's body four spaces deeper than its definition.

Help me check the inline Python indentation inside Code.ZipFile.

The handler reads the invocation context plus the current validation mode. Its missing-customer branch produces the controlled failure behavior used later in the incident exercise.

  • Place your cursor on a blank line beneath print(json.dumps(payload)).
  • Add the request setup plus the validation branch by pasting this section:
          def lambda_handler(event, context):
              request_id = context.aws_request_id
              validation_mode = os.environ.get("VALIDATION_MODE", "safe")
              customer_id = event.get("customer_id")

              write_log(
                  "INFO",
                  "request_received",
                  request_id=request_id,
                  validation_mode=validation_mode,
                  has_customer_id=bool(customer_id),
              )

              if not customer_id:
                  write_log(
                      "ERROR",
                      "validation_failed",
                      request_id=request_id,
                      reason="missing_customer_id",
                  )

                  if validation_mode == "unsafe":
                      raise ValueError("customer_id is required")

                  return {
                      "statusCode": 400,
                      "message": "customer_id is required",
                      "request_id": request_id,
                  }

How does validation work?

  • context.aws_request_id gives each invocation an identifier for log correlation.
  • os.environ.get() reads the deployed value of VALIDATION_MODE.
  • event.get() reads customer_id from the invocation payload.
  • validation_failed records structured evidence when that identifier is missing.
  • The missing-customer branch raises an exception in unsafe mode. It returns a controlled 400 response in safe mode.
  • Save template.yaml.
  • Confirm def lambda_handler aligns with def write_log.

Does the handler sit inside the helper?

Align both function definitions at the same indentation beneath ZipFile: |. Keep the statements inside each function four spaces deeper.

Help me separate write_log and lambda_handler in the inline Python code.

A valid customer identifier sends execution down the healthy branch. That branch records the completed checkout before returning the successful response.

  • Place your cursor on a blank line beneath the controlled 400 response.
  • Add the healthy checkout branch by pasting this section:
              write_log(
                  "INFO",
                  "checkout_processed",
                  request_id=request_id,
                  customer_id=customer_id,
              )

              return {
                  "statusCode": 200,
                  "message": "checkout processed",
                  "customer_id": customer_id,
                  "request_id": request_id,
              }

What does the healthy branch return?

  • checkout_processed records that the function accepted the event.
  • statusCode is set to 200 for the successful checkout path.
  • customer_id is copied into the response so you can trace which event succeeded.
  • Save template.yaml.
  • Confirm the healthy branch remains inside lambda_handler().

Is the healthy branch nested under validation?

Align the final write_log() call with if not customer_id:. A deeper indent would keep the success path inside the missing-customer branch.

Help me align the successful checkout branch in lambda_handler.

The alarm turns function errors into a visible operational signal. It evaluates the Lambda Errors metric over one-minute periods.

  • Place your cursor on a blank line beneath the healthy response.
  • Add the error alarm by pasting this final template section:
  IncidentErrorAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: ai-devops-lambda-errors
      AlarmDescription: Detects unhandled errors in the incident-triage Lambda function.
      Namespace: AWS/Lambda
      MetricName: Errors
      Dimensions:
        - Name: FunctionName
          Value: !Ref IncidentFunction
      Statistic: Sum
      Period: 60
      EvaluationPeriods: 1
      DatapointsToAlarm: 1
      Threshold: 1
      ComparisonOperator: GreaterThanOrEqualToThreshold
      TreatMissingData: notBreaching

How will the alarm detect errors?

  • ai-devops-lambda-errors is the alarm name you inspect during the incident.
  • Errors counts invocations that finish with a function error.
  • Sum counts the errors recorded during each 60-second period.
  • GreaterThanOrEqualToThreshold causes one error to satisfy the alarm threshold.
  • notBreaching keeps missing metric data from being treated as an error.
  • Save template.yaml.
  • Confirm IncidentErrorAlarm aligns with the other resources beneath Resources:.

Is the alarm inside the function resource?

Move IncidentErrorAlarm left until it aligns with IncidentFunction. Both entries need two spaces beneath Resources:.

Help me verify that IncidentErrorAlarm is a separate CloudFormation resource.

✔️ Awesome, I've got everything!

That is the largest file complete. Your template now defines the workload plus every monitoring resource required for the incident exercise.

ⓧ I'd like to double check the full code

  • Compare your saved template.yaml with this complete reference:
AWSTemplateFormatVersion: '2010-09-09'
Description: AI-assisted serverless incident triage lab.

Parameters:
  ValidationMode:
    Type: String
    Default: safe
    AllowedValues:
      - unsafe
      - safe
    Description: Controls whether malformed events raise an exception or return a controlled response.

Resources:
  IncidentLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: /aws/lambda/ai-devops-incident-triage
      RetentionInDays: 1

  IncidentFunctionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

  IncidentFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: ai-devops-incident-triage
      Runtime: python3.14
      Handler: index.lambda_handler
      Role: !GetAtt IncidentFunctionRole.Arn
      MemorySize: 128
      Timeout: 5
      Environment:
        Variables:
          VALIDATION_MODE: !Ref ValidationMode
      LoggingConfig:
        ApplicationLogLevel: INFO
        LogFormat: JSON
        LogGroup: !Ref IncidentLogGroup
        SystemLogLevel: INFO
      Code:
        ZipFile: |
          import json
          import os

          def write_log(level, event_name, **fields):
              payload = {"level": level, "event": event_name, **fields}
              print(json.dumps(payload))

          def lambda_handler(event, context):
              request_id = context.aws_request_id
              validation_mode = os.environ.get("VALIDATION_MODE", "safe")
              customer_id = event.get("customer_id")

              write_log(
                  "INFO",
                  "request_received",
                  request_id=request_id,
                  validation_mode=validation_mode,
                  has_customer_id=bool(customer_id),
              )

              if not customer_id:
                  write_log(
                      "ERROR",
                      "validation_failed",
                      request_id=request_id,
                      reason="missing_customer_id",
                  )

                  if validation_mode == "unsafe":
                      raise ValueError("customer_id is required")

                  return {
                      "statusCode": 400,
                      "message": "customer_id is required",
                      "request_id": request_id,
                  }

              write_log(
                  "INFO",
                  "checkout_processed",
                  request_id=request_id,
                  customer_id=customer_id,
              )

              return {
                  "statusCode": 200,
                  "message": "checkout processed",
                  "customer_id": customer_id,
                  "request_id": request_id,
              }

  IncidentErrorAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: ai-devops-lambda-errors
      AlarmDescription: Detects unhandled errors in the incident-triage Lambda function.
      Namespace: AWS/Lambda
      MetricName: Errors
      Dimensions:
        - Name: FunctionName
          Value: !Ref IncidentFunction
      Statistic: Sum
      Period: 60
      EvaluationPeriods: 1
      DatapointsToAlarm: 1
      Threshold: 1
      ComparisonOperator: GreaterThanOrEqualToThreshold
      TreatMissingData: notBreaching

How to compare the template

Check the resource order plus indentation against this reference. The final line should remain inside the alarm's Properties section.

The event files keep the successful request separate from the malformed request used later. Reusing fixed inputs makes the healthy baseline plus the incident reproducible.

  • Select the existing events folder in the Explorer sidebar.
  • Click New File in the Explorer toolbar.
  • Type healthy.json into the file name field.
  • Press Enter to create the file.
  • Define the healthy checkout event by pasting this content:
{
  "action": "checkout",
  "customer_id": "CUST-1001"
}

What makes this event healthy?

The event includes the required customer_id field. Its value identifies this repeatable test request as CUST-1001.

  • Save events/healthy.json.
  • Confirm the file appears beneath the events folder in the Explorer sidebar.

Is healthy.json outside the events folder?

Move healthy.json into the existing events folder. The invocation command depends on the exact path events/healthy.json.

Help me place healthy.json in the correct Visual Studio Code folder.

  • Select the events folder again.
  • Click New File in the Explorer toolbar.
  • Type failure.json into the file name field.
  • Press Enter to create the file.
  • Define the malformed checkout event by pasting this content:
{
  "action": "checkout"
}

What makes this event malformed?

The event still requests the checkout action. Its missing customer_id field creates the controlled incident input used in the next step.

  • Save events/failure.json.
  • Confirm both JSON files appear beneath the events folder.

Does the JSON show a syntax warning?

Check that each property name plus string value uses double quotes. Remove any trailing comma after "checkout" in failure.json.

Help me compare my two event files with valid JSON syntax.

✔️ Awesome, I've got everything!

You now have one event for the healthy path plus one fixed event for the incident path. Both inputs are ready for repeatable invocations.

ⓧ I'd like to double check the full code

  • Compare events/healthy.json with this complete reference:
{
  "action": "checkout",
  "customer_id": "CUST-1001"
}

Healthy event checkpoint

This file contains the checkout action plus the required customer identifier. Keep both property names exactly as shown.

  • Compare events/failure.json with this complete reference:
{
  "action": "checkout"
}

Failure event checkpoint

This file deliberately contains only the checkout action. Keep the malformed event unchanged throughout the incident plus remediation checks.

Deploy the monitored workload

Deploying cloud resources can feel risky because they have a billing surface. This lab's intended usage fits within the verified AWS Free Tier allowances.

Related telemetry or API usage beyond applicable allowances can incur charges. The first deployment overrides the template default so the stack runs in unsafe mode.

CloudFormation needs time to create the role plus the dependent resources. Expect the terminal to remain busy for several minutes while the stack deployment completes.

  • Switch back to the Visual Studio Code PowerShell terminal from the previous step.
  • Deploy the stack in temporary unsafe mode by running this command:
aws cloudformation deploy --template-file template.yaml --stack-name ai-devops-incident-triage --parameter-overrides ValidationMode=unsafe --capabilities CAPABILITY_IAM --region us-east-1

What does this deployment command do?

  • --template-file template.yaml sends your saved infrastructure definition to CloudFormation.
  • --stack-name ai-devops-incident-triage gives the deployed stack its fixed project name.
  • --parameter-overrides ValidationMode=unsafe replaces the template's safe default for this first release.
  • --capabilities CAPABILITY_IAM acknowledges that the template creates an IAM role.
  • --region us-east-1 keeps the workload in the Region verified during workspace setup.
  • Confirm the terminal reports a successful stack deployment before continuing.

Did the stack deployment fail?

  • Confirm template.yaml is saved inside ai-devops-incident-triage.
  • Check the reported template line for misplaced YAML indentation.
  • Verify your current AWS identity has permission to create CloudFormation stacks plus the listed IAM resources.

Help me diagnose my CloudFormation deployment failure without exposing account identifiers.

Invoke the healthy checkout

A deployed stack only establishes that the resources exist. The healthy event now tests the function's request path plus its response contract.

Before you invoke the function, do you expect the required customer identifier to reach the success branch?

  • Invoke the deployed function with the healthy event by running this command:
aws lambda invoke --function-name ai-devops-incident-triage --payload fileb://events/healthy.json --region us-east-1 artifacts/healthy-response.json

What does this invocation test?

  • --function-name ai-devops-incident-triage targets the function created by your stack.
  • --payload fileb://events/healthy.json sends the event file as the invocation payload.
  • artifacts/healthy-response.json stores the function's response inside your existing evidence folder.

The terminal response contains no FunctionError. That confirms the invocation completed without an unhandled exception.

  • Expand the artifacts folder in the Visual Studio Code Explorer sidebar.
  • Select healthy-response.json to inspect the saved payload.

You'll see "statusCode": 200 plus "customer_id": "CUST-1001" in the response. Your baseline now proves that valid checkout events reach the successful path.

Missing the healthy response?

  • Confirm the terminal is operating from the ai-devops-incident-triage folder.
  • Check that events/healthy.json includes "customer_id": "CUST-1001".
  • Refresh the Explorer sidebar if the generated response file is not listed.

Help me troubleshoot the healthy Lambda invocation without sharing account details.

Your monitored checkout workload is live in unsafe mode. Next, you'll send the fixed malformed event until the hidden validation flaw produces logs plus an alarm signal.

Trigger and Observe the Incident

Your healthy checkout proved that the AWS Lambda workload handles one well-formed event. Production workloads also receive malformed events that can expose hidden failure paths.

This step sends the unchanged malformed event through the unsafe validation path. You will trace the failure through direct logs before confirming that the CloudWatch alarm detects it.

In this step, get ready to:
  • Invoke the malformed event three times to capture separate failure artifacts.
  • Correlate the structured application entry with the runtime error.
  • Confirm that the error alarm enters ALARM.
Test the unsafe validation path

Controlled failure injection uses known bad input to expose how a system fails under repeatable conditions. The existing events/failure.json file omits customer_id so every invocation tests the same condition.

  • Return to the Visual Studio Code terminal from earlier.

Before you run these invocations, pause to predict whether the healthy result survives when customer_id is missing.

  • Send the malformed payload three times by running these commands:
aws lambda invoke --function-name ai-devops-incident-triage --payload fileb://events/failure.json --region us-east-1 artifacts/failure-1.json
aws lambda invoke --function-name ai-devops-incident-triage --payload fileb://events/failure.json --region us-east-1 artifacts/failure-2.json
aws lambda invoke --function-name ai-devops-incident-triage --payload fileb://events/failure.json --region us-east-1 artifacts/failure-3.json

What Do These Invocations Prove?

Each command sends the same malformed event to the deployed function. Each output path preserves a separate response payload.

Unsafe validation raises a ValueError before the handler can return a controlled response. The CLI therefore includes FunctionError in every invocation response.

  • Confirm that every terminal response includes FunctionError.
  • Expand the artifacts folder in the Visual Studio Code Explorer sidebar.
  • Open failure-1.json from the artifacts folder.

You should see a failed invocation payload describing the missing customer identifier. The three artifact files now preserve repeatable evidence from the unsafe path.

Missing FunctionError?

  • Confirm that the stack is still deployed with ValidationMode=unsafe.
  • Confirm that events/failure.json omits customer_id.
  • Confirm that the command targets ai-devops-incident-triage in us-east-1.

Help me diagnose why my malformed Lambda invocation does not include FunctionError.

Correlate the application and runtime evidence

Structured JSON logs record the application's view of a failure. Amazon CloudWatch Logs also captures the runtime exception that turns the same invocation into an error metric.

Before you tail the logs, pause to predict which application event should appear beside the runtime exception.

  • Read the most recent function logs by running this command:
aws logs tail /aws/lambda/ai-devops-incident-triage --since 5m --format short --region us-east-1

How Does the Evidence Connect?

  • The validation_failed entry comes from write_log() after the handler detects the missing field.
  • The ValueError entry shows that unsafe validation allowed the problem to escape as an unhandled exception.
  • The shared invocation path connects the application signal to the runtime failure.

You should see validation_failed application entries near runtime records containing ValueError. This correlation shows where the failure began before the platform counted it as an error.

Logs Missing the Failure?

  • Repeat one malformed invocation from the previous substep.
  • Repeat the log-tail command within the 5m window.
  • Confirm that the log group name is /aws/lambda/ai-devops-incident-triage.

Help me find the missing validation failure in CloudWatch Logs.

Confirm the alarm detects the incident

The Lambda Errors metric counts invocations that end with function errors. The alarm evaluates that metric to turn repeated failures into a visible incident signal.

The state can take more than one 60-second evaluation period to update. That delay is expected while CloudWatch assesses the new metric data.

Before you open the alarm, pause to predict whether three unhandled failures are enough to cross its threshold.

  • Sign in to the AWS Management Console in your browser.
  • Set the console Region to US East (N. Virginia).
  • Enter CloudWatch in the console search field.
  • Select CloudWatch from the search results.
  • Select Alarms from the left navigation pane.
  • Select ai-devops-lambda-errors from the alarm list.
  • Watch the alarm state until it changes to ALARM.

That is your detection pipeline working. The alarm has turned three controlled failures into a visible incident signal.

Alarm Has Not Changed?

  • Allow another 60-second evaluation period for the alarm to process the metric.
  • Confirm that the console Region is US East (N. Virginia).
  • Confirm that all three CLI responses included FunctionError.

Help me diagnose why the CloudWatch alarm has not entered ALARM.

Your controlled incident now has failed responses, correlated logs, and an alarm signal. Next, you will use that telemetry to guide an AI-assisted investigation.

Run an AI-Assisted Investigation

Your Lambda alarm now proves that the checkout workload is failing. Manually switching between metrics and logs slows incident triage.

In this step, you'll use Amazon CloudWatch investigations to examine the alarm telemetry. You'll test its AI-generated findings against the direct evidence from your controlled incident.

In this step, get ready to:
  • Start a CloudWatch investigation from the alarm.
  • Compare the AI-generated findings with the direct log evidence.
  • Capture a redacted screenshot of the investigation.
Start the investigation from the alarm

CloudWatch investigations uses the permissions from your current console session. The default investigation path creates no additional AWS resources.

Why CloudWatch investigations?

CloudWatch investigations is built for operational incident response. It scans related telemetry to surface observations or hypotheses.

A custom model integration would add infrastructure before you could investigate the failure. This approach keeps your attention on evaluating incident evidence.

  • In the CloudWatch alarm page from earlier, confirm the Region selector shows US East (N. Virginia).
  • Confirm the ai-devops-lambda-errors alarm remains in ALARM.
  • Choose Investigate to start the investigation.

You now have an investigation running from the same alarm that detected your controlled incident. The investigation view includes the selected alarm while CloudWatch scans related telemetry.

Investigation not starting?

Confirm that the console still uses US East (N. Virginia). Check that your signed-in identity can read the alarm plus its related telemetry.

Get help checking the current console permissions.

Compare the findings with direct evidence

AI-generated findings are investigative leads. Your direct CloudWatch Logs evidence remains the source of truth for the root cause.

  • Review the metrics surfaced for the ai-devops-lambda-errors alarm.
  • Review any surfaced logs for the validation_failed application event.
  • Compare each AI-generated observation or hypothesis with the runtime ValueError evidence.
  • Classify each finding as supported by the logs.
  • Identify any finding that adds useful context.
  • Identify any finding that lacks enough evidence.

Your comparison now separates direct incident evidence from AI-generated leads. That evidence discipline keeps the final root-cause statement accurate.

Capture redacted investigation evidence

A screenshot makes the investigation evidence reusable in your case study. Console chrome can expose account identifiers.

This deserves a careful privacy check before the image leaves your computer. Keep the alarm evidence visible while removing identifying details.

  • Arrange the investigation view so the selected alarm is visible.
  • Expand one AI-generated observation or hypothesis that you evaluated.
  • Capture the investigation view with your preferred Windows screenshot tool.
  • Crop the screenshot to the investigation content.
  • Obscure any visible account identifiers.
  • Check the screenshot for credentials or secrets.
  • Save the redacted screenshot in a location you can find during documentation.

Before your final check, which parts of the investigation should match the evidence from the controlled failure?

  • Confirm the investigation view includes the ai-devops-lambda-errors alarm.
  • Confirm the view includes an AI-generated telemetry observation or hypothesis.
  • Compare the final claim with the validation_failed evidence.
  • Compare the final claim with the ValueError evidence.

You should see the selected alarm plus AI-generated investigation findings that you can test against the direct logs. Your redacted screenshot now preserves that evidence without exposing account details.

Your investigation evidence is ready. Next, you'll remediate the failure before publishing the case study.

Remediate and Publish the Evidence

Your Amazon CloudWatch investigations session connected the alarm to the unhandled customer validation path.

The deployed AWS Lambda function still uses unsafe validation. The same AWS CloudFormation stack can switch the release parameter without changing the malformed event.

A recovery claim needs reproducible evidence. You will verify the fix before publishing the redacted case study on GitHub.

In this step, get ready to:
  • Redeploy the existing stack with safe validation.
  • Invoke the same malformed event to verify a controlled response.
  • Publish the redacted incident evidence in a public GitHub repository.
Redeploy in safe validation mode

The handler already contains a controlled validation path. The stack parameter selects that path for the deployed function.

  • Switch back to the PowerShell terminal in Visual Studio Code.
  • Update the existing stack with safe validation by running:
aws cloudformation deploy --template-file template.yaml --stack-name ai-devops-incident-triage --parameter-overrides ValidationMode=safe --capabilities CAPABILITY_IAM --region us-east-1

What does this command change?

  • The ValidationMode=safe override updates the deployed release mode.
  • The CAPABILITY_IAM value acknowledges the existing IAM resource in the template.
  • The us-east-1 value keeps the update in the Region used throughout the incident.

This stack update can take a few minutes while CloudFormation checks each resource. The terminal reports a successful deployment when safe validation is active.

Good progress. The deployed function now uses the controlled validation path.

Stack update failed?

Confirm the terminal still uses the authenticated AWS CLI session from earlier. Check that the command targets us-east-1.

Make sure your account can update the stack resources.

Help me diagnose my CloudFormation stack update.

Verify the controlled recovery

The malformed event stays unchanged for this test. That isolates the deployment parameter as the only remediation variable.

Before you rerun the event, ask yourself whether safe mode returns a response or throws another exception.

  • Invoke the remediated function by running:
aws lambda invoke --function-name ai-devops-incident-triage --payload fileb://events/failure.json --region us-east-1 artifacts/remediated-response.json

What does this invocation prove?

  • The fileb://events/failure.json payload reuses the malformed event from the incident.
  • The command saves the handler response in artifacts/remediated-response.json.
  • A CLI response without FunctionError confirms that the handler completed normally.

The CLI response has no FunctionError. The same malformed event now completes without an unhandled exception.

  • Select artifacts/remediated-response.json in the Visual Studio Code Explorer sidebar.

You should see "statusCode": 400 with the controlled customer_id is required message.

  • Check the latest function logs by running:
aws logs tail /aws/lambda/ai-devops-incident-triage --since 5m --format short --region us-east-1

How do the logs confirm recovery?

Safe validation still records validation_failed because the event is malformed. The handler now returns the controlled response after writing that entry.

Focus on entries created after the remediated invocation. There should be no later ValueError for that request.

Still seeing a new exception?

Compare the newest log timestamp with the remediated invocation. Older ValueError entries belong to the original incident.

If the newest request still raises an exception, confirm the stack deployment completed before the invocation.

Help me compare my remediated Lambda response with the latest logs.

Alarm recovery can lag behind the fixed invocation because the alarm evaluates one-minute periods. A short wait here is expected.

  • Switch back to the CloudWatch console from earlier.
  • Select the ai-devops-lambda-errors alarm.
  • Wait for the alarm state to return to OK.

Recovery is now proven. The malformed event returns a controlled client-error response without creating another Lambda function error.

Document and publish the incident

An incident case study needs enough context for another operator to reproduce the failure. It also needs direct evidence that separates verified facts from AI-generated leads.

  • Create README.md inside ai-devops-incident-triage with the new-file control in the Visual Studio Code Explorer sidebar.
  • Add the project overview and architecture by pasting:
# AI-Assisted Serverless Incident Triage on AWS

This project demonstrates a controlled Lambda failure, CloudWatch detection, AI-assisted investigation, safe remediation, and recovery verification.

## Enterprise problem

Operations teams lose time when malformed application events fail without a repeatable detection and investigation path. This lab creates that path as a small, reproducible case study.

## Architecture

```text
Test event
    |
    v
AWS Lambda Event Function
    |                \
    v                 v
CloudWatch Logs   Lambda Errors metric
                          |
                          v
                  CloudWatch alarm
                          |
                          v
               CloudWatch investigation
```

What does this section explain?

  • The opening gives readers the project outcome before they inspect the implementation.
  • The enterprise problem connects malformed events to repeatable incident response.
  • The Markdown diagram traces the signal from the test event to the investigation.
  • Save README.md.
  • Confirm the editor shows the architecture from the test event through the CloudWatch investigation.

Diagram missing a connection?

Check that the architecture begins with ```text and ends with a matching closing fence. Keep the backslash after the Lambda function line.

Help me compare the architecture section in my README.

  • Place your cursor after the architecture code fence.
  • Add the incident lifecycle and deployment commands by pasting:
## Incident lifecycle

1. Deploy the CloudFormation stack with `ValidationMode=unsafe`.
2. Prove the healthy checkout event works.
3. Send the malformed event three times.
4. Correlate the `validation_failed` log, `ValueError`, and alarm.
5. Use CloudWatch investigations to review observations or hypotheses.
6. Redeploy with `ValidationMode=safe`.
7. Re-run the same malformed event and verify a controlled `400` response.

## Deployment

```text
aws cloudformation deploy --template-file template.yaml --stack-name ai-devops-incident-triage --parameter-overrides ValidationMode=unsafe --capabilities CAPABILITY_IAM --region us-east-1
```

## Remediation

```text
aws cloudformation deploy --template-file template.yaml --stack-name ai-devops-incident-triage --parameter-overrides ValidationMode=safe --capabilities CAPABILITY_IAM --region us-east-1
```

What does the lifecycle preserve?

  • The numbered sequence records the incident from baseline deployment through recovery.
  • The two deployment commands make the unsafe release and its remediation reproducible.
  • The unchanged malformed event keeps the validation mode as the controlled variable.
  • Save README.md.
  • Confirm the editor shows seven lifecycle stages followed by the two deployment commands.

Lifecycle section misplaced?

Make sure this section starts after the architecture fence. An unclosed fence can make the remaining headings appear as plain text.

Help me repair the lifecycle formatting in my README.

  • Place your cursor after the remediation code fence.
  • Add the evidence and cleanup guidance by pasting:
## Evidence

See `incident-report.md` for the detection signal, direct logs, AI-assisted findings, remediation, and recovery checks. Redact account identifiers and credentials from every screenshot.

## Cost and cleanup

The lab is designed for AWS Free Tier-scale usage. Related API and telemetry usage can incur charges beyond applicable allowances.

```text
aws cloudformation delete-stack --stack-name ai-devops-incident-triage --region us-east-1
aws cloudformation wait stack-delete-complete --stack-name ai-devops-incident-triage --region us-east-1
```

Why include evidence and cleanup?

The evidence section directs readers to the detailed incident record. The redaction reminder protects account identifiers.

The cleanup section records the cost boundary. It also gives readers a complete teardown path.

  • Save README.md.
  • Confirm the final headings are Evidence and Cost and cleanup.

Cleanup text missing?

Check that the evidence paragraph ends before the cost heading. Confirm both cleanup commands sit inside one text fence.

Help me check the final README sections.

✔️ Awesome, I've got everything!

Your README.md now explains the architecture, incident lifecycle, remediation, evidence boundary, cost boundary, and cleanup path.

ⓧ I'd like to double check the full code

  • Compare your complete README.md with this reference:
# AI-Assisted Serverless Incident Triage on AWS

This project demonstrates a controlled Lambda failure, CloudWatch detection, AI-assisted investigation, safe remediation, and recovery verification.

## Enterprise problem

Operations teams lose time when malformed application events fail without a repeatable detection and investigation path. This lab creates that path as a small, reproducible case study.

## Architecture

```text
Test event
    |
    v
AWS Lambda Event Function
    |                \
    v                 v
CloudWatch Logs   Lambda Errors metric
                          |
                          v
                  CloudWatch alarm
                          |
                          v
               CloudWatch investigation
```

## Incident lifecycle

1. Deploy the CloudFormation stack with `ValidationMode=unsafe`.
2. Prove the healthy checkout event works.
3. Send the malformed event three times.
4. Correlate the `validation_failed` log, `ValueError`, and alarm.
5. Use CloudWatch investigations to review observations or hypotheses.
6. Redeploy with `ValidationMode=safe`.
7. Re-run the same malformed event and verify a controlled `400` response.

## Deployment

```text
aws cloudformation deploy --template-file template.yaml --stack-name ai-devops-incident-triage --parameter-overrides ValidationMode=unsafe --capabilities CAPABILITY_IAM --region us-east-1
```

## Remediation

```text
aws cloudformation deploy --template-file template.yaml --stack-name ai-devops-incident-triage --parameter-overrides ValidationMode=safe --capabilities CAPABILITY_IAM --region us-east-1
```

## Evidence

See `incident-report.md` for the detection signal, direct logs, AI-assisted findings, remediation, and recovery checks. Redact account identifiers and credentials from every screenshot.

## Cost and cleanup

The lab is designed for AWS Free Tier-scale usage. Related API and telemetry usage can incur charges beyond applicable allowances.

```text
aws cloudformation delete-stack --stack-name ai-devops-incident-triage --region us-east-1
aws cloudformation wait stack-delete-complete --stack-name ai-devops-incident-triage --region us-east-1
```

What does the complete file prove?

This reference combines the project narrative with the exact deployment and cleanup paths. It gives repository visitors one reproducible starting point.

The incident report separates direct telemetry from AI-assisted interpretation. That distinction keeps the root-cause claim tied to observable evidence.

  • Create incident-report.md inside ai-devops-incident-triage with the new-file control in the Visual Studio Code Explorer sidebar.
  • Add the summary and detection evidence by pasting:
# Incident Report: Missing Customer Identifier

## Summary

A controlled checkout event omitted `customer_id`. With `ValidationMode=unsafe`, the Lambda handler raised `ValueError`, produced an `Errors` metric, and caused the `ai-devops-lambda-errors` alarm to enter `ALARM`.

## Scope and impact

- Environment: Personal AWS learning account in `us-east-1`
- Workload: `ai-devops-incident-triage`
- Impact: Test invocations failed. No production traffic or customer data was used.

## Detection evidence

- Alarm: `ai-devops-lambda-errors`
- Application log event: `validation_failed`
- Runtime error: `ValueError: customer_id is required`
- Incident timestamp: Add the UTC timestamp from your alarm or logs.
- Evidence image: Add a redacted alarm or log screenshot.

## AI-assisted findings

Record the observation, suggestion, or hypothesis produced by CloudWatch investigations. State whether direct logs support it, and note any uncertainty instead of presenting AI output as fact.

How is the evidence organized?

  • The summary names the malformed input and its observable effect.
  • The scope states that the incident used test invocations without customer data.
  • The detection section records the alarm, application event, runtime error, timestamp, and redacted image.
  • The AI-assisted section requires comparison with direct logs before accepting a finding.
  • Save incident-report.md.
  • Confirm the editor shows separate sections for detection evidence and AI-assisted findings.

Evidence sections out of order?

Keep the detection evidence above the AI-assisted findings. This order establishes the direct telemetry before the interpretation.

Help me check the evidence structure in my incident report.

  • Place your cursor after the AI-assisted findings paragraph.
  • Add the root cause and recovery record by pasting:
## Root cause

The malformed event did not contain `customer_id`. Unsafe validation mode raised an exception instead of returning a controlled client-error response.

## Remediation

The same CloudFormation stack was redeployed with `ValidationMode=safe`. The handler then returned `statusCode: 400` for the unchanged malformed event without raising another exception.

## Recovery validation

- The remediated invocation had no CLI `FunctionError`.
- `artifacts/remediated-response.json` contained a controlled `400` response.
- No new unhandled exception appeared in the tailed logs.
- The CloudWatch alarm returned to `OK` after evaluation.

## Preventive actions

- Validate required fields before business logic runs.
- Add contract tests for malformed events.
- Keep validation mode safe by default.
- Review deployment parameter overrides during release approval.

What does this recovery record prove?

  • The root cause identifies the missing required field.
  • The remediation records the safe release parameter.
  • The recovery checklist connects the controlled response to clean logs and alarm recovery.
  • The preventive actions turn the incident into reusable engineering guidance.
  • Save incident-report.md.
  • Confirm the recovery section lists the response file, clean logs, and alarm state.

Recovery checklist incomplete?

Confirm the checklist includes the CLI result, response file, log result, and alarm result. Each item records a different recovery signal.

Help me verify the recovery section in my incident report.

✔️ Awesome, I've got everything!

Your incident-report.md now records the incident from detection through verified recovery.

ⓧ I'd like to double check the full code

  • Compare your complete incident-report.md with this reference:
# Incident Report: Missing Customer Identifier

## Summary

A controlled checkout event omitted `customer_id`. With `ValidationMode=unsafe`, the Lambda handler raised `ValueError`, produced an `Errors` metric, and caused the `ai-devops-lambda-errors` alarm to enter `ALARM`.

## Scope and impact

- Environment: Personal AWS learning account in `us-east-1`
- Workload: `ai-devops-incident-triage`
- Impact: Test invocations failed. No production traffic or customer data was used.

## Detection evidence

- Alarm: `ai-devops-lambda-errors`
- Application log event: `validation_failed`
- Runtime error: `ValueError: customer_id is required`
- Incident timestamp: Add the UTC timestamp from your alarm or logs.
- Evidence image: Add a redacted alarm or log screenshot.

## AI-assisted findings

Record the observation, suggestion, or hypothesis produced by CloudWatch investigations. State whether direct logs support it, and note any uncertainty instead of presenting AI output as fact.

## Root cause

The malformed event did not contain `customer_id`. Unsafe validation mode raised an exception instead of returning a controlled client-error response.

## Remediation

The same CloudFormation stack was redeployed with `ValidationMode=safe`. The handler then returned `statusCode: 400` for the unchanged malformed event without raising another exception.

## Recovery validation

- The remediated invocation had no CLI `FunctionError`.
- `artifacts/remediated-response.json` contained a controlled `400` response.
- No new unhandled exception appeared in the tailed logs.
- The CloudWatch alarm returned to `OK` after evaluation.

## Preventive actions

- Validate required fields before business logic runs.
- Add contract tests for malformed events.
- Keep validation mode safe by default.
- Review deployment parameter overrides during release approval.

What does the complete report preserve?

This reference keeps the direct incident signals beside the remediation and recovery checks. It also preserves the boundary between evidence and AI-assisted interpretation.

Publishing makes the evidence visible to anyone. Your investigation image must stay redacted so the case study shows the incident without exposing account identifiers.

  • Open GitHub in your browser.
  • Sign in to your GitHub account if prompted.
  • Select New repository.
  • Enter ai-devops-incident-triage as the repository name.
  • Select Public as the repository visibility.
  • Click Create repository.

You now have an empty public repository ready to hold the case study.

  • Click Add file on the repository page.
  • Click Upload files.
  • Use choose your files to select the project source and documentation.
  • Include the redacted investigation screenshot from earlier.
  • Enter Publish incident triage evidence as the commit message.
  • Select the option that commits the files to the default branch.
  • Click Propose changes.

Upload missing a project file?

Confirm the upload includes template.yaml, README.md, incident-report.md, and the events folder. Include only redacted visual evidence.

Help me check my GitHub incident-response repository before publishing.

Before you refresh the repository, ask yourself which evidence would convince another operator that the incident was detected and recovered.

  • Refresh the public repository page.
  • Check the repository visibility label.
  • Select README.md in the repository file list.

The visibility is public. The rendered README explains the architecture, incident lifecycle, remediation, evidence boundary, cost boundary, and cleanup path.

Your incident-response case study is live. It now demonstrates detection, AI-assisted investigation, evidence-based remediation, and verified recovery.

Secret mission

Query the Incident Evidence

Turn the historical validation failures into evidence that another operator can retrieve. You will create a reusable CloudWatch Logs Insights query, run it over the incident window, and publish the result with your case study.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to keep your resources available, remove only the AWS stack for a later return, or delete the project resources entirely. This project's intended use fits within AWS Free Tier allowances.

Related telemetry or API usage beyond those allowances can incur charges.

Cost warning

The stack is currently deployed with ValidationMode=safe. Its one-day log retention and standard alarm remain active until you delete the stack.

CloudWatch investigations can make telemetry and resource API calls. AWS can bill usage that exceeds applicable Free Tier limits, so choose Pause or Delete if you have finished the lab.

Resources you used:

  • AWS CloudFormation stack ai-devops-incident-triage in us-east-1.
  • AWS Lambda Event Function named ai-devops-incident-triage.
  • Amazon CloudWatch log group /aws/lambda/ai-devops-incident-triage.
  • IAM execution role represented by IncidentFunctionRole in template.yaml.
  • CloudWatch alarm ai-devops-lambda-errors.
  • Local ai-devops-incident-triage workspace open in Visual Studio Code.
  • Public GitHub repository named ai-devops-incident-triage.

Keep everything running

No action needed. Choose this if you still want to demonstrate the live incident-response workflow.

  • Leave the stack deployed with ValidationMode=safe.
  • Keep the local ai-devops-incident-triage workspace for future updates.
  • Keep the public repository available as your incident-response case study.
  • Avoid extra test invocations unless you need fresh evidence.

Pause - I'll come back to this later

Remove the AWS stack to stop retaining its cloud resources. Your local workspace and public repository remain available for a later redeployment.

Deleting the stack is permanent, but your saved template.yaml provides a reproducible rebuild path.

  • Delete the AWS stack by running these commands in the Visual Studio Code terminal:
aws cloudformation delete-stack --stack-name ai-devops-incident-triage --region us-east-1
aws cloudformation wait stack-delete-complete --stack-name ai-devops-incident-triage --region us-east-1

What do these commands do?

  • The first command starts deleting the ai-devops-incident-triage stack.
  • The second command waits until the stack reaches DELETE_COMPLETE.
  • The terminal can stay quiet while AWS removes the resources.

Once the waiter returns, the Lambda function, log group, execution role, and alarm have been removed with the stack. Your project evidence remains available locally and on GitHub.

Delete - I don't want to use this again

Remove the deployed AWS resources, the public repository, and the local workspace. Your redacted screenshots need a separate copy if you want to keep them.

Deleting these resources is permanent. Make sure you have saved anything you want to revisit before continuing.

Delete the AWS resources:

  • Delete the AWS stack by running these commands in the Visual Studio Code terminal:
aws cloudformation delete-stack --stack-name ai-devops-incident-triage --region us-east-1
aws cloudformation wait stack-delete-complete --stack-name ai-devops-incident-triage --region us-east-1

What do these commands do?

  • The first command starts deleting every deployed resource managed by the stack.
  • The second command waits until the stack reaches DELETE_COMPLETE.
  • When the waiter returns, the stack and its managed resources are gone.

Delete the public repository:

  • Return to the public ai-devops-incident-triage repository page from earlier.
  • Open the repository settings.
  • Find the permanent repository deletion control.
  • Start the repository deletion.
  • Enter ai-devops-incident-triage when GitHub asks for confirmation.
  • Confirm the repository deletion.
  • Verify that the old public repository URL no longer opens the project.

Delete the local workspace:

  • Return to the open ai-devops-incident-triage workspace in Visual Studio Code.
  • Close the workspace.
  • Locate the ai-devops-incident-triage folder in Windows File Explorer at the location where you created it.
  • Delete the ai-devops-incident-triage folder.
  • Confirm that the folder no longer appears in that location.

Nice Work!

Nice Work!

You did it! Your public incident-response case study now proves how a controlled AWS Lambda failure moves from detection to evidence-based recovery.

You've learned how to:

  • Built a reproducible AWS CloudFormation stack that deploys a monitored checkout workload on AWS Lambda. Proved the healthy path returned 200 before injecting a controlled malformed event.
  • Diagnosed a controlled validation incident by correlating validation_failed, ValueError, the Lambda Errors metric, and the ai-devops-lambda-errors alarm. Evaluated Amazon CloudWatch investigations output against direct log evidence before recording the root cause.
  • Remediated the workload with ValidationMode=safe. Proved the same malformed event returned a controlled 400 response without FunctionError. Published the redacted architecture, incident evidence, and recovery path in a public GitHub repository.
  • Secret Mission: Added a reusable CloudWatch Logs Insights query that isolates validation_failed events for repeatable incident evidence.

Ready to quiz yourself?