Deploy an Incident-Ready AWS Service

Deploy a monitored Lambda service, diagnose failures, and repair it with IaC.

Introduction

30 Second Summary

A service can look healthy until one bad request exposes a hidden problem. Without a clear signal or response plan, finding the cause becomes guesswork.

In this project, you will deploy a small Python service to AWS Lambda with AWS CloudFormation. You will guide that service through a complete incident response cycle using Amazon CloudWatch, Infrastructure as Code, and an operations runbook.

What You'll Build

Your finished lab turns a Lambda request into a visible incident-response story from healthy service to verified recovery.

By the end of this project, you'll have:

  • A repeatable health check that invokes your deployed Lambda function. It returns a JSON health response with status code 200.
  • An incident investigation that traces invalid input from the failed response into CloudWatch Logs. You can show the CloudWatch alarm moving to ALARM.
  • A verified recovery where invalid input returns a controlled 400 response. You can show the alarm returning to OK. Your OPERATIONS.md runbook captures triage, mitigation, verification, rollback, and root cause.
  • Secret Mission: Create out-of-band configuration drift in the Lambda timeout. Detect the drift with CloudFormation. Reconcile the function to the template value.

Are there any prerequisites?

You need an activated AWS account plus a non-root AWS Identity and Access Management (IAM) identity with approval to create and delete the resources in this lab. Account activation can take up to 24 hours, so a long wait does not mean your signup failed.

The browser-based AWS CloudShell path removes the need for a local command-line installation.

Before We Start

This checkpoint connects the monitored AWS service ahead to the incident diagnosis skills expected in a CloudOps role. That purpose gives the later hands-on work a practical goal.

Set Up AWS CloudShell

This operations lab needs a shell that can call AWS with the correct identity. Local installations or credentials would add setup differences before you reach the incident work.

An authenticated AWS CloudShell session gives you a regional Bash environment with AWS CLI version 2 already available. You will leave this step with an authenticated empty workspace in one Region.

In this step, get ready to:
  • Sign in with the approved non-root IAM identity.
  • Launch a regional Bash session with AWS CLI version 2.
  • Prepare an empty workspace under the verified caller identity.
Sign in with the approved identity

The approved non-root AWS Identity and Access Management identity has the sandbox permissions needed later in the lab. Using that identity keeps everyday work separate from the root user.

  • In your browser, go to the AWS Management Console sign-in page.
  • Enter the approved non-root IAM identity details provided for this lab.
  • Complete any authentication check required by your account.

You should reach the AWS Management Console home page under the approved identity. This confirms that the account is ready for the lab.

Launch a regional Bash session

An AWS Region is the geographic location where your AWS resources run. CloudShell files are also regional, so keeping one Region selected prevents your workspace from appearing to disappear.

  • Use the Region selector in the console toolbar to choose one CloudShell-supported AWS Region.
  • Keep this Region selected for the rest of the project.
  • Select AWS CloudShell from the console toolbar.

The first launch can take a moment while AWS prepares the environment. You will see a Bash prompt when the session is ready.

  • Check the AWS CLI version provided by CloudShell by running this command:
aws --version

How Do I Read the Version Check?

The command prints the installed AWS CLI release. Version 2 output starts with aws-cli/2.

✔️ I see the expected version

CloudShell already has the required AWS CLI version. Your command-line environment is ready for the project.

ⓧ I see an older version

A managed CloudShell session should provide the current AWS CLI. An older major version does not meet this project's requirement.

  • Reload the current CloudShell browser tab.
  • Repeat the version check shown above.
  • Install AWS CLI version 2 in the CloudShell home directory by following the official Linux installation instructions if the older version remains.

ⓧ Command not found

A missing AWS CLI command means the CloudShell environment is incomplete. Restore the command before continuing.

  • Reload the current CloudShell browser tab.
  • Repeat the version check shown above.
  • Install AWS CLI version 2 in the CloudShell home directory by following the official Linux installation instructions if the command remains unavailable.
Prepare the authenticated workspace

A successful console sign-in does not tell you which identity the shell is using. The caller identity check connects your terminal session to the exact AWS account and principal behind each command.

Before you run the identity check, make a quick prediction about the account and caller AWS will report.

  • Confirm the identity used by CloudShell by running this command:
aws sts get-caller-identity

What Does This Command Prove?

  • The Account field identifies the AWS account receiving your commands.
  • The Arn field identifies the IAM caller behind the session.

You should see output containing your Account and caller Arn. Good, CloudShell is acting as the approved identity for this lab.

Identity Output Looks Wrong?

  • Sign out of the AWS Management Console if the output belongs to a different account.
  • Sign back in with the approved non-root IAM identity.
  • Launch CloudShell again from the same chosen Region.

Still stuck? Help me diagnose why AWS CloudShell is using the wrong caller identity.

The workspace gives every template and response file in this lab one predictable home. Creating it now keeps later deployment commands focused on the correct files.

  • Create the workspace and enter it by running these commands:
mkdir incident-ready-service
cd incident-ready-service

What Do These Commands Do?

  • The mkdir command creates the incident-ready-service folder inside your CloudShell home directory.
  • The cd command makes that folder your current location.

Before checking the location, predict how the new workspace name will appear at the end of the path.

  • Confirm your current directory by running this command:
pwd

What Does This Check Show?

The pwd command prints the full path of your current directory. It confirms that later files will be created inside the intended workspace.

You should see a path ending in /incident-ready-service. The newly created workspace is empty and ready for the first service template.

Path Ends Somewhere Else?

  • Check that the folder name is exactly incident-ready-service.
  • Repeat the workspace commands above if you remained in your CloudShell home directory.

Need another pair of eyes? Help me reach the incident-ready-service workspace in AWS CloudShell.

You have a verified AWS identity inside the empty incident-ready-service workspace. Next, you will use AWS CloudFormation to deploy the first working version of the service.

Deploy the First Working Service

Your AWS CloudShell session is authenticated in the chosen Region. The empty workspace is ready for its first deployable service.

A successful invocation gives you a known-good baseline before monitoring enters the picture. That baseline helps you separate deployment problems from operational incidents later.

You'll use AWS CloudFormation to deploy the service from template.yaml. The service runs as an AWS Lambda function.

In this step, get ready to:
  • Author the starter CloudFormation template.
  • Validate the template before deploying its stack.
  • Invoke the healthy path to confirm the service works.
Author the starter CloudFormation template

An Infrastructure as Code template records the service's desired state in a repeatable file. This starter state gives you a healthy path that you can deploy before adding monitoring.

Why CloudFormation for this lab?

CloudFormation creates AWS resources from one template. CloudShell already provides authenticated command-line access for deploying that template.

This keeps the lab focused on deployment plus incident diagnosis. The same template becomes your record of the service's intended configuration.

  • Create template.yaml in your existing ~/incident-ready-service workspace by running this command:
edit template.yaml

What does this command do?

The command opens template.yaml in the CloudShell editor. Saving the editor creates the file in your current workspace.

  • Paste the template header plus execution role into template.yaml using this first block:
AWSTemplateFormatVersion: "2010-09-09"
Description: Incident-ready Lambda service with logging, IAM, and error monitoring

Resources:
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

What does this section define?

  • The template version tells CloudFormation how to interpret the file.
  • The LambdaExecutionRole resource creates the function's IAM execution role.
  • The trust policy allows Lambda to assume the role.
  • The managed policy gives the function permission to write basic operational logs.
  • Save template.yaml with the editor's save control.
  • Confirm the editor shows LambdaExecutionRole beneath Resources.

Does the role section look misaligned?

YAML uses spaces to show nesting. Make sure Type plus Properties line up beneath LambdaExecutionRole.

Still stuck? Help me check the YAML indentation in my LambdaExecutionRole resource.

  • Paste the log group plus function configuration below the managed policy line:

  ServiceLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: /aws/lambda/incident-ready-service-handler
      RetentionInDays: 7

  ServiceFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: incident-ready-service-handler
      Runtime: python3.14
      Handler: index.handler
      Role: !GetAtt LambdaExecutionRole.Arn
      MemorySize: 128
      Timeout: 3

What does this configuration create?

  • The ServiceLogGroup resource creates a CloudWatch Logs group with seven-day retention.
  • The ServiceFunction resource fixes the function name at incident-ready-service-handler.
  • The runtime executes Python 3.14 code through index.handler.
  • The function receives 128 MB of memory. Its timeout is three seconds.
  • Save template.yaml with the editor's save control.
  • Confirm the editor shows ServiceLogGroup above ServiceFunction.

Are the resources nested incorrectly?

Make sure ServiceLogGroup plus ServiceFunction align with LambdaExecutionRole. Each resource belongs directly beneath Resources.

Still stuck? Help me align the resources in my CloudFormation template.

  • Paste the inline handler below the Timeout: 3 line:
      Code:
        ZipFile: |
          import json

          def handler(event, context):
              request_type = event.get("requestType", "health")

              if request_type != "health":
                  raise ValueError(f"Unsupported requestType: {request_type}")

              response = {
                  "message": "Service healthy",
                  "requestType": request_type
              }
              print(json.dumps({
                  "level": "INFO",
                  "message": "Health request completed"
              }))
              return {
                  "statusCode": 200,
                  "body": json.dumps(response)
              }

What does this handler do?

  • CloudFormation places the inline source in an index file. The handler(event, context) function matches the configured entry point.
  • The request_type variable reads the incoming request type. It defaults to health.
  • The initial invalid-input branch raises ValueError for unsupported request types.
  • A health request logs its completion. It returns status code 200 with the service message.
  • Save template.yaml with the editor's save control.
  • Confirm the final line of the handler closes the returned response beneath body.

Does the inline Python look uneven?

Keep every Python line nested beneath ZipFile: |. Keep the statements inside handler(event, context) indented farther than the function definition.

Still stuck? Help me check the indentation of the inline Python in my CloudFormation template.

  • Paste the stack outputs below the closing brace of the handler:

Outputs:
  FunctionName:
    Description: Name of the deployed Lambda function
    Value: !Ref ServiceFunction
  LogGroupName:
    Description: Name of the Lambda log group
    Value: !Ref ServiceLogGroup

What do these outputs provide?

The outputs expose the deployed function name plus log group name through the CloudFormation stack. They give operators stable references to the service's key resources.

  • Save the completed template.yaml with the editor's save control.
  • Confirm Outputs aligns with Resources at the left edge.

Are the outputs inside a resource?

Move Outputs to the left edge if it appears beneath ServiceFunction. It is a top-level template section.

Still stuck? Help me position the Outputs section in my CloudFormation template.

✔️ Awesome, I've got everything!

Your starter template now defines the execution role, retained log group, inline Lambda function, plus stack outputs.

ⓧ I'd like to double check the full code

  • Compare your saved template.yaml with this complete starter version:
AWSTemplateFormatVersion: "2010-09-09"
Description: Incident-ready Lambda service with logging, IAM, and error monitoring

Resources:
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

  ServiceLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: /aws/lambda/incident-ready-service-handler
      RetentionInDays: 7

  ServiceFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: incident-ready-service-handler
      Runtime: python3.14
      Handler: index.handler
      Role: !GetAtt LambdaExecutionRole.Arn
      MemorySize: 128
      Timeout: 3
      Code:
        ZipFile: |
          import json

          def handler(event, context):
              request_type = event.get("requestType", "health")

              if request_type != "health":
                  raise ValueError(f"Unsupported requestType: {request_type}")

              response = {
                  "message": "Service healthy",
                  "requestType": request_type
              }
              print(json.dumps({
                  "level": "INFO",
                  "message": "Health request completed"
              }))
              return {
                  "statusCode": 200,
                  "body": json.dumps(response)
              }

Outputs:
  FunctionName:
    Description: Name of the deployed Lambda function
    Value: !Ref ServiceFunction
  LogGroupName:
    Description: Name of the Lambda log group
    Value: !Ref ServiceLogGroup

How should I use this reference?

Check the resource order plus indentation against your saved file. The reference represents the exact starter stack for this step.

Validate and deploy the stack

Template validation catches structural problems before AWS starts creating resources. This gives you a shorter feedback loop for indentation plus property mistakes.

  • Validate the saved template by running this command:
aws cloudformation validate-template --template-body file://template.yaml

What does this command check?

CloudFormation reads the local template through file://template.yaml. It checks whether the document follows the expected template structure before deployment.

The command returns template details without reporting a validation error. That result means the file is ready for a deployment attempt.

Did template validation fail?

  • Check that Resources plus Outputs begin at the left edge.
  • Check that each resource's Type plus Properties lines use matching indentation.

Still stuck? Help me diagnose my CloudFormation template validation failure.

Before you deploy

This deployment creates usage-priced Lambda plus CloudWatch resources in your AWS account. The cleanup section removes the stack when you finish.

  • Deploy template.yaml as the incident-ready-service stack by running this command:
aws cloudformation deploy --template-file template.yaml --stack-name incident-ready-service --capabilities CAPABILITY_IAM

What does this deployment create?

  • The stack creates the Lambda execution role.
  • It creates the log group with seven-day retention.
  • It deploys the function from the inline Python source.
  • The CAPABILITY_IAM capability acknowledges that the template creates an IAM resource.

CloudFormation may spend several minutes creating the resources. A quiet pause during this deployment is expected.

When the command finishes, you'll see a report that the stack was created successfully.

That's the service foundation in place. Your stack now manages a working function plus its supporting resources.

Did the stack deployment fail?

  • Read the deployment output for the resource that failed.
  • Confirm your approved IAM identity can create CloudFormation, Lambda, IAM role, plus CloudWatch Logs resources.
  • Confirm the template validation command succeeded before retrying the deployment.

Still stuck? Help me diagnose my CloudFormation deployment failure.

Invoke the healthy path

A synchronous invocation writes the function's application response into success.json. Inspecting that JSON file proves the deployed code can handle a health request.

Before you run the invocation, what status code plus message do you expect the healthy path to return?

  • Invoke the deployed function with the health payload by running this command:
aws lambda invoke --function-name incident-ready-service-handler --cli-binary-format raw-in-base64-out --payload '{"requestType":"health"}' success.json

What does this invocation do?

  • The command invokes incident-ready-service-handler synchronously.
  • The payload sends health as the request type.
  • The application response is written to success.json in the current workspace.
  • Select success.json from the CloudShell editor's file list.

You'll see this exact healthy response:

{"statusCode": 200, "body": "{\"message\": \"Service healthy\", \"requestType\": \"health\"}"}

How do I read this response?

  • The outer statusCode confirms the handler returned 200.
  • The body contains a JSON-encoded application response.
  • The message confirms the deployed service completed its health path.
  • The request type confirms the function processed the value sent in the payload.

Is the health response missing?

  • Confirm the deployment completed before invoking the function.
  • Confirm success.json is inside ~/incident-ready-service.
  • Confirm the function name in the command matches incident-ready-service-handler exactly.

Still stuck? Help me diagnose my Lambda health invocation.

Your smallest deployable service now works from template to invocation. Next, you'll send an invalid request plus follow its failure into the logs.

Trigger and Diagnose an Incident

The healthy response from the previous step proves your Lambda function can serve valid traffic. Your existing AWS CloudShell session now becomes your incident console.

The starter handler still treats unsupported input as an unhandled exception. You need to experience that reactive troubleshooting gap before improving the service.

First, you will trigger the exception. Then you will trace its evidence through the response and CloudWatch Logs.

In this step, get ready to:
  • Reproduce an invalid request.
  • Identify the exception in CloudWatch Logs.
  • Confirm that no error alarm exists.
Reproduce the failure

A synchronous Lambda invocation returns the function's execution result to the caller. An unsupported request takes the current handler down its exception branch.

Before you run this, do you think the API call will fail or the function execution will report the failure separately?

  • Reproduce the unsupported request by running this command:
aws lambda invoke --function-name incident-ready-service-handler --cli-binary-format raw-in-base64-out --payload '{"requestType":"invalid"}' failure.json

What Does This Command Prove?

The command invokes incident-ready-service-handler synchronously. It writes the function's response to failure.json.

Lambda reports execution failures separately from the API status code. The invocation metadata therefore includes FunctionError when the handler crashes.

  • Use the AWS CloudShell file browser to open failure.json.

You'll see ValueError as the error type. You'll also see Unsupported requestType: invalid as the message.

That failure is intentional. The invalid request has exposed the handler's unprotected validation path.

Does the File Show a Healthy Response?

  • Check that the payload contains "requestType":"invalid".
  • Rerun the command above if failure.json was not created.

Still stuck? Help me diagnose why my invalid Lambda invocation did not produce the expected execution error.

Trace the exception in CloudWatch Logs

The response proves what failed. CloudWatch Logs shows where the failure occurred inside the running function.

Before you query the logs, which details do you expect to connect the invalid request to the handler exception?

  • Inspect the function's recent log events by running this command:
aws logs tail /aws/lambda/incident-ready-service-handler --since 5m --format short

How Does This Query Narrow the Evidence?

  • The log group name targets only events from incident-ready-service-handler.
  • The --since 5m option limits the search to the incident window.
  • The --format short option keeps the output focused on each event's timestamp and message.

You'll see ValueError in the recent events. You'll also see Unsupported requestType: invalid in the exception details.

You have pinned down the incident. The request value and exception now point to the same validation branch.

Are the Recent Logs Empty?

  • Wait a few seconds for the new log event to become available.
  • Run the log command above again.
  • Confirm that AWS CloudShell remains in the Region where you deployed the stack.

Still stuck? Help me find the missing Lambda exception in CloudWatch Logs.

Confirm the monitoring gap

A CloudWatch alarm turns a metric threshold into an operational state. Checking for an alarm now reveals whether this incident could surface without manual investigation.

Before you run this check, do you expect the service to have raised an operational signal automatically?

  • Search for the service's error alarm by running this command:
aws cloudwatch describe-alarms --alarm-names incident-ready-service-errors

What Does This Alarm Check Reveal?

The command asks CloudWatch for an alarm named incident-ready-service-errors. The result tells you whether an operational signal exists for this specific service.

You'll see no matching metric alarm. The failure is discoverable only when someone inspects the response or searches the logs.

Did a Matching Alarm Appear?

A matching alarm means your AWS environment differs from the expected project state. Confirm that its name is exactly incident-ready-service-errors.

Need help comparing the states? Help me identify why an error alarm already exists in my project environment.

Before you run the final evidence check, which two strings should connect the invalid input to the exception?

  • Verify the incident evidence one final time by running this command:
aws logs tail /aws/lambda/incident-ready-service-handler --since 5m --format short

Why Repeat the Log Check?

This final query acts as your incident checkpoint. It proves that the recent exception remains available as operational evidence.

Your output includes ValueError. It also includes Unsupported requestType: invalid.

Strong work. You reproduced the failure and followed its evidence to the exact invalid input without guessing.

Is the Final Exception Missing?

  • Confirm that the log group path begins with /aws/lambda/.
  • Repeat the invalid invocation if the five-minute window has passed.

Need another set of eyes? Help me verify the final CloudWatch Logs evidence for this incident.

Your template.yaml file remains unchanged. Your new failure.json file records the execution error while success.json preserves the healthy response.

You can now diagnose this incident manually. Next, you'll add the proactive signal that the service currently lacks.

Add Proactive Error Monitoring

Your previous investigation traced the invalid request to an AWS Lambda exception through Amazon CloudWatch logs. The evidence only appeared after manual inspection.

This step adds an error alarm through AWS CloudFormation. The alarm turns function errors into an operational signal that you can check without searching through logs.

In this step, get ready to:
  • Define a CloudWatch alarm for Lambda execution errors.
  • Update the existing CloudFormation stack with the monitoring resource.
  • Prove the alarm reaches ALARM after a repeated function error.
Define the alarm as code

The Lambda Errors metric counts failed function executions. Your template can connect that metric to an alarm with a one-minute evaluation period.

  • Return to your CloudShell workspace editor by running this command:
edit template.yaml

What does this command do?

This command loads the existing template.yaml file in the CloudShell editor. You can update the deployed infrastructure from the same regional workspace.

  • In template.yaml, find the Outputs: line near the bottom of the file.
  • Add the alarm resource directly above Outputs: by pasting this block:
  ServiceErrorAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: incident-ready-service-errors
      AlarmDescription: Alarm when the Lambda function records an execution error
      Namespace: AWS/Lambda
      MetricName: Errors
      Dimensions:
        - Name: FunctionName
          Value: !Ref ServiceFunction
      Statistic: Sum
      Period: 60
      EvaluationPeriods: 1
      DatapointsToAlarm: 1
      Threshold: 1
      ComparisonOperator: GreaterThanOrEqualToThreshold
      TreatMissingData: notBreaching

What does this alarm monitor?

  • The AWS/Lambda namespace points the alarm at standard Lambda metrics.
  • The FunctionName dimension limits the alarm to incident-ready-service-handler.
  • The Sum statistic counts execution errors during each 60-second period.
  • The threshold moves the alarm to ALARM when at least one error occurs.
  • The notBreaching setting treats idle periods as healthy.
  • Save template.yaml in the CloudShell editor.
  • Return to the CloudShell terminal panel.
  • Check the alarm block for valid YAML by running:
aws cloudformation validate-template --template-body file://template.yaml

What does this validation prove?

CloudFormation parses the local template without changing the deployed stack. A successful response confirms that the new alarm block uses valid template structure.

You should see template details in the response. This confirms the alarm definition is valid YAML.

Is the alarm block failing validation?

  • Check that ServiceErrorAlarm aligns with the other resource names under Resources.
  • Check that each property under Properties uses consistent two-space indentation.
  • Confirm that Outputs: remains at the start of its line.

Still stuck? Help me fix the YAML structure in my CloudFormation alarm block.

The alarm output makes the monitored resource visible in the stack results. This gives the template a named reference to the alarm it creates.

  • Return to template.yaml in the CloudShell editor.
  • Find the existing LogGroupName output at the bottom of the file.
  • Add the alarm output below the LogGroupName output by pasting:
  AlarmName:
    Description: Name of the Lambda error alarm
    Value: !Ref ServiceErrorAlarm

What does this output expose?

The AlarmName output returns the physical name created for ServiceErrorAlarm. This keeps the alarm discoverable from the stack's outputs.

  • Save template.yaml in the CloudShell editor.

✔️ Awesome, I've got everything!

Your saved template now defines the error alarm. It also exposes the alarm name as a stack output.

ⓧ I'd like to double check the full code

  • Compare your saved template.yaml file with this complete version:
AWSTemplateFormatVersion: "2010-09-09"
Description: Incident-ready Lambda service with logging, IAM, and error monitoring

Resources:
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

  ServiceLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: /aws/lambda/incident-ready-service-handler
      RetentionInDays: 7

  ServiceFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: incident-ready-service-handler
      Runtime: python3.14
      Handler: index.handler
      Role: !GetAtt LambdaExecutionRole.Arn
      MemorySize: 128
      Timeout: 3
      Code:
        ZipFile: |
          import json

          def handler(event, context):
              request_type = event.get("requestType", "health")

              if request_type != "health":
                  raise ValueError(f"Unsupported requestType: {request_type}")

              response = {
                  "message": "Service healthy",
                  "requestType": request_type
              }
              print(json.dumps({
                  "level": "INFO",
                  "message": "Health request completed"
              }))
              return {
                  "statusCode": 200,
                  "body": json.dumps(response)
              }

  ServiceErrorAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: incident-ready-service-errors
      AlarmDescription: Alarm when the Lambda function records an execution error
      Namespace: AWS/Lambda
      MetricName: Errors
      Dimensions:
        - Name: FunctionName
          Value: !Ref ServiceFunction
      Statistic: Sum
      Period: 60
      EvaluationPeriods: 1
      DatapointsToAlarm: 1
      Threshold: 1
      ComparisonOperator: GreaterThanOrEqualToThreshold
      TreatMissingData: notBreaching

Outputs:
  FunctionName:
    Description: Name of the deployed Lambda function
    Value: !Ref ServiceFunction
  LogGroupName:
    Description: Name of the Lambda log group
    Value: !Ref ServiceLogGroup
  AlarmName:
    Description: Name of the Lambda error alarm
    Value: !Ref ServiceErrorAlarm
Validate and deploy the monitoring update

The saved template now describes the complete monitoring change. CloudFormation can validate the final file before updating the deployed stack.

  • Return to the CloudShell terminal panel.
  • Validate the complete template by running:
aws cloudformation validate-template --template-body file://template.yaml

Why validate again?

The first validation checked the alarm resource. This validation checks the saved alarm output as part of the complete template.

You should see template details in the response. Your complete monitoring update is ready to deploy.

The stack update can take a few minutes while CloudFormation creates the alarm. Keep the terminal open until the deployment finishes.

  • Update the existing incident-ready-service stack by running:
aws cloudformation deploy --template-file template.yaml --stack-name incident-ready-service --capabilities CAPABILITY_IAM

What does this deployment change?

CloudFormation recognizes the existing stack name. It updates that stack with ServiceErrorAlarm while preserving the deployed function.

The CAPABILITY_IAM acknowledgement remains necessary because the template contains the Lambda execution role.

The deployment output confirms that the existing stack was updated successfully. Your alarm is now managed by the same template as the service.

Did the stack update fail?

  • Confirm that the terminal path still ends in /incident-ready-service.
  • Run the template validation command again if the deployment reports a template problem.
  • Check that your approved IAM identity can update CloudFormation resources.

Need another pair of eyes? Help me diagnose my CloudFormation stack update.

Reproduce the incident under monitoring

The same invalid request now has an alarm watching its execution. Repeating the failure proves that the monitoring resource responds to a real Lambda error.

Before you run this, make a quick prediction about whether the new alarm will react to the invalid request.

  • Invoke the invalid request again by running:
aws lambda invoke --function-name incident-ready-service-handler --cli-binary-format raw-in-base64-out --payload '{"requestType":"invalid"}' failure.json

What does this invocation prove?

The request enters the same unhandled ValueError branch as before. Lambda records the failed execution in the Errors metric.

  • Return to the CloudShell editor.
  • Select failure.json from the file list.

You should see ValueError with Unsupported requestType: invalid. The repeated failure gives the new alarm a metric datapoint to evaluate.

CloudWatch delivers Lambda metrics at one-minute intervals. A short wait here means the metric is still being evaluated.

  • Wait for one minute before checking the alarm.

Before you check, predict which alarm state now matches the error you just generated.

  • Inspect the alarm state by running:
aws cloudwatch describe-alarms --alarm-names incident-ready-service-errors

What should you inspect?

The response contains the current StateValue for incident-ready-service-errors. This field shows whether the error threshold was breached during the evaluated period.

You should see StateValue set to ALARM. The failure now creates a visible operational signal.

Is the alarm still waiting?

  • Wait for another one-minute metric interval.
  • Run the alarm description command again.
  • Confirm that the invalid invocation targeted incident-ready-service-handler.

Still missing the state change? Help me diagnose why my Lambda error alarm has not reached ALARM.

Your service now raises an operational signal when the handler crashes. Next, you will repair the validation path and confirm that the alarm recovers.

Repair the Service and Write the Runbook

Your CloudWatch alarm exposed the repeated function failure. Now you can turn that evidence into a lasting repair.

Detection alone does not restore service. You'll use CloudFormation to repair the AWS Lambda handler. Then you'll capture the incident procedure in a runbook.

In this step, get ready to:
  • Deploy a handled invalid-input branch through CloudFormation.
  • Verify the controlled response.
  • Record the incident procedure in an operations runbook.
Deploy the validation fix

The existing handler treats unsupported input as an application crash. The repaired branch records a warning before returning a controlled response.

  • Return to your AWS CloudShell terminal from earlier.
  • Open the existing template.yaml file in the CloudShell editor by running:
edit template.yaml

What does this command do?

The command loads the existing CloudFormation template into the CloudShell editor. This keeps the repair inside the file that defines the stack's desired state.

  • In template.yaml, find the line raise ValueError(f"Unsupported requestType: {request_type}").
  • Replace that single line with this handled branch:
              if request_type != "health":
                  warning = {
                      "level": "WARNING",
                      "message": "Rejected unsupported request type",
                      "requestType": request_type
                  }
                  print(json.dumps(warning))
                  return {
                      "statusCode": 400,
                      "body": json.dumps({
                          "message": "Unsupported request type",
                          "requestType": request_type
                      })
                  }

What does this repair do?

  • The warning object creates structured evidence for troubleshooting.
  • The call to json.dumps() writes that evidence to the function log.
  • The return value gives the caller a controlled 400 response.
  • Save template.yaml.
  • Validate the repaired template by running:
aws cloudformation validate-template --template-body file://template.yaml

What does this validation check?

CloudFormation parses the saved template before any stack update begins. A successful result shows template details without a validation error.

You should see the template description in the command output. That confirms the repaired file passed validation.

Template validation failing?

  • Check that the replacement branch remains indented beneath ZipFile: |.
  • Check that every key inside warning lines up at the same indentation level.
  • Confirm that the old raise ValueError line has been removed.

Still stuck? Help me find the YAML indentation problem in my repaired Lambda branch.

The next command updates the existing stack named incident-ready-service. It keeps the same environment while CloudFormation applies the repaired function code.

Expect the update to take several minutes while CloudFormation publishes the new Lambda configuration.

  • Deploy the repaired template to the existing stack by running:
aws cloudformation deploy --template-file template.yaml --stack-name incident-ready-service --capabilities CAPABILITY_IAM

What does this deployment change?

The stack name points CloudFormation at the environment you already deployed. The update replaces the inline function code while preserving the stack's other resources.

You should see a message confirming that the stack update completed successfully.

Stack update failing?

  • Confirm that the validation command completed successfully before retrying the deployment.
  • Confirm that your current identity has permission to update CloudFormation resources.
  • Confirm that CAPABILITY_IAM remains present in the command.

Need help with the update? Help me diagnose my failed CloudFormation deployment.

✔️ Awesome, I've got everything!

The repaired template is saved. The existing stack has completed its update.

ⓧ I'd like to double check the full code

  • Compare your saved template.yaml with this complete repaired version:
AWSTemplateFormatVersion: "2010-09-09"
Description: Incident-ready Lambda service with logging, IAM, and error monitoring

Resources:
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

  ServiceLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: /aws/lambda/incident-ready-service-handler
      RetentionInDays: 7

  ServiceFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: incident-ready-service-handler
      Runtime: python3.14
      Handler: index.handler
      Role: !GetAtt LambdaExecutionRole.Arn
      MemorySize: 128
      Timeout: 3
      Code:
        ZipFile: |
          import json

          def handler(event, context):
              request_type = event.get("requestType", "health")

              if request_type != "health":
                  warning = {
                      "level": "WARNING",
                      "message": "Rejected unsupported request type",
                      "requestType": request_type
                  }
                  print(json.dumps(warning))
                  return {
                      "statusCode": 400,
                      "body": json.dumps({
                          "message": "Unsupported request type",
                          "requestType": request_type
                      })
                  }

              response = {
                  "message": "Service healthy",
                  "requestType": request_type
              }
              print(json.dumps({
                  "level": "INFO",
                  "message": "Health request completed"
              }))
              return {
                  "statusCode": 200,
                  "body": json.dumps(response)
              }

  ServiceErrorAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: incident-ready-service-errors
      AlarmDescription: Alarm when the Lambda function records an execution error
      Namespace: AWS/Lambda
      MetricName: Errors
      Dimensions:
        - Name: FunctionName
          Value: !Ref ServiceFunction
      Statistic: Sum
      Period: 60
      EvaluationPeriods: 1
      DatapointsToAlarm: 1
      Threshold: 1
      ComparisonOperator: GreaterThanOrEqualToThreshold
      TreatMissingData: notBreaching

Outputs:
  FunctionName:
    Description: Name of the deployed Lambda function
    Value: !Ref ServiceFunction
  LogGroupName:
    Description: Name of the Lambda log group
    Value: !Ref ServiceLogGroup
  AlarmName:
    Description: Name of the Lambda error alarm
    Value: !Ref ServiceErrorAlarm

This reference includes the repaired branch plus the existing role, log group, function, alarm, and outputs.

Verify recovery

A controlled invalid-input response proves that the request is rejected without crashing the function. The logs plus the alarm show whether that change restored operational health.

Before you run the invalid request, do you expect a function error or a controlled application response?

  • Invoke the repaired function with the invalid payload by running:
aws lambda invoke --function-name incident-ready-service-handler --cli-binary-format raw-in-base64-out --payload '{"requestType":"invalid"}' repair.json

What does this invocation prove?

The command sends the same invalid request that previously caused the incident. The function response is written to repair.json for inspection.

  • Select repair.json from the CloudShell file list.

The file should read {"statusCode": 400, "body": "{\"message\": \"Unsupported request type\", \"requestType\": \"invalid\"}"}.

Still seeing a function error?

  • Confirm that the deployment reported a successful stack update.
  • Confirm that template.yaml contains the complete warning branch.
  • Confirm that the invocation targets incident-ready-service-handler.

Need another pair of eyes? Help me diagnose why repair.json still contains a Lambda function error.

The controlled response confirms that the application path is repaired. Next, the log stream must show a warning without an exception traceback.

  • Inspect the latest function logs by running:
aws logs tail /aws/lambda/incident-ready-service-handler --since 5m --format short

What does the log check prove?

The log stream reveals how the repaired handler classified the request. Structured warning evidence makes the rejected input visible without turning it into a function error.

You should see WARNING plus Rejected unsupported request type. The latest invocation should contain no Python traceback.

Warning missing from the logs?

  • Confirm that repair.json contains the controlled response.
  • Confirm that the log group name is /aws/lambda/incident-ready-service-handler.
  • Run the invalid invocation again if the five-minute log window has expired.

Still missing the entry? Help me trace my repaired invocation into CloudWatch Logs.

Lambda sends invocation metrics to CloudWatch at one-minute intervals. The alarm can require another evaluation before it leaves ALARM.

Before you poll the alarm, which state do you expect after a period without another function error?

  • Check the current alarm state by running:
aws cloudwatch describe-alarms --alarm-names incident-ready-service-errors

What does the alarm check prove?

The command retrieves the metric alarm created by your stack. Its state shows whether the latest evaluation still contains a Lambda execution error.

After the next evaluation completes, you should see "StateValue": "OK". That state confirms the repaired invocation did not register as another function error.

Alarm still showing ALARM?

  • Wait for the next one-minute metric evaluation.
  • Poll the alarm again by rerunning the same command.
  • Avoid invoking the old exception path while the alarm is recovering.

Need help reading the state? Help me determine why my Lambda error alarm has not recovered to OK.

That closes the recovery loop. Your service now rejects unsupported input without creating a Lambda execution error.

Capture operational evidence

A repaired service can fail again if the response procedure lives only in memory. A Markdown runbook turns this incident into a repeatable operating process.

  • Create OPERATIONS.md in the current workspace by running:
edit OPERATIONS.md

What does this command create?

The command creates OPERATIONS.md if it is missing. It opens the file in the CloudShell editor for your incident documentation.

The first part of the runbook identifies the deployed resources. It also gives an operator a known-good health check.

  • Paste this service inventory plus health check into OPERATIONS.md.
# Incident-Ready Service Operations

## Service inventory

- CloudFormation stack: `incident-ready-service`
- Lambda function: `incident-ready-service-handler`
- CloudWatch log group: `/aws/lambda/incident-ready-service-handler`
- CloudWatch alarm: `incident-ready-service-errors`

## Healthy check

Run:

```bash
aws lambda invoke \
  --function-name incident-ready-service-handler \
  --cli-binary-format raw-in-base64-out \
  --payload '{"requestType":"health"}' \
  health-check.json
```

Expected application response: status code 200 with the message `Service healthy`.

What does this section capture?

  • The service inventory gives an operator the stable names needed during an incident.
  • The healthy check provides a repeatable test for the normal request path.
  • Save OPERATIONS.md.
  • Scroll to the top of the file.

You should see the service title followed by the Service inventory section. The Healthy check section should contain the invocation command.

Markdown formatting looking broken?

  • Confirm that each heading begins on a new line.
  • Confirm that the health command remains between matching code fences.

Need help with the layout? Help me fix the Markdown structure in my service inventory and health check.

The triage section tells an operator where to begin gathering evidence. It keeps the investigation tied to the alarm plus recent logs.

  • Place the cursor on a new line after the healthy-check response.
  • Paste this incident triage section:
## Incident triage

1. Inspect the alarm:

```bash
aws cloudwatch describe-alarms --alarm-names incident-ready-service-errors
```

2. Inspect recent function logs:

```bash
aws logs tail /aws/lambda/incident-ready-service-handler --since 5m --format short
```

3. Correlate the alarm state, request payload, exception or warning, and the most recent infrastructure change.

Why use this triage order?

The alarm establishes the operational signal. The logs then supply the request-level evidence needed to identify the failure.

  • Save OPERATIONS.md.
  • Scroll to the Incident triage heading.

You should see three numbered triage actions. The first two actions include the exact commands you used during the incident.

Triage steps out of order?

  • Place the alarm inspection before the log inspection.
  • Keep the evidence-correlation instruction as the third numbered item.

Need help checking the sequence? Help me correct the incident triage order in OPERATIONS.md.

Mitigation should preserve the template as the source of truth. Recovery then repeats the health check before the alarm is considered clear.

  • Place the cursor on a new line after the triage section.
  • Paste this mitigation plus recovery section:
## Mitigation and recovery

1. Keep the desired configuration in `template.yaml`.
2. Validate it:

```bash
aws cloudformation validate-template --template-body file://template.yaml
```

3. Deploy it to the existing stack:

```bash
aws cloudformation deploy \
  --template-file template.yaml \
  --stack-name incident-ready-service \
  --capabilities CAPABILITY_IAM
```

4. Repeat the healthy check and inspect the alarm until it reports `OK`.

What does this recovery sequence protect?

The sequence keeps the desired configuration in template.yaml. Validation reduces deployment mistakes before the existing stack is updated.

  • Save OPERATIONS.md.
  • Scroll to the Mitigation and recovery heading.

You should see four numbered recovery actions. The last action connects the health response to the alarm state.

Recovery commands incomplete?

  • Confirm that the validation command references template.yaml.
  • Confirm that the deployment command uses incident-ready-service.
  • Confirm that CAPABILITY_IAM appears beneath the deployment command.

Need help comparing the commands? Help me correct the mitigation and recovery section in OPERATIONS.md.

The final sections preserve the reason behind the repair. They also define rollback plus cleanup boundaries for future operators.

  • Place the cursor on a new line after the recovery section.
  • Paste these final runbook sections:
## Rollback

Restore the last known working contents of `template.yaml`, validate the file, and deploy it to the same stack name. Do not repair a CloudFormation-managed resource manually unless an incident requires immediate mitigation. If a manual change is necessary, record it and run drift detection afterward.

## Root-cause analysis

- Incident: An unsupported request type caused an unhandled Lambda exception.
- Impact: The request failed and the Lambda `Errors` metric increased.
- Detection gap: The first deployment produced logs but had no alarm.
- Root cause: The handler raised an exception instead of validating unsupported input.
- Correction: The handler now logs a warning and returns a controlled 400 response.
- Prevention: Keep the error alarm in CloudFormation, test valid and invalid inputs after deployments, and review drift after emergency changes.

## Cleanup

```bash
aws cloudformation delete-stack --stack-name incident-ready-service
```

What do these sections preserve?

  • Rollback defines how to restore a known working template.
  • Root-cause analysis records the incident impact plus the correction.
  • Cleanup records the stack deletion command for the end of the lab.
  • Save OPERATIONS.md.
  • Scroll to the bottom of the file.

You should see the Rollback heading followed by Root-cause analysis. The file should end with the Cleanup section.

Final sections missing?

  • Confirm that the rollback paragraph appears before the root-cause list.
  • Confirm that the cleanup command remains inside its code fence.
  • Confirm that the file ends after the cleanup code fence.

Need help checking the ending? Help me compare the final sections of OPERATIONS.md with the required runbook.

✔️ Awesome, I've got everything!

Your runbook now covers the service inventory through cleanup. Double check that OPERATIONS.md is saved.

ⓧ I'd like to double check the full code

  • Compare your saved OPERATIONS.md with this complete runbook:
# Incident-Ready Service Operations

## Service inventory

- CloudFormation stack: `incident-ready-service`
- Lambda function: `incident-ready-service-handler`
- CloudWatch log group: `/aws/lambda/incident-ready-service-handler`
- CloudWatch alarm: `incident-ready-service-errors`

## Healthy check

Run:

```bash
aws lambda invoke \
  --function-name incident-ready-service-handler \
  --cli-binary-format raw-in-base64-out \
  --payload '{"requestType":"health"}' \
  health-check.json
```

Expected application response: status code 200 with the message `Service healthy`.

## Incident triage

1. Inspect the alarm:

```bash
aws cloudwatch describe-alarms --alarm-names incident-ready-service-errors
```

2. Inspect recent function logs:

```bash
aws logs tail /aws/lambda/incident-ready-service-handler --since 5m --format short
```

3. Correlate the alarm state, request payload, exception or warning, and the most recent infrastructure change.

## Mitigation and recovery

1. Keep the desired configuration in `template.yaml`.
2. Validate it:

```bash
aws cloudformation validate-template --template-body file://template.yaml
```

3. Deploy it to the existing stack:

```bash
aws cloudformation deploy \
  --template-file template.yaml \
  --stack-name incident-ready-service \
  --capabilities CAPABILITY_IAM
```

4. Repeat the healthy check and inspect the alarm until it reports `OK`.

## Rollback

Restore the last known working contents of `template.yaml`, validate the file, and deploy it to the same stack name. Do not repair a CloudFormation-managed resource manually unless an incident requires immediate mitigation. If a manual change is necessary, record it and run drift detection afterward.

## Root-cause analysis

- Incident: An unsupported request type caused an unhandled Lambda exception.
- Impact: The request failed and the Lambda `Errors` metric increased.
- Detection gap: The first deployment produced logs but had no alarm.
- Root cause: The handler raised an exception instead of validating unsupported input.
- Correction: The handler now logs a warning and returns a controlled 400 response.
- Prevention: Keep the error alarm in CloudFormation, test valid and invalid inputs after deployments, and review drift after emergency changes.

## Cleanup

```bash
aws cloudformation delete-stack --stack-name incident-ready-service
```

This reference is the complete operational artifact for the repaired service. It preserves the exact resource names plus the tested incident commands.

Before you inspect the saved file, which section would you use first when the alarm changes state?

  • Open the saved runbook for the final check by running:
edit OPERATIONS.md

What should the final file show?

The editor should show seven operational sections beneath the main title. The file should begin with service inventory information before ending with the cleanup command.

You have closed the full incident loop. Your stack now handles bad input safely while your runbook explains how to detect, repair, verify, and document the failure.

Secret mission

Detect and Reconcile Configuration Drift

Create an out-of-band timeout change on the deployed Lambda function. Use CloudFormation drift detection to find the mismatch. Restore the declared value to prove the stack is back in sync.

Clean Up Your Resources

AWS CloudFormation and AWS CloudShell have no additional charge.

AWS Lambda execution, Amazon CloudWatch Logs storage, and CloudWatch alarms can incur usage-based charges.

Account credits or free-tier eligibility vary, so do not assume the deployed stack is free.

Nice Work!

Nice Work!

Mission accomplished! You built an incident-ready service on AWS. Your monitoring now surfaces execution failures.

You've learned how to:

  • Deployed a repeatable Infrastructure as Code stack with AWS CloudFormation. Your stack runs a Python 3.14 AWS Lambda function. Its IAM execution role writes operational logs to Amazon CloudWatch Logs.
  • Performed incident diagnosis on an intentional function failure. You traced the unhandled exception through the response and logs. You identified the unsupported input as the root cause.
  • Added proactive error monitoring with a CloudWatch alarm on the Lambda Errors metric. You watched the alarm move to ALARM after the incident. You deployed a controlled 400 response through CloudFormation. You verified recovery when the alarm returned to OK. Your OPERATIONS.md runbook records the response process and root cause.
  • Completed the optional Secret Mission by creating out-of-band configuration drift on the Lambda timeout. You detected the modified Timeout property through CloudFormation. You reconciled the live timeout with the template. The final drift check reported IN_SYNC.

Ready to quiz yourself?