AWS Portal: Detect, Isolate, Recover

Detect, isolate, and recover a fictional AWS dealer portal incident.

Introduction

30 Second Summary

One altered role can quietly turn a healthy-looking dealer portal into an access risk. Responders need clear evidence before they can safely contain the damage.

In this project, you will practise incident response through a controlled game day against a serverless dealer portal on AWS. You will take the portal from a clean baseline through a controlled compromise to an evidence-backed recovery.

What You'll Build

Your finished demo takes the dealer portal from a visible admin compromise to a clean dealer response from its recovered table.

By the end of this project, you'll have:

  • A live HTTP dealer route that returns only the three fake DLR-001 to DLR-003 records.
  • A separate status route that remains available while dealer processing is isolated.
  • An evidence-backed incident drill with a labeled EventBridge signal, narrow Lambda containment, negative controls, Terraform recovery, backup restore, and credential-free CI.
  • Secret Mission: Choose one hardening extension covering DynamoDB PITR, GuardDuty samples, GitHub OIDC, HTTP API access logs, or a Lambda throttles alarm.

This complete weekend lab is titled Detect, Isolate, and Recover an AWS Dealer Portal. It expands the direct-invocation exercise into a public HTTP data plane, a separate status path, automated containment, tested recovery, and a credential-free CI security gate.

The full build also proves these observable outcomes:

  • A live HTTP API where /dealers returns fake dealer records while /status reports whether the data path is healthy or isolated.
  • A labeled Amazon EventBridge drill that invokes a narrowly scoped containment function, sends an Amazon SNS notification, and writes evidence to Amazon CloudWatch Logs.
  • A blast-radius proof showing that dealers traffic is blocked while the status route stays healthy, plus negative tests proving unrelated events cannot trigger isolation.
  • Two tested recovery paths: Terraform restores control-plane drift, while an on-demand DynamoDB backup restores fake data into a new table without overwriting the source.
Reported Facts and Lab Fiction

The real-world context is limited to publicly reported Honda incidents. BleepingComputer reported a June 2022 incident affecting Honda and Acura systems, while Cyble reported a separate November 2023 dealer-portal claim.

Those reports provide scenario context only. This project does not claim access to Honda's internal architecture, forensic evidence, root cause, containment actions, or recovery process.

Everything you build is lab fiction: the AWS architecture, dealer records, event payload, log messages, isolation workflow, and recovery evidence are invented for learning. The drill proves your lab controls against synthetic input, not detection of a real attacker.

Time, Cost, and Safety
  • Plan for 8–12 hours over one weekend. The estimate includes reading, deployment waits, evidence capture, troubleshooting, recovery tests, and project-only cleanup.
  • Budget up to $2 USD. AWS usage, Free Tier eligibility, credits, Region, and account age can change the final amount, so zero cost is not guaranteed.
  • Use only fake dealer IDs and fake roles. Never paste customer data, production logs, access keys, session tokens, passwords, or private incident evidence into this lab.
  • Keep Project 1 separate. Do not reuse, import, rename, edit, or destroy its resources, state, repository, or evidence.
  • Treat this as cloud engineering and incident-response practice. It is not a full governance, risk, and compliance implementation.
  • Describe interview results precisely. Say you built a synthetic detection drill unless you separately validated a detector against real telemetry in an authorized environment.
Prerequisites
  • An AWS account with a non-root identity that can manage IAM roles, Lambda functions, API Gateway HTTP APIs, DynamoDB tables and backups, EventBridge rules, SNS topics, and CloudWatch Logs.
  • Access to AWS CloudShell in us-east-2 and permission to read the current caller identity.
  • A GitHub account, a private repository for this project, and email access for confirming the SNS subscription.
  • The canonical project-02-assets/ bundle from the guide, including Terraform, Lambda source, tests, workflow, and cleanup script files.
  • Basic familiarity with Git, Bash, JSON, Python, and Terraform. The project still explains each required command and flag.

Are there any prerequisites?

You'll need an AWS account with permission to manage every resource used by this lab. The project checks your environment before creating anything.

Before We Start

This first checkpoint sets a clear safety boundary before any hands-on work begins. Your written commitment keeps the simulated incident focused on lab-only resources and away from production.

Verify the Incident Lab Environment

This lab uses one AWS account, one Region, and one local Terraform root. Verifying that boundary before deployment keeps every later action scoped to p02-portal.

AWS CloudShell provides the authenticated shell. You will pin Terraform 1.9.8 so local checks match the canonical CI workflow.

In this step, get ready to:
  • Verify the caller identity and Ohio Region.
  • Check the required tools and install Terraform 1.9.8 when needed.
  • Create the private repository and local p02-portal work area.
Verify the account and Region
  • Sign in to the AWS Management Console with a non-root identity.
  • Select us-east-2 in the console header.
  • Open AWS CloudShell from the console toolbar.
  • Confirm the shell selector shows Bash.
  • Fix both AWS Region variables and verify the authenticated caller by running:
export AWS_REGION=us-east-2
export AWS_DEFAULT_REGION="$AWS_REGION"
aws sts get-caller-identity --output json
aws configure get region
echo "$AWS_REGION"

What should the identity check prove?

The caller command should return one account ID and a non-root ARN. The final line must print us-east-2.

The configured Region line can be blank in CloudShell. The explicit environment variable is the value this guide relies on.

🙋‍♀️ Wrong account or Region?

If the caller is root or the wrong account, stop and sign in with the intended IAM or IAM Identity Center identity. If the last line differs, rerun the two Region exports.

Help me verify the caller and Region.

Verify the toolchain
  • Check every required command by running:
aws --version
terraform version
python3 --version
git --version
curl --version | head -n 1
zip -v | head -n 2

CloudShell should provide AWS CLI v2, Python 3, Git, curl, and zip. Terraform must report v1.9.8 for this guide.

  • Install Terraform 1.9.8 in your CloudShell home directory only when the version check is missing or different:
TF_VERSION=1.9.8
curl -fsSLo terraform.zip "https://releases.hashicorp.com/terraform/${TF_VERSION}/terraform_${TF_VERSION}_linux_amd64.zip"
unzip -o terraform.zip
mkdir -p "$HOME/bin"
mv terraform "$HOME/bin/terraform"
export PATH="$HOME/bin:$PATH"
terraform version

How does the temporary install work?

  • The curl -f flag fails on an HTTP error.
  • The -sS flags hide progress while retaining error messages.
  • The -L flag follows redirects, while -o writes terraform.zip.
  • The unzip -o command replaces a same-named local extraction without prompting.
  • The PATH export makes $HOME/bin/terraform available in this session.
Create the isolated work area

Keep this project separate from Project 1. The repository and local directory both use the exact name p02-portal.

  • Create a new private GitHub repository named p02-portal.
  • Leave GitHub's README, license, and .gitignore generation disabled.
  • Create the local work area and initialize Git by running:
mkdir -p "$HOME/p02-portal"
cd "$HOME/p02-portal"
pwd
git init
git branch -M main

The printed path should end with /p02-portal. Git should report an initialized repository on branch main.

Your account, Region, tools, and repository boundary are ready. Next, you will assemble and validate the canonical source before creating AWS resources.

Deploy a Clean Dealer Portal Backend

Your account, Region, tools, and p02-portal directory are ready. You can now assemble the complete lab without touching Project 1.

The canonical stack separates dealer traffic, status checks, and containment into three Lambda functions. Terraform records the API, table, event rule, topic, logs, and least-privilege roles as one reviewable unit.

In this step, get ready to:
  • Create every canonical repository file.
  • Run offline tests and Terraform validation.
  • Deploy only p02-portal resources and capture healthy baselines.
Assemble the canonical repository

Create the directories from Appendix A. Then use the full-file references below to save each source exactly as shown.

  • Create the canonical directory tree by running:
cd "$HOME/p02-portal"
mkdir -p lambda/dealers lambda/status lambda/contain data scripts tests .github/workflows build

Terraform

terraform {
  required_version = ">= 1.5.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.80"
    }
    archive = {
      source  = "hashicorp/archive"
      version = "~> 2.6"
    }
  }
}
provider "aws" {
  region = var.aws_region

  default_tags {
    tags = {
      Project     = var.project_prefix
      ManagedBy   = "terraform"
      Environment = "practice-lab"
    }
  }
}
variable "aws_region" {
  description = "AWS Region for every resource in this lab. Use us-east-2 to match CloudShell."
  type        = string
  default     = "us-east-2"
}

variable "project_prefix" {
  description = "Name prefix so cleanup can target only this project's resources."
  type        = string
  default     = "p02-portal"

  validation {
    condition     = can(regex("^[a-z0-9-]{3,20}$", var.project_prefix))
    error_message = "Use 3-20 lowercase letters, numbers, or hyphens."
  }
}

variable "alert_email" {
  description = "Email that receives drill isolation notices. You must confirm the SNS subscription."
  type        = string

  validation {
    condition     = length(regexall("^[^@]+@[^@]+\\.[^@]+$", var.alert_email)) > 0
    error_message = "Provide a reachable email address."
  }
}

variable "dealers_reserved_concurrency" {
  description = "Steady-state reserved concurrency for the dealers function. Isolation sets this to 0; recovery sets it back to this value."
  type        = number
  default     = 5
}
output "api_base_url" {
  description = "HTTP API base URL (no trailing slash)."
  value       = aws_apigatewayv2_api.portal.api_endpoint
}

output "dealers_url" {
  description = "Dealers lookup URL."
  value       = "${aws_apigatewayv2_api.portal.api_endpoint}/dealers"
}

output "status_url" {
  description = "Status page URL."
  value       = "${aws_apigatewayv2_api.portal.api_endpoint}/status"
}

output "dealers_function_name" {
  value = aws_lambda_function.dealers.function_name
}

output "status_function_name" {
  value = aws_lambda_function.status.function_name
}

output "contain_function_name" {
  value = aws_lambda_function.contain.function_name
}

output "dealers_table_name" {
  value = aws_dynamodb_table.dealers.name
}

output "alerts_topic_arn" {
  value = aws_sns_topic.alerts.arn
}
data "aws_caller_identity" "current" {}
data "aws_partition" "current" {}
data "aws_region" "current" {}

locals {
  dealers_fn = "${var.project_prefix}-dealers"
  status_fn  = "${var.project_prefix}-status"
  contain_fn = "${var.project_prefix}-contain"
  table_name = "${var.project_prefix}-dealers"
  account_id = data.aws_caller_identity.current.account_id
  partition  = data.aws_partition.current.partition
  region     = data.aws_region.current.id
}

# ---------- Fake data store ----------
resource "aws_dynamodb_table" "dealers" {
  name         = local.table_name
  billing_mode = "PROVISIONED"
  hash_key     = "dealer_id"

  read_capacity  = 5
  write_capacity = 5

  attribute {
    name = "dealer_id"
    type = "S"
  }

  point_in_time_recovery {
    enabled = false
  }

  # checkov:skip=CKV_AWS_119:SSE-S3/AWS-owned encryption is the default and avoids a dedicated KMS key charge for this weekend lab.
  # checkov:skip=CKV2_AWS_16:PITR is an optional paid extension; MVP uses an on-demand backup you create during the recovery test.
}

# ---------- Alerts ----------
resource "aws_sns_topic" "alerts" {
  name = "${var.project_prefix}-alerts"
  # checkov:skip=CKV_AWS_26:Default AWS-managed encryption avoids a customer-managed KMS key charge for this short lab.
}

resource "aws_sns_topic_subscription" "email" {
  topic_arn = aws_sns_topic.alerts.arn
  protocol  = "email"
  endpoint  = var.alert_email
}

# ---------- Shared Lambda trust ----------
data "aws_iam_policy_document" "lambda_assume" {
  statement {
    sid     = "AllowLambdaAssume"
    effect  = "Allow"
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }
}

# ---------- Dealers API role ----------
resource "aws_iam_role" "dealers" {
  name               = "${var.project_prefix}-dealers-role"
  assume_role_policy = data.aws_iam_policy_document.lambda_assume.json
}

data "aws_iam_policy_document" "dealers" {
  statement {
    sid       = "WriteOwnLogs"
    effect    = "Allow"
    actions   = ["logs:CreateLogStream", "logs:PutLogEvents"]
    resources = ["${aws_cloudwatch_log_group.dealers.arn}:*"]
  }

  statement {
    sid    = "ReadFakeDealers"
    effect = "Allow"
    actions = [
      "dynamodb:GetItem",
      "dynamodb:Scan"
    ]
    resources = [aws_dynamodb_table.dealers.arn]
  }
}

resource "aws_iam_role_policy" "dealers" {
  name   = "dealers-least-privilege"
  role   = aws_iam_role.dealers.id
  policy = data.aws_iam_policy_document.dealers.json
}

# ---------- Status role ----------
resource "aws_iam_role" "status" {
  name               = "${var.project_prefix}-status-role"
  assume_role_policy = data.aws_iam_policy_document.lambda_assume.json
}

data "aws_iam_policy_document" "status" {
  statement {
    sid       = "WriteOwnLogs"
    effect    = "Allow"
    actions   = ["logs:CreateLogStream", "logs:PutLogEvents"]
    resources = ["${aws_cloudwatch_log_group.status.arn}:*"]
  }

  statement {
    sid       = "ReadDealersConcurrencyOnly"
    effect    = "Allow"
    actions   = ["lambda:GetFunctionConcurrency"]
    resources = ["arn:${local.partition}:lambda:${local.region}:${local.account_id}:function:${local.dealers_fn}"]
  }
}

resource "aws_iam_role_policy" "status" {
  name   = "status-least-privilege"
  role   = aws_iam_role.status.id
  policy = data.aws_iam_policy_document.status.json
}

# ---------- Containment role ----------
resource "aws_iam_role" "contain" {
  name               = "${var.project_prefix}-contain-role"
  assume_role_policy = data.aws_iam_policy_document.lambda_assume.json
}

data "aws_iam_policy_document" "contain" {
  statement {
    sid       = "WriteOwnLogs"
    effect    = "Allow"
    actions   = ["logs:CreateLogStream", "logs:PutLogEvents"]
    resources = ["${aws_cloudwatch_log_group.contain.arn}:*"]
  }

  statement {
    sid    = "IsolateDealersFunctionOnly"
    effect = "Allow"
    actions = [
      "lambda:PutFunctionConcurrency",
      "lambda:GetFunctionConcurrency"
    ]
    resources = ["arn:${local.partition}:lambda:${local.region}:${local.account_id}:function:${local.dealers_fn}"]
  }

  statement {
    sid       = "PublishDrillAlert"
    effect    = "Allow"
    actions   = ["sns:Publish"]
    resources = [aws_sns_topic.alerts.arn]
  }
}

resource "aws_iam_role_policy" "contain" {
  name   = "contain-least-privilege"
  role   = aws_iam_role.contain.id
  policy = data.aws_iam_policy_document.contain.json
}

# ---------- Log groups ----------
resource "aws_cloudwatch_log_group" "dealers" {
  name              = "/aws/lambda/${local.dealers_fn}"
  retention_in_days = 7
}

resource "aws_cloudwatch_log_group" "status" {
  name              = "/aws/lambda/${local.status_fn}"
  retention_in_days = 7
}

resource "aws_cloudwatch_log_group" "contain" {
  name              = "/aws/lambda/${local.contain_fn}"
  retention_in_days = 7
}

# ---------- Packages ----------
data "archive_file" "dealers" {
  type        = "zip"
  source_dir  = "${path.module}/lambda/dealers"
  output_path = "${path.module}/build/dealers.zip"
}

data "archive_file" "status" {
  type        = "zip"
  source_dir  = "${path.module}/lambda/status"
  output_path = "${path.module}/build/status.zip"
}

data "archive_file" "contain" {
  type        = "zip"
  source_dir  = "${path.module}/lambda/contain"
  output_path = "${path.module}/build/contain.zip"
}

# ---------- Functions ----------
resource "aws_lambda_function" "dealers" {
  function_name    = local.dealers_fn
  role             = aws_iam_role.dealers.arn
  filename         = data.archive_file.dealers.output_path
  source_code_hash = data.archive_file.dealers.output_base64sha256
  handler          = "handler.handler"
  runtime          = "python3.12"
  timeout          = 10
  memory_size      = 128

  reserved_concurrent_executions = var.dealers_reserved_concurrency

  environment {
    variables = {
      DEALERS_TABLE = aws_dynamodb_table.dealers.name
    }
  }

  depends_on = [aws_cloudwatch_log_group.dealers]

  # checkov:skip=CKV_AWS_115:Reserved concurrency is set. Isolation later sets it to 0 on purpose.
  # checkov:skip=CKV_AWS_116:No DLQ in the MVP; failed invokes are visible in CloudWatch Logs.
  # checkov:skip=CKV_AWS_117:No VPC. This function only calls DynamoDB's public AWS API.
  # checkov:skip=CKV_AWS_50:X-Ray is omitted to avoid extra tracing cost in a weekend lab.
  # checkov:skip=CKV_AWS_173:Environment variable is a table name, not a secret.
  # checkov:skip=CKV_AWS_272:Code signing is omitted for this lab.
}

resource "aws_lambda_function" "status" {
  function_name    = local.status_fn
  role             = aws_iam_role.status.arn
  filename         = data.archive_file.status.output_path
  source_code_hash = data.archive_file.status.output_base64sha256
  handler          = "handler.handler"
  runtime          = "python3.12"
  timeout          = 10
  memory_size      = 128

  reserved_concurrent_executions = 2

  environment {
    variables = {
      DEALERS_FUNCTION_NAME = aws_lambda_function.dealers.function_name
    }
  }

  depends_on = [aws_cloudwatch_log_group.status]

  # checkov:skip=CKV_AWS_116:No DLQ in the MVP.
  # checkov:skip=CKV_AWS_117:No VPC; only calls GetFunctionConcurrency.
  # checkov:skip=CKV_AWS_50:X-Ray omitted.
  # checkov:skip=CKV_AWS_173:Environment variable is a function name, not a secret.
  # checkov:skip=CKV_AWS_272:Code signing omitted.
}

resource "aws_lambda_function" "contain" {
  function_name    = local.contain_fn
  role             = aws_iam_role.contain.arn
  filename         = data.archive_file.contain.output_path
  source_code_hash = data.archive_file.contain.output_base64sha256
  handler          = "handler.handler"
  runtime          = "python3.12"
  timeout          = 15
  memory_size      = 128

  environment {
    variables = {
      DEALERS_FUNCTION_NAME = aws_lambda_function.dealers.function_name
      TOPIC_ARN             = aws_sns_topic.alerts.arn
    }
  }

  depends_on = [aws_cloudwatch_log_group.contain]

  # checkov:skip=CKV_AWS_115:Unreserved is fine; this function runs only during a drill.
  # checkov:skip=CKV_AWS_116:No DLQ in the MVP.
  # checkov:skip=CKV_AWS_117:No VPC.
  # checkov:skip=CKV_AWS_50:X-Ray omitted.
  # checkov:skip=CKV_AWS_173:No secrets in environment variables.
  # checkov:skip=CKV_AWS_272:Code signing omitted.
}

# ---------- HTTP API ----------
resource "aws_apigatewayv2_api" "portal" {
  name          = "${var.project_prefix}-http"
  protocol_type = "HTTP"
}

resource "aws_apigatewayv2_integration" "dealers" {
  api_id                 = aws_apigatewayv2_api.portal.id
  integration_type       = "AWS_PROXY"
  integration_uri        = aws_lambda_function.dealers.invoke_arn
  payload_format_version = "2.0"
}

resource "aws_apigatewayv2_integration" "status" {
  api_id                 = aws_apigatewayv2_api.portal.id
  integration_type       = "AWS_PROXY"
  integration_uri        = aws_lambda_function.status.invoke_arn
  payload_format_version = "2.0"
}

resource "aws_apigatewayv2_route" "dealers" {
  api_id    = aws_apigatewayv2_api.portal.id
  route_key = "GET /dealers"
  target    = "integrations/${aws_apigatewayv2_integration.dealers.id}"
}

resource "aws_apigatewayv2_route" "status" {
  api_id    = aws_apigatewayv2_api.portal.id
  route_key = "GET /status"
  target    = "integrations/${aws_apigatewayv2_integration.status.id}"
}

resource "aws_apigatewayv2_stage" "default" {
  api_id      = aws_apigatewayv2_api.portal.id
  name        = "$default"
  auto_deploy = true

  default_route_settings {
    throttling_burst_limit = 50
    throttling_rate_limit  = 25
  }
}

resource "aws_lambda_permission" "dealers_apigw" {
  statement_id  = "AllowHttpApiDealers"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.dealers.function_name
  principal     = "apigateway.amazonaws.com"
  source_arn    = "${aws_apigatewayv2_api.portal.execution_arn}/*/*/dealers"
}

resource "aws_lambda_permission" "status_apigw" {
  statement_id  = "AllowHttpApiStatus"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.status.function_name
  principal     = "apigateway.amazonaws.com"
  source_arn    = "${aws_apigatewayv2_api.portal.execution_arn}/*/*/status"
}

# ---------- Synthetic event -> containment ----------
resource "aws_cloudwatch_event_rule" "isolation_drill" {
  name        = "${var.project_prefix}-isolation-drill"
  description = "Matches only labeled practice isolation-drill events. Not a real-attack detector."

  event_pattern = jsonencode({
    source      = ["portal.security.drill"]
    detail-type = ["IsolationDrill"]
    detail = {
      mode = ["isolate"]
    }
  })
}

resource "aws_cloudwatch_event_target" "contain" {
  rule      = aws_cloudwatch_event_rule.isolation_drill.name
  target_id = "containment-lambda"
  arn       = aws_lambda_function.contain.arn
}

resource "aws_lambda_permission" "contain_events" {
  statement_id  = "AllowEventBridgeContain"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.contain.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.isolation_drill.arn
}
aws_region     = "us-east-2"
project_prefix = "p02-portal"
alert_email    = "YOUR_EMAIL@example.com"

Application

"""Practice dealers API. Reads only fake lab records. No real dealer or customer data."""

from __future__ import annotations

import json
import os
from decimal import Decimal

import boto3
from botocore.exceptions import ClientError

TABLE_NAME = os.environ.get("DEALERS_TABLE", "")
dynamodb = boto3.resource("dynamodb")


class DecimalEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, Decimal):
            return int(o) if o % 1 == 0 else float(o)
        return super().default(o)


def _response(status: int, body: dict) -> dict:
    return {
        "statusCode": status,
        "headers": {
            "Content-Type": "application/json",
            "Cache-Control": "no-store",
        },
        "body": json.dumps(body, cls=DecimalEncoder),
    }


def handler(event, context):
    if not TABLE_NAME:
        return _response(500, {"error": "missing_table_env"})

    table = dynamodb.Table(TABLE_NAME)
    params = event.get("queryStringParameters") or {}
    dealer_id = params.get("id")

    try:
        if dealer_id:
            result = table.get_item(Key={"dealer_id": dealer_id})
            item = result.get("Item")
            if not item:
                return _response(404, {"error": "not_found", "dealer_id": dealer_id})
            return _response(200, {"dealer": item, "note": "FAKE LAB RECORD"})

        result = table.scan(Limit=25)
        return _response(
            200,
            {
                "dealers": result.get("Items", []),
                "count": result.get("Count", 0),
                "note": "FAKE LAB RECORDS",
            },
        )
    except ClientError as exc:
        print(json.dumps({"level": "error", "error": str(exc)}))
        return _response(500, {"error": "backend_unavailable"})
"""Practice status page. Reports whether the dealers function is isolated."""

from __future__ import annotations

import html
import json
import os

import boto3
from botocore.exceptions import ClientError

DEALERS_FUNCTION_NAME = os.environ.get("DEALERS_FUNCTION_NAME", "")
lambda_client = boto3.client("lambda")


def _dealers_state():
    if not DEALERS_FUNCTION_NAME:
        return "unknown", None
    try:
        response = lambda_client.get_function_concurrency(FunctionName=DEALERS_FUNCTION_NAME)
    except ClientError:
        return "unknown", None
    reserved = response.get("ReservedConcurrentExecutions")
    return ("isolated" if reserved == 0 else "available"), reserved


def _payload(dealers_state, reserved):
    if dealers_state == "isolated":
        message = (
            "Dealers lookup is isolated (reserved concurrency 0). "
            "This is a practice drill, not a report about any real company outage."
        )
    else:
        message = "Practice portal components report available."
    return {
        "component": "practice-dealer-portal",
        "drill_status_page": True,
        "dealers_api": dealers_state,
        "status_api": "available",
        "dealers_reserved_concurrency": reserved,
        "message": message,
    }


def _html(payload: dict) -> str:
    color = "#b45309" if payload["dealers_api"] == "isolated" else "#047857"
    return f"""<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Practice dealer portal status</title>
  <style>
    body {{ font-family: Georgia, serif; background: #f8fafc; color: #0f172a; margin: 0; }}
    main {{ max-width: 40rem; margin: 3rem auto; background: white; padding: 2rem; border-radius: 12px; box-shadow: 0 8px 24px rgba(15,23,42,.08); }}
    .badge {{ display: inline-block; padding: .25rem .75rem; border-radius: 999px; background: {color}; color: white; font-weight: 700; }}
    code {{ background: #e2e8f0; padding: .1rem .35rem; border-radius: 4px; }}
  </style>
</head>
<body>
<main>
  <p class="badge">{html.escape(payload['dealers_api'].upper())}</p>
  <h1>Practice dealer portal</h1>
  <p>{html.escape(payload['message'])}</p>
  <ul>
    <li>Dealers API: <code>{html.escape(payload['dealers_api'])}</code></li>
    <li>Status API: <code>{html.escape(payload['status_api'])}</code></li>
    <li>Dealers reserved concurrency: <code>{html.escape(str(payload['dealers_reserved_concurrency']))}</code></li>
  </ul>
  <p>Invented lab architecture. Not an official company status page.</p>
</main>
</body>
</html>"""


def _wants_html(event) -> bool:
    params = event.get("queryStringParameters") or {}
    fmt = (params.get("format") or "").lower()
    if fmt == "json":
        return False
    if fmt == "html":
        return True
    headers = {str(k).lower(): str(v) for k, v in (event.get("headers") or {}).items()}
    accept = headers.get("accept", "")
    return "text/html" in accept or "application/json" not in accept


def handler(event, context):
    dealers_state, reserved = _dealers_state()
    payload = _payload(dealers_state, reserved)
    if _wants_html(event):
        return {
            "statusCode": 200,
            "headers": {"Content-Type": "text/html; charset=utf-8", "Cache-Control": "no-store"},
            "body": _html(payload),
        }
    return {
        "statusCode": 200,
        "headers": {"Content-Type": "application/json", "Cache-Control": "no-store"},
        "body": json.dumps(payload),
    }
"""Isolate the dealers function only after a labeled drill event."""

from __future__ import annotations

import json
import os

import boto3
from botocore.exceptions import ClientError

ALLOWED_SOURCE = "portal.security.drill"
ALLOWED_DETAIL_TYPE = "IsolationDrill"
TARGET_FUNCTION = os.environ.get("DEALERS_FUNCTION_NAME", "")
TOPIC_ARN = os.environ.get("TOPIC_ARN", "")

lambda_client = boto3.client("lambda")
sns_client = boto3.client("sns")


def handler(event, context):
    source = event.get("source")
    detail_type = event.get("detail-type")
    detail = event.get("detail") or {}
    if isinstance(detail, str):
        try:
            detail = json.loads(detail)
        except json.JSONDecodeError:
            detail = {}

    if source != ALLOWED_SOURCE or detail_type != ALLOWED_DETAIL_TYPE:
        print(json.dumps({
            "action": "ignored",
            "reason": "not_a_labeled_drill",
            "source": source,
            "detail_type": detail_type,
        }))
        return {"action": "ignored", "reason": "not_a_labeled_drill"}

    if detail.get("mode") != "isolate":
        print(json.dumps({"action": "ignored", "reason": "mode_not_isolate"}))
        return {"action": "ignored", "reason": "mode_not_isolate"}

    if not TARGET_FUNCTION:
        return {"action": "error", "reason": "missing_target_env"}

    if detail.get("function_name") and detail.get("function_name") != TARGET_FUNCTION:
        print(json.dumps({"action": "ignored", "reason": "event_cannot_choose_target"}))
        return {"action": "ignored", "reason": "event_cannot_choose_target"}

    try:
        lambda_client.put_function_concurrency(
            FunctionName=TARGET_FUNCTION,
            ReservedConcurrentExecutions=0,
        )
    except ClientError as exc:
        print(json.dumps({"action": "error", "error": str(exc)}))
        return {"action": "error", "reason": "put_function_concurrency_failed"}

    message = (
        "DRILL ONLY. Isolated " + TARGET_FUNCTION + " by setting reserved concurrency to 0. "
        "This is not a real-attack detection and is not a statement about any company incident. "
        "Re-enable with: aws lambda put-function-concurrency --function-name "
        + TARGET_FUNCTION + " --reserved-concurrent-executions 5"
    )

    if TOPIC_ARN:
        try:
            sns_client.publish(
                TopicArn=TOPIC_ARN,
                Subject="DRILL: practice dealers API isolated",
                Message=message,
            )
        except ClientError as exc:
            print(json.dumps({"level": "warn", "sns_error": str(exc)}))

    result = {
        "action": "isolated",
        "function": TARGET_FUNCTION,
        "reserved_concurrent_executions": 0,
    }
    print(json.dumps(result))
    return result
[
  {
    "dealer_id": "DLR-001",
    "name": "Sampleville Practice Motors",
    "city": "Sampleville",
    "state": "OH",
    "fake_parts_ready": 12,
    "note": "FAKE LAB RECORD — not a real dealership"
  },
  {
    "dealer_id": "DLR-002",
    "name": "North Example Autos",
    "city": "Exampleton",
    "state": "OH",
    "fake_parts_ready": 4,
    "note": "FAKE LAB RECORD — not a real dealership"
  },
  {
    "dealer_id": "DLR-003",
    "name": "Lakeside Training Garage",
    "city": "Trainington",
    "state": "OH",
    "fake_parts_ready": 9,
    "note": "FAKE LAB RECORD — not a real dealership"
  }
]
[
  {
    "Source": "portal.security.drill",
    "DetailType": "IsolationDrill",
    "Detail": "{\"mode\":\"isolate\",\"reason\":\"labeled practice drill — not a real attack\"}",
    "EventBusName": "default"
  }
]
[
  {
    "Source": "portal.security.unrelated",
    "DetailType": "IsolationDrill",
    "Detail": "{\"mode\":\"isolate\"}",
    "EventBusName": "default"
  }
]
[
  {
    "Source": "portal.security.drill",
    "DetailType": "IsolationDrill",
    "Detail": "{\"mode\":\"isolate\",\"function_name\":\"p02-portal-status\"}",
    "EventBusName": "default"
  }
]

Automation

#!/usr/bin/env bash
# Load only fake records from data/fake_dealers.json into the project table.
set -euo pipefail
REGION="${AWS_REGION:-us-east-2}"
TABLE="${TABLE_NAME:-$(terraform output -raw dealers_table_name)}"
DATA_FILE="${1:-data/fake_dealers.json}"

if [[ ! -f "$DATA_FILE" ]]; then
  echo "Missing $DATA_FILE" >&2
  exit 1
fi

echo "Seeding fake records into ${TABLE} in ${REGION}"
python3 - "$TABLE" "$REGION" "$DATA_FILE" <<'PY'
import json
import sys
import boto3

table_name, region, path = sys.argv[1], sys.argv[2], sys.argv[3]
items = json.loads(open(path, encoding="utf-8").read())
table = boto3.resource("dynamodb", region_name=region).Table(table_name)

with table.batch_writer() as batch:
    for item in items:
        batch.put_item(Item=item)
        print(f"wrote {item['dealer_id']} (FAKE LAB RECORD)")

print(f"seeded {len(items)} fake items")
PY
#!/usr/bin/env bash
# Capture repeatable evidence for the dealers route, status route, and isolation control.
set -euo pipefail
REGION="${AWS_REGION:-us-east-2}"
mkdir -p evidence
API="${API_BASE_URL:-$(terraform output -raw api_base_url)}"
DEALERS_FN="${DEALERS_FN:-$(terraform output -raw dealers_function_name)}"
LABEL="${1:-baseline}"

curl -sS -D "evidence/${LABEL}-dealers.headers" -o "evidence/${LABEL}-dealers.body" "${API}/dealers"
curl -sS -D "evidence/${LABEL}-status.headers" -o "evidence/${LABEL}-status.body" "${API}/status?format=json"
aws lambda get-function-concurrency --region "$REGION" --function-name "$DEALERS_FN" > "evidence/${LABEL}-concurrency.json"

echo "Saved evidence/${LABEL}-* for ${API}"
echo "Dealers HTTP $(awk 'NR==1{print $2}' evidence/${LABEL}-dealers.headers)"
echo "Status HTTP $(awk 'NR==1{print $2}' evidence/${LABEL}-status.headers)"
cat "evidence/${LABEL}-concurrency.json"
#!/usr/bin/env bash
# Destroy ONLY resources created by this Terraform root. Leaves unrelated account resources alone.
set -euo pipefail
REGION="${AWS_REGION:-us-east-2}"
PREFIX="${PROJECT_PREFIX:-p02-portal}"

echo "This will destroy the Terraform-managed ${PREFIX} stack in ${REGION}."
echo "It will not delete CloudTrail, Config, SCPs, or other projects."
read -r -p "Type DESTROY to continue: " confirm
if [[ "$confirm" != "DESTROY" ]]; then
  echo "Aborted."
  exit 1
fi

TABLE="${PREFIX}-dealers"
if aws dynamodb describe-table --region "$REGION" --table-name "$TABLE" >/dev/null 2>&1; then
  mapfile -t ARNS < <(aws dynamodb list-backups --region "$REGION" --table-name "$TABLE" --query 'BackupSummaries[].BackupArn' --output text | tr '\t' '\n')
  for arn in "${ARNS[@]:-}"; do
    [[ -z "$arn" || "$arn" == "None" ]] && continue
    echo "Deleting leftover backup ${arn}"
    aws dynamodb delete-backup --region "$REGION" --backup-arn "$arn" || true
  done
fi

RESTORE="${PREFIX}-dealers-restored"
if aws dynamodb describe-table --region "$REGION" --table-name "$RESTORE" >/dev/null 2>&1; then
  echo "Deleting leftover restored table ${RESTORE}"
  aws dynamodb delete-table --region "$REGION" --table-name "$RESTORE"
fi

terraform destroy -auto-approve
echo "Cleanup finished. Unrelated account resources were not targeted."
name: security-gate

# Lint and scan Terraform plus offline unit tests.
# This workflow does not call AWS. It needs no access keys and no OIDC.
# If you later add a deploy job that assumes an AWS role, use GitHub OIDC
# (permissions: id-token: write) instead of long-lived access keys. See the guide.

on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - name: Check out the repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: "1.9.8"
          terraform_wrapper: false

      - name: terraform fmt
        run: terraform fmt -check -recursive

      - name: terraform init without a backend
        run: terraform init -backend=false -input=false

      - name: terraform validate
        run: terraform validate

      - name: Setup TFLint
        uses: terraform-linters/setup-tflint@v4
        with:
          tflint_version: v0.55.1

      - name: Run TFLint
        run: |
          tflint --init
          tflint --recursive

      - name: Run Checkov
        uses: bridgecrewio/checkov-action@v12
        with:
          directory: .
          framework: terraform
          soft_fail: false
          skip_path: /tests/

      - name: Install test dependencies
        run: pip install boto3 pytest

      - name: Run offline handler tests
        run: python -m pytest -q tests/test_handlers.py
plugin "aws" {
  enabled = true
  version = "0.37.0"
  source  = "github.com/terraform-linters/tflint-ruleset-aws"
}
# Terraform local state and cache
.terraform/
*.tfstate
*.tfstate.*
crash.log
crash.*.log
override.tf
override.tf.json
*_override.tf
*_override.tf.json
tfplan

# Build and local Python artifacts
build/
*.zip
__pycache__/
.pytest_cache/
.venv/
*.pyc

# Local values and evidence (may include account IDs, URLs, email, log excerpts)
terraform.tfvars
evidence/
"""Offline tests. These do not deploy to AWS and do not prove live API behavior."""

from __future__ import annotations

import importlib.util
import json
import os
import sys
from decimal import Decimal
from pathlib import Path
from unittest.mock import MagicMock, patch

ROOT = Path(__file__).resolve().parents[1]

os.environ.setdefault("AWS_DEFAULT_REGION", "us-east-2")
os.environ.setdefault("AWS_ACCESS_KEY_ID", "testing")
os.environ.setdefault("AWS_SECRET_ACCESS_KEY", "testing")
os.environ["DEALERS_TABLE"] = "p02-portal-dealers"
os.environ["DEALERS_FUNCTION_NAME"] = "p02-portal-dealers"
os.environ["TOPIC_ARN"] = "arn:aws:sns:us-east-2:111122223333:p02-portal-alerts"


def load(name, relative):
    path = ROOT / relative
    spec = importlib.util.spec_from_file_location(name, path)
    module = importlib.util.module_from_spec(spec)
    sys.modules[name] = module
    spec.loader.exec_module(module)
    return module


dealers = load("p02_dealers", "lambda/dealers/handler.py")
status = load("p02_status", "lambda/status/handler.py")
contain = load("p02_contain", "lambda/contain/handler.py")


def test_dealers_get_one():
    table = MagicMock()
    table.get_item.return_value = {
        "Item": {"dealer_id": "DLR-001", "fake_parts_ready": Decimal("12")}
    }
    with patch.object(dealers.dynamodb, "Table", return_value=table):
        result = dealers.handler({"queryStringParameters": {"id": "DLR-001"}}, None)
    assert result["statusCode"] == 200
    body = json.loads(result["body"])
    assert body["dealer"]["fake_parts_ready"] == 12
    assert body["note"] == "FAKE LAB RECORD"


def test_dealers_not_found():
    table = MagicMock()
    table.get_item.return_value = {}
    with patch.object(dealers.dynamodb, "Table", return_value=table):
        result = dealers.handler({"queryStringParameters": {"id": "DLR-999"}}, None)
    assert result["statusCode"] == 404


def test_dealers_scan_list():
    table = MagicMock()
    table.scan.return_value = {"Items": [{"dealer_id": "DLR-001"}], "Count": 1}
    with patch.object(dealers.dynamodb, "Table", return_value=table):
        result = dealers.handler({"queryStringParameters": None}, None)
    body = json.loads(result["body"])
    assert body["count"] == 1
    assert body["note"] == "FAKE LAB RECORDS"


def test_status_reports_isolated_html_and_json():
    with patch.object(
        status.lambda_client,
        "get_function_concurrency",
        return_value={"ReservedConcurrentExecutions": 0},
    ):
        html_result = status.handler(
            {"queryStringParameters": {"format": "html"}, "headers": {}}, None
        )
        json_result = status.handler(
            {"queryStringParameters": {"format": "json"}, "headers": {}}, None
        )
    assert html_result["headers"]["Content-Type"].startswith("text/html")
    assert "isolated" in html_result["body"]
    assert json.loads(json_result["body"])["status_api"] == "available"


def test_status_reports_available():
    with patch.object(
        status.lambda_client,
        "get_function_concurrency",
        return_value={"ReservedConcurrentExecutions": 5},
    ):
        result = status.handler({"queryStringParameters": {"format": "json"}}, None)
    body = json.loads(result["body"])
    assert body["dealers_api"] == "available"
    assert body["dealers_reserved_concurrency"] == 5


def test_contain_ignores_unlabeled_event():
    with patch.object(contain.lambda_client, "put_function_concurrency") as put:
        result = contain.handler(
            {"source": "something.else", "detail-type": "IsolationDrill", "detail": {"mode": "isolate"}},
            None,
        )
    assert result["action"] == "ignored"
    put.assert_not_called()


def test_contain_ignores_event_chosen_target():
    event = {
        "source": "portal.security.drill",
        "detail-type": "IsolationDrill",
        "detail": {"mode": "isolate", "function_name": "p02-portal-status"},
    }
    with patch.object(contain.lambda_client, "put_function_concurrency") as put:
        result = contain.handler(event, None)
    assert result["action"] == "ignored"
    put.assert_not_called()


def test_contain_isolates_env_target_only():
    event = {
        "source": "portal.security.drill",
        "detail-type": "IsolationDrill",
        "detail": {"mode": "isolate"},
    }
    with patch.object(contain.lambda_client, "put_function_concurrency") as put, patch.object(
        contain.sns_client, "publish", return_value={"MessageId": "1"}
    ):
        result = contain.handler(event, None)
    assert result["action"] == "isolated"
    put.assert_called_once_with(
        FunctionName="p02-portal-dealers", ReservedConcurrentExecutions=0
    )
  • Add the canonical repository overview and dependency lock files:
# Project 02 — Detect, Isolate, and Recover a Serverless Dealer Portal

Weekend lab inspired by Honda’s 2026 dealer-portal incident. Everything here is **invented practice data and architecture**. It is not Honda’s design, evidence, timeline, or root cause.

## MVP architecture

```text
Browser / curl
   ├── GET /dealers ──> API Gateway HTTP API ──> Lambda dealers ──> DynamoDB (fake data)
   └── GET /status  ──> API Gateway HTTP API ──> Lambda status ──> reads dealers Lambda concurrency

Synthetic labeled drill event
   └── EventBridge rule ──> containment Lambda
         ├── sets ONLY dealers Lambda reserved concurrency to 0
         ├── publishes "DRILL:" SNS email
         └── logs the action
```

## What this proves

- Public HTTP data path returns only fake dealer records.
- A separate status path stays available while dealers processing is isolated.
- The MVP “detector” is a narrow EventBridge matcher for a **synthetic event you publish yourself**. It is not a real-attack detector.
- Containment changes one lab-owned function only.
- Recovery is tested two ways:
  1. fast re-enable / Terraform drift correction, and
  2. DynamoDB on-demand backup restore into a new table.
- GitHub Actions performs linting, scanning, and offline tests **without AWS credentials**.

## Safe order

1. Copy `terraform.tfvars.example` to `terraform.tfvars`; add a reachable email.
2. Run the local validation commands from the lab guide.
3. Review `terraform plan`; all created names should begin with `p02-portal`.
4. Apply.
5. Confirm the SNS subscription.
6. Seed fake data with `scripts/seed_dealers.sh`.
7. Run the labeled drill.
8. Test containment, recovery, backup restore, and negative cases.
9. Run `scripts/cleanup.sh`.

## Cost note

Small AWS usage may be free under an existing Free Tier or credits, but zero cost is not guaranteed. On-demand backup restore, optional GuardDuty, PITR, extra logs, and alarms can add charges. Clean up promptly.

## Project boundary

The cleanup script destroys only resources recorded in this Terraform root and the optional fixed-name restore table/backup. It does not delete CloudTrail, AWS Config, SCPs, Project 1 resources, or unrelated account controls.
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/archive" {
  version     = "2.7.1"
  constraints = "~> 2.6"
  hashes = [
    "h1:1bBJdihS3Y7rINlt6JS1uXxCDSGlRgF8AObhxbPJvQs=",
    "zh:1983ef0e453eecd96ceaa5fc6b82484d6df0ca21e1f0c64de1373422636bd8f4",
    "zh:2dd03b433fd3f9ed6ccc0f2a499752121572c247e815786d2df3e4d958d23e70",
    "zh:2ff7c86a6b68f491c43b0db154e502493c59b626a5bbf12292b1a364e2545ef8",
    "zh:653ad57f9566c8ac2f6d17dd3b976b49d4c712446a2e32c45e909da3ac0e96c6",
    "zh:6886e8dc59b2c5082d337e1cb27b077d94a9883d156af3772d208ff651b2888c",
    "zh:793938513effa5e08d3b93a54339188218414330816e5aaf78e04f89328f363b",
    "zh:8456a7230ad4dddea42e47e9c443dd549dad0df8831cd03894537b9e9a43c8b4",
    "zh:8b4b2b2c8f3daf7a81d0ffc6c30e2f44004ae4129e2e3b63c057b459b72e04e6",
    "zh:9e00d6bdea16b598a0284f0c05bb8218ced01cf030df61d4625cfb7284bc1f78",
    "zh:ad75bd9755e14b1f94b61ef76a57ae658d81455ff10cb0c35bb3c47454185f30",
    "zh:c537aa1715644b10d7a5681b373801712b4cda3e2e7b8ae38eb14a6ac57d1091",
    "zh:e9be3da6d2f19d52e18a73470fe91ce71eb92e4251429f258ed0809942737424",
  ]
}

provider "registry.terraform.io/hashicorp/aws" {
  version     = "5.100.0"
  constraints = "~> 5.80"
  hashes = [
    "h1:fhHFjOukY6jZiVuLQ42svuBFuTdWzoRVRTmoCCcNcIg=",
    "zh:0c054e62179343c9ecf5f753b0d5c5e8760bf0af594b57f9ecb2bed7b4d63a28",
    "zh:1130f73138b0470be8776b54cb9b2b7ce34e313efdde1c8ff7baa523e309ed0b",
    "zh:28700070745da9169a7ed6a29e0e22dc96aa5a26c88002b0c50acc18520700e9",
    "zh:496485b0bf2c33d3762852dc6c9714b2da2dbacbf0877c6b50f93ac54e8397c3",
    "zh:66dbfc24685926a9d6142008c8c41cb4f70ff98c0250222d6890281f6a8d7c14",
    "zh:6c52c4289089862b67b9337eb6ab95a12d1b7a2a42f938110fe176ab1d2dd0f4",
    "zh:9fc2f5fa4d1c2b52a509a99bf02eabf0cf7ad564847347d3e0644eb455d67f37",
    "zh:a14f403e24dc7f911e46517317f279df4ee7c2330d8071c287ce1301508a1a5d",
    "zh:a1c98bdcc41e06d759f1fb7c9e1806d028519ff184ea79b03586df63f9f89d93",
    "zh:a899853aa04a0fcb4b7151fe4cffea6b2fd5adff0ca1f512f7fa3be0a1f11c42",
    "zh:eec4d354166f27321ba661e98d398bef2bb733e117d7ddbe0098b0f55dba7e20",
    "zh:fde9f3a614e1ee26b524591d08a1238ff03c957bfac51b1a3f9eedf103e1e261",
    "zh:fe6f1713c085e14fdb18d6378214a70b055d92ea4a95fc39dd91c20c5782c5c9",
    "zh:ff9b4c78ef5bc70bc4e2edf3b7fe3a1281022d9c36f7ea02556092b6685647d4",
  ]
}
Validate and deploy the stack
  • Install the test dependencies and run all eight offline checks:
python3 -m pip install --user pytest boto3
python3 -m pytest -q tests/test_handlers.py
terraform fmt -check -recursive
terraform init
terraform validate

You should see 8 passed and Success! The configuration is valid..

  • Record the SNS notification email: your notification email address.
  • Create the ignored local variable file:
cat > terraform.tfvars <<'EOF'
aws_region     = "us-east-2"
project_prefix = "p02-portal"
alert_email    = "[[ALERT_EMAIL=\"your notification email address\"]]"
EOF
terraform plan -out=tfplan
terraform show -no-color tfplan

Review the plan before billing starts

The plan should add only names beginning with p02-portal and must not destroy or modify Project 1 resources.

  • Apply only the saved plan and print the outputs:
terraform apply tfplan
terraform output
export API_BASE_URL=$(terraform output -raw api_base_url)

Expect several minutes while IAM, Lambda, API Gateway, DynamoDB, EventBridge, SNS, and log groups become ready.

  • Open the SNS confirmation email for the p02-portal-alerts topic.
  • Select Confirm subscription after its topic ARN matches the Terraform output.
Seed and verify the healthy baseline
  • Make the project scripts executable, seed the fake table, and capture the baseline:
chmod +x scripts/*.sh
./scripts/seed_dealers.sh
aws dynamodb scan --region us-east-2 --table-name p02-portal-dealers --consistent-read
./scripts/capture_baseline.sh before

What proves the baseline is healthy?

  • The scan should contain only DLR-001, DLR-002, and DLR-003.
  • The capture should report dealers HTTP 200 and status HTTP 200.
  • The concurrency file should show the reviewed value 5.

Your complete source, infrastructure, fake data, and healthy evidence are ready. Next, you will publish the labeled isolation drill.

Compromise the Dealer Record and Detect It

Your Terraform-managed dealer portal now has a healthy baseline. Both HTTP routes work, and the table contains only fake records.

The MVP detects a signal you publish yourself. A narrow Amazon EventBridge rule accepts only a labeled isolation drill before invoking the containment function.

In this step, get ready to:
  • Inspect the canonical drill event.
  • Publish the labeled event to the default bus.
  • Verify containment logs and the drill-only notification.
Publish the labeled isolation drill

Keep the claim accurate

This event is synthetic input from your own repository. It does not represent Honda telemetry, a real intrusion, or proof that EventBridge detected malicious activity.

  • Inspect the canonical event file by running:
cat data/drill-isolate.json

What must match the rule?

  • The source is portal.security.drill.
  • The detail type is IsolationDrill.
  • The detail JSON sets mode to isolate and labels the reason as a practice drill.
  • The event targets the default event bus.

Before you send the event, do you think EventBridge acceptance alone proves that containment completed?

  • Publish the labeled drill event by running:
aws events put-events \
  --region us-east-2 \
  --entries file://data/drill-isolate.json

What does PutEvents prove?

The --region flag keeps the event in Ohio. The --entries flag reads one or more events from the file.

A successful request returns FailedEntryCount set to 0 plus an event ID. Acceptance proves only that EventBridge received the event.

Verify the containment evidence

Give the rule about ten seconds to invoke Lambda and deliver its logs. A short delay is normal.

  • Read the recent containment logs by running:
aws logs tail /aws/lambda/p02-portal-contain \
  --since 5m \
  --region us-east-2

What should the logs prove?

Look for "action": "isolated" and function p02-portal-dealers.

The function reads the target from its environment rather than the event. It sets reserved concurrency to 0 and publishes a subject beginning with DRILL:.

  • Confirm the SNS email subject is DRILL: practice dealers API isolated.
  • Confirm the body names only p02-portal-dealers.
  • Keep the notification inside your authorized lab context.

Drill event did not complete?

If the event has a nonzero failed count, read its per-entry code and message. If no containment log appears, compare the file's source, detail type, and mode with the Terraform event pattern.

If logs appear but email does not, confirm the subscription is no longer pending.

Help me trace the missing drill evidence.

Your labeled signal reached the narrow containment path. Next, you will prove that only the dealers route stopped while the status route stayed healthy.

Isolate the Portal

The labeled EventBridge drill invoked the containment function. The dealers Lambda now has reserved concurrency 0.

A useful containment control must have a narrow blast radius. You will prove the dealers route is blocked while the independent status route remains healthy.

In this step, get ready to:
  • Capture the affected and healthy route results.
  • Run wrong-source and wrong-target negative tests.
  • Inspect the containment role's least-privilege policy.
Prove the isolation boundary

Before you check, which route should fail and which route should remain HTTP 200?

  • Capture the isolated state and both HTTP responses by running:
./scripts/capture_baseline.sh after-isolate
aws lambda get-function-concurrency \
  --region us-east-2 \
  --function-name p02-portal-dealers
curl -sS -D - -o /tmp/dealers-isolated.json "$API_BASE_URL/dealers"
echo
curl -sS "$API_BASE_URL/status?format=json"

What proves the blast radius is narrow?

  • The concurrency output should contain "ReservedConcurrentExecutions": 0.
  • The dealers request should not return a successful data response. Record the live 429 or 5xx status instead of assuming one.
  • The status request should return HTTP 200 with "dealers_api": "isolated" and "status_api": "available".
  • Confirm that no successful new handler output appears for the rejected dealers request:
aws logs tail /aws/lambda/p02-portal-dealers --since 5m --region us-east-2

Throttled requests can appear in Lambda metrics rather than Python output because the handler never starts.

Run the negative controls
  • Restore the reviewed steady-state concurrency and confirm the dealers route works:
aws lambda put-function-concurrency \
  --region us-east-2 \
  --function-name p02-portal-dealers \
  --reserved-concurrent-executions 5
curl -sS -o /dev/null -w "%{http_code}\n" "$API_BASE_URL/dealers"

You should see concurrency 5 and HTTP 200 before testing near-miss events.

  • Publish the wrong-source event and verify that concurrency remains 5:
aws events put-events --region us-east-2 --entries file://data/drill-negative-wrong-source.json
sleep 10
curl -sS -o /dev/null -w "%{http_code}\n" "$API_BASE_URL/dealers"
aws lambda get-function-concurrency --region us-east-2 --function-name p02-portal-dealers
  • Publish the event-supplied wrong target and prove that neither function is isolated:
aws events put-events --region us-east-2 --entries file://data/drill-negative-wrong-target.json
sleep 10
curl -sS -o /dev/null -w "dealers %{http_code}\n" "$API_BASE_URL/dealers"
curl -sS "$API_BASE_URL/status?format=json"
aws lambda get-function-concurrency --region us-east-2 --function-name p02-portal-status

What do the negative tests prove?

EventBridge can accept both events with FailedEntryCount set to 0 even when no rule matches.

The wrong-source event should leave dealers concurrency at 5. The wrong-target event should produce no isolation because the containment function ignores an event-supplied function name.

Inspect least privilege
  • Read the exact inline policy Terraform attached to the containment role:
aws iam get-role-policy \
  --role-name p02-portal-contain-role \
  --policy-name contain-least-privilege

What should the IAM proof show?

  • Concurrency actions should target only the p02-portal-dealers function ARN.
  • SNS publishing should target only the p02-portal-alerts topic ARN.
  • The status function ARN must not appear as a concurrency target.
  • A wildcard Lambda resource would weaken the blast-radius claim and must be fixed in Terraform.
  • Return the portal to the isolated state for the recovery exercise:
aws events put-events --region us-east-2 --entries file://data/drill-isolate.json
sleep 10
aws lambda get-function-concurrency --region us-east-2 --function-name p02-portal-dealers
curl -sS "$API_BASE_URL/status?format=json"

You proved targeted containment, healthy-component survival, negative controls, and least privilege. Next, you will recover the service state and restore a deleted fake record from backup.

Restore and Reopen the Portal

Your containment proof left the dealers function isolated while the status route stayed available. The negative tests showed that near-miss events do not trigger the control.

Recovery now covers two different failure modes. Terraform restores control-plane drift, while a DynamoDB backup restores deleted fake data into a new table.

In this step, get ready to:
  • Reopen the dealers function and repair Terraform drift.
  • Restore a deleted fake dealer record from an on-demand backup.
  • Push the canonical repository through its credential-free security gate.
Recover the service control plane
  • Restore the reviewed concurrency value and capture the healthy routes:
aws lambda put-function-concurrency \
  --region us-east-2 \
  --function-name p02-portal-dealers \
  --reserved-concurrent-executions 5
./scripts/capture_baseline.sh after-reenable

What proves fast recovery worked?

The concurrency response should show 5. The capture files should show HTTP 200 for dealers and status.

Putting the value back to 5 matches Terraform. Removing the limit with delete-function-concurrency would reopen the function but create configuration drift.

  • Run the labeled event again to recreate only the reviewed concurrency drift:
aws events put-events --region us-east-2 --entries file://data/drill-isolate.json
sleep 10
aws lambda get-function-concurrency --region us-east-2 --function-name p02-portal-dealers
  • Preview the Terraform recovery and review the plan:
terraform plan

Read the recovery plan

Terraform should propose changing only the dealers function's reserved concurrency from 0 to 5.

Stop if the plan destroys a resource, changes the status function, or names anything outside p02-portal.

  • Apply the reviewed drift repair and retest both routes:
terraform apply -auto-approve
curl -i "$API_BASE_URL/dealers"
curl -i "$API_BASE_URL/status"

The -auto-approve flag skips Terraform's second prompt. Use it here only because you reviewed the immediately preceding plan.

Restore the fake data

Pause before backup and restore

Creating an on-demand backup and restoring it can incur backup-storage and restore charges. The table is tiny, but zero cost is not guaranteed.

DynamoDB restore creates a new table. It cannot overwrite p02-portal-dealers.

  • Create the pre-delete backup and save its ARN:
export BACKUP_ARN=$(aws dynamodb create-backup \
  --region us-east-2 \
  --table-name p02-portal-dealers \
  --backup-name p02-portal-dealers-predelete \
  --query 'BackupDetails.BackupArn' \
  --output text)
echo "$BACKUP_ARN"
  • Wait until the backup becomes restorable:
until [ "$(aws dynamodb describe-backup --region us-east-2 --backup-arn "$BACKUP_ARN" --query BackupDescription.BackupDetails.BackupStatus --output text)" = "AVAILABLE" ]; do
  echo "Backup is still preparing. Waiting 20 seconds."
  sleep 20
done
echo "Backup is AVAILABLE."

Expect this asynchronous step to take a few minutes. Each message means DynamoDB is still preparing the backup.

  • Delete only the fake DLR-002 item and prove it is absent:
aws dynamodb delete-item \
  --region us-east-2 \
  --table-name p02-portal-dealers \
  --key '{"dealer_id":{"S":"DLR-002"}}'
aws dynamodb get-item \
  --region us-east-2 \
  --table-name p02-portal-dealers \
  --key '{"dealer_id":{"S":"DLR-002"}}'

The second response should omit Item. That is the controlled data-loss checkpoint.

  • Restore the backup into the fixed proof table and wait for it to become active:
aws dynamodb restore-table-from-backup \
  --region us-east-2 \
  --target-table-name p02-portal-dealers-restored \
  --backup-arn "$BACKUP_ARN"
aws dynamodb wait table-exists \
  --region us-east-2 \
  --table-name p02-portal-dealers-restored
  • Prove the restored table contains the deleted fake record:
aws dynamodb get-item \
  --region us-east-2 \
  --table-name p02-portal-dealers-restored \
  --key '{"dealer_id":{"S":"DLR-002"}}'

What does the restored item prove?

The response should contain DLR-002 from the snapshot taken before deletion.

The live source table remains separate. Re-run ./scripts/seed_dealers.sh if you want to restore the fake item to the live demo.

  • Delete the temporary proof table and backup after capturing the result:
aws dynamodb delete-table --region us-east-2 --table-name p02-portal-dealers-restored
aws dynamodb delete-backup --region us-east-2 --backup-arn "$BACKUP_ARN"
Run the repository security gate

The canonical .github/workflows/security-gate.yml workflow runs Terraform format, init, validate, TFLint, Checkov, and eight offline pytest checks. It does not call AWS and has contents: read only.

  • Copy your private GitHub repository URL and record it here: your private GitHub repository URL.
  • Connect the local repository and push the reviewed source:
git remote add origin [[YOUR_REPO_URL="your private GitHub repository URL"]]
git branch -M main
git add .
git commit -m "Record lab files"
git push -u origin main

What happens after the push?

  • The remote command connects this lab repository to your private GitHub repository.
  • The branch command standardizes the branch as main.
  • The commit records the reviewed files and stops if Git still needs your author identity.
  • The push triggers security-gate without AWS credentials.
  • Open the repository's Actions tab.
  • Open the latest security-gate run.
  • Confirm format, validate, TFLint, Checkov, and pytest pass.
  • Confirm no AWS access key, secret key, or OIDC deployment permission appears in this MVP workflow.

Security workflow failed?

A failing job names the command and file that needs work. Reproduce that command locally before committing the correction.

Help me diagnose the failed CI job.

Recovery is proven across service rollback, Terraform drift repair, and backup restore. Your source is also checked by a credential-free CI gate.

Secret mission

Choose an Optional Hardening Extension

Choose one professional hardening extension: DynamoDB PITR, GuardDuty samples, GitHub OIDC, HTTP API access logs, or a Lambda throttles alarm. Test it, capture evidence, and remove its resources.

Clean Up Your Resources

Clean Up Your Resources

Your AWS resources can keep generating usage-based charges after this lab. Choose whether to keep them available, pause portal processing, or delete the lab completely.

Cost warning

Keeping either Amazon DynamoDB table can continue to generate usage-based charges. Retaining the on-demand backup can add storage charges.

Future AWS Lambda invocations can add request or duration charges. Amazon CloudWatch Logs can add log-storage charges.

Free Tier allowances or credits might cover some usage. Zero cost is not guaranteed.

Resources you used:

  • The Terraform-managed p02-portal-http HTTP API with /dealers and /status routes.
  • The p02-portal-dealers, p02-portal-status, and p02-portal-contain Lambda functions plus their seven-day log groups.
  • The provisioned-capacity p02-portal-dealers DynamoDB table holding three fake dealer records.
  • Any on-demand backup created from p02-portal-dealers during the recovery proof.
  • The optional p02-portal-dealers-restored recovery table when it still exists.
  • The p02-portal-isolation-drill EventBridge rule, containment target, and Lambda invoke permission.
  • The p02-portal-alerts SNS topic and email subscription.
  • The three project IAM roles and their inline least-privilege policies.
  • The local p02-portal repository, Terraform state, plans, build files, and evidence files.

Keep everything running

No action is needed. Choose this option while you are still testing the portal and its recovery workflow.

  • Keep the dealers, status, and containment Lambda functions available.
  • Keep the HTTP API, DynamoDB table, EventBridge rule, SNS topic, IAM roles, and seven-day log groups available.
  • Keep the local Terraform state because it records exactly what this project owns.
  • Expect usage-based charges to continue while the resources remain.

Pause - I'll come back to this later

Leave the portal available while stopping new isolation drills. Disabling the EventBridge rule preserves the API, Lambda functions, table, topic, and Terraform state.

  • Disable only the project drill rule and read its state by running:
aws events disable-rule --region us-east-2 --name p02-portal-isolation-drill
aws events describe-rule --region us-east-2 --name p02-portal-isolation-drill --query State --output text

What does pausing change?

The first command leaves the rule in place but stops it from matching events. EventBridge can take a short time to propagate the change.

The second command reads the rule state. You should see DISABLED.

  • Re-enable the same drill rule and read its state by running:
aws events enable-rule --region us-east-2 --name p02-portal-isolation-drill
aws events describe-rule --region us-east-2 --name p02-portal-isolation-drill --query State --output text

What proves drills can resume?

The first command allows matching events again. The second command should print ENABLED after the change propagates.

Delete - I don't want to use this again

Deletion is permanent. The project boundary makes it safe: this path targets the local Terraform state, leftover backups for p02-portal-dealers, and the optional p02-portal-dealers-restored table.

Destroy only this project

This cleanup does not touch CloudTrail, AWS Config, service control policies, unrelated Lambda functions, other DynamoDB tables, or Project 1 buckets. Stop if your current directory or Terraform state belongs to another project.

  • Enter the repository and inspect its state before deletion by running:
cd "$HOME/p02-portal"
pwd
terraform state list
terraform output -json

Confirm the ownership boundary

The directory should end with /p02-portal. Review the state and outputs before continuing.

Every AWS name should use the p02-portal prefix. Stop if a Project 1 name or unrelated resource appears.

  • Compare your saved cleanup helper with the complete canonical file:
#!/usr/bin/env bash
# Destroy ONLY resources created by this Terraform root. Leaves unrelated account resources alone.
set -euo pipefail
REGION="${AWS_REGION:-us-east-2}"
PREFIX="${PROJECT_PREFIX:-p02-portal}"

echo "This will destroy the Terraform-managed ${PREFIX} stack in ${REGION}."
echo "It will not delete CloudTrail, Config, SCPs, or other projects."
read -r -p "Type DESTROY to continue: " confirm
if [[ "$confirm" != "DESTROY" ]]; then
  echo "Aborted."
  exit 1
fi

TABLE="${PREFIX}-dealers"
if aws dynamodb describe-table --region "$REGION" --table-name "$TABLE" >/dev/null 2>&1; then
  mapfile -t ARNS < <(aws dynamodb list-backups --region "$REGION" --table-name "$TABLE" --query 'BackupSummaries[].BackupArn' --output text | tr '\t' '\n')
  for arn in "${ARNS[@]:-}"; do
    [[ -z "$arn" || "$arn" == "None" ]] && continue
    echo "Deleting leftover backup ${arn}"
    aws dynamodb delete-backup --region "$REGION" --backup-arn "$arn" || true
  done
fi

RESTORE="${PREFIX}-dealers-restored"
if aws dynamodb describe-table --region "$REGION" --table-name "$RESTORE" >/dev/null 2>&1; then
  echo "Deleting leftover restored table ${RESTORE}"
  aws dynamodb delete-table --region "$REGION" --table-name "$RESTORE"
fi

terraform destroy -auto-approve
echo "Cleanup finished. Unrelated account resources were not targeted."

How does every line and flag work?

  • The shebang selects Bash. The comment states the project-only ownership boundary.
  • The set -euo pipefail line stops on failed commands, unset variables, and failed pipeline stages.
  • The REGION and PREFIX assignments use existing environment values or fall back to us-east-2 and p02-portal.
  • The two echo lines display the target and ownership statement before deletion.
  • The read -r -p command shows a prompt, reads backslashes literally, and saves the answer in confirm.
  • The confirmation condition accepts only the exact word DESTROY. Any other input exits with status 1.
  • The TABLE assignment constructs p02-portal-dealers from the reviewed prefix.
  • The first describe-table call uses --region and --table-name. Redirection hides output because the condition only needs the exit status.
  • The mapfile -t command reads backup ARNs into an array and removes trailing newlines.
  • Process substitution runs list-backups. Its --query selects backup ARNs, --output text removes JSON formatting, and tr converts tab separators into lines.
  • The for loop checks each ARN. The guard skips blank and None values.
  • The backup deletion uses the reviewed Region and ARN. The || true suffix lets cleanup continue when a backup has already disappeared.
  • The RESTORE assignment constructs p02-portal-dealers-restored. The next condition deletes that table only when it exists.
  • The terraform destroy -auto-approve command removes resources recorded in this root's state without a second Terraform prompt. The earlier DESTROY prompt is the safety gate.
  • The final line confirms completion and restates that unrelated account resources were outside the target.
  • Run the canonical helper and type DESTROY only after reviewing the project boundary:
chmod +x scripts/cleanup.sh
./scripts/cleanup.sh

Terraform can take several minutes to remove API integrations, Lambda permissions, IAM roles, log groups, EventBridge targets, the SNS topic, and the DynamoDB table in dependency order.

  • Use this reviewed Terraform alternative to inspect a saved destroy plan:
terraform plan -destroy -out=destroy.tfplan
terraform show -no-color destroy.tfplan | tee destroy-plan.txt
grep -E 'Plan:|p02-portal|destroy' destroy-plan.txt

Review every planned deletion

The -destroy flag plans removal. The -out flag saves the reviewed actions, while show -no-color creates a plain-text copy.

Review the complete plan, not only the grep output. Continue only when every named lab resource starts with p02-portal and no Project 1 resource appears.

  • Apply only the saved destroy plan after completing that review:
terraform apply destroy.tfplan
  • Verify the project resources, optional restore table, and leftover backup are gone by running:
aws lambda get-function --function-name p02-portal-dealers --region us-east-2 || true
aws apigatewayv2 get-apis --region us-east-2 --query "Items[?Name=='p02-portal-http']"
aws dynamodb describe-table --table-name p02-portal-dealers --region us-east-2 || true
aws dynamodb describe-table --table-name p02-portal-dealers-restored --region us-east-2 || true
aws dynamodb list-backups --table-name p02-portal-dealers --region us-east-2 --query 'BackupSummaries[].BackupArn' --output text || true
Remove optional extension resources

The canonical cleanup script removes the MVP plus the on-demand proof table and backups. Run only the matching commands below for extensions you actually created.

  • Remove the PITR restore table, throttles alarm, and API access-log group when they exist:
aws dynamodb delete-table --region us-east-2 --table-name p02-portal-dealers-pitr-restored || true
aws cloudwatch delete-alarms --region us-east-2 --alarm-names p02-portal-dealers-throttles
aws logs delete-log-group --region us-east-2 --log-group-name /aws/apigateway/p02-portal-http || true
  • Remove the GuardDuty sample-notification target and rule when you created them:
aws events remove-targets --region us-east-2 --rule p02-portal-guardduty-samples --ids guardduty-sns || true
aws events delete-rule --region us-east-2 --name p02-portal-guardduty-samples || true
  • Open IAM, select Roles, open p02-portal-github-deploy, and delete it after removing any attached or inline policies.
  • Open IAM, select Identity providers, and remove token.actions.githubusercontent.com only if no other repository or role uses that provider.
  • Open GuardDuty and disable it only if you enabled it solely for this extension and no other authorized workload depends on it.

Keep shared controls outside the blast radius

The || true suffix lets the sequence continue when an optional resource is already absent. Commands without that suffix are idempotent delete operations for the named project alarm.

Shared IAM identity providers and GuardDuty detectors can serve other workloads. The ownership check prevents project cleanup from deleting shared account security controls.

💡 What proves cleanup is complete?

The Lambda and DynamoDB description commands should return ResourceNotFoundException. The HTTP API query should return an empty list.

The backup query should print no ARN. If DynamoDB reports ResourceNotFoundException because the source table is gone, that is also an expected clean result.

🙋‍♀️ A project resource remains?

If any command still returns a resource, confirm the account and us-east-2 Region. Match the name or ARN against this project's saved state before deleting anything manually.

Help me verify the remaining project resource safely.

Nice Work!

Nice Work!

Excellent work! You built and tested a Terraform-managed AWS incident-response lab with a public dealer route, an independent status route, targeted containment, and two recovery methods.

You've learned how to:

  • Built a serverless dealer portal where /dealers serves three fake DynamoDB records and /status reports the isolation state.
  • Ran a labeled synthetic drill that isolated only p02-portal-dealers, sent a drill-only SNS message, and preserved the status route.
  • Proved recovery and repository controls through concurrency rollback, Terraform drift repair, DynamoDB backup restore, and a credential-free CI security gate.
  • Secret Mission: Evaluated one optional hardening path covering PITR, GuardDuty samples, GitHub OIDC, API access logs, or a Lambda throttles alarm. You paired the extension with evidence and cleanup.

You also proved the full weekend architecture:

  • Built a Terraform-managed HTTP data plane with separate dealer and status paths, fake DynamoDB data, EventBridge routing, SNS notification, CloudWatch evidence, and narrowly scoped IAM roles.
  • Ran a labeled synthetic incident drill and kept the claim separate from the reported Honda facts and from any assertion of real threat detection.
  • Proved a narrow and reversible blast radius by blocking the dealers route, keeping status at HTTP 200, rejecting wrong-source and wrong-target events, inspecting IAM, and rolling concurrency back to 5.
  • Tested three recovery layers: fast service re-enable, Terraform drift repair, and DynamoDB backup restore into a new proof table.
  • Protected the repository with a credential-free CI gate covering Terraform format and validation, TFLint, Checkov, and offline Python tests.
  • Cleaned up only the project-owned resources recorded in Terraform state and verified that unrelated AWS and Project 1 resources remained untouched.

A precise way to describe this project:

  • Say, "I built a synthetic AWS incident-response lab that detects a labeled EventBridge drill, isolates one Lambda path, preserves a healthy status route, and tests two recovery methods."
  • Say, "I used fake dealer data and public incident reporting only as scenario context. I did not reproduce Honda's architecture or analyze Honda telemetry."
  • Support each claim with the saved plan, HTTP outputs, containment logs, negative-test results, IAM policy, recovery proof, and passing CI run.
  • Avoid saying you detected a real attacker, completed a forensic investigation, or implemented a full governance, risk, and compliance program.

Ready to quiz yourself?