Troubleshoot Private EC2 with Terraform

Use Terraform to diagnose Session Manager and S3 access on private EC2.

Introduction

30 Second Summary

A green status light can hide a system you still cannot reach. During an outage, strong evidence turns uncertainty into a fix you can explain.

In this project, you will build a disposable private Amazon EC2 operations lab on Amazon Web Services (AWS) using Terraform. You will use evidence-driven Terraform changes to restore AWS Systems Manager Session Manager access before limiting an Amazon S3 report upload to one approved prefix.

What You'll Build

Your final demo shows a private instance recovering remote administration through a reviewed permissions change before uploading its system report through a prefix-scoped policy.

By the end of this project, you'll have:

  • A repeatable private environment you can provision from version-controlled Terraform. You can prove the EC2 instance has no public administrative path.
  • A failure-driven troubleshooting workflow that traces missing Session Manager permissions across the compute-to-IAM dependency chain. The same workflow pinpoints why an S3 action fails against the wrong resource type.
  • An interview-ready evidence pack that pairs an uploaded system report with focused Git history. Your README.md, runbook.md, troubleshooting-debrief.md, and cleanup-checklist.md preserve the failure-to-repair story.
  • Secret Mission: a five-minute reproduction of the Systems Manager permission regression. You'll restore service without rereading the walkthrough. You'll add a 60-second interview answer to troubleshooting-debrief.md.

Are there any prerequisites?

You need a Windows workstation plus an AWS administrator account that can use browser-based AWS CLI login. The guide covers local tool verification plus immediate cleanup of billable resources.

Before We Start

Before any hands-on work, this checkpoint locks in the first failure: Session Manager fails because enable_ssm_core_policy is false, leaving AmazonSSMManagedInstanceCore absent. Your diagnosis will use EC2, IAM, route, and SSM status as evidence while the outbound SSM Agent connection provides private administration without SSH.

Verify the Toolchain and Create the Repository

A failed cloud command only becomes useful evidence when you trust the computer running it. This lab depends on Terraform reaching the intended AWS account with temporary credentials.

You will verify the Windows PowerShell toolchain before any infrastructure exists. You will also create a Git repository so every later repair has a clear history.

In this step, get ready to:
  • Verify every required local tool against its pinned version.
  • Authenticate with temporary AWS credentials for the intended account.
  • Create a version-controlled repository with the baseline ignore rules.
Verify the Windows tools

Version checks separate workstation problems from infrastructure problems. Each command must succeed before you begin provisioning resources.

  • Press the Windows key to open the search bar.
  • Type Windows PowerShell into the search bar.
  • Press Enter to open Windows PowerShell.
  • Check the installed tool versions by running these commands:
terraform version
aws --version
session-manager-plugin --version
git --version
code --version

What do these checks prove?

  • The Terraform check confirms that the infrastructure CLI is available through your system path.
  • The AWS CLI check confirms that your workstation can run AWS API commands.
  • The Session Manager plugin check confirms that interactive private-instance sessions can open later.
  • The Git check confirms that your repository workflow is available.
  • The Visual Studio Code check confirms that the code command can open a folder from PowerShell.

✔️ Every required version is ready

Your toolchain is ready for the lab. Terraform reports 1.16.2. The AWS CLI reports v2.36.42.

The Session Manager plugin reports 1.2.764.0 or later. Git reports 2.55.0(5). Visual Studio Code returns its version details.

ⓧ I see an older version

Upgrade each tool whose output is below the required version. Leave tools that already meet the requirement unchanged.

ⓧ A command is not found

A missing command means the tool is absent from your system path. Install each missing tool from its official source.

Still seeing a missing command?

Close every PowerShell window after installation. A new PowerShell session reloads the system path created by the installer.

Check that you used the Windows installer for your workstation architecture. Reinstall only the command that remains unavailable.

Help me diagnose a Windows tool that is still missing from PowerShell.

Create the temporary AWS profile

The AWS CLI can authenticate through a browser-based login. The separate process profile lets Terraform request temporary credentials without storing long-term access keys.

Credential setup can feel sensitive. This flow keeps your password in the browser sign-in process while the CLI caches temporary credentials.

The login command opens your browser for authentication. It waits in PowerShell until the browser flow completes.

  • Start the browser-based sign-in for the signin profile by running this command:
aws login --profile signin

What does this command do?

The command creates or signs in to the named signin profile with your AWS console identity. The resulting credentials are temporary.

  • Complete the AWS sign-in in the browser window.
  • Select the administrator account intended for this disposable lab.
  • Return to Windows PowerShell after the login succeeds.

Terraform needs a standard credential source that it can read from the shared AWS configuration. A credential process bridges the browser-authenticated profile into that format.

  • Use Visual Studio Code's file picker to open %USERPROFILE%\.aws\config.
  • Append the following profile block without removing the signin profile:
[profile process]
credential_process = aws configure export-credentials --profile signin --format process
region = us-east-1

How does the process profile work?

  • The process profile runs the credential export command whenever a compatible tool requests AWS credentials.
  • The credential command reads the temporary browser login from the signin profile.
  • The region setting keeps this lab in us-east-1.
  • Save %USERPROFILE%\.aws\config.

Before you check, which account identity do you expect the process profile to return?

  • Reveal the account and principal used by the process profile by running this command:
aws sts get-caller-identity --profile process

What should the identity show?

The response identifies the AWS account and principal represented by the temporary credentials. These values prove which execution context Terraform uses in the next step.

  • Confirm that the returned account matches the administrator account intended for the lab.
  • Confirm that the returned ARN represents the principal you expected to use.

Identity check failed?

Check that the profile heading is saved exactly as shown. Confirm that the credential process line still refers to the signin profile.

Repeat the browser sign-in if the temporary login has expired. Save the configuration file before checking the identity again.

Help me troubleshoot the temporary AWS process profile.

Create the Git repository

A focused commit history preserves the difference between the broken baseline and each repair. The repository begins with ignore rules that keep local state files and saved plans out of Git.

  • Move to your Desktop by running this command:
cd ~/Desktop

Why start on the Desktop?

Starting from the Desktop gives the repository a predictable location. You can find the lab folder again without depending on PowerShell's previous directory.

  • Create the terraform-private-ec2-lab folder by running these commands:
mkdir terraform-private-ec2-lab
cd terraform-private-ec2-lab

What do these commands do?

  • The first command creates the repository folder on your Desktop.
  • The second command makes that folder the active PowerShell location.
  • Confirm that the PowerShell prompt ends with terraform-private-ec2-lab.

Folder already exists?

Confirm that the existing terraform-private-ec2-lab folder is empty before using it. Remove unrelated files from the folder before continuing.

Help me check whether my existing lab folder is safe to use.

  • Initialize the repository with main as its first branch by running this command:
git init -b main

What does repository initialization create?

Git creates its internal repository metadata inside terraform-private-ec2-lab. The initial branch is named main.

  • Confirm that PowerShell reports successful repository initialization.
  • Open the current repository in Visual Studio Code by running this command:
code .

What does the dot mean?

The dot represents the current terraform-private-ec2-lab folder. Visual Studio Code opens that folder as the active workspace.

  • Confirm that Visual Studio Code shows terraform-private-ec2-lab in the Explorer sidebar.
  • Create .gitignore inside terraform-private-ec2-lab from the Explorer sidebar.
  • Paste these ignore rules into .gitignore.
.terraform/
*.tfstate
*.tfstate.*
*.tfplan
crash.log
crash.*.log
.terraform.tfstate.lock.info
override.tf
override.tf.json
*_override.tf
*_override.tf.json

What do these ignore rules protect?

  • The .terraform/ rule excludes downloaded provider files.
  • The state patterns exclude local Terraform state from Git.
  • The plan patterns exclude saved plans that can contain sensitive values.
  • The remaining patterns exclude crash output and local override files.
  • Save .gitignore.
  • Confirm that the Explorer sidebar lists .gitignore inside terraform-private-ec2-lab.

Cannot see the ignore file?

Check that the filename starts with a dot. Confirm that Visual Studio Code saved the file inside terraform-private-ec2-lab.

Help me fix a missing or incorrectly named .gitignore file.

✔️ Awesome, I've got everything!

Great, your .gitignore now protects the repository from local Terraform artifacts. Double-check that the file is saved.

ⓧ I'd like to double check the full code

.terraform/
*.tfstate
*.tfstate.*
*.tfplan
crash.log
crash.*.log
.terraform.tfstate.lock.info
override.tf
override.tf.json
*_override.tf
*_override.tf.json
  • Stage and commit .gitignore by running these commands:
git add .gitignore
git commit -m "chore: initialize Terraform lab"

What does this commit capture?

  • The staging command selects .gitignore for the next commit.
  • The commit records the repository's baseline protection rules with a focused message.

Git needs your author identity?

Use the official Git first-time setup guide to set your global author name and email. Retry the commit after saving both values.

Help me configure my Git author identity on Windows.

  • Create the feature branch by running this command:
git switch -c feature/private-ec2-core

Why use a feature branch?

The feature/private-ec2-core branch contains the infrastructure baseline and its later repairs. The separate branch keeps the lab workflow explicit.

  • Confirm that Visual Studio Code shows feature/private-ec2-core as the active repository branch.

Before the final check, do you expect every command to succeed without creating any AWS infrastructure?

  • Verify the complete workstation context by running these commands in Windows PowerShell:
terraform version
aws --version
session-manager-plugin --version
git --version
code --version
aws sts get-caller-identity --profile process
git log --oneline

What should you see?

  • Terraform reports 1.16.2.
  • The AWS CLI reports v2.36.42.
  • The Session Manager plugin reports 1.2.764.0 or later.
  • Git reports 2.55.0(5).
  • Visual Studio Code reports its version details.
  • The caller identity shows the intended AWS account and principal.
  • The Git history includes chore: initialize Terraform lab.

That is the workstation boundary proven. Your tools, temporary identity, repository history, and feature branch are ready while AWS still contains no lab infrastructure.

Your execution context is ready. Next, you will provision the deliberately broken private EC2 baseline and collect the first failure evidence.

Provision the Broken Private EC2 Baseline

Your verified workstation gives this lab a trustworthy execution context. Your feature branch is also ready to capture the infrastructure baseline.

A running private instance can still be unavailable through its management plane. In this step, Terraform provisions the network and EC2 resources while an incomplete Systems Manager policy creates that visible operational gap.

In this step, get ready to:
  • Define the provider settings and controlled failure inputs.
  • Provision the private network and workload resources.
  • Verify the running instance and capture the intended Systems Manager gap.
Define the baseline configuration

Version constraints keep future runs consistent. Input variables separate reusable infrastructure settings from the two controlled failure switches.

  • Switch back to Visual Studio Code with the terraform-private-ec2-lab repository from earlier.
  • Create provider.tf from the Explorer sidebar.
  • Add the Terraform requirements by pasting this code into provider.tf:
terraform {
  required_version = "= 1.16.2"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "= 6.63.0"
    }
  }
}

What do these requirements control?

The Terraform block pins Terraform CLI 1.16.2 and HashiCorp AWS Provider 6.63.0. Initialization uses these constraints when selecting the provider package.

  • Add the provider and Availability Zone configuration below the Terraform block:
provider "aws" {
  region  = var.aws_region
  profile = var.aws_profile

  default_tags {
    tags = {
      Project     = var.project_name
      Environment = "InterviewLab"
      ManagedBy   = "Terraform"
    }
  }
}

data "aws_availability_zones" "available" {
  state = "available"

  filter {
    name   = "opt-in-status"
    values = ["opt-in-not-required"]
  }
}

How does Terraform reach AWS?

The provider uses the temporary profile and Region supplied by variables. Default tags make the lab resources easier to identify during verification and cleanup.

The Availability Zones data source selects zones that are available without account opt-in. Both subnets use the first matching zone.

  • Save provider.tf.
  • Format the provider configuration by running this command in the PowerShell terminal from earlier:
terraform fmt -recursive

What does formatting confirm?

Terraform applies its canonical layout to supported configuration files. You may see provider.tf in the output if its spacing changed.

  • Create variables.tf from the Explorer sidebar.
  • Add the provider input variables by pasting this code into variables.tf:
variable "aws_region" {
  description = "AWS Region for the lab."
  type        = string
  default     = "us-east-1"
}

variable "aws_profile" {
  description = "AWS shared configuration profile used by Terraform."
  type        = string
  default     = "process"
}

variable "project_name" {
  description = "Name prefix and tag value for lab resources."
  type        = string
  default     = "tf-private-ec2"
}

What do the provider inputs hold?

These variables point Terraform at us-east-1 through the temporary process profile. The project name supplies a shared naming prefix and tag value.

  • Add the network variables below the project-name variable:
variable "vpc_cidr" {
  description = "IPv4 CIDR for the lab VPC."
  type        = string
  default     = "10.42.0.0/16"
}

variable "public_subnet_cidr" {
  description = "IPv4 CIDR for the public subnet."
  type        = string
  default     = "10.42.1.0/24"
}

variable "private_subnet_cidr" {
  description = "IPv4 CIDR for the private subnet."
  type        = string
  default     = "10.42.2.0/24"
}

How is the address space divided?

The VPC receives one public subnet range and one private subnet range. Their non-overlapping CIDRs create distinct paths for public egress infrastructure and the private workload.

  • Save variables.tf.
  • Format the current configuration by running:
terraform fmt -recursive

What should you observe?

The command completes without an HCL parsing error. Any printed filename identifies a file Terraform reformatted.

  • Add the instance and failure-control variables at the end of variables.tf:
variable "instance_type" {
  description = "EC2 instance type for the lab host."
  type        = string
  default     = "t3.micro"
}

variable "enable_ssm_core_policy" {
  description = "Controls whether the complete Systems Manager managed policy is attached."
  type        = bool
  default     = true
}

variable "s3_object_arn_suffix" {
  description = "Object ARN suffix. Use an empty value for the failure and /reports/* for the repair."
  type        = string
  default     = "/reports/*"
}

What do the scenario variables control?

The instance type selects the lab host size. The final two variables control the Systems Manager permission path and the S3 object resource suffix.

  • Create scenario.auto.tfvars from the Explorer sidebar.
  • Select the initial failure states by pasting these assignments into scenario.auto.tfvars:
enable_ssm_core_policy = false
s3_object_arn_suffix   = ""

What does the initial scenario select?

The false value selects the intentionally incomplete Systems Manager policy. The empty suffix makes the upload statement resolve to the bucket ARN.

  • Save variables.tf.
  • Save scenario.auto.tfvars.
  • Initialize the Terraform directory by running:
terraform init

What does initialization create?

Terraform downloads HashiCorp AWS Provider 6.63.0 and creates .terraform.lock.hcl with its recorded checksums. Committing that lock file keeps future provider selection consistent.

  • Confirm that PowerShell reports successful initialization.
  • Confirm that .terraform.lock.hcl appears in the Explorer sidebar.
  • Validate the foundation configuration by running:
terraform validate

What does validation prove?

Validation checks the initialized configuration for valid syntax and internally consistent references. A successful result proves Terraform can read the provider and input files.

Good progress. Your pinned provider and controlled scenario inputs now form a reproducible foundation for the infrastructure.

Initialization or validation failed?

Confirm that PowerShell is still in terraform-private-ec2-lab. Use the filename and line number from the message to locate the affected block.

Share the exact message here: Help me diagnose this Terraform initialization or validation error.

Build the private infrastructure

The network places a NAT Gateway in the public subnet. The private subnet uses that gateway for outbound traffic while the workload remains without a public address.

  • Create networking.tf from the Explorer sidebar.
  • Add the VPC and public subnet by pasting this code into networking.tf:
resource "aws_vpc" "lab" {
  cidr_block           = var.vpc_cidr
  enable_dns_support   = true
  enable_dns_hostnames = true

  tags = {
    Name = "${var.project_name}-vpc"
  }
}

resource "aws_subnet" "public" {
  vpc_id                  = aws_vpc.lab.id
  cidr_block              = var.public_subnet_cidr
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.project_name}-public"
  }
}

How does the public network begin?

The VPC enables DNS support and hostnames. The public subnet uses the selected Availability Zone and permits public addressing for resources placed there.

  • Add the private subnet and internet gateway below the public subnet:
resource "aws_subnet" "private" {
  vpc_id                  = aws_vpc.lab.id
  cidr_block              = var.private_subnet_cidr
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = false

  tags = {
    Name = "${var.project_name}-private"
  }
}

resource "aws_internet_gateway" "lab" {
  vpc_id = aws_vpc.lab.id

  tags = {
    Name = "${var.project_name}-igw"
  }
}

What keeps the workload subnet private?

The private subnet disables automatic public addressing. The internet gateway supports the public side of the egress path without assigning a public address to the workload.

  • Save networking.tf.
  • Validate the subnet and gateway references by running:
terraform validate

What does this checkpoint prove?

A successful result confirms that Terraform can resolve the VPC, both subnets, the Availability Zone data source, and the internet gateway.

  • Add the Elastic IP and NAT Gateway below the internet gateway:
resource "aws_eip" "nat" {
  domain = "vpc"

  depends_on = [aws_internet_gateway.lab]

  tags = {
    Name = "${var.project_name}-nat-eip"
  }
}

resource "aws_nat_gateway" "lab" {
  allocation_id = aws_eip.nat.allocation_id
  subnet_id     = aws_subnet.public.id

  depends_on = [aws_internet_gateway.lab]

  tags = {
    Name = "${var.project_name}-nat"
  }
}

How does private egress work?

The Elastic IP supplies the NAT Gateway's public address. The NAT Gateway sits in the public subnet so private resources can initiate outbound connections.

  • Add the two route tables below the NAT Gateway:
resource "aws_route_table" "public" {
  vpc_id = aws_vpc.lab.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.lab.id
  }

  tags = {
    Name = "${var.project_name}-public-rt"
  }
}

resource "aws_route_table" "private" {
  vpc_id = aws_vpc.lab.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.lab.id
  }

  tags = {
    Name = "${var.project_name}-private-rt"
  }
}

Why are there two default routes?

The public route sends internet-bound traffic to the internet gateway. The private route sends outbound traffic through the NAT Gateway.

  • Save networking.tf.
  • Validate the gateway and route references by running:
terraform validate

What does the route check prove?

Terraform confirms that each route target exists in the configuration. It also confirms that the NAT Gateway uses the public subnet and Elastic IP.

  • Add the route-table associations at the end of networking.tf:
resource "aws_route_table_association" "public" {
  subnet_id      = aws_subnet.public.id
  route_table_id = aws_route_table.public.id
}

resource "aws_route_table_association" "private" {
  subnet_id      = aws_subnet.private.id
  route_table_id = aws_route_table.private.id
}

What do the associations complete?

Each association connects a subnet to its intended route table. This completes the separate public and private traffic paths.

  • Create s3.tf from the Explorer sidebar.
  • Add the report bucket and Public Access Block by pasting this code into s3.tf:
resource "aws_s3_bucket" "reports" {
  bucket_prefix = "tf-private-ec2-reports-"
  force_destroy = false

  tags = {
    Name = "${var.project_name}-reports"
  }
}

resource "aws_s3_bucket_public_access_block" "reports" {
  bucket = aws_s3_bucket.reports.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

How is the report bucket protected?

The S3 bucket receives a generated name. All four Public Access Block controls are enabled.

Automatic object deletion remains disabled. Cleanup must empty the bucket before Terraform removes it.

  • Save networking.tf.
  • Save s3.tf.
  • Validate the network and bucket configuration by running:
terraform validate

What does this validation cover?

Terraform now checks the completed network graph and private bucket references. A successful result gives you a clean checkpoint before adding workload permissions.

An IAM instance profile gives the workload temporary role credentials. Conditional resources select the intentionally incomplete Systems Manager permission path for this baseline.

  • Create iam.tf from the Explorer sidebar.
  • Add the EC2 trust role and instance profile by pasting this code into iam.tf:
resource "aws_iam_role" "ec2" {
  name_prefix = "${var.project_name}-role-"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = "sts:AssumeRole"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_instance_profile" "ec2" {
  name_prefix = "${var.project_name}-profile-"
  role        = aws_iam_role.ec2.name
}

How does the instance receive a role?

The trust policy allows the EC2 service to assume the role. The instance profile packages that role for attachment to the private instance.

  • Add the conditional Systems Manager resources below the instance profile:
resource "aws_iam_role_policy" "ssm_incomplete" {
  count = var.enable_ssm_core_policy ? 0 : 1

  name = "private-ec2-ssm-incomplete"
  role = aws_iam_role.ec2.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["ssm:UpdateInstanceInformation"]
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "ssm_core" {
  count = var.enable_ssm_core_policy ? 1 : 0

  role       = aws_iam_role.ec2.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}

How does the permission switch behave?

The baseline value creates the inline policy containing only ssm:UpdateInstanceInformation. The managed policy attachment has a count of zero.

This combination preserves an instance profile while leaving the role without the complete Session Manager permission set. The resulting gap is deliberate evidence for the next troubleshooting step.

  • Save iam.tf.
  • Validate the role and Systems Manager resources by running:
terraform validate

What does the IAM checkpoint prove?

Terraform can resolve the trust policy and instance profile. It can also evaluate both conditional permission resources from the selected scenario value.

  • Add the S3 upload policy at the end of iam.tf:
resource "aws_iam_role_policy" "s3_upload" {
  name = "private-ec2-s3-upload"
  role = aws_iam_role.ec2.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid      = "ListReportsPrefix"
        Effect   = "Allow"
        Action   = ["s3:ListBucket"]
        Resource = aws_s3_bucket.reports.arn
        Condition = {
          StringLike = {
            "s3:prefix" = ["reports/*"]
          }
        }
      },
      {
        Sid      = "PutReports"
        Effect   = "Allow"
        Action   = ["s3:PutObject"]
        Resource = "${aws_s3_bucket.reports.arn}${var.s3_object_arn_suffix}"
      }
    ]
  })
}

Why does the policy use two statements?

Bucket listing uses the bucket ARN with a prefix condition. Object upload builds its resource from the bucket ARN and the scenario-controlled suffix.

The empty suffix makes the upload resource resolve to the bucket ARN. You will preserve that second planned defect for later diagnosis.

  • Save iam.tf.
  • Validate the completed IAM configuration by running:
terraform validate

What does the policy check prove?

A successful result confirms that Terraform can render the trust policy and inline policies. It also confirms that the upload policy can reference the report bucket.

The compute layer combines a no-ingress security group with temporary outbound access. The instance launches in the private subnet without a public address or SSH key.

  • Create ec2.tf from the Explorer sidebar.
  • Add the security group and egress rule by pasting this code into ec2.tf:
resource "aws_security_group" "private_instance" {
  name_prefix = "${var.project_name}-sg-"
  description = "No inbound access; outbound access supports private administration through NAT"
  vpc_id      = aws_vpc.lab.id

  tags = {
    Name = "${var.project_name}-sg"
  }
}

resource "aws_vpc_security_group_egress_rule" "private_instance_all_ipv4" {
  security_group_id = aws_security_group.private_instance.id
  description       = "Temporary lab egress through NAT"
  cidr_ipv4         = "0.0.0.0/0"
  ip_protocol       = "-1"
}

What traffic does the security group permit?

The security group declares no inbound rules. The separate egress resource allows the instance to initiate IPv4 traffic through the private route.

  • Add the initial private instance resource below the egress rule:
resource "aws_instance" "private" {
  ami                         = "resolve:ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64"
  instance_type               = var.instance_type
  subnet_id                   = aws_subnet.private.id
  associate_public_ip_address = false
  vpc_security_group_ids      = [aws_security_group.private_instance.id]
  iam_instance_profile        = aws_iam_instance_profile.ec2.name

  tags = {
    Name = "${var.project_name}-private-instance"
  }
}

What defines the private instance?

The dynamic AMI reference selects Amazon Linux 2023. The subnet and addressing settings keep the host private while the instance profile supplies its workload identity.

  • Save ec2.tf.
  • Validate the initial compute resources by running:
terraform validate

What does this compute check prove?

Terraform can resolve the private subnet and security group. It can also connect the instance to the IAM instance profile.

  • Insert the metadata and root-volume blocks above the tags block inside aws_instance.private:
  metadata_options {
    http_endpoint = "enabled"
    http_tokens   = "required"
  }

  root_block_device {
    delete_on_termination = true
    encrypted             = true
    volume_size           = 8
    volume_type           = "gp3"
  }

What do the instance safeguards enforce?

The metadata settings require IMDSv2 tokens. The encrypted 8 GiB gp3 root volume is deleted when the instance is terminated.

  • Insert the startup and dependency blocks below the root-volume block:
  user_data = <<-EOF
    #!/bin/bash
    systemctl start amazon-ssm-agent
  EOF

  depends_on = [
    aws_route_table_association.private,
    aws_iam_role_policy.ssm_incomplete,
    aws_iam_role_policy_attachment.ssm_core
  ]

What does startup depend on?

The startup script starts SSM Agent. The dependency list waits for the private route association and the selected IAM permission path before creating the instance.

  • Save ec2.tf.
  • Validate the hardened compute configuration by running:
terraform validate

What does the completed compute check prove?

Terraform can resolve the network and IAM dependencies. It also accepts the metadata settings and encrypted root-volume configuration.

Outputs turn generated resource attributes into reusable evidence. Later diagnostics use them to identify the instance and each supporting resource.

  • Create outputs.tf from the Explorer sidebar.
  • Add the location and workload outputs by pasting this code into outputs.tf:
output "availability_zone" {
  description = "Availability Zone used by both lab subnets."
  value       = data.aws_availability_zones.available.names[0]
}

output "instance_id" {
  description = "Private EC2 instance ID used for Session Manager commands."
  value       = aws_instance.private.id
}

output "instance_private_ip" {
  description = "Private IPv4 address assigned to the EC2 instance."
  value       = aws_instance.private.private_ip
}

output "bucket_name" {
  description = "Generated S3 bucket name for system reports."
  value       = aws_s3_bucket.reports.id
}

What does the first output group expose?

These outputs identify the shared Availability Zone and private instance. They also expose the private address and generated report bucket name.

  • Add the IAM and network evidence outputs at the end of outputs.tf:
output "iam_role_name" {
  description = "IAM role attached through the EC2 instance profile."
  value       = aws_iam_role.ec2.name
}

output "private_route_table_id" {
  description = "Private route table used to verify the NAT default route."
  value       = aws_route_table.private.id
}

output "nat_gateway_public_ip" {
  description = "Public IPv4 address used only by the NAT Gateway."
  value       = aws_eip.nat.public_ip
}

What does the second output group expose?

These outputs identify the workload role and private route table. The final output confirms that public addressing belongs to the NAT Gateway.

  • Save outputs.tf.
  • Validate the complete resource graph by running:
terraform validate

What does full validation prove?

A successful result confirms that Terraform understands every provider and variable reference. It also confirms the networking and IAM dependencies used by the workload.

Seeing a configuration error?

Start with the filename and line number in the message. Confirm that every pasted block sits inside the file named above it.

Share the exact error here: Help me trace this Terraform baseline configuration error.

Use this checkpoint to compare the complete baseline before Terraform creates anything in AWS. The generated .terraform.lock.hcl remains managed by Terraform.

✔️ Awesome, I've got everything!

Great. Confirm that all eight Terraform configuration files are saved. Confirm that .terraform.lock.hcl is present.

ⓧ I'd like to double check the full code

.terraform/
*.tfstate
*.tfstate.*
*.tfplan
crash.log
crash.*.log
.terraform.tfstate.lock.info
override.tf
override.tf.json
*_override.tf
*_override.tf.json
terraform {
  required_version = "= 1.16.2"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "= 6.63.0"
    }
  }
}

provider "aws" {
  region  = var.aws_region
  profile = var.aws_profile

  default_tags {
    tags = {
      Project     = var.project_name
      Environment = "InterviewLab"
      ManagedBy   = "Terraform"
    }
  }
}

data "aws_availability_zones" "available" {
  state = "available"

  filter {
    name   = "opt-in-status"
    values = ["opt-in-not-required"]
  }
}
variable "aws_region" {
  description = "AWS Region for the lab."
  type        = string
  default     = "us-east-1"
}

variable "aws_profile" {
  description = "AWS shared configuration profile used by Terraform."
  type        = string
  default     = "process"
}

variable "project_name" {
  description = "Name prefix and tag value for lab resources."
  type        = string
  default     = "tf-private-ec2"
}

variable "vpc_cidr" {
  description = "IPv4 CIDR for the lab VPC."
  type        = string
  default     = "10.42.0.0/16"
}

variable "public_subnet_cidr" {
  description = "IPv4 CIDR for the public subnet."
  type        = string
  default     = "10.42.1.0/24"
}

variable "private_subnet_cidr" {
  description = "IPv4 CIDR for the private subnet."
  type        = string
  default     = "10.42.2.0/24"
}

variable "instance_type" {
  description = "EC2 instance type for the lab host."
  type        = string
  default     = "t3.micro"
}

variable "enable_ssm_core_policy" {
  description = "Controls whether the complete Systems Manager managed policy is attached."
  type        = bool
  default     = true
}

variable "s3_object_arn_suffix" {
  description = "Object ARN suffix. Use an empty value for the failure and /reports/* for the repair."
  type        = string
  default     = "/reports/*"
}
enable_ssm_core_policy = false
s3_object_arn_suffix   = ""
resource "aws_vpc" "lab" {
  cidr_block           = var.vpc_cidr
  enable_dns_support   = true
  enable_dns_hostnames = true

  tags = {
    Name = "${var.project_name}-vpc"
  }
}

resource "aws_subnet" "public" {
  vpc_id                  = aws_vpc.lab.id
  cidr_block              = var.public_subnet_cidr
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.project_name}-public"
  }
}

resource "aws_subnet" "private" {
  vpc_id                  = aws_vpc.lab.id
  cidr_block              = var.private_subnet_cidr
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = false

  tags = {
    Name = "${var.project_name}-private"
  }
}

resource "aws_internet_gateway" "lab" {
  vpc_id = aws_vpc.lab.id

  tags = {
    Name = "${var.project_name}-igw"
  }
}

resource "aws_eip" "nat" {
  domain = "vpc"

  depends_on = [aws_internet_gateway.lab]

  tags = {
    Name = "${var.project_name}-nat-eip"
  }
}

resource "aws_nat_gateway" "lab" {
  allocation_id = aws_eip.nat.allocation_id
  subnet_id     = aws_subnet.public.id

  depends_on = [aws_internet_gateway.lab]

  tags = {
    Name = "${var.project_name}-nat"
  }
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.lab.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.lab.id
  }

  tags = {
    Name = "${var.project_name}-public-rt"
  }
}

resource "aws_route_table" "private" {
  vpc_id = aws_vpc.lab.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.lab.id
  }

  tags = {
    Name = "${var.project_name}-private-rt"
  }
}

resource "aws_route_table_association" "public" {
  subnet_id      = aws_subnet.public.id
  route_table_id = aws_route_table.public.id
}

resource "aws_route_table_association" "private" {
  subnet_id      = aws_subnet.private.id
  route_table_id = aws_route_table.private.id
}
resource "aws_iam_role" "ec2" {
  name_prefix = "${var.project_name}-role-"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = "sts:AssumeRole"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_instance_profile" "ec2" {
  name_prefix = "${var.project_name}-profile-"
  role        = aws_iam_role.ec2.name
}

resource "aws_iam_role_policy" "ssm_incomplete" {
  count = var.enable_ssm_core_policy ? 0 : 1

  name = "private-ec2-ssm-incomplete"
  role = aws_iam_role.ec2.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["ssm:UpdateInstanceInformation"]
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "ssm_core" {
  count = var.enable_ssm_core_policy ? 1 : 0

  role       = aws_iam_role.ec2.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}

resource "aws_iam_role_policy" "s3_upload" {
  name = "private-ec2-s3-upload"
  role = aws_iam_role.ec2.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid      = "ListReportsPrefix"
        Effect   = "Allow"
        Action   = ["s3:ListBucket"]
        Resource = aws_s3_bucket.reports.arn
        Condition = {
          StringLike = {
            "s3:prefix" = ["reports/*"]
          }
        }
      },
      {
        Sid      = "PutReports"
        Effect   = "Allow"
        Action   = ["s3:PutObject"]
        Resource = "${aws_s3_bucket.reports.arn}${var.s3_object_arn_suffix}"
      }
    ]
  })
}
resource "aws_security_group" "private_instance" {
  name_prefix = "${var.project_name}-sg-"
  description = "No inbound access; outbound access supports private administration through NAT"
  vpc_id      = aws_vpc.lab.id

  tags = {
    Name = "${var.project_name}-sg"
  }
}

resource "aws_vpc_security_group_egress_rule" "private_instance_all_ipv4" {
  security_group_id = aws_security_group.private_instance.id
  description       = "Temporary lab egress through NAT"
  cidr_ipv4         = "0.0.0.0/0"
  ip_protocol       = "-1"
}

resource "aws_instance" "private" {
  ami                         = "resolve:ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64"
  instance_type               = var.instance_type
  subnet_id                   = aws_subnet.private.id
  associate_public_ip_address = false
  vpc_security_group_ids      = [aws_security_group.private_instance.id]
  iam_instance_profile        = aws_iam_instance_profile.ec2.name

  metadata_options {
    http_endpoint = "enabled"
    http_tokens   = "required"
  }

  root_block_device {
    delete_on_termination = true
    encrypted             = true
    volume_size           = 8
    volume_type           = "gp3"
  }

  user_data = <<-EOF
    #!/bin/bash
    systemctl start amazon-ssm-agent
  EOF

  depends_on = [
    aws_route_table_association.private,
    aws_iam_role_policy.ssm_incomplete,
    aws_iam_role_policy_attachment.ssm_core
  ]

  tags = {
    Name = "${var.project_name}-private-instance"
  }
}
resource "aws_s3_bucket" "reports" {
  bucket_prefix = "tf-private-ec2-reports-"
  force_destroy = false

  tags = {
    Name = "${var.project_name}-reports"
  }
}

resource "aws_s3_bucket_public_access_block" "reports" {
  bucket = aws_s3_bucket.reports.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}
output "availability_zone" {
  description = "Availability Zone used by both lab subnets."
  value       = data.aws_availability_zones.available.names[0]
}

output "instance_id" {
  description = "Private EC2 instance ID used for Session Manager commands."
  value       = aws_instance.private.id
}

output "instance_private_ip" {
  description = "Private IPv4 address assigned to the EC2 instance."
  value       = aws_instance.private.private_ip
}

output "bucket_name" {
  description = "Generated S3 bucket name for system reports."
  value       = aws_s3_bucket.reports.id
}

output "iam_role_name" {
  description = "IAM role attached through the EC2 instance profile."
  value       = aws_iam_role.ec2.name
}

output "private_route_table_id" {
  description = "Private route table used to verify the NAT default route."
  value       = aws_route_table.private.id
}

output "nat_gateway_public_ip" {
  description = "Public IPv4 address used only by the NAT Gateway."
  value       = aws_eip.nat.public_ip
}
Apply and verify the intended gap

The configuration now describes healthy compute and routing with an incomplete management permission set. Planning gives you one final chance to inspect that exact baseline before AWS begins creating billable resources.

  • Format every Terraform file by running:
terraform fmt -recursive

What should formatting show?

Terraform may print files whose layout changed. The command should finish without an HCL parsing error.

  • Validate the complete baseline by running:
terraform validate

What should validation show?

Terraform reports that the configuration is valid. This confirms the complete graph can be evaluated before planning.

Before you create the plan, which actions do you expect Terraform to propose for an account with no existing lab resources?

  • Create the saved baseline plan by running:
terraform plan -out=baseline.tfplan

What should the plan contain?

The plan should contain creation actions for the VPC and two subnets. It should also include the internet gateway and NAT Gateway.

You should also find the private EC2 instance and no-ingress security group. The IAM resources and private S3 bucket should appear as creation actions.

Confirm that the plan contains no update or deletion actions. Confirm that the incomplete Systems Manager policy is selected.

Seeing an unexpected plan action?

Stop before applying. Compare the affected resource with the full-code checkpoint and confirm both assignments in scenario.auto.tfvars.

Review the unexpected action here: Help me inspect this Terraform baseline plan.

Charges Start After Apply

The NAT Gateway and its public IPv4 address accrue hourly charges while provisioned. EC2 and EBS usage may also be charged.

S3 requests and storage can add smaller charges. Keep the Terraform state intact so the cleanup section can remove every managed resource.

Before you apply the plan, which resource do you expect to take the longest to provision?

  • Create the planned infrastructure by running:
terraform apply baseline.tfplan

What happens during apply?

Terraform executes the exact actions stored in baseline.tfplan. It does not request another approval for the saved plan.

The NAT Gateway is commonly the slowest resource in this stack. A quiet wait while AWS provisions it is expected.

  • Display the generated infrastructure values by running:
terraform output

What should the outputs show?

You should see the Availability Zone and instance ID. You should also see the private address and generated bucket name.

The remaining values identify the IAM role and private route table. The NAT Gateway public address belongs to the egress resource instead of the instance.

The infrastructure is live. You now have a private workload with a public egress path and reusable diagnostic identifiers.

  • Capture the instance ID for the CLI checks by running:
$InstanceId = terraform output -raw instance_id

What does this variable hold?

The PowerShell variable holds the generated EC2 instance ID from Terraform state. The next checks target that exact workload.

  • Inspect the instance state and addressing by running:
aws ec2 describe-instances --instance-ids $InstanceId --profile process --region us-east-1 --query "Reservations[0].Instances[0].{State:State.Name,PrivateIp:PrivateIpAddress,PublicIp:PublicIpAddress,IamProfile:IamInstanceProfile.Arn}" --output table

What should EC2 report?

The state should be running. The table should contain a private address and an instance profile ARN.

The public address should be empty. This proves the workload is running without direct public addressing.

Before you query Systems Manager, do you expect this running instance to appear as an online managed node?

  • Check the instance's managed-node status by running:
aws ssm describe-instance-information --filters "Key=InstanceIds,Values=$InstanceId" --profile process --region us-east-1

This Gap Is Intentional

The command does not report the new instance as an online managed node. EC2 and the private egress path exist while the role contains only the incomplete Systems Manager permission.

This is the planned shortfall. A running instance and an outbound route do not establish Session Manager access.

  • Switch back to the signed-in AWS Management Console from earlier.
  • Select the us-east-1 Region.
  • Open the EC2 service.
  • Select Instances from the left navigation.
  • Select the instance named tf-private-ec2-private-instance.
  • Confirm that its instance state is Running.
  • Confirm that it has a private IPv4 address.
  • Confirm that it has no public IPv4 address.

The console evidence matches the CLI evidence. The instance is healthy at the compute layer while Systems Manager still lacks an online managed node.

  • Stage the baseline configuration and provider lock file by running:
git add provider.tf variables.tf networking.tf iam.tf ec2.tf s3.tf outputs.tf scenario.auto.tfvars .terraform.lock.hcl

What does this stage?

Git stages the complete baseline configuration and generated provider lock file. Local state and saved plans remain excluded by .gitignore.

  • Commit the failure baseline by running:
git commit -m "feat: provision private EC2 failure baseline"

What does this commit preserve?

The commit records the exact infrastructure configuration that produced the Systems Manager gap. Later remediation remains separate from baseline provisioning.

Apply or commit failed?

Use the failed resource or file name to narrow the cause. Authentication errors point to the temporary profile while authorization errors can involve resource creation or IAM role passing.

Share the exact failure here: Help me diagnose this baseline apply or Git commit failure.

Your committed baseline now proves that healthy compute and routing do not guarantee manageability. Next, you will diagnose the missing permission from EC2, IAM, routing, and Systems Manager evidence.

Restore Session Manager Access

Your Terraform baseline proves that the private Amazon EC2 instance can run without a public address. It also proves that the private route provides outbound access through the NAT Gateway.

A running instance with a working route still needs complete AWS Systems Manager permissions. You will isolate that missing AWS IAM dependency before restoring Session Manager access through Terraform.

In this step, get ready to:
  • Gather evidence across the private access dependency chain.
  • Apply the Systems Manager permission repair through Terraform.
  • Prove that Session Manager reaches the private instance.
Collect the failure evidence

A layered diagnosis separates healthy compute from broken access. The Terraform outputs connect your local state to the exact AWS resources you need to inspect.

  • Return to Windows PowerShell in the existing Visual Studio Code terminal.
  • Capture the Terraform outputs and reusable identifiers by running:
terraform output
$Region = "us-east-1"
$InstanceId = terraform output -raw instance_id
$RoleName = terraform output -raw iam_role_name
$PrivateRouteTableId = terraform output -raw private_route_table_id

What do these values prove?

  • The Terraform outputs identify the infrastructure currently tracked in local state.
  • The $InstanceId value connects the Terraform resource to the running EC2 instance.
  • The $RoleName value identifies the role whose permissions control Systems Manager access.
  • The $PrivateRouteTableId value identifies the route table that provides private outbound access.

You should see values for the instance ID, private address, IAM role name, private route table ID, bucket name, Availability Zone, and NAT Gateway public address.

Terraform outputs unavailable?

Confirm that PowerShell is still inside the terraform-private-ec2-lab folder. Terraform reads the local state from this folder.

If an output is missing, confirm that the baseline apply completed successfully before continuing.

Help me diagnose missing Terraform outputs for this private EC2 lab.

  • Gather the initial EC2, Systems Manager, and IAM evidence by running:
aws ec2 describe-instances --instance-ids $InstanceId
aws ssm describe-instance-information --filters "Key=InstanceIds,Values=$InstanceId"
aws iam list-attached-role-policies --role-name $RoleName

What does this evidence isolate?

  • The EC2 response proves that the instance exists under the expected instance ID.
  • The Systems Manager response shows whether that instance is registered as a managed node.
  • The IAM response lists the managed policies attached to the instance role.

The instance data should show a running private instance with an attached instance profile. The managed-node response should not report the instance as online.

The role policy list should omit AmazonSSMManagedInstanceCore. That missing attachment is the permission gap you are looking for.

Diagnostic command failed?

Confirm that the temporary process profile is still authenticated. An expired login can make a healthy AWS resource look unavailable from PowerShell.

Check that $InstanceId and $RoleName contain values from the current Terraform state.

Help me troubleshoot the failed AWS diagnostic command.

  • Return to the AWS Management Console from earlier.
  • Switch to the details page for the instance matching $InstanceId.
  • Confirm that the instance state is running.
  • Confirm that the instance has a private IPv4 address.
  • Confirm that the instance has no public IPv4 address.
  • Confirm that the expected instance profile is attached.
  • Confirm that the attached security group has no inbound rules.

What have you ruled out?

The console evidence rules out a stopped instance, accidental public placement, a missing instance profile, and an inbound administration path. The investigation can now focus on outbound routing and role permissions.

  • Switch to the private route table matching $PrivateRouteTableId.
  • Confirm that 0.0.0.0/0 targets the NAT Gateway.

You have now isolated the fault to the instance role. Compute placement, private addressing, security group ingress, the instance profile, and outbound routing all match the intended architecture.

Apply the permission repair

The existing IAM configuration uses conditional resource counts. Enabling the complete policy removes the deliberately incomplete inline policy and creates the managed-policy attachment.

  • Switch back to scenario.auto.tfvars in Visual Studio Code.
  • Find enable_ssm_core_policy = false.
  • Replace false with true.
  • Leave s3_object_arn_suffix = "" unchanged.
  • Compare the file with this final version:
enable_ssm_core_policy = true
s3_object_arn_suffix   = ""

What does this change control?

The first value activates the complete Systems Manager policy attachment. The empty S3 suffix preserves the separate authorization failure for the next investigation.

  • Save scenario.auto.tfvars.

✔️ Awesome, I've got everything!

Your saved scenario now enables the Systems Manager repair while preserving the planned S3 failure.

ⓧ I'd like to double check the full code

enable_ssm_core_policy = true
s3_object_arn_suffix   = ""

What should match?

The Systems Manager toggle should be true. The S3 suffix should remain an empty string.

  • Format the configuration and validate its internal consistency by running:
terraform fmt -recursive
terraform validate

What do these checks protect?

  • Formatting keeps the Terraform files in canonical form.
  • Validation catches configuration errors before Terraform contacts AWS for the plan.

You should see a successful validation result. The format command may finish quietly when every file already matches Terraform formatting.

Validation failed?

Check that the toggle contains the Boolean value true without quotation marks. Confirm that the S3 suffix still contains two quotation marks with no characters between them.

Help me fix the Terraform validation failure in scenario.auto.tfvars.

  • Create the saved remediation plan by running:
terraform plan -out=ssm-fix.tfplan

What does the saved plan show?

The plan should remove the incomplete inline permission and add the managed-policy attachment. The EC2 instance should remain the same private instance.

  • Inspect the plan for the removal of aws_iam_role_policy.ssm_incomplete.
  • Inspect the plan for the creation of aws_iam_role_policy_attachment.ssm_core.
  • Confirm that the plan does not replace aws_instance.private.

The proposed change should focus on the role permissions. That small scope matches the evidence you collected.

Plan shows an instance replacement?

Stop before applying the plan. Confirm that scenario.auto.tfvars is the only file you changed in this step.

Help me understand why the SSM remediation plan replaces my EC2 instance.

  • Apply the reviewed permission repair by running:
terraform apply ssm-fix.tfplan

What does this apply change?

Terraform executes the exact saved plan without requesting another approval. The incomplete inline policy disappears while arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore becomes attached to the instance role.

The permission repair is now managed by Terraform. Your private networking and EC2 instance remain unchanged.

Apply failed?

Confirm that ssm-fix.tfplan was created by the immediately preceding plan. A saved plan can become unusable when the configuration or infrastructure changes after planning.

Help me diagnose the failed Terraform apply for the SSM policy attachment.

Prove private session access

The repair succeeds only when the instance becomes an online managed node and accepts a fresh session. These checks test the same role that failed before the Terraform change.

  • Repeat the Systems Manager and IAM checks by running:
aws ssm describe-instance-information --filters "Key=InstanceIds,Values=$InstanceId"
aws iam list-attached-role-policies --role-name $RoleName

What changed after the apply?

  • The managed-node information should now report the instance with an Online status.
  • The role policy list should now include AmazonSSMManagedInstanceCore.

If the node has not reported Online yet, wait briefly before repeating the same checks. The policy attachment must reach the instance before registration completes.

Managed node still absent?

Confirm that the managed policy appears in the role policy list. Then recheck the private NAT route and the instance profile association you recorded earlier.

Help me diagnose why the private EC2 instance is still absent from Systems Manager.

Before you start the session, do you expect the private instance to need a public IP or an inbound rule?

  • Start a fresh Session Manager shell by running:
aws ssm start-session --target $InstanceId --profile process --region $Region

What does this command prove?

The command targets the Terraform-managed instance through the authenticated process profile. A successful shell proves that the agent, instance identity, IAM permissions, DNS, and outbound connectivity now form a working access path.

You should see a shell prompt open without a public IP, inbound rule, SSH key, or bastion host. That is the access path restored.

  • Check the SSM Agent service from inside the session by running:
sudo systemctl status amazon-ssm-agent

What does the service check confirm?

The service status confirms that the agent responsible for the managed-node connection is running on the instance.

  • Return to the shell prompt after reading the service status.
  • Verify the installed AWS CLI and the instance role identity by running:
aws --version
aws sts get-caller-identity

What does the instance identity prove?

The version check confirms that the instance can run the AWS CLI. The caller identity response proves that commands inside the session use temporary credentials from the EC2 role.

You should see an AWS CLI version followed by the account and role identity associated with the instance profile.

  • Close the CLI session after capturing your evidence.
  • Return to Windows PowerShell in the Terraform lab folder.
  • Commit the reviewed remediation by running:
git add scenario.auto.tfvars
git commit -m "fix: attach AmazonSSMManagedInstanceCore"

Why commit this change separately?

A focused Git commit preserves the exact remediation that restored access. The history now separates baseline provisioning from the Systems Manager fix.

You should see a new commit with the message fix: attach AmazonSSMManagedInstanceCore.

Commit blocked by Git?

If Git reports a missing author identity, configure your user name and email using the setup from the toolchain step. Then run the commit again.

Help me resolve the Git commit failure for the SSM remediation.

Before you test the console path, do you expect it to reach the same private instance as the CLI session?

  • Return to the EC2 instance details page in the AWS Management Console.
  • Select Connect.
  • Choose the Session Manager path.
  • Start a new console session.

You should see another shell prompt for the same private instance. The CLI path and console path now confirm the repaired access independently.

Session Manager now reaches your private instance through the role attached by Terraform. Next, you will use this shell to test the report upload and investigate the planned S3 authorization failure.

Diagnose the S3 Upload Failure

Your private EC2 instance is now Online in Systems Manager. Session Manager gives you a working administrative shell through the instance role.

An IAM permission names an action. It also scopes that action to a resource.

In this step, get ready to:
  • Create a system report from the EC2 role identity.
  • Attempt the report upload to S3 from the private instance.
  • Inspect the inline IAM policy to identify the resource mismatch.
Prepare the report evidence

The generated bucket name connects your local Terraform state to the active instance shell. Recording it first keeps the test pointed at the bucket created by this lab.

  • Read the generated bucket name in Windows PowerShell by running this command:
terraform output -raw bucket_name

What does this command show?

The command reads the current bucket_name output as a plain value. This is the generated S3 bucket managed by your Terraform state.

  • Record the returned value here: your generated S3 bucket name.
  • Switch back to the active Session Manager shell.
  • Replace your-bucket-name-here below with your generated S3 bucket name.
  • Set the session variables by running these commands:
export AWS_DEFAULT_REGION="us-east-1"
BUCKET_NAME="your-bucket-name-here"

What do these variables control?

  • The AWS_DEFAULT_REGION variable directs commands to us-east-1.
  • The BUCKET_NAME variable holds the generated destination bucket.

The report captures system evidence from inside the private instance. Its caller identity also proves which role attempts the upload.

  • Generate /tmp/system-report.txt by running these commands:
date -u > /tmp/system-report.txt
hostname >> /tmp/system-report.txt
uname -a >> /tmp/system-report.txt
df -h >> /tmp/system-report.txt
aws sts get-caller-identity >> /tmp/system-report.txt
cat /tmp/system-report.txt

What does this report prove?

  • The date -u output timestamps the evidence in UTC.
  • The hostname output identifies the host that produced the report.
  • The uname -a output records the running kernel.
  • The df -h output records filesystem capacity.
  • The caller identity output confirms that the EC2 role produced the report.

Report missing a section?

  • Run cat /tmp/system-report.txt again to inspect the complete file.
  • Check each redirect operator if one command replaced earlier evidence.

Ask for help with the report commands.

Test the current S3 permissions

The AWS CLI now sends the report request with temporary credentials from the EC2 role. The result tests the policy from the workload identity that needs access.

Before you run the upload, do you expect the current role policy to authorize the object request?

  • Attempt the report upload by running this command:
aws s3 cp /tmp/system-report.txt "s3://${BUCKET_NAME}/reports/system-report.txt"

The Denial Is the Evidence

The command returns an AccessDenied response. This planned failure proves that the current EC2 role cannot authorize the target object upload.

Listing the prefix exercises a separate S3 permission. This comparison helps isolate the failed operation.

  • Test the authorized prefix listing by running this command:
aws s3 ls "s3://${BUCKET_NAME}/reports/"

What does the listing prove?

The command returns to the shell prompt without AccessDenied. The role can list the allowed prefix even though it cannot upload the report object.

Seeing a Different Result?

  • Confirm that BUCKET_NAME matches the generated Terraform output if the listing is denied.
  • Confirm that the active caller is the EC2 role if the upload uses an unexpected identity.
  • Check that s3_object_arn_suffix remains an empty string if the upload succeeds.

Help me compare my upload result with the intended failure.

Inspect the rendered IAM policy

The workload test shows which operation fails. The inline policy now provides the configuration evidence needed to explain why.

  • Leave the active Session Manager shell connected.
  • Switch back to the Windows PowerShell terminal in terraform-private-ec2-lab from earlier.
  • Load the generated IAM role name by running this command:
$RoleName = terraform output -raw iam_role_name

What does this variable hold?

The RoleName variable holds the IAM role attached through the instance profile. Reading it from Terraform avoids guessing the generated name.

Before you inspect the policy, which resource value do you expect the failed object statement to contain?

  • Retrieve the inline upload policy by running this command:
aws iam get-role-policy --role-name $RoleName --policy-name private-ec2-s3-upload --profile process

What does the policy output show?

The response contains the rendered private-ec2-s3-upload policy from the EC2 role. Its two statements let you compare bucket-level listing with object-level uploading.

  • Locate the statement whose action is s3:ListBucket.
  • Confirm that its resource is the generated bucket ARN.
  • Locate the statement whose action is s3:PutObject.
  • Confirm that its resource is the same bucket ARN.
  • Identify /reports/* as the missing object path.

Why Does the Upload Fail?

The s3:ListBucket statement correctly targets the bucket ARN with a prefix condition. That resource shape authorizes the successful listing test.

The s3:PutObject statement also targets the bucket ARN because s3_object_arn_suffix is empty. An object upload requires the missing /reports/* path.

No infrastructure changed during this diagnosis. The failure evidence points to one resource-scope input in Terraform.

Policy Lookup Failed?

  • Confirm that RoleName contains the current Terraform output.
  • Confirm that the process profile still represents the intended AWS account.
  • Confirm that the policy name is private-ec2-s3-upload.

Help me diagnose the IAM policy lookup.

  • Switch back to the active Session Manager shell.

That is a disciplined diagnosis. Next, you will make the smallest Terraform correction to restore the report upload.

Repair S3 Access and Document the Lab

You traced the AccessDenied result to an Amazon S3 object permission that uses the bucket ARN. The evidence points to one precise IAM policy correction.

This step applies that correction through Terraform. You will prove the upload from the EC2 role. You will then preserve the incident story in documentation and Git history.

In this step, get ready to:
  • Scope the EC2 role's upload permission to objects under the reports/ prefix.
  • Upload the system report from the private instance and confirm the object in the AWS CLI and console.
  • Document both incidents and commit the evidence with a focused Git history.
Repair the object ARN

The s3:PutObject action applies to an object ARN. The /reports/* suffix narrows that permission to objects inside the report prefix.

  • Return to scenario.auto.tfvars in your terraform-private-ec2-lab folder.
  • Find these existing lines:
enable_ssm_core_policy = true
s3_object_arn_suffix   = ""

Why did the empty value fail?

Terraform appends s3_object_arn_suffix to the generated bucket ARN. An empty value leaves s3:PutObject pointed at the bucket instead of an object path.

  • Replace the empty suffix with /reports/* so the file looks like this:
enable_ssm_core_policy = true
s3_object_arn_suffix   = "/reports/*"

What does this change do?

The SSM repair remains enabled. The upload statement now resolves to the generated bucket ARN followed by /reports/*.

The separate s3:ListBucket statement stays scoped to the bucket ARN. Each permission now targets the correct S3 resource type.

  • Save scenario.auto.tfvars.
  • Return to Windows PowerShell from earlier.
  • Format the Terraform files by running this command:
terraform fmt -recursive

What does formatting check?

Terraform rewrites the configuration into its canonical format. This keeps the policy repair consistent with the rest of the repository.

  • Validate the configuration by running this command:
terraform validate

What does validation prove?

Terraform checks the initialized configuration for syntax and internal consistency. A successful result confirms that the new suffix is valid Terraform input.

Before you create the plan, predict whether this repair needs to replace the EC2 instance.

  • Create the saved S3 repair plan by running this command:
terraform plan -out=s3-fix.tfplan

What should the plan show?

The plan should update the inline S3 role policy. It should preserve the EC2 instance and networking resources.

Inspect the rendered resource value carefully. The object permission should resolve to the generated bucket ARN followed by /reports/*.

  • Review the saved plan for unexpected resource changes.
  • Apply the reviewed plan by running this command:
terraform apply s3-fix.tfplan

What did Terraform repair?

Terraform updates aws_iam_role_policy.s3_upload with the corrected object resource. The workload retains its existing EC2 role and private network placement.

That is the smallest safe repair. The role gains access only to report objects in the intended prefix.

Did the plan or apply fail?

Confirm that scenario.auto.tfvars contains matching double quotes around /reports/*. Confirm that your process profile is still authenticated.

If Terraform reports an unexpected replacement, stop before applying. Compare the plan with the single variable change.

Help me diagnose my S3 policy repair plan before I apply it.

Retry the report upload

The active Session Manager shell still uses temporary credentials from the EC2 role. Retrying from that shell proves the workload identity can use the repaired policy.

Before you retry the upload, predict whether the same report path now matches the policy resource.

  • Switch back to the active Session Manager shell.
  • Retry the system report upload by running this command:
aws s3 cp /tmp/system-report.txt "s3://${BUCKET_NAME}/reports/system-report.txt"

What does this upload prove?

The command sends /tmp/system-report.txt to the reports/system-report.txt object key. Success proves that the EC2 role can put an object inside the authorized prefix.

No embedded credentials are involved. The instance profile supplies temporary role credentials.

You should see confirmation that the report was uploaded. The same operation that returned AccessDenied now succeeds.

  • List the authorized report prefix by running this command:
aws s3 ls "s3://${BUCKET_NAME}/reports/"

What does the listing confirm?

The listing should include system-report.txt under the report prefix. This confirms the object exists at the path authorized by the final policy.

Strong result. You have now repaired least-privilege access and verified it from the workload that needed the permission.

  • Return to the AWS Management Console from earlier.
  • Use the console search bar to find Amazon S3.
  • Select the generated private reports bucket.
  • Open the reports/ prefix.
  • Confirm that system-report.txt is listed.
  • Return to Windows PowerShell from earlier.
  • Stage the Terraform remediation by running this command:
git add scenario.auto.tfvars

What are you staging?

Git stages only the scenario variable change. This keeps the permission repair separate from the documentation commit.

  • Commit the verified S3 repair by running this command:
git commit -m "fix: scope S3 uploads to the reports prefix"

Why use a focused commit?

The commit message records the specific policy correction. A reviewer can connect that change directly to the failed upload and successful retry.

Did Git refuse the commit?

Confirm that scenario.auto.tfvars is saved with the repaired suffix. A commit also needs the file to be staged.

Help me diagnose why my focused S3 remediation commit is failing.

Document and commit the evidence

Operational evidence loses value when another engineer cannot reproduce the reasoning. Your documentation connects the architecture to both failures and their smallest safe repairs.

Your Visual Studio Code workspace becomes the interview package. Each file answers a different question about the lab.

  • Create README.md inside the terraform-private-ec2-lab folder.
  • Create runbook.md inside the same folder.

You should now see both files in the Explorer alongside the Terraform configuration.

  • Create troubleshooting-debrief.md inside the terraform-private-ec2-lab folder.
  • Create cleanup-checklist.md inside the same folder.

You should now see all four evidence files in the Explorer.

Draft each document from the evidence you collected. Use the full-file references if you want to compare every required section.

✔️ Awesome, I've got everything!

Build the documentation from your own incident notes. Keep every claim tied to evidence you collected in the lab.

  • Describe the private EC2 architecture and both failure switches in README.md.
  • Record the repeatable diagnostic and operating commands in runbook.md.
  • Explain both root causes and repairs in troubleshooting-debrief.md.
  • Record the safe destruction sequence in cleanup-checklist.md.

ⓧ I'd like to double check the full code

Check scenario.auto.tfvars
enable_ssm_core_policy = true
s3_object_arn_suffix   = "/reports/*"

What should this file preserve?

The final scenario keeps Session Manager access enabled. It also limits report uploads to the intended object prefix.

Check README.md
# Terraform Private EC2 Troubleshooting Lab

This repository provisions a one-AZ AWS lab for practicing private EC2 operations, IAM troubleshooting, Systems Manager Session Manager, and S3 least-privilege access.

## Architecture

- One VPC in `us-east-1`
- One public subnet containing a NAT Gateway and Elastic IP
- One private subnet containing an Amazon Linux 2023 EC2 instance
- No public IP, SSH key, or inbound security group rules on the instance
- An EC2 instance profile with controlled Systems Manager and S3 permissions
- One private S3 bucket for reports under the `reports/` prefix

## Failure sequence

1. `enable_ssm_core_policy = false` leaves the role with only an intentionally incomplete Systems Manager permission. The instance runs, but a new Session Manager session is unavailable.
2. `enable_ssm_core_policy = true` attaches `AmazonSSMManagedInstanceCore` and restores Session Manager.
3. `s3_object_arn_suffix = ""` gives `s3:PutObject` a bucket ARN, so upload fails.
4. `s3_object_arn_suffix = "/reports/*"` scopes `s3:PutObject` to objects under the report prefix and restores upload.

## Workflow

Run `terraform init`, `terraform fmt -recursive`, `terraform validate`, `terraform plan -out=tfplan`, inspect the plan, and run `terraform apply tfplan`.

Commit `.terraform.lock.hcl`. Do not commit `.terraform/`, state files, or saved plan files.

## Evidence

See `runbook.md` for commands, `troubleshooting-debrief.md` for the root-cause narrative, and `cleanup-checklist.md` before leaving resources provisioned.

What does the README capture?

The README connects the one-AZ private architecture to the two deterministic failures. It also records the repeatable Terraform workflow and evidence locations.

Check runbook.md
# Operations Runbook

## Local variables

Run in Windows PowerShell from the Terraform root directory.

```powershell
$Region = "us-east-1"
$InstanceId = terraform output -raw instance_id
$RoleName = terraform output -raw iam_role_name
$BucketName = terraform output -raw bucket_name
$PrivateRouteTableId = terraform output -raw private_route_table_id
```

## Verify local identity

```powershell
aws sts get-caller-identity --profile process
```

## Inspect EC2 placement and identity

```powershell
aws ec2 describe-instances --instance-ids $InstanceId --profile process --region $Region --query "Reservations[0].Instances[0].{State:State.Name,PrivateIp:PrivateIpAddress,PublicIp:PublicIpAddress,IamProfile:IamInstanceProfile.Arn}" --output table
```

## Inspect the private default route

```powershell
aws ec2 describe-route-tables --route-table-ids $PrivateRouteTableId --profile process --region $Region
```

## Inspect managed-node registration

```powershell
aws ssm describe-instance-information --filters "Key=InstanceIds,Values=$InstanceId" --profile process --region $Region
```

## Inspect role policies

```powershell
aws iam list-attached-role-policies --role-name $RoleName --profile process
aws iam get-role-policy --role-name $RoleName --policy-name private-ec2-s3-upload --profile process
```

## Start a Session Manager shell

```powershell
aws ssm start-session --target $InstanceId --profile process --region $Region
```

## Verify the instance and create a report

Run inside the Session Manager shell. Replace the bucket placeholder with the value of `terraform output -raw bucket_name`.

```bash
export AWS_DEFAULT_REGION="us-east-1"
BUCKET_NAME="your-bucket-name-here"
sudo systemctl status amazon-ssm-agent
aws --version
aws sts get-caller-identity
date -u > /tmp/system-report.txt
hostname >> /tmp/system-report.txt
uname -a >> /tmp/system-report.txt
df -h >> /tmp/system-report.txt
aws sts get-caller-identity >> /tmp/system-report.txt
cat /tmp/system-report.txt
```

## Test upload and listing

```bash
aws s3 cp /tmp/system-report.txt "s3://${BUCKET_NAME}/reports/system-report.txt"
aws s3 ls "s3://${BUCKET_NAME}/reports/"
```

What does the runbook capture?

The runbook turns the investigation into a repeatable sequence. It covers identity, EC2 placement, routing, managed-node status, IAM policies, Session Manager access, report creation, and S3 verification.

Check troubleshooting-debrief.md
# Troubleshooting Debrief

## Incident 1: Session Manager unavailable

What I built: A private EC2 instance with NAT egress, no public IP, no inbound rules, and an attached instance profile.

What failed: The instance did not become an online Systems Manager managed node, so a new Session Manager shell could not start.

How I diagnosed it: I verified EC2 was running, confirmed the private address and absent public address, inspected the no-ingress security group, checked the private route table for a NAT default route, confirmed the instance profile ARN, and listed the role's attached managed policies.

Root cause: `AmazonSSMManagedInstanceCore` was absent. The intentionally incomplete inline permission was insufficient for Session Manager.

How I fixed it: I set `enable_ssm_core_policy = true`, reviewed the Terraform plan, applied it, waited for the managed node to report `Online`, and opened a new session.

Interview explanation: A running EC2 instance does not prove manageability. Session Manager depends on a running agent, an effective instance identity, sufficient SSM permissions, DNS, and outbound HTTPS connectivity.

## Incident 2: S3 upload denied

What I built: An EC2 role with `s3:ListBucket` on one bucket and `s3:PutObject` intended for one prefix.

What failed: Uploading `/tmp/system-report.txt` returned AccessDenied even though `s3:PutObject` appeared in the policy.

How I diagnosed it: I confirmed the caller identity, verified that listing and uploading are different operations, and inspected the inline policy rendered on the role.

Root cause: `s3:PutObject` targeted the bucket ARN. Object operations require an object ARN containing a slash after the bucket name.

How I fixed it: I set `s3_object_arn_suffix = "/reports/*"`, reviewed and applied the plan, retried the upload, and listed the resulting object.

Interview explanation: For an S3 403, I inspect principal, action, resource, condition, and explicit denies. I avoid broadening to `s3:*` and instead correct the resource type and prefix.

What does the debrief capture?

The debrief preserves the symptom, evidence, root cause, repair, and interview explanation for each incident. It shows a methodical troubleshooting process instead of a guessed fix.

Check cleanup-checklist.md
# Cleanup Checklist

- Exit every Session Manager shell.
- Capture the bucket name before destroying Terraform outputs.
- Empty the S3 bucket.
- Review a Terraform destroy plan.
- Apply the destroy plan.
- Confirm deletion of the EC2 instance and its root volume.
- Confirm deletion of the IAM role, inline policies, managed policy attachment, and instance profile.
- Confirm deletion of the S3 bucket and Public Access Block configuration.
- Confirm deletion of the NAT Gateway and release of its Elastic IP.
- Confirm deletion of both route tables and associations, both subnets, the internet gateway, and the VPC.
- Run `terraform state list` and confirm no managed resources are printed.
- Keep configuration and `.terraform.lock.hcl` in Git.
- Do not delete state before a successful destroy.
- Remove local saved plan files after cleanup.
- Run `aws logout --profile signin` when finished.

## Cleanup commands

Run in Windows PowerShell from the Terraform root directory.

```powershell
$BucketName = terraform output -raw bucket_name
aws s3 rm "s3://$BucketName" --recursive --profile process --region us-east-1
terraform plan -destroy -out=destroy.tfplan
terraform apply destroy.tfplan
terraform state list
aws logout --profile signin
```

What does the checklist protect?

The checklist preserves the order required for a safe destroy. It empties the bucket before Terraform removes it. It also keeps state available until infrastructure deletion succeeds.

  • Update README.md to match the full-file reference.
  • Update runbook.md to match the full-file reference.
  • Save both files.
  • Confirm that the README links to the runbook and evidence documents.
  • Update troubleshooting-debrief.md to match the full-file reference.
  • Update cleanup-checklist.md to match the full-file reference.
  • Save both files.
  • Confirm that the debrief contains two incidents and the cleanup file ends with the PowerShell commands.
  • Stage the four documentation files by running this command:
git add README.md runbook.md troubleshooting-debrief.md cleanup-checklist.md

What are you staging?

Git stages the architecture overview, operations runbook, troubleshooting debrief, and cleanup checklist together. The commit represents the evidence package as one documentation change.

  • Commit the documentation package by running this command:
git commit -m "docs: add runbook and interview debrief"

Why separate the documentation commit?

The infrastructure repair already has its own commit. This second commit lets reviewers inspect the operational explanation without mixing it into the policy change.

Are documentation files missing from the commit?

Confirm that all four files are saved inside terraform-private-ec2-lab. Confirm that each filename matches the staged command exactly.

Help me diagnose missing files or an incomplete documentation commit.

Before you inspect the history, predict how the baseline and two repairs should read as a short incident story.

  • Display the compact Git history by running this command:
git log --oneline

What should the history show?

You should see separate entries for the private EC2 failure baseline, the Session Manager repair, the S3 prefix repair, and the documentation package. The abbreviated commit IDs can differ.

The sequence should include feat: provision private EC2 failure baseline, fix: attach AmazonSSMManagedInstanceCore, fix: scope S3 uploads to the reports prefix, and docs: add runbook and interview debrief.

You now have a working least-privilege report path and evidence that explains every repair. The provisioned lab remains ready for the optional Secret Mission.

Secret mission

Reproduce the SSM Regression Under Interview Conditions

Recreate the Session Manager permission regression under a five-minute clock. Use evidence to isolate the missing policy. Restore access through Terraform before delivering a 60-second incident answer.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to keep the lab available for more practice, hold it briefly, or remove it completely. This project uses billable AWS resources, so deleting the infrastructure immediately is the safest option.

Cost warning

Leaving the lab provisioned keeps hourly charges active for the NAT Gateway and its public IPv4 address. The EC2 instance, EBS volume, S3 storage, requests, and NAT data processing may also incur charges.

A stopped instance still leaves several chargeable resources provisioned. Destroy the lab after collecting your evidence.

Resources you used:

  • Terraform-managed networking with one VPC, one public subnet, one private subnet, one internet gateway, two route tables, two route-table associations, one NAT Gateway, and one Elastic IP.
  • Private compute with aws_instance.private, its encrypted root volume, and its no-ingress security group.
  • IAM resources with one EC2 role, one instance profile, the AmazonSSMManagedInstanceCore attachment, and the private-ec2-s3-upload inline policy.
  • Private S3 storage with the generated report bucket, its Public Access Block configuration, and reports/system-report.txt.
  • Local Terraform artifacts with saved .tfplan files and local .tfstate data.

Keep everything running

No action is needed to keep the environment available. Choose this only if you are actively repeating the incident exercises and accept the ongoing AWS charges.

  • Record the resource owner in cleanup-checklist.md.
  • Record a firm expiry time in cleanup-checklist.md.
  • Record us-east-1 as the active Region.
  • Record the expected ongoing NAT and compute charges.
  • Keep the Terraform configuration in Git.
  • Keep .terraform.lock.hcl with the configuration.
  • Keep local state available for the eventual destroy operation.

Set a short expiry

This disposable lab becomes easy to forget once the troubleshooting work is complete. Use the recorded expiry time as the point when you destroy every AWS resource.

Pause - I'll come back to this later

This lab has no effective low-cost pause. Stopping the EC2 instance leaves the NAT Gateway, Elastic IP, EBS volume, and S3 resources provisioned.

  • Exit the Session Manager shell from earlier.
  • Treat this option as a short holding period only.
  • Record a near-term deletion time in cleanup-checklist.md.
  • Keep the Terraform configuration unchanged.
  • Keep local state until the destroy operation succeeds.
  • Return to the delete option before the recorded expiry time.

Stopping EC2 is not enough

The NAT Gateway and public IPv4 address continue accruing hourly charges while the instance is stopped. Use the delete workflow when you finish your short break.

Delete - I don't want to use this again

Deletion removes the billable cloud infrastructure while preserving your configuration and interview evidence in Git. The saved destroy plan lets you review the full removal before Terraform changes anything.

  • Exit the Session Manager shell from earlier.
  • Switch back to the Windows PowerShell terminal you used for Terraform.
  • Empty the generated S3 bucket by running these commands:
$BucketName = terraform output -raw bucket_name
aws s3 rm "s3://$BucketName" --recursive --profile process --region us-east-1

Why empty the bucket first?

  • The first command reads the generated bucket name from the Terraform output.
  • The second command recursively deletes reports/system-report.txt from that bucket.
  • The bucket uses force_destroy = false, so Terraform can remove it only after it is empty.

Bucket cleanup failed?

Confirm that the process profile is still authenticated. Refresh the temporary browser login from the setup step if authentication has expired.

Keep the bucket and local state intact until the object deletion succeeds. Help me diagnose why the generated S3 bucket did not empty.

  • Create the saved destroy plan by running this command:
terraform plan -destroy -out=destroy.tfplan

What does this plan do?

Terraform previews the infrastructure it intends to delete. It saves that exact proposal in destroy.tfplan for the apply step.

  • Inspect the destroy plan for unexpected changes.
  • Confirm that the plan removes only the resources from this lab.

Unexpected resources in the plan?

Stop before applying the plan. Confirm that Windows PowerShell still points to the Terraform folder containing this lab's files and state.

Do not delete local state to force a clean result. Help me review an unexpected Terraform destroy plan.

  • Apply the reviewed destroy plan by running this command:
terraform apply destroy.tfplan

What does this apply do?

Terraform executes the reviewed deletion sequence from destroy.tfplan. Dependency ordering lets it remove the compute, identity, storage, and networking resources safely.

Destroy did not complete?

Keep the configuration and state files in place. Resolve the reported resource failure before running another reviewed destroy plan.

A remaining S3 object can prevent the bucket from being deleted. Help me recover from a failed Terraform destroy.

  • Confirm in the AWS Management Console that the EC2 instance and encrypted root volume are gone.
  • Confirm in the AWS Management Console that the IAM role and instance profile are gone.
  • Confirm in the AWS Management Console that the S3 bucket is gone.
  • Confirm in the AWS Management Console that the NAT Gateway and Elastic IP are gone.
  • Confirm in the AWS Management Console that both route tables and associations are gone.
  • Confirm in the AWS Management Console that both subnets are gone.
  • Confirm in the AWS Management Console that the internet gateway and VPC are gone.
  • Verify that Terraform tracks no managed resources by running this command:
terraform state list

What does an empty result mean?

An empty result confirms that Terraform has no managed resource addresses left in local state. This is the final infrastructure cleanup check.

You should see no managed resource addresses in the output.

That is the cost-bearing infrastructure removed. Your configuration and troubleshooting evidence remain available in Git.

  • Delete local files ending in .tfplan from the Terraform folder with Windows File Explorer.
  • Delete the empty local state file only if no recovery or audit need remains.
  • Keep the Terraform configuration in Git.
  • Keep .terraform.lock.hcl in Git.
  • End the temporary browser-authenticated AWS sign-in by running this command:
aws logout --profile signin

What does logout do?

This ends the temporary sign-in session for the signin profile after the cleanup checks are complete.

Nice Work!

Nice Work!

You did it! Your Terraform-managed private Amazon EC2 lab now supports administration through Session Manager plus least-privilege report uploads to Amazon S3.

What you learned:

  • Built a private EC2 environment with one-AZ networking plus NAT-based egress. Kept administrative access private with no public instance address, SSH key, or inbound security group rules.
  • Diagnosed an AWS Systems Manager failure by checking compute health, identity, routing, permissions, and managed-node evidence. Restored Session Manager through a reviewed Terraform plan that attached AmazonSSMManagedInstanceCore.
  • Traced an S3 authorization failure to a bucket ARN used for an object operation. Repaired the policy with upload access limited to reports/*. Preserved the investigation in a focused Git history plus an interview-ready runbook and troubleshooting debrief.
  • Secret Mission: Reproduced the SSM permission regression within a five-minute timebox. Restored service through Terraform. Added a 60-second incident narrative covering the evidence, root cause, fix, and prevention.

Ready to quiz yourself?