Terraform Troubleshooting Lab

Diagnose and fix real-world Terraform failures: drift, cycles, and broken state.

Introduction

30 Second Summary

When Terraform breaks in production, the error messages can feel cryptic and the stakes are high. Knowing how to systematically diagnose and fix failures is what separates day-one users from engineers who can keep infrastructure running.

In this project, you will build a troubleshooting lab where you deploy real AWS infrastructure, deliberately break it in common ways, and fix each failure using production-grade workflows.

What You'll Build

A Terraform troubleshooting lab that deploys S3 buckets and security groups to AWS, then walks you through diagnosing and resolving drift, dependency cycles, and broken state.

By the end of this project, you'll have:

  • A Terraform configuration managing an S3 bucket and security groups in AWS.
  • A documented workflow for detecting and reconciling infrastructure drift with refresh-only plans.
  • Experience breaking and fixing dependency cycles using terraform graph and completed state surgery using import, moved, and removed blocks.
  • Secret Mission: Combine TF_LOG=TRACE debugging with multiple simultaneous failures to diagnose and fix a broken configuration under pressure.

Are there any prerequisites?

This project assumes you can write a basic Terraform resource block and run init/plan/apply. You also need an AWS account with the AWS CLI configured and Terraform installed.

Set Up the Project

With Terraform and the AWS CLI confirmed working, you have the two core tools this lab depends on.

Now you need a project directory with the configuration files that tell Terraform which providers to download and which cloud region to target. In this step, you'll create the project structure and initialize Terraform so it's ready to manage real AWS resources.

In this step, get ready to:
  • Create the project directory and configuration files.
  • Write the Terraform block with provider requirements.
  • Initialize Terraform to download the AWS and random providers.
Create the project directory and files

Every Terraform project lives in its own directory. You'll create one on your Desktop so it's easy to find.

  • Open your terminal by pressing Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project directory and move into it by running these commands:
mkdir terraform-troubleshooting-lab
cd terraform-troubleshooting-lab
  • Create the three configuration files your project needs:

macOS

touch main.tf variables.tf outputs.tf

Windows

New-Item -ItemType File -Name "main.tf", "variables.tf", "outputs.tf"
  • Confirm the files exist by running:
ls

You should see main.tf, variables.tf, and outputs.tf in the output.

  • Open this project in VS Code by running:
code .

Command not found for code?

If code . does not work, open VS Code manually. Press Cmd+Space (macOS) or the Windows key (Windows), type VS Code, and press Enter.

Then click File in the top menu bar, click Open Folder, and select the terraform-troubleshooting-lab folder on your Desktop.

How do I add VS Code to my PATH?

Write the Terraform and provider configuration

The main.tf file is where Terraform looks for its core settings. You need to tell it which version of Terraform to require and which providers to download.

  • In the VS Code Explorer sidebar, click main.tf to open it.
  • Paste the following configuration into main.tf:
# Define Terraform settings and required providers
terraform {
  required_version = ">= 1.15.0"

  required_providers {
    # AWS provider for managing cloud infrastructure
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
    # Random provider for generating unique resource name suffixes
    random = {
      source  = "hashicorp/random"
      version = "~> 3.6"
    }
  }
}

# Set the AWS region where resources will be created
provider "aws" {
  region = "us-east-1"
}

What does this configuration do?

  • required_version = ">= 1.15.0" ensures anyone running this project has Terraform 1.15.0 or newer installed.
  • The aws provider (~> 6.0) gives Terraform the ability to create and manage AWS resources like S3 buckets and security groups.
  • The random provider (~> 3.6) generates random values you'll use later to create globally unique resource names.
  • The provider "aws" block sets us-east-1 as the region where all AWS resources will be created.
  • Save main.tf by pressing Cmd+S (macOS) or Ctrl+S (Windows).

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

terraform {
  required_version = ">= 1.15.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.6"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}
Initialize Terraform

Terraform needs to download the provider plugins before it can manage any resources. The terraform init command reads your configuration and fetches everything listed in required_providers.

  • Open the integrated terminal in VS Code by pressing Ctrl+` (backtick key, same shortcut on macOS and Windows).
  • Download the providers by running this command:
terraform init

You should see a success message confirming Terraform has been initialized. The output also lists the provider versions it downloaded.

  • Confirm the initialization created the expected files by running:
ls

You should see a .terraform directory and a .terraform.lock.hcl file alongside your three .tf files.

What are these new files?

The .terraform directory stores the downloaded provider plugins. Terraform reads from this directory every time you run plan or apply.

The .terraform.lock.hcl file locks the exact provider versions so every team member gets identical builds. Think of it like a package-lock.json for infrastructure.

Seeing a provider installation error?

Make sure your main.tf is saved. Terraform reads from disk, so unsaved changes in your editor won't be picked up.

Also confirm your terminal's working directory is inside terraform-troubleshooting-lab. Run pwd to check your current path.

Help me troubleshoot a terraform init failure

Your project skeleton is ready and Terraform has its providers. Next up, you'll write the actual resource configuration and deploy real AWS infrastructure.

Deploy Baseline Infrastructure

Your Terraform project is initialized and ready to manage infrastructure. Now you need to define the actual AWS resources that form your troubleshooting lab.

In this step, you will deploy a small but realistic environment: an S3 bucket with versioning and two security groups that simulate a web tier and an app tier. These resources give you real infrastructure to break and fix in later steps.

In this step, get ready to:
  • Write the baseline Terraform configuration with an S3 bucket and two security groups.
  • Deploy the infrastructure to AWS with terraform apply.
  • Record resource identifiers using state and output commands.
Write the baseline configuration

You will build up the configuration across three files. Start with the variable that names all your resources, then add the AWS resources themselves.

  • In VS Code, open variables.tf and add the following code:
variable "project_name" {
  description = "Name prefix for all resources in this lab"
  type        = string
  default     = "tf-troubleshoot-lab"
}

What does this code do?

This defines a reusable variable called project_name that prefixes every resource name in your lab.

The default value tf-troubleshoot-lab is used automatically unless you override it at the command line. This keeps your resource names consistent and easy to identify in the AWS console.

  • Save variables.tf.
  • You won't see a change yet. The next chunk uses this variable to name your AWS resources.
  • Switch to main.tf and add the following code below the provider "aws" block:
resource "random_id" "suffix" {
  byte_length = 4
}

resource "aws_s3_bucket" "lab" {
  bucket = "${var.project_name}-${random_id.suffix.hex}"
}

resource "aws_s3_bucket_versioning" "lab" {
  bucket = aws_s3_bucket.lab.id

  versioning_configuration {
    status = "Enabled"
  }
}

What does this code do?

  • random_id.suffix generates a random 4-byte hex string. S3 bucket names must be globally unique, so appending this random suffix prevents naming collisions.
  • aws_s3_bucket.lab creates the bucket using your project name plus the random suffix.
  • aws_s3_bucket_versioning.lab enables versioning on the bucket. Versioning keeps a history of object changes, which is common in production setups.
  • Save main.tf.
  • Confirm your configuration is valid so far by running this in your terminal:
terraform validate

You should see Success! The configuration is valid. in the output.

Seeing a validation error?

  • Check that variables.tf is saved. If var.project_name is referenced in main.tf but the variable file is empty, Terraform cannot resolve it.
  • Make sure the random_id resource block comes before the aws_s3_bucket block that references it.

Help me fix my Terraform validation error

  • Still in main.tf, add the web tier security group below the versioning resource:
resource "aws_security_group" "web_sg" {
  name        = "${var.project_name}-web-sg"
  description = "Web tier security group for troubleshooting lab"

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "${var.project_name}-web-sg"
  }
}

What does this code do?

This creates a security group that simulates a web-facing tier. It allows inbound HTTP traffic on port 80 from anywhere and permits all outbound traffic.

The protocol = "-1" in the egress block means "all protocols". This is AWS shorthand for allowing any type of outbound connection.

  • Save main.tf.
  • You won't see a change yet. The next chunk adds the app tier security group.
  • Still in main.tf, add the app tier security group below web_sg:
resource "aws_security_group" "app_sg" {
  name        = "${var.project_name}-app-sg"
  description = "App tier security group for troubleshooting lab"

  ingress {
    from_port   = 8080
    to_port     = 8080
    protocol    = "tcp"
    cidr_blocks = ["10.0.0.0/16"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "${var.project_name}-app-sg"
  }
}

What does this code do?

This creates a security group for an internal app tier. It only accepts inbound traffic on port 8080 from the private subnet range 10.0.0.0/16.

Together, web_sg and app_sg simulate a two-tier architecture. In later steps, you will create dependency cycles between them to practice troubleshooting.

  • Save main.tf.
  • Confirm the full configuration is valid by running:
terraform validate

You should see Success! The configuration is valid. confirming all resources are syntactically correct.

Seeing an error about the security group?

  • Make sure every opening brace { has a matching closing brace }. Each resource, ingress, egress, and tags block needs its own pair.
  • Check that cidr_blocks uses square brackets (it is a list): ["0.0.0.0/0"] not "0.0.0.0/0".

Help me fix my security group configuration

✔️ Awesome, I've got everything!

Great. Double check you have saved both variables.tf and main.tf.

ⓧ I'd like to double check the full code

Here is what variables.tf should look like:

variable "project_name" {
  description = "Name prefix for all resources in this lab"
  type        = string
  default     = "tf-troubleshoot-lab"
}

Here is what main.tf should look like:

terraform {
  required_version = ">= 1.15.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.6"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

resource "random_id" "suffix" {
  byte_length = 4
}

resource "aws_s3_bucket" "lab" {
  bucket = "${var.project_name}-${random_id.suffix.hex}"
}

resource "aws_s3_bucket_versioning" "lab" {
  bucket = aws_s3_bucket.lab.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_security_group" "web_sg" {
  name        = "${var.project_name}-web-sg"
  description = "Web tier security group for troubleshooting lab"

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "${var.project_name}-web-sg"
  }
}

resource "aws_security_group" "app_sg" {
  name        = "${var.project_name}-app-sg"
  description = "App tier security group for troubleshooting lab"

  ingress {
    from_port   = 8080
    to_port     = 8080
    protocol    = "tcp"
    cidr_blocks = ["10.0.0.0/16"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "${var.project_name}-app-sg"
  }
}
Deploy the infrastructure

Your configuration defines what you want. Now you will ask Terraform to create it in AWS.

  • Preview the resources Terraform will create by running:
terraform plan

You should see Plan: 5 to add, 0 to change, 0 to destroy. in the output. The five resources are the random ID, the S3 bucket, the versioning configuration, and the two security groups.

What is terraform plan?

terraform plan is a dry run. It compares your configuration to the current state and shows what Terraform would change without actually touching AWS.

Always review the plan before applying to catch mistakes early.

Seeing fewer than 5 resources?

  • Make sure both security groups are present in main.tf. If you only see 3 resources, one or both security group blocks may be missing.
  • If you see an error instead of a plan, run terraform validate first to pinpoint the syntax issue.

Help me figure out why my terraform plan is wrong

  • Deploy all five resources to AWS by running:
terraform apply -auto-approve

You should see Apply complete! Resources: 5 added, 0 changed, 0 destroyed. at the end of the output.

What does -auto-approve do?

Normally terraform apply asks you to type yes before making changes.

The -auto-approve flag skips that prompt, which is useful in labs and CI/CD pipelines where you have already reviewed the plan.

Apply failed with an access denied error?

  • Confirm your AWS credentials are still valid by running aws sts get-caller-identity. If that fails, re-run aws configure with fresh credentials.
  • Make sure the IAM user or role attached to your credentials has permissions to create S3 buckets and EC2 security groups.

Help me fix my terraform apply access error

Record resource identifiers

Terraform tracks every resource it manages in a state file. You need the resource addresses and IDs to introduce drift and perform state surgery in later steps.

  • List all managed resources by running:
terraform state list

You should see five entries including aws_s3_bucket.lab, aws_security_group.web_sg, and aws_security_group.app_sg.

  • Now define outputs so you can retrieve the resource IDs easily. Open outputs.tf and add the following code:
output "bucket_name" {
  description = "Name of the S3 bucket"
  value       = aws_s3_bucket.lab.bucket
}

output "web_sg_id" {
  description = "ID of the web security group"
  value       = aws_security_group.web_sg.id
}

output "app_sg_id" {
  description = "ID of the app security group"
  value       = aws_security_group.app_sg.id
}

What do outputs do?

Outputs expose values from your Terraform state so you can reference them easily.

After applying, you can run terraform output to see bucket names, security group IDs, and any other values you need for later commands.

  • Save outputs.tf.
  • Apply the outputs by running:
terraform apply -auto-approve
  • View your resource identifiers by running:
terraform output

You should see three values: your bucket name (something like tf-troubleshoot-lab-a1b2c3d4), and the two security group IDs (starting with sg-).

Save these IDs for later

You will use the bucket name and security group IDs in the next step when you create drift. Keep this terminal open or note down the values from terraform output.

Outputs showing empty or missing?

  • Make sure you ran terraform apply after adding the outputs. Terraform only registers new outputs after an apply.
  • Check that the value lines reference the correct resource names: aws_s3_bucket.lab.bucket, aws_security_group.web_sg.id, and aws_security_group.app_sg.id.

Help me fix my Terraform outputs

✔️ Awesome, I've got everything!

Great. Make sure all three files are saved.

ⓧ I'd like to double check the full code

Here is what outputs.tf should look like:

output "bucket_name" {
  description = "Name of the S3 bucket"
  value       = aws_s3_bucket.lab.bucket
}

output "web_sg_id" {
  description = "ID of the web security group"
  value       = aws_security_group.web_sg.id
}

output "app_sg_id" {
  description = "ID of the app security group"
  value       = aws_security_group.app_sg.id
}

Your baseline lab environment is live in AWS. Next up, you will break things on purpose by introducing drift between what Terraform expects and what actually exists in AWS.

Introduce and Resolve Drift

Your baseline infrastructure is live in AWS. Now it is time to simulate the most common production headache: state drift.

Drift happens when someone (or an automated process) changes infrastructure outside of Terraform. Terraform's state file no longer matches reality, and your next plan may propose unexpected changes. In this step, you will deliberately introduce drift, detect it, and reconcile it using the exact workflow production engineers follow.

In this step, get ready to:
  • Create real drift by modifying AWS resources outside Terraform.
  • Detect the drift using a refresh-only plan.
  • Reconcile the drift by accepting or re-imposing your configuration.
Create drift by modifying infrastructure outside Terraform

In production, drift usually happens when a teammate makes a quick fix in the AWS console or runs a one-off CLI command. You will simulate this by adding a tag to your S3 bucket and an unauthorized ingress rule to your web security group.

  • Add a tag to your S3 bucket that Terraform does not know about by running this command:
aws s3api put-bucket-tagging --bucket [[BUCKET_NAME="your bucket name from terraform output"]] --tagging "TagSet=[{Key=Team,Value=platform-eng}]"

What does this command do?

The aws s3api put-bucket-tagging command adds a tag (Team=platform-eng) directly to your bucket via the AWS CLI. Terraform has no record of this tag in its state file or your configuration.

Find your bucket name quickly

Run terraform output bucket_name in your terminal to see the exact bucket name. Copy and paste it into the command above.

  • Add an unauthorized ingress rule to your web security group by running this command:
aws ec2 authorize-security-group-ingress --group-id [[WEB_SG_ID="your web_sg_id from terraform output"]] --protocol tcp --port 9090 --cidr 0.0.0.0/0

What does this command do?

This opens port 9090 to the entire internet on your web security group. Your Terraform configuration only defines port 80 as allowed. This simulates someone bypassing the Terraform workflow to make a "quick fix" that introduces security risk.

You should see a JSON response confirming the new rule was added, including a SecurityGroupRuleId.

Command failed with an error?

Make sure you replaced the placeholder with your actual security group ID. Run terraform output web_sg_id to retrieve it.

If you see a permissions error, confirm your AWS CLI is using the same credentials that created the resources.

Help me fix an AWS CLI authorization error

Detect drift with a refresh-only plan

Your state file still reflects the original infrastructure. Terraform does not know about the changes you just made. A refresh-only plan compares state to reality without proposing any infrastructure changes.

  • Detect the drift by running this command:
terraform plan -refresh-only

You should see a message that says "Note: Objects have changed outside of Terraform" followed by a list of detected differences.

What does the output tell you?

Terraform refreshes each resource against AWS and reports what changed. You will see the S3 bucket's new tag and the security group's extra ingress rule highlighted with ~ (update) markers.

The key line is: "This is a refresh-only plan, so Terraform will not take any actions to undo these." This means you are only observing the drift, not fixing it yet.

What are my two choices?

When you detect drift, you have two options. You can accept reality into state (update state to match what actually exists in AWS). Or you can re-impose your config (run a normal apply to force AWS back to what your .tf files define).

In the next substep, you will do both: accept the harmless tag drift, but reject the unauthorized security group rule.

Reconcile the drift

Now you will apply a two-part reconciliation strategy. First, accept the tag into state because it is a harmless team label. Second, re-impose your config to remove the unauthorized port 9090 rule.

  • Accept all detected drift into your state file by running this command:
terraform apply -refresh-only

What does this do?

This updates your Terraform state file to match what actually exists in AWS right now. It does not change any real infrastructure. After this command, Terraform "knows" about the tag and the extra security group rule.

Terraform will prompt you to confirm. Type yes and press Enter.

You should see a message confirming the state was updated without modifying any real infrastructure.

  • Re-impose your configuration to remove the unauthorized port 9090 rule by running:
terraform apply

What happens here?

A normal terraform apply compares your .tf configuration to the current state. Since your config only defines port 80 for web_sg, Terraform will remove the port 9090 rule you added via the CLI.

The S3 tag stays because it is now in your state but not explicitly defined in your config. Terraform will not remove attributes it does not manage.

  • Type yes when prompted to confirm the apply.

You should see Terraform report that it updated the security group in-place to remove the unauthorized rule.

  • Confirm zero drift remains by running:
terraform plan

You should see "No changes. Your infrastructure matches the configuration." This confirms your state, config, and AWS reality are all in sync.

Still seeing planned changes?

If Terraform still shows changes, you may have missed the terraform apply -refresh-only step. Run it again to sync state, then run terraform apply to re-impose your config.

Help me resolve lingering drift in my Terraform plan

Your drift is fully reconciled. Next up, you will tackle a more structural problem: dependency cycles that prevent Terraform from planning at all.

Create and Fix Dependency Cycles

In the previous step you detected and reconciled drift between your Terraform state and real infrastructure. Now you'll tackle a different class of failure: dependency cycles.

A dependency cycle happens when two resources reference each other, creating a circular loop that Terraform cannot resolve. Security groups that need to allow traffic to each other are the most common source of cycles in real AWS configurations. In this step, you'll deliberately introduce a cycle, diagnose it, and apply the production-grade fix.

In this step, get ready to:
  • Introduce a circular dependency between two security groups.
  • Diagnose the cycle using terraform graph.
  • Fix the cycle by decoupling inline rules into separate resources.
Introduce a circular dependency

To see how Terraform handles cycles, you'll modify both security groups to reference each other's ID in inline ingress rules. This is the exact pattern that causes cycles in production AWS configs.

  • In VS Code, open main.tf.
  • Find the aws_security_group.web_sg resource block.
  • Add a new ingress block directly below the existing ingress block (before the egress block) that references app_sg:
  ingress {
    from_port       = 8080
    to_port         = 8080
    protocol        = "tcp"
    security_groups = [aws_security_group.app_sg.id]
  }
  • Now find the aws_security_group.app_sg resource block.
  • Add a new ingress block directly below the existing ingress block (before the egress block) that references web_sg:
  ingress {
    from_port       = 80
    to_port         = 80
    protocol        = "tcp"
    security_groups = [aws_security_group.web_sg.id]
  }

What do these changes do?

You've made web_sg depend on app_sg (to read its ID), and app_sg depend on web_sg (to read its ID).

Neither can be created first because each needs the other to already exist. Terraform detects this loop and refuses to plan.

  • Save main.tf.
  • Run terraform plan in your terminal to trigger the cycle error:
terraform plan

Terraform outputs an error beginning with Error: Cycle: followed by the resources involved in the circular dependency.

Not seeing the cycle error?

Make sure both security groups reference each other. If only one references the other, Terraform can resolve the order and won't report a cycle.

Double-check that you used aws_security_group.app_sg.id inside web_sg and aws_security_group.web_sg.id inside app_sg.

help me fix my Terraform cycle error

Diagnose with terraform graph

When you hit a cycle in a real project, the error message alone might not make the root cause obvious. The terraform graph command outputs the full dependency graph so you can trace exactly which resources form the loop.

  • Generate the dependency graph and save it to a file by running:
terraform graph > graph.dot

What does this command do?

terraform graph outputs the dependency graph in DOT format, a text-based graph description language.

The > graph.dot part redirects the output into a file instead of printing it to your terminal. You can open this file to inspect the relationships between your resources.

  • Open graph.dot in VS Code.
  • Look for the two security group nodes pointing at each other. Lines like aws_security_group.web_sg -> aws_security_group.app_sg and aws_security_group.app_sg -> aws_security_group.web_sg confirm the cycle.

Why not just read the error?

For two resources the error is obvious. In production configs with hundreds of resources, cycles can span three or more nodes. The graph file lets you trace the full chain.

Fix the cycle by decoupling rules

The fix is a standard pattern: remove the cross-referencing inline rules and create separate aws_security_group_rule resources instead. Separate rule resources break the cycle because Terraform can create the security groups first, then create the rules that reference them.

  • In main.tf, find the ingress block you added to web_sg (the one with security_groups = [aws_security_group.app_sg.id]).
  • Delete that entire ingress block.
  • Find the ingress block you added to app_sg (the one with security_groups = [aws_security_group.web_sg.id]).
  • Delete that entire ingress block as well.

Both security groups should now look exactly as they did at the start of this step, with their original CIDR-based ingress rules.

  • At the bottom of main.tf, add the two separate security group rule resources by copying this code:
# After Step 4 fix: separate security group rules replacing inline rules
resource "aws_security_group_rule" "web_to_app" {
  type                     = "ingress"
  from_port                = 8080
  to_port                  = 8080
  protocol                 = "tcp"
  security_group_id        = aws_security_group.app_sg.id
  source_security_group_id = aws_security_group.web_sg.id
}

resource "aws_security_group_rule" "app_to_web" {
  type                     = "ingress"
  from_port                = 80
  to_port                  = 80
  protocol                 = "tcp"
  security_group_id        = aws_security_group.web_sg.id
  source_security_group_id = aws_security_group.app_sg.id
}

Why does this fix the cycle?

The aws_security_group_rule resources are separate from the security groups themselves. Terraform creates the security groups first (no cross-references between them), then creates the rule resources that reference both.

The security_group_id field says which group the rule attaches to. The source_security_group_id field says which group is allowed as the traffic source.

  • Save main.tf.
  • Confirm the cycle is resolved by running:
terraform plan

Terraform should show a successful plan with 2 resources to add (the two new rule resources). No cycle error this time.

  • Apply the fixed configuration by running:
terraform apply -auto-approve
  • Verify the state now includes the new rules by running:
terraform state list

You should see 7 resources listed, including aws_security_group_rule.web_to_app and aws_security_group_rule.app_to_web.

Still seeing a cycle error?

Make sure you completely removed the inline ingress blocks that contained security_groups references. The original CIDR-based ingress blocks (port 80 with 0.0.0.0/0 and port 8080 with 10.0.0.0/16) should remain.

Check that your new aws_security_group_rule resources are outside of any security group block, at the top level of your configuration.

help me fix my Terraform dependency cycle

✔️ Awesome, I've got everything!

Great. Double-check you've saved main.tf and that terraform state list shows 7 resources.

ⓧ I'd like to double check the full code

Here is the complete main.tf after fixing the cycle:

terraform {
  required_version = ">= 1.15.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.6"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

resource "random_id" "suffix" {
  byte_length = 4
}

resource "aws_s3_bucket" "lab" {
  bucket = "${var.project_name}-${random_id.suffix.hex}"
}

resource "aws_s3_bucket_versioning" "lab" {
  bucket = aws_s3_bucket.lab.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_security_group" "web_sg" {
  name        = "${var.project_name}-web-sg"
  description = "Web tier security group for troubleshooting lab"

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "${var.project_name}-web-sg"
  }
}

resource "aws_security_group" "app_sg" {
  name        = "${var.project_name}-app-sg"
  description = "App tier security group for troubleshooting lab"

  ingress {
    from_port   = 8080
    to_port     = 8080
    protocol    = "tcp"
    cidr_blocks = ["10.0.0.0/16"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "${var.project_name}-app-sg"
  }
}

# After Step 4 fix: separate security group rules replacing inline rules
resource "aws_security_group_rule" "web_to_app" {
  type                     = "ingress"
  from_port                = 8080
  to_port                  = 8080
  protocol                 = "tcp"
  security_group_id        = aws_security_group.app_sg.id
  source_security_group_id = aws_security_group.web_sg.id
}

resource "aws_security_group_rule" "app_to_web" {
  type                     = "ingress"
  from_port                = 80
  to_port                  = 80
  protocol                 = "tcp"
  security_group_id        = aws_security_group.web_sg.id
  source_security_group_id = aws_security_group.app_sg.id
}

You've now experienced the full lifecycle of a dependency cycle: creation, diagnosis, and fix. Next up, you'll learn state surgery techniques to import, rename, and safely remove resources from Terraform's management.

Perform State Surgery

You now have a clean Terraform state with zero drift and no dependency cycles. The next challenge is managing resources that exist outside your configuration.

In production, teams frequently need to bring existing cloud resources under Terraform management, rename resources without destroying them, or stop managing a resource while keeping it alive. Terraform provides three declarative blocks for this: import, moved, and removed.

In this step, get ready to:
  • Import an unmanaged S3 bucket into Terraform state using an import block.
  • Rename a resource in state using a moved block (no destroy/create).
  • Remove a resource from state without destroying it using a removed block.
Import an unmanaged resource

Sometimes infrastructure already exists in AWS before anyone writes Terraform for it. Import blocks let you bring those resources under Terraform management without recreating them.

  • Create a new S3 bucket manually via the AWS CLI by running this command:
aws s3 mb s3://[[IMPORT_BUCKET="tf-lab-import-test-your-unique-suffix"]]

What does this command do?

The aws s3 mb command creates ("makes") a new S3 bucket directly in AWS. This simulates a resource that someone created manually through the console or CLI, outside of Terraform.

  • You should see output confirming the bucket was created:
make_bucket: s3://tf-lab-import-test-your-unique-suffix

Bucket creation failed?

S3 bucket names must be globally unique across all AWS accounts. If you see a "BucketAlreadyExists" error, try a different suffix (add more random characters).

Also confirm your AWS CLI is still authenticated by running aws sts get-caller-identity.

help me fix my aws s3 mb command

Now you need to tell Terraform about this bucket. You will add a resource block that describes the bucket, plus an import block that tells Terraform where to find it.

  • In main.tf, add the following code at the bottom of the file:
# Import the manually-created bucket into Terraform management
resource "aws_s3_bucket" "imported" {
  bucket = "[[IMPORT_BUCKET=\"tf-lab-import-test-your-unique-suffix\"]]"
}

import {
  to = aws_s3_bucket.imported
  id = "[[IMPORT_BUCKET=\"tf-lab-import-test-your-unique-suffix\"]]"
}

What do these blocks do?

The resource block tells Terraform the desired configuration for the bucket. The import block maps that resource address to the real bucket's ID in AWS.

When you run terraform plan, Terraform reads the real bucket's state and compares it against your resource block. If they match, it simply adopts the bucket into state with no changes.

  • Save main.tf.
  • Preview the import by running:
terraform plan
  • Look for the line that says the resource "will be imported" in the plan output.
  • Apply the import by running:
terraform apply -auto-approve
  • Confirm the bucket is now in state by running:
terraform state list

You should see aws_s3_bucket.imported listed alongside your other managed resources.

Don't see the imported bucket in state?

Make sure the bucket name in your import block's id field exactly matches the bucket you created with aws s3 mb. Even a single character difference causes the import to fail.

Also check that the bucket attribute in the resource block matches the same name.

help me debug my Terraform import block

Why import blocks instead of the CLI import command?

The older terraform import CLI command still works, but it bypasses the plan/apply workflow. Import blocks are declarative. You can preview exactly what will happen before applying.

HashiCorp recommends import blocks as the modern approach for bringing existing resources under management.

Refactor with a moved block

In production, you often rename resources as configurations grow. Without a moved block, renaming a resource address tells Terraform to destroy the old one and create a new one. That is dangerous for stateful resources like databases or buckets.

  • In main.tf, find the resource block you just added:
resource "aws_s3_bucket" "imported" {
  bucket = "tf-lab-import-test-your-unique-suffix"
}
  • Change "imported" to "managed_bucket", so the resource block now reads:
resource "aws_s3_bucket" "managed_bucket" {
  bucket = "tf-lab-import-test-your-unique-suffix"
}
  • Remove the import block (it is no longer needed since the resource is already in state).
  • Add a moved block below the resource to tell Terraform this is a rename:
# Tell Terraform this resource was renamed, not deleted and recreated
moved {
  from = aws_s3_bucket.imported
  to   = aws_s3_bucket.managed_bucket
}

What does the moved block do?

The moved block tells Terraform that aws_s3_bucket.imported and aws_s3_bucket.managed_bucket are the same resource with a new address. Terraform updates the state file to reflect the new name without touching the actual infrastructure.

This is far safer than the legacy terraform state mv command because you can preview the operation with terraform plan before committing.

  • Save main.tf.
  • Preview the rename by running:
terraform plan

The plan output should show the resource has moved from its old address to its new address. You should NOT see any destroy or create actions.

  • Apply the rename by running:
terraform apply -auto-approve
  • Confirm the rename succeeded by running:
terraform state list

You should see aws_s3_bucket.managed_bucket in the list. The old name aws_s3_bucket.imported is gone.

Seeing a destroy/create instead of a move?

Make sure your moved block's from address exactly matches the old resource name (aws_s3_bucket.imported). Also confirm you removed the import block from the previous substep.

If the plan shows a destroy, do NOT apply it. Fix the moved block first.

help me fix my Terraform moved block

Remove from state without destroying the resource

Sometimes you need Terraform to stop managing a resource without deleting it. The removed block tells Terraform to forget about the resource in state while leaving the actual infrastructure intact.

  • In main.tf, find the aws_s3_bucket.managed_bucket resource block and the moved block below it.
  • Delete both the resource block and the moved block.
  • Replace them with the following removed block:
# Stop managing this bucket without destroying it in AWS
removed {
  from = aws_s3_bucket.managed_bucket

  lifecycle {
    destroy = false
  }
}

What does destroy = false mean?

Setting destroy = false inside the lifecycle block tells Terraform to remove the resource from state only. The actual bucket continues to exist in AWS.

If you set destroy = true instead, Terraform would both remove the resource from state AND delete it from AWS. This gives you explicit control over the outcome.

  • Save main.tf.
  • Preview the removal by running:
terraform plan

The plan should confirm Terraform will stop managing the bucket without destroying it.

  • Apply the removal by running:
terraform apply -auto-approve
  • Now remove the removed block from main.tf entirely (it has served its purpose).
  • Save main.tf.
  • Verify the bucket still exists in AWS by running:

macOS

aws s3 ls | grep tf-lab-import

Windows

aws s3 ls | findstr tf-lab-import

You should see your bucket name in the output. The bucket is alive in AWS, but Terraform no longer knows about it.

  • Confirm Terraform's state is clean by running:
terraform state list

The aws_s3_bucket.managed_bucket resource should no longer appear. Your state is back to the 7 original resources.

Bucket not showing up in aws s3 ls?

If you do not see the bucket, it is possible the removed block was applied with destroy = true instead of destroy = false. Double check the lifecycle block in your removed block before applying.

If the bucket was accidentally deleted, you can recreate it with aws s3 mb and repeat the substep.

help me verify my Terraform removed block worked correctly

✔️ Awesome, I've got everything!

Great. Double check you have saved main.tf with the removed block cleaned out. Your file should match the final code below.

ⓧ I'd like to double check the full code

terraform {
  required_version = ">= 1.15.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.6"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

resource "random_id" "suffix" {
  byte_length = 4
}

resource "aws_s3_bucket" "lab" {
  bucket = "${var.project_name}-${random_id.suffix.hex}"
}

resource "aws_s3_bucket_versioning" "lab" {
  bucket = aws_s3_bucket.lab.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_security_group" "web_sg" {
  name        = "${var.project_name}-web-sg"
  description = "Web tier security group for troubleshooting lab"

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "${var.project_name}-web-sg"
  }
}

resource "aws_security_group" "app_sg" {
  name        = "${var.project_name}-app-sg"
  description = "App tier security group for troubleshooting lab"

  ingress {
    from_port   = 8080
    to_port     = 8080
    protocol    = "tcp"
    cidr_blocks = ["10.0.0.0/16"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "${var.project_name}-app-sg"
  }
}

# After Step 4 fix: separate security group rules replacing inline rules
resource "aws_security_group_rule" "web_to_app" {
  type                     = "ingress"
  from_port                = 8080
  to_port                  = 8080
  protocol                 = "tcp"
  security_group_id        = aws_security_group.app_sg.id
  source_security_group_id = aws_security_group.web_sg.id
}

resource "aws_security_group_rule" "app_to_web" {
  type                     = "ingress"
  from_port                = 80
  to_port                  = 80
  protocol                 = "tcp"
  security_group_id        = aws_security_group.web_sg.id
  source_security_group_id = aws_security_group.app_sg.id
}

You have just completed the three core state surgery operations. You can now import, rename, and release resources safely using declarative blocks that integrate with the plan/apply workflow. Next up, you will push your troubleshooting skills further by tackling multiple simultaneous failures in the Secret Mission.

Secret mission

Multi-Failure Diagnosis with TF_LOG

In production, Terraform failures rarely come one at a time. What happens when a provider version conflict, a state-vs-reality mismatch, and a broken config reference all hit at once? In this secret mission, you will enable TRACE-level debug logging and use it to untangle multiple simultaneous failures that mask each other.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses only AWS Free Tier resources (S3 buckets and default VPC security groups), so there are no ongoing costs.

Resources you used:

  • S3 bucket managed by Terraform (free tier).
  • 2 security groups with 4 security group rules in the default VPC (no cost).
  • Orphaned S3 bucket tf-lab-import-test-<RANDOM> still in AWS but outside Terraform state (free tier).
  • Local project directory terraform-troubleshooting-lab on your Desktop.

Keep everything running

No action needed. Choose this if you want to revisit these resources or continue experimenting with Terraform troubleshooting patterns.

  • All resources are within AWS Free Tier, so there is no cost to leaving them running.
  • Your local project files remain on your Desktop for future reference.
  • The orphaned bucket continues to exist in AWS at no cost.

Pause - I'll come back to this later

Remove the AWS resources to keep your account clean, but keep your local configuration files so you can redeploy later with terraform apply.

  • Open your terminal and make sure you are inside the terraform-troubleshooting-lab directory.
  • Run the following command to destroy all Terraform-managed resources:
terraform destroy
  • Type yes when prompted to confirm.
  • Delete the orphaned bucket that Terraform no longer manages:
aws s3 rb s3://[[ORPHAN_BUCKET="tf-lab-import-test-your-random-suffix"]]
  • Your local files (main.tf, variables.tf, outputs.tf) remain on disk so you can run terraform init and terraform apply again later.

Finding the orphaned bucket name

If you forgot the name of your orphaned bucket, run the following to find it:

aws s3 ls | findstr tf-lab-import

The bucket name will appear in the output.

Delete - I don't want to use this again

Remove all AWS resources and delete your local project files entirely.

  • Open your terminal and make sure you are inside the terraform-troubleshooting-lab directory.
  • Run the following command to destroy all Terraform-managed resources:
terraform destroy
  • Type yes when prompted to confirm.
  • Delete the orphaned bucket that Terraform no longer manages:
aws s3 rb s3://[[ORPHAN_BUCKET="tf-lab-import-test-your-random-suffix"]]
  • Navigate up one level and delete the entire project directory.

macOS

cd ~/Desktop
rm -rf terraform-troubleshooting-lab

Windows

cd ~\Desktop
Remove-Item -Recurse -Force terraform-troubleshooting-lab
  • Confirm the directory is gone by checking your Desktop.

Finding the orphaned bucket name

If you forgot the name of your orphaned bucket, run the following before deleting the project folder:

aws s3 ls | findstr tf-lab-import

The bucket name will appear in the output.

That's a Wrap!

That's a Wrap!

Nice work! You've just built a battle-tested Terraform troubleshooting toolkit by diagnosing and resolving real-world failures against live AWS infrastructure.

You've learned how to:

  • Detected and reconciled infrastructure drift using terraform plan -refresh-only and terraform apply -refresh-only to sync state with reality.
  • Diagnosed a dependency cycle with terraform graph and fixed it by decoupling inline rules into separate aws_security_group_rule resources.
  • Performed state surgery using import blocks, moved blocks, and removed blocks to manage resources without destroying infrastructure.
  • Secret Mission: Used TF_LOG=TRACE to diagnose and fix multiple simultaneous failures in a broken Terraform configuration.

Ready to quiz yourself?