AWS Hybrid Cloud Network Architecture
Build a Transit Gateway hub-spoke network with VPN, firewall, and PrivateLink.
Introduction
30 Second Summary
Every time you stream a movie, place an order, or check your bank balance, your request crosses a network designed by a solutions architect. At Fortune 500 scale, that network connects hundreds of engineering teams across multiple cloud accounts and on-premises datacenters.
In this project, you will build a production-grade AWS hybrid network reference architecture using Terraform. Your finished architecture includes a Transit Gateway hub-spoke with 4 VPCs, centralized firewall inspection, hybrid VPN connectivity, and automated reachability proofs.
What You'll Build
A production-grade AWS network reference architecture spanning 5 artifact classes: a core Terraform network module, hybrid connectivity layer, centralized security inspection, automated validation suite, and an executive-ready pitch deck outline.
By the end of this project, you'll have:
- A Transit Gateway hub-spoke network with 4 VPCs, centralized Network Firewall inspection, and segmented route tables deployed entirely via Terraform.
- Hybrid Site-to-Site VPN connectivity with BGP, bidirectional DNS via Route 53 Resolver, and private service exposure through PrivateLink producer/consumer patterns.
- Automated compliance proofs using Reachability Analyzer (positive and negative assertions), a Mermaid architecture diagram, and a 6-slide pitch deck outline ready for engineering leadership.
- Secret Mission: a VPN failover drill proving BGP reconvergence under 60 seconds, plus a Gateway Load Balancer east-west inspection path using GENEVE encapsulation.
Are there any prerequisites?
This is an advanced project. You need an AWS account with admin-level IAM access, Terraform v1.15.2 installed, AWS CLI v2 configured with credentials, and Cursor IDE installed.
You should also be comfortable reading Terraform configuration files and navigating the AWS console. Step 1 verifies all tools are operational before you build anything.
Verify Environment and Scaffold Project
Before you deploy a single resource, a Principal Solutions Architect locks down one thing: a repeatable, version-controlled foundation. In Fortune 500 environments, infrastructure drift starts at project scaffolding.
In this step, you will confirm your tools are operational, create the full Terraform project structure, and configure multi-account provider aliases. This gives you an audit-ready starting point and a scaffold that 4 parallel Cursor agents can build on simultaneously.
In this step, get ready to:
- Verify Terraform and AWS CLI installations with captured proof.
- Create the project directory structure with root files and 8 module directories.
- Configure multi-account providers with default resource tagging.
Run verification commands and capture output
Every enterprise deployment starts with a tool verification gate. You need Terraform 1.15.x for the latest provider compatibility, and AWS CLI v2 for full API coverage.
- Check your Terraform version by running this command:
terraform version✔️ I see version 1.15 or higher
Your Terraform installation is ready. Move on to the AWS CLI check below.
ⓧ I see an older version
You need Terraform 1.15.0 or higher for this project. The provider features and syntax this project uses require the latest major release.
- On macOS, upgrade via Homebrew by running:
- On Windows, download the latest binary from the official Terraform install page and replace your existing terraform.exe in your PATH.
brew tap hashicorp/tap
brew install hashicorp/tap/terraform- After upgrading, run terraform version again to confirm you see 1.15.x.
ⓧ Command not found
Terraform is not installed on your machine. This project requires Terraform 1.15.0 or higher.
- On macOS, install via Homebrew by running:
brew tap hashicorp/tap
brew install hashicorp/tap/terraform- On Windows, download the installer from the official Terraform install page and add it to your system PATH.
- After installing, close and reopen your terminal, then run terraform version to confirm you see 1.15.x.
- Check your AWS CLI version by running:
aws --version✔️ I see version 2.x
Your AWS CLI is ready. Move on to the identity check below.
ⓧ I see version 1.x
AWS CLI v1 is missing critical commands this project uses. You need to upgrade to v2.
- Follow the official upgrade instructions at the AWS CLI install page.
- After upgrading, run aws --version again to confirm the output starts with aws-cli/2..
ⓧ Command not found
AWS CLI is not installed. This project requires AWS CLI v2.
- Follow the official install instructions for your operating system at the AWS CLI install page.
- After installing, close and reopen your terminal, then run aws --version to confirm the output starts with aws-cli/2..
- Confirm your AWS credentials and admin-level access by running:
aws sts get-caller-identityWhat should I see?
You should see a JSON object with three fields: UserId, Account, and Arn. The Arn field should show an admin-level role or user.
If you see an error about credentials, run aws configure and enter your Access Key ID, Secret Access Key, and set the default region to us-east-1.
- Save all three verification outputs to a single file by running these commands:
# Save tool versions and identity proof to an audit file
terraform version > env-verification.txt
aws --version >> env-verification.txt
aws sts get-caller-identity >> env-verification.txtWhy save verification output?
In enterprise environments, auditors ask for proof that infrastructure was deployed from a known-good toolchain. This file serves as your audit trail showing exact versions used during deployment.
- Confirm the file was created by running:
cat env-verification.txtYou should see your Terraform version, AWS CLI version, and AWS identity JSON printed together.
Seeing a credentials error?
If aws sts get-caller-identity fails, your AWS CLI is not configured with valid credentials. Run aws configure and enter your IAM access key, secret key, and region us-east-1.
Make sure the IAM user or role has admin-level permissions. Help me troubleshoot my AWS CLI credentials
Create the project directory structure
A Principal SA's Terraform project mirrors the architecture it deploys. Each module directory maps to one domain of the network design.
- Move to your Desktop by running this command:
cd ~/Desktop- Create your project folder and move into it by running these commands:
mkdir sap-network-ref
cd sap-network-ref- Create the root Terraform files by running:
touch main.tf variables.tf outputs.tf providers.tf terraform.tfvars- Create all module directories by running:
# Each module maps to one architectural domain
mkdir -p modules/vpc-blueprint
mkdir -p modules/tgw-hub
mkdir -p modules/network-firewall
mkdir -p modules/vpn-onprem
mkdir -p modules/endpoints
mkdir -p modules/privatelink
mkdir -p modules/resolver
mkdir -p modules/reachability- Create the documentation and diagram directories by running:
mkdir diagrams docsWhat does this structure give you?
Each modules/ subdirectory will contain a self-contained Terraform module. diagrams/ holds your Mermaid architecture output. docs/ holds the pitch deck outline for executive presentation.
This separation lets Cursor's parallel agents work on different modules simultaneously without merge conflicts.
- Confirm the full structure exists by running:
lsYou should see main.tf, variables.tf, outputs.tf, providers.tf, terraform.tfvars, plus the modules/, diagrams/, and docs/ directories listed.
Missing files or directories?
Make sure you are inside the sap-network-ref folder when running ls. Run pwd to confirm your current location shows Desktop/sap-network-ref.
If directories are missing, re-run the mkdir -p commands above. The -p flag creates parent directories without error if they already exist. Help me fix my project structure
Configure providers and initialize Terraform
The provider configuration tells Terraform which cloud APIs to use and how to authenticate. You will set up two provider blocks to simulate cross-account architecture.
- Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
- Type Cursor and press Enter to open it.
- In Cursor, click File in the top menu bar.
- Click Open Folder.
- Select the sap-network-ref folder on your Desktop, then click Open.
- Open providers.tf from the Explorer sidebar and paste the following configuration:
terraform {
required_version = ">= 1.15.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
}
}
provider "aws" {
region = "us-east-1"
default_tags {
tags = {
Project = "sap-network-ref"
}
}
}
provider "aws" {
alias = "account_b"
region = "us-east-1"
default_tags {
tags = {
Project = "sap-network-ref"
}
}
}What does this configuration do?
- required_version = ">= 1.15.0" prevents anyone from running this code with an older Terraform that might lack features you depend on.
- hashicorp/aws ~> 5.0 pins the provider to any 5.x release, giving you patch updates without breaking changes.
- The second provider with alias = "account_b" simulates a cross-account deployment pattern. In production, this would use a different IAM role ARN to assume into a second account.
- default_tags ensures every resource gets a Project = "sap-network-ref" tag automatically. This is how you identify and clean up all lab resources later.
- Open main.tf and paste:
# Root main.tf - module calls to be added in subsequent steps- Open variables.tf and paste:
# Root variables.tf - variables to be added in subsequent steps- Open outputs.tf and paste:
# Root outputs.tf - outputs to be added in subsequent steps- Open terraform.tfvars and paste:
# terraform.tfvars - variable values to be added in subsequent steps- Save all files.
- Initialize the Terraform project by running this command in your terminal (inside the sap-network-ref folder):
terraform initYou should see output confirming the AWS provider was installed, ending with a success message.
Provider installation failed?
If you see a network error, check your internet connection. Terraform downloads the provider plugin from the HashiCorp registry during init.
If you see a version constraint error, confirm your providers.tf has version = "~> 5.0" (not a specific patch version). Help me debug terraform init
💡 SAP-C02 exam tie-in
The exam assumes you can evaluate connectivity across multiple accounts and regions. Provider aliases mirror how real multi-account Terraform configurations work at Fortune 500 scale.
The default_tags pattern is an operational excellence best practice tested under Domain 3 (Continuous Improvement for Existing Solutions, 25% of scored content).
✔️ Awesome, I've got everything!
Great. Double check you have saved all five root files.
ⓧ I'd like to double check the full code
Here is the complete providers.tf for comparison:
terraform {
required_version = ">= 1.15.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
}
}
provider "aws" {
region = "us-east-1"
default_tags {
tags = {
Project = "sap-network-ref"
}
}
}
provider "aws" {
alias = "account_b"
region = "us-east-1"
default_tags {
tags = {
Project = "sap-network-ref"
}
}
}And the other root files (each is a single comment for now):
# Root main.tf - module calls to be added in subsequent steps# Root variables.tf - variables to be added in subsequent steps# Root outputs.tf - outputs to be added in subsequent steps# terraform.tfvars - variable values to be added in subsequent stepsYour scaffold is locked in. Next up, you will deploy the Transit Gateway hub and 4 spoke VPCs, then prove exactly why VPC peering fails at scale.
Deploy Transit Gateway Hub and Spoke VPCs
Your environment is ready and your project scaffolding is in place. Now you need the core network fabric that connects everything together.
A $50B media conglomerate with 200+ engineering teams cannot rely on point-to-point VPC peering connections. With 12 VPCs, that would require 66 peering connections (n*(n-1)/2), and none of them support transitive routing. In this step, you will deploy a Transit Gateway hub with 4 spoke VPCs and then prove exactly why peering fails at scale.
In this step, get ready to:
- Deploy a Transit Gateway with explicit route table management and 4 spoke VPCs with non-overlapping CIDRs.
- Create TGW VPC attachments and segmented route tables (spoke-rt and inspection-rt).
- Prove the VPC peering trap-pattern by demonstrating non-transitive routing failure.
Deploy the Transit Gateway and 4 spoke VPCs
The Transit Gateway acts as a regional network router. Every VPC attaches to it and gains full-mesh connectivity through a single attachment point. You will disable the default route table behavior so you can manage routing explicitly.
- Open the Cursor Composer panel and assign Agent A to build the TGW hub module and Agent B to build the VPC blueprint module simultaneously.
- Create the reusable VPC blueprint module by adding the following code to modules/vpc-blueprint/main.tf:
# Reusable VPC module - creates one VPC with TGW and private subnets
variable "vpc_name" { type = string }
variable "cidr_block" { type = string }
variable "azs" { type = list(string) }
variable "tgw_subnet_cidrs" { type = list(string) }
variable "private_subnet_cidrs" { type = list(string) }
variable "tgw_id" {
type = string
default = ""
}
# VPC with DNS support enabled
resource "aws_vpc" "this" {
cidr_block = var.cidr_block
enable_dns_hostnames = true
enable_dns_support = true
tags = { Name = var.vpc_name }
}
# Dedicated /28 TGW attachment subnets (one per AZ)
resource "aws_subnet" "tgw" {
count = length(var.azs)
vpc_id = aws_vpc.this.id
cidr_block = var.tgw_subnet_cidrs[count.index]
availability_zone = var.azs[count.index]
tags = { Name = "${var.vpc_name}-tgw-${var.azs[count.index]}" }
}
# Private /24 workload subnets (one per AZ)
resource "aws_subnet" "private" {
count = length(var.azs)
vpc_id = aws_vpc.this.id
cidr_block = var.private_subnet_cidrs[count.index]
availability_zone = var.azs[count.index]
tags = { Name = "${var.vpc_name}-private-${var.azs[count.index]}" }
}
# Route table for private subnets
resource "aws_route_table" "private" {
vpc_id = aws_vpc.this.id
tags = { Name = "${var.vpc_name}-private-rt" }
}
# Default route sends all traffic to Transit Gateway
resource "aws_route" "to_tgw" {
count = var.tgw_id != "" ? 1 : 0
route_table_id = aws_route_table.private.id
destination_cidr_block = "0.0.0.0/0"
transit_gateway_id = var.tgw_id
}
# Associate private subnets with the route table
resource "aws_route_table_association" "private" {
count = length(var.azs)
subnet_id = aws_subnet.private[count.index].id
route_table_id = aws_route_table.private.id
}
# Outputs for other modules to consume
output "vpc_id" { value = aws_vpc.this.id }
output "vpc_cidr" { value = aws_vpc.this.cidr_block }
output "tgw_subnet_ids" { value = aws_subnet.tgw[*].id }
output "private_subnet_ids" { value = aws_subnet.private[*].id }
output "private_route_table_id" { value = aws_route_table.private.id }What does this module do?
This reusable module creates a single VPC with two subnet tiers. The /28 TGW subnets are tiny (16 IPs) because they only hold the Transit Gateway ENIs. The /24 private subnets hold your actual workloads.
The conditional aws_route.to_tgw resource only creates the default route if a TGW ID is provided. This prevents errors during initial terraform apply before the TGW exists.
- Now create the TGW hub module by adding the following code to modules/tgw-hub/main.tf:
# Transit Gateway hub module - central router for all spoke VPCs
variable "vpc_attachments" {
type = map(object({
vpc_id = string
subnet_ids = list(string)
vpc_cidr = string
}))
}
# Transit Gateway with explicit route table management
resource "aws_ec2_transit_gateway" "hub" {
amazon_side_asn = 64512
auto_accept_shared_attachments = "enable"
default_route_table_association = "disable"
default_route_table_propagation = "disable"
dns_support = "enable"
vpn_ecmp_support = "enable"
tags = { Name = "network-hub-tgw" }
}
# VPC attachments - one per spoke (using dedicated /28 TGW subnets)
resource "aws_ec2_transit_gateway_vpc_attachment" "spokes" {
for_each = var.vpc_attachments
transit_gateway_id = aws_ec2_transit_gateway.hub.id
vpc_id = each.value.vpc_id
subnet_ids = each.value.subnet_ids
transit_gateway_default_route_table_association = false
transit_gateway_default_route_table_propagation = false
tags = { Name = "${each.key}-attachment" }
}Why disable default route table association?
By setting default_route_table_association = "disable" and default_route_table_propagation = "disable" on the TGW, you take full control of which VPCs can communicate. This is the segmentation model the SAP-C02 exam tests.
In a real Fortune 500 environment, you never want a new VPC attachment to auto-propagate into the default route table. That creates accidental full-mesh connectivity between VPCs that should be isolated.
💡 SAP-C02 exam tie-in
The exam asks: "A company has 12 VPCs that all need to communicate. An architect proposes VPC peering. What is the primary limitation?" Answer: VPC peering is non-transitive. With 12 VPCs, you need 66 peering connections. Transit Gateway provides full-mesh connectivity with a single attachment per VPC.
- Save both module files.
Create TGW route tables and VPC attachments
TGW route tables control which traffic flows where. You need two: one for spoke VPCs (that sends internet-bound traffic to the firewall) and one for the firewall attachment (that knows how to return traffic to each spoke).
- Add the route table configuration to the bottom of modules/tgw-hub/main.tf, below the attachment resources:
# Spoke route table - used by App, Data, and SaaS-Producer VPCs
resource "aws_ec2_transit_gateway_route_table" "spoke_rt" {
transit_gateway_id = aws_ec2_transit_gateway.hub.id
tags = { Name = "spoke-rt" }
}
# Inspection route table - used by the Network Firewall attachment
resource "aws_ec2_transit_gateway_route_table" "inspection_rt" {
transit_gateway_id = aws_ec2_transit_gateway.hub.id
tags = { Name = "inspection-rt" }
}
# Associate spoke VPCs with the spoke route table
resource "aws_ec2_transit_gateway_route_table_association" "spokes" {
for_each = var.vpc_attachments
transit_gateway_attachment_id = aws_ec2_transit_gateway_vpc_attachment.spokes[each.key].id
transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.spoke_rt.id
}
# Propagate all VPC CIDRs into the spoke route table
resource "aws_ec2_transit_gateway_route_table_propagation" "spoke_propagations" {
for_each = var.vpc_attachments
transit_gateway_attachment_id = aws_ec2_transit_gateway_vpc_attachment.spokes[each.key].id
transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.spoke_rt.id
}
# Propagate all VPC CIDRs into the inspection route table (return path)
resource "aws_ec2_transit_gateway_route_table_propagation" "inspection_propagations" {
for_each = var.vpc_attachments
transit_gateway_attachment_id = aws_ec2_transit_gateway_vpc_attachment.spokes[each.key].id
transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.inspection_rt.id
}
# Outputs
output "tgw_id" { value = aws_ec2_transit_gateway.hub.id }
output "spoke_rt_id" { value = aws_ec2_transit_gateway_route_table.spoke_rt.id }
output "inspection_rt_id" { value = aws_ec2_transit_gateway_route_table.inspection_rt.id }What do these route tables do?
The spoke-rt table is associated with all spoke VPC attachments. It propagates every VPC CIDR so spokes can reach each other. In Step 4, you will add a static 0.0.0.0/0 route pointing to the Network Firewall attachment for centralized egress inspection.
The inspection-rt table is associated with the firewall attachment. It propagates all spoke CIDRs so return traffic finds its way back to the correct spoke after inspection.
- Now wire everything together in the root main.tf by adding these module calls:
# Root main.tf - orchestrates VPC blueprints and TGW hub
locals {
azs = ["us-east-1a", "us-east-1b"]
}
# App VPC (10.1.0.0/16)
module "app_vpc" {
source = "./modules/vpc-blueprint"
vpc_name = "app-vpc"
cidr_block = "10.1.0.0/16"
azs = local.azs
tgw_subnet_cidrs = ["10.1.255.0/28", "10.1.255.16/28"]
private_subnet_cidrs = ["10.1.1.0/24", "10.1.2.0/24"]
tgw_id = module.tgw_hub.tgw_id
}
# Data VPC (10.2.0.0/16)
module "data_vpc" {
source = "./modules/vpc-blueprint"
vpc_name = "data-vpc"
cidr_block = "10.2.0.0/16"
azs = local.azs
tgw_subnet_cidrs = ["10.2.255.0/28", "10.2.255.16/28"]
private_subnet_cidrs = ["10.2.1.0/24", "10.2.2.0/24"]
tgw_id = module.tgw_hub.tgw_id
}
# SaaS-Producer VPC (10.3.0.0/16)
module "saas_producer_vpc" {
source = "./modules/vpc-blueprint"
vpc_name = "saas-producer-vpc"
cidr_block = "10.3.0.0/16"
azs = local.azs
tgw_subnet_cidrs = ["10.3.255.0/28", "10.3.255.16/28"]
private_subnet_cidrs = ["10.3.1.0/24", "10.3.2.0/24"]
tgw_id = module.tgw_hub.tgw_id
}
# On-Prem Simulation VPC (192.168.0.0/16)
module "onprem_sim_vpc" {
source = "./modules/vpc-blueprint"
vpc_name = "onprem-sim-vpc"
cidr_block = "192.168.0.0/16"
azs = local.azs
tgw_subnet_cidrs = ["192.168.255.0/28", "192.168.255.16/28"]
private_subnet_cidrs = ["192.168.1.0/24", "192.168.2.0/24"]
tgw_id = module.tgw_hub.tgw_id
}
# Transit Gateway Hub
module "tgw_hub" {
source = "./modules/tgw-hub"
vpc_attachments = {
app = {
vpc_id = module.app_vpc.vpc_id
subnet_ids = module.app_vpc.tgw_subnet_ids
vpc_cidr = module.app_vpc.vpc_cidr
}
data = {
vpc_id = module.data_vpc.vpc_id
subnet_ids = module.data_vpc.tgw_subnet_ids
vpc_cidr = module.data_vpc.vpc_cidr
}
saas-producer = {
vpc_id = module.saas_producer_vpc.vpc_id
subnet_ids = module.saas_producer_vpc.tgw_subnet_ids
vpc_cidr = module.saas_producer_vpc.vpc_cidr
}
onprem-sim = {
vpc_id = module.onprem_sim_vpc.vpc_id
subnet_ids = module.onprem_sim_vpc.tgw_subnet_ids
vpc_cidr = module.onprem_sim_vpc.vpc_cidr
}
}
}What does this root configuration do?
The root main.tf instantiates the VPC blueprint 4 times with non-overlapping CIDRs. Each VPC gets /28 TGW attachment subnets (minimal, just for ENIs) and /24 workload subnets.
The TGW hub module receives all VPC outputs and creates the attachments plus route tables. Terraform resolves the dependency graph automatically. The VPC blueprint's conditional route waits for the TGW to exist before creating the 0.0.0.0/0 route.
- Save main.tf.
- Initialize and apply the Terraform configuration by running:
terraform init
terraform apply -auto-approveThis will take 3-5 minutes. Terraform creates the VPCs, subnets, route tables, the Transit Gateway, all 4 VPC attachments, and both TGW route tables with associations and propagations.
Seeing a cycle error?
If Terraform reports a circular dependency between the VPC modules and the TGW hub module, confirm that modules/vpc-blueprint/main.tf uses the conditional count = var.tgw_id != "" ? 1 : 0 on the aws_route.to_tgw resource. This breaks the cycle by making the route optional during the first plan.
help me debug a Terraform cycle error between my VPC and TGW modules
- After the apply completes, verify the TGW and attachments exist by running:
aws ec2 describe-transit-gateways --filters Name=tag:Project,Values=sap-network-ref --query "TransitGateways[0].{ID:TransitGatewayId,State:State,ASN:Options.AmazonSideAsn}"You should see output showing a Transit Gateway in the available state with ASN 64512.
Prove the VPC peering trap-pattern
Now for the most critical SAP-C02 exam demonstration. You will create a VPC peering connection, prove it works for direct communication, and then prove it FAILS for transitive routing. This is the exact wrong-answer trap the exam uses.
- Create a temporary Terraform file to deploy the peering trap-pattern. Add the following to a new file peering-trap.tf in your project root:
# TRAP PATTERN: VPC peering - proves non-transitive routing failure
# This file is temporary and will be removed after demonstration
# Create peering connection between App VPC and Data VPC
resource "aws_vpc_peering_connection" "app_to_data" {
vpc_id = module.app_vpc.vpc_id
peer_vpc_id = module.data_vpc.vpc_id
auto_accept = true
tags = { Name = "trap-app-to-data-peering" }
}
# Add route in App VPC to reach Data VPC via peering
resource "aws_route" "app_to_data" {
route_table_id = module.app_vpc.private_route_table_id
destination_cidr_block = "10.2.0.0/16"
vpc_peering_connection_id = aws_vpc_peering_connection.app_to_data.id
}
# Add route in Data VPC to reach App VPC via peering
resource "aws_route" "data_to_app" {
route_table_id = module.data_vpc.private_route_table_id
destination_cidr_block = "10.1.0.0/16"
vpc_peering_connection_id = aws_vpc_peering_connection.app_to_data.id
}What does this prove?
This peering connection lets App VPC (10.1.0.0/16) communicate directly with Data VPC (10.2.0.0/16). That part works fine. The trap is: can App VPC reach SaaS-Producer VPC (10.3.0.0/16) by hopping through the Data VPC peering path?
The answer is no. VPC peering does NOT support transitive routing. Traffic entering a VPC via a peering connection cannot exit via another peering connection, a VPN, or a gateway. This is the fundamental reason Transit Gateway exists.
- Apply the peering trap resources by running:
terraform apply -auto-approve- After the apply succeeds, verify the peering connection is active by running:
aws ec2 describe-vpc-peering-connections --filters Name=tag:Name,Values=trap-app-to-data-peering --query "VpcPeeringConnections[0].Status.Code"You should see "active" in the output. The peering works for direct App-to-Data communication. But there is no way to reach SaaS-Producer VPC (10.3.0.0/16) through this peering. Adding a route for 10.3.0.0/16 pointing to the peering connection would be rejected by AWS because peering is non-transitive.
SAP-C02 exam trap
The exam will present VPC peering as a wrong answer for multi-VPC connectivity. The correct answer is Transit Gateway when 3+ VPCs need to communicate or when transitive routing is required. VPC peering is only appropriate for 2-VPC low-latency connections where the TGW hop would add unnecessary latency.
Additional trap: "Can two VPCs with overlapping CIDR blocks be peered?" No. This is another reason TGW planning requires non-overlapping CIDR allocation upfront.
- Now remove the trap demonstration. Delete peering-trap.tf from your project root and run:
rm peering-trap.tf
terraform apply -auto-approveTerraform removes the peering connection and routes. Your architecture now relies exclusively on the Transit Gateway for inter-VPC connectivity.
Peering connection stuck in pending-acceptance?
If the peering status shows pending-acceptance instead of active, confirm that auto_accept = true is set in the aws_vpc_peering_connection resource. Auto-accept only works when both VPCs are in the same account and region.
help me fix a VPC peering connection stuck in pending-acceptance state
✔️ Awesome, I've got everything!
Great. Double check you have saved all files and your terminal shows a successful terraform apply with the peering resources destroyed.
ⓧ I'd like to double check the full code
Here is the complete modules/vpc-blueprint/main.tf:
# Reusable VPC module - creates one VPC with TGW and private subnets
variable "vpc_name" { type = string }
variable "cidr_block" { type = string }
variable "azs" { type = list(string) }
variable "tgw_subnet_cidrs" { type = list(string) }
variable "private_subnet_cidrs" { type = list(string) }
variable "tgw_id" {
type = string
default = ""
}
# VPC with DNS support enabled
resource "aws_vpc" "this" {
cidr_block = var.cidr_block
enable_dns_hostnames = true
enable_dns_support = true
tags = { Name = var.vpc_name }
}
# Dedicated /28 TGW attachment subnets (one per AZ)
resource "aws_subnet" "tgw" {
count = length(var.azs)
vpc_id = aws_vpc.this.id
cidr_block = var.tgw_subnet_cidrs[count.index]
availability_zone = var.azs[count.index]
tags = { Name = "${var.vpc_name}-tgw-${var.azs[count.index]}" }
}
# Private /24 workload subnets (one per AZ)
resource "aws_subnet" "private" {
count = length(var.azs)
vpc_id = aws_vpc.this.id
cidr_block = var.private_subnet_cidrs[count.index]
availability_zone = var.azs[count.index]
tags = { Name = "${var.vpc_name}-private-${var.azs[count.index]}" }
}
# Route table for private subnets
resource "aws_route_table" "private" {
vpc_id = aws_vpc.this.id
tags = { Name = "${var.vpc_name}-private-rt" }
}
# Default route sends all traffic to Transit Gateway
resource "aws_route" "to_tgw" {
count = var.tgw_id != "" ? 1 : 0
route_table_id = aws_route_table.private.id
destination_cidr_block = "0.0.0.0/0"
transit_gateway_id = var.tgw_id
}
# Associate private subnets with the route table
resource "aws_route_table_association" "private" {
count = length(var.azs)
subnet_id = aws_subnet.private[count.index].id
route_table_id = aws_route_table.private.id
}
# Outputs for other modules to consume
output "vpc_id" { value = aws_vpc.this.id }
output "vpc_cidr" { value = aws_vpc.this.cidr_block }
output "tgw_subnet_ids" { value = aws_subnet.tgw[*].id }
output "private_subnet_ids" { value = aws_subnet.private[*].id }
output "private_route_table_id" { value = aws_route_table.private.id }Here is the complete modules/tgw-hub/main.tf:
# Transit Gateway hub module - central router for all spoke VPCs
variable "vpc_attachments" {
type = map(object({
vpc_id = string
subnet_ids = list(string)
vpc_cidr = string
}))
}
# Transit Gateway with explicit route table management
resource "aws_ec2_transit_gateway" "hub" {
amazon_side_asn = 64512
auto_accept_shared_attachments = "enable"
default_route_table_association = "disable"
default_route_table_propagation = "disable"
dns_support = "enable"
vpn_ecmp_support = "enable"
tags = { Name = "network-hub-tgw" }
}
# VPC attachments - one per spoke (using dedicated /28 TGW subnets)
resource "aws_ec2_transit_gateway_vpc_attachment" "spokes" {
for_each = var.vpc_attachments
transit_gateway_id = aws_ec2_transit_gateway.hub.id
vpc_id = each.value.vpc_id
subnet_ids = each.value.subnet_ids
transit_gateway_default_route_table_association = false
transit_gateway_default_route_table_propagation = false
tags = { Name = "${each.key}-attachment" }
}
# Spoke route table - used by App, Data, and SaaS-Producer VPCs
resource "aws_ec2_transit_gateway_route_table" "spoke_rt" {
transit_gateway_id = aws_ec2_transit_gateway.hub.id
tags = { Name = "spoke-rt" }
}
# Inspection route table - used by the Network Firewall attachment
resource "aws_ec2_transit_gateway_route_table" "inspection_rt" {
transit_gateway_id = aws_ec2_transit_gateway.hub.id
tags = { Name = "inspection-rt" }
}
# Associate spoke VPCs with the spoke route table
resource "aws_ec2_transit_gateway_route_table_association" "spokes" {
for_each = var.vpc_attachments
transit_gateway_attachment_id = aws_ec2_transit_gateway_vpc_attachment.spokes[each.key].id
transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.spoke_rt.id
}
# Propagate all VPC CIDRs into the spoke route table
resource "aws_ec2_transit_gateway_route_table_propagation" "spoke_propagations" {
for_each = var.vpc_attachments
transit_gateway_attachment_id = aws_ec2_transit_gateway_vpc_attachment.spokes[each.key].id
transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.spoke_rt.id
}
# Propagate all VPC CIDRs into the inspection route table (return path)
resource "aws_ec2_transit_gateway_route_table_propagation" "inspection_propagations" {
for_each = var.vpc_attachments
transit_gateway_attachment_id = aws_ec2_transit_gateway_vpc_attachment.spokes[each.key].id
transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.inspection_rt.id
}
# Outputs
output "tgw_id" { value = aws_ec2_transit_gateway.hub.id }
output "spoke_rt_id" { value = aws_ec2_transit_gateway_route_table.spoke_rt.id }
output "inspection_rt_id" { value = aws_ec2_transit_gateway_route_table.inspection_rt.id }Here is the complete root main.tf:
# Root main.tf - orchestrates VPC blueprints and TGW hub
locals {
azs = ["us-east-1a", "us-east-1b"]
}
# App VPC (10.1.0.0/16)
module "app_vpc" {
source = "./modules/vpc-blueprint"
vpc_name = "app-vpc"
cidr_block = "10.1.0.0/16"
azs = local.azs
tgw_subnet_cidrs = ["10.1.255.0/28", "10.1.255.16/28"]
private_subnet_cidrs = ["10.1.1.0/24", "10.1.2.0/24"]
tgw_id = module.tgw_hub.tgw_id
}
# Data VPC (10.2.0.0/16)
module "data_vpc" {
source = "./modules/vpc-blueprint"
vpc_name = "data-vpc"
cidr_block = "10.2.0.0/16"
azs = local.azs
tgw_subnet_cidrs = ["10.2.255.0/28", "10.2.255.16/28"]
private_subnet_cidrs = ["10.2.1.0/24", "10.2.2.0/24"]
tgw_id = module.tgw_hub.tgw_id
}
# SaaS-Producer VPC (10.3.0.0/16)
module "saas_producer_vpc" {
source = "./modules/vpc-blueprint"
vpc_name = "saas-producer-vpc"
cidr_block = "10.3.0.0/16"
azs = local.azs
tgw_subnet_cidrs = ["10.3.255.0/28", "10.3.255.16/28"]
private_subnet_cidrs = ["10.3.1.0/24", "10.3.2.0/24"]
tgw_id = module.tgw_hub.tgw_id
}
# On-Prem Simulation VPC (192.168.0.0/16)
module "onprem_sim_vpc" {
source = "./modules/vpc-blueprint"
vpc_name = "onprem-sim-vpc"
cidr_block = "192.168.0.0/16"
azs = local.azs
tgw_subnet_cidrs = ["192.168.255.0/28", "192.168.255.16/28"]
private_subnet_cidrs = ["192.168.1.0/24", "192.168.2.0/24"]
tgw_id = module.tgw_hub.tgw_id
}
# Transit Gateway Hub
module "tgw_hub" {
source = "./modules/tgw-hub"
vpc_attachments = {
app = {
vpc_id = module.app_vpc.vpc_id
subnet_ids = module.app_vpc.tgw_subnet_ids
vpc_cidr = module.app_vpc.vpc_cidr
}
data = {
vpc_id = module.data_vpc.vpc_id
subnet_ids = module.data_vpc.tgw_subnet_ids
vpc_cidr = module.data_vpc.vpc_cidr
}
saas-producer = {
vpc_id = module.saas_producer_vpc.vpc_id
subnet_ids = module.saas_producer_vpc.tgw_subnet_ids
vpc_cidr = module.saas_producer_vpc.vpc_cidr
}
onprem-sim = {
vpc_id = module.onprem_sim_vpc.vpc_id
subnet_ids = module.onprem_sim_vpc.tgw_subnet_ids
vpc_cidr = module.onprem_sim_vpc.vpc_cidr
}
}
}Your Transit Gateway hub-spoke network is live. You have 4 VPCs with explicit route table segmentation, and you have documented proof that VPC peering fails at scale. Next up, you will establish hybrid connectivity by simulating a Direct Connect connection with a Site-to-Site VPN using BGP.
Establish Hybrid Connectivity via Site-to-Site VPN
Your Transit Gateway hub-spoke is deployed and the peering trap-pattern proved that transitive routing demands TGW at scale.
Now you need hybrid connectivity. The $50B media conglomerate has 5 on-premises datacenters that must reach AWS workloads with encrypted tunnels and automatic failover. In this step, you will simulate that datacenter link by deploying a Site-to-Site VPN from the On-Prem-Sim VPC to the Transit Gateway, using strongSwan for IPsec and FRRouting for BGP route advertisement.
In this step, get ready to:
- Deploy an EC2 instance as the customer gateway device with IPsec-ready security group rules.
- Create a Site-to-Site VPN connection attached to the Transit Gateway with BGP enabled.
- Configure strongSwan and FRRouting to establish tunnels and advertise on-premises routes.
Deploy the customer gateway EC2 instance
A customer gateway in AWS terminology represents your on-premises VPN device. You need an EC2 instance with a public Elastic IP that AWS can target with IPsec tunnel negotiations.
- Create the file modules/vpn-onprem/main.tf and add the security group, EIP, AMI lookup, and EC2 instance by pasting this code:
# Look up latest Amazon Linux 2023 AMI
data "aws_ami" "al2023" {
most_recent = true
owners = ["amazon"]
filter {
name = "name"
values = ["al2023-ami-*-x86_64"]
}
}
# Security group allowing IPsec negotiation traffic
resource "aws_security_group" "cgw_sg" {
name_prefix = "cgw-device-"
description = "Allow IKE, NAT-T, and ESP for VPN tunnels"
vpc_id = var.vpc_id
# IKE negotiation (UDP 500)
ingress {
from_port = 500
to_port = 500
protocol = "udp"
cidr_blocks = ["0.0.0.0/0"]
}
# NAT Traversal (UDP 4500)
ingress {
from_port = 4500
to_port = 4500
protocol = "udp"
cidr_blocks = ["0.0.0.0/0"]
}
# ESP - Encapsulating Security Payload (IP protocol 50)
ingress {
from_port = 0
to_port = 0
protocol = "50"
cidr_blocks = ["0.0.0.0/0"]
}
# Allow all outbound
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
Name = "cgw-device-sg"
}
}
# Elastic IP for the customer gateway device
resource "aws_eip" "cgw" {
domain = "vpc"
tags = {
Name = "cgw-device-eip"
}
}
# EC2 instance acting as the on-premises VPN device
resource "aws_instance" "cgw_device" {
ami = data.aws_ami.al2023.id
instance_type = "t3.micro"
subnet_id = var.public_subnet_id
vpc_security_group_ids = [aws_security_group.cgw_sg.id]
source_dest_check = false
user_data = local.user_data_script
tags = {
Name = "onprem-cgw-device"
}
}
# Associate the EIP with the CGW device
resource "aws_eip_association" "cgw" {
instance_id = aws_instance.cgw_device.id
allocation_id = aws_eip.cgw.id
}What does this code do?
The security group opens the three protocols required for IPsec VPN tunnels. UDP 500 handles IKE (Internet Key Exchange) negotiation. UDP 4500 handles NAT Traversal for when the device sits behind a NAT. IP protocol 50 carries the encrypted ESP payload.
The source_dest_check = false setting is critical. Without it, AWS drops traffic whose source or destination does not match the instance's own IP. A VPN gateway routes traffic for other hosts, so this check must be disabled.
Create the VPN connection with BGP
The VPN connection must attach to the Transit Gateway (not a Virtual Private Gateway). This is the modern pattern for hub-spoke architectures. You also enable BGP so routes propagate automatically and failover works without manual intervention.
- Add the customer gateway resource, VPN connection, and Terraform outputs to the same file by pasting this code below the EIP association:
# AWS Customer Gateway pointing to the on-prem device EIP
resource "aws_customer_gateway" "onprem" {
bgp_asn = 65000
ip_address = aws_eip.cgw.public_ip
type = "ipsec.1"
tags = {
Name = "onprem-cgw"
}
}
# Site-to-Site VPN attached to Transit Gateway with BGP
resource "aws_vpn_connection" "onprem" {
customer_gateway_id = aws_customer_gateway.onprem.id
transit_gateway_id = var.transit_gateway_id
type = "ipsec.1"
static_routes_only = false
tags = {
Name = "onprem-vpn"
}
}What does this code do?
The aws_customer_gateway resource tells AWS what IP address and BGP ASN your on-premises device uses. ASN 65000 is in the private ASN range (64,512-65,534).
The aws_vpn_connection uses transit_gateway_id (not vpn_gateway_id) because TGW supports transitive routing across all spoke VPCs. Setting static_routes_only = false enables BGP, which gives you automatic failover between the two tunnels.
💡 Why BGP instead of static routing?
Static routing requires you to manually configure every route and provides no automatic failover. BGP detects dead peers within seconds and reroutes traffic to the surviving tunnel automatically.
VPN ECMP (Equal-Cost Multi-Path) on the Transit Gateway also requires dynamic routing. ECMP lets you aggregate bandwidth across both tunnels simultaneously.
- Enable VPN ECMP on the Transit Gateway by opening modules/tgw-hub/main.tf and finding the aws_ec2_transit_gateway resource.
- Find this line:
default_route_table_propagation = "disable"- Add the ECMP setting directly below it, so the resource now includes:
default_route_table_propagation = "disable"
vpn_ecmp_support = "enable"Configure strongSwan and FRRouting via user-data
The EC2 instance needs IPsec tunnels (strongSwan) and BGP route advertisement (FRRouting) configured at boot. The user-data script references VPN tunnel attributes that Terraform populates after creating the VPN connection.
- Add the user-data local block at the top of modules/vpn-onprem/main.tf (before the data source) by pasting this code:
locals {
user_data_script = <<-EOF
#!/bin/bash
# Install strongSwan for IPsec tunnel management
yum install -y strongswan
# Install FRRouting for BGP route advertisement
yum install -y frr
# Write strongSwan IPsec config with two tunnels (IKEv2, AES256, SHA256, DH14)
cat > /etc/strongswan/ipsec.conf <<'IPSEC'
config setup
charondebug="ike 2, knl 2, cfg 2"
conn tunnel1
auto=start
type=tunnel
authby=secret
keyexchange=ikev2
ike=aes256-sha256-modp2048
esp=aes256-sha256-modp2048
left=%defaultroute
leftid=${aws_eip.cgw.public_ip}
right=${aws_vpn_connection.onprem.tunnel1_address}
rightid=${aws_vpn_connection.onprem.tunnel1_address}
leftsubnet=192.168.0.0/16
rightsubnet=0.0.0.0/0
mark=100
conn tunnel2
auto=start
type=tunnel
authby=secret
keyexchange=ikev2
ike=aes256-sha256-modp2048
esp=aes256-sha256-modp2048
left=%defaultroute
leftid=${aws_eip.cgw.public_ip}
right=${aws_vpn_connection.onprem.tunnel2_address}
rightid=${aws_vpn_connection.onprem.tunnel2_address}
leftsubnet=192.168.0.0/16
rightsubnet=0.0.0.0/0
mark=200
IPSEC
# Write pre-shared keys for both tunnels
cat > /etc/strongswan/ipsec.secrets <<SECRETS
${aws_eip.cgw.public_ip} ${aws_vpn_connection.onprem.tunnel1_address} : PSK "${aws_vpn_connection.onprem.tunnel1_preshared_key}"
${aws_eip.cgw.public_ip} ${aws_vpn_connection.onprem.tunnel2_address} : PSK "${aws_vpn_connection.onprem.tunnel2_preshared_key}"
SECRETS
# Enable BGP daemon in FRRouting
sed -i 's/bgpd=no/bgpd=yes/' /etc/frr/daemons
# Write BGP config advertising 192.168.0.0/16 from ASN 65000
cat > /etc/frr/bgpd.conf <<BGP
router bgp 65000
bgp router-id ${aws_instance.cgw_device.private_ip}
neighbor ${aws_vpn_connection.onprem.tunnel1_vgw_inside_address} remote-as 64512
neighbor ${aws_vpn_connection.onprem.tunnel2_vgw_inside_address} remote-as 64512
address-family ipv4 unicast
network 192.168.0.0/16
neighbor ${aws_vpn_connection.onprem.tunnel1_vgw_inside_address} activate
neighbor ${aws_vpn_connection.onprem.tunnel2_vgw_inside_address} activate
exit-address-family
BGP
# Start IPsec and routing services
systemctl start strongswan
systemctl start frr
EOF
}What does this user-data script do?
The script runs at first boot and configures the instance as a full VPN gateway. strongSwan manages the two IPsec tunnels using IKEv2 with AES-256 encryption, SHA-256 integrity, and DH group 14 (2048-bit MODP) for key exchange.
FRRouting advertises the on-premises CIDR (192.168.0.0/16) via BGP to both tunnel peers. The TGW side uses ASN 64512 (the Amazon-side ASN you configured in Step 2). Your customer gateway uses ASN 65000.
Terraform interpolates the tunnel addresses and pre-shared keys from the aws_vpn_connection resource attributes. Each VPN connection provides two tunnels for redundancy, each with its own /30 CIDR from the 169.254.0.0/16 range.
- Add the vpn-onprem module call to your root main.tf by adding this block after your existing module calls:
# Hybrid VPN connectivity module
module "vpn-onprem" {
source = "./modules/vpn-onprem"
vpc_id = module.onprem-sim-vpc.vpc_id
public_subnet_id = module.onprem-sim-vpc.public_subnet_id
transit_gateway_id = module.tgw-hub.transit_gateway_id
}- Save all files.
- Apply the configuration by running:
terraform apply -auto-approveThis apply takes 3-5 minutes because AWS provisions the VPN tunnels and waits for the EC2 instance to boot.
- Verify the VPN tunnel status by running:
aws ec2 describe-vpn-connections --filters Name=tag:Name,Values=onprem-vpn --query "VpnConnections[0].VgwTelemetry[*].{Tunnel:OutsideIpAddress,Status:Status}" --output tableYou should see at least one tunnel with Status UP in the output table.
Both tunnels showing DOWN?
The EC2 instance user-data script takes 1-2 minutes to complete after the instance launches. Wait 2 minutes and re-run the describe command.
If tunnels remain DOWN after 3 minutes, verify the security group allows UDP 500, UDP 4500, and protocol 50 inbound. Also confirm the instance has source_dest_check = false set.
help me troubleshoot why my VPN tunnels are not coming up
💡 SAP-C02 Exam Tie-in
The exam asks: "A company needs hybrid connectivity provisioned within hours. What should they use?" The answer is Site-to-Site VPN. Direct Connect takes weeks for physical cross-connect installation.
"The company later establishes Direct Connect. How does failover work?" Both DX and VPN advertise the same prefix via BGP. TGW prefers DX (shorter AS_PATH). If DX fails, VPN becomes the active path automatically.
"What is the maximum number of BGP routes a customer gateway can advertise to a TGW VPN?" The answer is 1,000.
✔️ Awesome, I've got everything!
Great. Double check you have saved all three files: modules/vpn-onprem/main.tf, the updated modules/tgw-hub/main.tf, and the root main.tf.
ⓧ I'd like to double check the full code
locals {
user_data_script = <<-EOF
#!/bin/bash
# Install strongSwan for IPsec tunnel management
yum install -y strongswan
# Install FRRouting for BGP route advertisement
yum install -y frr
# Write strongSwan IPsec config with two tunnels (IKEv2, AES256, SHA256, DH14)
cat > /etc/strongswan/ipsec.conf <<'IPSEC'
config setup
charondebug="ike 2, knl 2, cfg 2"
conn tunnel1
auto=start
type=tunnel
authby=secret
keyexchange=ikev2
ike=aes256-sha256-modp2048
esp=aes256-sha256-modp2048
left=%defaultroute
leftid=${aws_eip.cgw.public_ip}
right=${aws_vpn_connection.onprem.tunnel1_address}
rightid=${aws_vpn_connection.onprem.tunnel1_address}
leftsubnet=192.168.0.0/16
rightsubnet=0.0.0.0/0
mark=100
conn tunnel2
auto=start
type=tunnel
authby=secret
keyexchange=ikev2
ike=aes256-sha256-modp2048
esp=aes256-sha256-modp2048
left=%defaultroute
leftid=${aws_eip.cgw.public_ip}
right=${aws_vpn_connection.onprem.tunnel2_address}
rightid=${aws_vpn_connection.onprem.tunnel2_address}
leftsubnet=192.168.0.0/16
rightsubnet=0.0.0.0/0
mark=200
IPSEC
# Write pre-shared keys for both tunnels
cat > /etc/strongswan/ipsec.secrets <<SECRETS
${aws_eip.cgw.public_ip} ${aws_vpn_connection.onprem.tunnel1_address} : PSK "${aws_vpn_connection.onprem.tunnel1_preshared_key}"
${aws_eip.cgw.public_ip} ${aws_vpn_connection.onprem.tunnel2_address} : PSK "${aws_vpn_connection.onprem.tunnel2_preshared_key}"
SECRETS
# Enable BGP daemon in FRRouting
sed -i 's/bgpd=no/bgpd=yes/' /etc/frr/daemons
# Write BGP config advertising 192.168.0.0/16 from ASN 65000
cat > /etc/frr/bgpd.conf <<BGP
router bgp 65000
bgp router-id ${aws_instance.cgw_device.private_ip}
neighbor ${aws_vpn_connection.onprem.tunnel1_vgw_inside_address} remote-as 64512
neighbor ${aws_vpn_connection.onprem.tunnel2_vgw_inside_address} remote-as 64512
address-family ipv4 unicast
network 192.168.0.0/16
neighbor ${aws_vpn_connection.onprem.tunnel1_vgw_inside_address} activate
neighbor ${aws_vpn_connection.onprem.tunnel2_vgw_inside_address} activate
exit-address-family
BGP
# Start IPsec and routing services
systemctl start strongswan
systemctl start frr
EOF
}
# Look up latest Amazon Linux 2023 AMI
data "aws_ami" "al2023" {
most_recent = true
owners = ["amazon"]
filter {
name = "name"
values = ["al2023-ami-*-x86_64"]
}
}
# Security group allowing IPsec negotiation traffic
resource "aws_security_group" "cgw_sg" {
name_prefix = "cgw-device-"
description = "Allow IKE, NAT-T, and ESP for VPN tunnels"
vpc_id = var.vpc_id
# IKE negotiation (UDP 500)
ingress {
from_port = 500
to_port = 500
protocol = "udp"
cidr_blocks = ["0.0.0.0/0"]
}
# NAT Traversal (UDP 4500)
ingress {
from_port = 4500
to_port = 4500
protocol = "udp"
cidr_blocks = ["0.0.0.0/0"]
}
# ESP - Encapsulating Security Payload (IP protocol 50)
ingress {
from_port = 0
to_port = 0
protocol = "50"
cidr_blocks = ["0.0.0.0/0"]
}
# Allow all outbound
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
Name = "cgw-device-sg"
}
}
# Elastic IP for the customer gateway device
resource "aws_eip" "cgw" {
domain = "vpc"
tags = {
Name = "cgw-device-eip"
}
}
# EC2 instance acting as the on-premises VPN device
resource "aws_instance" "cgw_device" {
ami = data.aws_ami.al2023.id
instance_type = "t3.micro"
subnet_id = var.public_subnet_id
vpc_security_group_ids = [aws_security_group.cgw_sg.id]
source_dest_check = false
user_data = local.user_data_script
tags = {
Name = "onprem-cgw-device"
}
}
# Associate the EIP with the CGW device
resource "aws_eip_association" "cgw" {
instance_id = aws_instance.cgw_device.id
allocation_id = aws_eip.cgw.id
}
# AWS Customer Gateway pointing to the on-prem device EIP
resource "aws_customer_gateway" "onprem" {
bgp_asn = 65000
ip_address = aws_eip.cgw.public_ip
type = "ipsec.1"
tags = {
Name = "onprem-cgw"
}
}
# Site-to-Site VPN attached to Transit Gateway with BGP
resource "aws_vpn_connection" "onprem" {
customer_gateway_id = aws_customer_gateway.onprem.id
transit_gateway_id = var.transit_gateway_id
type = "ipsec.1"
static_routes_only = false
tags = {
Name = "onprem-vpn"
}
}Your hybrid VPN link is live. On-premises routes are propagating into the Transit Gateway via BGP. Next up, you will deploy centralized Network Firewall inspection so all egress traffic from spoke VPCs passes through a security control plane before reaching the internet.
Deploy Centralized Network Firewall Inspection
Your hybrid connectivity is live. Traffic flows between the on-premises simulation and AWS spoke VPCs via encrypted VPN tunnels with BGP failover.
The CISO has flagged a compliance gap: egress traffic from spoke VPCs reaches the internet without any inspection. PCI-DSS Requirement 1.3.4 mandates that all outbound traffic must pass through an authorized inspection point. In this step, you will deploy AWS Network Firewall centrally on the Transit Gateway using the modern network function attachment pattern. Every packet leaving your spoke VPCs for the internet will pass through Suricata-based stateful rules before reaching a NAT Gateway.
In this step, get ready to:
- Create a Network Firewall policy with Suricata stateful rules that block traffic to known-bad domains.
- Deploy the firewall with a TGW network function attachment and update the spoke route table.
- Configure egress routing with NAT, enable flow logging, and verify inspection behavior.
Create the Network Firewall rule group and policy
AWS Network Firewall evaluates traffic in two stages. First, stateless rules run (by priority). Then, traffic forwarded to the stateful engine is matched against Suricata-compatible rules. You will create a rule group that blocks TLS connections to a specific domain and allows all other HTTPS traffic.
- Create the Network Firewall module file by running:
touch modules/network-firewall/main.tf- In modules/network-firewall/main.tf, add the stateful rule group with Suricata syntax:
# Stateful rule group using Suricata-compatible rules
# Blocks TLS to badexample.com, allows all other HTTPS
resource "aws_networkfirewall_rule_group" "egress_filtering" {
capacity = 100
name = "egress-domain-filtering"
type = "STATEFUL"
rule_group {
# Define HOME_NET and EXTERNAL_NET variables for Suricata
rule_variables {
ip_sets {
key = "HOME_NET"
ip_set {
definition = var.vpc_cidr_blocks
}
}
ip_sets {
key = "EXTERNAL_NET"
ip_set {
definition = ["0.0.0.0/0"]
}
}
}
# Suricata rules: drop bad domain, pass all other HTTPS
rules_source {
rules_string = <<-EOT
drop tls $HOME_NET any -> $EXTERNAL_NET any (tls.sni; content:"badexample.com"; endswith; msg:"Blocked bad domain"; sid:1; rev:1;)
pass tls $HOME_NET any -> $EXTERNAL_NET 443 (msg:"Allow HTTPS"; sid:2; rev:1;)
EOT
}
# STRICT_ORDER required for priority-based evaluation
stateful_rule_options {
rule_order = "STRICT_ORDER"
}
}
tags = {
Project = "sap-network-ref"
}
}What does this code do?
The rules_string block contains two Suricata rules evaluated in priority order (by sid).
- sid:1 drops any TLS connection whose Server Name Indication (SNI) matches badexample.com. The endswith keyword ensures subdomains also match.
- sid:2 passes all remaining TLS traffic to port 443. Because STRICT_ORDER is set, sid:1 is evaluated first. Anything not explicitly passed is dropped by the policy default action.
- Below the rule group resource, add the firewall policy:
# Firewall policy: forward all stateless traffic to stateful engine
# Drop any stateful traffic that doesn't match a pass rule
resource "aws_networkfirewall_firewall_policy" "central_inspection" {
name = "central-egress-inspection"
firewall_policy {
# Forward ALL traffic to the stateful engine (no stateless filtering)
stateless_default_actions = ["aws:forward_to_sfe"]
stateless_fragment_default_actions = ["aws:forward_to_sfe"]
# Reference the Suricata rule group with priority 1
stateful_rule_group_reference {
resource_arn = aws_networkfirewall_rule_group.egress_filtering.arn
priority = 1
}
# STRICT_ORDER enables priority-based evaluation and default actions
stateful_engine_options {
rule_order = "STRICT_ORDER"
}
# Drop established connections that don't match any pass rule
stateful_default_actions = ["aws:drop_established"]
}
tags = {
Project = "sap-network-ref"
}
}What does this policy do?
The policy has two layers. The stateless layer uses aws:forward_to_sfe to send every packet to the stateful engine without filtering. The stateful layer evaluates packets against your Suricata rules.
Any established connection that does not match a pass rule is dropped by aws:drop_established. This is the "deny by default" posture required for PCI-DSS compliance.
Deploy the firewall with TGW network function attachment
The modern deployment pattern attaches Network Firewall directly to the Transit Gateway. AWS provisions a service-managed buffer VPC behind the scenes. Appliance mode is auto-enabled, which prevents asymmetric routing issues across Availability Zones.
- In modules/network-firewall/main.tf, add the firewall resource below the policy:
# Look up available AZs for firewall endpoint placement
data "aws_availability_zones" "available" {
state = "available"
}
# Network Firewall attached directly to Transit Gateway
# AWS manages a buffer VPC and appliance mode is auto-enabled
resource "aws_networkfirewall_firewall" "central" {
name = "central-egress-firewall"
firewall_policy_arn = aws_networkfirewall_firewall_policy.central_inspection.arn
transit_gateway_id = var.transit_gateway_id
# Place firewall endpoints in two AZs for high availability
availability_zone_mapping {
availability_zone_id = data.aws_availability_zones.available.zone_ids[0]
}
availability_zone_mapping {
availability_zone_id = data.aws_availability_zones.available.zone_ids[1]
}
tags = {
Project = "sap-network-ref"
}
}Why use a TGW network function attachment?
Before this feature existed, architects had to create a dedicated inspection VPC with firewall subnets and manually enable appliance mode on the TGW VPC attachment. The network function attachment eliminates that complexity.
The firewall connects directly to the TGW. AWS manages the underlying buffer VPC and automatically ensures traffic stays in the same Availability Zone (appliance mode). This is the pattern you should recommend on the SAP-C02 exam for any new centralized inspection deployment.
- Now update the TGW spoke route table to send all internet-bound traffic through the firewall. In modules/tgw-hub/main.tf, add this static route to the spoke-rt route table:
# Route all internet-bound egress through the Network Firewall
resource "aws_ec2_transit_gateway_route" "egress_to_firewall" {
destination_cidr_block = "0.0.0.0/0"
transit_gateway_attachment_id = module.network_firewall.tgw_attachment_id
transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.spoke_rt.id
}What does this route do?
Every packet from a spoke VPC destined for 0.0.0.0/0 (any internet address) is now forwarded to the Network Firewall attachment. The firewall inspects the packet against your Suricata rules before forwarding allowed traffic to the egress path.
Configure egress routing, enable logging, and verify
After the firewall allows a packet, it needs a path to the internet. You will deploy a NAT Gateway and Internet Gateway in the egress path, enable flow logging to CloudWatch, and verify that blocking works.
- In modules/network-firewall/main.tf, add the egress infrastructure and logging resources:
# --- Egress path: IGW + NAT for post-inspection internet access ---
resource "aws_vpc" "egress" {
cidr_block = "10.255.0.0/16"
enable_dns_hostnames = true
enable_dns_support = true
tags = {
Name = "egress-vpc"
Project = "sap-network-ref"
}
}
resource "aws_subnet" "egress_public" {
vpc_id = aws_vpc.egress.id
cidr_block = "10.255.1.0/24"
availability_zone = data.aws_availability_zones.available.names[0]
tags = {
Name = "egress-public"
Project = "sap-network-ref"
}
}
# Internet Gateway for outbound connectivity
resource "aws_internet_gateway" "egress" {
vpc_id = aws_vpc.egress.id
tags = {
Name = "egress-igw"
Project = "sap-network-ref"
}
}
# Elastic IP for the NAT Gateway
resource "aws_eip" "nat" {
domain = "vpc"
tags = {
Name = "egress-nat-eip"
Project = "sap-network-ref"
}
}
# NAT Gateway provides internet access for inspected traffic
resource "aws_nat_gateway" "egress" {
allocation_id = aws_eip.nat.id
subnet_id = aws_subnet.egress_public.id
tags = {
Name = "egress-nat"
Project = "sap-network-ref"
}
depends_on = [aws_internet_gateway.egress]
}- Below the NAT Gateway, add the CloudWatch logging configuration:
# --- Flow logging to CloudWatch ---
# Log group for Network Firewall flow logs
resource "aws_cloudwatch_log_group" "firewall_flows" {
name = "/aws/network-firewall/flows"
retention_in_days = 7
tags = {
Project = "sap-network-ref"
}
}
# Send FLOW logs to CloudWatch for visibility
resource "aws_networkfirewall_logging_configuration" "central" {
firewall_arn = aws_networkfirewall_firewall.central.arn
logging_configuration {
log_destination_config {
log_destination = {
logGroup = aws_cloudwatch_log_group.firewall_flows.name
}
log_destination_type = "CloudWatchLogs"
log_type = "FLOW"
}
}
}What does the logging configuration do?
The FLOW log type captures all network traffic decisions (allowed and dropped). These logs appear in the /aws/network-firewall/flows log group in CloudWatch. You could also add an ALERT log type to capture only rule-matched traffic for SOC analysis.
- Save all files and deploy the firewall module by running:
terraform apply -auto-approveFirewall provisioning takes time
Network Firewall takes approximately 5-10 minutes to provision endpoints. Terraform will wait until the firewall reaches the READY state before completing.
- After terraform apply completes, verify the firewall blocks traffic to the bad domain. From the App VPC instance (via Session Manager), run:
curl https://badexample.comThis should time out or return a connection reset. The firewall's sid:1 rule drops TLS connections matching that SNI.
- Verify that allowed HTTPS traffic passes through. Run:
curl https://aws.amazon.comThis should return HTML content. The traffic matched sid:2 (pass all HTTPS on port 443) and was forwarded through the NAT Gateway to the internet.
Curl times out for both URLs?
- Confirm the TGW route table update applied correctly. Run terraform state show on the aws_ec2_transit_gateway_route.egress_to_firewall resource and verify the attachment ID points to the firewall.
- Check that the egress VPC route table has a 0.0.0.0/0 route pointing to the Internet Gateway.
- Verify the NAT Gateway is in the available state in the AWS Console under VPC > NAT Gateways.
Still stuck? Help me troubleshoot why my Network Firewall is blocking all traffic instead of just badexample.com
💡 SAP-C02 exam pattern: centralized vs. distributed firewall
The exam asks: "A company needs centralized egress inspection for PCI-DSS compliance across 50 VPCs. What architecture should they use?" The answer is AWS Network Firewall deployed centrally via Transit Gateway. Deploying per-VPC would cost 50x more.
The traditional pattern (inspection VPC with appliance_mode_support = enable on the TGW VPC attachment) is still valid and still tested. The key difference is that the network function attachment handles appliance mode automatically. Know both patterns for exam day.
Trap: Network Firewall does not support inspection of VPC peering traffic. This is another reason Transit Gateway is required for centralized inspection.
✔️ Awesome, I've got everything!
Great. Double check you have saved all your files.
ⓧ I'd like to double check the full code
Here is the complete modules/network-firewall/main.tf at the end of this step:
# Stateful rule group using Suricata-compatible rules
# Blocks TLS to badexample.com, allows all other HTTPS
resource "aws_networkfirewall_rule_group" "egress_filtering" {
capacity = 100
name = "egress-domain-filtering"
type = "STATEFUL"
rule_group {
# Define HOME_NET and EXTERNAL_NET variables for Suricata
rule_variables {
ip_sets {
key = "HOME_NET"
ip_set {
definition = var.vpc_cidr_blocks
}
}
ip_sets {
key = "EXTERNAL_NET"
ip_set {
definition = ["0.0.0.0/0"]
}
}
}
# Suricata rules: drop bad domain, pass all other HTTPS
rules_source {
rules_string = <<-EOT
drop tls $HOME_NET any -> $EXTERNAL_NET any (tls.sni; content:"badexample.com"; endswith; msg:"Blocked bad domain"; sid:1; rev:1;)
pass tls $HOME_NET any -> $EXTERNAL_NET 443 (msg:"Allow HTTPS"; sid:2; rev:1;)
EOT
}
# STRICT_ORDER required for priority-based evaluation
stateful_rule_options {
rule_order = "STRICT_ORDER"
}
}
tags = {
Project = "sap-network-ref"
}
}
# Firewall policy: forward all stateless traffic to stateful engine
# Drop any stateful traffic that doesn't match a pass rule
resource "aws_networkfirewall_firewall_policy" "central_inspection" {
name = "central-egress-inspection"
firewall_policy {
# Forward ALL traffic to the stateful engine (no stateless filtering)
stateless_default_actions = ["aws:forward_to_sfe"]
stateless_fragment_default_actions = ["aws:forward_to_sfe"]
# Reference the Suricata rule group with priority 1
stateful_rule_group_reference {
resource_arn = aws_networkfirewall_rule_group.egress_filtering.arn
priority = 1
}
# STRICT_ORDER enables priority-based evaluation and default actions
stateful_engine_options {
rule_order = "STRICT_ORDER"
}
# Drop established connections that don't match any pass rule
stateful_default_actions = ["aws:drop_established"]
}
tags = {
Project = "sap-network-ref"
}
}
# Look up available AZs for firewall endpoint placement
data "aws_availability_zones" "available" {
state = "available"
}
# Network Firewall attached directly to Transit Gateway
# AWS manages a buffer VPC and appliance mode is auto-enabled
resource "aws_networkfirewall_firewall" "central" {
name = "central-egress-firewall"
firewall_policy_arn = aws_networkfirewall_firewall_policy.central_inspection.arn
transit_gateway_id = var.transit_gateway_id
# Place firewall endpoints in two AZs for high availability
availability_zone_mapping {
availability_zone_id = data.aws_availability_zones.available.zone_ids[0]
}
availability_zone_mapping {
availability_zone_id = data.aws_availability_zones.available.zone_ids[1]
}
tags = {
Project = "sap-network-ref"
}
}
# --- Egress path: IGW + NAT for post-inspection internet access ---
resource "aws_vpc" "egress" {
cidr_block = "10.255.0.0/16"
enable_dns_hostnames = true
enable_dns_support = true
tags = {
Name = "egress-vpc"
Project = "sap-network-ref"
}
}
resource "aws_subnet" "egress_public" {
vpc_id = aws_vpc.egress.id
cidr_block = "10.255.1.0/24"
availability_zone = data.aws_availability_zones.available.names[0]
tags = {
Name = "egress-public"
Project = "sap-network-ref"
}
}
# Internet Gateway for outbound connectivity
resource "aws_internet_gateway" "egress" {
vpc_id = aws_vpc.egress.id
tags = {
Name = "egress-igw"
Project = "sap-network-ref"
}
}
# Elastic IP for the NAT Gateway
resource "aws_eip" "nat" {
domain = "vpc"
tags = {
Name = "egress-nat-eip"
Project = "sap-network-ref"
}
}
# NAT Gateway provides internet access for inspected traffic
resource "aws_nat_gateway" "egress" {
allocation_id = aws_eip.nat.id
subnet_id = aws_subnet.egress_public.id
tags = {
Name = "egress-nat"
Project = "sap-network-ref"
}
depends_on = [aws_internet_gateway.egress]
}
# --- Flow logging to CloudWatch ---
# Log group for Network Firewall flow logs
resource "aws_cloudwatch_log_group" "firewall_flows" {
name = "/aws/network-firewall/flows"
retention_in_days = 7
tags = {
Project = "sap-network-ref"
}
}
# Send FLOW logs to CloudWatch for visibility
resource "aws_networkfirewall_logging_configuration" "central" {
firewall_arn = aws_networkfirewall_firewall.central.arn
logging_configuration {
log_destination_config {
log_destination = {
logGroup = aws_cloudwatch_log_group.firewall_flows.name
}
log_destination_type = "CloudWatchLogs"
log_type = "FLOW"
}
}
}And here is the static route added to modules/tgw-hub/main.tf:
# Route all internet-bound egress through the Network Firewall
resource "aws_ec2_transit_gateway_route" "egress_to_firewall" {
destination_cidr_block = "0.0.0.0/0"
transit_gateway_attachment_id = module.network_firewall.tgw_attachment_id
transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.spoke_rt.id
}Your centralized inspection architecture is live. Every spoke VPC's internet-bound traffic now passes through Suricata rules before reaching the NAT Gateway. Next up, you will configure service endpoints and PrivateLink to eliminate NAT costs for S3 and DynamoDB access and expose a private SaaS service.
Configure Service Endpoints and PrivateLink
Your network hub is inspecting all egress traffic through the centralized firewall. The next cost optimization opportunity is massive: every VPC accessing S3 or DynamoDB through NAT Gateway pays per-gigabyte data processing fees.
At Fortune 500 scale with 12+ VPCs, this adds up to $600K/year in unnecessary NAT charges. In this step, you will deploy gateway endpoints that route S3 and DynamoDB traffic for free, then build a full PrivateLink producer/consumer pair that exposes an internal SaaS service without any internet attack surface.
In this step, get ready to:
- Deploy S3 and DynamoDB gateway endpoints in the App VPC and verify NAT-free access.
- Build a PrivateLink producer service backed by an internal Network Load Balancer.
- Consume the PrivateLink service from a separate VPC and prove 100% private connectivity.
Deploy gateway endpoints for S3 and DynamoDB
Gateway endpoints are free, route-table-based shortcuts that keep traffic between your VPC and S3/DynamoDB entirely on the AWS backbone. No NAT Gateway, no internet gateway, no data processing charges.
- Create the file modules/endpoints/main.tf and paste the following Terraform configuration:
# --- Variables for the endpoints module ---
variable "app_vpc_id" {
description = "ID of the App VPC"
type = string
}
variable "app_private_subnet_ids" {
description = "List of App VPC private subnet IDs"
type = list(string)
}
variable "app_private_route_table_ids" {
description = "Route table IDs for App VPC private subnets"
type = list(string)
}
# --- S3 Gateway Endpoint (free, route-table-based) ---
resource "aws_vpc_endpoint" "s3" {
vpc_id = var.app_vpc_id
service_name = "com.amazonaws.us-east-1.s3"
vpc_endpoint_type = "Gateway"
route_table_ids = var.app_private_route_table_ids
tags = { Name = "app-vpc-s3-endpoint" }
}
# --- DynamoDB Gateway Endpoint (free, route-table-based) ---
resource "aws_vpc_endpoint" "dynamodb" {
vpc_id = var.app_vpc_id
service_name = "com.amazonaws.us-east-1.dynamodb"
vpc_endpoint_type = "Gateway"
route_table_ids = var.app_private_route_table_ids
tags = { Name = "app-vpc-dynamodb-endpoint" }
}
# --- Test S3 bucket to prove endpoint works ---
resource "aws_s3_bucket" "test" {
bucket_prefix = "sap-network-ref-test-"
force_destroy = true
}
# --- IAM role for Session Manager access (no SSH needed) ---
resource "aws_iam_role" "ssm" {
name_prefix = "ssm-instance-"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = { Service = "ec2.amazonaws.com" }
}]
})
}
resource "aws_iam_role_policy_attachment" "ssm_core" {
role = aws_iam_role.ssm.name
policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}
resource "aws_iam_role_policy_attachment" "s3_read" {
role = aws_iam_role.ssm.name
policy_arn = "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"
}
resource "aws_iam_instance_profile" "ssm" {
name_prefix = "ssm-instance-"
role = aws_iam_role.ssm.name
}
# --- Security group for App VPC test instance ---
resource "aws_security_group" "app_instance" {
name_prefix = "app-instance-"
vpc_id = var.app_vpc_id
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = { Name = "app-instance-sg" }
}
# --- App VPC test instance (private subnet, no public IP) ---
data "aws_ami" "al2023" {
most_recent = true
owners = ["amazon"]
filter {
name = "name"
values = ["al2023-ami-*-x86_64"]
}
}
resource "aws_instance" "app_test" {
ami = data.aws_ami.al2023.id
instance_type = "t3.micro"
subnet_id = var.app_private_subnet_ids[0]
vpc_security_group_ids = [aws_security_group.app_instance.id]
iam_instance_profile = aws_iam_instance_profile.ssm.name
associate_public_ip_address = false
tags = { Name = "app-vpc-test" }
}What does this code do?
The two aws_vpc_endpoint resources with type Gateway add prefix-list entries to the specified route tables. Any traffic destined for S3 or DynamoDB follows these routes directly to the service, bypassing NAT entirely.
The EC2 instance has no public IP and no NAT route. It can only reach S3 through the gateway endpoint. Session Manager provides shell access without SSH, so no inbound security group rules are needed.
- Add the endpoints module call to your root main.tf by adding this block after the existing module calls:
# --- Service Endpoints (S3/DynamoDB gateway + test instance) ---
module "endpoints" {
source = "./modules/endpoints"
app_vpc_id = module.app_vpc.vpc_id
app_private_subnet_ids = module.app_vpc.private_subnet_ids
app_private_route_table_ids = module.app_vpc.private_route_table_ids
}What does this module call do?
This wires the endpoints module to the App VPC outputs from your vpc-blueprint module. The route table IDs tell Terraform which route tables receive the gateway endpoint prefix-list entries.
- Apply the endpoints module by running:
terraform apply -auto-approve- Wait for the apply to complete. You should see resources created for the S3 endpoint, DynamoDB endpoint, S3 bucket, IAM role, and EC2 instance.
- Verify gateway endpoint access by connecting to the instance via Session Manager and running:
aws ssm start-session --target $(terraform output -raw app_test_instance_id)- Once connected to the instance shell, confirm S3 access without NAT by running:
aws s3 lsYou should see the test bucket (sap-network-ref-test- prefix) listed in the output. This proves traffic flows through the gateway endpoint because the instance has no other internet path.
Session Manager not connecting?
Session Manager requires the SSM agent to register with the Systems Manager service. Make sure the instance's IAM role has the AmazonSSMManagedInstanceCore policy attached. The instance also needs a route to the SSM endpoints, which the VPC endpoint for S3 alone does not cover. If Session Manager fails, add an interface endpoint for com.amazonaws.us-east-1.ssmmessages in the App VPC.
help me troubleshoot why Session Manager cannot connect to my App VPC instance
💡 SAP-C02 exam pattern: Gateway vs Interface endpoints
The exam asks: 'A company accesses S3 from private subnets via a NAT Gateway. Monthly data transfer costs are $45,000. How can they reduce costs?' The answer is an S3 gateway endpoint (free, added to route table).
Both gateway and interface endpoint types exist for S3. Gateway is free and preferred for in-region access. Interface is needed when on-premises clients access S3 via Direct Connect, because gateway endpoints are not routable from outside the VPC (but interface endpoints have private IPs that are routable).
Build the PrivateLink producer service
A PrivateLink producer exposes a service through an internal Network Load Balancer, making it consumable from other VPCs without any internet exposure. The producer has the NLB. The consumer creates the interface endpoint.
- Create the file modules/privatelink/main.tf and paste the producer resources first:
# --- Variables for the privatelink module ---
variable "saas_producer_vpc_id" {
description = "ID of the SaaS-Producer VPC"
type = string
}
variable "saas_producer_vpc_cidr" {
description = "CIDR block of SaaS-Producer VPC"
type = string
default = "10.3.0.0/16"
}
variable "saas_producer_private_subnet_ids" {
description = "Private subnet IDs in SaaS-Producer VPC"
type = list(string)
}
variable "app_vpc_id" {
description = "ID of the App VPC (consumer)"
type = string
}
variable "app_vpc_cidr" {
description = "CIDR block of App VPC"
type = string
default = "10.1.0.0/16"
}
variable "app_private_subnet_ids" {
description = "Private subnet IDs in App VPC"
type = list(string)
}
# --- SaaS Producer instance running httpd ---
data "aws_ami" "al2023" {
most_recent = true
owners = ["amazon"]
filter {
name = "name"
values = ["al2023-ami-*-x86_64"]
}
}
resource "aws_security_group" "producer" {
name_prefix = "saas-producer-"
vpc_id = var.saas_producer_vpc_id
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = [var.saas_producer_vpc_cidr]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = { Name = "saas-producer-sg" }
}
resource "aws_instance" "producer" {
ami = data.aws_ami.al2023.id
instance_type = "t3.micro"
subnet_id = var.saas_producer_private_subnet_ids[0]
vpc_security_group_ids = [aws_security_group.producer.id]
user_data = <<-EOF
#!/bin/bash
yum install -y httpd
echo "Hello from SaaS Producer" > /var/www/html/index.html
systemctl start httpd
systemctl enable httpd
EOF
tags = { Name = "saas-producer" }
}
# --- Internal Network Load Balancer (not internet-facing) ---
resource "aws_lb" "producer" {
name_prefix = "saas-"
internal = true
load_balancer_type = "network"
subnets = var.saas_producer_private_subnet_ids
tags = { Name = "saas-producer-nlb" }
}
resource "aws_lb_target_group" "producer" {
name_prefix = "saas-"
port = 80
protocol = "TCP"
vpc_id = var.saas_producer_vpc_id
target_type = "instance"
health_check {
protocol = "TCP"
port = "80"
}
}
resource "aws_lb_target_group_attachment" "producer" {
target_group_arn = aws_lb_target_group.producer.arn
target_id = aws_instance.producer.id
port = 80
}
resource "aws_lb_listener" "producer" {
load_balancer_arn = aws_lb.producer.arn
port = 80
protocol = "TCP"
default_action {
type = "forward"
target_group_arn = aws_lb_target_group.producer.arn
}
}
# --- VPC Endpoint Service (PrivateLink producer) ---
resource "aws_vpc_endpoint_service" "producer" {
acceptance_required = false
network_load_balancer_arns = [aws_lb.producer.arn]
tags = { Name = "saas-producer-endpoint-service" }
}What does this code do?
The EC2 instance runs Apache serving a simple page on port 80. The internal NLB forwards TCP 80 traffic to this instance. The aws_vpc_endpoint_service wraps the NLB as a PrivateLink producer with acceptance_required = false so consumers can connect immediately without manual approval.
The endpoint service name (format: com.amazonaws.vpce.us-east-1.vpce-svc-xxxx) is what consumers reference to create their interface endpoint.
💡 Why NLB instead of ALB for PrivateLink?
AWS PrivateLink requires a Network Load Balancer or Gateway Load Balancer as the backing resource. Application Load Balancers are not supported. This is because PrivateLink operates at layer 4 (TCP/UDP), passing traffic through without terminating HTTP connections.
Deploy the PrivateLink consumer endpoint
The consumer side creates an interface endpoint in the App VPC that points to the producer's endpoint service name. Traffic between consumer and producer stays entirely on the AWS network.
- Add the consumer resources to the bottom of modules/privatelink/main.tf:
# --- Interface VPC Endpoint in App VPC (PrivateLink consumer) ---
resource "aws_security_group" "endpoint" {
name_prefix = "privatelink-endpoint-"
vpc_id = var.app_vpc_id
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = [var.app_vpc_cidr]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = { Name = "privatelink-endpoint-sg" }
}
resource "aws_vpc_endpoint" "consumer" {
vpc_id = var.app_vpc_id
service_name = aws_vpc_endpoint_service.producer.service_name
vpc_endpoint_type = "Interface"
subnet_ids = var.app_private_subnet_ids
security_group_ids = [aws_security_group.endpoint.id]
private_dns_enabled = false
tags = { Name = "app-vpc-privatelink-consumer" }
}
# --- Outputs ---
output "endpoint_service_name" {
description = "PrivateLink endpoint service name for consumers"
value = aws_vpc_endpoint_service.producer.service_name
}
output "consumer_endpoint_dns" {
description = "DNS name to reach the producer via PrivateLink"
value = aws_vpc_endpoint.consumer.dns_entry[0].dns_name
}What does this code do?
The security group allows TCP 80 inbound only from the App VPC CIDR (10.1.0.0/16). The interface endpoint creates ENIs in the App VPC private subnets that act as local entry points to the producer service. The dns_entry[0].dns_name output gives the DNS name consumers use to reach the service.
- Add the privatelink module call to your root main.tf after the endpoints module:
# --- PrivateLink Producer/Consumer ---
module "privatelink" {
source = "./modules/privatelink"
saas_producer_vpc_id = module.saas_producer_vpc.vpc_id
saas_producer_private_subnet_ids = module.saas_producer_vpc.private_subnet_ids
app_vpc_id = module.app_vpc.vpc_id
app_private_subnet_ids = module.app_vpc.private_subnet_ids
}- Apply the full configuration by running:
terraform apply -auto-approve- Wait for the apply to complete. The NLB provisioning takes 2-3 minutes. Once finished, verify PrivateLink connectivity from the App VPC instance.
- Get the PrivateLink DNS name from Terraform output:
terraform output -raw consumer_endpoint_dns- Connect to the App VPC instance via Session Manager and test the PrivateLink path:
curl http://<endpoint-dns-name>:80You should see Hello from SaaS Producer in the response. This traffic never touches the internet. It flows from the App VPC ENI through the AWS PrivateLink fabric directly to the NLB in the SaaS-Producer VPC.
Seeing connection timeout on curl?
The NLB target group health check must pass before the NLB routes traffic. Wait 2-3 minutes after the apply for the target to register as healthy. Check target health with aws elbv2 describe-target-health.
Also verify the endpoint security group allows inbound TCP 80 from 10.1.0.0/16 (the App VPC CIDR). If the security group references a different CIDR, the connection will silently drop.
help me debug why my PrivateLink curl is timing out
💡 SAP-C02 exam trap: PrivateLink direction
The PRODUCER has the NLB. The CONSUMER has the interface endpoint. The producer cannot initiate connections to the consumer. If the exam asks 'Service A in Account 1 needs to be accessed by Service B in Account 2 privately', the answer is: Account 1 creates an NLB plus endpoint service. Account 2 creates an interface endpoint.
Getting this backwards is one of the most common exam mistakes. Another trap: 'Can a gateway endpoint be accessed from a peered VPC?' No. But it CAN be accessed from a VPC connected via Transit Gateway, because TGW routes are added to the subnet route table where the gateway endpoint prefix list applies.
✔️ Awesome, I've got everything!
Great. Double check you have saved both modules/endpoints/main.tf and modules/privatelink/main.tf, and that your root main.tf includes both module calls.
ⓧ I'd like to double check the full code
Here is the complete modules/endpoints/main.tf:
# --- Variables for the endpoints module ---
variable "app_vpc_id" {
description = "ID of the App VPC"
type = string
}
variable "app_private_subnet_ids" {
description = "List of App VPC private subnet IDs"
type = list(string)
}
variable "app_private_route_table_ids" {
description = "Route table IDs for App VPC private subnets"
type = list(string)
}
# --- S3 Gateway Endpoint (free, route-table-based) ---
resource "aws_vpc_endpoint" "s3" {
vpc_id = var.app_vpc_id
service_name = "com.amazonaws.us-east-1.s3"
vpc_endpoint_type = "Gateway"
route_table_ids = var.app_private_route_table_ids
tags = { Name = "app-vpc-s3-endpoint" }
}
# --- DynamoDB Gateway Endpoint (free, route-table-based) ---
resource "aws_vpc_endpoint" "dynamodb" {
vpc_id = var.app_vpc_id
service_name = "com.amazonaws.us-east-1.dynamodb"
vpc_endpoint_type = "Gateway"
route_table_ids = var.app_private_route_table_ids
tags = { Name = "app-vpc-dynamodb-endpoint" }
}
# --- Test S3 bucket to prove endpoint works ---
resource "aws_s3_bucket" "test" {
bucket_prefix = "sap-network-ref-test-"
force_destroy = true
}
# --- IAM role for Session Manager access (no SSH needed) ---
resource "aws_iam_role" "ssm" {
name_prefix = "ssm-instance-"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = { Service = "ec2.amazonaws.com" }
}]
})
}
resource "aws_iam_role_policy_attachment" "ssm_core" {
role = aws_iam_role.ssm.name
policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}
resource "aws_iam_role_policy_attachment" "s3_read" {
role = aws_iam_role.ssm.name
policy_arn = "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"
}
resource "aws_iam_instance_profile" "ssm" {
name_prefix = "ssm-instance-"
role = aws_iam_role.ssm.name
}
# --- Security group for App VPC test instance ---
resource "aws_security_group" "app_instance" {
name_prefix = "app-instance-"
vpc_id = var.app_vpc_id
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = { Name = "app-instance-sg" }
}
# --- App VPC test instance (private subnet, no public IP) ---
data "aws_ami" "al2023" {
most_recent = true
owners = ["amazon"]
filter {
name = "name"
values = ["al2023-ami-*-x86_64"]
}
}
resource "aws_instance" "app_test" {
ami = data.aws_ami.al2023.id
instance_type = "t3.micro"
subnet_id = var.app_private_subnet_ids[0]
vpc_security_group_ids = [aws_security_group.app_instance.id]
iam_instance_profile = aws_iam_instance_profile.ssm.name
associate_public_ip_address = false
tags = { Name = "app-vpc-test" }
}Here is the complete modules/privatelink/main.tf:
# --- Variables for the privatelink module ---
variable "saas_producer_vpc_id" {
description = "ID of the SaaS-Producer VPC"
type = string
}
variable "saas_producer_vpc_cidr" {
description = "CIDR block of SaaS-Producer VPC"
type = string
default = "10.3.0.0/16"
}
variable "saas_producer_private_subnet_ids" {
description = "Private subnet IDs in SaaS-Producer VPC"
type = list(string)
}
variable "app_vpc_id" {
description = "ID of the App VPC (consumer)"
type = string
}
variable "app_vpc_cidr" {
description = "CIDR block of App VPC"
type = string
default = "10.1.0.0/16"
}
variable "app_private_subnet_ids" {
description = "Private subnet IDs in App VPC"
type = list(string)
}
# --- SaaS Producer instance running httpd ---
data "aws_ami" "al2023" {
most_recent = true
owners = ["amazon"]
filter {
name = "name"
values = ["al2023-ami-*-x86_64"]
}
}
resource "aws_security_group" "producer" {
name_prefix = "saas-producer-"
vpc_id = var.saas_producer_vpc_id
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = [var.saas_producer_vpc_cidr]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = { Name = "saas-producer-sg" }
}
resource "aws_instance" "producer" {
ami = data.aws_ami.al2023.id
instance_type = "t3.micro"
subnet_id = var.saas_producer_private_subnet_ids[0]
vpc_security_group_ids = [aws_security_group.producer.id]
user_data = <<-EOF
#!/bin/bash
yum install -y httpd
echo "Hello from SaaS Producer" > /var/www/html/index.html
systemctl start httpd
systemctl enable httpd
EOF
tags = { Name = "saas-producer" }
}
# --- Internal Network Load Balancer (not internet-facing) ---
resource "aws_lb" "producer" {
name_prefix = "saas-"
internal = true
load_balancer_type = "network"
subnets = var.saas_producer_private_subnet_ids
tags = { Name = "saas-producer-nlb" }
}
resource "aws_lb_target_group" "producer" {
name_prefix = "saas-"
port = 80
protocol = "TCP"
vpc_id = var.saas_producer_vpc_id
target_type = "instance"
health_check {
protocol = "TCP"
port = "80"
}
}
resource "aws_lb_target_group_attachment" "producer" {
target_group_arn = aws_lb_target_group.producer.arn
target_id = aws_instance.producer.id
port = 80
}
resource "aws_lb_listener" "producer" {
load_balancer_arn = aws_lb.producer.arn
port = 80
protocol = "TCP"
default_action {
type = "forward"
target_group_arn = aws_lb_target_group.producer.arn
}
}
# --- VPC Endpoint Service (PrivateLink producer) ---
resource "aws_vpc_endpoint_service" "producer" {
acceptance_required = false
network_load_balancer_arns = [aws_lb.producer.arn]
tags = { Name = "saas-producer-endpoint-service" }
}
# --- Interface VPC Endpoint in App VPC (PrivateLink consumer) ---
resource "aws_security_group" "endpoint" {
name_prefix = "privatelink-endpoint-"
vpc_id = var.app_vpc_id
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = [var.app_vpc_cidr]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = { Name = "privatelink-endpoint-sg" }
}
resource "aws_vpc_endpoint" "consumer" {
vpc_id = var.app_vpc_id
service_name = aws_vpc_endpoint_service.producer.service_name
vpc_endpoint_type = "Interface"
subnet_ids = var.app_private_subnet_ids
security_group_ids = [aws_security_group.endpoint.id]
private_dns_enabled = false
tags = { Name = "app-vpc-privatelink-consumer" }
}
# --- Outputs ---
output "endpoint_service_name" {
description = "PrivateLink endpoint service name for consumers"
value = aws_vpc_endpoint_service.producer.service_name
}
output "consumer_endpoint_dns" {
description = "DNS name to reach the producer via PrivateLink"
value = aws_vpc_endpoint.consumer.dns_entry[0].dns_name
}Your network now has free S3/DynamoDB access through gateway endpoints and a fully private SaaS exposure pattern through PrivateLink. Next up, you will share this Transit Gateway cross-account via AWS RAM and configure bidirectional hybrid DNS with Route 53 Resolver endpoints.
Configure Cross-Account Governance and Hybrid DNS
Your hub-spoke network is live with VPN connectivity, centralized firewall inspection, and PrivateLink. Two operational gaps remain.
First, at a $50B conglomerate with 200+ teams across multiple AWS accounts, the Network Services account must share the Transit Gateway to spoke accounts without granting console access. Second, hybrid DNS resolution must work bidirectionally so on-premises servers can resolve AWS private hostnames, and AWS workloads can resolve on-premises domains like Active Directory. In this step, you will deploy AWS RAM for cross-account sharing and Route 53 Resolver endpoints for hybrid DNS.
In this step, get ready to:
- Share the Transit Gateway cross-account via AWS RAM.
- Deploy Route 53 Resolver inbound and outbound endpoints for bidirectional DNS.
- Validate hybrid DNS resolution end-to-end across the VPN tunnel.
Share the Transit Gateway via AWS RAM
In a multi-account organization, the Network Services account owns shared infrastructure like the Transit Gateway. Spoke accounts attach their VPCs to it without needing admin access to the networking account. AWS RAM (Resource Access Manager) enables this by creating a resource share that grants specific accounts permission to use the TGW.
- Add the following RAM resource share code to modules/tgw-hub/main.tf, below your existing TGW route table resources:
# --- Cross-Account TGW Sharing via AWS RAM ---
# Look up the current AWS account ID
data "aws_caller_identity" "current" {}
# Create a RAM resource share for the Transit Gateway
resource "aws_ram_resource_share" "network_hub" {
name = "network-hub-share"
allow_external_principals = true
tags = {
Name = "network-hub-share"
}
}
# Attach the Transit Gateway to the share
resource "aws_ram_resource_association" "tgw" {
resource_arn = aws_ec2_transit_gateway.main.arn
resource_share_arn = aws_ram_resource_share.network_hub.arn
}
# Add the current account as a principal (simulates cross-account)
resource "aws_ram_principal_association" "account_b" {
principal = data.aws_caller_identity.current.account_id
resource_share_arn = aws_ram_resource_share.network_hub.arn
}What does this code do?
This creates a RAM resource share named network-hub-share containing the Transit Gateway ARN. The aws_ram_principal_association adds your own account ID as a principal to demonstrate the cross-account flow.
In production, you would replace the account ID with an AWS Organization ID or specific spoke account IDs. Spoke accounts would then create their own VPC attachments to this shared TGW without needing any access to the Network Services account.
💡 SAP-C02 exam pattern: cross-account networking
The exam asks: "A company has 50 accounts that need VPC connectivity through a shared Transit Gateway. How should they share it?" The answer is always AWS RAM. RAM lets you share TGW, subnets, and other resources across accounts without duplicating infrastructure.
The Network Services account pattern (one account owns all shared network infrastructure) is the reference architecture expected on SAP-C02 Domain 1.4 questions about multi-account environments.
- Save modules/tgw-hub/main.tf.
RAM share not creating?
If Terraform reports an error about allow_external_principals, confirm your AWS account is not inside an Organization that restricts external sharing. In that case, set allow_external_principals = false and use your Organization ARN as the principal instead.
help me troubleshoot my AWS RAM resource share configuration
Deploy Route 53 Resolver endpoints for hybrid DNS
Hybrid DNS has two directions. On-premises servers need to resolve AWS private hosted zone records (like api.app.internal). AWS workloads need to resolve on-premises domain names (like server.onprem.corp). Each direction requires its own Resolver endpoint.
What is the Resolver direction mnemonic?
INBOUND = traffic coming IN to AWS. On-premises DNS servers forward queries to the inbound endpoint IPs, which resolve against AWS private hosted zones.
OUTBOUND = traffic going OUT of AWS. AWS resources use the outbound endpoint to forward queries for on-premises domains to your corporate DNS servers.
This mnemonic is critical for SAP-C02. Getting the direction backwards is one of the most common exam mistakes.
- Create the file modules/resolver/main.tf and add the security group and inbound endpoint:
# --- Route 53 Resolver Endpoints for Hybrid DNS ---
# Security group allowing DNS traffic from on-premises network
resource "aws_security_group" "resolver" {
name = "resolver-endpoints-sg"
description = "Allow DNS traffic from on-premises"
vpc_id = var.app_vpc_id
ingress {
from_port = 53
to_port = 53
protocol = "udp"
cidr_blocks = ["192.168.0.0/16"]
}
ingress {
from_port = 53
to_port = 53
protocol = "tcp"
cidr_blocks = ["192.168.0.0/16"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
Name = "resolver-endpoints-sg"
}
}
# Inbound endpoint - receives DNS queries FROM on-premises
resource "aws_route53_resolver_endpoint" "inbound" {
name = "hybrid-dns-inbound"
direction = "INBOUND"
security_group_ids = [aws_security_group.resolver.id]
ip_address {
subnet_id = var.app_vpc_private_subnet_az1_id
}
ip_address {
subnet_id = var.app_vpc_private_subnet_az2_id
}
tags = {
Name = "hybrid-dns-inbound"
}
}What does this code do?
The security group allows DNS traffic (UDP and TCP port 53) from the on-premises CIDR 192.168.0.0/16. The inbound endpoint deploys 2 IPs across 2 Availability Zones in the App VPC private subnets for high availability.
On-premises DNS servers will forward queries for AWS domains to these IPs. The Resolver then looks up the answer in Route 53 private hosted zones and returns it.
- Below the inbound endpoint in modules/resolver/main.tf, add the outbound endpoint and forwarding rule:
# Outbound endpoint - forwards DNS queries TO on-premises
resource "aws_route53_resolver_endpoint" "outbound" {
name = "hybrid-dns-outbound"
direction = "OUTBOUND"
security_group_ids = [aws_security_group.resolver.id]
ip_address {
subnet_id = var.app_vpc_private_subnet_az1_id
}
ip_address {
subnet_id = var.app_vpc_private_subnet_az2_id
}
tags = {
Name = "hybrid-dns-outbound"
}
}
# Forwarding rule: send onprem.corp queries to the on-prem DNS
resource "aws_route53_resolver_rule" "onprem_forward" {
domain_name = "onprem.corp"
name = "forward-onprem-corp"
rule_type = "FORWARD"
resolver_endpoint_id = aws_route53_resolver_endpoint.outbound.id
target_ip {
ip = var.strongswan_private_ip
}
tags = {
Name = "forward-onprem-corp"
}
}
# Associate the forwarding rule with the App VPC
resource "aws_route53_resolver_rule_association" "onprem_forward" {
resolver_rule_id = aws_route53_resolver_rule.onprem_forward.id
vpc_id = var.app_vpc_id
}What does this code do?
The outbound endpoint gives AWS workloads a path to forward DNS queries out of the VPC. The forwarding rule says: any query for onprem.corp should be sent to the strongSwan instance private IP, which runs dnsmasq as a lightweight DNS server.
The rule association binds this forwarding rule to the App VPC so all resources in that VPC automatically use it.
- Still in modules/resolver/main.tf, add the private hosted zone and A record at the bottom:
# Private hosted zone for AWS-side DNS
resource "aws_route53_zone" "app_internal" {
name = "app.internal"
vpc {
vpc_id = var.app_vpc_id
}
tags = {
Name = "app.internal"
}
}
# A record for the internal API service
resource "aws_route53_record" "api" {
zone_id = aws_route53_zone.app_internal.zone_id
name = "api.app.internal"
type = "A"
ttl = 300
records = ["10.1.1.100"]
}What does this code do?
The private hosted zone app.internal is only resolvable from within the associated VPC (and from on-premises via the Resolver inbound endpoint). The A record maps api.app.internal to 10.1.1.100, simulating an internal microservice endpoint.
- Update your root main.tf to call the new resolver module, passing in the App VPC ID, subnet IDs, and strongSwan private IP as variables.
- Update the modules/vpn-onprem/main.tf user-data script to install dnsmasq and add a test record for server.onprem.corp. Also configure resolv.conf on the strongSwan instance to point to the Resolver inbound endpoint IPs (these will be available as Terraform outputs after apply).
- Save all modified files.
- Deploy all resources by running:
terraform apply -auto-approveWhat resources does this deploy?
This single apply creates the RAM resource share, both Resolver endpoints (4 ENIs total across 2 AZs), the forwarding rule, the private hosted zone, and updates the on-prem instance with dnsmasq. Expect the apply to take 3-5 minutes due to Resolver endpoint provisioning.
- Confirm the Resolver endpoints are operational by running:
terraform output resolver_inbound_ipsYou should see two IP addresses from the App VPC private subnets. Note these down for the DNS validation in the next substep.
Resolver endpoint stuck in CREATING?
Resolver endpoints can take up to 4 minutes to provision. If Terraform times out, increase the timeout in your resource block or re-run terraform apply. The default timeout is 10 minutes, which should be sufficient.
If the endpoint fails with a subnet error, confirm your App VPC private subnets have available IP addresses. Each endpoint consumes one IP per AZ.
help me debug my Route 53 Resolver endpoint deployment
💡 SAP-C02 exam pattern: hybrid DNS direction
"On-premises applications need to resolve records in an AWS private hosted zone. What should the architect configure?" Answer: Route 53 Resolver inbound endpoint. On-premises DNS servers forward queries for the AWS domain to the inbound endpoint IPs.
"AWS applications need to resolve on-premises Active Directory DNS names. What should the architect configure?" Answer: Route 53 Resolver outbound endpoint plus a forwarding rule for the on-premises domain pointing to on-premises DNS server IPs.
Validate hybrid DNS end-to-end
Both Resolver endpoints are deployed. The private hosted zone has a test record. The strongSwan instance is running dnsmasq. Now you need to prove bidirectional DNS works across the hybrid connection.
- Connect to your App VPC instance via Session Manager and test outbound resolution by running:
dig server.onprem.corpYou should see a valid A record response. The query travels from the App VPC instance to the Resolver outbound endpoint, which forwards it to the strongSwan private IP running dnsmasq.
- Now SSH into the strongSwan on-prem simulation instance and test inbound resolution by running:
dig api.app.internalYou should see 10.1.1.100 in the response. The query travels from the on-prem instance to the Resolver inbound endpoint IPs (configured in its resolv.conf), which resolves against the app.internal private hosted zone.
Why this proves bidirectional hybrid DNS
The first test proves AWS-to-on-prem resolution (outbound). The second proves on-prem-to-AWS resolution (inbound). Together, they demonstrate the complete hybrid DNS architecture that SAP-C02 expects you to design.
In production, the outbound rule would forward Active Directory domains to your corporate DC. The inbound endpoint would let on-prem services discover AWS-hosted microservices by private DNS name without any internet exposure.
🙋♀️ DNS query returning SERVFAIL?
If dig server.onprem.corp returns SERVFAIL, confirm the strongSwan instance security group allows UDP/TCP port 53 inbound from the App VPC CIDR (10.1.0.0/16). Also verify dnsmasq is running on the instance.
If dig api.app.internal fails from the on-prem instance, check that its resolv.conf nameserver entries match the Resolver inbound endpoint IPs from your Terraform output. Also confirm the VPN tunnel is UP so traffic can reach the App VPC.
help me troubleshoot hybrid DNS resolution across my VPN tunnel
✔️ Awesome, I've got everything!
Great. Double check you have saved all your files and that both dig tests returned valid responses.
ⓧ I'd like to double check the full code
Here is the complete modules/resolver/main.tf for comparison:
# --- Route 53 Resolver Endpoints for Hybrid DNS ---
# Security group allowing DNS traffic from on-premises network
resource "aws_security_group" "resolver" {
name = "resolver-endpoints-sg"
description = "Allow DNS traffic from on-premises"
vpc_id = var.app_vpc_id
ingress {
from_port = 53
to_port = 53
protocol = "udp"
cidr_blocks = ["192.168.0.0/16"]
}
ingress {
from_port = 53
to_port = 53
protocol = "tcp"
cidr_blocks = ["192.168.0.0/16"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
Name = "resolver-endpoints-sg"
}
}
# Inbound endpoint - receives DNS queries FROM on-premises
resource "aws_route53_resolver_endpoint" "inbound" {
name = "hybrid-dns-inbound"
direction = "INBOUND"
security_group_ids = [aws_security_group.resolver.id]
ip_address {
subnet_id = var.app_vpc_private_subnet_az1_id
}
ip_address {
subnet_id = var.app_vpc_private_subnet_az2_id
}
tags = {
Name = "hybrid-dns-inbound"
}
}
# Outbound endpoint - forwards DNS queries TO on-premises
resource "aws_route53_resolver_endpoint" "outbound" {
name = "hybrid-dns-outbound"
direction = "OUTBOUND"
security_group_ids = [aws_security_group.resolver.id]
ip_address {
subnet_id = var.app_vpc_private_subnet_az1_id
}
ip_address {
subnet_id = var.app_vpc_private_subnet_az2_id
}
tags = {
Name = "hybrid-dns-outbound"
}
}
# Forwarding rule: send onprem.corp queries to the on-prem DNS
resource "aws_route53_resolver_rule" "onprem_forward" {
domain_name = "onprem.corp"
name = "forward-onprem-corp"
rule_type = "FORWARD"
resolver_endpoint_id = aws_route53_resolver_endpoint.outbound.id
target_ip {
ip = var.strongswan_private_ip
}
tags = {
Name = "forward-onprem-corp"
}
}
# Associate the forwarding rule with the App VPC
resource "aws_route53_resolver_rule_association" "onprem_forward" {
resolver_rule_id = aws_route53_resolver_rule.onprem_forward.id
vpc_id = var.app_vpc_id
}
# Private hosted zone for AWS-side DNS
resource "aws_route53_zone" "app_internal" {
name = "app.internal"
vpc {
vpc_id = var.app_vpc_id
}
tags = {
Name = "app.internal"
}
}
# A record for the internal API service
resource "aws_route53_record" "api" {
zone_id = aws_route53_zone.app_internal.zone_id
name = "api.app.internal"
type = "A"
ttl = 300
records = ["10.1.1.100"]
}And the RAM additions to modules/tgw-hub/main.tf:
# --- Cross-Account TGW Sharing via AWS RAM ---
# Look up the current AWS account ID
data "aws_caller_identity" "current" {}
# Create a RAM resource share for the Transit Gateway
resource "aws_ram_resource_share" "network_hub" {
name = "network-hub-share"
allow_external_principals = true
tags = {
Name = "network-hub-share"
}
}
# Attach the Transit Gateway to the share
resource "aws_ram_resource_association" "tgw" {
resource_arn = aws_ec2_transit_gateway.main.arn
resource_share_arn = aws_ram_resource_share.network_hub.arn
}
# Add the current account as a principal (simulates cross-account)
resource "aws_ram_principal_association" "account_b" {
principal = data.aws_caller_identity.current.account_id
resource_share_arn = aws_ram_resource_share.network_hub.arn
}Cross-account governance and bidirectional hybrid DNS are both proven. Next up, you will validate every connectivity path with Reachability Analyzer and data-plane tests to produce automated compliance proofs.
Validate with Reachability Analyzer and Generate Architecture Diagram
You have built the entire network architecture. Now you need to prove it works.
In a Principal SA role, deploying infrastructure is only half the job. The other half is producing verifiable evidence that every intended path is reachable and every blocked path stays blocked. Reachability Analyzer performs static configuration analysis without sending any packets, giving you a hop-by-hop proof of connectivity that serves as an automated compliance gate.
In this step, get ready to:
- Create Reachability Analyzer positive assertions for 3 critical paths.
- Prove the negative assertion (security group removal causes analysis failure).
- Run data-plane tests and generate the architecture diagram, pitch deck, and exam decision matrix.
Create positive Reachability Analyzer assertions
Reachability Analyzer uses static analysis to check whether a network path between two resources is feasible based on your current configuration. It does not send packets. It evaluates route tables, security groups, NACLs, and TGW route tables hop by hop.
- Create the modules/reachability directory by running this command:
mkdir -p modules/reachability- Create modules/reachability/main.tf with the following Terraform code defining 3 network insight paths and their analyses:
# Reachability Analyzer paths and analyses for network validation
variable "app_vpc_eni_id" {
description = "ENI ID of the App VPC instance"
type = string
}
variable "data_vpc_eni_id" {
description = "ENI ID of the Data VPC instance"
type = string
}
variable "onprem_eni_id" {
description = "ENI ID of the On-Prem simulation instance"
type = string
}
variable "privatelink_endpoint_eni_id" {
description = "ENI ID of the PrivateLink interface endpoint"
type = string
}
# Path 1: App VPC -> Data VPC (TCP 443 via TGW)
resource "aws_ec2_network_insights_path" "app_to_data" {
source = var.app_vpc_eni_id
destination = var.data_vpc_eni_id
protocol = "tcp"
destination_port = 443
tags = {
Name = "app-to-data-tcp443"
}
}
resource "aws_ec2_network_insights_analysis" "app_to_data" {
network_insights_path_id = aws_ec2_network_insights_path.app_to_data.id
tags = {
Name = "app-to-data-analysis"
}
}
# Path 2: On-Prem -> App VPC (TCP 22 via VPN + TGW)
resource "aws_ec2_network_insights_path" "onprem_to_app" {
source = var.onprem_eni_id
destination = var.app_vpc_eni_id
protocol = "tcp"
destination_port = 22
tags = {
Name = "onprem-to-app-tcp22"
}
}
resource "aws_ec2_network_insights_analysis" "onprem_to_app" {
network_insights_path_id = aws_ec2_network_insights_path.onprem_to_app.id
tags = {
Name = "onprem-to-app-analysis"
}
}
# Path 3: App VPC -> PrivateLink endpoint (TCP 80)
resource "aws_ec2_network_insights_path" "app_to_privatelink" {
source = var.app_vpc_eni_id
destination = var.privatelink_endpoint_eni_id
protocol = "tcp"
destination_port = 80
tags = {
Name = "app-to-privatelink-tcp80"
}
}
resource "aws_ec2_network_insights_analysis" "app_to_privatelink" {
network_insights_path_id = aws_ec2_network_insights_path.app_to_privatelink.id
tags = {
Name = "app-to-privatelink-analysis"
}
}
# Outputs for validation
output "app_to_data_path_found" {
value = aws_ec2_network_insights_analysis.app_to_data.path_found
}
output "onprem_to_app_path_found" {
value = aws_ec2_network_insights_analysis.onprem_to_app.path_found
}
output "app_to_privatelink_path_found" {
value = aws_ec2_network_insights_analysis.app_to_privatelink.path_found
}
output "app_to_data_analysis_id" {
value = aws_ec2_network_insights_analysis.app_to_data.id
}
output "onprem_to_app_analysis_id" {
value = aws_ec2_network_insights_analysis.onprem_to_app.id
}
output "app_to_privatelink_analysis_id" {
value = aws_ec2_network_insights_analysis.app_to_privatelink.id
}What does this code do?
Each aws_ec2_network_insights_path defines a source ENI, destination ENI, protocol, and port. This tells Reachability Analyzer which path to evaluate.
Each aws_ec2_network_insights_analysis triggers the actual static analysis on that path. When Terraform applies, it waits for the analysis to complete and exposes the path_found attribute as true or false.
- Add the reachability module to your root main.tf by adding this module block (passing ENI IDs from your other module outputs):
# Reachability Analyzer validation paths
module "reachability" {
source = "./modules/reachability"
app_vpc_eni_id = module.endpoints.app_instance_eni_id
data_vpc_eni_id = module.vpc_data.workload_eni_id
onprem_eni_id = module.vpn_onprem.onprem_instance_eni_id
privatelink_endpoint_eni_id = module.privatelink.endpoint_eni_id
}What does this module call do?
This passes the ENI IDs from your existing modules into the reachability module. Each ENI represents a specific network interface on an EC2 instance or VPC endpoint that Reachability Analyzer will use as a source or destination.
- Apply the reachability module by running:
terraform apply -target=module.reachability -auto-approve- Verify all three paths show path_found = true in the Terraform output.
- For detailed hop-by-hop analysis, run this command with one of the analysis IDs from the output:
aws ec2 describe-network-insights-analyses --network-insights-analysis-ids [[ANALYSIS_ID="your analysis ID from terraform output"]]- Confirm the response shows "NetworkPathFound": true along with ForwardPathComponents listing each hop (security group, route table, TGW attachment, destination ENI).
Why use Reachability Analyzer as a compliance gate?
Static analysis means you can run these checks on a schedule or in CI/CD without generating any network traffic. If someone modifies a security group or route table, the next analysis run catches the drift immediately.
This is the pattern that separates a Principal SA from a Senior SA. You do not just deploy infrastructure. You build automated governance that proves it stays correct over time.
🙋♀️ Seeing path_found = false on a positive assertion?
Check that your security groups on both source and destination allow the specified protocol and port. Reachability Analyzer evaluates security groups bidirectionally.
Verify your TGW route tables have propagations enabled for the relevant VPC attachments. A missing route propagation is the most common cause of unexpected path failures.
Help me troubleshoot this
💡 SAP-C02 Exam Tie-in
"An architect needs to verify that a security group change allows traffic between two VPCs connected via Transit Gateway, without sending any traffic. Which AWS service should they use?" Answer: VPC Reachability Analyzer. It performs static configuration analysis, identifies blocking components hop-by-hop, and does not send packets.
"Reachability Analyzer shows the path is reachable, but the application still cannot connect. What should the architect investigate?" Answer: Data-plane issues that static analysis cannot detect. MTU mismatches, NACLs with ephemeral port restrictions, application-layer timeouts, DNS resolution failures, or TLS certificate mismatches.
Prove the negative assertion as a compliance gate
A positive assertion proves connectivity exists. A negative assertion proves that your security controls actively block unauthorized paths. This is the compliance evidence auditors look for.
- Temporarily remove the inbound TCP 443 rule from the Data VPC instance security group by running:
aws ec2 revoke-security-group-ingress --group-id [[DATA_SG_ID="your Data VPC instance security group ID"]] --protocol tcp --port 443 --cidr 10.1.0.0/16- Re-run the Reachability Analyzer analysis for the App-to-Data path using the AWS CLI:
aws ec2 start-network-insights-analysis --network-insights-path-id [[PATH_ID="your app-to-data network insights path ID"]]- Wait about 30 seconds for the analysis to complete, then check the result:
aws ec2 describe-network-insights-analyses --network-insights-analysis-ids [[NEG_ANALYSIS_ID="the analysis ID returned from the previous command"]]- Confirm the response shows "NetworkPathFound": false with explanation code ENI_SG_RULES_MISMATCH. This proves the security group is actively blocking the path.
- Restore the security group rule by running:
aws ec2 authorize-security-group-ingress --group-id [[DATA_SG_ID="your Data VPC instance security group ID"]] --protocol tcp --port 443 --cidr 10.1.0.0/16- Re-run the analysis one more time to confirm the path returns to "NetworkPathFound": true:
aws ec2 start-network-insights-analysis --network-insights-path-id [[PATH_ID="your app-to-data network insights path ID"]]What did you just prove?
You demonstrated a full compliance cycle: path works, security control removes access, Reachability Analyzer detects the change with a specific explanation code, you restore the control, path works again. This is exactly how automated drift detection works at scale.
The ENI_SG_RULES_MISMATCH code tells you exactly which component blocked the path. In production, you would run these analyses on a CloudWatch Events schedule and alert when any positive path flips to false.
🙋♀️ Not seeing ENI_SG_RULES_MISMATCH?
If the analysis still shows NetworkPathFound: true after revoking the rule, verify you targeted the correct security group. Run aws ec2 describe-security-groups --group-ids to confirm the ingress rule is actually removed.
Help me debug this security group issue
Run data-plane tests and generate architecture artifacts
Reachability Analyzer has known limitations. It does not evaluate Network Firewall Suricata rules, does not check GWLB inspection paths, and only analyzes forward paths through TGW. Data-plane tests fill these gaps by sending real traffic.
- SSH into your App VPC instance via Session Manager and run the following data-plane tests.
- Test the S3 gateway endpoint (confirms traffic flows without NAT):
aws s3 ls s3://[[BUCKET_NAME="your test S3 bucket name"]]- Test the PrivateLink connection (confirms private connectivity to the SaaS producer):
curl http://[[PRIVATELINK_DNS="your PrivateLink endpoint DNS name"]]:80- You should see Hello from SaaS Producer returned from the NLB-backed endpoint service.
- From the on-prem simulation instance, test VPN + TGW routing:
ping -c 4 10.1.1.100- Test the Network Firewall block (from the App VPC instance):
curl --max-time 5 https://badexample.com- This should timeout or show a connection reset. The Network Firewall Suricata rule drops TLS traffic to badexample.com before it reaches the internet.
Why do you need both Reachability Analyzer and data-plane tests?
Reachability Analyzer is static configuration analysis. It checks route tables and security groups but cannot evaluate Network Firewall Suricata rules, domain lists, or rule options. Its output even includes a PATH_THROUGH_GWLB_NOT_CHECKED code when a GWLB is in the path.
Data-plane tests confirm real traffic behavior. The curl https://badexample.com test proves the Suricata rule works at runtime. Static analysis alone would show this path as reachable because the security groups and route tables allow it.
🙋♀️ S3 ls command failing from the private instance?
Verify the S3 gateway endpoint is associated with the correct private subnet route table. Run aws ec2 describe-route-tables and look for the pl- prefix list entry pointing to the gateway endpoint.
If the instance has no route to S3 via the endpoint, traffic attempts to go through the TGW default route (which leads to the firewall then NAT). Ensure the endpoint route table association is correct.
Help me fix my S3 gateway endpoint connectivity
Now generate the architecture documentation artifacts. These are the visual and presentation deliverables that a Principal SA presents to leadership.
- Create diagrams/topology.md with the following Mermaid diagram documenting all validated paths:
# Network Architecture Topology
```mermaid
graph TD
subgraph "AWS Cloud - us-east-1"
TGW[Transit Gateway<br/>ASN 64512]
subgraph "App VPC - 10.1.0.0/16"
APP_INST[App Instance]
APP_S3EP[S3 Gateway Endpoint]
APP_DYNEP[DynamoDB Gateway Endpoint]
APP_PLEP[PrivateLink Endpoint]
R53_IN[Route 53 Resolver<br/>Inbound Endpoint]
R53_OUT[Route 53 Resolver<br/>Outbound Endpoint]
end
subgraph "Data VPC - 10.2.0.0/16"
DATA_INST[Data Instance]
end
subgraph "SaaS-Producer VPC - 10.3.0.0/16"
SAAS_NLB[Network Load Balancer]
SAAS_INST[SaaS Producer Instance]
SAAS_EPSERV[Endpoint Service]
end
subgraph "Network Firewall - TGW Attachment"
NFW[AWS Network Firewall<br/>Suricata Rules]
NAT[NAT Gateway]
IGW[Internet Gateway]
end
end
subgraph "On-Premises Simulation - 192.168.0.0/16"
ONPREM[strongSwan + FRRouting<br/>ASN 65000]
ONPREM_DNS[dnsmasq DNS]
end
%% TGW Attachments
APP_INST -->|TGW Attachment| TGW
DATA_INST -->|TGW Attachment| TGW
SAAS_NLB -->|TGW Attachment| TGW
%% Network Firewall path
TGW -->|Network Function<br/>Attachment| NFW
NFW -->|Allowed Traffic| NAT
NAT --> IGW
%% VPN Tunnels
ONPREM ==>|"VPN Tunnel 1 (IPsec/BGP)"| TGW
ONPREM ==>|"VPN Tunnel 2 (IPsec/BGP)"| TGW
%% Gateway Endpoints (free, route-table based)
APP_S3EP -.->|"S3 Access (FREE)"| S3[(Amazon S3)]
APP_DYNEP -.->|"DynamoDB Access (FREE)"| DDB[(DynamoDB)]
%% PrivateLink (private path, no internet)
APP_PLEP -->|"PrivateLink TCP 80"| SAAS_EPSERV
SAAS_EPSERV --> SAAS_NLB
SAAS_NLB --> SAAS_INST
%% DNS Resolution
R53_IN -->|"Inbound: On-Prem resolves AWS"| APP_INST
R53_OUT -->|"Outbound: AWS resolves On-Prem"| ONPREM_DNS
%% Validated Paths (Green = Pass)
APP_INST -->|"PASS: TCP 443 via TGW"| DATA_INST
ONPREM -->|"PASS: TCP 22 via VPN+TGW"| APP_INST
APP_INST -->|"PASS: TCP 80 PrivateLink"| APP_PLEP
%% Blocked Path (Red = Firewall Block)
APP_INST -.->x|"BLOCKED: badexample.com"| IGW
%% Styling
classDef pass fill:#90EE90,stroke:#006400
classDef blocked fill:#FFB6C1,stroke:#8B0000
classDef free fill:#98FB98,stroke:#006400,stroke-dasharray: 5 5
```
## Validated Paths Summary
| # | Source | Destination | Protocol | Result | Method |
|---|--------|-------------|----------|--------|--------|
| 1 | App VPC ENI | Data VPC ENI | TCP 443 | PASS | Reachability Analyzer |
| 2 | On-Prem ENI | App VPC ENI | TCP 22 | PASS | Reachability Analyzer |
| 3 | App VPC ENI | PrivateLink Endpoint | TCP 80 | PASS | Reachability Analyzer |
| 4 | App VPC | S3 Bucket | HTTPS | PASS | Data-plane (aws s3 ls) |
| 5 | App VPC | SaaS Producer | HTTP 80 | PASS | Data-plane (curl) |
| 6 | On-Prem | App VPC 10.1.1.100 | ICMP | PASS | Data-plane (ping) |
| 7 | App VPC | badexample.com | TLS 443 | BLOCKED | Data-plane (curl timeout) |What does this diagram communicate?
This Mermaid diagram shows every connection type in your architecture with clear labels. Green paths are validated as reachable. Red paths are confirmed blocked. Dotted green lines represent free gateway endpoints. Orange double-lines represent encrypted VPN tunnels.
When presenting to leadership, this single diagram answers: "Show me all the ways traffic flows and prove each one works."
- Create docs/pitch-deck.md with the 6-slide executive presentation outline:
# Network Reference Architecture - Executive Pitch Deck
## Slide 1: Problem Statement
- 200+ engineering teams deploying AWS VPCs independently
- No centralized egress inspection (compliance gap: PCI-DSS 1.3.4, SOC2 CC6.6)
- NAT Gateway spend: $600K/year across 12 high-volume VPCs
- 14 internet-facing internal services that should be private
- No consistent network architecture or connectivity standard
## Slide 2: Solution Architecture
- Transit Gateway hub-spoke with 4 reference VPCs (template for all future deployments)
- Centralized Network Firewall via TGW network function attachment
- Gateway endpoints for S3/DynamoDB (eliminate NAT cost for these services)
- PrivateLink for internal SaaS services (zero internet exposure)
- Site-to-Site VPN with BGP for hybrid connectivity (Direct Connect ready)
- Route 53 Resolver for bidirectional hybrid DNS
- Architecture diagram: see diagrams/topology.md
## Slide 3: Cost Impact (Annual Savings)
- NAT Gateway elimination for S3/DynamoDB: -$600K/year
- Centralized vs. per-VPC Network Firewall: -60% firewall OpEx
- PrivateLink vs. public endpoints: eliminated data exfiltration insurance liability
- Total first-year savings estimate: $700K+
## Slide 4: Compliance Posture Improvement
- PCI-DSS Requirement 1.3.4: All egress inspected centrally via Network Firewall
- SOC2 CC6.6: Boundary protection with Suricata stateful rules
- Zero internet exposure for internal services (PrivateLink replaces public endpoints)
- Encryption in transit: IPsec VPN (AES-256), TLS via NLB, private network paths
- Automated compliance verification: Reachability Analyzer scheduled assertions
## Slide 5: Architecture Decision Matrix
| Decision | Option A | Option B | When to Use A | When to Use B |
|----------|----------|----------|---------------|---------------|
| Multi-VPC Connectivity | Transit Gateway | VPC Peering | 3+ VPCs, transitive routing needed | 2 VPCs only, low-latency (no TGW hop) |
| S3/DynamoDB Access | Gateway Endpoint | Interface Endpoint | In-region access (free, route-table) | On-premises access via DX needed |
| Private Service Exposure | PrivateLink | NAT Gateway | Exposing a service privately | General internet egress only |
| Traffic Inspection | Centralized Firewall | Distributed Firewall | Cost + policy consistency | Latency-sensitive workloads |
| Hybrid Connectivity | Direct Connect | Site-to-Site VPN | Production (consistent latency, 100 Gbps) | Fast setup, backup, or cost-sensitive |
## Slide 6: Next Steps and Timeline
1. Week 1: Deploy reference architecture to Network Services account (us-east-1)
2. Week 2: Onboard first 3 spoke accounts via RAM TGW share
3. Weeks 3-4: Migrate existing VPC peering connections to TGW attachments
4. Weeks 5-8: Establish Direct Connect to replace VPN for production traffic
5. Months 3-6 (Phase 2): Enable Cloud WAN for multi-region expansion
6. Phase 2: Implement GWLB with Palo Alto for layer-7 east-west inspection- Create docs/exam-decision-matrix.md with the SAP-C02 trap-pattern reference card:
# SAP-C02 Exam Decision Matrix and Trap-Pattern Reference
## Trap Patterns (6 Total)
### Trap 1: VPC Peering When TGW Required
- **Wrong answer the exam tempts you with:** "Use VPC peering for all inter-VPC communication"
- **Why it fails:** VPC peering is non-transitive. Traffic cannot hop through an intermediate VPC. With 12 VPCs, you need 66 peering connections (n*(n-1)/2).
- **Correct answer:** Transit Gateway provides full-mesh connectivity with a single attachment per VPC.
- **Proven in:** Step 2.3 (peering deployed, transitive failure documented, peering removed)
### Trap 2: Interface Endpoint for S3 When Gateway Is Free
- **Wrong answer:** "Create an interface endpoint for S3 to keep traffic private"
- **Why it fails:** Gateway endpoints for S3 are free and route-table-based. Interface endpoints cost per hour and per GB.
- **Correct answer:** Use gateway endpoint for in-region S3 access. Interface endpoint ONLY when on-premises needs to reach S3 via Direct Connect (gateway endpoints are not routable from outside the VPC).
- **Proven in:** Step 5.1 (gateway endpoint works from private subnet without NAT)
### Trap 3: PrivateLink Direction Confusion
- **Wrong answer:** "The consumer creates the NLB and the producer creates the endpoint"
- **Why it fails:** Direction is backwards. The PRODUCER has the NLB + endpoint service. The CONSUMER has the interface endpoint.
- **Correct answer:** Producer = NLB + VPC Endpoint Service. Consumer = Interface VPC Endpoint.
- **Proven in:** Steps 5.2-5.3 (SaaS-Producer VPC has NLB, App VPC has interface endpoint)
### Trap 4: Missing Appliance Mode for Firewall
- **Wrong answer:** "Deploy Network Firewall in an inspection VPC without enabling appliance mode"
- **Why it fails:** Without appliance mode, asymmetric routing causes packets to bypass the firewall on the return path.
- **Correct answer:** Enable appliance_mode_support on the TGW VPC attachment for any VPC hosting firewall endpoints. With TGW network function attachments, appliance mode is auto-enabled.
- **Proven in:** Step 4.2 (network function attachment auto-enables appliance mode)
### Trap 5: Reachability Analyzer False Confidence
- **Wrong answer:** "Reachability Analyzer shows path is reachable, therefore the application works"
- **Why it fails:** RA does not evaluate Suricata rules, domain lists, GWLB inspection paths (PATH_THROUGH_GWLB_NOT_CHECKED), or data-plane issues (MTU, ephemeral ports, TLS).
- **Correct answer:** Supplement RA with data-plane tests for Network Firewall rules and application-layer connectivity.
- **Proven in:** Step 7.3 (data-plane tests catch what RA cannot)
### Trap 6: VPN Static Routing When BGP Enables Failover
- **Wrong answer:** "Use static routing for the VPN connection for simplicity"
- **Why it fails:** Static routing requires manual failover. No ECMP support. No automatic reconvergence.
- **Correct answer:** Use BGP (dynamic routing) for automatic failover between tunnels and ECMP load balancing.
- **Proven in:** Step 3.2 (static_routes_only = false enables BGP)
## Decision Matrix (Quick Reference)
| Scenario | Choose This | Not This | Key Reason |
|----------|-------------|----------|------------|
| 3+ VPCs need connectivity | Transit Gateway | VPC Peering | Non-transitive routing limitation |
| S3 access from private subnet | Gateway Endpoint | Interface Endpoint | Free vs. $0.01/hr per AZ |
| On-prem needs S3 via DX | Interface Endpoint | Gateway Endpoint | Gateway not routable from outside VPC |
| Expose internal service privately | PrivateLink (NLB + Endpoint Service) | NAT + public endpoint | Zero internet attack surface |
| Inspect egress for 50 VPCs | Centralized NFW via TGW | Per-VPC NFW | Cost: 1x vs. 50x firewall endpoints |
| Production hybrid connectivity | Direct Connect | VPN | Consistent latency, up to 100 Gbps |
| Fast hybrid setup or DX backup | VPN with BGP | VPN with static routes | Automatic failover, ECMP support |How to use these artifacts
The pitch deck outline is structured for a 15-minute executive presentation. Each slide has a single theme and can be expanded into 2-3 minutes of commentary. The decision matrix serves double duty: Slide 5 content for leadership AND a personal reference card for SAP-C02 exam day.
- Save all three files.
- Verify the artifacts exist by running:
ls diagrams/ docs/- You should see topology.md in diagrams/ and pitch-deck.md plus exam-decision-matrix.md in docs/.
Reachability Analyzer limitations to remember
- IPv4 only. IPv6 resources are ignored.
- Does not evaluate Suricata rules, domain lists, or rule options in Network Firewall.
- Reports PATH_THROUGH_GWLB_NOT_CHECKED when a Gateway Load Balancer is in the path.
- Forward-only TCP analysis through TGW route tables. The reverse path is not checked.
- These limitations are why data-plane tests remain necessary even with passing Reachability Analyzer results.
✔️ Awesome, I've got everything!
Great. Double check you have saved all files including modules/reachability/main.tf, diagrams/topology.md, docs/pitch-deck.md, and docs/exam-decision-matrix.md.
ⓧ I'd like to double check the full code
Here is the complete modules/reachability/main.tf:
# Reachability Analyzer paths and analyses for network validation
variable "app_vpc_eni_id" {
description = "ENI ID of the App VPC instance"
type = string
}
variable "data_vpc_eni_id" {
description = "ENI ID of the Data VPC instance"
type = string
}
variable "onprem_eni_id" {
description = "ENI ID of the On-Prem simulation instance"
type = string
}
variable "privatelink_endpoint_eni_id" {
description = "ENI ID of the PrivateLink interface endpoint"
type = string
}
# Path 1: App VPC -> Data VPC (TCP 443 via TGW)
resource "aws_ec2_network_insights_path" "app_to_data" {
source = var.app_vpc_eni_id
destination = var.data_vpc_eni_id
protocol = "tcp"
destination_port = 443
tags = {
Name = "app-to-data-tcp443"
}
}
resource "aws_ec2_network_insights_analysis" "app_to_data" {
network_insights_path_id = aws_ec2_network_insights_path.app_to_data.id
tags = {
Name = "app-to-data-analysis"
}
}
# Path 2: On-Prem -> App VPC (TCP 22 via VPN + TGW)
resource "aws_ec2_network_insights_path" "onprem_to_app" {
source = var.onprem_eni_id
destination = var.app_vpc_eni_id
protocol = "tcp"
destination_port = 22
tags = {
Name = "onprem-to-app-tcp22"
}
}
resource "aws_ec2_network_insights_analysis" "onprem_to_app" {
network_insights_path_id = aws_ec2_network_insights_path.onprem_to_app.id
tags = {
Name = "onprem-to-app-analysis"
}
}
# Path 3: App VPC -> PrivateLink endpoint (TCP 80)
resource "aws_ec2_network_insights_path" "app_to_privatelink" {
source = var.app_vpc_eni_id
destination = var.privatelink_endpoint_eni_id
protocol = "tcp"
destination_port = 80
tags = {
Name = "app-to-privatelink-tcp80"
}
}
resource "aws_ec2_network_insights_analysis" "app_to_privatelink" {
network_insights_path_id = aws_ec2_network_insights_path.app_to_privatelink.id
tags = {
Name = "app-to-privatelink-analysis"
}
}
# Outputs for validation
output "app_to_data_path_found" {
value = aws_ec2_network_insights_analysis.app_to_data.path_found
}
output "onprem_to_app_path_found" {
value = aws_ec2_network_insights_analysis.onprem_to_app.path_found
}
output "app_to_privatelink_path_found" {
value = aws_ec2_network_insights_analysis.app_to_privatelink.path_found
}
output "app_to_data_analysis_id" {
value = aws_ec2_network_insights_analysis.app_to_data.id
}
output "onprem_to_app_analysis_id" {
value = aws_ec2_network_insights_analysis.onprem_to_app.id
}
output "app_to_privatelink_analysis_id" {
value = aws_ec2_network_insights_analysis.app_to_privatelink.id
}You now have a fully validated network architecture with three classes of proof: static path analysis (Reachability Analyzer), negative compliance assertions, and real traffic data-plane tests. Plus an executive-ready pitch deck and exam reference card. Next up, the Secret Mission pushes your architecture further with VPN failover drills and GWLB east-west inspection.
Secret mission
VPN Failover Drill + GWLB East-West Inspection
What happens when a VPN tunnel goes down unexpectedly? And how do Fortune 500 companies inspect east-west traffic between VPCs using third-party security appliances? In this mission, you will prove BGP reconvergence within 60 seconds and deploy a Gateway Load Balancer with GENEVE encapsulation for transparent inline inspection.
Clean Up Your Resources
Clean Up Your Resources
This project provisions multiple AWS resources that incur hourly charges. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.
Cost warning
Your Network Firewall and Route 53 Resolver endpoints together cost approximately $1.40/hr while running. If you leave everything up overnight (8 hours), that adds roughly $11 to your bill.
Stopping EC2 instances does NOT stop VPN, NLB, Network Firewall, or Resolver endpoint charges. The recommended option is Delete unless you are actively continuing work.
Resources you used:
- Transit Gateway with 4 VPC attachments and 2 route tables.
- 4 VPCs (App, Data, SaaS-Producer, On-Prem-Sim) with subnets, route tables, and security groups.
- Site-to-Site VPN connection (2 tunnels) and Customer Gateway.
- EC2 instances (t3.micro, 3-4 total) including the strongSwan/FRRouting instance and test instances.
- AWS Network Firewall with firewall policy and stateful rule group.
- NAT Gateway and Internet Gateway for egress.
- Network Load Balancer and PrivateLink endpoint service.
- Route 53 Resolver inbound and outbound endpoints, private hosted zone, and forwarding rule.
- S3 gateway endpoint, DynamoDB gateway endpoint, and PrivateLink interface endpoint.
- AWS RAM resource share.
- S3 test bucket.
- Local project files (Terraform state, modules, diagrams, docs).
Keep everything running
No action needed. Choose this if you are actively continuing work on this architecture within the next hour or two.
- Your resources will continue to incur charges at approximately $1.40-$1.60/hr (Network Firewall + Resolver endpoints dominate).
- EC2 instances (t3.micro) are free-tier eligible up to 750 hours/month total across all instances.
- S3 gateway and DynamoDB gateway endpoints are free.
- Reachability Analyzer does not charge for analyses.
Pause - I'll come back to this later
Stop EC2 instances to reduce compute charges, but be aware that most networking resources still charge hourly even when idle.
- Stop all EC2 instances from the AWS Console or by running the command below.
aws ec2 describe-instances --filters "Name=tag:Project,Values=sap-network-ref" --query "Reservations[].Instances[].InstanceId" --output text | xargs -n1 aws ec2 stop-instances --instance-ids- VPN tunnels, NAT Gateway, Network Firewall, NLB, Resolver endpoints, and PrivateLink interface endpoints all continue to charge hourly while they exist.
- Pausing is only useful if you plan to return within a few hours. For overnight or longer, use the Delete option instead.
Delete - I don't want to use this again
Remove all project resources completely. This is the recommended option once you have finished the project.
- Navigate to your project directory in the terminal.
- Run Terraform destroy to delete all AWS resources:
terraform destroy -auto-approve- Wait for Terraform to finish. This may take 5-10 minutes as some resources (Network Firewall, VPN) take time to deprovision.
- Verify all resources are gone by running:
aws ec2 describe-vpcs --filters "Name=tag:Project,Values=sap-network-ref"- Confirm the output shows an empty VPCs list. If any VPCs remain, check the Terraform output for errors and re-run the destroy command.
- Empty the S3 test bucket before destroy if Terraform reports an error about a non-empty bucket:
aws s3 rm s3://your-test-bucket-name --recursive- Delete your local project files (macOS/Linux):
cd ~/Desktop
rm -rf sap-network-ref- Delete your local project files (Windows):
cd ~\Desktop
rmdir /s /q sap-network-refThat's a Wrap!
That's a Wrap!
You just built the AWS hybrid network reference architecture that Fortune 500 companies pay consulting firms seven figures to design. You did it in 90 minutes for under $3.
Here's what you learned:
- Deployed a Transit Gateway hub-spoke with 4 VPCs, proved why VPC peering fails at scale through the non-transitive routing trap, and established hybrid connectivity via Site-to-Site VPN with BGP.
- Centralized egress inspection with AWS Network Firewall using the modern TGW network function attachment, eliminated NAT Gateway costs with gateway endpoints, and exposed a private SaaS service via the PrivateLink producer/consumer pattern.
- Validated all intended paths with Reachability Analyzer positive and negative assertions, configured bidirectional hybrid DNS with Route 53 Resolver endpoints, and produced an executive-ready pitch deck outline with a SAP-C02 decision matrix.
- Secret Mission: Proved VPN failover reconvergence under 60 seconds and deployed Gateway Load Balancer east-west inspection with GENEVE encapsulation.
Ready to quiz yourself?