Multi-Cloud AWS + GCP Network & CI/CD
Connect AWS and GCP with HA VPN, BGP, and a zero-trust CI/CD pipeline.
Introduction
30 Second Summary
Large organizations rarely bet everything on a single cloud provider. They spread workloads across AWS and GCP for resilience, regulatory compliance, and leverage in vendor negotiations. The hard part is connecting those clouds securely so services talk to each other as if they were on the same private network.
In this project, you will build an enterprise-grade multi-cloud infrastructure that connects AWS and GCP through encrypted VPN tunnels with dynamic routing. You will deploy a distributed application that discovers services by hostname across clouds, and automate everything with a zero-trust CI/CD pipeline.
What You'll Build
A fully automated cross-cloud network connecting AWS and GCP via Transit Gateway and HA VPN with BGP dynamic routing, a distributed product catalog application with cross-cloud DNS resolution, and a GitHub Actions CI/CD pipeline with OIDC federation and security scanning.
By the end of this project, you'll have:
- A cross-cloud private network with 4 IPsec tunnels, BGP sessions, and ECMP load balancing between AWS and GCP, deployed entirely through Terraform.
- A distributed application where a Node.js API on AWS and an nginx frontend on GCP discover each other by hostname over the private VPN path using cross-cloud DNS.
- A zero-trust CI/CD pipeline with OIDC federation to both clouds, tfsec security scanning, Infracost cost estimation on every PR, and Architecture Decision Records documenting your reasoning.
- Secret Mission: Add cross-cloud health monitoring with automated tunnel alerting and a latency dashboard.
Are there any prerequisites?
This project assumes you have hands-on experience with Terraform, an active AWS account, a GCP account, and a GitHub account. You should be comfortable with core networking concepts like VPCs, subnets, routing, and BGP.
You will also need the Terraform CLI, AWS CLI, and gcloud CLI installed on your machine. Step 1 walks you through verifying these tools.
Build the AWS VPC and Transit Gateway Foundation
Your repository structure is in place. Now it is time to build the core AWS networking layer that will anchor your cross-cloud connectivity.
Every cross-cloud architecture needs a well-defined network boundary on each side. On the AWS side, that means a VPC with a private subnet (no public IP addresses) and a Transit Gateway that acts as the central routing hub for all cross-cloud traffic.
In this step, get ready to:
- Create the AWS VPC module with a private subnet and route table.
- Create an AWS Transit Gateway with a custom ASN and ECMP support enabled.
- Configure security groups for cross-cloud traffic and attach the VPC to the Transit Gateway.
Create the AWS VPC module with a private subnet and route table
The VPC module defines a private network on AWS with a 10.0.0.0/16 CIDR block. A single private subnet lives inside it with no public IP assignment, ensuring that the only way traffic enters or exits is through the VPN tunnel you will build later.
- Create the VPC module file by adding the following code to modules/aws-vpc/main.tf:
# AWS VPC for cross-cloud connectivity
resource "aws_vpc" "main" {
cidr_block = var.vpc_cidr
enable_dns_support = true
enable_dns_hostnames = true
tags = {
Name = "${var.project_name}-vpc"
}
}
# Private subnet with no public IP assignment
resource "aws_subnet" "private" {
vpc_id = aws_vpc.main.id
cidr_block = var.private_subnet_cidr
availability_zone = var.availability_zone
map_public_ip_on_launch = false
tags = {
Name = "${var.project_name}-private-subnet"
}
}
# Route table for the private subnet
resource "aws_route_table" "private" {
vpc_id = aws_vpc.main.id
tags = {
Name = "${var.project_name}-private-rt"
}
}
# Associate route table with the private subnet
resource "aws_route_table_association" "private" {
subnet_id = aws_subnet.private.id
route_table_id = aws_route_table.private.id
}What does this code do?
This module creates the foundational AWS network. The VPC uses 10.0.0.0/16 as its address space, giving you 65,536 private IP addresses to work with.
The private subnet (10.0.1.0/24) has map_public_ip_on_launch = false, meaning instances here are invisible to the public internet. They can only be reached through the VPN tunnel. DNS support is enabled because you will need it for cross-cloud DNS resolution in a later step.
- Create the variables file by adding the following code to modules/aws-vpc/variables.tf:
variable "vpc_cidr" {
description = "CIDR block for the AWS VPC"
type = string
default = "10.0.0.0/16"
}
variable "private_subnet_cidr" {
description = "CIDR block for the private subnet"
type = string
default = "10.0.1.0/24"
}
variable "availability_zone" {
description = "Availability zone for the subnet"
type = string
default = "us-east-1a"
}
variable "project_name" {
description = "Project name used for resource tagging"
type = string
default = "multicloud"
}
variable "gcp_cidr" {
description = "GCP CIDR range to allow inbound traffic from"
type = string
default = "10.1.0.0/16"
}- Create the outputs file by adding the following code to modules/aws-vpc/outputs.tf:
output "vpc_id" {
description = "ID of the created VPC"
value = aws_vpc.main.id
}
output "private_subnet_id" {
description = "ID of the private subnet"
value = aws_subnet.private.id
}
output "private_route_table_id" {
description = "ID of the private route table"
value = aws_route_table.private.id
}
output "security_group_id" {
description = "ID of the cross-cloud security group"
value = aws_security_group.cross_cloud.id
}- Save all three files in the modules/aws-vpc/ directory.
- Confirm the files exist by running:
ls modules/aws-vpc/You should see main.tf, variables.tf, and outputs.tf listed.
Files not showing up?
Make sure you created the files inside the modules/aws-vpc/ directory, not in the repository root. If you saved them in the wrong location, move them with mv.
help me find where my Terraform module files ended up
Create an AWS Transit Gateway with a custom ASN and ECMP support
The Transit Gateway is a regional routing hub that connects your VPC to VPN tunnels. Unlike a Virtual Private Gateway (which only allows one active VPN tunnel at a time), a Transit Gateway supports ECMP (Equal Cost Multi-Path) routing. This distributes traffic across all active tunnels simultaneously for higher throughput.
- Create the Transit Gateway module by adding the following code to modules/aws-tgw/main.tf:
# Transit Gateway with custom ASN and ECMP support
resource "aws_ec2_transit_gateway" "main" {
amazon_side_asn = var.tgw_asn
default_route_table_association = "enable"
default_route_table_propagation = "enable"
vpn_ecmp_support = "enable"
dns_support = "enable"
tags = {
Name = "${var.project_name}-tgw"
}
}
# Attach the VPC to the Transit Gateway
resource "aws_ec2_transit_gateway_vpc_attachment" "main" {
subnet_ids = [var.subnet_id]
transit_gateway_id = aws_ec2_transit_gateway.main.id
vpc_id = var.vpc_id
tags = {
Name = "${var.project_name}-tgw-vpc-attachment"
}
}What does this code do?
The Transit Gateway is created with ASN 64512 (a private BGP ASN). The vpn_ecmp_support = "enable" setting tells the Transit Gateway to distribute traffic equally across all active VPN tunnels instead of using only one.
The VPC attachment connects your private subnet to the Transit Gateway so that traffic can flow from the VPC into the VPN tunnels (and vice versa). Route table association and propagation are both enabled so that routes learned from VPN peers automatically appear in the Transit Gateway route table.
Why Transit Gateway over Virtual Private Gateway?
A Virtual Private Gateway (VGW) only uses one active tunnel at a time. The second tunnel sits idle as a standby. Transit Gateway uses ECMP to send traffic across all 4 tunnels simultaneously, giving you up to 4x the throughput.
Transit Gateway also supports connecting multiple VPCs to the same VPN, future-proofing the architecture. The trade-off is a small additional cost ($0.05/hr per attachment plus $0.02/GB data processing), but for enterprise architectures the scalability and performance justify it.
- Create the variables file by adding the following code to modules/aws-tgw/variables.tf:
variable "tgw_asn" {
description = "BGP ASN for the Transit Gateway"
type = number
default = 64512
}
variable "vpc_id" {
description = "ID of the VPC to attach to the Transit Gateway"
type = string
}
variable "subnet_id" {
description = "ID of the subnet for the TGW VPC attachment"
type = string
}
variable "project_name" {
description = "Project name used for resource tagging"
type = string
default = "multicloud"
}- Create the outputs file by adding the following code to modules/aws-tgw/outputs.tf:
output "transit_gateway_id" {
description = "ID of the Transit Gateway"
value = aws_ec2_transit_gateway.main.id
}
output "transit_gateway_asn" {
description = "BGP ASN of the Transit Gateway"
value = aws_ec2_transit_gateway.main.amazon_side_asn
}- Save all three files in the modules/aws-tgw/ directory.
- Confirm the files exist by running:
ls modules/aws-tgw/You should see main.tf, variables.tf, and outputs.tf listed.
Directory not found?
Ensure you are working from the root of your repository. Run ls modules/ to confirm the aws-tgw directory exists. If it does not, create it with mkdir -p modules/aws-tgw.
help me fix my Terraform module directory structure
Configure security groups and attach the VPC to the Transit Gateway
The final piece of this step ties everything together. You need a security group that allows traffic from GCP to reach your AWS instances, and a root Terraform configuration that calls both modules and applies them.
- Add the security group resource to the bottom of modules/aws-vpc/main.tf, below the route table association:
# Security group allowing cross-cloud traffic from GCP
resource "aws_security_group" "cross_cloud" {
name = "${var.project_name}-cross-cloud-sg"
description = "Allow inbound traffic from GCP CIDR ranges"
vpc_id = aws_vpc.main.id
ingress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = [var.gcp_cidr]
description = "Allow all traffic from GCP"
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
description = "Allow all outbound traffic"
}
tags = {
Name = "${var.project_name}-cross-cloud-sg"
}
}What does this security group do?
The security group allows all inbound traffic from 10.1.0.0/16 (the GCP hub VPC CIDR). This means any packet arriving through the VPN from GCP can reach instances in this VPC. The protocol = "-1" means all protocols (TCP, UDP, ICMP) are permitted.
Outbound traffic is unrestricted so that responses and health checks can flow back to GCP without being blocked.
- Create the root configuration that wires both modules together. Add the following code to environments/dev/main.tf:
# Configure the AWS provider
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
}
}
provider "aws" {
region = "us-east-1"
}
# AWS VPC module
module "aws_vpc" {
source = "../../modules/aws-vpc"
vpc_cidr = "10.0.0.0/16"
private_subnet_cidr = "10.0.1.0/24"
availability_zone = "us-east-1a"
project_name = "multicloud"
gcp_cidr = "10.1.0.0/16"
}
# AWS Transit Gateway module
module "aws_tgw" {
source = "../../modules/aws-tgw"
tgw_asn = 64512
vpc_id = module.aws_vpc.vpc_id
subnet_id = module.aws_vpc.private_subnet_id
project_name = "multicloud"
}What does this root configuration do?
This file is the entry point for Terraform. It declares the AWS provider and calls both modules you just created, passing outputs from the VPC module (the VPC ID and subnet ID) into the Transit Gateway module. This is how Terraform knows to create the VPC first, then attach it to the Transit Gateway.
- Initialize Terraform by running this from the environments/dev directory:
cd environments/dev
terraform init- Preview what Terraform will create by running:
terraform planYou should see a plan showing 6 resources to add (VPC, subnet, route table, route table association, security group, Transit Gateway, and VPC attachment).
- Apply the infrastructure by running:
terraform apply- When prompted, type yes and press Enter to confirm.
Terraform will create all the resources. You should see output ending with Apply complete! Resources: 7 added, 0 changed, 0 destroyed. (the count includes both module resources).
Seeing authentication errors?
Make sure your AWS CLI session is still active. Run aws sts get-caller-identity to verify. If it fails, re-authenticate using the same method from Step 1.
If you see a permissions error mentioning Transit Gateway, your IAM user or role needs the ec2:CreateTransitGateway permission. Ensure you have administrative access or a policy that includes EC2 Transit Gateway actions.
help me debug my Terraform apply failure for the AWS VPC and Transit Gateway
✔️ Awesome, I've got everything!
Great. Double check you have saved all your files and your terraform apply completed successfully.
ⓧ I'd like to double check the full code
Here is the complete code for each file at this point:
# AWS VPC for cross-cloud connectivity
resource "aws_vpc" "main" {
cidr_block = var.vpc_cidr
enable_dns_support = true
enable_dns_hostnames = true
tags = {
Name = "${var.project_name}-vpc"
}
}
# Private subnet with no public IP assignment
resource "aws_subnet" "private" {
vpc_id = aws_vpc.main.id
cidr_block = var.private_subnet_cidr
availability_zone = var.availability_zone
map_public_ip_on_launch = false
tags = {
Name = "${var.project_name}-private-subnet"
}
}
# Route table for the private subnet
resource "aws_route_table" "private" {
vpc_id = aws_vpc.main.id
tags = {
Name = "${var.project_name}-private-rt"
}
}
# Associate route table with the private subnet
resource "aws_route_table_association" "private" {
subnet_id = aws_subnet.private.id
route_table_id = aws_route_table.private.id
}
# Security group allowing cross-cloud traffic from GCP
resource "aws_security_group" "cross_cloud" {
name = "${var.project_name}-cross-cloud-sg"
description = "Allow inbound traffic from GCP CIDR ranges"
vpc_id = aws_vpc.main.id
ingress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = [var.gcp_cidr]
description = "Allow all traffic from GCP"
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
description = "Allow all outbound traffic"
}
tags = {
Name = "${var.project_name}-cross-cloud-sg"
}
}variable "vpc_cidr" {
description = "CIDR block for the AWS VPC"
type = string
default = "10.0.0.0/16"
}
variable "private_subnet_cidr" {
description = "CIDR block for the private subnet"
type = string
default = "10.0.1.0/24"
}
variable "availability_zone" {
description = "Availability zone for the subnet"
type = string
default = "us-east-1a"
}
variable "project_name" {
description = "Project name used for resource tagging"
type = string
default = "multicloud"
}
variable "gcp_cidr" {
description = "GCP CIDR range to allow inbound traffic from"
type = string
default = "10.1.0.0/16"
}output "vpc_id" {
description = "ID of the created VPC"
value = aws_vpc.main.id
}
output "private_subnet_id" {
description = "ID of the private subnet"
value = aws_subnet.private.id
}
output "private_route_table_id" {
description = "ID of the private route table"
value = aws_route_table.private.id
}
output "security_group_id" {
description = "ID of the cross-cloud security group"
value = aws_security_group.cross_cloud.id
}# Transit Gateway with custom ASN and ECMP support
resource "aws_ec2_transit_gateway" "main" {
amazon_side_asn = var.tgw_asn
default_route_table_association = "enable"
default_route_table_propagation = "enable"
vpn_ecmp_support = "enable"
dns_support = "enable"
tags = {
Name = "${var.project_name}-tgw"
}
}
# Attach the VPC to the Transit Gateway
resource "aws_ec2_transit_gateway_vpc_attachment" "main" {
subnet_ids = [var.subnet_id]
transit_gateway_id = aws_ec2_transit_gateway.main.id
vpc_id = var.vpc_id
tags = {
Name = "${var.project_name}-tgw-vpc-attachment"
}
}variable "tgw_asn" {
description = "BGP ASN for the Transit Gateway"
type = number
default = 64512
}
variable "vpc_id" {
description = "ID of the VPC to attach to the Transit Gateway"
type = string
}
variable "subnet_id" {
description = "ID of the subnet for the TGW VPC attachment"
type = string
}
variable "project_name" {
description = "Project name used for resource tagging"
type = string
default = "multicloud"
}output "transit_gateway_id" {
description = "ID of the Transit Gateway"
value = aws_ec2_transit_gateway.main.id
}
output "transit_gateway_asn" {
description = "BGP ASN of the Transit Gateway"
value = aws_ec2_transit_gateway.main.amazon_side_asn
}# Configure the AWS provider
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
}
}
provider "aws" {
region = "us-east-1"
}
# AWS VPC module
module "aws_vpc" {
source = "../../modules/aws-vpc"
vpc_cidr = "10.0.0.0/16"
private_subnet_cidr = "10.0.1.0/24"
availability_zone = "us-east-1a"
project_name = "multicloud"
gcp_cidr = "10.1.0.0/16"
}
# AWS Transit Gateway module
module "aws_tgw" {
source = "../../modules/aws-tgw"
tgw_asn = 64512
vpc_id = module.aws_vpc.vpc_id
subnet_id = module.aws_vpc.private_subnet_id
project_name = "multicloud"
}Your AWS foundation is solid. You have a private VPC with a Transit Gateway ready to receive VPN connections. Next up, you will build the matching GCP hub-and-spoke VPC topology that connects to this Transit Gateway through an HA VPN tunnel mesh.
Build the GCP Hub-and-Spoke VPC Topology
Your AWS foundation is solid. The Transit Gateway is running with ECMP enabled, and the VPC is ready to accept cross-cloud traffic.
Now you need the GCP side of the tunnel. Enterprise GCP deployments use a hub-and-spoke VPC topology where the hub handles network connectivity (VPN endpoints) and spokes handle application workloads. This separation means you can add more spoke VPCs later without touching VPN configuration.
In this step, get ready to:
- Create a hub VPC for VPN connectivity and a spoke VPC for application workloads.
- Configure VPC peering with custom route export so BGP-learned routes propagate to the spoke.
- Set up firewall rules allowing traffic from the AWS CIDR range.
Create the GCP hub and spoke VPCs
The hub VPC hosts VPN gateway resources. The spoke VPC hosts your application instances. Keeping them separate follows Google's recommended multi-VPC architecture for security isolation.
- Create the file modules/gcp-vpc/main.tf and add the network and subnet definitions by pasting the following code:
# Hub VPC - VPN endpoint for cross-cloud connectivity
resource "google_compute_network" "hub" {
name = "hub-vpc"
auto_create_subnetworks = false
}
# Spoke VPC - Application workload
resource "google_compute_network" "spoke" {
name = "spoke-vpc"
auto_create_subnetworks = false
}
# Hub subnet in us-central1
resource "google_compute_subnetwork" "hub" {
name = "hub-subnet"
ip_cidr_range = "10.1.0.0/24"
region = "us-central1"
network = google_compute_network.hub.id
}
# Spoke subnet in us-central1
resource "google_compute_subnetwork" "spoke" {
name = "spoke-subnet"
ip_cidr_range = "10.2.0.0/24"
region = "us-central1"
network = google_compute_network.spoke.id
}What does this code do?
You define two VPCs in custom subnet mode. Custom mode means GCP will not auto-create subnets in every region. You control exactly which regions get subnets.
The hub subnet (10.1.0.0/24) and spoke subnet (10.2.0.0/24) are both in us-central1. These CIDRs are non-overlapping with the AWS VPC (10.0.0.0/16), which is a requirement for cross-cloud routing.
- Save modules/gcp-vpc/main.tf.
- Run terraform validate from your project directory. You should see Success! The configuration is valid. confirming no syntax errors.
Seeing a validation error?
Check that auto_create_subnetworks is set to false (not "false" in quotes). Terraform booleans do not use quotes.
Also verify that each google_compute_subnetwork references the correct network via google_compute_network.hub.id or google_compute_network.spoke.id.
Help me debug my GCP VPC Terraform validation error.
Configure VPC peering with custom route export
VPC peering connects two VPCs so traffic can flow between them using internal IPs. The critical detail is enabling custom route export. Without it, BGP-learned routes from the Cloud Router (added in the next step) will not propagate to the spoke VPC.
- Add the peering configuration below the subnet definitions in modules/gcp-vpc/main.tf:
# Hub to spoke peering - exports learned routes from Cloud Router
resource "google_compute_network_peering" "hub_to_spoke" {
name = "hub-to-spoke"
network = google_compute_network.hub.self_link
peer_network = google_compute_network.spoke.self_link
export_custom_routes = true
import_custom_routes = true
}
# Spoke to hub peering - imports learned routes for cross-cloud connectivity
resource "google_compute_network_peering" "spoke_to_hub" {
name = "spoke-to-hub"
network = google_compute_network.spoke.self_link
peer_network = google_compute_network.hub.self_link
export_custom_routes = true
import_custom_routes = true
}Why do both sides need peering?
GCP VPC peering is not transitive. Both networks must explicitly create a peering with each other for the connection to become active.
Setting export_custom_routes = true on the hub peering means routes learned by the Cloud Router (via BGP from AWS) will be advertised to the spoke VPC. Setting import_custom_routes = true on the spoke side means the spoke accepts those routes. This is how spoke instances learn the path to 10.0.0.0/16 (the AWS VPC).
Why hub-and-spoke instead of a flat VPC?
A flat VPC puts VPN gateways and application workloads in the same network. This works but creates a blast radius problem. Any firewall misconfiguration exposes everything.
The hub-and-spoke pattern isolates connectivity infrastructure from application workloads. You can add new spoke VPCs (for new teams or environments) without modifying the VPN configuration. Google recommends this topology for production multi-cloud architectures.
- Save modules/gcp-vpc/main.tf.
- You will not see a change yet. The next chunk adds firewall rules, and then you will apply everything together.
Set up firewall rules for cross-cloud traffic
GCP denies all ingress traffic by default. Without explicit firewall rules, traffic arriving from AWS over the VPN tunnels will be dropped at the VPC boundary.
- Add the firewall rules below the peering resources in modules/gcp-vpc/main.tf:
# Allow all traffic from AWS VPC into hub VPC
resource "google_compute_firewall" "hub_allow_aws" {
name = "hub-allow-aws"
network = google_compute_network.hub.name
allow {
protocol = "all"
}
source_ranges = ["10.0.0.0/16"]
direction = "INGRESS"
}
# Allow all traffic from AWS VPC into spoke VPC
resource "google_compute_firewall" "spoke_allow_aws" {
name = "spoke-allow-aws"
network = google_compute_network.spoke.name
allow {
protocol = "all"
}
source_ranges = ["10.0.0.0/16"]
direction = "INGRESS"
}What does this code do?
Each firewall rule allows all protocols and all ports from the AWS VPC CIDR (10.0.0.0/16). The source_ranges field accepts CIDR notation to match incoming packet source IPs.
Both VPCs need their own rule because GCP firewall rules are scoped to a single network. Traffic entering the spoke VPC (via peering) still needs an explicit allow rule on the spoke network.
- Save modules/gcp-vpc/main.tf.
- Apply the module by running:
terraform init
terraform apply- Type yes when prompted to confirm. Terraform will create 8 resources (2 networks, 2 subnets, 2 peerings, 2 firewall rules).
- Verify the VPCs and peering were created by running:
gcloud compute networks list
gcloud compute networks peerings list --network=hub-vpc- You should see hub-vpc and spoke-vpc in the networks list. The peerings command should show hub-to-spoke with state ACTIVE.
Peering state shows INACTIVE?
Both sides of the peering must exist for it to become ACTIVE. If Terraform created one peering but errored on the second, the first will show INACTIVE. Re-run terraform apply to complete the creation.
Also check that the subnet CIDRs do not overlap. Overlapping ranges between peered networks will cause the peering to fail.
Help me troubleshoot my GCP VPC peering showing INACTIVE state.
✔️ Awesome, I've got everything!
Great. Double check you have saved modules/gcp-vpc/main.tf and that all 8 resources were created successfully.
ⓧ I'd like to double check the full code
# Hub VPC - VPN endpoint for cross-cloud connectivity
resource "google_compute_network" "hub" {
name = "hub-vpc"
auto_create_subnetworks = false
}
# Spoke VPC - Application workload
resource "google_compute_network" "spoke" {
name = "spoke-vpc"
auto_create_subnetworks = false
}
# Hub subnet in us-central1
resource "google_compute_subnetwork" "hub" {
name = "hub-subnet"
ip_cidr_range = "10.1.0.0/24"
region = "us-central1"
network = google_compute_network.hub.id
}
# Spoke subnet in us-central1
resource "google_compute_subnetwork" "spoke" {
name = "spoke-subnet"
ip_cidr_range = "10.2.0.0/24"
region = "us-central1"
network = google_compute_network.spoke.id
}
# Hub to spoke peering - exports learned routes from Cloud Router
resource "google_compute_network_peering" "hub_to_spoke" {
name = "hub-to-spoke"
network = google_compute_network.hub.self_link
peer_network = google_compute_network.spoke.self_link
export_custom_routes = true
import_custom_routes = true
}
# Spoke to hub peering - imports learned routes for cross-cloud connectivity
resource "google_compute_network_peering" "spoke_to_hub" {
name = "spoke-to-hub"
network = google_compute_network.spoke.self_link
peer_network = google_compute_network.hub.self_link
export_custom_routes = true
import_custom_routes = true
}
# Allow all traffic from AWS VPC into hub VPC
resource "google_compute_firewall" "hub_allow_aws" {
name = "hub-allow-aws"
network = google_compute_network.hub.name
allow {
protocol = "all"
}
source_ranges = ["10.0.0.0/16"]
direction = "INGRESS"
}
# Allow all traffic from AWS VPC into spoke VPC
resource "google_compute_firewall" "spoke_allow_aws" {
name = "spoke-allow-aws"
network = google_compute_network.spoke.name
allow {
protocol = "all"
}
source_ranges = ["10.0.0.0/16"]
direction = "INGRESS"
}Your GCP network topology is ready. The hub VPC will host the HA VPN gateway, and the spoke VPC will host your application. Next up, you will connect these two clouds by building the HA VPN tunnel mesh with BGP dynamic routing.
Set Up HA VPN with BGP and ECMP
Your GCP hub-and-spoke VPC topology is in place. Now you need to connect it to AWS through an encrypted tunnel that carries traffic between both clouds privately.
In this step, you will create an HA VPN connection with 4 IPsec tunnels and BGP dynamic routing between AWS and GCP. The Transit Gateway enables ECMP (Equal Cost Multi-Path) routing, which load-balances traffic across all 4 tunnels simultaneously instead of using one active and three standby.
In this step, get ready to:
- Create the GCP HA VPN Gateway and Cloud Router with BGP.
- Create AWS Customer Gateways and Site-to-Site VPN connections with IKEv2.
- Configure GCP VPN tunnels, BGP sessions, and verify ECMP across tunnels.
Create the GCP HA VPN Gateway and Cloud Router
The HA VPN Gateway provides two external interfaces for redundancy. The Cloud Router manages BGP sessions and dynamically exchanges routes with AWS.
- Create a new file modules/gcp-vpn/main.tf and add the HA VPN Gateway and Cloud Router resources by pasting this code:
# HA VPN Gateway with two interfaces for high availability
resource "google_compute_ha_vpn_gateway" "ha_vpn_gw" {
name = "ha-vpn-gw"
network = var.hub_vpc_id
region = var.region
}
# Cloud Router for BGP dynamic routing with GCP-side ASN
resource "google_compute_router" "cloud_router" {
name = "cloud-router"
network = var.hub_vpc_id
region = var.region
bgp {
asn = 65000
}
}What does this code do?
The google_compute_ha_vpn_gateway automatically assigns two external IP addresses (one per interface). These IPs become the endpoints that AWS connects to.
The google_compute_router uses ASN 65000 for the GCP side. AWS Transit Gateway uses ASN 64512. BGP sessions between these two ASNs dynamically exchange route information so each cloud knows how to reach the other.
- Save modules/gcp-vpn/main.tf.
Why two interfaces?
GCP HA VPN requires two interfaces for 99.99% availability. Each interface connects to separate AWS VPN tunnels. If one interface fails, traffic automatically routes through the other.
Create AWS Customer Gateways and VPN connections
AWS needs to know the IP addresses of the GCP HA VPN Gateway interfaces. Customer Gateways represent the GCP endpoints from the AWS perspective. Each VPN connection creates two tunnels, giving you 4 tunnels total for ECMP.
- Create a new file modules/aws-tgw/vpn.tf and add the Customer Gateways and VPN connections by pasting this code:
# Customer Gateways representing GCP HA VPN interface IPs
resource "aws_customer_gateway" "gcp_gw_0" {
bgp_asn = 65000
ip_address = var.gcp_ha_vpn_ip_0
type = "ipsec.1"
tags = {
Name = "gcp-customer-gw-0"
}
}
resource "aws_customer_gateway" "gcp_gw_1" {
bgp_asn = 65000
ip_address = var.gcp_ha_vpn_ip_1
type = "ipsec.1"
tags = {
Name = "gcp-customer-gw-1"
}
}
# Site-to-Site VPN connection 0 attached to Transit Gateway (IKEv2)
resource "aws_vpn_connection" "to_gcp_0" {
transit_gateway_id = var.transit_gateway_id
customer_gateway_id = aws_customer_gateway.gcp_gw_0.id
type = "ipsec.1"
static_routes_only = false
# Use IKEv2 to avoid rekey failures with GCP
tunnel1_ike_versions = ["ikev2"]
tunnel2_ike_versions = ["ikev2"]
tags = {
Name = "vpn-to-gcp-0"
}
}
# Site-to-Site VPN connection 1 attached to Transit Gateway (IKEv2)
resource "aws_vpn_connection" "to_gcp_1" {
transit_gateway_id = var.transit_gateway_id
customer_gateway_id = aws_customer_gateway.gcp_gw_1.id
type = "ipsec.1"
static_routes_only = false
tunnel1_ike_versions = ["ikev2"]
tunnel2_ike_versions = ["ikev2"]
tags = {
Name = "vpn-to-gcp-1"
}
}
# Enable route propagation so BGP-learned routes appear in the TGW route table
resource "aws_ec2_transit_gateway_route_table_propagation" "vpn_0" {
transit_gateway_attachment_id = aws_vpn_connection.to_gcp_0.transit_gateway_attachment_id
transit_gateway_route_table_id = var.transit_gateway_route_table_id
}
resource "aws_ec2_transit_gateway_route_table_propagation" "vpn_1" {
transit_gateway_attachment_id = aws_vpn_connection.to_gcp_1.transit_gateway_attachment_id
transit_gateway_route_table_id = var.transit_gateway_route_table_id
}What does this code do?
Two aws_customer_gateway resources tell AWS about the GCP HA VPN interfaces. The bgp_asn of 65000 matches the Cloud Router ASN so BGP sessions can establish.
Each aws_vpn_connection creates 2 IPsec tunnels (4 total across both connections). Setting static_routes_only = false enables BGP dynamic routing. The tunnel1_ike_versions and tunnel2_ike_versions attributes enforce IKEv2 to avoid rekey failures with GCP.
The aws_ec2_transit_gateway_route_table_propagation resources ensure that routes learned via BGP from GCP automatically appear in the Transit Gateway route table.
- Save modules/aws-tgw/vpn.tf.
Why IKEv2 instead of IKEv1?
GCP recommends IKEv2 with a reduced set of transform sets on the AWS side. IKEv1 can cause rekey failures due to large SA payloads and IP fragmentation. IKEv2 handles rekeying more reliably for cross-cloud tunnels.
Seeing a Customer Gateway error?
If Terraform reports an invalid IP address for the Customer Gateway, the GCP HA VPN Gateway must be applied first so its interface IPs are available as outputs. Run terraform apply on the GCP VPN module before the AWS TGW VPN module.
help me troubleshoot my AWS Customer Gateway creation error
Configure GCP VPN tunnels, BGP sessions, and verify ECMP
Now you need to tell GCP about the 4 AWS tunnel endpoints and create the actual VPN tunnels with BGP peers. GCP uses an external VPN gateway resource to represent all 4 AWS tunnel outside IPs.
- Return to modules/gcp-vpn/main.tf and add the external VPN gateway, tunnels, and BGP configuration below the Cloud Router resource:
# External VPN gateway representing all 4 AWS tunnel outside IPs
resource "google_compute_external_vpn_gateway" "aws_gw" {
name = "aws-external-gw"
redundancy_type = "FOUR_IPS_REDUNDANCY"
interface {
id = 0
ip_address = var.aws_tunnel_ips[0]
}
interface {
id = 1
ip_address = var.aws_tunnel_ips[1]
}
interface {
id = 2
ip_address = var.aws_tunnel_ips[2]
}
interface {
id = 3
ip_address = var.aws_tunnel_ips[3]
}
}
# VPN Tunnel 0 (HA VPN interface 0 to AWS tunnel endpoint 0)
resource "google_compute_vpn_tunnel" "tunnel_0" {
name = "vpn-tunnel-0"
vpn_gateway = google_compute_ha_vpn_gateway.ha_vpn_gw.id
vpn_gateway_interface = 0
peer_external_gateway = google_compute_external_vpn_gateway.aws_gw.id
peer_external_gateway_interface = 0
shared_secret = var.tunnel_secrets[0]
router = google_compute_router.cloud_router.id
ike_version = 2
}
# VPN Tunnel 1 (HA VPN interface 0 to AWS tunnel endpoint 1)
resource "google_compute_vpn_tunnel" "tunnel_1" {
name = "vpn-tunnel-1"
vpn_gateway = google_compute_ha_vpn_gateway.ha_vpn_gw.id
vpn_gateway_interface = 0
peer_external_gateway = google_compute_external_vpn_gateway.aws_gw.id
peer_external_gateway_interface = 1
shared_secret = var.tunnel_secrets[1]
router = google_compute_router.cloud_router.id
ike_version = 2
}
# VPN Tunnel 2 (HA VPN interface 1 to AWS tunnel endpoint 2)
resource "google_compute_vpn_tunnel" "tunnel_2" {
name = "vpn-tunnel-2"
vpn_gateway = google_compute_ha_vpn_gateway.ha_vpn_gw.id
vpn_gateway_interface = 1
peer_external_gateway = google_compute_external_vpn_gateway.aws_gw.id
peer_external_gateway_interface = 2
shared_secret = var.tunnel_secrets[2]
router = google_compute_router.cloud_router.id
ike_version = 2
}
# VPN Tunnel 3 (HA VPN interface 1 to AWS tunnel endpoint 3)
resource "google_compute_vpn_tunnel" "tunnel_3" {
name = "vpn-tunnel-3"
vpn_gateway = google_compute_ha_vpn_gateway.ha_vpn_gw.id
vpn_gateway_interface = 1
peer_external_gateway = google_compute_external_vpn_gateway.aws_gw.id
peer_external_gateway_interface = 3
shared_secret = var.tunnel_secrets[3]
router = google_compute_router.cloud_router.id
ike_version = 2
}What does this code do?
The google_compute_external_vpn_gateway with FOUR_IPS_REDUNDANCY tells GCP that the AWS peer has 4 tunnel endpoints. This is the required redundancy type when connecting HA VPN to AWS for the 99.99% SLA.
Each google_compute_vpn_tunnel maps one GCP HA VPN interface to one AWS tunnel endpoint. Tunnels 0-1 use GCP interface 0. Tunnels 2-3 use GCP interface 1. This distribution ensures redundancy across both GCP interfaces.
- Now add the BGP router interfaces and peers below the tunnel resources in the same file:
# Router interfaces for BGP sessions (one per tunnel)
resource "google_compute_router_interface" "interface_0" {
name = "bgp-interface-0"
router = google_compute_router.cloud_router.name
region = var.region
ip_range = var.bgp_interface_ips[0]
vpn_tunnel = google_compute_vpn_tunnel.tunnel_0.name
}
resource "google_compute_router_interface" "interface_1" {
name = "bgp-interface-1"
router = google_compute_router.cloud_router.name
region = var.region
ip_range = var.bgp_interface_ips[1]
vpn_tunnel = google_compute_vpn_tunnel.tunnel_1.name
}
resource "google_compute_router_interface" "interface_2" {
name = "bgp-interface-2"
router = google_compute_router.cloud_router.name
region = var.region
ip_range = var.bgp_interface_ips[2]
vpn_tunnel = google_compute_vpn_tunnel.tunnel_2.name
}
resource "google_compute_router_interface" "interface_3" {
name = "bgp-interface-3"
router = google_compute_router.cloud_router.name
region = var.region
ip_range = var.bgp_interface_ips[3]
vpn_tunnel = google_compute_vpn_tunnel.tunnel_3.name
}
# BGP peers (one per tunnel, peering with AWS TGW ASN 64512)
resource "google_compute_router_peer" "peer_0" {
name = "bgp-peer-0"
router = google_compute_router.cloud_router.name
region = var.region
peer_ip_address = var.bgp_peer_ips[0]
peer_asn = 64512
advertised_route_priority = 100
interface = google_compute_router_interface.interface_0.name
}
resource "google_compute_router_peer" "peer_1" {
name = "bgp-peer-1"
router = google_compute_router.cloud_router.name
region = var.region
peer_ip_address = var.bgp_peer_ips[1]
peer_asn = 64512
advertised_route_priority = 100
interface = google_compute_router_interface.interface_1.name
}
resource "google_compute_router_peer" "peer_2" {
name = "bgp-peer-2"
router = google_compute_router.cloud_router.name
region = var.region
peer_ip_address = var.bgp_peer_ips[2]
peer_asn = 64512
advertised_route_priority = 100
interface = google_compute_router_interface.interface_2.name
}
resource "google_compute_router_peer" "peer_3" {
name = "bgp-peer-3"
router = google_compute_router.cloud_router.name
region = var.region
peer_ip_address = var.bgp_peer_ips[3]
peer_asn = 64512
advertised_route_priority = 100
interface = google_compute_router_interface.interface_3.name
}What does the BGP configuration do?
Each google_compute_router_interface binds a BGP session to a specific VPN tunnel. The ip_range uses /30 CIDR blocks from the 169.254.0.0/16 range that AWS assigns to each tunnel.
Each google_compute_router_peer establishes a BGP session with the AWS side (ASN 64512). The advertised_route_priority of 100 is equal across all peers. This equal priority is what enables ECMP. The Transit Gateway distributes traffic across all active tunnels simultaneously.
- Save modules/gcp-vpn/main.tf.
- Apply the Terraform configuration by running:
terraform apply- After the apply completes, verify the VPN tunnel status in the GCP console by running:
gcloud compute vpn-tunnels list --filter="region:us-central1"You should see 4 tunnels listed with their status. Once BGP sessions establish, the status changes to ESTABLISHED.
Tunnels showing as NEGOTIATION_FAILURE?
Verify that your shared secrets match between the AWS VPN connection outputs and the GCP tunnel configuration. Also confirm that the IKE version is set to 2 on both sides.
Check that your security group and firewall rules allow UDP port 500 and UDP port 4500 (used by IKE and NAT traversal).
help me debug my VPN tunnel negotiation failure between AWS and GCP
✔️ Awesome, I've got everything!
Great. Double check you have saved both modules/gcp-vpn/main.tf and modules/aws-tgw/vpn.tf.
ⓧ I'd like to double check the full code
Here is the complete modules/gcp-vpn/main.tf:
# HA VPN Gateway with two interfaces for high availability
resource "google_compute_ha_vpn_gateway" "ha_vpn_gw" {
name = "ha-vpn-gw"
network = var.hub_vpc_id
region = var.region
}
# Cloud Router for BGP dynamic routing with GCP-side ASN
resource "google_compute_router" "cloud_router" {
name = "cloud-router"
network = var.hub_vpc_id
region = var.region
bgp {
asn = 65000
}
}
# External VPN gateway representing all 4 AWS tunnel outside IPs
resource "google_compute_external_vpn_gateway" "aws_gw" {
name = "aws-external-gw"
redundancy_type = "FOUR_IPS_REDUNDANCY"
interface {
id = 0
ip_address = var.aws_tunnel_ips[0]
}
interface {
id = 1
ip_address = var.aws_tunnel_ips[1]
}
interface {
id = 2
ip_address = var.aws_tunnel_ips[2]
}
interface {
id = 3
ip_address = var.aws_tunnel_ips[3]
}
}
# VPN Tunnel 0 (HA VPN interface 0 to AWS tunnel endpoint 0)
resource "google_compute_vpn_tunnel" "tunnel_0" {
name = "vpn-tunnel-0"
vpn_gateway = google_compute_ha_vpn_gateway.ha_vpn_gw.id
vpn_gateway_interface = 0
peer_external_gateway = google_compute_external_vpn_gateway.aws_gw.id
peer_external_gateway_interface = 0
shared_secret = var.tunnel_secrets[0]
router = google_compute_router.cloud_router.id
ike_version = 2
}
# VPN Tunnel 1 (HA VPN interface 0 to AWS tunnel endpoint 1)
resource "google_compute_vpn_tunnel" "tunnel_1" {
name = "vpn-tunnel-1"
vpn_gateway = google_compute_ha_vpn_gateway.ha_vpn_gw.id
vpn_gateway_interface = 0
peer_external_gateway = google_compute_external_vpn_gateway.aws_gw.id
peer_external_gateway_interface = 1
shared_secret = var.tunnel_secrets[1]
router = google_compute_router.cloud_router.id
ike_version = 2
}
# VPN Tunnel 2 (HA VPN interface 1 to AWS tunnel endpoint 2)
resource "google_compute_vpn_tunnel" "tunnel_2" {
name = "vpn-tunnel-2"
vpn_gateway = google_compute_ha_vpn_gateway.ha_vpn_gw.id
vpn_gateway_interface = 1
peer_external_gateway = google_compute_external_vpn_gateway.aws_gw.id
peer_external_gateway_interface = 2
shared_secret = var.tunnel_secrets[2]
router = google_compute_router.cloud_router.id
ike_version = 2
}
# VPN Tunnel 3 (HA VPN interface 1 to AWS tunnel endpoint 3)
resource "google_compute_vpn_tunnel" "tunnel_3" {
name = "vpn-tunnel-3"
vpn_gateway = google_compute_ha_vpn_gateway.ha_vpn_gw.id
vpn_gateway_interface = 1
peer_external_gateway = google_compute_external_vpn_gateway.aws_gw.id
peer_external_gateway_interface = 3
shared_secret = var.tunnel_secrets[3]
router = google_compute_router.cloud_router.id
ike_version = 2
}
# Router interfaces for BGP sessions (one per tunnel)
resource "google_compute_router_interface" "interface_0" {
name = "bgp-interface-0"
router = google_compute_router.cloud_router.name
region = var.region
ip_range = var.bgp_interface_ips[0]
vpn_tunnel = google_compute_vpn_tunnel.tunnel_0.name
}
resource "google_compute_router_interface" "interface_1" {
name = "bgp-interface-1"
router = google_compute_router.cloud_router.name
region = var.region
ip_range = var.bgp_interface_ips[1]
vpn_tunnel = google_compute_vpn_tunnel.tunnel_1.name
}
resource "google_compute_router_interface" "interface_2" {
name = "bgp-interface-2"
router = google_compute_router.cloud_router.name
region = var.region
ip_range = var.bgp_interface_ips[2]
vpn_tunnel = google_compute_vpn_tunnel.tunnel_2.name
}
resource "google_compute_router_interface" "interface_3" {
name = "bgp-interface-3"
router = google_compute_router.cloud_router.name
region = var.region
ip_range = var.bgp_interface_ips[3]
vpn_tunnel = google_compute_vpn_tunnel.tunnel_3.name
}
# BGP peers (one per tunnel, peering with AWS TGW ASN 64512)
resource "google_compute_router_peer" "peer_0" {
name = "bgp-peer-0"
router = google_compute_router.cloud_router.name
region = var.region
peer_ip_address = var.bgp_peer_ips[0]
peer_asn = 64512
advertised_route_priority = 100
interface = google_compute_router_interface.interface_0.name
}
resource "google_compute_router_peer" "peer_1" {
name = "bgp-peer-1"
router = google_compute_router.cloud_router.name
region = var.region
peer_ip_address = var.bgp_peer_ips[1]
peer_asn = 64512
advertised_route_priority = 100
interface = google_compute_router_interface.interface_1.name
}
resource "google_compute_router_peer" "peer_2" {
name = "bgp-peer-2"
router = google_compute_router.cloud_router.name
region = var.region
peer_ip_address = var.bgp_peer_ips[2]
peer_asn = 64512
advertised_route_priority = 100
interface = google_compute_router_interface.interface_2.name
}
resource "google_compute_router_peer" "peer_3" {
name = "bgp-peer-3"
router = google_compute_router.cloud_router.name
region = var.region
peer_ip_address = var.bgp_peer_ips[3]
peer_asn = 64512
advertised_route_priority = 100
interface = google_compute_router_interface.interface_3.name
}And here is the complete modules/aws-tgw/vpn.tf:
# Customer Gateways representing GCP HA VPN interface IPs
resource "aws_customer_gateway" "gcp_gw_0" {
bgp_asn = 65000
ip_address = var.gcp_ha_vpn_ip_0
type = "ipsec.1"
tags = {
Name = "gcp-customer-gw-0"
}
}
resource "aws_customer_gateway" "gcp_gw_1" {
bgp_asn = 65000
ip_address = var.gcp_ha_vpn_ip_1
type = "ipsec.1"
tags = {
Name = "gcp-customer-gw-1"
}
}
# Site-to-Site VPN connection 0 attached to Transit Gateway (IKEv2)
resource "aws_vpn_connection" "to_gcp_0" {
transit_gateway_id = var.transit_gateway_id
customer_gateway_id = aws_customer_gateway.gcp_gw_0.id
type = "ipsec.1"
static_routes_only = false
# Use IKEv2 to avoid rekey failures with GCP
tunnel1_ike_versions = ["ikev2"]
tunnel2_ike_versions = ["ikev2"]
tags = {
Name = "vpn-to-gcp-0"
}
}
# Site-to-Site VPN connection 1 attached to Transit Gateway (IKEv2)
resource "aws_vpn_connection" "to_gcp_1" {
transit_gateway_id = var.transit_gateway_id
customer_gateway_id = aws_customer_gateway.gcp_gw_1.id
type = "ipsec.1"
static_routes_only = false
tunnel1_ike_versions = ["ikev2"]
tunnel2_ike_versions = ["ikev2"]
tags = {
Name = "vpn-to-gcp-1"
}
}
# Enable route propagation so BGP-learned routes appear in the TGW route table
resource "aws_ec2_transit_gateway_route_table_propagation" "vpn_0" {
transit_gateway_attachment_id = aws_vpn_connection.to_gcp_0.transit_gateway_attachment_id
transit_gateway_route_table_id = var.transit_gateway_route_table_id
}
resource "aws_ec2_transit_gateway_route_table_propagation" "vpn_1" {
transit_gateway_attachment_id = aws_vpn_connection.to_gcp_1.transit_gateway_attachment_id
transit_gateway_route_table_id = var.transit_gateway_route_table_id
}- Verify ECMP is working by checking the Cloud Router status:
gcloud compute routers get-status cloud-router --region=us-central1You should see 4 BGP peers listed with routes learned from AWS (the 10.0.0.0/16 CIDR). All peers having the same route with equal priority confirms ECMP is active.
Why Transit Gateway over Virtual Private Gateway?
A Virtual Private Gateway (VGW) only uses one active tunnel at a time. The second tunnel is standby for failover. Transit Gateway uses all tunnels simultaneously via ECMP, giving you higher throughput and better utilization.
Transit Gateway also supports connecting multiple VPCs to the same VPN, making it future-proof for growing architectures. The trade-off is an additional $0.05/hr attachment fee plus $0.02/GB data processing.
- Write your first Architecture Decision Record. Create the file docs/adr/0001-transit-gateway-over-vpgw.md with this content:
# ADR 0001: Transit Gateway over Virtual Private Gateway
## Status
Accepted
## Context
We need to connect our AWS VPC to GCP via Site-to-Site VPN. AWS offers two
options: Virtual Private Gateway (VGW) and Transit Gateway (TGW).
## Decision
We chose Transit Gateway for the cross-cloud VPN attachment.
## Rationale
- **ECMP support**: TGW distributes traffic across all 4 tunnels simultaneously.
VGW uses only one active tunnel (others are standby).
- **Multi-VPC attachment**: TGW allows multiple VPCs to share the same VPN
connection without additional VPN tunnels.
- **Enterprise scalability**: TGW supports up to 5,000 attachments vs VGW's
single-VPC limitation.
## Consequences
- Additional cost: $0.05/hr per VPN attachment + $0.02/GB data processing.
- Slightly more complex Terraform configuration.
- Higher throughput ceiling for cross-cloud communication.
What is an Architecture Decision Record?
ADRs document the reasoning behind significant technical decisions. They capture the context at the time the decision was made, the alternatives considered, and why one was chosen. Interviewers value ADRs because they demonstrate structured engineering thinking beyond just writing code.
- Save docs/adr/0001-transit-gateway-over-vpgw.md.
Your two clouds are now connected through 4 encrypted IPsec tunnels with BGP dynamically exchanging routes. Next up, you will configure cross-cloud DNS so services can discover each other by hostname instead of hardcoded IPs.
Configure Cross-Cloud DNS Resolution
Your VPN tunnels are up, BGP routes are propagating, and ECMP is load-balancing traffic across all four tunnels. The network pipe between AWS and GCP is working.
Now comes the problem: services on each cloud still need to find each other by IP address. In a real enterprise, hardcoding IPs breaks the moment you scale or redeploy. Cross-cloud DNS resolution lets services discover each other by hostname over the private VPN path.
In this step, get ready to:
- Create a Route 53 Resolver inbound endpoint and private hosted zone on AWS.
- Create a GCP Cloud DNS forwarding zone and DNS peering between hub and spoke.
- Verify the DNS infrastructure deploys successfully with Terraform.
Create the Route 53 Resolver inbound endpoint and private hosted zone
For GCP to resolve AWS hostnames over the VPN, AWS needs a DNS endpoint that accepts incoming queries from outside the VPC. Route 53 Resolver inbound endpoints do exactly this.
- Create a new file modules/dns/variables.tf to define the module inputs by pasting this code:
- This file already exists inside the modules/dns directory you created earlier.
variable "aws_vpc_id" {
description = "ID of the AWS VPC"
type = string
}
variable "aws_subnet_id" {
description = "ID of the AWS private subnet for Resolver endpoints"
type = string
}
variable "gcp_hub_network_id" {
description = "Self-link of the GCP hub VPC network"
type = string
}
variable "gcp_spoke_network_id" {
description = "Self-link of the GCP spoke VPC network"
type = string
}What do these variables do?
The DNS module spans both clouds, so it needs references to resources from each. aws_vpc_id and aws_subnet_id place the Resolver endpoint in the correct VPC and subnet. gcp_hub_network_id and gcp_spoke_network_id attach the Cloud DNS zones to the correct GCP networks.
- Create modules/dns/main.tf and add the AWS DNS infrastructure by pasting this code:
# Security group for Route 53 Resolver inbound endpoint
resource "aws_security_group" "resolver" {
name = "route53-resolver-sg"
description = "Allow DNS queries from GCP over VPN"
vpc_id = var.aws_vpc_id
# Allow DNS over UDP from GCP subnets and Cloud DNS source range
ingress {
description = "DNS UDP from GCP"
from_port = 53
to_port = 53
protocol = "udp"
cidr_blocks = ["10.1.0.0/24", "10.2.0.0/24", "35.199.192.0/19"]
}
# Allow DNS over TCP from GCP subnets and Cloud DNS source range
ingress {
description = "DNS TCP from GCP"
from_port = 53
to_port = 53
protocol = "tcp"
cidr_blocks = ["10.1.0.0/24", "10.2.0.0/24", "35.199.192.0/19"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
# Route 53 Resolver inbound endpoint - accepts DNS queries from GCP over VPN
resource "aws_route53_resolver_endpoint" "inbound" {
name = "cross-cloud-inbound"
direction = "INBOUND"
security_group_ids = [aws_security_group.resolver.id]
# Two IP addresses in the private subnet for high availability
ip_address {
subnet_id = var.aws_subnet_id
ip = "10.0.1.10"
}
ip_address {
subnet_id = var.aws_subnet_id
ip = "10.0.1.11"
}
}
# Private hosted zone for AWS internal services
resource "aws_route53_zone" "internal_aws" {
name = "internal.aws"
vpc {
vpc_id = var.aws_vpc_id
}
}What does this code do?
The security group allows DNS traffic (port 53) from GCP subnet ranges and from 35.199.192.0/19. That last range is critical. GCP Cloud DNS sends forwarded queries from those source IPs.
The Resolver inbound endpoint gets two static IPs (10.0.1.10 and 10.0.1.11) for redundancy. GCP will forward DNS queries to both of these over the VPN.
The private hosted zone internal.aws is where you will register service hostnames in the next step.
Why two Resolver IP addresses?
Route 53 Resolver inbound endpoints require a minimum of two ENIs (Elastic Network Interfaces) for high availability. If one ENI becomes unhealthy, queries automatically fail over to the other. This adds approximately $0.125/hr per ENI to your running costs.
- Save modules/dns/variables.tf and modules/dns/main.tf.
Seeing a validation error on ip_address?
Make sure each ip_address block has a valid IP within the 10.0.1.0/24 subnet range. The IPs 10.0.1.10 and 10.0.1.11 should not conflict with any existing resources in your subnet.
help me debug a Terraform validation error on my Route 53 Resolver endpoint ip_address blocks
Create the GCP Cloud DNS forwarding zone and DNS peering
The GCP side needs a forwarding zone that sends internal.aws queries to the Route 53 Resolver over the VPN. It also needs DNS peering so the spoke VPC can resolve these names through the hub.
- In modules/dns/main.tf, add the GCP Cloud DNS forwarding zone below the AWS resources:
# --- GCP Cloud DNS ---
# Forwarding zone: sends internal.aws queries to Route 53 Resolver over VPN
resource "google_dns_managed_zone" "forward_to_aws" {
name = "forward-to-aws"
dns_name = "internal.aws."
description = "Forwards DNS queries to AWS Route 53 Resolver over VPN"
visibility = "private"
private_visibility_config {
networks {
network_url = var.gcp_hub_network_id
}
}
forwarding_config {
target_name_servers {
ipv4_address = "10.0.1.10"
forwarding_path = "private"
}
target_name_servers {
ipv4_address = "10.0.1.11"
forwarding_path = "private"
}
}
}What does forwarding_path = private do?
Setting forwarding_path to private forces Cloud DNS to always route queries through the VPC (and over the VPN tunnel) to reach those target IPs. Without this, Cloud DNS would try to reach the 10.0.1.x addresses over the public internet, which would fail.
The two target IPs match the static addresses you assigned to the Route 53 Resolver inbound endpoint. Cloud DNS picks the best available server if both are reachable.
- Still in modules/dns/main.tf, add the GCP private zone and DNS peering below the forwarding zone:
# Private zone for GCP internal services
resource "google_dns_managed_zone" "internal_gcp" {
name = "internal-gcp"
dns_name = "internal.gcp."
description = "Private zone for GCP internal services"
visibility = "private"
private_visibility_config {
networks {
network_url = var.gcp_hub_network_id
}
}
}
# DNS peering: spoke VPC resolves names through the hub VPC
resource "google_dns_managed_zone" "spoke_peering" {
name = "spoke-dns-peering"
dns_name = "internal.gcp."
description = "DNS peering from spoke to hub for cross-cloud resolution"
visibility = "private"
private_visibility_config {
networks {
network_url = var.gcp_spoke_network_id
}
}
peering_config {
target_network {
network_url = var.gcp_hub_network_id
}
}
}Why DNS peering from spoke to hub?
The forwarding zone for internal.aws lives on the hub VPC. Without DNS peering, instances in the spoke VPC cannot see it. The peering zone tells Cloud DNS to route the spoke's DNS queries for internal.gcp through the hub network, which has visibility into both the forwarding zone and the private zone.
This follows Google's best practice for multi-VPC DNS: place forwarding zones in a central hub and peer other VPCs to it.
- Create modules/dns/outputs.tf to expose the Resolver IPs and zone ID:
output "resolver_ips" {
description = "Route 53 Resolver inbound endpoint IP addresses"
value = ["10.0.1.10", "10.0.1.11"]
}
output "internal_aws_zone_id" {
description = "Route 53 private hosted zone ID for internal.aws"
value = aws_route53_zone.internal_aws.zone_id
}- Save all three files: modules/dns/variables.tf, modules/dns/main.tf, and modules/dns/outputs.tf.
Terraform can't find the google provider?
The DNS module uses both AWS and Google providers. Make sure your root module passes both providers to the DNS module, or that you have a required_providers block in the module specifying both hashicorp/aws and hashicorp/google.
help me configure Terraform providers for a module that uses both AWS and Google resources
Apply and verify the DNS infrastructure
With all the DNS resources defined, apply the module and confirm everything deploys correctly.
- Initialize the new module by running:
terraform init- Preview the changes by running:
terraform plan- Confirm the plan shows 5 new resources: the security group, Resolver endpoint, Route 53 zone, forwarding zone, private zone, and peering zone.
- Apply the infrastructure by running:
terraform apply- Type yes when prompted to confirm.
- After the apply completes, verify the Resolver endpoint is operational by running:
aws route53resolver list-resolver-endpoints --query "ResolverEndpoints[?Name=='cross-cloud-inbound'].{Name:Name,Status:Status,IPs:IpAddressCount}"- You should see the endpoint with Status: OPERATIONAL and IPs: 2.
- Verify the GCP forwarding zone was created by running:
gcloud dns managed-zones describe forward-to-aws --format="value(name,visibility,forwardingConfig)"- You should see the zone name forward-to-aws with private visibility and your two target IPs listed.
When will full DNS resolution work end-to-end?
The DNS infrastructure is deployed, but there are no A records in the internal.aws hosted zone yet. In the next two steps, you will deploy the backend API on AWS and register its hostname. Once that record exists, GCP instances will resolve it through this forwarding chain over the VPN.
Resolver endpoint stuck in CREATING?
Route 53 Resolver endpoints can take 2-5 minutes to transition from CREATING to OPERATIONAL. Wait a few minutes and re-run the status check.
If it fails, check that your subnet 10.0.1.0/24 has available IP addresses and that the security group allows inbound port 53 traffic.
help me debug why my Route 53 Resolver endpoint is stuck in CREATING status
✔️ Awesome, I've got everything!
Great. Make sure you have saved all files in modules/dns/ and that your terraform apply completed without errors.
ⓧ I'd like to double check the full code
Here is the complete modules/dns/main.tf for comparison:
# Security group for Route 53 Resolver inbound endpoint
resource "aws_security_group" "resolver" {
name = "route53-resolver-sg"
description = "Allow DNS queries from GCP over VPN"
vpc_id = var.aws_vpc_id
# Allow DNS over UDP from GCP subnets and Cloud DNS source range
ingress {
description = "DNS UDP from GCP"
from_port = 53
to_port = 53
protocol = "udp"
cidr_blocks = ["10.1.0.0/24", "10.2.0.0/24", "35.199.192.0/19"]
}
# Allow DNS over TCP from GCP subnets and Cloud DNS source range
ingress {
description = "DNS TCP from GCP"
from_port = 53
to_port = 53
protocol = "tcp"
cidr_blocks = ["10.1.0.0/24", "10.2.0.0/24", "35.199.192.0/19"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
# Route 53 Resolver inbound endpoint - accepts DNS queries from GCP over VPN
resource "aws_route53_resolver_endpoint" "inbound" {
name = "cross-cloud-inbound"
direction = "INBOUND"
security_group_ids = [aws_security_group.resolver.id]
# Two IP addresses in the private subnet for high availability
ip_address {
subnet_id = var.aws_subnet_id
ip = "10.0.1.10"
}
ip_address {
subnet_id = var.aws_subnet_id
ip = "10.0.1.11"
}
}
# Private hosted zone for AWS internal services
resource "aws_route53_zone" "internal_aws" {
name = "internal.aws"
vpc {
vpc_id = var.aws_vpc_id
}
}
# --- GCP Cloud DNS ---
# Forwarding zone: sends internal.aws queries to Route 53 Resolver over VPN
resource "google_dns_managed_zone" "forward_to_aws" {
name = "forward-to-aws"
dns_name = "internal.aws."
description = "Forwards DNS queries to AWS Route 53 Resolver over VPN"
visibility = "private"
private_visibility_config {
networks {
network_url = var.gcp_hub_network_id
}
}
forwarding_config {
target_name_servers {
ipv4_address = "10.0.1.10"
forwarding_path = "private"
}
target_name_servers {
ipv4_address = "10.0.1.11"
forwarding_path = "private"
}
}
}
# Private zone for GCP internal services
resource "google_dns_managed_zone" "internal_gcp" {
name = "internal-gcp"
dns_name = "internal.gcp."
description = "Private zone for GCP internal services"
visibility = "private"
private_visibility_config {
networks {
network_url = var.gcp_hub_network_id
}
}
}
# DNS peering: spoke VPC resolves names through the hub VPC
resource "google_dns_managed_zone" "spoke_peering" {
name = "spoke-dns-peering"
dns_name = "internal.gcp."
description = "DNS peering from spoke to hub for cross-cloud resolution"
visibility = "private"
private_visibility_config {
networks {
network_url = var.gcp_spoke_network_id
}
}
peering_config {
target_network {
network_url = var.gcp_hub_network_id
}
}
}Your cross-cloud DNS infrastructure is in place. Next up, you will deploy the backend API on AWS and register it in the internal.aws hosted zone so GCP services can discover it by name.
Deploy the Backend API on AWS
Your cross-cloud DNS is now resolving hostnames across the VPN tunnel. GCP instances can look up api.internal.aws and get a private IP back over the encrypted path.
Now you need something at that hostname to respond. In this step, you will deploy a Node.js product catalog API on an EC2 instance in the AWS private subnet. The instance has no public IP, so it is reachable only through the VPN tunnel.
In this step, get ready to:
- Write a Node.js API with health, latency, and product endpoints.
- Create the EC2 Terraform configuration with IAM role and Route 53 record.
- Deploy the instance and verify the API responds.
Write the product catalog API
The backend API is a lightweight Express server that exposes three endpoints. The EC2 instance will run this code automatically at boot via user_data.
- Create the application directory by running:
mkdir -p app/api- Create the API server file by running:
touch app/api/server.js- Confirm the file exists by running:
ls app/apiYou should see server.js in the output.
- Open app/api/server.js and add the following code:
const express = require('express');
const app = express();
const port = 3000;
// Health check endpoint - confirms the API is running
app.get('/health', (req, res) => {
res.json({ status: 'ok', timestamp: new Date().toISOString() });
});
// Product catalog endpoint - returns sample product data
app.get('/products', (req, res) => {
const products = [
{ id: 1, name: 'Widget', price: 9.99 },
{ id: 2, name: 'Gadget', price: 24.99 },
{ id: 3, name: 'Doohickey', price: 14.99 }
];
res.json(products);
});
// Latency measurement endpoint - reports round-trip time
app.get('/latency', (req, res) => {
const start = Date.now();
const latency = Date.now() - start;
res.json({ latency_ms: latency, source: 'aws-backend' });
});
// Start the server on all interfaces
app.listen(port, '0.0.0.0', () => {
console.log(`Product catalog API running on port ${port}`);
});What does this code do?
This Express server runs on port 3000 and exposes three GET endpoints.
- /health returns a JSON object with status: 'ok' and the current timestamp. The GCP frontend uses this to confirm the backend is reachable.
- /products returns a sample product catalog array. This simulates a real database-backed endpoint.
- /latency measures server-side processing time and reports source: 'aws-backend' so the frontend knows which cloud responded.
- Save app/api/server.js.
Create the EC2 Terraform configuration
The instance needs an IAM role for Systems Manager access so you can manage it without SSH or a public IP. The user_data script installs Node.js and starts the API at boot.
- Create the EC2 configuration file by running:
touch modules/aws-vpc/ec2.tf- Open modules/aws-vpc/ec2.tf and add the IAM role and instance profile:
# IAM role allowing EC2 to assume it
resource "aws_iam_role" "backend_role" {
name = "backend-ec2-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = "ec2.amazonaws.com"
}
}
]
})
}
# Attach SSM managed policy for remote management without SSH
resource "aws_iam_role_policy_attachment" "ssm_policy" {
role = aws_iam_role.backend_role.name
policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}
# Instance profile to attach the role to the EC2 instance
resource "aws_iam_instance_profile" "backend_profile" {
name = "backend-ec2-profile"
role = aws_iam_role.backend_role.name
}What does this code do?
The IAM role grants the EC2 instance permission to communicate with AWS Systems Manager. The AmazonSSMManagedInstanceCore managed policy enables Session Manager access.
This means you can connect to the instance from the AWS Console or CLI using aws ssm start-session without needing a bastion host, SSH keys, or a public IP address.
- Save ec2.tf.
- Below the instance profile in modules/aws-vpc/ec2.tf, add the AMI data source, EC2 instance, and Route 53 record:
# Look up the latest Amazon Linux 2 AMI
data "aws_ami" "amazon_linux_2" {
most_recent = true
owners = ["amazon"]
filter {
name = "name"
values = ["amzn2-ami-hvm-*-x86_64-gp2"]
}
}
# Backend API EC2 instance in the private subnet
resource "aws_instance" "backend" {
ami = data.aws_ami.amazon_linux_2.id
instance_type = "t3.micro"
subnet_id = aws_subnet.private.id
vpc_security_group_ids = [aws_security_group.cross_cloud.id]
iam_instance_profile = aws_iam_instance_profile.backend_profile.name
associate_public_ip_address = false
user_data = <<-EOF
#!/bin/bash
# Install Node.js 18 from NodeSource
curl -fsSL https://rpm.nodesource.com/setup_18.x | bash -
yum install -y nodejs
# Create application directory
mkdir -p /opt/api
cd /opt/api
# Initialize project and install Express
npm init -y
npm install express
# Write the product catalog API server
cat > server.js << 'APPEOF'
const express = require('express');
const app = express();
const port = 3000;
app.get('/health', (req, res) => {
res.json({ status: 'ok', timestamp: new Date().toISOString() });
});
app.get('/products', (req, res) => {
const products = [
{ id: 1, name: 'Widget', price: 9.99 },
{ id: 2, name: 'Gadget', price: 24.99 },
{ id: 3, name: 'Doohickey', price: 14.99 }
];
res.json(products);
});
app.get('/latency', (req, res) => {
const start = Date.now();
const latency = Date.now() - start;
res.json({ latency_ms: latency, source: 'aws-backend' });
});
app.listen(port, '0.0.0.0', () => {
console.log(`Product catalog API running on port ${port}`);
});
APPEOF
# Start the API server
node server.js &
EOF
tags = {
Name = "backend-api"
}
}
# Register the API hostname in the Route 53 private zone
resource "aws_route53_record" "api" {
zone_id = var.route53_zone_id
name = "api.internal.aws"
type = "A"
ttl = 300
records = [aws_instance.backend.private_ip]
}What does this code do?
The data "aws_ami" block dynamically finds the latest Amazon Linux 2 AMI so you never hardcode an AMI ID that could go stale.
The aws_instance resource launches a t3.micro in the private subnet with no public IP. The user_data script runs at first boot to install Node.js 18, create the Express app, and start it on port 3000.
The aws_route53_record creates an A record api.internal.aws pointing to the instance's private IP. GCP instances resolve this hostname through the DNS forwarding zone you set up earlier.
- Save modules/aws-vpc/ec2.tf.
Why SSM instead of SSH?
The instance has no public IP and lives in a private subnet. Traditional SSH would require a bastion host or VPN tunnel to your workstation.
SSM Session Manager lets you open a shell directly from the AWS Console or CLI. It authenticates through IAM, encrypts traffic with TLS, and requires zero inbound ports. This is the enterprise-standard approach for managing private instances.
✔️ Awesome, I've got everything!
Great. Double check you have saved modules/aws-vpc/ec2.tf with both the IAM resources and the EC2 instance.
ⓧ I'd like to double check the full code
# IAM role allowing EC2 to assume it
resource "aws_iam_role" "backend_role" {
name = "backend-ec2-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = "ec2.amazonaws.com"
}
}
]
})
}
# Attach SSM managed policy for remote management without SSH
resource "aws_iam_role_policy_attachment" "ssm_policy" {
role = aws_iam_role.backend_role.name
policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}
# Instance profile to attach the role to the EC2 instance
resource "aws_iam_instance_profile" "backend_profile" {
name = "backend-ec2-profile"
role = aws_iam_role.backend_role.name
}
# Look up the latest Amazon Linux 2 AMI
data "aws_ami" "amazon_linux_2" {
most_recent = true
owners = ["amazon"]
filter {
name = "name"
values = ["amzn2-ami-hvm-*-x86_64-gp2"]
}
}
# Backend API EC2 instance in the private subnet
resource "aws_instance" "backend" {
ami = data.aws_ami.amazon_linux_2.id
instance_type = "t3.micro"
subnet_id = aws_subnet.private.id
vpc_security_group_ids = [aws_security_group.cross_cloud.id]
iam_instance_profile = aws_iam_instance_profile.backend_profile.name
associate_public_ip_address = false
user_data = <<-EOF
#!/bin/bash
# Install Node.js 18 from NodeSource
curl -fsSL https://rpm.nodesource.com/setup_18.x | bash -
yum install -y nodejs
# Create application directory
mkdir -p /opt/api
cd /opt/api
# Initialize project and install Express
npm init -y
npm install express
# Write the product catalog API server
cat > server.js << 'APPEOF'
const express = require('express');
const app = express();
const port = 3000;
app.get('/health', (req, res) => {
res.json({ status: 'ok', timestamp: new Date().toISOString() });
});
app.get('/products', (req, res) => {
const products = [
{ id: 1, name: 'Widget', price: 9.99 },
{ id: 2, name: 'Gadget', price: 24.99 },
{ id: 3, name: 'Doohickey', price: 14.99 }
];
res.json(products);
});
app.get('/latency', (req, res) => {
const start = Date.now();
const latency = Date.now() - start;
res.json({ latency_ms: latency, source: 'aws-backend' });
});
app.listen(port, '0.0.0.0', () => {
console.log(`Product catalog API running on port ${port}`);
});
APPEOF
# Start the API server
node server.js &
EOF
tags = {
Name = "backend-api"
}
}
# Register the API hostname in the Route 53 private zone
resource "aws_route53_record" "api" {
zone_id = var.route53_zone_id
name = "api.internal.aws"
type = "A"
ttl = 300
records = [aws_instance.backend.private_ip]
}Deploy and verify the backend API
With the Terraform configuration in place, you can now deploy the EC2 instance and confirm the API is running.
- Initialize and apply the updated Terraform configuration by running:
terraform init
terraform apply- Terraform will show you the planned changes. Review the plan and type yes when prompted to confirm.
What should I see?
Terraform creates the IAM role, instance profile, EC2 instance, and Route 53 A record. The instance takes 2-3 minutes to fully boot and start the API because user_data installs Node.js from scratch.
- Wait 2-3 minutes for the instance to boot, then verify the API health endpoint from within the VPC. Connect to the instance using SSM Session Manager by running:
aws ssm start-session --target $(terraform output -raw backend_instance_id)- Once connected, test the health endpoint by running:
curl http://localhost:3000/healthYou should see a JSON response like {"status":"ok","timestamp":"2026-06-06T..."}.
- Test the products endpoint by running:
curl http://localhost:3000/productsYou should see the product catalog array with Widget, Gadget, and Doohickey.
- Exit the SSM session by typing exit.
- Verify the Route 53 record was created by running:
aws route53 list-resource-record-sets --hosted-zone-id $(terraform output -raw route53_zone_id) --query "ResourceRecordSets[?Name=='api.internal.aws.']"You should see an A record pointing to the EC2 instance's private IP address in the 10.0.1.0/24 subnet.
SSM session not connecting?
The instance needs 2-3 minutes after launch for the SSM agent to register. If you get a TargetNotConnected error, wait and try again.
If the instance never appears in SSM, confirm the IAM instance profile is attached and the instance has outbound internet access (or VPC endpoints for SSM). The security group must allow outbound HTTPS on port 443.
help me troubleshoot why my EC2 instance is not connecting to SSM Session Manager in a private subnet
Health endpoint returning connection refused?
The user_data script may still be running. Check the install log on the instance:
cat /var/log/cloud-init-output.log | tail -20Look for errors related to Node.js installation or npm install express failures. The instance needs outbound internet access to download packages.
help me debug why my Node.js API is not starting on the EC2 instance after user_data runs
Your backend API is live on the private subnet and registered in DNS. Next, you will deploy the GCP frontend that discovers this API by resolving api.internal.aws over the VPN tunnel.
Deploy the Frontend on GCP with DNS-Based Discovery
Your AWS backend is running in its private subnet, responding to health checks and serving product data. Now you need a frontend on GCP that can discover and communicate with that backend by hostname, not by hardcoded IP address.
In this step, you will deploy a Compute Engine instance running nginx in the spoke VPC. The instance has no external IP. It resolves api.internal.aws through the cross-cloud DNS forwarding you configured earlier, then proxies requests to the AWS backend over the encrypted VPN tunnels.
In this step, get ready to:
- Configure Cloud NAT so the instance can download packages without an external IP.
- Deploy a Compute Engine instance with nginx proxying to the AWS backend via DNS-based discovery.
- Verify end-to-end connectivity where the GCP frontend calls the AWS backend by hostname over VPN.
Configure Cloud NAT for outbound internet access
Your frontend instance will have no external IP address. Without Cloud NAT, it cannot reach the internet to install packages like nginx. Cloud NAT provides outbound internet access for instances that lack public IPs.
- Create a new file called compute.tf inside the modules/gcp-vpc/ directory.
- Add the Cloud NAT resource by pasting this code:
# Cloud NAT for outbound internet access (package installation)
resource "google_compute_router_nat" "hub_nat" {
name = "hub-nat"
router = var.cloud_router_name
region = var.region
nat_ip_allocate_option = "AUTO_ONLY"
source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"
}What does this code do?
This creates a Cloud NAT gateway on the hub VPC's Cloud Router. It automatically allocates external IPs and applies NAT to all subnetworks in the VPC.
The var.cloud_router_name references the Cloud Router you created in the VPN module. The AUTO_ONLY option lets GCP manage the NAT IP addresses automatically.
Deploy the Compute Engine frontend with nginx
The frontend instance runs in the spoke VPC with no external IP. Its startup script installs nginx and configures it to proxy API requests to api.internal.aws:3000, which resolves over the cross-cloud DNS forwarding path you built in the previous step.
- First, create the nginx configuration file. Create a new directory app/frontend/ in your repository and add the following file:
# nginx reverse proxy configuration for cross-cloud API discovery
events {
worker_connections 1024;
}
http {
# Use GCP internal DNS resolver for hostname resolution
resolver 169.254.169.254;
server {
listen 80;
# Proxy API requests to AWS backend via cross-cloud DNS
location /api/ {
proxy_pass http://api.internal.aws:3000/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
# Measure cross-cloud round-trip time
location /latency {
proxy_pass http://api.internal.aws:3000/health;
add_header X-Cross-Cloud-RTT $upstream_response_time always;
}
# Frontend health check
location /health {
return 200 '{"status": "ok", "service": "gcp-frontend"}';
add_header Content-Type application/json;
}
# Default page
location / {
return 200 'Multi-Cloud Frontend - GCP\n';
add_header Content-Type text/plain;
}
}
}How does DNS-based discovery work here?
The resolver 169.254.169.254 directive tells nginx to use the GCP metadata DNS resolver. When nginx resolves api.internal.aws, the query flows through Cloud DNS to the forwarding zone, which sends it over the VPN to the Route 53 Resolver inbound endpoint on AWS.
The /latency endpoint proxies to the backend's /health endpoint and adds the $upstream_response_time header. This variable captures how long the round trip to AWS took, giving you cross-cloud latency measurement.
- Now add the Compute Engine instance resource to modules/gcp-vpc/compute.tf below the Cloud NAT resource:
# Frontend Compute Engine instance in spoke VPC
resource "google_compute_instance" "frontend" {
name = "gcp-frontend"
machine_type = "e2-micro"
zone = "${var.region}-a"
boot_disk {
initialize_params {
image = "debian-cloud/debian-11"
}
}
network_interface {
subnetwork = google_compute_subnetwork.spoke.id
# No access_config block = no external IP
}
metadata_startup_script = <<-EOF
#!/bin/bash
apt-get update
apt-get install -y nginx
# Configure nginx to proxy to AWS backend over VPN
cat > /etc/nginx/nginx.conf << 'NGINX'
events {
worker_connections 1024;
}
http {
resolver 169.254.169.254;
server {
listen 80;
location /api/ {
proxy_pass http://api.internal.aws:3000/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
location /latency {
proxy_pass http://api.internal.aws:3000/health;
add_header X-Cross-Cloud-RTT $upstream_response_time always;
}
location /health {
return 200 '{"status": "ok", "service": "gcp-frontend"}';
add_header Content-Type application/json;
}
location / {
return 200 'Multi-Cloud Frontend - GCP\n';
add_header Content-Type text/plain;
}
}
}
NGINX
systemctl restart nginx
EOF
tags = ["frontend"]
}What does this code do?
This creates an e2-micro instance in the spoke VPC subnet with no external IP (notice the missing access_config block). The startup script installs nginx and writes the proxy configuration inline.
The instance uses the spoke VPC's subnet (10.2.0.0/24), meaning all traffic to the AWS backend (10.0.0.0/16) routes through the VPC peering to the hub, then over the VPN tunnels.
- Add the DNS record resource below the instance to register the frontend hostname:
# DNS record for frontend discovery
resource "google_dns_record_set" "frontend" {
name = "frontend.internal.gcp."
managed_zone = var.dns_zone_name
type = "A"
ttl = 300
rrdatas = [google_compute_instance.frontend.network_interface[0].network_ip]
}What does this code do?
This creates an A record frontend.internal.gcp pointing to the Compute Engine instance's private IP. Other services in the network can now discover the frontend by hostname instead of a hardcoded IP.
- Save modules/gcp-vpc/compute.tf and app/frontend/nginx.conf.
✔️ Awesome, I've got everything!
Great. Double check you have saved both files before continuing.
ⓧ I'd like to double check the full code
Here is the complete modules/gcp-vpc/compute.tf file:
# Cloud NAT for outbound internet access (package installation)
resource "google_compute_router_nat" "hub_nat" {
name = "hub-nat"
router = var.cloud_router_name
region = var.region
nat_ip_allocate_option = "AUTO_ONLY"
source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"
}
# Frontend Compute Engine instance in spoke VPC
resource "google_compute_instance" "frontend" {
name = "gcp-frontend"
machine_type = "e2-micro"
zone = "${var.region}-a"
boot_disk {
initialize_params {
image = "debian-cloud/debian-11"
}
}
network_interface {
subnetwork = google_compute_subnetwork.spoke.id
# No access_config block = no external IP
}
metadata_startup_script = <<-EOF
#!/bin/bash
apt-get update
apt-get install -y nginx
# Configure nginx to proxy to AWS backend over VPN
cat > /etc/nginx/nginx.conf << 'NGINX'
events {
worker_connections 1024;
}
http {
resolver 169.254.169.254;
server {
listen 80;
location /api/ {
proxy_pass http://api.internal.aws:3000/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
location /latency {
proxy_pass http://api.internal.aws:3000/health;
add_header X-Cross-Cloud-RTT $upstream_response_time always;
}
location /health {
return 200 '{"status": "ok", "service": "gcp-frontend"}';
add_header Content-Type application/json;
}
location / {
return 200 'Multi-Cloud Frontend - GCP\n';
add_header Content-Type text/plain;
}
}
}
NGINX
systemctl restart nginx
EOF
tags = ["frontend"]
}
# DNS record for frontend discovery
resource "google_dns_record_set" "frontend" {
name = "frontend.internal.gcp."
managed_zone = var.dns_zone_name
type = "A"
ttl = 300
rrdatas = [google_compute_instance.frontend.network_interface[0].network_ip]
}Seeing variable errors?
Make sure your modules/gcp-vpc/variables.tf file declares var.cloud_router_name, var.region, and var.dns_zone_name. These values are passed from the environment composition in environments/dev/main.tf.
help me debug Terraform variable errors in my GCP VPC module
Apply and verify end-to-end cross-cloud connectivity
Time to deploy the frontend and prove that cross-cloud DNS-based discovery works. The GCP instance will resolve api.internal.aws through the DNS forwarding chain and reach the AWS backend over the VPN.
- Deploy the new infrastructure by running:
cd environments/dev
terraform apply- Terraform will prompt you to approve the plan. Review the new resources (Cloud NAT, Compute Engine instance, DNS record) and enter yes.
- Wait for the instance to finish provisioning. The startup script needs a few minutes to install nginx.
- Create a temporary firewall rule for IAP SSH access so you can reach the instance without an external IP:
gcloud compute firewall-rules create allow-ssh-iap-spoke \
--network=spoke-vpc \
--direction=INGRESS \
--action=allow \
--rules=tcp:22 \
--source-ranges=35.235.240.0/20What is IAP tunneling?
Identity-Aware Proxy (IAP) lets you SSH into instances that have no external IP. GCP routes your SSH connection through its 35.235.240.0/20 IP range, so you need a firewall rule allowing that range on port 22.
- SSH into the frontend instance using IAP tunneling:
gcloud compute ssh gcp-frontend \
--zone=us-central1-a \
--tunnel-through-iap- Once connected, test DNS resolution of the AWS backend hostname:
nslookup api.internal.aws- You should see the private IP of the AWS EC2 instance (in the 10.0.1.0/24 range) returned by the DNS query. This proves the cross-cloud DNS forwarding chain works.
- Test the nginx proxy to the AWS backend:
curl http://localhost/api/products- You should see the product catalog JSON response from the AWS backend.
- Test the cross-cloud latency measurement endpoint:
curl -v http://localhost/latency- Look for the X-Cross-Cloud-RTT header in the response. This value (in seconds) represents the round-trip time from GCP to AWS over the VPN tunnel.
- Test the frontend health check:
curl http://localhost/health- You should see {"status": "ok", "service": "gcp-frontend"} confirming nginx is running correctly.
- Exit the SSH session by typing exit.
DNS resolution failing?
If nslookup api.internal.aws fails, check that the VPN tunnels are up and that the DNS forwarding zone targets the correct Route 53 Resolver inbound IPs. Also verify the Cloud DNS forwarding zone has the 35.199.192.0/19 route advertised via the Cloud Router for DNS response routing.
help me troubleshoot cross-cloud DNS resolution from GCP to AWS over VPN
Your distributed application is now communicating across clouds using DNS-based service discovery over private encrypted tunnels. Next up, you will configure Terraform remote state and set up environment separation for production-readiness.
Configure Terraform Remote State and Environment Separation
Your infrastructure is now fully deployed across both clouds with end-to-end connectivity verified. But right now, your Terraform state lives in a local file on your machine.
If that file gets deleted or corrupted, you lose all knowledge of what Terraform manages. In this step, you will move your state to a remote GCS bucket and set up environment separation so the same codebase can deploy both dev and prod infrastructure without conflicts.
In this step, get ready to:
- Create a GCS bucket for Terraform state with versioning enabled.
- Configure the Terraform backend and environment-specific tfvars files.
- Migrate your local state to remote and write ADR #2.
Create the GCS state bucket with versioning
Terraform remote state needs a pre-existing storage bucket. The bucket stores your .tfstate file remotely so your team (and your CI/CD pipeline) can access it.
- Create the GCS bucket by running this command:
gcloud storage buckets create gs://tf-state-multicloud-[[GCP_PROJECT_ID="your-gcp-project-id"]] --location=US- Enable versioning on the bucket so you can recover previous state versions:
gcloud storage buckets update gs://tf-state-multicloud-[[GCP_PROJECT_ID="your-gcp-project-id"]] --versioningWhy enable versioning on the state bucket?
Versioning keeps a history of every state file upload. If a bad terraform apply corrupts your state, you can roll back to a previous version instead of rebuilding from scratch.
- Verify the bucket was created with versioning by running:
gcloud storage buckets describe gs://tf-state-multicloud-[[GCP_PROJECT_ID="your-gcp-project-id"]] --format="default(versioning_enabled)"You should see versioning_enabled: True in the output.
Bucket creation failed?
GCS bucket names must be globally unique. If you see a conflict error, your project ID appended to tf-state-multicloud- should make it unique. Double-check you are using your actual GCP project ID (not the project name).
help me fix my GCS bucket creation error
Configure backend and environment tfvars
Terraform backends cannot use variables or expressions. The standard pattern is to use a partial backend configuration in your code and supply the environment-specific prefix via -backend-config during initialization.
- Create a file called backend.tf in the root of your repository with this backend configuration:
# backend.tf - Remote state configuration using GCS
# The prefix is supplied via -backend-config during terraform init
# to separate dev and prod state files in the same bucket
terraform {
backend "gcs" {
bucket = "tf-state-multicloud-[[GCP_PROJECT_ID="your-gcp-project-id"]]"
}
}What does this configuration do?
The backend "gcs" block tells Terraform to store state in your GCS bucket instead of a local file. The prefix is intentionally omitted here because you will supply it at init time.
This lets you run terraform init -backend-config="prefix=environments/dev" for dev and terraform init -backend-config="prefix=environments/prod" for prod, keeping state files completely isolated.
- Create the dev environment tfvars file at environments/dev/terraform.tfvars:
# environments/dev/terraform.tfvars - Development environment values
# Smaller CIDRs and instance sizes to minimize cost during development
environment = "dev"
aws_vpc_cidr = "10.0.0.0/16"
gcp_hub_cidr = "10.1.0.0/24"
gcp_spoke_cidr = "10.2.0.0/24"
aws_instance_type = "t3.micro"
gcp_instance_type = "e2-micro"- Create the prod environment tfvars file at environments/prod/terraform.tfvars:
# environments/prod/terraform.tfvars - Production environment values
# Larger CIDRs and beefier instances for production workloads
environment = "prod"
aws_vpc_cidr = "10.10.0.0/16"
gcp_hub_cidr = "10.11.0.0/24"
gcp_spoke_cidr = "10.12.0.0/24"
aws_instance_type = "t3.small"
gcp_instance_type = "e2-small"Why separate CIDR ranges for prod?
Production uses a completely different IP address space (10.10.0.0/16, 10.11.0.0/24, 10.12.0.0/24) so that dev and prod VPCs never have overlapping addresses. This means you could connect both environments to the same Transit Gateway without routing conflicts.
- Save both files.
- Verify the files exist by running:
ls environments/dev/terraform.tfvars environments/prod/terraform.tfvarsYou should see both file paths listed in the output.
File not found?
Make sure you are running the ls command from your repository root. The environments/dev and environments/prod directories should already exist from your initial repository setup in Step 1.
help me find my tfvars files
Migrate local state to remote and write ADR #2
With the bucket and backend configuration ready, you can now migrate your existing local state to GCS. Terraform handles this automatically when you re-initialize with the new backend.
- Initialize Terraform with the new GCS backend and the dev environment prefix by running:
terraform init -backend-config="prefix=environments/dev" -migrate-stateWhat happens during migration?
Terraform detects that you are changing from the local backend to GCS. It asks whether you want to copy your existing state to the new backend. Type yes when prompted.
After migration, your local terraform.tfstate file becomes empty and all state is stored in GCS at the path environments/dev/default.tfstate.
- When Terraform asks if you want to copy existing state to the new backend, type yes and press Enter.
- Verify the migration succeeded by running:
terraform planYou should see no changes planned. This confirms Terraform is reading state from GCS and recognizes all existing resources.
Seeing unexpected changes in the plan?
If Terraform wants to recreate resources, the migration may have failed silently. Run terraform state list to check if your resources appear. If empty, re-run the init command and confirm the copy prompt with yes.
help me debug my state migration issue
Now write ADR #2 to document your environment separation strategy.
- Create the file docs/adr/0002-environment-separation-strategy.md with this content:
# ADR #2: Environment Separation Strategy
## Status
Accepted
## Context
We need to deploy the same multi-cloud infrastructure to dev and prod
environments without state conflicts or CIDR collisions.
## Decision
We use a single GCS bucket with environment-specific path prefixes
(environments/dev/, environments/prod/) for state isolation. Each
environment has its own terraform.tfvars file defining CIDR ranges
and instance sizes. The backend prefix is supplied at init time via
-backend-config so the same Terraform code serves both environments.
## Consequences
- State is isolated per environment in the same bucket (simple billing).
- CIDR ranges are non-overlapping, enabling future cross-env connectivity.
- No code duplication: a single set of modules handles all environments.
- Developers must remember to pass the correct -backend-config at init.
- Workspaces are not used, avoiding implicit state-path assumptions.What is an Architecture Decision Record?
ADRs document key design choices and their rationale. In interviews, ADRs demonstrate that you think beyond just making things work. They show you consider trade-offs, alternatives, and long-term consequences.
- Save docs/adr/0002-environment-separation-strategy.md.
✔️ Awesome, I've got everything!
Great. Double check you have saved backend.tf, both tfvars files, and the ADR.
ⓧ I'd like to double check the full code
Here is the full backend.tf file:
# backend.tf - Remote state configuration using GCS
# The prefix is supplied via -backend-config during terraform init
# to separate dev and prod state files in the same bucket
terraform {
backend "gcs" {
bucket = "tf-state-multicloud-[[GCP_PROJECT_ID="your-gcp-project-id"]]"
}
}Dev tfvars at environments/dev/terraform.tfvars:
# environments/dev/terraform.tfvars - Development environment values
# Smaller CIDRs and instance sizes to minimize cost during development
environment = "dev"
aws_vpc_cidr = "10.0.0.0/16"
gcp_hub_cidr = "10.1.0.0/24"
gcp_spoke_cidr = "10.2.0.0/24"
aws_instance_type = "t3.micro"
gcp_instance_type = "e2-micro"Prod tfvars at environments/prod/terraform.tfvars:
# environments/prod/terraform.tfvars - Production environment values
# Larger CIDRs and beefier instances for production workloads
environment = "prod"
aws_vpc_cidr = "10.10.0.0/16"
gcp_hub_cidr = "10.11.0.0/24"
gcp_spoke_cidr = "10.12.0.0/24"
aws_instance_type = "t3.small"
gcp_instance_type = "e2-small"ADR at docs/adr/0002-environment-separation-strategy.md:
# ADR #2: Environment Separation Strategy
## Status
Accepted
## Context
We need to deploy the same multi-cloud infrastructure to dev and prod
environments without state conflicts or CIDR collisions.
## Decision
We use a single GCS bucket with environment-specific path prefixes
(environments/dev/, environments/prod/) for state isolation. Each
environment has its own terraform.tfvars file defining CIDR ranges
and instance sizes. The backend prefix is supplied at init time via
-backend-config so the same Terraform code serves both environments.
## Consequences
- State is isolated per environment in the same bucket (simple billing).
- CIDR ranges are non-overlapping, enabling future cross-env connectivity.
- No code duplication: a single set of modules handles all environments.
- Developers must remember to pass the correct -backend-config at init.
- Workspaces are not used, avoiding implicit state-path assumptions.Your Terraform state is now safely stored in GCS with versioning protection and environment-specific paths. Next up, you will configure zero-trust authentication so your CI/CD pipeline can deploy to both clouds without any stored secrets.
Set Up Zero-Trust CI/CD Authentication
Your infrastructure is now stored in remote state with environment separation. The next challenge is deploying it automatically without storing any credentials in your CI/CD system.
Traditional CI/CD relies on long-lived access keys stored as secrets. Zero-trust authentication uses OIDC federation instead, where your pipeline proves its identity through short-lived tokens that each cloud verifies independently. In this step, you will configure both GCP and AWS to trust GitHub Actions as an identity provider.
In this step, get ready to:
- Create GCP Workload Identity Federation for GitHub Actions.
- Create an AWS IAM OIDC provider and role scoped to your repository.
- Test authentication from a GitHub Actions workflow to both clouds.
Create GCP Workload Identity Federation and service account
Workload Identity Federation lets external workloads (like GitHub Actions runners) authenticate to GCP without a service account key. You create a pool and provider that map GitHub's OIDC tokens to a GCP identity, then bind that identity to a service account with the roles your pipeline needs.
- Create a new module directory for IAM resources by running these commands:
mkdir modules/iam- Create a file called gcp-wif.tf inside modules/iam and add the following Terraform configuration:
# Workload Identity Pool for GitHub Actions
resource "google_iam_workload_identity_pool" "github" {
workload_identity_pool_id = "github-pool"
display_name = "GitHub Actions Pool"
description = "Identity pool for GitHub Actions CI/CD"
}
# OIDC Provider within the pool
resource "google_iam_workload_identity_pool_provider" "github" {
workload_identity_pool_id = google_iam_workload_identity_pool.github.workload_identity_pool_id
workload_identity_pool_provider_id = "github-provider"
display_name = "GitHub OIDC Provider"
# Map GitHub token claims to Google attributes
attribute_mapping = {
"google.subject" = "assertion.sub"
"attribute.repository" = "assertion.repository"
}
oidc {
issuer_uri = "https://token.actions.githubusercontent.com"
}
}
# Service account for GitHub Actions to impersonate
resource "google_service_account" "github_actions" {
account_id = "github-actions-sa"
display_name = "GitHub Actions Service Account"
}
# Grant minimum required roles to the service account
resource "google_project_iam_member" "compute_admin" {
project = var.gcp_project_id
role = "roles/compute.admin"
member = "serviceAccount:${google_service_account.github_actions.email}"
}
resource "google_project_iam_member" "dns_admin" {
project = var.gcp_project_id
role = "roles/dns.admin"
member = "serviceAccount:${google_service_account.github_actions.email}"
}
resource "google_project_iam_member" "sa_user" {
project = var.gcp_project_id
role = "roles/iam.serviceAccountUser"
member = "serviceAccount:${google_service_account.github_actions.email}"
}
# Allow GitHub Actions from your specific repo to impersonate this SA
resource "google_service_account_iam_binding" "workload_identity" {
service_account_id = google_service_account.github_actions.name
role = "roles/iam.workloadIdentityUser"
members = [
"principalSet://iam.googleapis.com/${google_iam_workload_identity_pool.github.name}/attribute.repository/${var.github_repo}"
]
}What does this code do?
The Workload Identity Pool (github-pool) is a container that holds external identity providers. The OIDC Provider (github-provider) tells GCP to trust tokens issued by https://token.actions.githubusercontent.com.
The attribute_mapping maps claims from GitHub's JWT to attributes GCP can use for access control. The google.subject=assertion.sub mapping is the minimum required by GCP.
The service account binding scopes access to your specific repository using attribute.repository. Only workflows running in that repo can impersonate this service account.
- Save modules/iam/gcp-wif.tf.
- Run terraform validate to confirm the syntax is correct.
Seeing a variable not found error?
Make sure you have var.gcp_project_id and var.github_repo defined in your variables file. The github_repo variable should be in the format your-org/your-repo.
Help me debug my Workload Identity Federation Terraform configuration.
Why Workload Identity Federation instead of a service account key?
Service account keys are long-lived credentials that can be stolen and reused. Workload Identity Federation issues short-lived tokens (valid for about 1 hour) that cannot be reused outside the specific GitHub Actions workflow run. If your repository is compromised, the attacker cannot extract a persistent credential.
This is the zero-trust approach: verify identity on every request rather than trusting a stored secret.
Create AWS IAM OIDC provider and role
The AWS side follows the same principle. You register GitHub's OIDC issuer as a trusted identity provider in IAM, then create a role with a trust policy that only allows your specific repository to assume it.
- Create a file called aws-oidc.tf inside modules/iam and add the following configuration:
# OIDC Identity Provider for GitHub Actions
# AWS validates GitHub's tokens using its own trusted CA library
resource "aws_iam_openid_connect_provider" "github" {
url = "https://token.actions.githubusercontent.com"
client_id_list = ["sts.amazonaws.com"]
}
# IAM Role that GitHub Actions will assume
resource "aws_iam_role" "github_actions" {
name = "GitHubActionsRole"
# Trust policy scoped to your specific repository
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Principal = {
Federated = aws_iam_openid_connect_provider.github.arn
}
Action = "sts:AssumeRoleWithWebIdentity"
Condition = {
StringLike = {
"token.actions.githubusercontent.com:sub" = "repo:${var.github_repo}:*"
}
StringEquals = {
"token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
}
}
}
]
})
}
# Policy granting permissions for EC2, VPC, Route53, and Terraform state
resource "aws_iam_role_policy" "github_actions" {
name = "GitHubActionsPolicy"
role = aws_iam_role.github_actions.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Action = [
"ec2:*",
"route53:*",
"route53resolver:*"
]
Resource = "*"
},
{
Effect = "Allow"
Action = [
"s3:GetObject",
"s3:PutObject",
"s3:ListBucket"
]
Resource = [
"arn:aws:s3:::tf-state-multicloud-*",
"arn:aws:s3:::tf-state-multicloud-*/*"
]
}
]
})
}What does this code do?
The aws_iam_openid_connect_provider registers GitHub's token issuer (token.actions.githubusercontent.com) with AWS. The audience sts.amazonaws.com tells AWS this provider is used for STS role assumption. AWS validates GitHub's certificates using its own trusted CA library, so no thumbprint configuration is needed.
The trust policy has two conditions. StringLike on the sub claim restricts access to your specific repository. StringEquals on the aud claim ensures the token was issued for AWS specifically. Together, these prevent any other repository or service from assuming this role.
The S3 permissions are scoped to your Terraform state bucket pattern (tf-state-multicloud-*) so the pipeline can read and write remote state.
- Save modules/iam/aws-oidc.tf.
- Run terraform validate to confirm both IAM module files are syntactically correct.
Seeing a provider error?
Make sure your root module references both the aws and google providers. The IAM module uses resources from both clouds. Also ensure var.github_repo is declared (format: owner/repo-name).
Help me fix provider errors in my IAM module.
- Apply the IAM module to both clouds by running:
terraform apply -var-file=environments/dev/terraform.tfvars- Review the plan output carefully. You should see new resources for the Workload Identity Pool, OIDC providers, service account, role bindings, and the AWS IAM role.
- Type yes to confirm the apply.
What should I see?
The apply should create approximately 8-10 new resources across both clouds. Look for successful creation of google_iam_workload_identity_pool.github, google_service_account.github_actions, and aws_iam_role.github_actions in the output.
Test authentication from GitHub Actions
With both identity providers configured, you need to verify that a GitHub Actions workflow can actually authenticate to both clouds using zero stored secrets.
- Create the workflow directory by running:
mkdir -p .github/workflows- Create a file called auth-test.yml inside .github/workflows and add the following workflow:
name: Test OIDC Authentication
on:
workflow_dispatch:
jobs:
test-auth:
runs-on: ubuntu-latest
# Required for OIDC token generation
permissions:
contents: read
id-token: write
steps:
- uses: actions/checkout@v4
# Authenticate to GCP via Workload Identity Federation
- id: gcp-auth
name: Authenticate to Google Cloud
uses: google-github-actions/auth@v3
with:
workload_identity_provider: projects/${{ vars.GCP_PROJECT_NUMBER }}/locations/global/workloadIdentityPools/github-pool/providers/github-provider
service_account: github-actions-sa@${{ vars.GCP_PROJECT_ID }}.iam.gserviceaccount.com
# Verify GCP authentication works
- name: Verify GCP Access
run: gcloud compute instances list --format="table(name,zone,status)" --limit=5
# Authenticate to AWS via OIDC
- name: Authenticate to AWS
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/GitHubActionsRole
aws-region: us-east-1
# Verify AWS authentication works
- name: Verify AWS Access
run: aws sts get-caller-identityWhat does this workflow do?
The permissions: id-token: write setting enables GitHub to issue OIDC tokens for this workflow. Without it, the authentication steps will fail because no token can be generated.
The google-github-actions/auth@v3 action exchanges GitHub's OIDC token for GCP credentials via Workload Identity Federation. The aws-actions/configure-aws-credentials@v4 action does the same for AWS, assuming the GitHubActionsRole via sts:AssumeRoleWithWebIdentity.
Notice there are zero secrets referenced. The workflow uses vars (repository variables) for non-sensitive identifiers like project IDs and account numbers.
- Save .github/workflows/auth-test.yml.
- In your GitHub repository settings, navigate to Settings then Secrets and variables then Actions.
- Click the Variables tab and add three repository variables:
- Add GCP_PROJECT_NUMBER with your GCP project number (numeric ID, not the project ID string).
- Add GCP_PROJECT_ID with your GCP project ID string.
- Add AWS_ACCOUNT_ID with your 12-digit AWS account number.
Find your GCP project number
Your project number is different from your project ID. Run gcloud projects describe YOUR_PROJECT_ID --format="value(projectNumber)" to get the numeric project number. The Workload Identity Provider path requires the number, not the ID.
- Commit and push all the changes to your repository by running:
git add .
git commit -m "Add zero-trust OIDC authentication for CI/CD"
git push origin main- In your GitHub repository, click the Actions tab.
- Select Test OIDC Authentication from the left sidebar.
- Click Run workflow and then Run workflow again in the dropdown.
- Wait for the workflow to complete. Both the GCP and AWS verification steps should show green checkmarks.
What should I see?
The Verify GCP Access step should list your Compute Engine instances (or an empty table if none exist in the default project view). The Verify AWS Access step should output your assumed role ARN containing GitHubActionsRole and your AWS account ID. Both steps completing without errors confirms zero-trust authentication is working.
Workflow failing on the GCP auth step?
The most common cause is a mismatch between your GCP_PROJECT_NUMBER variable and the actual project where you created the Workload Identity Pool. Double-check you used the numeric project number, not the project ID string.
Also verify the repository attribute in your google_service_account_iam_binding matches your GitHub repo exactly (case-sensitive, format owner/repo-name).
Help me debug my GitHub Actions OIDC authentication workflow.
Both clouds now trust your GitHub repository as an identity provider. Next up, you will build the full CI/CD pipeline that uses this authentication to plan, scan, and apply your infrastructure automatically.
Build the CI/CD Pipeline with Security Scanning and Cost Estimation
In the previous step, you verified that GitHub Actions can authenticate to both AWS and GCP using OIDC federation with zero stored secrets. That was the foundation.
Now you will build the full CI/CD pipeline that enforces quality gates on every change. The pipeline runs format checks, security scanning with tfsec, cost estimation with Infracost, and environment-gated Terraform plan/apply.
In this step, get ready to:
- Create a GitHub Actions workflow with validation, security, and cost estimation jobs.
- Test the full pipeline end-to-end with a minor infrastructure change.
- Write ADR #3 documenting your CI/CD pipeline design decisions.
Create the GitHub Actions workflow
A production CI/CD pipeline does more than just run terraform apply. It validates formatting, scans for security misconfigurations, estimates cost impact, and gates deployments by environment.
- Create the workflow file by running these commands from your repository root:
mkdir -p .github/workflows
touch .github/workflows/terraform.yml- Confirm the file exists by running:
ls .github/workflows/You should see terraform.yml in the output.
- Open .github/workflows/terraform.yml in your editor and add the workflow triggers and permissions:
# CI/CD pipeline for multi-cloud Terraform infrastructure
name: Terraform CI/CD
# Run on PRs and pushes to main
on:
push:
branches: [main]
pull_request:
branches: [main]
# OIDC requires id-token write; pull-requests write for Infracost comments
permissions:
id-token: write
contents: read
pull-requests: writeWhat does this code do?
The on block triggers the workflow on pull requests targeting main (for plan and review) and on pushes to main (for apply).
The permissions block grants id-token: write so the workflow can request OIDC tokens for both clouds. pull-requests: write allows Infracost to post cost comments on PRs.
- Below the permissions block, add the validate job:
jobs:
# Job 1: Format and validate Terraform code
validate:
name: Terraform Validate
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Terraform
uses: hashicorp/setup-terraform@v3
- name: Terraform Format Check
run: terraform fmt -check -recursive
- name: Terraform Init
run: terraform init -backend=false
- name: Terraform Validate
run: terraform validateWhat does this code do?
The validate job catches formatting drift and syntax errors before anything else runs. terraform fmt -check fails the job if any file is not canonically formatted.
The -backend=false flag on init skips backend configuration so the job can validate without cloud credentials.
- Below the validate job, add the security scanning job:
# Job 2: Security scanning with tfsec
security:
name: Security Scan
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Run tfsec
uses: aquasecurity/tfsec-action@v1.0.0
with:
format: sarifWhat does this code do?
The security job uses tfsec to scan your Terraform code for common misconfigurations like overly permissive security groups, missing encryption, or public access on private resources.
The format: sarif output produces a structured format that integrates with GitHub's code scanning features. If tfsec finds issues, the job fails and blocks the PR from merging.
- Below the security job, add the cost estimation job:
# Job 3: Cost estimation with Infracost
cost:
name: Cost Estimation
runs-on: ubuntu-latest
if: github.event_name == 'pull_request'
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Infracost
uses: infracost/actions/setup@v3
with:
api-key: ${{ secrets.INFRACOST_API_KEY }}
- name: Generate cost breakdown
run: infracost breakdown --path=. --format=json --out-file=/tmp/infracost.json
- name: Post cost comment
uses: infracost/actions/comment@v1
with:
path: /tmp/infracost.json
behavior: updateWhat does this code do?
The cost job only runs on pull requests (controlled by the if condition). It uses Infracost to estimate the monthly cost impact of your infrastructure changes.
The infracost breakdown command generates a JSON cost report. The infracost/actions/comment@v1 action posts (or updates) a comment on the PR showing the cost delta. This demonstrates FinOps awareness to interviewers.
Get your Infracost API key
Sign up at infracost.io for a free API key. Add it as a GitHub repository secret named INFRACOST_API_KEY under your repository's Settings > Secrets and variables > Actions.
- Below the cost job, add the plan and apply jobs:
# Job 4: Terraform Plan (runs on PRs)
plan:
name: Terraform Plan
runs-on: ubuntu-latest
needs: [validate, security]
if: github.event_name == 'pull_request'
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Terraform
uses: hashicorp/setup-terraform@v3
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
aws-region: us-east-1
- name: Authenticate to GCP
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Terraform Init
run: terraform init
- name: Terraform Plan
run: terraform plan -var-file=environments/dev/terraform.tfvars -out=tfplan
# Job 5: Terraform Apply (runs on push to main)
apply:
name: Terraform Apply
runs-on: ubuntu-latest
needs: [validate, security]
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Terraform
uses: hashicorp/setup-terraform@v3
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
aws-region: us-east-1
- name: Authenticate to GCP
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Terraform Init
run: terraform init
- name: Terraform Apply
run: terraform apply -var-file=environments/dev/terraform.tfvars -auto-approveWhat does this code do?
Both the plan and apply jobs authenticate to AWS using aws-actions/configure-aws-credentials@v4 with OIDC (the role-to-assume input triggers the OIDC flow automatically). They authenticate to GCP using google-github-actions/auth@v2 with Workload Identity Federation.
The needs: [validate, security] line ensures both validation and security scanning pass before Terraform touches any cloud resources. The plan job runs on PRs so reviewers can see the diff. The apply job only runs on merges to main.
Add the required repository secrets
You need three repository secrets (not environment variables, not API keys stored in the workflow file):
- AWS_ROLE_ARN: the ARN of your GitHubActionsRole (e.g. arn:aws:iam::123456789012:role/GitHubActionsRole).
- GCP_WORKLOAD_IDENTITY_PROVIDER: the full identifier of your Workload Identity Provider.
- GCP_SERVICE_ACCOUNT: the email of your github-actions-sa service account.
- Save .github/workflows/terraform.yml.
✔️ Awesome, I've got everything!
Great. Double check you have saved the file and that all five jobs (validate, security, cost, plan, apply) are present.
ⓧ I'd like to double check the full code
# CI/CD pipeline for multi-cloud Terraform infrastructure
name: Terraform CI/CD
# Run on PRs and pushes to main
on:
push:
branches: [main]
pull_request:
branches: [main]
# OIDC requires id-token write; pull-requests write for Infracost comments
permissions:
id-token: write
contents: read
pull-requests: write
jobs:
# Job 1: Format and validate Terraform code
validate:
name: Terraform Validate
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Terraform
uses: hashicorp/setup-terraform@v3
- name: Terraform Format Check
run: terraform fmt -check -recursive
- name: Terraform Init
run: terraform init -backend=false
- name: Terraform Validate
run: terraform validate
# Job 2: Security scanning with tfsec
security:
name: Security Scan
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Run tfsec
uses: aquasecurity/tfsec-action@v1.0.0
with:
format: sarif
# Job 3: Cost estimation with Infracost
cost:
name: Cost Estimation
runs-on: ubuntu-latest
if: github.event_name == 'pull_request'
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Infracost
uses: infracost/actions/setup@v3
with:
api-key: ${{ secrets.INFRACOST_API_KEY }}
- name: Generate cost breakdown
run: infracost breakdown --path=. --format=json --out-file=/tmp/infracost.json
- name: Post cost comment
uses: infracost/actions/comment@v1
with:
path: /tmp/infracost.json
behavior: update
# Job 4: Terraform Plan (runs on PRs)
plan:
name: Terraform Plan
runs-on: ubuntu-latest
needs: [validate, security]
if: github.event_name == 'pull_request'
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Terraform
uses: hashicorp/setup-terraform@v3
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
aws-region: us-east-1
- name: Authenticate to GCP
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Terraform Init
run: terraform init
- name: Terraform Plan
run: terraform plan -var-file=environments/dev/terraform.tfvars -out=tfplan
# Job 5: Terraform Apply (runs on push to main)
apply:
name: Terraform Apply
runs-on: ubuntu-latest
needs: [validate, security]
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Terraform
uses: hashicorp/setup-terraform@v3
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
aws-region: us-east-1
- name: Authenticate to GCP
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Terraform Init
run: terraform init
- name: Terraform Apply
run: terraform apply -var-file=environments/dev/terraform.tfvars -auto-approveWorkflow file not detected by GitHub?
Make sure the file is at exactly .github/workflows/terraform.yml (case-sensitive) in your repository root. The .github directory must start with a dot.
If you see YAML syntax errors, check that indentation uses spaces (not tabs) and that all colons have a space after them. Help me debug my GitHub Actions YAML syntax
Test the pipeline end-to-end
The best way to verify your pipeline works is to push a real change through it. You will create a branch, make a minor infrastructure tweak, and push it to trigger the workflow.
- Create a feature branch by running:
git checkout -b test/pipeline-verification- Make a minor change to test the pipeline. Add a tag to your AWS VPC module by opening modules/aws-vpc/main.tf and adding a new tag to the VPC resource (e.g. Pipeline = "tested").
- Stage, commit, and push your changes by running:
git add .
git commit -m "test: verify CI/CD pipeline with minor tag change"
git push origin test/pipeline-verification- Open your repository on GitHub and create a pull request from test/pipeline-verification into main.
- Click the Actions tab in your repository to watch the workflow run.
You should see the validate, security, cost, and plan jobs all trigger. The validate job checks formatting. The security job scans for misconfigurations. The cost job posts a comment on your PR with an estimated monthly cost. The plan job shows the Terraform execution plan.
What should I see?
In the Actions tab, you should see a workflow run with green checkmarks (or amber if tfsec finds security warnings in your Terraform code). On the PR itself, look for an Infracost comment showing the estimated monthly cost of your infrastructure.
If the security job fails, that is actually a good sign. It means tfsec is catching real misconfigurations in your code. You can add soft_fail: true to the tfsec action temporarily while you address the findings.
OIDC authentication failing in the plan job?
Verify that your repository secrets (AWS_ROLE_ARN, GCP_WORKLOAD_IDENTITY_PROVIDER, GCP_SERVICE_ACCOUNT) are set correctly under Settings > Secrets and variables > Actions.
Check that the trust policy on your GitHubActionsRole allows your repository and branch pattern. Also confirm that the permissions block at the top of the workflow includes id-token: write. Help me debug OIDC authentication in my GitHub Actions workflow
Write ADR #3: CI/CD pipeline design decisions
Architecture Decision Records capture the reasoning behind key technical choices. This ADR documents why you chose OIDC over stored secrets, tfsec for security, Infracost for cost awareness, and environment-gated apply.
- Create the ADR file by running:
touch docs/adr/0003-cicd-pipeline-design.md- Open docs/adr/0003-cicd-pipeline-design.md and add the following content:
# ADR 3: CI/CD Pipeline Design
## Status
Accepted
## Context
We need a CI/CD pipeline that deploys multi-cloud infrastructure safely,
with security scanning and cost visibility on every change.
## Decision
### OIDC over stored secrets
GitHub Actions authenticates to AWS and GCP using OIDC federation
(Workload Identity Federation for GCP, IAM OIDC Provider for AWS).
No long-lived credentials are stored as GitHub secrets.
This eliminates credential rotation burden and reduces blast radius
if the repository is compromised.
### tfsec for security scanning
tfsec runs on every PR as a required check. It catches common
misconfigurations (public access, missing encryption, overly permissive IAM)
before they reach any environment. This demonstrates shift-left security.
### Infracost for FinOps
Infracost posts cost impact comments on every PR. Engineers see the
financial consequence of their changes before merging. This builds
cost awareness into the development workflow.
### Environment-gated apply
terraform plan runs on PRs for review. terraform apply only runs
on merge to main, targeting the dev environment. Production requires
a separate approval workflow (future enhancement).
## Consequences
- Zero stored credentials reduces security risk
- Security issues caught before merge, not after deployment
- Cost visibility prevents surprise bills
- Apply only on main prevents partial or accidental deploymentsWhy write ADRs?
ADRs show interviewers that you think about trade-offs, not just implementation. Documenting why you chose OIDC over stored secrets, or why tfsec over Checkov, demonstrates the engineering judgment that separates senior candidates from junior ones.
- Save docs/adr/0003-cicd-pipeline-design.md.
- Commit the ADR by running:
git add docs/adr/0003-cicd-pipeline-design.md
git commit -m "docs: add ADR 3 - CI/CD pipeline design decisions"Not sure what to write in your ADR?
Focus on the alternatives you considered and why you rejected them. For example, you chose OIDC over storing AWS access keys as GitHub secrets because OIDC tokens are short-lived and scoped to individual workflow runs. Help me improve my Architecture Decision Record
Your CI/CD pipeline is now a security and cost quality gate for every infrastructure change. Next up, you will verify end-to-end connectivity across the VPN and document the full architecture.
Verify Connectivity and Document the Architecture
Your CI/CD pipeline is fully operational with OIDC federation, security scanning, and cost estimation. The infrastructure is deployed across both clouds.
Before packaging this project for your portfolio, you need to prove it actually works. In this step, you will run end-to-end connectivity tests across the VPN, verify BGP route propagation with ECMP, and document everything in a comprehensive README with an architecture diagram.
In this step, get ready to:
- Run end-to-end connectivity tests (ping, HTTP, DNS) across the VPN.
- Verify BGP route propagation, ECMP, and VPN tunnel status in both clouds.
- Document the architecture with a Mermaid diagram and comprehensive README.
Run end-to-end connectivity tests across the VPN
Both instances have no public IP address. You will use IAP TCP tunneling to connect to the GCP Compute Engine instance for testing.
- Connect to your GCP frontend instance via IAP by running this command:
gcloud compute ssh frontend-instance \
--zone=us-central1-a \
--tunnel-through-iap- Once connected, test DNS resolution for the AWS backend by running:
# Resolve the AWS backend hostname over cross-cloud DNS
nslookup api.internal.awsYou should see the private IP address of your AWS EC2 instance (in the 10.0.1.0/24 range) returned in the response.
- Test HTTP connectivity to the AWS backend API over the VPN by running:
# Call the AWS backend health endpoint over the VPN
curl http://api.internal.aws:3000/healthYou should see a JSON response with {"status":"ok"} and a timestamp.
- Test latency between the two clouds by running:
# Measure cross-cloud round-trip time
curl http://api.internal.aws:3000/latency- Test ICMP ping to the AWS private subnet by running:
# Ping the AWS subnet to confirm Layer 3 connectivity
ping -c 4 10.0.1.0- Type exit to disconnect from the GCP instance.
What do these tests prove?
DNS resolution confirms your Cloud DNS forwarding zone is correctly sending internal.aws queries to the Route 53 Resolver over the VPN tunnel.
HTTP connectivity proves end-to-end application-layer communication between the GCP spoke VPC and the AWS private subnet. The traffic flows from spoke-vpc through VPC peering to hub-vpc, through the HA VPN tunnels, to the Transit Gateway, and into the AWS VPC.
DNS resolution failing?
Verify the Cloud DNS forwarding zone targets the correct Route 53 Resolver inbound endpoint IPs. Check that the Cloud Router is advertising 35.199.192.0/19 as a custom route so DNS responses can route back to GCP.
Help me debug cross-cloud DNS
Verify BGP route propagation and VPN tunnel status
Beyond application-level tests, you need to confirm the network layer is healthy. This means verifying that BGP has propagated routes into the Transit Gateway route table, that ECMP is distributing traffic across all 4 tunnels, and that every tunnel reports UP status.
- Check VPN tunnel status on the GCP side by running:
gcloud compute vpn-tunnels list --region=us-central1You should see all 4 tunnels listed with status ESTABLISHED.
- Verify BGP session status and learned routes on the Cloud Router by running:
gcloud compute routers get-status cloud-router \
--region=us-central1The output shows BGP peer status, advertised routes, and learned routes. You should see 10.0.0.0/16 (the AWS VPC CIDR) in the learned routes from the Transit Gateway.
- Check VPN tunnel status on the AWS side by running:
aws ec2 describe-vpn-connections \
--query "VpnConnections[*].VgwTelemetry" \
--output tableYou should see all tunnel entries with status UP. Each VPN connection has 2 tunnels, so you should see 4 entries total across both connections.
- Find your Transit Gateway route table ID by running:
aws ec2 describe-transit-gateway-route-tables \
--query "TransitGatewayRouteTables[0].TransitGatewayRouteTableId" \
--output text- Search for BGP-propagated routes using that route table ID by running:
aws ec2 search-transit-gateway-routes \
--transit-gateway-route-table-id [[TGW_RTB_ID="your Transit Gateway route table ID"]] \
--filters "Name=type,Values=propagated"You should see the GCP CIDR ranges (10.1.0.0/24 and 10.2.0.0/24) propagated via BGP. Multiple routes to the same destination with different attachments confirm ECMP is active.
How does ECMP show up in the route table?
When ECMP is working, you see the same destination CIDR listed multiple times, each with a different VPN attachment ID. Transit Gateway distributes traffic across these equal-cost paths for higher throughput and redundancy.
Tunnels showing DOWN?
VPN tunnels can take 1-2 minutes to establish after infrastructure changes. If tunnels remain DOWN, check that the pre-shared keys match between AWS and GCP, and verify that security groups allow UDP port 500 and 4500 (IKE/IPsec).
Help me troubleshoot VPN tunnels
Document the architecture in the README
With connectivity verified, it is time to document everything. A comprehensive README with a visual architecture diagram transforms this from a working project into a portfolio piece that interviewers can understand at a glance.
- Open README.md in the root of your repository and replace its contents with the following:
# Enterprise Multi-Cloud Infrastructure: AWS + GCP
## Architecture Overview
This project implements enterprise-grade multi-cloud networking between AWS and GCP
with encrypted HA VPN tunnels, BGP dynamic routing, and ECMP load balancing.
```mermaid
graph LR
subgraph AWS["AWS us-east-1"]
EC2[EC2 Backend API<br/>10.0.1.0/24]
R53[Route 53<br/>internal.aws]
VPC_AWS[VPC 10.0.0.0/16]
TGW[Transit Gateway<br/>ASN 64512 + ECMP]
end
subgraph GCP["GCP us-central1"]
subgraph Hub["hub-vpc 10.1.0.0/24"]
HAVPN[HA VPN Gateway]
CR[Cloud Router<br/>ASN 65000]
CDNS[Cloud DNS<br/>Forwarding]
end
subgraph Spoke["spoke-vpc 10.2.0.0/24"]
GCE[Compute Engine<br/>Frontend]
end
end
TGW <-->|4 IPsec Tunnels<br/>IKEv2 + BGP| HAVPN
VPC_AWS --- TGW
Hub <-->|VPC Peering<br/>Custom Routes| Spoke
GCE -->|api.internal.aws| CDNS
CDNS -->|Over VPN| R53
```
## Key Components
| Component | Purpose |
|-----------|--------|
| AWS Transit Gateway | ECMP load balancing across VPN tunnels |
| GCP HA VPN | 99.99% SLA with 4 IPsec tunnels |
| BGP Dynamic Routing | Automatic route propagation between clouds |
| Cross-Cloud DNS | Service discovery by hostname over VPN |
| Hub-and-Spoke VPC | Centralized VPN endpoint with workload isolation |
## Cost Breakdown
| Resource | Hourly Cost |
|----------|------------|
| GCP HA VPN (4 tunnels) | $0.20/hr |
| AWS VPN (2 connections) | $0.10/hr |
| AWS Transit Gateway | $0.05/hr |
| VMs + other | ~$0.05/hr |
| **Total** | **~$0.40/hr (~$10/day)** |
## Deployment Instructions
1. Clone this repository.
2. Configure AWS and GCP credentials.
3. Initialize Terraform: `terraform init`
4. Deploy dev environment: `terraform apply -var-file=environments/dev/terraform.tfvars`
5. Verify connectivity: ping and curl between instances over VPN.
## Architecture Decision Records
| ADR | Decision |
|-----|----------|
| [ADR-0001](docs/adr/0001-transit-gateway-over-vpgw.md) | Transit Gateway over Virtual Private Gateway |
| [ADR-0002](docs/adr/0002-environment-separation-strategy.md) | Environment separation strategy |
| [ADR-0003](docs/adr/0003-cicd-pipeline-design.md) | CI/CD pipeline design decisions |
## CI/CD Pipeline
- **OIDC Federation**: Zero stored secrets for AWS and GCP authentication.
- **Security Scanning**: tfsec runs on every PR.
- **Cost Estimation**: Infracost posts cost impact comments on PRs.
- **Environment Gating**: Plan on PRs, apply on merge to main.
## Teardown
To destroy all resources:
```bash
terraform destroy -var-file=environments/dev/terraform.tfvars
```Why a Mermaid diagram?
Mermaid diagrams render natively on GitHub. Anyone viewing your repository sees a visual architecture diagram without needing external tools or image files.
This makes your project immediately understandable to interviewers and collaborators browsing the repository.
- Save README.md.
- Commit and push the changes to your repository by running:
git add README.md
git commit -m "Add comprehensive architecture documentation"
git push origin main- Open your repository on GitHub and verify the Mermaid diagram renders as a visual flowchart in the README.
Mermaid diagram not rendering?
GitHub renders Mermaid inside fenced code blocks marked with the mermaid language identifier. Make sure there are no extra spaces before the opening backticks and that the diagram syntax has no typos.
Help me fix my Mermaid diagram
✔️ Awesome, I've got everything!
Great. Double check you have saved README.md and pushed to GitHub.
ⓧ I'd like to double check the full code
# Enterprise Multi-Cloud Infrastructure: AWS + GCP
## Architecture Overview
This project implements enterprise-grade multi-cloud networking between AWS and GCP
with encrypted HA VPN tunnels, BGP dynamic routing, and ECMP load balancing.
```mermaid
graph LR
subgraph AWS["AWS us-east-1"]
EC2[EC2 Backend API<br/>10.0.1.0/24]
R53[Route 53<br/>internal.aws]
VPC_AWS[VPC 10.0.0.0/16]
TGW[Transit Gateway<br/>ASN 64512 + ECMP]
end
subgraph GCP["GCP us-central1"]
subgraph Hub["hub-vpc 10.1.0.0/24"]
HAVPN[HA VPN Gateway]
CR[Cloud Router<br/>ASN 65000]
CDNS[Cloud DNS<br/>Forwarding]
end
subgraph Spoke["spoke-vpc 10.2.0.0/24"]
GCE[Compute Engine<br/>Frontend]
end
end
TGW <-->|4 IPsec Tunnels<br/>IKEv2 + BGP| HAVPN
VPC_AWS --- TGW
Hub <-->|VPC Peering<br/>Custom Routes| Spoke
GCE -->|api.internal.aws| CDNS
CDNS -->|Over VPN| R53
```
## Key Components
| Component | Purpose |
|-----------|--------|
| AWS Transit Gateway | ECMP load balancing across VPN tunnels |
| GCP HA VPN | 99.99% SLA with 4 IPsec tunnels |
| BGP Dynamic Routing | Automatic route propagation between clouds |
| Cross-Cloud DNS | Service discovery by hostname over VPN |
| Hub-and-Spoke VPC | Centralized VPN endpoint with workload isolation |
## Cost Breakdown
| Resource | Hourly Cost |
|----------|------------|
| GCP HA VPN (4 tunnels) | $0.20/hr |
| AWS VPN (2 connections) | $0.10/hr |
| AWS Transit Gateway | $0.05/hr |
| VMs + other | ~$0.05/hr |
| **Total** | **~$0.40/hr (~$10/day)** |
## Deployment Instructions
1. Clone this repository.
2. Configure AWS and GCP credentials.
3. Initialize Terraform: `terraform init`
4. Deploy dev environment: `terraform apply -var-file=environments/dev/terraform.tfvars`
5. Verify connectivity: ping and curl between instances over VPN.
## Architecture Decision Records
| ADR | Decision |
|-----|----------|
| [ADR-0001](docs/adr/0001-transit-gateway-over-vpgw.md) | Transit Gateway over Virtual Private Gateway |
| [ADR-0002](docs/adr/0002-environment-separation-strategy.md) | Environment separation strategy |
| [ADR-0003](docs/adr/0003-cicd-pipeline-design.md) | CI/CD pipeline design decisions |
## CI/CD Pipeline
- **OIDC Federation**: Zero stored secrets for AWS and GCP authentication.
- **Security Scanning**: tfsec runs on every PR.
- **Cost Estimation**: Infracost posts cost impact comments on PRs.
- **Environment Gating**: Plan on PRs, apply on merge to main.
## Teardown
To destroy all resources:
```bash
terraform destroy -var-file=environments/dev/terraform.tfvars
```Your multi-cloud infrastructure is fully verified and documented. Next up, you will harden security, audit IAM permissions, and finalize the repository for interviews.
Harden Security, Add Tagging, and Package for Portfolio
Your cross-cloud network is verified and documented. The final step before this project is portfolio-ready is hardening: confirming least-privilege access, applying consistent resource tagging for cost tracking, and running a security scan to catch any misconfigurations.
Interviewers look for three signals in infrastructure projects: security-first thinking, operational maturity (tagging and cost awareness), and clean documentation. This step delivers all three.
In this step, get ready to:
- Audit IAM roles and verify no resources have unnecessary public access.
- Add consistent resource tagging across both clouds for cost tracking and ownership.
- Review security rules, run tfsec, and finalize the repository for interviews.
Audit IAM roles and verify no public access
Least-privilege means every identity has exactly the permissions it needs and nothing more. Over-permissioned roles are the most common finding in cloud security audits.
- Open modules/iam/aws-oidc.tf and review the GitHubActionsRole policy.
- Confirm the policy grants only EC2, VPC, Route53, and S3 (state bucket) permissions.
- Verify there are no wildcard resource ARNs (Resource: "*") on destructive actions. Scope actions to the specific resources this project creates where possible.
- Open modules/iam/gcp-wif.tf and review the service account role bindings.
- Confirm github-actions-sa uses compute.admin, dns.admin, and iam.serviceAccountUser. These are the minimum roles needed to manage VPCs, VPN, DNS, and impersonate the service account.
- Verify the Workload Identity binding is scoped to your specific repository (not all of GitHub).
- Open modules/aws-vpc/ec2.tf and confirm the EC2 instance IAM role uses only the AmazonSSMManagedInstanceCore managed policy.
- Verify the EC2 instance has no public IP (associate_public_ip_address = false or no setting, which defaults to false in a private subnet).
- Verify the GCP Compute Engine instance in modules/gcp-vpc/compute.tf has no external IP assigned.
What does least-privilege look like here?
The EC2 instance role grants SSM access only. It does not need S3, Lambda, or any other service permissions because it just runs the Node.js API.
The GitHub Actions role is broader because it provisions infrastructure across multiple services. Scoping Resource ARNs to specific VPCs, subnets, or hosted zones (instead of *) is the next level of hardening you can highlight in interviews.
Add consistent resource tagging across both clouds
Tagging is the foundation of FinOps. Without consistent tags, you cannot allocate costs to teams, environments, or projects in a shared account.
- Add an owner variable to your root variables.tf (or in the environment-level variables file) by adding this code:
variable "owner" {
description = "Resource owner name for cost tracking"
type = string
}What does this variable do?
The owner variable lets you stamp every resource with who created it. In a shared AWS account, this makes cost reports filterable by team member.
For AWS, the default_tags block in the provider configuration automatically applies tags to every resource the provider manages. This is far cleaner than adding tags blocks to each individual resource.
- Update your AWS provider block (in your environment's providers.tf or root main.tf) to include default_tags:
provider "aws" {
region = var.aws_region
default_tags {
tags = {
Project = "multicloud-infra"
Environment = var.environment
Owner = var.owner
ManagedBy = "terraform"
}
}
}What does default_tags do?
The default_tags block applies these four tags to every AWS resource this provider creates. You do not need to add a tags block to each aws_vpc, aws_instance, or aws_ec2_transit_gateway individually.
Any resource-level tags you still define (like a Name tag) merge with the provider defaults automatically.
GCP does not have an equivalent provider-level default_labels that applies to all resources. You need to add labels directly to each resource that supports them.
- Add a locals block to your GCP modules (or a shared locals.tf) defining common labels:
locals {
common_labels = {
project = "multicloud-infra"
environment = var.environment
managed_by = "terraform"
}
}- Apply labels = local.common_labels to each GCP resource that supports labels. Here is an example for the HA VPN gateway in modules/gcp-vpn/main.tf:
resource "google_compute_ha_vpn_gateway" "ha_vpn_gw" {
region = var.region
name = "ha-vpn-gw"
network = google_compute_network.hub.id
# Consistent labels for cost tracking
labels = local.common_labels
}Which GCP resources support labels?
The resources you need to label include: google_compute_ha_vpn_gateway, google_compute_vpn_tunnel, google_compute_instance, and google_dns_managed_zone.
Some resources like google_compute_network and google_compute_subnetwork do not support labels. For those, the label coverage comes from the higher-level resources that use them.
- Apply labels = local.common_labels to all labeled GCP resources across modules/gcp-vpn/main.tf, modules/gcp-vpc/main.tf, modules/gcp-vpc/compute.tf, and modules/dns/main.tf.
- Add owner = "your-name" to your environments/dev/terraform.tfvars file.
- Run terraform plan to preview the tagging changes:
terraform plan -var-file=environments/dev/terraform.tfvars- You should see in-place updates adding tags/labels to existing resources. No resources should be destroyed.
- Apply the changes by running:
terraform apply -var-file=environments/dev/terraform.tfvarsSeeing resources marked for replacement instead of update?
Tags and labels are in-place updates for nearly all resources. If terraform plan shows a destroy/create cycle, check that you have not changed a name or network argument accidentally while adding labels.
Help me debug why Terraform wants to replace my resources after adding tags.
Review security rules, run tfsec, and finalize for portfolio
The final hardening pass confirms your firewall rules and security groups expose only what is necessary, and that tfsec finds no critical misconfigurations.
- Open modules/aws-vpc/main.tf and review the aws_security_group.
- Confirm inbound traffic is restricted to the GCP CIDR range 10.1.0.0/16 only. No 0.0.0.0/0 rules should exist.
- Open modules/gcp-vpc/main.tf and review the google_compute_firewall rules.
- Confirm ingress is restricted to 10.0.0.0/16 (the AWS VPC CIDR) on both hub and spoke VPCs.
Why is CIDR-scoped access acceptable here?
Both security groups and firewall rules allow all ports from the peer cloud's CIDR. This is acceptable because all traffic crosses an encrypted IPsec tunnel. The tunnel itself acts as the trust boundary.
In production, you would further restrict to specific ports (3000 for the API, 53 for DNS). For this project, the cross-cloud CIDR scope demonstrates the pattern clearly without obscuring it behind many port-specific rules.
- Run tfsec locally to scan your entire project for misconfigurations:
tfsec .- Review the output. Address any CRITICAL or HIGH severity findings. Common findings to expect and resolve:
- Security group with overly broad ingress: acceptable here because it is scoped to a private CIDR, not 0.0.0.0/0.
- VPC flow logs not enabled: add flow logs if tfsec flags it, or add a #tfsec:ignore comment with a justification.
- Missing encryption: confirm the GCS state bucket has default encryption (it does by default in GCP).
Document tfsec exceptions
When you suppress a tfsec finding, add a comment explaining why. For example: #tfsec:ignore:aws-vpc-no-public-ingress-sgr -- ingress scoped to 10.1.0.0/16 cross-cloud CIDR only. This shows interviewers you made a deliberate decision, not an oversight.
- Verify all three ADRs are finalized in docs/adr/:
- 0001-transit-gateway-over-vpgw.md documents the Transit Gateway decision.
- 0002-environment-separation-strategy.md documents the environment separation approach.
- 0003-cicd-pipeline-design.md documents the CI/CD pipeline decisions.
- Commit all changes with a clean message by running:
git add -A
git commit -m "Harden security, add consistent resource tagging, finalize ADRs"- Push to your repository by running:
git push origin main- Your CI/CD pipeline triggers automatically. Confirm the tfsec step passes in the GitHub Actions run.
Pipeline failing on the security scan?
If tfsec passes locally but fails in the pipeline, check that your #tfsec:ignore comments are on the line directly above the flagged resource block. Also confirm you committed the file with the ignore comment (not just saved locally).
Help me fix tfsec failures in my GitHub Actions pipeline.
Secret mission
Cross-Cloud Health Monitoring and Latency Dashboard
Your infrastructure is production-ready, but how would you know if a VPN tunnel drops at 3 AM? Enterprise teams never deploy infrastructure without automated alerting. In this secret mission, you will add cross-cloud monitoring that watches your VPN tunnels in both AWS and GCP, sends alerts when something breaks, and visualizes latency on a dashboard.
Clean Up Your Resources
Clean Up Your Resources
This project provisions paid cloud resources on both AWS and GCP that incur charges while running. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.
Cost warning
Your multi-cloud infrastructure costs approximately $0.40/hour while running. That adds up to roughly $10/day.
The main cost drivers are the 4 GCP HA VPN tunnels ($0.05/tunnel/hr), 2 AWS VPN connections ($0.05/connection/hr), the Transit Gateway VPN attachment ($0.05/hr), and the Route 53 Resolver inbound endpoint. If you are not actively working on this project, choose Pause or Delete below to avoid unnecessary charges.
Resources you used:
- AWS VPC, Transit Gateway, Site-to-Site VPN connections (4 IPsec tunnels).
- AWS EC2 instance (backend API in private subnet).
- AWS Route 53 private hosted zone and Resolver inbound endpoint.
- AWS CloudWatch alarms, SNS topic, and IAM OIDC provider with GitHubActionsRole.
- GCP hub and spoke VPCs with VPC peering, firewall rules, and Cloud NAT.
- GCP HA VPN gateway, Cloud Router, and 4 VPN tunnels with BGP sessions.
- GCP Compute Engine instance (frontend in spoke VPC).
- GCP Cloud DNS zones (forwarding, private, peering) and Cloud Monitoring resources (alert policy, dashboard, notification channel).
- GCP Workload Identity Pool and service account.
- GCS Terraform state bucket.
- GitHub repository.
Keep everything running
No action needed. Choose this if you are still actively building or testing your multi-cloud infrastructure.
- Your resources will continue running and incurring charges at approximately $10/day.
- Monitor your AWS and GCP billing dashboards to track spending.
- Your Terraform CLI, AWS CLI, and gcloud CLI installations remain on your machine.
Pause - I'll come back to this later
Destroy all cloud resources to stop charges, but keep your state file and code so you can recreate everything later. This is the recommended option between work sessions.
- Navigate to your project's environment directory in the terminal.
- Run the following command to destroy all Terraform-managed resources:
terraform destroy- Confirm the destruction when prompted by typing yes.
- Your GCS state bucket remains intact so Terraform knows what to recreate.
- When you are ready to resume, run terraform apply to bring everything back up.
Delete - I don't want to use this again
Remove all project resources from both clouds and start fresh. This cannot be undone.
- Destroy all Terraform-managed resources by running:
terraform destroy- Confirm the destruction when prompted by typing yes.
- Delete the GCS Terraform state bucket by running:
gcloud storage rm --recursive gs://[[STATE_BUCKET="your-terraform-state-bucket-name"]]- Delete the local project folder:
cd ~/Desktop
rm -rf multicloud-infrastructureOn Windows, use this instead:
cd ~\Desktop
rmdir /s /q multicloud-infrastructure- Delete the GitHub repository by opening your repository page, clicking Settings, scrolling to the Danger Zone section, and clicking Delete this repository.
- Confirm by typing the repository name when prompted.
That's a Wrap!
That's a Wrap!
Nice work! You just built an enterprise-grade multi-cloud infrastructure that most engineers only read about.
You've learned how to:
- Deploy AWS Transit Gateway with ECMP and HA VPN with 4 IPsec tunnels and BGP dynamic routing between AWS and GCP, creating high-availability cross-cloud private connectivity.
- Implement cross-cloud DNS resolution with Route 53 Resolver and Cloud DNS forwarding zones so distributed services discover each other by hostname over private encrypted tunnels.
- Build a zero-trust CI/CD pipeline with OIDC federation, tfsec security scanning, and Infracost cost estimation on every pull request, deployed through GitHub Actions with zero stored secrets.
- Secret Mission: Add cross-cloud health monitoring with automated tunnel alerting and a latency dashboard using CloudWatch and Cloud Monitoring.
Ready to quiz yourself?