Multi-Cloud GovRAMP Modernization Lab
Build a state government multi-cloud Kubernetes platform with GitOps and DR.
Introduction
30 Second Summary
When a state government agency needs to modernize its IT systems, it can't just pick one cloud provider and hope for the best. It needs to prove that citizen data stays available, recoverable, and compliant no matter what happens to any single vendor.
In this project, you will build a multi-cloud Kubernetes platform for a fictional state health department, deploy a citizen benefits portal across four environments, run a real disaster recovery drill, and author the compliance documentation needed for a GovRAMP Moderate security review.
What You'll Build
A lab-simulated state modernization platform spanning an on-prem data center simulation and three cloud providers, provisioned with OpenTofu, managed by Argo CD GitOps, and protected by Velero disaster recovery.
By the end of this project, you'll have:
- A four-environment multi-cloud Kubernetes platform (on-prem + AWS + Azure + GCP) provisioned entirely through OpenTofu and managed by Argo CD GitOps.
- A tested Continuity of Operations (COOP) disaster recovery procedure with documented RTO/RPO metrics from a real backup/restore drill using Velero.
- A complete GovRAMP Moderate controls crosswalk, migration wave plan, and portability scorecard authored with Cursor IDE Composer 2 and validated against industry benchmarks.
- A Crossplane vs. OpenTofu portability comparison that evaluates two infrastructure management paradigms side by side.
Are there any prerequisites?
No prior experience with government compliance frameworks or multi-cloud architecture is required. This guide includes full setup instructions for every tool. You will need a Windows machine with WSL2, free-tier accounts on AWS, Azure, and GCP, and Cursor IDE with a Pro subscription for Composer 2 parallel mode.
Before We Start
Before We Start...
Set Up Your Lab Environment
Before you can build a multi-cloud platform, you need a consistent development environment with all the right tools installed and verified.
This step gets your local machine ready with WSL2, Cursor IDE, cloud CLIs, infrastructure tools, and a GitHub repository.
By the end, you will have a running k3s cluster simulating your on-prem data center, three cloud CLIs authenticated, and a repo ready for code.
In This Step, Get Ready To:
- Install WSL2, Cursor IDE, and all CLI tools needed for the build.
- Install OpenTofu, k3s, and the Velero CLI, and verify your on-prem cluster is running.
- Authenticate to AWS, Azure, and GCP, set billing alerts, and create the project repository.
Install WSL2, Cursor IDE, and Core CLI Tools
The entire lab runs inside a Linux environment. WSL2 gives you a full Ubuntu distribution on Windows, which is where all your CLI tools, clusters, and infrastructure code will live.
Cursor IDE runs on Windows but connects to WSL2 as its terminal, so you get a native Linux environment with a graphical code editor.
- Open PowerShell as Administrator and install WSL2 with Ubuntu:
wsl --install -d UbuntuWhat does this command do?
This installs the WSL2 subsystem and downloads an Ubuntu distribution.
If WSL2 is already installed, the command updates it.
You may need to restart your computer after installation.
- After the restart, open the Ubuntu terminal from the Start menu.
- Create a username and password when prompted.
- Verify WSL2 is running:
wsl --list --verboseYou should see Ubuntu listed with VERSION 2.
- Download and install Cursor IDE on Windows.
- Open Cursor IDE and open the integrated terminal.
- Set the default terminal profile to WSL2 Ubuntu by pressing Ctrl+Shift+P, typing Terminal: Select Default Profile, and selecting Ubuntu (WSL).
Now enable Composer 2 with agent manager and parallel mode.
- Open Cursor Settings and navigate to the Features section.
- Enable Composer 2.
- Enable Agent Manager.
- Enable Parallel Mode.
What Is Composer 2?
Composer 2 is Cursor's AI assistant that can generate and edit code directly in your editor.
The agent manager lets you run multiple AI agents simultaneously. Parallel mode means you can have one agent generating OpenTofu modules while another drafts compliance documents at the same time.
Cursor Pro Required for Full Parallel Mode
Composer 2 with agent manager and parallel mode requires a Cursor Pro subscription. The free tier includes limited Composer usage.
If you prefer not to use Cursor, the project is fully buildable with any AI code assistant. The Cursor-specific features (parallel agents, repository-wide context) are acceleration patterns, not hard dependencies.
Next, install the core CLI tools inside your WSL2 terminal.
- Open a new WSL2 terminal in Cursor.
- Update your package list and install Git:
sudo apt update && sudo apt install -y git- Install kubectl:
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl- Install the AWS CLI:
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install- Install the Azure CLI:
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash- Install the gcloud CLI:
curl https://sdk.cloud.google.com | bash
exec -l $SHELL
gcloud init- Verify each tool is installed by running its version command:
git --version
kubectl version --client
aws --version
az --version
gcloud --versionI see version numbers for all tools
All your core CLI tools are installed and working.
A command was not found
If any command returns "command not found", the installation did not complete successfully.
- Re-run the install command for the failing tool.
- Make sure you are running commands inside the WSL2 terminal, not PowerShell.
- For the gcloud CLI, run exec -l $SHELL to reload your shell after installation.
Still Having Trouble?
Paste the exact error message into Cursor Composer and ask it to help you debug the installation.
Now set up your project repository on GitHub.
- Go to GitHub and create a new repository called columbia-modernization.
- Set it to Public and initialize it with a README.
- Clone the repository inside WSL2:
git clone [[YOUR_REPO_URL="https://github.com/YOUR_USERNAME/columbia-modernization.git"]]
cd columbia-modernization- Open the cloned repository in Cursor IDE by selecting File, then Open Folder, and navigating to the columbia-modernization directory inside your WSL2 file system.
Install OpenTofu, k3s, and the Velero CLI
These three tools form the infrastructure backbone of your platform. OpenTofu provisions cloud VMs. k3s runs lightweight Kubernetes clusters on those VMs. Velero handles backup and disaster recovery across clusters.
You can use Cursor Composer to generate the install commands, or run the following commands directly in your WSL2 terminal.
- Open Composer with Cmd+I (macOS) or Ctrl+I (Windows) and prompt it:
# Prompt for Composer:
# "Generate install commands for OpenTofu v1.11.6, Velero CLI v1.18.0,
# and k3s v1.35.3 on Ubuntu WSL2. Include verification commands."Alternatively, run the following commands directly.
- Install OpenTofu v1.11.6:
# Add the OpenTofu repository and install
curl --proto '=https' --tlsv1.2 -fsSL https://get.opentofu.org/install-opentofu.sh -o install-opentofu.sh
chmod +x install-opentofu.sh
./install-opentofu.sh --install-method deb
rm install-opentofu.sh- Verify the installation:
tofu --versionWhat should I see?
You should see OpenTofu v1.11.6 or similar output confirming the version.
- Install k3s to create your columbia-onprem cluster:
curl -sfL https://get.k3s.io | sh -What does this command do?
This downloads and installs k3s, a lightweight Kubernetes distribution. It starts a single-node cluster automatically.
This WSL2 cluster simulates the Columbia HHS on-premises data center.
Why k3s Instead of Managed Kubernetes?
Managed Kubernetes services like EKS, AKS, and GKE charge $73/month or more for control plane fees alone.
k3s runs the same Kubernetes API on small VMs at near-zero cost. The same k3s binary runs identically on all four targets, which proves the workload is truly cloud-agnostic.
- Verify the cluster is running:
sudo kubectl get nodesI see a node in Ready status
Your columbia-onprem cluster is up and running.
I see an error or no nodes
k3s may still be starting up.
- Wait 30 seconds and try the command again.
- Check the k3s service status with sudo systemctl status k3s.
- If the service failed, check the logs with sudo journalctl -u k3s.
Still Not Working?
Paste the output of sudo journalctl -u k3s --no-pager | tail -20 into Cursor Composer and ask it to help diagnose the k3s startup failure.
- Set up your kubeconfig so kubectl works without sudo:
mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
export KUBECONFIG=~/.kube/configPro tip: persist KUBECONFIG
Add export KUBECONFIG=~/.kube/config to your ~/.bashrc so it loads automatically every time you open a terminal.
- Verify kubectl works without sudo:
kubectl get nodes- Install the Velero CLI v1.18.0:
curl -LO https://github.com/vmware-tanzu/velero/releases/download/v1.18.0/velero-v1.18.0-linux-amd64.tar.gz
tar -xzf velero-v1.18.0-linux-amd64.tar.gz
sudo mv velero-v1.18.0-linux-amd64/velero /usr/local/bin/
rm -rf velero-v1.18.0-linux-amd64*- Verify the Velero CLI:
velero versionWhat should I see?
You should see the client version as v1.18.0.
The server version will show an error because Velero is not installed on a cluster yet. That is expected at this stage.
Configure Cloud Accounts and Set Cost Guards
You need active accounts on all three cloud providers. Each provider offers free-tier resources that keep this lab at near-zero cost, but you must set billing alerts to prevent accidental charges if you forget to tear down resources.
- If you do not have accounts, create free-tier accounts at the following links:
- Log in to each cloud CLI from your WSL2 terminal:
# AWS - configure with your access key
aws configure
# Azure - browser-based login
az login
# GCP - browser-based login
gcloud auth loginWhat does this command do?
Each command authenticates your local CLI with the corresponding cloud provider.
For AWS, you will need your Access Key ID and Secret Access Key from the AWS IAM console. Azure and GCP open a browser window for authentication.
- Verify each login succeeded:
aws sts get-caller-identity
az account show
gcloud config get-value projectEach command should return your account details without errors.
Set Billing Alerts Now
Set a $1 billing alert on each cloud account before proceeding. Even with free-tier VMs, forgetting to tear down resources can lead to unexpected charges. A t3.small left running on AWS costs about $15/month.
- AWS: Go to Billing, select Budgets, click Create budget, and set a $1 monthly budget.
- Azure: Go to Cost Management, select Budgets, and create a $1 alert.
- GCP: Go to Billing, select Budgets & alerts, and create a $1 budget.
Finally, use Cursor Composer to generate the project README.
- Open Composer and prompt it with the following context:
# Prompt for Composer:
# "Generate a README.md for the columbia-modernization repository.
# Include:
# - Project title: State of Columbia HHS Multi-Cloud Modernization Platform
# - Scenario description: The Columbia HHS department operates a legacy citizen
# benefits portal handling Medicaid eligibility and SNAP benefit status for
# 2.4M residents. Data is classified Moderate Impact under NIST SP 800-60
# (PII and PHI).
# - System boundary summary: 4 environments (columbia-onprem, columbia-aws,
# columbia-azure, columbia-gcp)
# - Tool versions: OpenTofu v1.11.6, k3s v1.35.3, Argo CD v3.3.7, Velero v1.18.0
# - Embed a Mermaid diagram showing the system topology with all four
# environments, the Argo CD hub, Velero backup paths, and the OpenTofu
# provisioning layer. Use government terminology (System Boundary, CSP,
# COOP Backup Path)."- Review the generated README in Cursor's diff view.
- Accept the changes and verify the Mermaid diagram renders correctly.
- Commit and push to GitHub:
git add .
git commit -m "Initialize repository with README and system topology diagram"
git push origin main- Open your repository on GitHub and confirm the Mermaid diagram renders in the README.
What should I see?
Your GitHub README should display a rendered Mermaid diagram showing the system boundary with all four environments, the Argo CD control plane, Velero backup paths, and the OpenTofu provisioning layer.
GitHub natively renders Mermaid diagrams in markdown files.
Your lab environment is fully set up. You have a running k3s cluster simulating the on-prem data center, all three cloud CLIs authenticated, billing alerts in place, and a GitHub repository with a system topology diagram.
Next up, you will use OpenTofu to provision the three cloud VMs and install k3s on each one, creating the full multi-cloud system boundary.
Provision the Multi-Cloud System Boundary with OpenTofu
The Columbia HHS modernization platform needs to run across three cloud providers to prove that the citizen benefits portal is truly portable.
In this step, you will use OpenTofu to provision three virtual machines, one on each cloud provider, with k3s installed automatically via startup scripts.
By the end, you will have four healthy Kubernetes clusters (including your local columbia-onprem) forming the system boundary for the State of Columbia HHS.
In This Step, Get Ready To:
- Write a multi-provider OpenTofu configuration with separate modules for AWS, Azure, and GCP.
- Plan and apply the infrastructure to provision three cloud VMs with k3s installed.
- Verify all four Kubernetes clusters are healthy and accessible from a single kubeconfig.
Write the OpenTofu Multi-Provider Configuration
OpenTofu uses a modular structure to keep cloud-specific logic separated from the root configuration.
Your target structure matches Diagram 5 from the architecture diagrams: a root module under infra/ with three child modules (modules/aws-cluster/, modules/azure-cluster/, and modules/gcp-cluster/).
Each module provisions a single VM with a security group or firewall rule (opening ports 6443, 22, and 80) and a startup script that installs k3s with a TLS SAN for remote access.
- Open Cursor IDE and switch to your columbia-modernization repository.
- Create the directory structure for your OpenTofu modules.
mkdir -p infra/modules/aws-cluster
mkdir -p infra/modules/azure-cluster
mkdir -p infra/modules/gcp-clusterNow use Cursor Composer to generate the OpenTofu files.
Composer 2's parallel mode lets you run multiple agents at the same time. One agent can generate the AWS module while another generates the Azure module and a third generates the GCP module simultaneously.
- Open Cursor Composer with Cmd+Shift+I (macOS) or Ctrl+Shift+I (Windows).
- Send the following prompt to generate the root module configuration.
Generate an OpenTofu root module in infra/ with:
1. main.tf - Declare three providers:
- aws (region us-east-1)
- azurerm (features {})
- google (project from variable, region us-central1)
Use a required_providers block with source and version constraints for all three.
2. variables.tf - Define variables for:
- aws_region (default us-east-1)
- azure_location (default eastus)
- gcp_project (no default, required)
- gcp_region (default us-central1)
- k3s_version (default v1.35.3+k3s1)
- ssh_public_key_path (default ~/.ssh/id_rsa.pub)
3. outputs.tf - Output the public IP and kubeconfig path for each cloud VM.
4. Module calls for modules/aws-cluster, modules/azure-cluster, modules/gcp-cluster.
Pass the relevant variables to each module.
The k3s install script for each VM should use:
curl -sfL https://get.k3s.io | sh -s - server --tls-san <EXTERNAL-IP>
Each module should provision:
- AWS: t3.small instance in us-east-1 with a security group opening ports 6443, 22, 80
- Azure: B1s VM in eastus with an NSG opening ports 6443, 22, 80
- GCP: e2-micro in us-central1 with a firewall rule opening ports 6443, 22, 80Pro tip: use parallel mode
If you have Cursor Pro, enable parallel mode in Composer settings.
You can split this into three separate prompts, one per cloud module, and run them simultaneously. This mirrors how real platform teams use AI tooling to accelerate multi-cloud development.
- Review the generated files in Cursor's diff view before accepting the changes.
- Verify that each module has a main.tf, variables.tf, and outputs.tf.
VM Sizing Warning for Azure and GCP
The k3s server requires approximately 1596 MB of RAM with a workload running.
Azure B1s (1 GB RAM) and GCP e2-micro (1 GB RAM, 0.25 vCPU) may be too small to run k3s reliably. If you see k3s failing to start or pods stuck in Pending on these clusters, upgrade to Azure B2s (2 vCPU, 4 GB, ~$0.0416/hr) or GCP e2-small (2 GB, ~$0.0168/hr).
Update the instance type in the relevant module's variables.tf if you need to resize.
Why k3s Instead of Managed Kubernetes?
Managed Kubernetes services like EKS, AKS, and GKE charge $73+/month per cluster for the control plane alone. k3s on small VMs achieves the same Kubernetes conformance at near-zero cost.
More importantly, the same k3s binary runs identically on all four targets. This proves that the workload is truly cloud-agnostic.
Your root module's main.tf should look similar to this structure.
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
azurerm = {
source = "hashicorp/azurerm"
version = "~> 4.0"
}
google = {
source = "hashicorp/google"
version = "~> 6.0"
}
}
}
provider "aws" {
region = var.aws_region
}
provider "azurerm" {
features {}
}
provider "google" {
project = var.gcp_project
region = var.gcp_region
}
# Each module provisions a single VM with k3s installed
module "aws_cluster" {
source = "./modules/aws-cluster"
region = var.aws_region
k3s_version = var.k3s_version
ssh_public_key = file(var.ssh_public_key_path)
}
module "azure_cluster" {
source = "./modules/azure-cluster"
location = var.azure_location
k3s_version = var.k3s_version
ssh_public_key = file(var.ssh_public_key_path)
}
module "gcp_cluster" {
source = "./modules/gcp-cluster"
project = var.gcp_project
region = var.gcp_region
k3s_version = var.k3s_version
ssh_public_key = file(var.ssh_public_key_path)
}What Does This Configuration Do?
The required_providers block tells OpenTofu which provider plugins to download from the OpenTofu Registry.
Each module block calls a child module that contains the cloud-specific logic for provisioning a VM. The root module passes shared variables (like k3s_version and ssh_public_key) down to each child module.
Each cloud module follows the same pattern: provision a VM, attach a firewall/security group, and run a startup script that installs k3s. Here is the core of the AWS module as an example.
# Security group allowing k3s API, SSH, and HTTP traffic
resource "aws_security_group" "k3s" {
name = "columbia-k3s-sg"
description = "Allow k3s API, SSH, and HTTP"
ingress {
from_port = 6443
to_port = 6443
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
# EC2 instance with k3s installed via user_data startup script
resource "aws_instance" "k3s" {
ami = data.aws_ami.ubuntu.id
instance_type = "t3.small"
vpc_security_group_ids = [aws_security_group.k3s.id]
key_name = aws_key_pair.k3s.key_name
user_data = <<-EOF
#!/bin/bash
curl -sfL https://get.k3s.io | sh -s - server --tls-san $(curl -s http://169.254.169.254/latest/meta-data/public-ipv4)
EOF
tags = {
Name = "columbia-aws-k3s"
}
}What Is the TLS SAN Flag?
The --tls-san flag adds the VM's public IP address to the k3s server's TLS certificate as a Subject Alternative Name.
Without this, your local kubectl would reject the connection with a certificate error when you try to manage the cluster remotely.
- Verify that the Azure and GCP modules follow the same pattern with their respective resources (azurerm_linux_virtual_machine and google_compute_instance).
- Commit the generated files to your repository.
cd infra
git add .
git commit -m "Add OpenTofu multi-cloud modules for AWS, Azure, GCP"Plan and Apply the Infrastructure
OpenTofu's workflow has three stages: init downloads provider plugins, plan previews what will be created, and apply executes the changes.
- Initialize OpenTofu to download the AWS, Azure, and GCP provider plugins.
tofu initWhat Should I See?
You should see output confirming that OpenTofu downloaded three provider plugins: hashicorp/aws, hashicorp/azurerm, and hashicorp/google.
The final line should read "OpenTofu has been successfully initialized!"
- Run the plan to preview the resources OpenTofu will create.
tofu plan -var="gcp_project=[[GCP_PROJECT="your-gcp-project-id"]]"What Does This Command Do?
The tofu plan command reads your configuration and shows exactly what resources will be created, modified, or destroyed without making any changes.
The -var flag passes your GCP project ID since it has no default value. You should see a plan to create approximately 6-10 resources per cloud provider (VMs, security groups, network interfaces, SSH keys).
- Review the plan output in the Cursor terminal.
- Confirm that you see resources for all three clouds: an aws_instance, an azurerm_linux_virtual_machine, and a google_compute_instance.
Seeing Errors in the Plan Output?
Copy the full error message from the terminal and paste it into Cursor Composer. Composer can diagnose provider authentication issues, missing variable values, and syntax errors in your HCL files.
Common issues include expired cloud CLI sessions (re-run your login commands from Step 1) and missing SSH key files. Help me debug my OpenTofu plan errors.
- Apply the infrastructure once the plan looks correct.
tofu apply -var="gcp_project=[[GCP_PROJECT="your-gcp-project-id"]]"- Type yes when prompted to confirm the apply.
OpenTofu will now create all three VMs across AWS, Azure, and GCP. The startup scripts install k3s automatically on each VM. This typically takes 2-3 minutes to complete.
What Should I See?
The apply output shows each resource being created with a progress indicator. When complete, you should see "Apply complete! Resources: X added, 0 changed, 0 destroyed."
The outputs section displays the public IP addresses for each VM. Save these IP addresses. You will need them in the next substep.
Your Cloud Resources Are Now Running
From this point forward, your three cloud VMs are incurring charges (AWS t3.small at ~$0.0208/hr, Azure B1s at ~$0.0104/hr, GCP e2-micro at $0.00 on the always-free tier).
Make sure your $1 billing alerts from Step 1 are active. When you are done with the project, run tofu destroy to remove everything.
Verify All Four Clusters
You now have four separate Kubernetes clusters: one local (columbia-onprem) and three in the cloud. To manage them all from a single terminal, you need to merge their kubeconfig files into one.
- SSH into each cloud VM and retrieve the k3s kubeconfig file.
# Retrieve kubeconfig from the AWS VM
ssh -i ~/.ssh/id_rsa ubuntu@[[AWS_VM_IP="your AWS VM public IP"]] "sudo cat /etc/rancher/k3s/k3s.yaml" > /tmp/kubeconfig-aws.yaml
# Retrieve kubeconfig from the Azure VM
ssh -i ~/.ssh/id_rsa ubuntu@[[AZURE_VM_IP="your Azure VM public IP"]] "sudo cat /etc/rancher/k3s/k3s.yaml" > /tmp/kubeconfig-azure.yaml
# Retrieve kubeconfig from the GCP VM
ssh -i ~/.ssh/id_rsa ubuntu@[[GCP_VM_IP="your GCP VM public IP"]] "sudo cat /etc/rancher/k3s/k3s.yaml" > /tmp/kubeconfig-gcp.yamlWhat Does This Command Do?
k3s stores its kubeconfig at /etc/rancher/k3s/k3s.yaml on the server. These commands SSH into each VM, read that file, and save it locally so you can merge them.
The raw kubeconfig files all point to 127.0.0.1:6443 as the server address and use default as the cluster name. You need to update each file to use the VM's public IP and a unique context name before merging.
- Use Cursor Composer to generate a shell script that automates the kubeconfig merge.
Generate a bash script called merge-kubeconfigs.sh that:
1. Takes the three kubeconfig files from /tmp/ (kubeconfig-aws.yaml, kubeconfig-azure.yaml, kubeconfig-gcp.yaml)
2. Updates each file:
- Replace the server address 127.0.0.1 with the VM's actual public IP
- Rename the cluster, context, and user to columbia-aws, columbia-azure, columbia-gcp respectively
3. Gets the local k3s kubeconfig from /etc/rancher/k3s/k3s.yaml and renames its context to columbia-onprem
4. Merges all four kubeconfigs into ~/.kube/config using KUBECONFIG env var and kubectl config view --merge --flatten
5. Prints all available contexts with kubectl config get-contexts
The script should accept the three VM IPs as arguments.- Review the generated script in Cursor's diff view.
- Run the merge script with your three VM IP addresses.
chmod +x merge-kubeconfigs.sh
./merge-kubeconfigs.sh [[AWS_VM_IP="your AWS VM public IP"]] [[AZURE_VM_IP="your Azure VM public IP"]] [[GCP_VM_IP="your GCP VM public IP"]]- Verify each cluster is healthy by checking its nodes.
kubectl get nodes --context columbia-onprem
kubectl get nodes --context columbia-aws
kubectl get nodes --context columbia-azure
kubectl get nodes --context columbia-gcpWhat Should I See?
Each command should return a single node in Ready status. The node name will be the hostname of each VM.
If any cluster shows NotReady or fails to connect, k3s may still be installing. Wait 1-2 minutes and try again.
I see Ready nodes on all four clusters
All four clusters in the Columbia HHS system boundary are online and accessible from your local terminal. You have a fully provisioned multi-cloud Kubernetes platform.
A cluster is not connecting
There are several common causes for cluster connection failures.
- Verify the VM is running by checking your cloud console.
- SSH into the failing VM and check if k3s is running with sudo systemctl status k3s.
- Check the k3s logs with sudo journalctl -u k3s -f for out-of-memory errors.
- Verify that port 6443 is open in the VM's security group or firewall rules.
- Confirm the kubeconfig has the correct public IP (not 127.0.0.1).
If the VM is running out of memory (common on Azure B1s and GCP e2-micro), you need to resize the VM. Update the instance type in your OpenTofu module and run tofu apply again.
Help me troubleshoot my k3s cluster connection.
Your multi-cloud system boundary is fully provisioned. Four Kubernetes clusters are running across an on-prem simulation and three cloud providers, all managed from a single kubeconfig. Next up, you will install Argo CD on the hub cluster and register all four clusters for centralized GitOps management.
Bootstrap Argo CD and Register All Clusters
Your four clusters are running, but right now each one is an island. To enforce consistent deployments across the entire system boundary, you need a single control plane that watches your Git repository and pushes changes to every cluster automatically. Argo CD is that control plane. In this step, you will install Argo CD on the on-prem hub cluster, register all three cloud clusters as deploy targets, and use the App of Apps pattern to deploy the benefits-portal application to every environment from a single Git push.
This step enforces Configuration Management controls CM-2 (Baseline Configuration) and CM-6 (Configuration Settings) from the NIST 800-53 Rev. 5 framework. Every deployment originates from version-controlled manifests in Git, creating an auditable trail of what changed, when, and by whom.
In This Step, Get Ready To:
- Install Argo CD on the columbia-onprem hub cluster and access its UI.
- Register all three cloud clusters as Argo CD deploy targets.
- Deploy the benefits-portal application to all four clusters using the App of Apps pattern.
Install Argo CD on the Hub Cluster
Argo CD runs inside a Kubernetes cluster and watches a Git repository for changes. When it detects a difference between what Git says and what the cluster has, it automatically reconciles the cluster to match Git. This is the core of GitOps. You will install it on the columbia-onprem cluster, which acts as the centralized hub for managing all four environments.
- Switch to the on-prem cluster context.
kubectl config use-context columbia-onprem- Create the argocd namespace and install Argo CD v3.3.7 using the standard install manifest.
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yamlWhat does this command do?
The first command creates a dedicated namespace for Argo CD's components. The second command downloads and applies the official Argo CD manifest, which installs the API server, repo server, application controller, and other components into that namespace.
- Wait for all Argo CD pods to reach the Running state. This typically takes 1-2 minutes.
kubectl get pods -n argocd -wWhat should I see?
You should see pods for argocd-server, argocd-repo-server, argocd-application-controller, and others all showing 1/1 Running. Press Ctrl+C to stop watching once all pods are ready.
- Retrieve the initial admin password for Argo CD.
argocd admin initial-password -n argocdPro tip: save this password
Copy the password output and save it somewhere safe. You will need it to log in to the Argo CD CLI and UI. The username is admin.
- Port-forward the Argo CD server so you can access the UI from your Windows browser.
kubectl port-forward svc/argocd-server -n argocd 8080:443 &What does this command do?
This forwards port 8080 on your local machine to the Argo CD server's HTTPS port (443) inside the cluster. The & at the end runs it in the background so you can keep using your terminal.
- Open your Windows browser and navigate to https://localhost:8080.
- Accept the self-signed certificate warning.
- Log in with username admin and the password you retrieved.
You should see the Argo CD dashboard with no applications listed yet. This is your GitOps control plane.
- Log in to the Argo CD CLI from your terminal as well.
argocd login localhost:8080 --username admin --password your-admin-password-here --insecureWhat does this command do?
This authenticates the argocd CLI against your local Argo CD server. The --insecure flag skips TLS certificate verification since you are using a self-signed certificate. Replace your-admin-password-here with the password you retrieved earlier.
Port-forward not working?
If you get a connection refused error, make sure all Argo CD pods are fully running by checking kubectl get pods -n argocd again. If a pod is stuck in CrashLoopBackOff, your WSL2 instance may be running low on memory. Help me troubleshoot Argo CD pod failures on k3s in WSL2.
Register the Three Cloud Clusters
Argo CD can deploy to any Kubernetes cluster it knows about. The columbia-onprem cluster is already registered automatically because Argo CD runs inside it (this is called the "in-cluster" target). You need to explicitly register the three cloud clusters so Argo CD can push deployments to them.
- Register the columbia-aws cluster with Argo CD.
argocd cluster add columbia-aws --yesWhat does this command do?
The argocd cluster add command reads the kubeconfig context you specify and creates a service account on that remote cluster. Argo CD then uses this service account to deploy resources. The --yes flag skips the confirmation prompt.
- Register the columbia-azure cluster.
argocd cluster add columbia-azure --yes- Register the columbia-gcp cluster.
argocd cluster add columbia-gcp --yes- Verify all clusters appear in the Argo CD cluster list.
argocd cluster listWhat should I see?
You should see four entries in the list. The columbia-onprem cluster appears as https://kubernetes.default.svc (the in-cluster default). The three cloud clusters appear with their external API server URLs. All should show a Successful connection status.
Cluster registration failing?
If argocd cluster add fails with a connection error, the most common cause is that the kubeconfig for that cluster uses an internal IP or localhost address instead of the VM's public IP. When you installed k3s on the cloud VMs, the --tls-san flag should have included the external IP. Verify your kubeconfig's server field points to the correct public IP and port 6443. Help me fix Argo CD cluster registration errors.
Deploy the Benefits Portal With the App of Apps Pattern
The App of Apps pattern is Argo CD's standard approach for bootstrapping many applications into multiple clusters. You create one parent Application that points to a directory of child Application manifests. Each child Application targets a specific cluster. When you sync the parent, Argo CD creates all the children, and each child deploys the actual workload to its target cluster.
You will use Cursor Composer in parallel mode to generate the manifests. One agent generates the Kubernetes Deployment and Service for the benefits-portal. Another agent generates the Argo CD Application manifests pointing to each cluster.
- Open Cursor Composer and enter the following prompt to generate the benefits-portal Kubernetes manifests.
Generate Kubernetes manifests for a benefits-portal application in the columbia-modernization repo.
Create the file apps/benefits-portal/base/deployment.yaml:
- Deployment named benefits-portal in namespace hhs-prod
- 1 replica
- Container image: nginx:stable-alpine
- Container port 80
- Add a label app: benefits-portal
Create the file apps/benefits-portal/base/service.yaml:
- Service named benefits-portal in namespace hhs-prod
- Type ClusterIP, port 80, targeting port 80
- Selector: app: benefits-portal
Create the file apps/benefits-portal/base/namespace.yaml:
- Namespace named hhs-prodWhy nginx for the benefits portal?
The benefits-portal in this lab is a lightweight nginx container serving a static page with mock Medicaid eligibility data. The goal is to prove that the same workload deploys identically across all four clusters. The application itself is intentionally simple so you can focus on the infrastructure and GitOps patterns.
- In a second Composer agent (use parallel mode), enter the following prompt to generate the Argo CD Application manifests.
Generate Argo CD Application manifests for the App of Apps pattern in the columbia-modernization repo.
Create apps/parent-app.yaml:
- Application named columbia-apps in namespace argocd
- Source repo: https://github.com/YOUR_GITHUB_USERNAME/columbia-modernization.git
- Source path: apps/children
- Destination: https://kubernetes.default.svc (in-cluster)
- Destination namespace: argocd
- Sync policy: automated with self-heal and prune enabled
Create four child Application manifests in apps/children/:
1. apps/children/benefits-portal-onprem.yaml
- Name: benefits-portal-onprem
- Source path: apps/benefits-portal/base
- Destination server: https://kubernetes.default.svc
- Destination namespace: hhs-prod
- Automated sync with self-heal and prune
2. apps/children/benefits-portal-aws.yaml
- Name: benefits-portal-aws
- Source path: apps/benefits-portal/base
- Destination server: USE THE columbia-aws CLUSTER URL FROM argocd cluster list
- Destination namespace: hhs-prod
- Automated sync with self-heal and prune
3. apps/children/benefits-portal-azure.yaml
- Name: benefits-portal-azure
- Source path: apps/benefits-portal/base
- Destination server: USE THE columbia-azure CLUSTER URL FROM argocd cluster list
- Destination namespace: hhs-prod
- Automated sync with self-heal and prune
4. apps/children/benefits-portal-gcp.yaml
- Name: benefits-portal-gcp
- Source path: apps/benefits-portal/base
- Destination server: USE THE columbia-gcp CLUSTER URL FROM argocd cluster list
- Destination namespace: hhs-prod
- Automated sync with self-heal and prunePro tip: get cluster URLs first
Before accepting the generated manifests, run argocd cluster list and copy the server URLs for each cloud cluster. Replace the placeholder URLs in the generated child Application manifests with the actual URLs. The on-prem cluster always uses https://kubernetes.default.svc.
- Review the generated files in Cursor's diff view.
- Verify that each child Application's spec.destination.server field points to the correct cluster URL.
- Verify that all Application manifests reference the same Git repository URL.
- Accept the changes, then commit and push to GitHub.
git add .
git commit -m "Add benefits-portal manifests and App of Apps configuration"
git push origin main- Create the parent Application in Argo CD.
kubectl apply -f apps/parent-app.yaml- Wait for Argo CD to sync the parent app and create all child applications. This typically takes 30-60 seconds.
- Check the sync status of all applications.
argocd app listWhat should I see?
You should see five applications: the parent columbia-apps and four children (benefits-portal-onprem, benefits-portal-aws, benefits-portal-azure, benefits-portal-gcp). All should show Synced and Healthy status.
- Verify the benefits-portal pods are running on all four clusters.
kubectl get pods -n hhs-prod --context columbia-onprem
kubectl get pods -n hhs-prod --context columbia-aws
kubectl get pods -n hhs-prod --context columbia-azure
kubectl get pods -n hhs-prod --context columbia-gcpEach command should show a benefits-portal pod in the Running state. The same container image is now running identically across your on-prem data center simulation and three cloud providers.
Why the App of Apps Pattern?
You could deploy each application manually with argocd app create commands. The App of Apps pattern is better because the parent Application manifest lives in Git. If someone adds a new cluster or a new application, they add a YAML file and push. Argo CD picks up the change automatically. This means your cluster fleet configuration is version-controlled and auditable, which is exactly what CM-2 (Baseline Configuration) requires.
Applications stuck in OutOfSync or Unknown?
If child applications show OutOfSync or Unknown status, check two things. First, make sure you pushed your manifests to GitHub and the repository URL in the Application manifests matches your actual repo. Second, verify the cluster URLs in the child manifests match the output of argocd cluster list exactly. Even a trailing slash difference can cause a mismatch. Help me debug Argo CD sync issues.
Your GitOps control plane is fully operational. Every change to the benefits-portal manifests in Git will automatically propagate to all four clusters. Next up, you will configure disaster recovery with Velero so the platform can survive a cloud provider outage.
Configure COOP Disaster Recovery with Velero
Your benefits portal is running across all four clusters, managed by a single Argo CD control plane. That handles the Configuration Management side of compliance. Now you need to answer a harder question: what happens when an entire cloud provider goes down?
In government IT, the answer to that question is a Continuity of Operations Plan (COOP). A COOP defines how a system recovers at an alternate processing site when the primary site becomes unavailable. In this step, you will implement and test a real COOP procedure using Velero for cross-cluster disaster recovery. This satisfies two critical NIST 800-53 controls: CP-7 (Alternate Processing Site) and CP-10 (Information System Recovery and Reconstitution).
In This Step, Get Ready To:
- Install Velero on two clusters and configure shared S3 backup storage.
- Run a full COOP disaster recovery drill following the sequence in Diagram 3.
- Document RTO and RPO metrics and validate them against government benchmarks.
Install Velero and Configure Shared Backup Storage
Velero backs up Kubernetes resources and restores them to any cluster that shares the same backup storage. For cross-cluster disaster recovery, both the primary site (columbia-aws) and the alternate processing site (columbia-azure) need Velero installed and pointed at the same S3 bucket.
Start by creating the shared backup bucket on AWS. This bucket acts as the single source of truth for all COOP backups.
- Create an S3 bucket for Velero backups.
aws s3api create-bucket \
--bucket [[VELERO_BUCKET="columbia-velero-backups-yourname"]] \
--region us-east-1What does this command do?
This creates an S3 bucket that both clusters will use as their BackupStorageLocation. S3 buckets must have globally unique names, so include your name or a random string. The bucket falls under the AWS free tier (5 GB for 12 months on new accounts).
Next, create an IAM user and policy so Velero can read and write to the bucket. Use Cursor Composer to generate the IAM policy if you are unsure of the exact permissions.
- Open Cursor Composer and prompt it to generate the Velero IAM policy and credentials file.
Generate an AWS IAM policy for Velero v1.18.0 that grants
read/write access to the S3 bucket I just created. Also generate
the credentials file format that Velero expects at install time.
The bucket is in us-east-1.- Create the IAM user and attach the policy Composer generated.
- Save the access key and secret key into a credentials file at ~/.velero/credentials-velero.
Do not commit credentials to Git
The credentials file contains your AWS access key and secret key. Add .velero/ to your .gitignore immediately. Never commit cloud credentials to a repository.
Now install Velero v1.18.0 on the columbia-aws cluster first. This is the primary site where backups originate.
- Switch to the columbia-aws context.
kubectl config use-context columbia-aws- Install Velero with the AWS plugin, pointing to your S3 bucket.
velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.11.0 \
--bucket [[VELERO_BUCKET="columbia-velero-backups-yourname"]] \
--backup-location-config region=us-east-1 \
--secret-file ~/.velero/credentials-velero \
--use-node-agent \
--default-volumes-to-fs-backupWhat does this command do?
This installs the Velero server components into the velero namespace on your current cluster. The --provider aws flag tells Velero to use the AWS plugin for S3 access. The --use-node-agent and --default-volumes-to-fs-backup flags enable File System Backup, which is required for cross-provider volume migration since Velero cannot natively migrate volume snapshots between cloud providers.
- Wait for the Velero pods to reach Running status.
kubectl get pods -n velero- Verify the backup storage location is available.
velero backup-location getYou should see a default backup storage location with phase Available.
Now install Velero on the columbia-azure cluster. This is the alternate processing site that will receive the restored workloads during a disaster.
- Switch to the columbia-azure context.
kubectl config use-context columbia-azure- Run the same Velero install command, pointing to the same S3 bucket.
velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.11.0 \
--bucket [[VELERO_BUCKET="columbia-velero-backups-yourname"]] \
--backup-location-config region=us-east-1 \
--secret-file ~/.velero/credentials-velero \
--use-node-agent \
--default-volumes-to-fs-backupWhy does the Azure cluster use the AWS plugin?
Both clusters use the AWS plugin because the shared backup storage is an S3 bucket. Velero does not care which cloud the cluster runs on. It only needs the correct plugin for the storage backend. This is what makes cross-cloud disaster recovery possible with a single shared bucket.
- Verify the Velero pods are running and the backup storage location is Available on the Azure cluster.
kubectl get pods -n velero
velero backup-location getBackup storage location shows Unavailable?
This usually means the IAM credentials are incorrect or the bucket name does not match. Check the Velero pod logs with kubectl logs deployment/velero -n velero for specific error messages. Paste the error into Cursor Composer for diagnosis.
Create Backup Schedule and Run the DR Drill
This is the core of the COOP procedure. You will follow the exact sequence from Diagram 3 (COOP Disaster Recovery Flow): create a scheduled backup on the primary site, trigger it, declare a simulated disaster, and restore the workload on the alternate processing site.
- Switch back to the columbia-aws context (the primary site).
kubectl config use-context columbia-aws- Create a backup schedule called coop-daily-backup that captures the hhs-prod namespace.
velero schedule create coop-daily-backup \
--schedule "0 7 * * *" \
--include-namespaces hhs-prod \
--ttl 720hWhat does this command do?
This creates a scheduled backup that runs daily at 7:00 AM UTC. The --include-namespaces hhs-prod flag limits the backup to only the namespace containing the benefits portal. The --ttl 720h sets a 30-day retention period (the Velero default). In production, this schedule would align with the department's RPO target.
You do not want to wait until 7:00 AM UTC for the schedule to trigger. Create an immediate on-demand backup to test with.
- Trigger an immediate backup.
velero backup create coop-drill-backup \
--include-namespaces hhs-prod \
--waitWhat does the --wait flag do?
The --wait flag blocks the terminal until the backup completes. This makes it easy to time the backup duration for your RTO/RPO calculations.
- Verify the backup completed successfully.
velero backup describe coop-drill-backupThe output should show Phase: Completed and list the resources backed up from the hhs-prod namespace. Note the backup completion timestamp. You will need it for RTO/RPO calculations.
Now simulate the disaster. Declare columbia-aws as unavailable and begin the restore on the alternate processing site.
- Record the current time as your disaster declaration timestamp.
date -u +"%Y-%m-%dT%H:%M:%SZ"
# Record this timestamp as DISASTER_DECLARED- Switch to the columbia-azure context (the alternate processing site).
kubectl config use-context columbia-azure- Set the backup storage location to ReadOnly to prevent any writes during the restore.
kubectl patch backupstoragelocation default \
--namespace velero \
--type merge \
--patch '{"spec":{"accessMode":"ReadOnly"}}'Why set the storage location to ReadOnly?
During a restore, you want to prevent the alternate site from creating new backups that could overwrite or conflict with the backup you are restoring from. Setting the storage location to ReadOnly ensures the restore site can only read existing backups, not modify them. This is a standard Velero disaster recovery procedure.
Velero syncs backup metadata from the shared bucket at a default interval of 1 minute. Wait briefly for the backup to appear on the Azure cluster.
- Check that the backup from columbia-aws is visible on the Azure cluster.
velero backup getYou should see coop-drill-backup listed with Phase: Completed. If it does not appear yet, wait 60 seconds and run the command again.
- Restore from the backup.
velero restore create coop-drill-restore \
--from-backup coop-drill-backup \
--wait- Verify the benefits-portal pods are running on columbia-azure.
kubectl get pods -n hhs-prodYou should see the benefits-portal pods in Running status, matching the same deployment configuration from the AWS cluster.
- Record the current time as your service restoration timestamp.
date -u +"%Y-%m-%dT%H:%M:%SZ"
# Record this timestamp as SERVICE_RESTORED- Revert the backup storage location back to ReadWrite.
kubectl patch backupstoragelocation default \
--namespace velero \
--type merge \
--patch '{"spec":{"accessMode":"ReadWrite"}}'Pro tip: verify the restore details
Run velero restore describe coop-drill-restore to see exactly which resources were restored, any warnings, and the restore duration. This information goes directly into your COOP report.
Document RTO/RPO and Validate Against Benchmarks
A COOP procedure is only as good as its documented metrics. RTO (Recovery Time Objective) measures the time from disaster declaration to service restoration. RPO (Recovery Point Objective) measures the maximum acceptable data loss, calculated as the time between the last backup and the disaster.
- Calculate your RTO by subtracting the disaster declaration timestamp from the service restoration timestamp.
- Calculate your RPO by subtracting the backup completion timestamp from the disaster declaration timestamp.
What are typical government benchmarks?
For state government systems at the Moderate Impact level, typical targets are an RTO under 4 hours and an RPO under 1 hour. Your lab drill should complete well under these thresholds since the benefits portal is a lightweight stateless deployment. In production, larger datasets and persistent volumes would increase these times.
Use Cursor Composer to generate the COOP report document with all the data you collected.
- Open Cursor Composer and prompt it to generate the report.
Generate a coop-report.md for the Columbia HHS modernization project.
Include these sections:
1. COOP Drill Summary - date, participants, primary site (columbia-aws),
alternate site (columbia-azure)
2. Timeline - backup start, backup complete, disaster declared,
restore initiated, restore complete, service verified
3. RTO Calculation - show the math, compare against 4-hour target
4. RPO Calculation - show the math, compare against 1-hour target
5. Pass/Fail Assessment - did we meet both targets?
6. Observations and Recommendations - what would change in production?
Use the timestamps I recorded during the drill.
Reference NIST 800-53 controls CP-7 and CP-10 in the summary.- Review the generated coop-report.md in Cursor's diff view.
- Replace the placeholder timestamps with your actual recorded values.
- Commit and push the report to your columbia-modernization repository.
Extra for Experts: What about stateful workloads?
The benefits portal in this lab is stateless (nginx serving static content), which makes backup and restore straightforward. In production, stateful workloads with persistent volumes require Velero's File System Backup mode. Velero does not natively support cross-provider persistent volume snapshot migration. The --use-node-agent flag you included during install enables this capability for when stateful workloads are added in later migration waves.
Your COOP disaster recovery procedure is tested and documented. You now have concrete evidence that the Columbia HHS platform can recover at an alternate processing site within government-mandated timeframes. Next up, you will use Cursor Composer to generate the GovRAMP compliance crosswalk and migration wave plan.
Author the GovRAMP Crosswalk and Migration Wave Plan
You have a working multi-cloud platform with tested disaster recovery. Now it is time to produce the compliance and planning artifacts that make this platform presentable to state government decision-makers. In real modernization programs, these documents take weeks of manual effort. Cursor IDE Composer 2 with parallel mode lets you generate multiple documents simultaneously, compressing that timeline from weeks to minutes.
In This Step, Get Ready To:
- Generate a GovRAMP Moderate controls crosswalk mapping 60 Core controls to your infrastructure artifacts.
- Create a three-wave migration plan with data classification, rollback procedures, and a Mermaid Gantt chart.
- Build a portability scorecard rating your platform across five dimensions and generate the system boundary diagram.
Generate the GovRAMP Moderate Controls Crosswalk
A controls crosswalk maps each required security control to the specific artifact in your system that satisfies it. GovRAMP Core validates 60 foundational controls from NIST 800-53 Rev. 5 at the Moderate Impact baseline, mapped to the MITRE ATT&CK Framework. The GovRAMP PMO conducts the Core assessment directly, so no 3PAO is required for Core status. Your crosswalk needs to reference real file paths and configurations from your repository so a reviewer can trace each control to its implementation.
What is GovRAMP?
GovRAMP is the 2025 rebrand of StateRAMP. It provides a standardized security verification program for cloud service providers working with state and local governments. The program upgraded from NIST 800-53 Rev. 4 to Rev. 5. You may still see "StateRAMP" in older RFPs and documentation.
- Open Cursor Composer 2 and give it access to the full columbia-modernization repository so it can reference actual file paths.
- Prompt Composer to generate a crosswalk mapping the 60 GovRAMP Core controls to your specific artifacts.
Use a prompt like the following to give Composer the full context it needs:
Generate a GovRAMP Moderate controls crosswalk in Markdown table format.
Map the 60 GovRAMP Core controls (NIST 800-53 Rev. 5, Moderate Impact baseline)
to the artifacts in this repository. For each control:
- List the Control ID (e.g., AC-1, CM-2, CP-7)
- List the Control Name
- Identify the specific artifact that satisfies it
(OpenTofu module, Argo CD config, Velero policy, or documented procedure)
- Reference the exact file path in the repo
- Note the implementation status (Implemented, Partially Implemented, Planned)
Include these control families at minimum:
- AC (Access Control)
- CM (Configuration Management)
- CP (Contingency Planning)
- SC (System and Communications Protection)
- IA (Identification and Authentication)
- AU (Audit and Accountability)
Save the output as govRAMP-crosswalk.md in the repository root.What does this prompt do?
This prompt instructs Composer to scan your entire repository and match each NIST control to a real file or configuration. For example, CM-2 (Baseline Configuration) maps to your Argo CD App of Apps manifests because GitOps enforces a single source of truth for all cluster configurations. CP-7 (Alternate Processing Site) maps to your Velero cross-cluster restore procedure.
- Review the generated govRAMP-crosswalk.md in Cursor's editor.
- Run a quality check by prompting Composer to identify any controls without a mapped artifact and suggest remediation.
Review govRAMP-crosswalk.md. Identify any controls that are listed as
"Planned" or have no mapped artifact. For each gap, suggest a specific
remediation action I could take in this lab environment to move the
control to "Implemented" status.Pro tip: expect gaps
A lab environment will not satisfy every control. Controls like AU-6 (Audit Review) or IA-5 (Authenticator Management) require organizational policies that go beyond infrastructure code. Marking these as "Planned" with a remediation note is exactly what a 3PAO reviewer expects to see in an early-stage crosswalk.
- Commit the crosswalk to your repository.
git add govRAMP-crosswalk.md
git commit -m "Add GovRAMP Moderate controls crosswalk"
git pushCreate the Migration Wave Plan
A migration wave plan sequences which systems move to the cloud and when. The Columbia HHS department has three systems to modernize across three waves. Wave 1 (the Benefits Portal) is what you built in this lab. Waves 2 and 3 are documented but not built. Each wave needs data sensitivity classification per NIST SP 800-60, interconnection security agreements, estimated migration velocity, and rollback procedures.
Use Composer's parallel mode to run this prompt at the same time as the crosswalk gap check from the previous substep. One agent refines the crosswalk while another drafts the migration plan.
- Open a new Composer agent in parallel mode.
- Prompt it to generate the three-wave migration plan.
Generate a three-wave migration plan for the Columbia HHS department.
Save it as migration-wave-plan.md.
Wave 1: Citizen Benefits Portal (built in this lab)
- System of Record: Medicaid eligibility checks, SNAP benefit status
- Data sensitivity: Moderate Impact (PII/PHI) per NIST SP 800-60
- Migration approach: Multi-cloud k3s with GitOps and COOP DR
- Rollback procedure: Revert Argo CD to previous Git commit
Wave 2: Eligibility Rules Engine (documented, not built)
- System of Record: Business rules for benefit determination
- Data sensitivity: Moderate Impact per NIST SP 800-60
- Requires ISA negotiation with CSPs
- Estimated timeline: 4 months after Wave 1 cutover
Wave 3: Data Warehouse and Reporting (documented, not built)
- System of Record: Historical claims data, analytics
- Data sensitivity: Moderate Impact per NIST SP 800-60
- Requires schema migration planning
- Estimated timeline: 6 months after Wave 2 cutover
For each wave, include:
- System of Record description
- Data sensitivity classification (NIST SP 800-60)
- Interconnection Security Agreements (ISAs) needed
- Estimated migration velocity
- Rollback procedures
- Dependencies on previous waves
Embed the following Mermaid Gantt chart directly in the document:
```mermaid
gantt
title Columbia HHS Modernization - Migration Wave Plan
dateFormat YYYY-MM
axisFormat %b %Y
section Wave 1 - Benefits Portal
Discovery and assessment :w1a, 2026-01, 1M
IaC module development :w1b, after w1a, 1M
GitOps pipeline setup :w1c, after w1a, 1M
COOP DR configuration :w1d, after w1b, 1M
GovRAMP crosswalk and ATO prep :w1e, after w1c, 1M
Production cutover :milestone, after w1d, 0d
section Wave 2 - Eligibility Rules Engine
Discovery and assessment :w2a, after w1e, 1M
Data classification (NIST 800-60):w2b, after w2a, 1M
ISA negotiation with CSPs :w2c, after w2a, 2M
Migration and testing :w2d, after w2b, 2M
Production cutover :milestone, after w2d, 0d
section Wave 3 - Data Warehouse
Discovery and assessment :w3a, after w2d, 1M
Data sensitivity review :w3b, after w3a, 1M
Schema migration planning :w3c, after w3a, 2M
Migration and testing :w3d, after w3b, 3M
Production cutover :milestone, after w3d, 0d
```Why document waves you are not building?
Real government modernization programs require a full migration roadmap before any single system gets its Authority to Operate (ATO). Decision-makers need to see the complete picture, including timelines, dependencies, and risk assessments for future waves. Documenting Waves 2 and 3 demonstrates strategic planning capability, not just technical execution.
- Review the generated migration-wave-plan.md and verify the Mermaid Gantt chart renders correctly in Cursor's Markdown preview.
- Commit and push the migration plan.
git add migration-wave-plan.md
git commit -m "Add three-wave migration plan with Gantt chart"
git pushGenerate the Portability Scorecard and System Boundary Diagram
The portability scorecard quantifies how cloud-agnostic your platform actually is. Instead of claiming portability, you measure it across five concrete dimensions. The system boundary diagram documents every environment and data flow using NIST terminology, which is the format government reviewers expect.
- Prompt Composer to create the portability scorecard.
Generate a portability scorecard as portability-scorecard.md.
Rate the platform across these five dimensions using a 1-5 scale
(1 = fully vendor-locked, 5 = fully portable):
1. Cloud-Agnostic IaC Coverage
- What percentage of resources are managed by OpenTofu vs.
provider-specific tools?
- Reference the infra/ directory structure.
2. DR Readiness
- Compare tested RTO/RPO from coop-report.md against targets
(RTO < 4 hours, RPO < 1 hour).
3. GitOps Drift Detection
- Report Argo CD sync status across all four clusters.
- Are all clusters in sync with the Git source of truth?
4. Vendor-Specific API Surface Area
- How many provider-specific APIs does the benefits-portal
application depend on? (Target: zero)
5. Compliance Coverage
- What percentage of GovRAMP Core controls are mapped in
govRAMP-crosswalk.md?
- Break down by Implemented vs. Planned.
Include a summary score and a list of concrete improvements
that would increase each dimension's rating.
After generating the scorecard, review it against real production
multi-cloud criteria and suggest improvements.What does each dimension measure?
IaC coverage measures how much of your infrastructure is defined in code versus manually created. DR readiness compares your actual recovery times against government benchmarks. GitOps drift detection checks whether all clusters match the declared state in Git.
Vendor API surface area counts how many cloud-specific APIs your application code calls directly. Compliance coverage tracks your progress toward full GovRAMP Core control mapping.
- Now generate the system boundary diagram as a standalone file.
Generate system-boundary.md containing the system boundary diagram
for the Columbia HHS modernization platform.
Use this Mermaid diagram as the base:
```mermaid
graph TB
subgraph SB["SYSTEM BOUNDARY -- State of Columbia HHS"]
subgraph ONPREM["columbia-onprem (On-Prem Data Center Sim / WSL2)"]
K3S_HUB["k3s Hub Cluster"]
ARGO["Argo CD v3.3.7<br/>GitOps Control Plane"]
VELERO_HUB["Velero CLI"]
BP_ONPREM["benefits-portal<br/>hhs-prod namespace"]
end
subgraph AWS["columbia-aws (CSP A / AWS us-east-1)"]
K3S_AWS["k3s on EC2 t3.small"]
BP_AWS["benefits-portal<br/>hhs-prod namespace"]
VELERO_AWS["Velero v1.18.0"]
end
subgraph AZURE["columbia-azure (CSP B / Azure eastus)"]
K3S_AZURE["k3s on Azure VM"]
BP_AZURE["benefits-portal<br/>hhs-prod namespace"]
VELERO_AZURE["Velero v1.18.0"]
end
subgraph GCP["columbia-gcp (CSP C / GCP us-central1)"]
K3S_GCP["k3s on GCE e2-micro"]
BP_GCP["benefits-portal<br/>hhs-prod namespace"]
end
end
subgraph DR["COOP Backup Storage"]
S3["AWS S3 Bucket<br/>Shared BackupStorageLocation"]
end
VELERO_AWS -->|backup| S3
VELERO_AZURE -->|restore| S3
```
Below the diagram, add annotations explaining each data flow
using NIST terminology:
- System Boundary (what it encompasses)
- CSP designations (A, B, C)
- COOP Backup Path (CP-7, CP-10 controls)
- GitOps Control Plane (CM-2, CM-6 controls)
- Data classification (Moderate Impact, PII/PHI)- Review both generated files in Cursor.
- Verify the Mermaid diagram in system-boundary.md renders correctly and matches the architecture you built.
- Commit and push both files.
git add portability-scorecard.md system-boundary.md
git commit -m "Add portability scorecard and system boundary diagram"
git pushComposer generating inaccurate control IDs?
AI models sometimes hallucinate NIST control IDs or descriptions. Always cross-check the generated control IDs against the NIST 800-53 Rev. 5 catalog. You will do a thorough validation in Step 6, but flag any obvious errors now. Help me verify my NIST control IDs.
Your compliance crosswalk, migration wave plan, portability scorecard, and system boundary diagram are now committed to the repository. These four documents, combined with the COOP report from Step 4, form a complete deliverable package. Next up, you will run a quality review across all deliverables to catch inconsistencies and validate everything against production standards.
Quality Review and ATO Readiness Check
You have built the infrastructure, deployed the application, tested disaster recovery, and authored the compliance documentation. Now you need to validate that everything holds together as a coherent deliverable set. In a real ATO process, a 3PAO reviewer would scrutinize every document for internal consistency, verify that referenced artifacts actually exist, and compare your claims against the NIST 800-53 Rev. 5 catalog. This step uses Cursor Composer's repository-wide context to perform that review automatically.
In This Step, Get Ready To:
- Run an ATO readiness review across all deliverables using Cursor Composer.
- Validate every factual claim, control ID, and tool version in your compliance documents.
- Generate a production readiness assessment comparing your lab platform against real government system benchmarks.
Run the ATO Readiness Review
An Authority to Operate (ATO) is the formal approval a government system needs before it can go live. Before requesting that approval, the platform team runs an internal readiness review to catch gaps before a 3PAO does. Cursor Composer can read your entire repository at once, making it ideal for cross-referencing documents that were generated in different steps.
- Open Cursor IDE and make sure you are in the columbia-modernization repository root.
- Open Composer and enter the following prompt.
Review all deliverables in this repository for ATO readiness. Specifically:
1. Check govRAMP-crosswalk.md for completeness against the 60 GovRAMP Core controls (NIST 800-53 Rev. 5, Moderate Impact baseline). Flag any controls without a mapped artifact.
2. Check coop-report.md for internal consistency: do the RTO/RPO numbers match the timestamps? Are the benchmark comparisons accurate?
3. Check migration-wave-plan.md for completeness: does each wave have data sensitivity classification, rollback procedures, and velocity estimates?
4. Check portability-scorecard.md for accuracy: do the scores reflect the actual state of the repository?
5. Verify that every file path referenced in any document actually exists in this repo.
6. Verify that all five Mermaid diagrams (system topology, GitOps flow, COOP DR sequence, migration wave Gantt, OpenTofu module structure) are present and accurately reflect the built system.
7. Identify gaps that a 3PAO reviewer would flag during a GovRAMP assessment.
For each issue found, provide the file name, the specific problem, and a suggested fix.What does this prompt do?
This prompt leverages Composer's ability to read across your full repository. It checks seven categories of quality issues that map directly to what a 3PAO reviewer would examine during a GovRAMP assessment. The output gives you a structured list of issues to fix before considering the deliverable set complete.
- Review Composer's output carefully. It will produce a list of issues organized by file.
- Fix any missing file path references by correcting the paths in the relevant documents.
- Fix any controls in govRAMP-crosswalk.md that Composer flagged as unmapped. For controls that genuinely cannot be satisfied by your lab artifacts, document them as gaps with a remediation plan.
- Fix any inconsistencies in coop-report.md between timestamps and calculated RTO/RPO values.
Pro tip: real gap documentation
In a real ATO process, not every control will be fully satisfied. The important thing is to document gaps honestly in a POA&M (Plan of Action and Milestones). A crosswalk that says "all controls satisfied" is less credible than one that identifies gaps and provides a remediation timeline.
Validate Claims and Cross-Check Sources
AI-generated compliance documents can contain plausible-sounding but incorrect control descriptions. A 3PAO reviewer will check every NIST control ID against the official catalog. You need to do the same before submitting.
- Open govRAMP-crosswalk.md in Cursor.
- Spot-check at least five control IDs manually. Verify that the control ID (e.g., CM-2, CP-7, CP-10, AC-1, SC-7) matches its description in the NIST 800-53 Rev. 5 catalog.
Where to find the NIST 800-53 Rev. 5 catalog
The official catalog is available at csrc.nist.gov/publications/detail/sp/800-53/rev-5/final. You can also use the NIST searchable database at csf.tools/reference/nist-sp-800-53/r5 for a faster lookup experience.
- Verify that all tool versions in your documents match what you actually installed. The correct versions are: OpenTofu v1.11.6, Argo CD v3.3.7, Velero v1.18.0, and k3s v1.35.3.
- Use Composer to run a targeted fact-check across the crosswalk and COOP report. Enter the following prompt.
Review govRAMP-crosswalk.md and coop-report.md for factual accuracy:
1. For each NIST 800-53 Rev. 5 control ID referenced, verify the control name and description match the official catalog. Flag any mismatches.
2. Check that all tool version numbers match: OpenTofu v1.11.6, Argo CD v3.3.7, Velero v1.18.0, k3s v1.35.3.
3. Verify that GovRAMP-specific claims are accurate: GovRAMP Core validates 60 controls, uses NIST 800-53 Rev. 5 (not Rev. 4), Core status is assessed by the GovRAMP PMO (not a 3PAO).
4. Flag any outdated references to StateRAMP that should say GovRAMP.
5. List every factual claim you cannot verify so I can check them manually.Why manual spot-checking matters
Composer can catch many inconsistencies, but it can also hallucinate control descriptions. The manual spot-check of five control IDs gives you a calibration point. If Composer's descriptions match the official catalog for those five, you can have higher confidence in the rest. If they do not match, you know to review more carefully.
- Fix any incorrect control descriptions, version mismatches, or outdated StateRAMP references that Composer or your manual review identified.
- Commit your corrected documents to the repository.
What Is the StateRAMP to GovRAMP Rebrand?
StateRAMP rebranded to GovRAMP in 2025. The program upgraded from NIST 800-53 Rev. 4 to Rev. 5 as part of the rebrand. You will still encounter "StateRAMP" in older RFPs, procurement documents, and vendor marketing materials. Your deliverables should use "GovRAMP" as the current name.
Generate the Industry Benchmark Comparison
Your platform is a lab simulation. A production system serving 2.4 million residents would have very different requirements for scalability, reliability, and cost. This final substep documents exactly what would need to change. This kind of honest assessment builds credibility with stakeholders.
- Open Composer and enter the following prompt to generate the production readiness assessment.
Generate a file called production-readiness-assessment.md that compares this lab platform against production requirements for a state government system serving 2.4 million residents. Evaluate four dimensions:
1. Scalability: Can this architecture handle production load? What changes are needed? Compare k3s on single-node VMs to managed Kubernetes (EKS at $73/month per cluster, AKS Standard at $73/month per cluster, GKE with $74.40/month credit). Discuss node pools, horizontal pod autoscaling, and load balancing.
2. Reliability: What SLA could this lab achieve vs. a 99.95% production target? What would need to change? Discuss multi-node clusters, health checks, automated failover, and monitoring.
3. Cost: Estimate production-grade managed Kubernetes costs across three clouds vs. the lab's near-zero cost. Include control plane fees, compute nodes, networking, and storage.
4. Performance: What bottlenecks exist in the current single-node k3s setup? What would production networking look like?
End with a concrete improvement checklist ranked by priority (critical, high, medium, low). Each item should describe what to change, why, and the estimated effort.What does this assessment do?
This document bridges the gap between your lab and a real deployment. It shows stakeholders that you understand the limitations of a lab environment and have a concrete plan for production.
The cost comparison is especially important. EKS alone costs approximately $73/month per cluster for the control plane with no free tier. AKS Standard tier is also $73/month per cluster. GKE provides a $74.40/month credit for cluster management but compute is extra.
- Review the generated production-readiness-assessment.md for accuracy. Verify that the managed Kubernetes pricing figures match the numbers above.
- Check that the improvement checklist is realistic. Each item should be actionable, not vague.
- Commit production-readiness-assessment.md to your repository.
Pro tip: priority rankings matter
A flat list of improvements is less useful than a ranked one. Critical items (like replacing single-node clusters with multi-node for high availability) should be clearly separated from medium-priority items (like adding monitoring dashboards). This ranking helps decision-makers allocate budget and schedule realistically.
Do a final check of your repository structure. You should have all of the following files committed.
- Run ls or use the Cursor file explorer to confirm these files exist in your repository:
ls README.md \
govRAMP-crosswalk.md \
coop-report.md \
migration-wave-plan.md \
portability-scorecard.md \
system-boundary.md \
production-readiness-assessment.md \
infra/ \
apps/- Push all final changes to GitHub.
git add -A
git commit -m "Step 6: ATO readiness review, fact-check corrections, production readiness assessment"
git push origin mainWhat does this command do?
This stages all changes across your repository, commits them with a descriptive message, and pushes to the remote. After this push, your GitHub repository contains the complete, validated deliverable set.
Your deliverable set is validated, fact-checked, and ready for review. Next up, you will tackle the Secret Mission to compare Crossplane against OpenTofu for infrastructure portability.
Secret mission
Compare Crossplane vs. OpenTofu Portability
You built the entire multi-cloud platform with OpenTofu. But there is a second paradigm for managing cloud resources: Crossplane, which treats cloud infrastructure as Kubernetes custom resources managed by the cluster API itself. Install Crossplane, provision an S3 bucket through it, and compare both approaches head-to-head to determine which the Columbia HHS department should adopt for each resource type.
Before you go
Clean Up Your Resources
This project uses three cloud VMs and an S3 bucket that can incur charges if left running. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.
Cost warning
Even though this lab costs $0.00 to $0.05 for a single session, leaving the cloud VMs running will accumulate charges. An AWS t3.small left running costs approximately $15/month. Always run tofu destroy when you are done to bring your ongoing cloud cost to $0.
Resources you used:
- AWS EC2 t3.small instance (columbia-aws cluster) in us-east-1.
- Azure VM B1s or B2s instance (columbia-azure cluster) in eastus.
- GCP Compute Engine e2-micro or e2-small instance (columbia-gcp cluster) in us-central1.
- AWS S3 bucket for Velero COOP backup storage.
- k3s cluster on WSL2 (columbia-onprem) with Argo CD and Velero installed. No cloud cost.
- GitHub repository (columbia-modernization). Free tier, no cost.
Keep everything running
No action needed. Choose this if you are still actively building or experimenting with the platform.
- Your three cloud VMs will continue running and incurring charges (approximately $0.02/hr for the AWS t3.small, $0.01/hr for the Azure VM, and $0.00 to $0.017/hr for the GCP instance).
- The S3 bucket remains under the 5 GB free tier for the first 12 months on new AWS accounts.
- The WSL2 k3s cluster runs locally at no cost.
- Verify your $1 billing alerts are active on AWS, Azure, and GCP so you get notified if charges start accumulating.
Pause - I'll come back to this later
Stop the cloud VMs to avoid compute charges while preserving your infrastructure state and project files.
- Stop the AWS EC2 instance from the AWS Console or by running aws ec2 stop-instances --instance-ids <INSTANCE_ID>. Stopped instances do not incur compute charges, but attached EBS volumes still incur small storage charges.
- Stop the Azure VM from the Azure Portal or by running az vm deallocate --resource-group <RG_NAME> --name <VM_NAME>. Deallocated VMs do not incur compute charges, but managed disks still incur storage charges.
- Stop the GCP instance from the GCP Console or by running gcloud compute instances stop <INSTANCE_NAME> --zone us-central1-a. Stopped instances do not incur compute charges, but persistent disks still incur storage charges.
- Stop the local k3s cluster on WSL2 by running sudo systemctl stop k3s to free up local memory.
- Keep the S3 bucket in place. Velero backups stored there remain accessible when you resume.
- Your OpenTofu state file and GitHub repository preserve all configuration, so you can restart the VMs and pick up where you left off.
Delete - I don't want to use this again
Remove all project resources and start fresh. This brings your ongoing cloud cost to $0.
- Navigate to the infra/ directory in your project and run tofu destroy. Confirm the destruction when prompted. This removes all three cloud VMs and the S3 bucket.
- Run tofu show to verify the state is empty and all resources have been removed.
- Check each cloud console (AWS, Azure, GCP) for any orphaned resources such as disks, static IPs, or buckets that OpenTofu may not have removed.
- If the S3 bucket was not removed by tofu destroy (common if the bucket still contains Velero backup objects), delete it manually from the AWS Console or by running aws s3 rb s3://<BUCKET_NAME> --force.
- Stop and uninstall k3s from WSL2 by running sudo /usr/local/bin/k3s-uninstall.sh.
- Verify your $1 billing alerts are still active on all three cloud accounts. Check back in 24 hours to confirm no charges appear.
- Optionally delete the columbia-modernization GitHub repository if you no longer need it.
Mission accomplished
That's a Wrap
Nice work! You just built a state-level multi-cloud modernization platform from scratch, complete with disaster recovery, compliance documentation, and a migration roadmap. Every piece was designed, generated, and validated using AI-accelerated development with Cursor IDE Composer 2.
You've learned how to:
- Provision a multi-cloud Kubernetes platform across four environments using OpenTofu IaC modules and manage deployments through Argo CD GitOps with the App of Apps pattern.
- Execute a COOP disaster recovery drill using Velero for cross-cluster backup and restore, with documented RTO/RPO metrics validated against industry benchmarks.
- Author a GovRAMP Moderate controls crosswalk mapping 60 Core controls to your infrastructure artifacts, alongside a three-wave migration plan and portability scorecard ready for stakeholder review.
- Compare Crossplane vs. OpenTofu portability models by provisioning an S3 bucket through Crossplane and evaluating which approach better serves government compliance and GitOps workflows.
Ready to quiz yourself?