Deploy an ERP Platform on AWS EKS

Provision EKS, RDS, and Secrets Manager with Terraform to run microservices.

Introduction

30 Second Summary

Every enterprise runs on platforms that connect inventory, orders, and data across teams. Behind the scenes, those platforms need secure databases, auto-scaling compute, and credentials that never get hardcoded into source code.

In this project, you will provision production-grade AWS infrastructure using Terraform and deploy two database-backed microservices on EKS with intelligent autoscaling and secure secrets injection.

What You'll Build

A fully Terraform-managed Kubernetes cluster on AWS with Karpenter autoscaling, RDS PostgreSQL for persistent storage, and AWS Secrets Manager for credential management.

By the end of this project, you'll have:

  • A Terraform codebase provisioning a VPC, EKS cluster, RDS PostgreSQL database, Secrets Manager, and Karpenter IAM roles.
  • Two containerized microservices (Order Service and Inventory Service) running on EKS with real database connectivity and credentials injected securely via the External Secrets Operator.
  • An interview-ready infrastructure project demonstrating the data layer, compute layer, and security layer of a complete enterprise DevOps platform.
  • Secret Mission: Add a Kubernetes CronJob that seeds the database with sample ERP data and configure Karpenter to scale the cluster by deploying a load generator.

Are there any prerequisites?

You'll need an AWS account with admin access and a GitHub account. Basic familiarity with terminal commands and Kubernetes concepts is helpful, but every step is explained.

This is Project 1 of 4 in an enterprise DevOps series covering Infrastructure, CI/CD and Security, Observability, and Service Mesh.

Install Tools and Create Microservices

Before you can provision cloud infrastructure, you need a local toolkit that speaks to AWS. This project uses five CLI tools working together: Terraform for infrastructure as code, the AWS CLI for AWS authentication, kubectl for Kubernetes cluster access, Helm for package management, and Docker for building container images.

You will also write the two microservices that will eventually run on EKS. The Inventory Service queries a PostgreSQL database for inventory items. The Order Service calls the Inventory Service to compose order data.

In this step, get ready to:
  • Install and verify Terraform, AWS CLI, kubectl, Helm, and Docker.
  • Create the project directory structure and write both microservices with Dockerfiles.
  • Write a database initialization SQL script for RDS PostgreSQL.
Install Terraform, AWS CLI, kubectl, Helm, and Docker

Each tool plays a distinct role in the deployment pipeline. You will install all five, verify they work, and configure the AWS CLI with your credentials.

Open your terminal. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar, type Terminal (macOS) or PowerShell (Windows), and press Enter.

  • Check whether Terraform is installed by running:
terraform version

✔️ I see version 1.15.5 or higher

Terraform is ready. Move on to the AWS CLI check below.

ⓧ I see an older version

You need Terraform 1.15.5 or later. Upgrade using the same method you originally installed with.

macOS

  • Upgrade Terraform via Homebrew by running:
brew tap hashicorp/tap
brew upgrade hashicorp/tap/terraform

If Homebrew still shows an older version, run brew update first to refresh the package index.

Windows

  • Download the latest Terraform binary from the official install page:

https://developer.hashicorp.com/terraform/install

  • Replace your existing terraform.exe with the new binary and ensure it is in your PATH.
  • After upgrading, run terraform version again to confirm you see 1.15.5 or higher.

ⓧ Command not found

Terraform is not installed. Install it now.

macOS

  • Install Terraform via Homebrew by running:
brew tap hashicorp/tap
brew install hashicorp/tap/terraform

Windows

  • Download the Terraform binary for Windows from the official install page:

https://developer.hashicorp.com/terraform/install

  • Unzip the downloaded file and move terraform.exe to a directory in your system PATH.
  • Open a new terminal session and run terraform version to confirm you see 1.15.5 or higher.
  • Check whether the AWS CLI is installed by running:
aws --version

✔️ I see a version number

AWS CLI is ready. Move on to the kubectl check below.

ⓧ Command not found

Install the AWS CLI v2 from the official page for your platform:

https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

  • Follow the instructions for your operating system and complete the installer.
  • Open a new terminal and run aws --version to confirm.
  • Check whether kubectl is installed by running:
kubectl version --client

✔️ I see a version number

kubectl is ready. Move on to the Helm check below.

ⓧ Command not found

Install kubectl from the official Kubernetes documentation:

https://kubernetes.io/docs/tasks/tools/

  • Follow the instructions for your operating system.
  • After installation, run kubectl version --client to confirm.
  • Check whether Helm is installed by running:
helm version

✔️ I see a version number

Helm is ready. Move on to the Docker check below.

ⓧ Command not found

Install Helm from the official site:

https://helm.sh/docs/intro/install/

  • Follow the instructions for your operating system.
  • After installation, run helm version to confirm.
  • Check whether Docker is installed by running:
docker --version

✔️ I see a version number

Docker is ready. Move on to the AWS CLI configuration below.

ⓧ Command not found

Install Docker Desktop from the official site:

https://docs.docker.com/get-docker/

  • Download and install Docker Desktop for your operating system.
  • After installation, open Docker Desktop and let it finish starting up.
  • Run docker --version to confirm.

Now configure the AWS CLI with your admin credentials so Terraform can provision resources.

  • Configure the AWS CLI by running:
aws configure

What does aws configure ask for?

The command prompts you for four values. Enter them one at a time when prompted:

  • AWS Access Key ID - paste your IAM admin access key.
  • AWS Secret Access Key - paste your IAM admin secret key.
  • Default region name - enter us-east-1.
  • Default output format - enter json.

AWS configure failing?

  • Make sure you have an IAM user with admin access and an active access key pair. You can create one in the AWS Console under IAM, then Users, then Security credentials.
  • If you already configured the AWS CLI before, this command overwrites the default profile. That is expected.

Help me troubleshoot my AWS CLI configuration

Create the project structure and write both microservices

Your project has three top-level directories. The infra/ folder holds Terraform code. The services/ folder holds the microservice source code. The k8s/ folder holds Kubernetes manifests.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and directory structure by running these commands:
mkdir -p erp-platform/{infra,services/inventory-service,services/order-service,k8s}
cd erp-platform
  • Initialize a Git repository by running:
git init
  • Confirm the directory structure exists by running:
ls

You should see infra, k8s, and services in the output.

Now open the project in your code editor. Press Cmd+Space (macOS) or the Windows key (Windows), type VS Code or Cursor, and press Enter.

  • Click File in the top menu bar.
  • Click Open Folder.
  • Select the erp-platform folder on your Desktop, then click Open.

Start with the Inventory Service. This Node.js HTTP server connects to PostgreSQL and returns inventory items.

  • Create a new file at services/inventory-service/index.js and paste the following code:
const http = require("http");
const { Client } = require("pg");

// Database connection config from environment variables
const dbConfig = {
  host: process.env.DB_HOST,
  port: parseInt(process.env.DB_PORT || "5432"),
  user: process.env.DB_USER,
  password: process.env.DB_PASSWORD,
  database: process.env.DB_NAME,
  ssl: { rejectUnauthorized: false },
};

// Query all inventory items from PostgreSQL
async function queryInventory() {
  const client = new Client(dbConfig);
  await client.connect();
  const result = await client.query(
    "SELECT id, name, quantity, warehouse FROM inventory"
  );
  await client.end();
  return result.rows;
}

// HTTP server with health check and inventory endpoint
const server = http.createServer(async (req, res) => {
  if (req.url === "/health") {
    res.writeHead(200, { "Content-Type": "application/json" });
    res.end(JSON.stringify({ status: "healthy" }));
    return;
  }

  if (req.url === "/inventory") {
    try {
      const items = await queryInventory();
      res.writeHead(200, { "Content-Type": "application/json" });
      res.end(JSON.stringify({ items }));
    } catch (err) {
      res.writeHead(500, { "Content-Type": "application/json" });
      res.end(JSON.stringify({ error: "Database error", details: err.message }));
    }
    return;
  }

  res.writeHead(404, { "Content-Type": "application/json" });
  res.end(JSON.stringify({ error: "Not found" }));
});

server.listen(8080, () => {
  console.log("Inventory Service running on port 8080");
});

What does this code do?

  • The dbConfig object reads database connection details from environment variables. These will be injected later by the External Secrets Operator.
  • The queryInventory() function opens a connection to PostgreSQL, runs a SELECT query, then closes the connection.
  • The HTTP server exposes two routes: /health for Kubernetes liveness probes and /inventory to return all inventory items as JSON.
  • Create a new file at services/inventory-service/package.json and paste the following:
{
  "name": "inventory-service",
  "version": "1.0.0",
  "description": "ERP Inventory Service",
  "main": "index.js",
  "dependencies": {
    "pg": "8.16.0"
  }
}

What does this file do?

The package.json declares the pg package (version 8.16.0) as a dependency. This is the Node.js PostgreSQL client that queryInventory() uses to talk to RDS.

  • Create a new file at services/inventory-service/Dockerfile and paste the following:
FROM node:20-alpine
WORKDIR /app
COPY package.json .
RUN npm install --production
COPY index.js .
EXPOSE 8080
CMD ["node", "index.js"]

What does this Dockerfile do?

  • Starts from a minimal Node.js 20 Alpine image to keep the container lightweight.
  • Copies package.json first and runs npm install --production to cache dependency layers separately from code changes.
  • Then copies index.js and sets the container to listen on port 8080.
  • Save all three Inventory Service files.

Next, write the Order Service. This service calls the Inventory Service to get items and returns composed order data.

  • Create a new file at services/order-service/index.js and paste the following code:
const http = require("http");

// URL of the Inventory Service (Kubernetes DNS resolves this)
const INVENTORY_SERVICE_URL =
  process.env.INVENTORY_SERVICE_URL || "http://inventory-service:8080";

// Fetch inventory data from the Inventory Service
function fetchInventory() {
  return new Promise((resolve, reject) => {
    http
      .get(`${INVENTORY_SERVICE_URL}/inventory`, (res) => {
        let data = "";
        res.on("data", (chunk) => {
          data += chunk;
        });
        res.on("end", () => {
          resolve(JSON.parse(data));
        });
      })
      .on("error", (err) => {
        reject(err);
      });
  });
}

// HTTP server with health check and orders endpoint
const server = http.createServer(async (req, res) => {
  if (req.url === "/health") {
    res.writeHead(200, { "Content-Type": "application/json" });
    res.end(JSON.stringify({ status: "healthy" }));
    return;
  }

  if (req.url === "/orders") {
    try {
      const inventory = await fetchInventory();
      const orders = {
        orders: [
          {
            id: "ORD-001",
            status: "processing",
            items: inventory.items.slice(0, 2),
          },
          {
            id: "ORD-002",
            status: "shipped",
            items: inventory.items.slice(2, 4),
          },
        ],
      };
      res.writeHead(200, { "Content-Type": "application/json" });
      res.end(JSON.stringify(orders));
    } catch (err) {
      res.writeHead(502, { "Content-Type": "application/json" });
      res.end(
        JSON.stringify({ error: "Failed to reach Inventory Service", details: err.message })
      );
    }
    return;
  }

  res.writeHead(404, { "Content-Type": "application/json" });
  res.end(JSON.stringify({ error: "Not found" }));
});

server.listen(8080, () => {
  console.log("Order Service running on port 8080");
});

What does this code do?

  • The INVENTORY_SERVICE_URL defaults to http://inventory-service:8080. Inside Kubernetes, the ClusterIP Service DNS automatically resolves this name.
  • The fetchInventory() function makes an HTTP GET request to the Inventory Service and parses the JSON response.
  • The /orders endpoint composes two orders from the fetched inventory data. This demonstrates real inter-service communication.
  • Create a new file at services/order-service/Dockerfile and paste the following:
FROM node:20-alpine
WORKDIR /app
COPY index.js .
EXPOSE 8080
CMD ["node", "index.js"]

Why is this Dockerfile simpler?

The Order Service has no external npm dependencies. It only uses the Node.js built-in http module. No package.json or npm install step is needed.

  • Save services/order-service/index.js and services/order-service/Dockerfile.

✔️ Awesome, I've got everything!

Great. Double check you have saved all five files across both services.

ⓧ I'd like to double check the full code

Here is each file for comparison:

const http = require("http");
const { Client } = require("pg");

const dbConfig = {
  host: process.env.DB_HOST,
  port: parseInt(process.env.DB_PORT || "5432"),
  user: process.env.DB_USER,
  password: process.env.DB_PASSWORD,
  database: process.env.DB_NAME,
  ssl: { rejectUnauthorized: false },
};

async function queryInventory() {
  const client = new Client(dbConfig);
  await client.connect();
  const result = await client.query(
    "SELECT id, name, quantity, warehouse FROM inventory"
  );
  await client.end();
  return result.rows;
}

const server = http.createServer(async (req, res) => {
  if (req.url === "/health") {
    res.writeHead(200, { "Content-Type": "application/json" });
    res.end(JSON.stringify({ status: "healthy" }));
    return;
  }

  if (req.url === "/inventory") {
    try {
      const items = await queryInventory();
      res.writeHead(200, { "Content-Type": "application/json" });
      res.end(JSON.stringify({ items }));
    } catch (err) {
      res.writeHead(500, { "Content-Type": "application/json" });
      res.end(JSON.stringify({ error: "Database error", details: err.message }));
    }
    return;
  }

  res.writeHead(404, { "Content-Type": "application/json" });
  res.end(JSON.stringify({ error: "Not found" }));
});

server.listen(8080, () => {
  console.log("Inventory Service running on port 8080");
});
{
  "name": "inventory-service",
  "version": "1.0.0",
  "description": "ERP Inventory Service",
  "main": "index.js",
  "dependencies": {
    "pg": "8.16.0"
  }
}
FROM node:20-alpine
WORKDIR /app
COPY package.json .
RUN npm install --production
COPY index.js .
EXPOSE 8080
CMD ["node", "index.js"]
const http = require("http");

const INVENTORY_SERVICE_URL =
  process.env.INVENTORY_SERVICE_URL || "http://inventory-service:8080";

function fetchInventory() {
  return new Promise((resolve, reject) => {
    http
      .get(`${INVENTORY_SERVICE_URL}/inventory`, (res) => {
        let data = "";
        res.on("data", (chunk) => {
          data += chunk;
        });
        res.on("end", () => {
          resolve(JSON.parse(data));
        });
      })
      .on("error", (err) => {
        reject(err);
      });
  });
}

const server = http.createServer(async (req, res) => {
  if (req.url === "/health") {
    res.writeHead(200, { "Content-Type": "application/json" });
    res.end(JSON.stringify({ status: "healthy" }));
    return;
  }

  if (req.url === "/orders") {
    try {
      const inventory = await fetchInventory();
      const orders = {
        orders: [
          {
            id: "ORD-001",
            status: "processing",
            items: inventory.items.slice(0, 2),
          },
          {
            id: "ORD-002",
            status: "shipped",
            items: inventory.items.slice(2, 4),
          },
        ],
      };
      res.writeHead(200, { "Content-Type": "application/json" });
      res.end(JSON.stringify(orders));
    } catch (err) {
      res.writeHead(502, { "Content-Type": "application/json" });
      res.end(
        JSON.stringify({ error: "Failed to reach Inventory Service", details: err.message })
      );
    }
    return;
  }

  res.writeHead(404, { "Content-Type": "application/json" });
  res.end(JSON.stringify({ error: "Not found" }));
});

server.listen(8080, () => {
  console.log("Order Service running on port 8080");
});
FROM node:20-alpine
WORKDIR /app
COPY index.js .
EXPOSE 8080
CMD ["node", "index.js"]
Write the database initialization script

The RDS PostgreSQL instance starts empty. You need a SQL script that creates the inventory table and seeds it with sample data. You will run this script against RDS after provisioning.

  • Create a new file at infra/init.sql and paste the following:
-- Create the inventory table
CREATE TABLE IF NOT EXISTS inventory (
  id SERIAL PRIMARY KEY,
  name VARCHAR(100) NOT NULL,
  quantity INTEGER NOT NULL DEFAULT 0,
  warehouse VARCHAR(20) NOT NULL,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Seed with sample ERP data
INSERT INTO inventory (name, quantity, warehouse) VALUES
  ('Widget A', 150, 'WH-01'),
  ('Widget B', 75, 'WH-02'),
  ('Gadget C', 200, 'WH-01'),
  ('Component D', 320, 'WH-03'),
  ('Assembly E', 50, 'WH-02');

What does this script do?

  • Creates the inventory table with columns for product name, quantity, warehouse location, and a timestamp.
  • Seeds five sample rows representing ERP inventory items across three warehouses.
  • The IF NOT EXISTS clause makes the script safe to rerun without errors.
  • Save infra/init.sql.

✔️ Awesome, I've got everything!

Great. Double check you have saved the file.

ⓧ I'd like to double check the full code

CREATE TABLE IF NOT EXISTS inventory (
  id SERIAL PRIMARY KEY,
  name VARCHAR(100) NOT NULL,
  quantity INTEGER NOT NULL DEFAULT 0,
  warehouse VARCHAR(20) NOT NULL,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

INSERT INTO inventory (name, quantity, warehouse) VALUES
  ('Widget A', 150, 'WH-01'),
  ('Widget B', 75, 'WH-02'),
  ('Gadget C', 200, 'WH-01'),
  ('Component D', 320, 'WH-03'),
  ('Assembly E', 50, 'WH-02');

Files not appearing in your editor?

  • Make sure you opened the erp-platform folder (not a parent or child folder) in your editor.
  • Check that you created files in the correct nested paths. The Inventory Service files live inside services/inventory-service/, not at the project root.

Help me find my project files in VS Code

Your local environment is fully set up with five CLI tools, two containerized microservices, and a database seed script. Next up, you will write the Terraform infrastructure code that provisions VPC, EKS, RDS, and Secrets Manager on AWS.

Write Terraform Infrastructure Code

Your microservices are ready, but they need a production platform to run on. Every enterprise platform has three layers: networking (where traffic flows), compute (where containers run), and data (where state lives).

In this step, you will define all three layers as Terraform code. By the end, you will have seven configuration files that describe your entire AWS infrastructure. Nothing gets provisioned yet. That happens in the next step.

In this step, get ready to:
  • Write the provider configuration and VPC networking with public, private, and database subnets.
  • Define the EKS cluster with a managed node group and Karpenter IAM roles.
  • Configure RDS PostgreSQL, a Secrets Manager secret for database credentials, and Terraform outputs.
Create the provider config and VPC

Every Terraform project starts with a provider configuration that tells Terraform which cloud to talk to and which API version to use.

  • Create a new file called versions.tf inside your infra/ folder and paste the following code:
terraform {
  required_version = ">= 1.5.7"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

provider "aws" {
  region = var.region
}

What does this code do?

The terraform block pins your project to Terraform 1.5.7 or higher and locks the AWS provider to major version 6.

The provider "aws" block references a var.region variable you will define next. This keeps region configurable without editing the provider block.

  • Save the file.

Next, you need to define all the variables that your Terraform files will reference. Variables keep values like the cluster name and region in one place so you never repeat yourself.

  • Create variables.tf inside infra/ and paste this code:
variable "region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

variable "cluster_name" {
  description = "EKS cluster name"
  type        = string
  default     = "erp-cluster"
}

variable "environment" {
  description = "Environment name"
  type        = string
  default     = "dev"
}

variable "db_username" {
  description = "Database master username"
  type        = string
  default     = "erpadmin"
}

variable "db_password" {
  description = "Database master password"
  type        = string
  sensitive   = true
}

What does this code do?

  • region defaults to us-east-1 and is used by the provider and every resource.
  • cluster_name defaults to erp-cluster and names the EKS cluster, VPC, RDS instance, and secrets.
  • db_password is marked sensitive = true so Terraform never prints it in plan output. It has no default because you will supply it at apply time.
  • Save variables.tf.

Now for the networking layer. A VPC isolates your resources in a private network. You need three subnet tiers: public subnets for load balancers, private subnets for EKS nodes, and database subnets for RDS.

  • Create vpc.tf inside infra/ and paste this code:
data "aws_availability_zones" "available" {
  filter {
    name   = "opt-in-status"
    values = ["opt-in-not-required"]
  }
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 6.0"

  name = "${var.cluster_name}-vpc"
  cidr = "10.0.0.0/16"

  azs              = slice(data.aws_availability_zones.available.names, 0, 2)
  private_subnets  = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets   = ["10.0.101.0/24", "10.0.102.0/24"]
  database_subnets = ["10.0.201.0/24", "10.0.202.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true

  create_database_subnet_group = true

  public_subnet_tags = {
    "kubernetes.io/role/elb" = 1
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = 1
    "karpenter.sh/discovery"         = var.cluster_name
  }

  tags = {
    Environment = var.environment
    Terraform   = "true"
  }
}

What does this code do?

  • The data source fetches available AZs so you don't hardcode zone names.
  • single_nat_gateway = true keeps costs low by routing all private traffic through one NAT gateway instead of one per AZ.
  • The subnet tags are required by EKS and Karpenter. EKS uses kubernetes.io/role/elb to find subnets for load balancers. Karpenter uses karpenter.sh/discovery to find subnets for new nodes.
  • create_database_subnet_group = true produces a named subnet group that the RDS instance will reference later.
  • Save vpc.tf.
  • Confirm all three files exist by running:
ls infra/*.tf

You should see versions.tf, variables.tf, and vpc.tf listed in the output.

Not seeing all three files?

  • Make sure you created the files inside the infra/ folder, not the project root.
  • Check each file has the .tf extension, not .tf.txt or another variant.

Help me troubleshoot my missing Terraform files.

✔️ Awesome, I've got everything!

Great. Make sure all three files are saved before moving on.

ⓧ I'd like to double check the full code

terraform {
  required_version = ">= 1.5.7"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

provider "aws" {
  region = var.region
}
variable "region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

variable "cluster_name" {
  description = "EKS cluster name"
  type        = string
  default     = "erp-cluster"
}

variable "environment" {
  description = "Environment name"
  type        = string
  default     = "dev"
}

variable "db_username" {
  description = "Database master username"
  type        = string
  default     = "erpadmin"
}

variable "db_password" {
  description = "Database master password"
  type        = string
  sensitive   = true
}
data "aws_availability_zones" "available" {
  filter {
    name   = "opt-in-status"
    values = ["opt-in-not-required"]
  }
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 6.0"

  name = "${var.cluster_name}-vpc"
  cidr = "10.0.0.0/16"

  azs              = slice(data.aws_availability_zones.available.names, 0, 2)
  private_subnets  = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets   = ["10.0.101.0/24", "10.0.102.0/24"]
  database_subnets = ["10.0.201.0/24", "10.0.202.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true

  create_database_subnet_group = true

  public_subnet_tags = {
    "kubernetes.io/role/elb" = 1
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = 1
    "karpenter.sh/discovery"         = var.cluster_name
  }

  tags = {
    Environment = var.environment
    Terraform   = "true"
  }
}
Define the EKS cluster and Karpenter IAM

The compute layer is an EKS cluster. You will create a small managed node group for system workloads (Karpenter itself, CoreDNS, and the External Secrets Operator). Application workloads will run on nodes that Karpenter provisions on demand.

  • Create eks.tf inside infra/ and paste this code:
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.0"

  name               = var.cluster_name
  kubernetes_version = "1.35"

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  endpoint_public_access = true

  enable_cluster_creator_admin_permissions = true

  addons = {
    coredns = {}
    eks-pod-identity-agent = {
      before_compute = true
    }
    kube-proxy = {}
    vpc-cni = {
      before_compute = true
    }
  }

  eks_managed_node_groups = {
    system = {
      ami_type       = "AL2023_x86_64_STANDARD"
      instance_types = ["t3.medium"]
      min_size       = 2
      max_size       = 3
      desired_size   = 2

      labels = {
        role = "system"
      }
    }
  }

  tags = {
    Environment              = var.environment
    Terraform                = "true"
    "karpenter.sh/discovery" = var.cluster_name
  }
}

What does this code do?

  • kubernetes_version = "1.35" pins the cluster to Kubernetes 1.35, which has standard support until March 2027.
  • The system node group runs 2 t3.medium instances. These host Karpenter and other system controllers. Application pods will land on separate Karpenter-provisioned nodes.
  • before_compute = true on vpc-cni and eks-pod-identity-agent ensures networking and identity are ready before nodes launch.
  • The karpenter.sh/discovery tag on the cluster lets Karpenter discover associated security groups.

Why Karpenter instead of Cluster Autoscaler?

Cluster Autoscaler scales entire node groups up or down. Karpenter evaluates each pending pod individually and provisions the cheapest instance type that satisfies its resource requests.

This bin-packing approach reduces waste and cost. Karpenter also handles spot interruptions natively, making it the standard for production EKS clusters.

  • Save eks.tf.

Karpenter needs its own IAM role to launch EC2 instances and an SQS queue to receive spot interruption notifications. The EKS module includes a dedicated Karpenter submodule that handles this.

  • Create karpenter-iam.tf inside infra/ and paste this code:
module "karpenter" {
  source  = "terraform-aws-modules/eks/aws//modules/karpenter"
  version = "~> 21.0"

  cluster_name = module.eks.cluster_name

  enable_irsa            = true
  irsa_oidc_provider_arn = module.eks.oidc_provider_arn

  node_iam_role_additional_policies = {
    AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
  }

  tags = {
    Environment = var.environment
    Terraform   = "true"
  }
}

resource "aws_sqs_queue" "karpenter" {
  name                      = var.cluster_name
  message_retention_seconds = 300
  sqs_managed_sse_enabled   = true
}

What does this code do?

  • enable_irsa = true creates an IRSA (IAM Roles for Service Accounts) role that lets the Karpenter pod assume AWS permissions without static credentials.
  • The AmazonSSMManagedInstanceCore policy gives Karpenter-launched nodes SSM access for debugging.
  • The SQS queue receives EC2 spot interruption warnings so Karpenter can gracefully drain nodes before termination.
  • Save karpenter-iam.tf.
  • Confirm the new files exist by running:
ls infra/*.tf

You should see five .tf files: eks.tf, karpenter-iam.tf, variables.tf, versions.tf, and vpc.tf.

Missing a file?

  • Double check you saved each file with the exact name shown (including the hyphen in karpenter-iam.tf).
  • Ensure your terminal is in the project root (the parent of the infra/ folder), not inside infra/ itself.

Help me find my missing Terraform files.

✔️ Awesome, I've got everything!

Great. Double check you have saved both files before continuing.

ⓧ I'd like to double check the full code

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.0"

  name               = var.cluster_name
  kubernetes_version = "1.35"

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  endpoint_public_access = true

  enable_cluster_creator_admin_permissions = true

  addons = {
    coredns = {}
    eks-pod-identity-agent = {
      before_compute = true
    }
    kube-proxy = {}
    vpc-cni = {
      before_compute = true
    }
  }

  eks_managed_node_groups = {
    system = {
      ami_type       = "AL2023_x86_64_STANDARD"
      instance_types = ["t3.medium"]
      min_size       = 2
      max_size       = 3
      desired_size   = 2

      labels = {
        role = "system"
      }
    }
  }

  tags = {
    Environment              = var.environment
    Terraform                = "true"
    "karpenter.sh/discovery" = var.cluster_name
  }
}
module "karpenter" {
  source  = "terraform-aws-modules/eks/aws//modules/karpenter"
  version = "~> 21.0"

  cluster_name = module.eks.cluster_name

  enable_irsa            = true
  irsa_oidc_provider_arn = module.eks.oidc_provider_arn

  node_iam_role_additional_policies = {
    AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
  }

  tags = {
    Environment = var.environment
    Terraform   = "true"
  }
}

resource "aws_sqs_queue" "karpenter" {
  name                      = var.cluster_name
  message_retention_seconds = 300
  sqs_managed_sse_enabled   = true
}
Configure RDS, Secrets Manager, and outputs

The data layer stores your application state. You will create an RDS PostgreSQL instance for persistent storage, then store its credentials in AWS Secrets Manager so they never appear in your Kubernetes manifests.

  • Create rds.tf inside infra/ and paste this code:
resource "aws_security_group" "rds" {
  name_prefix = "${var.cluster_name}-rds-"
  vpc_id      = module.vpc.vpc_id

  ingress {
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [module.eks.node_security_group_id]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name        = "${var.cluster_name}-rds"
    Environment = var.environment
  }
}

resource "aws_db_instance" "erp" {
  identifier = "${var.cluster_name}-db"

  engine         = "postgres"
  engine_version = "16"
  instance_class = "db.t3.micro"

  allocated_storage = 20
  storage_type      = "gp3"

  db_name  = "erpdb"
  username = var.db_username
  password = var.db_password

  db_subnet_group_name   = module.vpc.database_subnet_group_name
  vpc_security_group_ids = [aws_security_group.rds.id]

  skip_final_snapshot = true
  publicly_accessible = false

  tags = {
    Environment = var.environment
    Terraform   = "true"
  }
}

What does this code do?

  • The security group allows inbound PostgreSQL traffic (port 5432) only from EKS nodes. Nothing else can reach the database.
  • db.t3.micro is the smallest RDS instance class. It costs about $0.018/hr and is sufficient for a demo.
  • publicly_accessible = false ensures the database is only reachable from within the VPC. Combined with database subnets, this is defense in depth.
  • skip_final_snapshot = true lets Terraform destroy the database cleanly without requiring a snapshot. Fine for a dev environment.
  • Save rds.tf.

Now you need to store the database credentials in Secrets Manager. The External Secrets Operator will later sync this secret into your Kubernetes cluster as a native Secret object.

  • Create secrets.tf inside infra/ and paste this code:
resource "aws_secretsmanager_secret" "erp_db" {
  name = "${var.cluster_name}/db-credentials"

  tags = {
    Environment = var.environment
    Terraform   = "true"
  }
}

resource "aws_secretsmanager_secret_version" "erp_db" {
  secret_id = aws_secretsmanager_secret.erp_db.id
  secret_string = jsonencode({
    host     = aws_db_instance.erp.endpoint
    port     = 5432
    username = var.db_username
    password = var.db_password
    dbname   = "erpdb"
  })
}

What does this code do?

The secret stores a JSON object with five keys: host, port, username, password, and dbname. The External Secrets Operator will map each key to a Kubernetes Secret data field.

The host value comes directly from the RDS endpoint output. Terraform resolves this dependency automatically.

  • Save secrets.tf.

Finally, you need an outputs file. Outputs expose key values (like the cluster endpoint and Karpenter role ARN) so you can reference them from the command line in later steps.

  • Create outputs.tf inside infra/ and paste this code:
output "cluster_endpoint" {
  description = "EKS cluster endpoint"
  value       = module.eks.cluster_endpoint
}

output "cluster_name" {
  description = "EKS cluster name"
  value       = module.eks.cluster_name
}

output "region" {
  description = "AWS region"
  value       = var.region
}

output "karpenter_irsa_role_arn" {
  description = "Karpenter IRSA role ARN"
  value       = module.karpenter.iam_role_arn
}

output "karpenter_node_role_name" {
  description = "Karpenter node IAM role name"
  value       = module.karpenter.node_iam_role_name
}

output "rds_endpoint" {
  description = "RDS endpoint"
  value       = aws_db_instance.erp.endpoint
}

output "db_secret_arn" {
  description = "Secrets Manager secret ARN for DB credentials"
  value       = aws_secretsmanager_secret.erp_db.arn
}

What does this code do?

Each output exposes a value you will need in later steps. For example, karpenter_irsa_role_arn is passed to the Helm chart when you install Karpenter. db_secret_arn is used by the External Secrets Operator to know which secret to sync.

  • Save outputs.tf.
  • Confirm all Terraform files now exist by running:
ls infra/*.tf

You should see seven .tf files: eks.tf, karpenter-iam.tf, outputs.tf, rds.tf, secrets.tf, variables.tf, and versions.tf.

Not seeing seven files?

  • Check that you saved each file with the .tf extension. Some editors may append .txt automatically.
  • Verify you are running the command from the project root, not from inside the infra/ directory.

Help me figure out where my Terraform files went.

✔️ Awesome, I've got everything!

Great. Make sure all files are saved before continuing to the next step.

ⓧ I'd like to double check the full code

resource "aws_security_group" "rds" {
  name_prefix = "${var.cluster_name}-rds-"
  vpc_id      = module.vpc.vpc_id

  ingress {
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [module.eks.node_security_group_id]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name        = "${var.cluster_name}-rds"
    Environment = var.environment
  }
}

resource "aws_db_instance" "erp" {
  identifier = "${var.cluster_name}-db"

  engine         = "postgres"
  engine_version = "16"
  instance_class = "db.t3.micro"

  allocated_storage = 20
  storage_type      = "gp3"

  db_name  = "erpdb"
  username = var.db_username
  password = var.db_password

  db_subnet_group_name   = module.vpc.database_subnet_group_name
  vpc_security_group_ids = [aws_security_group.rds.id]

  skip_final_snapshot = true
  publicly_accessible = false

  tags = {
    Environment = var.environment
    Terraform   = "true"
  }
}
resource "aws_secretsmanager_secret" "erp_db" {
  name = "${var.cluster_name}/db-credentials"

  tags = {
    Environment = var.environment
    Terraform   = "true"
  }
}

resource "aws_secretsmanager_secret_version" "erp_db" {
  secret_id = aws_secretsmanager_secret.erp_db.id
  secret_string = jsonencode({
    host     = aws_db_instance.erp.endpoint
    port     = 5432
    username = var.db_username
    password = var.db_password
    dbname   = "erpdb"
  })
}
output "cluster_endpoint" {
  description = "EKS cluster endpoint"
  value       = module.eks.cluster_endpoint
}

output "cluster_name" {
  description = "EKS cluster name"
  value       = module.eks.cluster_name
}

output "region" {
  description = "AWS region"
  value       = var.region
}

output "karpenter_irsa_role_arn" {
  description = "Karpenter IRSA role ARN"
  value       = module.karpenter.iam_role_arn
}

output "karpenter_node_role_name" {
  description = "Karpenter node IAM role name"
  value       = module.karpenter.node_iam_role_name
}

output "rds_endpoint" {
  description = "RDS endpoint"
  value       = aws_db_instance.erp.endpoint
}

output "db_secret_arn" {
  description = "Secrets Manager secret ARN for DB credentials"
  value       = aws_secretsmanager_secret.erp_db.arn
}

Your entire infrastructure is now defined as code. Next up, you will initialize Terraform, review the execution plan, and provision all of these resources on AWS.

Provision Infrastructure with Terraform

Your Terraform configuration files define the entire infrastructure. Now it is time to turn that code into real AWS resources.

In this step you will initialize Terraform, review the execution plan, provision the full stack (VPC, EKS, RDS, Secrets Manager), and seed the database so it is ready for your microservices.

In this step, get ready to:
  • Initialize Terraform and review the execution plan.
  • Apply the infrastructure and wait for EKS, RDS, and VPC to provision.
  • Configure kubectl and seed the RDS database with the inventory table.
Initialize and plan the infrastructure

Before Terraform can create anything, it needs to download the provider plugins and modules your configuration references.

  • In your terminal, make sure you are inside the infra/ directory by running:
cd infra
  • The db_password variable in your configuration has no default value. Set it as an environment variable so Terraform can read it without prompting you. Choose a strong password (at least 8 characters, mix of letters and numbers) and remember it for later:
export TF_VAR_db_password="your-secure-db-password-here"

Why an environment variable for the password?

Terraform reads any environment variable prefixed with TF_VAR_ and maps it to the matching input variable. This avoids typing the password into the terminal on every plan and apply, and keeps it out of your shell history (unlike the -var flag).

  • Initialize Terraform by running:
terraform init
  • You should see Terraform downloading the AWS provider and the VPC and EKS modules. It finishes with a green message saying Terraform has been successfully initialized.

Seeing provider version errors?

Make sure your AWS CLI credentials are configured for us-east-1 and that you are running Terraform 1.5.7 or newer. Run terraform version to confirm.

help me fix Terraform init errors in my EKS project

  • Preview what Terraform will create by running:
terraform plan

Terraform shows every resource it plans to create. You should see a summary that includes roughly 50-70 resources to add, including VPC subnets, NAT gateway, EKS cluster, managed node group, RDS instance, Secrets Manager secret, and Karpenter IAM roles.

What should I look for in the plan?

  • VPC with 6 subnets (2 public, 2 private, 2 database) and a single NAT gateway.
  • EKS cluster named erp-cluster with Kubernetes 1.35 and a system node group of 2x t3.medium instances.
  • RDS db.t3.micro instance running PostgreSQL 16 with 20GB gp3 storage.
  • Secrets Manager secret at erp-cluster/db-credentials storing the connection details.
  • IAM roles for Karpenter (IRSA) and an SQS queue named erp-cluster.
Apply the infrastructure

The plan looks good. Time to provision everything on AWS.

  • Apply the infrastructure by running:
terraform apply -auto-approve

How long does this take?

This command takes approximately 15-20 minutes. EKS cluster creation takes about 12 minutes, and RDS provisioning takes about 8 minutes. They run in parallel, so total time is closer to 15 minutes.

The -auto-approve flag skips the interactive yes/no confirmation. You already reviewed the plan in the previous step.

What is Karpenter and why not Cluster Autoscaler?

While your infrastructure provisions, here is why this project uses Karpenter instead of the traditional Cluster Autoscaler. Cluster Autoscaler scales entire Auto Scaling Groups. It picks from pre-defined node group sizes, which often leads to over-provisioning.

Karpenter evaluates each pending pod's resource requests and provisions exactly the right instance type to fit them. It can mix instance families and purchase types (on-demand or spot) in a single NodePool. This makes it far more cost-efficient and responsive for enterprise workloads.

Why External Secrets Operator for credentials?

In enterprise environments, hardcoding database passwords into Kubernetes manifests or environment variable files is a security failure. The External Secrets Operator syncs secrets from AWS Secrets Manager into native Kubernetes Secrets automatically.

This means credentials rotate in one place (Secrets Manager) and propagate to all pods without redeploying. Your Terraform code already stored the database credentials there. In Step 4 you will install ESO to pull them into the cluster.

  • Wait for the apply to finish. You should see green output ending with Apply complete! followed by your outputs (cluster endpoint, RDS endpoint, secret ARN, and Karpenter role ARN).

Apply failed or timed out?

If you see a timeout or quota error, check that your AWS account has not hit EC2 or VPC limits in us-east-1. Common issues include hitting the default limit of 5 Elastic IPs or 5 VPCs per region.

If apply failed partway through, run terraform apply -auto-approve again. Terraform is idempotent and will pick up where it left off.

help me fix my terraform apply failure for EKS and RDS

Configure kubectl and seed the database

The cluster is live but your local kubectl does not know about it yet. You also need to initialize the RDS database with the inventory table your microservices expect.

  • Point kubectl at your new EKS cluster by running:
aws eks update-kubeconfig --name erp-cluster --region us-east-1

What does this command do?

This command fetches the cluster's CA certificate and API endpoint from AWS and writes them into your ~/.kube/config file. After this, every kubectl command targets erp-cluster.

  • Verify the system nodes are ready by running:
kubectl get nodes
  • You should see 2 nodes with status Ready. These are the t3.medium instances from your system node group.

Nodes not showing Ready?

If nodes show NotReady, wait 1-2 minutes for the kubelet to register. If no nodes appear at all, confirm that aws eks update-kubeconfig completed without errors and that you are using the same AWS credentials that created the cluster.

help me troubleshoot why my EKS nodes are not ready

Now you will initialize the RDS database. The database is in a private subnet and not reachable from the internet. You will connect from inside the cluster using a temporary PostgreSQL pod.

  • Store the RDS endpoint in a shell variable by running:
RDS_ENDPOINT=$(terraform output -raw rds_endpoint)
  • Seed the database by piping init.sql into a temporary PostgreSQL pod that connects to RDS:
cat init.sql | kubectl run psql-client --rm -i --image=postgres:16 --restart=Never -- psql "postgresql://erpadmin:${TF_VAR_db_password}@${RDS_ENDPOINT}/erpdb"

What is happening here?

  • kubectl run psql-client creates a temporary pod running the postgres:16 image inside your EKS cluster.
  • The --rm flag auto-deletes the pod when it exits. The -i flag passes stdin (your SQL file) into the pod.
  • Because the pod runs inside the VPC, it can reach the private RDS instance. The connection string uses the erpadmin username and the password you set earlier.
  • init.sql creates the inventory table and inserts 5 sample rows.
  • You should see output indicating the table was created and rows were inserted. Look for lines like CREATE TABLE and INSERT 0 5.

Connection timed out or refused?

If the pod cannot reach RDS, verify that the RDS security group allows inbound traffic on port 5432 from the EKS node security group. Your Terraform code already configures this, but if you modified anything, check rds.tf for the ingress rule.

Also confirm that the TF_VAR_db_password variable is still set in your current terminal session. Run echo $TF_VAR_db_password to check.

help me connect to my private RDS instance from an EKS pod

Your entire infrastructure stack is now live. The VPC, EKS cluster, RDS database, and Secrets Manager secret are all provisioned and verified. Next, you will install Karpenter and the External Secrets Operator to bring intelligent autoscaling and secure credential injection to the cluster.

Install Karpenter and External Secrets Operator

Your infrastructure is provisioned and your database is seeded. The next challenge is making the cluster intelligent about scaling and secure about secrets.

In this step, you will install two critical operators via Helm. Karpenter handles node autoscaling by provisioning right-sized EC2 instances for pending pods. External Secrets Operator syncs credentials from AWS Secrets Manager into Kubernetes Secrets so your services never touch hardcoded passwords.

In this step, get ready to:
  • Install Karpenter via Helm and configure IRSA for node provisioning.
  • Apply NodePool and EC2NodeClass CRDs to define scaling rules.
  • Install External Secrets Operator and sync RDS credentials into Kubernetes.
Install Karpenter via Helm

Karpenter needs to authenticate with AWS to launch and terminate EC2 instances. The Terraform module from the previous step already created an IRSA role for Karpenter. Now you pass that role ARN to the Helm chart so the Karpenter service account can assume it.

  • Retrieve the Karpenter IRSA role ARN from your Terraform outputs by running:
export KARPENTER_IRSA_ROLE_ARN=$(terraform -chdir=infra output -raw karpenter_irsa_role_arn)
  • Verify the value was exported by running:
echo $KARPENTER_IRSA_ROLE_ARN

You should see an ARN that looks like arn:aws:iam::123456789012:role/erp-cluster-karpenter-....

  • Install Karpenter v1.12.1 from the OCI registry by running:
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version "1.12.1" \
  --namespace "kube-system" \
  --create-namespace \
  --set "settings.clusterName=erp-cluster" \
  --set "settings.interruptionQueue=erp-cluster" \
  --set "serviceAccount.annotations.eks\.amazonaws\.com/role-arn=${KARPENTER_IRSA_ROLE_ARN}" \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

What does this command do?

This installs Karpenter into the kube-system namespace from the official OCI registry at public.ecr.aws/karpenter/karpenter.

  • settings.clusterName tells Karpenter which EKS cluster it manages.
  • settings.interruptionQueue points to the SQS queue that receives EC2 spot interruption events.
  • serviceAccount.annotations binds the Karpenter service account to the IRSA role so it can call AWS APIs.
  • --wait ensures the command only returns once all Karpenter pods are running.
  • Verify Karpenter is running by checking its pods:
kubectl get pods -n kube-system -l app.kubernetes.io/name=karpenter

You should see one or more Karpenter pods with a status of Running.

Karpenter pods not running?

  • Check that the IRSA role ARN was exported correctly. Run echo $KARPENTER_IRSA_ROLE_ARN and confirm it is not empty.
  • Verify your system node group has enough capacity. Two t3.medium nodes should be sufficient.
  • Run kubectl describe pod -n kube-system -l app.kubernetes.io/name=karpenter to check events for scheduling failures.

Help me troubleshoot why Karpenter pods are not reaching Running status in kube-system.

Apply NodePool and EC2NodeClass CRDs

Karpenter is running, but it does not know what kinds of nodes to provision yet. You define this with two custom resources: a NodePool (what constraints to apply) and an EC2NodeClass (which AWS resources to use).

  • Create the Karpenter configuration file by running:
cat > k8s/karpenter-nodepool.yaml << 'EOF'
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-size
          operator: In
          values: ["medium", "large", "xlarge"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 720h
  limits:
    cpu: 100
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiSelectorTerms:
    - alias: al2023@latest
  role: erp-cluster-karpenter-node
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: erp-cluster
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: erp-cluster
EOF

What do these resources define?

The NodePool sets the rules Karpenter uses to choose instances:

  • Allows both on-demand and spot capacity types for cost optimization.
  • Restricts to compute (c), memory (m), and general (r) instance families in medium, large, and xlarge sizes.
  • Caps total cluster CPU at 100 cores to prevent runaway scaling.
  • Enables consolidation so underutilized nodes get removed after 1 minute.

The EC2NodeClass tells Karpenter where to place new nodes. It discovers subnets and security groups using the karpenter.sh/discovery: erp-cluster tag you applied via Terraform in Step 2.

  • Apply the NodePool and EC2NodeClass by running:
kubectl apply -f k8s/karpenter-nodepool.yaml
  • Verify both resources were created by running:
kubectl get nodepools
kubectl get ec2nodeclasses

You should see the default NodePool and default EC2NodeClass listed.

Why Karpenter over Cluster Autoscaler?

Cluster Autoscaler scales entire node groups regardless of what pods need. Karpenter evaluates each pending pod's resource requests and provisions the cheapest instance type that satisfies all of them. This bin-packing approach saves money and responds faster.

Karpenter also handles spot interruptions natively via the SQS queue, gracefully draining nodes before AWS reclaims them.

NodePool shows a warning or error?

  • Confirm Karpenter is running first. The CRDs are only available after the Helm chart installs them.
  • Double check that your role field in the EC2NodeClass matches your Terraform output. Run terraform -chdir=infra output karpenter_node_role_name to confirm.

Help me fix my Karpenter NodePool or EC2NodeClass configuration errors.

Install External Secrets Operator and sync database credentials

Your RDS credentials live in AWS Secrets Manager. Instead of copying them manually into Kubernetes, the External Secrets Operator automatically syncs them into a native Kubernetes Secret that your pods can mount as environment variables.

  • Add the External Secrets Helm repository by running:
helm repo add external-secrets https://charts.external-secrets.io
  • Install External Secrets Operator into the external-secrets namespace by running:
helm install external-secrets external-secrets/external-secrets \
  -n external-secrets \
  --create-namespace \
  --wait

What does this install?

The Helm chart deploys three components: the core controller, a webhook for validation, and a cert controller for TLS. Together they watch for ExternalSecret resources and sync data from cloud providers into Kubernetes Secrets.

  • Verify the operator is running by checking pods in the external-secrets namespace:
kubectl get pods -n external-secrets

You should see three pods (controller, webhook, cert-controller) all in Running status.

  • Create the ExternalSecret manifest that connects to AWS Secrets Manager and syncs your database credentials by running:
cat > k8s/external-secret.yaml << 'EOF'
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: erp-db-credentials
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: erp-db-credentials
    creationPolicy: Owner
  data:
    - secretKey: DB_HOST
      remoteRef:
        key: erp-cluster/db-credentials
        property: host
    - secretKey: DB_PORT
      remoteRef:
        key: erp-cluster/db-credentials
        property: port
    - secretKey: DB_USER
      remoteRef:
        key: erp-cluster/db-credentials
        property: username
    - secretKey: DB_PASSWORD
      remoteRef:
        key: erp-cluster/db-credentials
        property: password
    - secretKey: DB_NAME
      remoteRef:
        key: erp-cluster/db-credentials
        property: dbname
EOF

What does this manifest do?

The ClusterSecretStore defines how to authenticate with AWS Secrets Manager. It uses IRSA via a service account reference so no static credentials are needed.

The ExternalSecret tells the operator to read the JSON secret at erp-cluster/db-credentials and extract each property (host, port, username, password, dbname) into a Kubernetes Secret named erp-db-credentials. The secret refreshes every hour.

  • Apply the manifest by running:
kubectl apply -f k8s/external-secret.yaml
  • Verify the ExternalSecret synced successfully by running:
kubectl get externalsecret erp-db-credentials

The STATUS column should show SecretSynced. This means the operator pulled data from Secrets Manager and created the Kubernetes Secret.

  • Confirm the Kubernetes Secret exists by running:
kubectl get secrets erp-db-credentials

You should see the erp-db-credentials secret listed with type Opaque and 5 data keys.

ExternalSecret stuck in error state?

  • Run kubectl describe externalsecret erp-db-credentials to see the error message in events.
  • The most common issue is IRSA permissions. Ensure the external-secrets-sa service account in the external-secrets namespace has an IAM role with secretsmanager:GetSecretValue permission for the erp-cluster/db-credentials secret.
  • Verify the secret name in Secrets Manager matches what you specified. Run aws secretsmanager list-secrets --query "SecretList[?Name=='erp-cluster/db-credentials']" to confirm.

Help me debug why my ExternalSecret is not syncing from AWS Secrets Manager.

✔️ Awesome, I've got everything!

Great. Double check you have Karpenter running in kube-system, both the NodePool and EC2NodeClass created, and the erp-db-credentials secret synced in the default namespace.

ⓧ I'd like to double check the full code

Here is the complete k8s/karpenter-nodepool.yaml file:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-size
          operator: In
          values: ["medium", "large", "xlarge"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 720h
  limits:
    cpu: 100
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiSelectorTerms:
    - alias: al2023@latest
  role: erp-cluster-karpenter-node
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: erp-cluster
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: erp-cluster

Here is the complete k8s/external-secret.yaml file:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: erp-db-credentials
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: erp-db-credentials
    creationPolicy: Owner
  data:
    - secretKey: DB_HOST
      remoteRef:
        key: erp-cluster/db-credentials
        property: host
    - secretKey: DB_PORT
      remoteRef:
        key: erp-cluster/db-credentials
        property: port
    - secretKey: DB_USER
      remoteRef:
        key: erp-cluster/db-credentials
        property: username
    - secretKey: DB_PASSWORD
      remoteRef:
        key: erp-cluster/db-credentials
        property: password
    - secretKey: DB_NAME
      remoteRef:
        key: erp-cluster/db-credentials
        property: dbname

Your cluster now has intelligent autoscaling and zero-trust secrets management. Next up, you will build and deploy the microservices that use these credentials to connect to PostgreSQL.

Build, Push, and Deploy Microservices

Your infrastructure is running, secrets are flowing from AWS Secrets Manager into Kubernetes, and your database is seeded with inventory data.

Now it is time to containerize the two microservices, push them to Amazon ECR, and deploy them to the cluster where they can connect to the database using the injected credentials.

In this step, get ready to:
  • Create ECR repositories and push both Docker images.
  • Deploy both microservices to EKS with secure database connectivity.
  • Verify end-to-end communication from Order Service to Inventory Service to PostgreSQL.
Create ECR repositories and push Docker images

Before Kubernetes can pull your container images, those images need to live in a registry that your EKS nodes can access. ECR is the native container registry for AWS, and your EKS nodes already have permission to pull from it.

  • First, retrieve your AWS account ID by running this command:
aws sts get-caller-identity --query Account --output text
  • Note down the 12-digit number that appears. You will use it in the next commands.
  • Store your account ID in a shell variable by running:
# Save your account ID for use in subsequent commands
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
echo $ACCOUNT_ID
  • Create both ECR repositories by running these commands:
aws ecr create-repository --repository-name erp/inventory-service --region us-east-1
aws ecr create-repository --repository-name erp/order-service --region us-east-1

What do these commands do?

Each command creates a private container registry in ECR using a namespaced path (erp/ prefix). This keeps both service images organized under a single namespace.

The full image URI will be ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/erp/inventory-service once pushed.

  • Authenticate Docker to your ECR registry by running:
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com

You should see Login Succeeded in your terminal output.

  • Build and push the Inventory Service image by running:
# Build the inventory-service image
docker build -t erp/inventory-service services/inventory-service/

# Tag it with the ECR repository URI
docker tag erp/inventory-service:latest $ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/erp/inventory-service:latest

# Push to ECR
docker push $ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/erp/inventory-service:latest
  • Build and push the Order Service image by running:
# Build the order-service image
docker build -t erp/order-service services/order-service/

# Tag it with the ECR repository URI
docker tag erp/order-service:latest $ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/erp/order-service:latest

# Push to ECR
docker push $ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/erp/order-service:latest

Each push should show layers being uploaded. Once complete, both images are available for your EKS cluster to pull.

Push failing with authorization errors?

  • ECR authentication tokens expire after 12 hours. Re-run the aws ecr get-login-password command to get a fresh token.
  • Confirm your AWS CLI region matches the ECR region (us-east-1).
  • Make sure Docker Desktop is running before attempting any build or push commands.

Why can't I push my Docker image to ECR?

Write and apply Kubernetes manifests

With images in ECR, you need Kubernetes Deployments and Services to run them on the cluster. The Inventory Service connects directly to PostgreSQL using the credentials from the erp-db-credentials secret. The Order Service calls the Inventory Service via internal cluster DNS.

  • Create the Inventory Service manifest by creating a file k8s/inventory-service.yaml with the following content:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inventory-service
  labels:
    app: inventory-service
spec:
  replicas: 2
  selector:
    matchLabels:
      app: inventory-service
  template:
    metadata:
      labels:
        app: inventory-service
    spec:
      containers:
        - name: inventory-service
          image: ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/erp/inventory-service:latest
          ports:
            - containerPort: 8080
          envFrom:
            - secretRef:
                name: erp-db-credentials
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 250m
              memory: 256Mi
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: inventory-service
spec:
  selector:
    app: inventory-service
  ports:
    - port: 8080
      targetPort: 8080
  type: ClusterIP

What does this manifest do?

  • envFrom with secretRef: erp-db-credentials injects all keys from the Kubernetes Secret (DB_HOST, DB_USER, DB_PASSWORD, DB_NAME, DB_PORT) as environment variables into the container.
  • Liveness and readiness probes hit the /health endpoint so Kubernetes can restart unhealthy pods and route traffic only to ready ones.
  • The ClusterIP Service makes the Inventory Service reachable at inventory-service:8080 from other pods inside the cluster.
  • Replace ACCOUNT_ID in the image field with your actual AWS account ID. You can use sed to do this automatically:
sed -i "s/ACCOUNT_ID/$ACCOUNT_ID/g" k8s/inventory-service.yaml
  • Create the Order Service manifest by creating a file k8s/order-service.yaml with the following content:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  labels:
    app: order-service
spec:
  replicas: 2
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      containers:
        - name: order-service
          image: ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/erp/order-service:latest
          ports:
            - containerPort: 8080
          env:
            - name: INVENTORY_SERVICE_URL
              value: "http://inventory-service:8080"
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 250m
              memory: 256Mi
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 3
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: order-service
spec:
  selector:
    app: order-service
  ports:
    - port: 8080
      targetPort: 8080
  type: ClusterIP

How does inter-service communication work?

The INVENTORY_SERVICE_URL environment variable is set to http://inventory-service:8080. Kubernetes DNS resolves inventory-service to the ClusterIP of the Inventory Service. No hardcoded IPs needed.

The Order Service does not need database credentials. It fetches inventory data by calling the Inventory Service over HTTP.

  • Replace ACCOUNT_ID in the Order Service manifest as well:
sed -i "s/ACCOUNT_ID/$ACCOUNT_ID/g" k8s/order-service.yaml
  • Apply both manifests to your cluster by running:
kubectl apply -f k8s/inventory-service.yaml
kubectl apply -f k8s/order-service.yaml
  • Wait for all pods to reach the Running state by running:
kubectl get pods -w

You should see 4 pods total (2 for each service) transition to Running with 1/1 in the READY column. Press Ctrl+C once all pods are ready.

Pods stuck in CrashLoopBackOff?

  • Run kubectl logs <pod-name> to check the container logs. A database connection error means the erp-db-credentials secret may not have synced yet.
  • Run kubectl get secret erp-db-credentials to verify the secret exists. If missing, check that the ExternalSecret is healthy with kubectl get externalsecret.
  • Confirm your image URI is correct. Run kubectl describe pod <pod-name> and look for image pull errors.

Why are my Kubernetes pods stuck in CrashLoopBackOff?

Verify the full end-to-end flow

Both services are running, but you need to confirm the complete data path works. The Order Service calls the Inventory Service, which queries PostgreSQL and returns real data.

  • Port-forward the Order Service to your local machine by running:
kubectl port-forward svc/order-service 8080:8080
  • Open a second terminal and test the /orders endpoint by running:
curl http://localhost:8080/orders

You should see a JSON response containing two orders with real inventory items from PostgreSQL (Widget A, Widget B, Gadget C, Component D).

What should the response look like?

The response contains an orders array with two entries. Order ORD-001 has status processing with the first two inventory items. Order ORD-002 has status shipped with the next two items. Each item includes id, name, quantity, and warehouse fields pulled directly from RDS.

  • Confirm the synced secret exists by running:
kubectl get secret erp-db-credentials

You should see the secret listed with type Opaque and 5 data keys. This proves the External Secrets Operator successfully synced credentials from AWS Secrets Manager into the cluster without any hardcoded values.

  • Press Ctrl+C in the first terminal to stop the port-forward.

Getting 502 or connection refused from curl?

  • A 502 response means the Order Service cannot reach the Inventory Service. Check that both Inventory Service pods are Running and Ready.
  • Connection refused means the port-forward dropped. Re-run the kubectl port-forward command.
  • If you see a database error in the response, the Inventory Service cannot reach RDS. Verify the RDS security group allows inbound from the EKS node security group on port 5432.

Help me debug my microservice communication on EKS

Your enterprise ERP platform is fully operational. Two microservices are communicating over cluster DNS, reading real data from PostgreSQL, with credentials managed entirely through AWS Secrets Manager and the External Secrets Operator. Next up, you will push the boundaries of this cluster with Karpenter autoscaling.

Secret mission

Scale with Karpenter and Seed the Database

Your EKS cluster can provision nodes on its own. Deploy a load generator to push Karpenter into action and watch it scale your infrastructure in real time.

Clean Up Your Resources

Clean Up Your Resources

This project runs on paid AWS infrastructure that charges continuously. Decide whether to keep your resources running for follow-on projects, or delete them entirely to stop costs.

Cost warning

Your EKS cluster, RDS instance, NAT Gateway, and EC2 nodes cost approximately $0.35/hr ($8.40/day) while running. EKS and RDS charge continuously and cannot be paused without deletion.

If you are not continuing to follow-on projects immediately, delete your resources now to avoid unexpected charges.

Resources you used:

  • EKS cluster (erp-cluster) with system node group (2x t3.medium).
  • NAT Gateway (single, in erp-cluster-vpc).
  • RDS PostgreSQL instance (erp-cluster-db, db.t3.micro).
  • Secrets Manager secret (erp-cluster/db-credentials).
  • SQS queue (erp-cluster).
  • ECR repositories (erp/inventory-service and erp/order-service).
  • VPC (erp-cluster-vpc) with subnets, route tables, and internet gateway.
  • Local project files on your Desktop.

Keep everything running

Choose this if you plan to continue with follow-on projects in this series (CI/CD, Observability, Service Mesh). No action needed.

  • Your infrastructure stays running and billed at approximately $0.35/hr.
  • Karpenter-provisioned nodes only incur cost when workloads demand them. The load generator is already scaled to 0.
  • The CronJob db-seeder continues running hourly, adding rows to PostgreSQL. This has no additional cost impact.

Pause - I'll come back to this later

EKS and RDS charge continuously and cannot be paused without deletion. There is no true pause option for this infrastructure.

If you want to stop costs temporarily, you must destroy and re-provision later. Follow the Delete tab instructions instead.

Delete - I don't want to use this again

Remove all AWS resources and local files. This is irreversible.

  • Delete the ECR repositories by running these commands:
aws ecr delete-repository --repository-name erp/inventory-service --force
aws ecr delete-repository --repository-name erp/order-service --force
  • Destroy all Terraform-managed infrastructure (EKS, RDS, VPC, Secrets Manager, SQS, IAM roles) by navigating to your infra/ directory and running:
cd infra
terraform destroy -auto-approve

What does terraform destroy remove?

This removes the EKS cluster, RDS instance, NAT Gateway, VPC, Secrets Manager secret, SQS queue, and all associated IAM roles. It takes approximately 10-15 minutes to complete.

  • After Terraform finishes, delete your local project files by running:
cd ~/Desktop
rm -rf infra services k8s

Nice Work!

Nice Work!

Nice work! You just built a production-grade enterprise ERP platform on AWS EKS, provisioned entirely through Terraform with intelligent autoscaling, secure secrets management, and real database connectivity.

You've learned how to:

  • Provision a complete VPC, EKS cluster, and RDS PostgreSQL database using Terraform modules and manage all infrastructure as code.
  • Install Karpenter and External Secrets Operator via Helm to deliver intelligent node autoscaling and zero-hardcoded-credential secret injection into Kubernetes pods.
  • Build, containerize, and deploy two database-backed microservices with real inter-service communication, health checks, and credentials sourced securely from AWS Secrets Manager.
  • (Secret Mission) Triggered Karpenter autoscaling with a load generator, observed the full node lifecycle from provisioning to consolidation, and created a CronJob to seed the database with sample ERP data.

Ready to quiz yourself?