Deploy a RAG Chatbot to AWS EKS

Build and deploy a containerized RAG chatbot on Kubernetes using Terraform.

Introduction

30 Second Summary

Every team building AI-powered tools eventually faces the same question: how do you get this thing running reliably in the cloud, not just on your laptop? The gap between a working prototype and a production deployment is where most projects stall.

In this project, you will deploy a RAG chatbot to a Kubernetes cluster on AWS. You will write the entire infrastructure from scratch using Terraform, containerize the app with Docker, and troubleshoot a real-world pod permissions failure along the way.

What You'll Build

By the end, you will have a live REST endpoint where you send a question about AWS services and get back an AI-generated answer pulled from documents you uploaded.

By the end of this project, you'll have:

  • A RAG chatbot running on Amazon EKS that you can query with a simple curl command and receive answers grounded in your own documents.
  • A Kubernetes pod that securely calls Amazon Bedrock using EKS Pod Identity with zero hardcoded credentials.
  • A complete Terraform configuration that creates and destroys your entire cloud infrastructure (VPC, cluster, container registry, IAM roles) with a single command.
  • Secret Mission: Add a GitHub Actions CI/CD pipeline that automatically builds, pushes, and deploys your chatbot on every git push.

Are there any prerequisites?

This project assumes basic familiarity with Docker, Kubernetes concepts, and Terraform syntax. You will also need an AWS account with billing enabled and Bedrock model access for Amazon Titan Text Embeddings V2 and Anthropic Claude 3 Haiku.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters. This step is your chance to commit to the goal before any hands-on work begins.

Set Up Tools and Scaffold the RAG App

Your RAG chatbot needs to work on your machine before you add any infrastructure complexity. Starting with a running app lets you verify Amazon Bedrock connectivity and test the question-and-answer flow locally.

In this step, you will scaffold a Flask app that loads documents into a FAISS vector store using Bedrock Titan embeddings, then answers questions using Claude 3 Haiku through LangChain.

In this step, get ready to:
  • Create the project directory and install Python dependencies.
  • Write a Flask RAG app that answers questions from your documents.
  • Test the app locally to confirm Bedrock connectivity works.
Create the project and install dependencies

Your tools are verified and ready. Now you need a project folder with the right Python packages installed.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir rag-chatbot
cd rag-chatbot
  • Create and activate a Python virtual environment by running:
python3 -m venv venv
source venv/bin/activate

Your terminal prompt now shows (venv) at the beginning, confirming the virtual environment is active.

What is a virtual environment?

A virtual environment isolates your project's Python packages from your system Python. Any packages you install stay inside the rag-chatbot folder and won't interfere with other projects.

  • Create the requirements.txt file by running:
cat > requirements.txt << 'EOF'
flask
langchain
langchain-aws
langchain-community
faiss-cpu
pypdf
boto3
EOF
  • Install the dependencies by running:
pip install -r requirements.txt

You should see pip downloading and installing all seven packages. The output ends with a success message listing the installed packages.

Seeing install errors?

Make sure your virtual environment is active. Your terminal prompt should show (venv) at the beginning. If it does not, run source venv/bin/activate again and retry the install.

help me fix a pip install error in my RAG chatbot project

Write the Flask RAG app

Now you will build the Flask server that loads your documents, creates embeddings, and exposes a question-answering endpoint. First you need a document for the chatbot to answer questions from.

  • Create the docs/ folder and a sample document by running:
mkdir docs
cat > docs/aws-services.txt << 'EOF'
Amazon S3 (Simple Storage Service) is an object storage service that offers industry-leading scalability, data availability, security, and performance. You can use S3 to store and retrieve any amount of data at any time from anywhere on the web. S3 stores data as objects within buckets.

Amazon EC2 (Elastic Compute Cloud) provides resizable compute capacity in the cloud. It allows you to launch virtual servers, configure security and networking, and manage storage. EC2 makes web-scale cloud computing easier for developers.

AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers. You pay only for the compute time you consume. Lambda runs your code on high-availability compute infrastructure.

Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. DynamoDB lets you offload the administrative burdens of operating and scaling a distributed database.

Amazon VPC (Virtual Private Cloud) lets you provision a logically isolated section of the AWS Cloud where you can launch AWS resources in a virtual network that you define. You have complete control over your virtual networking environment.
EOF
  • Confirm the file was created by running:
ls docs/

You should see aws-services.txt listed in the output.

  • Create app.py in your editor and paste the imports and document loading section:
import os
from flask import Flask, request, jsonify
from langchain_community.vectorstores import FAISS
from langchain_aws import BedrockEmbeddings, ChatBedrock
from langchain.chains import RetrievalQA
from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

app = Flask(__name__)

# Load documents from docs/ folder
loader = DirectoryLoader("docs/", glob="**/*.txt", loader_cls=TextLoader)
documents = loader.load()

# Split documents into smaller chunks for embedding
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = text_splitter.split_documents(documents)

What does this code do?

  • DirectoryLoader scans the docs/ folder for all .txt files and loads them as LangChain documents.
  • RecursiveCharacterTextSplitter breaks those documents into overlapping 1000-character chunks so each chunk fits within the embedding model's context window.
  • Below the code you just pasted, add the embeddings, vector store, LLM, and retrieval chain:
# Create embeddings using Bedrock Titan Text Embeddings V2
embeddings = BedrockEmbeddings(
    model_id="amazon.titan-embed-text-v2:0",
    region_name=os.environ.get("AWS_REGION", "us-east-1")
)

# Build FAISS vector store from document chunks
vectorstore = FAISS.from_documents(chunks, embeddings)

# Set up Claude 3 Haiku as the answering LLM
llm = ChatBedrock(
    model_id="anthropic.claude-3-haiku-20240307-v1:0",
    region_name=os.environ.get("AWS_REGION", "us-east-1")
)

# Create a retrieval QA chain
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever()
)

What does this code do?

  • BedrockEmbeddings converts each text chunk into a numerical vector using Amazon Titan Text Embeddings V2.
  • FAISS.from_documents stores those vectors in an in-memory index for fast similarity search.
  • ChatBedrock connects to Claude 3 Haiku, which generates the final answer.
  • RetrievalQA ties it all together: it retrieves the most relevant chunks, then passes them to Claude along with the user's question.
  • Save app.py and run the script to verify Bedrock connectivity:
python app.py

The script calls Bedrock to generate embeddings for your document chunks, then exits back to your prompt without errors. That silence means your AWS credentials and Bedrock model access are working.

Seeing a credentials or access error?

If the script crashes with an error about missing credentials or access denied, make sure your AWS CLI is configured with credentials that have Bedrock model access enabled for us-east-1.

Also verify you have enabled the Titan Text Embeddings V2 and Claude 3 Haiku models in the Bedrock console under Model access.

help me fix a Bedrock access error when running my RAG chatbot locally

  • Now add the Flask route and entry point below the existing code in app.py:
@app.route("/ask", methods=["POST"])
def ask():
    # Parse the question from the request body
    data = request.get_json()
    question = data.get("question", "")
    if not question:
        return jsonify({"error": "No question provided"}), 400
    # Run the question through the RAG chain
    result = qa_chain.invoke({"query": question})
    return jsonify({"answer": result["result"]})

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

What does this code do?

  • The /ask route accepts a POST request with a JSON body containing a question field.
  • It passes the question through the RAG chain and returns the AI-generated answer as JSON.
  • The if __name__ block starts the Flask server on port 5000 when you run the script directly.
  • Save app.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved app.py in the rag-chatbot folder.

ⓧ I'd like to double check the full code

Your app.py should contain all three chunks above in order. Scroll up and confirm you have:

  • The imports and document loading section (starting with import os).
  • The embeddings, vector store, LLM, and chain section (starting with the BedrockEmbeddings comment).
  • The Flask route and entry point section (starting with @app.route).

All three sections go into the same app.py file, one after the other.

Test the app locally

Everything is in place. Time to run the full app and see if your chatbot can answer questions about AWS services.

  • Start the Flask server by running:
python app.py

When the app starts, it calls Bedrock to generate embeddings for your document chunks. This takes a few seconds. You should then see output showing Flask is running on http://0.0.0.0:5000.

  • Open a second terminal tab (keep the Flask server running in the first one).
  • Navigate back to your project folder and activate the virtual environment:
cd ~/Desktop/rag-chatbot
source venv/bin/activate

Before you run the next command, predict what you think the chatbot will answer about S3. Will it use your document's content, or general knowledge?

  • Test the chatbot by running:
curl http://localhost:5000/ask -X POST -H "Content-Type: application/json" -d '{"question": "What is S3?"}'

You should see a JSON response with an AI-generated answer about Amazon S3, grounded in the document you created. The answer references details from your aws-services.txt file like object storage, buckets, and scalability.

What just happened?

Your question traveled through the full RAG pipeline: Flask received it, the retriever found the most relevant chunks from your FAISS vector store, and Claude 3 Haiku generated an answer using those chunks as context.

The answer is grounded in your documents rather than the model's general knowledge. This is what makes it retrieval-augmented generation.

Getting connection refused or an empty response?

Make sure the Flask server is still running in your first terminal tab. If you accidentally closed it, run python app.py again from the rag-chatbot directory with the virtual environment active.

help me debug why my RAG chatbot curl request is failing

Your RAG chatbot works locally. Next up, you will package it into a Docker container so it can run anywhere.

Containerize the App and Test with Docker

Your RAG chatbot runs on your machine and answers questions about AWS services. The next step toward production is packaging it so it can run anywhere.

Kubernetes runs containers, not raw Python scripts. To deploy your chatbot to EKS in the next step, you first need to wrap it in a Docker image and confirm it still works inside that container.

In this step, get ready to:
  • Write a Dockerfile that packages your Flask app and its dependencies.
  • Build the Docker image and run it locally to confirm the chatbot works inside a container.
Write a Dockerfile for the RAG chatbot

A Dockerfile is a recipe that tells Docker how to build your image. It specifies the base operating system, installs dependencies, copies your code in, and sets the command that runs when the container starts.

  • Create a new file called Dockerfile in your rag-chatbot folder (no file extension).
  • Add the following content to your Dockerfile:
# Use a lightweight Python 3.11 base image
FROM python:3.11-slim

# Set the working directory inside the container
WORKDIR /app

# Copy requirements first to leverage Docker layer caching
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of the application code
COPY . .

# Expose the Flask port
EXPOSE 5000

# Start the Flask app when the container runs
CMD ["python", "app.py"]

What does this Dockerfile do?

  • FROM python:3.11-slim starts from a minimal Linux image with Python 3.11 pre-installed.
  • WORKDIR /app creates and switches to the /app directory inside the container.
  • COPY requirements.txt . copies just the requirements file first. This lets Docker cache the installed packages so rebuilds are faster when only your code changes.
  • COPY . . copies everything else (your app code and the docs/ folder) into the container.
  • CMD ["python", "app.py"] tells Docker what command to run when the container starts.
  • Save Dockerfile.
  • Confirm the file exists by running:
ls Dockerfile

You should see Dockerfile printed in the output.

Don't see the file?

  • Make sure you are in the rag-chatbot directory. Run pwd to confirm.
  • The file must be named exactly Dockerfile with a capital D and no extension. Not dockerfile.txt or Dockerfile.txt.
  • Still stuck? Why can't Docker find my Dockerfile?

✔️ Awesome, I've got everything!

Great. Make sure you have saved the file before moving on.

ⓧ I'd like to double check the full code

Here is the complete Dockerfile you should have:

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 5000
CMD ["python", "app.py"]
Build the Docker image and run the container

With your Dockerfile ready, you can now build a Docker image from it. The image is a snapshot of your app and all its dependencies, frozen and ready to run anywhere Docker is installed.

  • Build the Docker image and tag it as rag-chatbot:latest by running:
docker build -t rag-chatbot:latest .

This takes a minute or two as Docker pulls the base image and installs your Python packages. You will see each Dockerfile instruction executing in sequence.

What does this command do?

  • -t rag-chatbot:latest tags (names) the resulting image so you can reference it easily.
  • . tells Docker to use the current directory as the build context, which includes your Dockerfile, app.py, requirements.txt, and docs/ folder.
  • Confirm the image was built successfully by running:
docker images rag-chatbot

You should see a row with rag-chatbot and the tag latest listed.

Build failed?

  • If you see a permissions error, make sure Docker Desktop is running. You can check by running docker info.
  • If pip install fails inside the build, double-check your requirements.txt has no typos. Each package name should be on its own line.
  • Help me fix my Docker build error

Now run the container to test your chatbot inside Docker. The container needs access to your AWS credentials so it can call Bedrock.

  • Start the container by running:
docker run -p 5000:5000 -e AWS_REGION=us-east-1 -v ~/.aws:/root/.aws:ro rag-chatbot:latest

What do these flags mean?

  • -p 5000:5000 maps port 5000 inside the container to port 5000 on your machine.
  • -e AWS_REGION=us-east-1 sets the AWS region environment variable inside the container.
  • -v ~/.aws:/root/.aws:ro mounts your local AWS credentials folder into the container as read-only. This lets the container call Bedrock without hardcoding any secrets into the image.

You should see Flask startup output in your terminal, ending with a line like Running on http://0.0.0.0:5000. The container is now running and listening for requests.

  • Open a second terminal window. Press Cmd+Space (macOS) or the Windows key (Windows), type Terminal, and press Enter.

Before you run the test, what do you think will happen? The app worked locally with Python. Will it also work inside the container?

  • Test the containerized chatbot by running:
curl http://localhost:5000/ask -X POST -H "Content-Type: application/json" -d '{"question": "What is S3?"}'

You should receive a JSON response with an AI-generated answer about Amazon S3, grounded in the documents from your docs/ folder. This confirms your RAG chatbot works correctly inside Docker.

Not getting a response?

  • Check the first terminal (where the container is running) for error messages. If you see a credentials error, make sure your ~/.aws folder exists and contains valid credentials.
  • If you see Connection refused, the container may not be running. Switch back to the first terminal and check for startup errors.
  • If port 5000 is already in use, stop any other process on that port. On Linux/macOS, run lsof -i :5000 to find the process.
  • Help me troubleshoot my containerized RAG chatbot
  • Once you have confirmed the response, stop the container by pressing Ctrl+C in the first terminal.

Your chatbot is containerized and proven to work inside Docker. Next up, you will write Terraform configuration to provision the EKS cluster that runs this container in the cloud.

Provision EKS Infrastructure with Terraform

You have a Docker container that runs your RAG chatbot locally. Now you need somewhere in the cloud to run it.

In this step, you will write a complete Terraform configuration that creates a VPC, an EKS cluster, an ECR repository, and an IAM role for pod permissions. One command provisions everything.

In this step, get ready to:
  • Write Terraform configuration files that define your entire cloud infrastructure.
  • Run terraform apply to provision a VPC, EKS cluster, and ECR repository in AWS.
  • Connect kubectl to the new cluster and verify nodes are running.
Write the Terraform variables file

Terraform uses variables to keep your configuration flexible. You will define three variables that control the region, cluster name, and node instance type.

  • Create a new directory called terraform inside your rag-chatbot folder by running these commands:
mkdir terraform
cd terraform
  • Confirm the directory exists by running:
ls ../

You should see terraform listed alongside your existing files like app.py and Dockerfile.

  • Create a file called variables.tf and add the following code:
# Region where all resources will be created
variable "region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

# Name for the EKS cluster
variable "cluster_name" {
  description = "EKS cluster name"
  type        = string
  default     = "rag-chatbot-cluster"
}

# EC2 instance type for the worker nodes
variable "node_instance_type" {
  description = "EC2 instance type for EKS nodes"
  type        = string
  default     = "t3.medium"
}

What does this code do?

Each variable block defines a configurable input for your Terraform configuration.

  • region sets where AWS creates your resources. us-east-1 is the default.
  • cluster_name gives your EKS cluster a consistent name across all resources that reference it.
  • node_instance_type controls the size of the EC2 machines that run your pods. t3.medium provides 2 vCPUs and 4 GB RAM.
  • Save variables.tf.
Write the main Terraform configuration: provider and VPC

The main configuration file defines the AWS provider and the network your cluster will live in. You will use the community VPC module to create public and private subnets across two availability zones.

  • Create a file called main.tf and add the Terraform and provider blocks:
# Pin Terraform and provider versions
terraform {
  required_version = ">= 1.3.2"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.95"
    }
  }
}

# Configure the AWS provider with the chosen region
provider "aws" {
  region = var.region
}

What does this code do?

The terraform block locks the minimum Terraform version and pins the AWS provider to 5.95+ (required by the EKS module v20.x). The provider block tells Terraform which region to target using the variable you just defined.

  • Below the provider block in main.tf, add the VPC module:
# Create a VPC with public and private subnets
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.cluster_name}-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["${var.region}a", "${var.region}b"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true

  # Tags required for EKS to discover subnets
  public_subnet_tags = {
    "kubernetes.io/role/elb" = 1
  }
  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = 1
  }
}

What does this code do?

The VPC module creates an isolated network for your cluster. Private subnets hold your EKS nodes (no direct internet access). Public subnets hold load balancers that route traffic to your pods.

The subnet tags tell EKS which subnets to use when creating load balancers. single_nat_gateway = true keeps costs down by sharing one NAT Gateway across both availability zones.

  • Save main.tf.
Add the EKS cluster module

The EKS module handles the heavy lifting of creating a Kubernetes control plane and managed node group. It also installs the Pod Identity Agent addon that your chatbot will need later.

  • Below the VPC module in main.tf, add the EKS module:
# Create the EKS cluster with a managed node group
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.31"

  cluster_name    = var.cluster_name
  cluster_version = "1.31"

  # Install required cluster addons
  cluster_addons = {
    coredns                = {}
    eks-pod-identity-agent = {}
    kube-proxy             = {}
    vpc-cni                = {}
  }

  cluster_endpoint_public_access           = true
  enable_cluster_creator_admin_permissions = true

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets
}

What does this code do?

This creates your EKS cluster running Kubernetes 1.31. The cluster_addons map installs four essential components: DNS resolution (coredns), pod-level AWS credentials (eks-pod-identity-agent), network proxying (kube-proxy), and pod networking (vpc-cni).

enable_cluster_creator_admin_permissions gives your current AWS identity admin access to the cluster. Without this, you would not be able to run kubectl commands.

  • Still in main.tf, add the managed node group block directly below the subnet_ids line (inside the same module block):
  # Managed node group for running pods
  eks_managed_node_groups = {
    rag_nodes = {
      ami_type       = "AL2023_x86_64_STANDARD"
      instance_types = [var.node_instance_type]

      min_size     = 2
      max_size     = 3
      desired_size = 2
    }
  }
}

What does this code do?

This creates a node group named rag_nodes with 2-3 t3.medium instances. AL2023 is the default AMI for EKS managed nodes. The closing } closes the entire module "eks" block.

  • Save main.tf.
Add the ECR repository and Pod Identity IAM role

Your container image needs a home in the cloud (ECR), and your pods need an IAM role to call Amazon Bedrock. You will create both resources now.

  • Below the EKS module in main.tf, add the ECR repository:
# ECR repository to store the chatbot Docker image
resource "aws_ecr_repository" "rag_chatbot" {
  name                 = "rag-chatbot"
  image_tag_mutability = "MUTABLE"
  force_delete         = true
}
  • Below the ECR resource, add the IAM role for EKS Pod Identity:
# IAM role that pods assume via EKS Pod Identity
data "aws_iam_policy_document" "pod_assume_role" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["pods.eks.amazonaws.com"]
    }

    actions = [
      "sts:AssumeRole",
      "sts:TagSession"
    ]
  }
}

resource "aws_iam_role" "pod_identity" {
  name               = "rag-chatbot-pod-identity-role"
  assume_role_policy = data.aws_iam_policy_document.pod_assume_role.json
}

What is this trust policy?

EKS Pod Identity lets pods securely assume an IAM role without hardcoded credentials. The trust policy grants the pods.eks.amazonaws.com service principal permission to assume this role on behalf of your pods.

sts:AssumeRole lets EKS assume the role. sts:TagSession lets EKS attach session tags (namespace, service account) so you can restrict which pods use which roles.

  • Add the policy attachment that gives this role Bedrock access:
# Attach Bedrock permissions to the pod identity role
resource "aws_iam_role_policy_attachment" "pod_bedrock" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonBedrockFullAccess"
  role       = aws_iam_role.pod_identity.name
}

Why AmazonBedrockFullAccess?

Your chatbot calls both Titan Embeddings and Claude via Bedrock. The managed policy grants access to all Bedrock models. In production, you would scope this down to specific model ARNs.

  • Save main.tf.
Write the outputs file
  • Create a file called outputs.tf and add the following:
# Values you will need for deployment
output "cluster_endpoint" {
  description = "EKS cluster API endpoint"
  value       = module.eks.cluster_endpoint
}

output "cluster_name" {
  description = "EKS cluster name"
  value       = module.eks.cluster_name
}

output "ecr_repository_url" {
  description = "ECR repository URL for pushing images"
  value       = aws_ecr_repository.rag_chatbot.repository_url
}

output "pod_identity_role_arn" {
  description = "IAM role ARN for EKS Pod Identity"
  value       = aws_iam_role.pod_identity.arn
}

What does this code do?

Outputs print key values after Terraform finishes. You will use ecr_repository_url to push your Docker image and pod_identity_role_arn to link your pods to the Bedrock IAM role.

  • Save outputs.tf.
  • Verify that all three files are present by running:
ls

You should see main.tf, variables.tf, and outputs.tf.

✔️ Awesome, I've got everything!

Great. Make sure all three files are saved before continuing.

ⓧ I'd like to double check the full code

Scroll up and confirm each chunk is present in the right file:

  • variables.tf has three variable blocks (region, cluster_name, node_instance_type).
  • main.tf has the terraform/provider blocks, the VPC module, the EKS module (with eks_managed_node_groups inside it), the ECR repository, the IAM policy document, the IAM role, and the policy attachment.
  • outputs.tf has four output blocks (cluster_endpoint, cluster_name, ecr_repository_url, pod_identity_role_arn).
Initialize and apply the Terraform configuration

Terraform needs to download the modules and provider plugins before it can create anything. Then you will run apply to provision the real AWS resources.

  • Initialize Terraform by running:
terraform init

You should see output ending with a success message along with lines about downloading the VPC and EKS modules.

Seeing a provider version error?

If you see an error about provider version constraints, make sure your Terraform version is 1.3.2 or higher. Run terraform --version to check.

help me fix a Terraform init error about provider version constraints

  • Apply the configuration to create all resources by running:
terraform apply

What happens during apply?

Terraform shows a plan listing every resource it will create. You will see approximately 50-60 resources including the VPC, subnets, route tables, NAT gateway, EKS cluster, node group, ECR repository, and IAM role.

When prompted, type yes and press Enter to confirm. The provisioning takes approximately 12-15 minutes. Most of that time is the EKS control plane starting up.

This will take 12-15 minutes

The EKS control plane takes about 10 minutes to fully provision. The node group takes another 2-3 minutes after that. This is normal for EKS. Use this time to review the plan output and understand what Terraform is creating.

When it completes, you will see a completion message followed by your four output values.

Apply failing with a permissions error?

Make sure your AWS CLI credentials have permissions to create VPCs, EKS clusters, ECR repositories, and IAM roles. If you see an AccessDenied error, check that your IAM user or role has the required policies attached.

help me fix an AWS permissions error when running terraform apply for EKS

Connect kubectl to the cluster and verify nodes

The cluster is running but your local kubectl does not know about it yet. You need to update your kubeconfig so kubectl can talk to the new EKS cluster.

  • Configure kubectl to connect to your cluster by running:
aws eks update-kubeconfig --name rag-chatbot-cluster --region us-east-1

You should see a message confirming the context was added to your kubeconfig file.

  • Before you check the nodes, predict what you expect to see. Do you think the nodes will be in Ready or NotReady state?
  • Verify your nodes are running by running:
kubectl get nodes

You should see 2 nodes listed with a Ready status. The nodes are t3.medium instances running in your private subnets.

Nodes showing NotReady?

If nodes show NotReady, wait 1-2 minutes and run kubectl get nodes again. Nodes can take a moment to fully register after the apply completes.

If you see no resources found, confirm your kubeconfig updated correctly by running kubectl config current-context. It should contain rag-chatbot-cluster.

help me troubleshoot my EKS nodes not reaching Ready state

Your EKS cluster is live with worker nodes ready to run pods. Next up, you will push your Docker image to ECR and deploy the chatbot to Kubernetes.

Deploy to EKS and Hit the Permissions Wall

Your EKS cluster is running and your Docker image is built. Pushing the container to ECR and deploying it to Kubernetes should be the final step to get a live chatbot.

But pods do not automatically inherit your AWS permissions. In this step, you will deploy the chatbot and discover firsthand why pod-level IAM matters.

In this step, get ready to:
  • Push the Docker image to your ECR repository.
  • Write and apply Kubernetes Deployment and Service manifests.
  • Check pod logs to observe a Bedrock access failure.
Push the Docker image to ECR

Your ECR repository is empty. You need to authenticate Docker to it, tag your local image with the ECR URL, and push it.

  • Retrieve your AWS account ID by running this command:
aws sts get-caller-identity --query Account --output text

You will see a 12-digit number printed. Copy this value. You will need it for the next few commands.

  • Authenticate Docker to your ECR registry by running this command:
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com

You should see Login Succeeded in your terminal output.

Seeing an authorization error?

Make sure your AWS CLI credentials are valid. Run aws sts get-caller-identity to confirm your identity. If it fails, reconfigure with aws configure.

Help me troubleshoot my ECR login error

  • Tag your local image with the ECR repository URL by running:
docker tag rag-chatbot:latest [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/rag-chatbot:latest
  • Push the tagged image to ECR by running:
docker push [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/rag-chatbot:latest

You will see layer upload progress in the terminal. When it finishes, the last line shows the image digest confirming the push succeeded.

What does this do?

The docker tag command gives your local image a second name that includes the full ECR URL. The docker push command uploads the image layers to that remote registry so Kubernetes can pull them.

Push denied or repository not found?

Double-check your account ID in the tag command matches your actual 12-digit AWS account ID. Also confirm the repository name is rag-chatbot (matching what Terraform created).

Help me debug my ECR push error

Write and apply Kubernetes manifests

Kubernetes needs manifest files that describe what to run and how to expose it. You will create a Deployment (tells Kubernetes which container to run) and a Service (exposes it to the internet via a load balancer).

  • Create a k8s directory in your project by running:
mkdir k8s
  • Create the Deployment manifest by running:
cat > k8s/deployment.yaml << 'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rag-chatbot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rag-chatbot
  template:
    metadata:
      labels:
        app: rag-chatbot
    spec:
      serviceAccountName: rag-chatbot-sa
      containers:
        - name: rag-chatbot
          image: [[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]].dkr.ecr.us-east-1.amazonaws.com/rag-chatbot:latest
          ports:
            - containerPort: 5000
          env:
            - name: AWS_REGION
              value: us-east-1
EOF

What does this manifest do?

  • replicas: 1 tells Kubernetes to run exactly one copy of your chatbot pod.
  • serviceAccountName: rag-chatbot-sa assigns a Kubernetes service account to the pod. This is the hook where Pod Identity attaches AWS permissions later.
  • The image field points to the ECR image you just pushed.
  • The AWS_REGION environment variable tells the Bedrock SDK which region to call.
  • Confirm the file was created by running:
ls k8s/

You should see deployment.yaml listed.

  • Create the Service manifest by running:
cat > k8s/service.yaml << 'EOF'
apiVersion: v1
kind: Service
metadata:
  name: rag-chatbot
spec:
  type: LoadBalancer
  selector:
    app: rag-chatbot
  ports:
    - port: 80
      targetPort: 5000
EOF

What does the Service do?

A LoadBalancer Service tells AWS to create an Elastic Load Balancer that routes internet traffic on port 80 to your pod's port 5000. This is how you will access the chatbot from outside the cluster.

  • Confirm both files exist by running:
ls k8s/

You should see both deployment.yaml and service.yaml listed.

  • Apply both manifests to your cluster by running:
kubectl apply -f k8s/

You should see output confirming both resources were created:

deployment.apps/rag-chatbot created
service/rag-chatbot created
  • Check that your pod is starting by running:
kubectl get pods

You should see a pod with a name starting with rag-chatbot- in the list. Its status might briefly show ContainerCreating or Running. Wait about 30 seconds, then run the command again until you see Running.

Pod stuck in ImagePullBackOff?

This means the cluster cannot pull the image from ECR. Verify that your image tag in deployment.yaml matches the exact ECR URL you pushed to (including account ID and region). Also confirm the node group IAM role has the AmazonEC2ContainerRegistryReadOnly policy attached (the EKS module does this by default).

Help me debug my ImagePullBackOff error

Check pod logs to see the Bedrock access failure

The pod shows Running status, which means Kubernetes successfully pulled your image and started the container. But is the chatbot actually working?

Before you check the logs, think about what the pod is doing. The Flask app tries to call Amazon Bedrock to build its vector store on startup. Does the pod have any AWS credentials?

  • Check the pod logs by running:
kubectl logs deployment/rag-chatbot

You will see an error in the output. The pod cannot call Bedrock because it has no AWS credentials. Look for a message containing NoCredentialProviders or AccessDeniedException.

Why does this happen?

When you ran the app locally, it used your machine's AWS credentials (from aws configure). When Docker ran it locally, you mounted credentials into the container.

Inside the Kubernetes cluster, the pod is a blank slate. It has no AWS credentials at all. The IAM role rag-chatbot-pod-identity-role exists, but nothing connects it to this pod yet. That connection is exactly what EKS Pod Identity provides.

This failure is expected and intentional. You have just hit the most common real-world EKS mistake: deploying a pod that needs AWS access without configuring pod-level permissions.

Your chatbot is deployed but broken. The infrastructure is ready and the container image is correct. The only missing piece is connecting that IAM role to your pod. Next up, you will fix this with EKS Pod Identity and get a working chatbot on a live URL.

Fix Pod Permissions with EKS Pod Identity

In the last step, you deployed your chatbot to EKS and discovered it crashes when trying to call Bedrock. The pod has no AWS credentials, so every request to the AI model fails.

EKS Pod Identity is the modern way to give pods scoped AWS permissions without hardcoding secrets. It links a Kubernetes ServiceAccount to an IAM role, and the EKS Pod Identity agent automatically injects temporary credentials into your containers. This is how production clusters handle AWS access.

In this step, get ready to:
  • Create a Kubernetes ServiceAccount and link it to your Bedrock IAM role via EKS Pod Identity.
  • Restart the deployment so pods pick up the new identity.
  • Test the live chatbot endpoint and get a RAG-generated answer.
Create the ServiceAccount and Pod Identity association

Your deployment manifest already references a ServiceAccount named rag-chatbot-sa. That ServiceAccount does not exist in the cluster yet. You need to create it and then link it to the IAM role that Terraform provisioned.

  • Create the Kubernetes ServiceAccount by running this command:
kubectl create serviceaccount rag-chatbot-sa

What does this command do?

This creates a ServiceAccount resource in the default namespace. Your deployment is already configured to use this ServiceAccount name, so once it exists, pods will be associated with it on their next restart.

  • Confirm the ServiceAccount was created by running:
kubectl get serviceaccount rag-chatbot-sa

You should see rag-chatbot-sa listed with an age of a few seconds.

Now you need to tell EKS to inject Bedrock credentials into any pod that uses this ServiceAccount. The aws eks create-pod-identity-association command links the ServiceAccount to your IAM role.

  • Get your IAM role ARN from Terraform outputs by running:
terraform -chdir=terraform output pod_identity_role_arn
  • Create the Pod Identity association by running this command (replace the role ARN with your output from above):
aws eks create-pod-identity-association \
  --cluster-name rag-chatbot-cluster \
  --namespace default \
  --service-account rag-chatbot-sa \
  --role-arn [[POD_IDENTITY_ROLE_ARN="arn:aws:iam::123456789012:role/rag-chatbot-pod-identity-role"]]

What does this command do?

This registers an association in the EKS control plane. When a pod starts with ServiceAccount rag-chatbot-sa, the Pod Identity agent injects environment variables that configure the AWS SDK to use temporary credentials scoped to your Bedrock role.

No annotations or Kubernetes-side config are needed. The association lives entirely in the EKS API, outside the cluster.

You should see JSON output containing your cluster name, namespace, service account, and role ARN confirming the association was created.

Seeing an AccessDeniedException?

Make sure your AWS CLI user has iam:PassRole permission on the rag-chatbot-pod-identity-role role. The create-pod-identity-association call requires this permission to link the role.

Also confirm you are in the correct region (us-east-1) where your cluster lives.

help me debug an AccessDeniedException when creating an EKS Pod Identity association

Restart the deployment to pick up the new identity

The existing pods were created before the Pod Identity association existed. They need to be recreated so the Pod Identity agent can inject the new credentials at startup.

  • Restart the deployment by running:
kubectl rollout restart deployment/rag-chatbot

You should see deployment.apps/rag-chatbot restarted in your terminal.

  • Wait for the new pods to become ready by running:
kubectl rollout status deployment/rag-chatbot

This command watches the rollout and exits when all new pods are running. You should see a message like deployment "rag-chatbot" successfully rolled out.

  • Confirm the pods are healthy by checking logs:
kubectl logs deployment/rag-chatbot --tail=20

This time, you should see your Flask app starting up without any credential errors. Look for a line indicating the server is running on port 5000.

Why do pods need a restart?

The Pod Identity agent injects credentials as environment variables when a pod starts. Existing pods that launched before the association was created never received these variables. The rolling restart creates fresh pods that pick up the new identity at boot time.

Pods still crashing?

Run kubectl describe pod -l app=rag-chatbot and check the Events section for errors. If you see ImagePullBackOff, the ECR image may not be accessible from the new pod. Verify the image exists in your ECR repository.

If you still see Bedrock access errors in the logs, verify the association was created correctly by running aws eks list-pod-identity-associations --cluster-name rag-chatbot-cluster.

help me debug pods that are still crashing after setting up EKS Pod Identity

Test the live chatbot endpoint

Your pods are running and have Bedrock access. The LoadBalancer service is routing external traffic to them on port 80. Time to get the URL and send a real question to your RAG chatbot.

  • Get the LoadBalancer URL by running:
kubectl get service rag-chatbot -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

Copy the hostname that appears. This is your chatbot's public endpoint.

What if the hostname is empty?

The LoadBalancer can take 1-2 minutes to provision. If the output is empty, wait a moment and run the command again. You can also run kubectl get service rag-chatbot to see the EXTERNAL-IP column update from <pending> to a hostname.

Before you run the test, what do you think will happen this time? Will the request succeed or fail?

  • Test the chatbot endpoint by running this curl command (replace the hostname with your LoadBalancer URL):
curl http://[[LB_HOSTNAME="your-loadbalancer-hostname"]]/ask \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"question": "What is S3?"}'

You should receive a JSON response containing an AI-generated answer about Amazon S3, grounded in the documents from your docs/ folder. This answer is being generated by Claude 3 Haiku via Amazon Bedrock, running inside a pod on your EKS cluster.

What just happened end-to-end?

Your curl request hit the AWS LoadBalancer, which routed it to the rag-chatbot pod. Inside the pod, the Flask app used FAISS to find relevant document chunks, then sent them along with your question to Bedrock's Claude model. The pod authenticated to Bedrock using temporary credentials injected by the Pod Identity agent.

No AWS access keys are stored anywhere in the container or the cluster. The credentials rotate automatically.

Getting a connection timeout?

Make sure you are using http:// (not https://) since the service is not configured with TLS. Also check that your EKS security groups allow inbound traffic on port 80 from your IP.

If you get a response but it is an error from the app, check kubectl logs deployment/rag-chatbot for details about what failed inside the container.

help me debug a connection timeout when curling my EKS LoadBalancer endpoint

Your RAG chatbot is live on Kubernetes, answering questions with AI-generated responses grounded in your documents. The entire stack is running on infrastructure you provisioned from scratch with Terraform. Next up, take on the Secret Mission and automate the whole deployment pipeline with GitHub Actions.

Secret mission

Add CI/CD with GitHub Actions

You deployed the chatbot manually. Every future code change means repeating the build-push-deploy dance by hand. In this secret mission, you set up a GitHub Actions pipeline that automatically builds your Docker image, pushes it to ECR, and updates the live deployment on EKS every time you push to main.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running or delete them entirely. This project provisions paid AWS infrastructure that costs approximately $0.25/hour while running.

Cost warning

Your EKS cluster charges $0.10/hour for the control plane alone. With EC2 nodes, a NAT Gateway, and a Load Balancer, the total cost is approximately $0.25/hour. If you leave resources running overnight (8 hours), that is roughly $2.

The recommended option is to delete everything by running terraform destroy after completing the project.

Resources you used:

  • AWS EKS cluster (rag-chatbot-cluster) with managed node group (t3.medium instances).
  • AWS ECR repository (rag-chatbot) with stored Docker images.
  • AWS VPC with public/private subnets and a NAT Gateway.
  • AWS IAM roles (rag-chatbot-pod-identity-role and github-actions-role).
  • AWS OIDC Identity Provider (token.actions.githubusercontent.com).
  • Elastic Load Balancer (created by the Kubernetes Service).
  • Local project files (rag-chatbot/, terraform/, k8s/, .github/).
  • GitHub repository.

Keep everything running

No action needed. Choose this if you are still actively building or testing your RAG chatbot on EKS.

  • Your EKS cluster and all AWS resources remain active.
  • Costs will continue to accrue at approximately $0.25/hour.
  • Your GitHub Actions pipeline will continue to deploy on every push to main.
  • Your local project files, Docker Desktop, Terraform, and kubectl remain installed on your machine.

Pause - I'll come back to this later

EKS does not support pausing. The control plane runs continuously and charges $0.10/hour regardless of workload. You cannot pause and resume an EKS cluster.

Your two options are to keep everything running (and accept the ongoing cost) or delete everything and re-create it later with terraform apply. Because your infrastructure is defined as code, re-provisioning takes a single command and about 15 minutes.

  • If you want to reduce costs while keeping the option to return, follow the Delete tab and keep your local terraform/ and rag-chatbot/ directories intact.
  • When you are ready to return, run terraform apply again to recreate all cloud resources.

Delete - I don't want to use this again

Remove all project resources and stop all charges. This is the recommended option after completing the project.

Follow these steps in order to avoid orphaned resources:

Delete the Kubernetes Load Balancer first

The Load Balancer was created by Kubernetes, not Terraform. Delete the Kubernetes Service so AWS removes the Load Balancer before Terraform destroys the underlying network.

  • Delete the Kubernetes Service by running:
kubectl delete service rag-chatbot
Destroy all Terraform-managed resources
  • Navigate to your terraform/ directory and run:
cd ~/Desktop/terraform
terraform destroy

Terraform will show you everything it plans to delete and ask for confirmation. Type yes and press Enter. This removes the EKS cluster, VPC, NAT Gateway, ECR repository, managed node group, and the Pod Identity IAM role.

Delete the GitHub Actions IAM role and OIDC provider

If you completed the Secret Mission, these resources were created manually and are not managed by Terraform. Delete them with the AWS CLI.

  • Detach the policies from the GitHub Actions role and delete it by running:
aws iam detach-role-policy --role-name github-actions-role --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser
aws iam delete-role-policy --role-name github-actions-role --policy-name eks-access
aws iam delete-role --role-name github-actions-role
  • Find and delete the OIDC Identity Provider by running:
aws iam list-open-id-connect-providers

Look for the entry containing token.actions.githubusercontent.com in the output. Copy its ARN, then run:

aws iam delete-open-id-connect-provider --open-id-connect-provider-arn [[OIDC_ARN="arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"]]
Remove local project files
  • Delete the local project directory by running:

macOS

cd ~/Desktop
rm -rf rag-chatbot terraform k8s .github

Windows

cd ~/Desktop
Remove-Item -Recurse -Force rag-chatbot, terraform, k8s, .github
Delete the GitHub repository (optional)
  • If you no longer need the GitHub repository, go to your repository page on GitHub.
  • Click Settings at the top of the repository.
  • Scroll to the bottom and click Delete this repository in the Danger Zone section.
  • Follow the confirmation prompts to permanently delete it.

Nice Work!

Nice Work!

You just deployed a production-style RAG chatbot to Amazon EKS, with its entire infrastructure defined as code in Terraform.

You've learned how to:

  • Build a RAG chatbot using LangChain, FAISS, and Amazon Bedrock, then containerize it with Docker for portable deployment.
  • Write a complete Terraform configuration from scratch to provision a VPC, EKS cluster, and ECR repository with a single command.
  • Diagnose a real-world pod permissions failure and fix it with EKS Pod Identity, the modern successor to IRSA for granting pods scoped AWS access.
  • Secret Mission: Automate the full build-push-deploy cycle with GitHub Actions and OIDC authentication, so every code push triggers a hands-free deployment to your cluster.

Ready to quiz yourself?