Automate CI/CD for EKS with GitHub Actions
Build a secure CI/CD pipeline deploying to EKS using OIDC—no stored AWS keys.
Introduction
30 Second Summary
Every time you manually run docker build, push an image, and update your cluster by hand, you introduce risk. A typo, a forgotten step, or a stale credential can take your app offline.
In this project, you will build a GitHub Actions CI/CD pipeline that automatically deploys your containerized app to Amazon EKS every time you push code to main. Instead of storing long-lived AWS credentials, your pipeline authenticates using OIDC federation for short-lived, automatically expiring credentials.
What You'll Build
A GitHub Actions workflow that authenticates to AWS using OIDC, builds and pushes Docker images to Amazon ECR tagged with the commit SHA, deploys to your EKS cluster, verifies the rollout, and automatically rolls back on failure.
By the end of this project, you'll have:
- An AWS IAM OIDC identity provider that lets GitHub Actions authenticate to your AWS account without any stored secrets.
- A complete CI/CD workflow that builds, pushes, and deploys your app on every push to main with rollout verification.
- A tested rollback mechanism that automatically reverts failed deployments without manual intervention.
- Secret Mission: Add a manual approval gate using GitHub Environments so production deployments require a reviewer.
Are there any prerequisites?
Yes. This project assumes you have completed Project 1 (containerizing a Next.js app and pushing it to ECR) and Project 2 (deploying to EKS with kubectl). You need an existing EKS cluster, an ECR repository, and a GitHub account.
Configure AWS OIDC Trust for GitHub Actions
Your EKS cluster and ECR repository are ready from previous projects. Now you need a way for GitHub Actions to authenticate with AWS without storing long-lived access keys as secrets.
OpenID Connect (OIDC) federation solves this by letting GitHub issue a short-lived token that AWS trusts directly. In this step, you will register GitHub as a trusted identity provider in your AWS account, then create an IAM role scoped to your specific repository.
In this step, get ready to:
- Register the GitHub OIDC identity provider in AWS IAM.
- Create an IAM role with a trust policy locked to your repository and branch.
- Map the IAM role to your EKS cluster so kubectl commands work from the workflow.
Create the GitHub OIDC identity provider in AWS IAM
Before GitHub Actions can request temporary credentials, AWS needs to know that GitHub's token service is trustworthy. You do this by registering GitHub's OIDC endpoint as an identity provider in IAM.
- Create the OIDC identity provider by running this command:
aws iam create-open-id-connect-provider \
--url https://token.actions.githubusercontent.com \
--client-id-list sts.amazonaws.com \
--thumbprint-list 6938fd4d98bab03faadb97b34396831e3780aea1
What does this command do?
This tells AWS to trust tokens issued by GitHub's OIDC service at https://token.actions.githubusercontent.com.
- --client-id-list sts.amazonaws.com sets the expected audience claim. AWS checks this to confirm the token was intended for STS.
- --thumbprint-list is a legacy requirement. AWS now validates GitHub's certificates directly, but the parameter is still required by the API.
- Confirm the provider was created by opening the AWS IAM Console and navigating to **Identity Providers** under **Access management**.
- You should see token.actions.githubusercontent.com listed as a provider.
Provider already exists?
If you see an error saying the provider already exists, that is fine. It means a previous project or admin already registered GitHub's OIDC endpoint. You can proceed to the next substep.
Help me troubleshoot the OIDC provider creation error.
Create the IAM role with a trust policy scoped to your repository
The IAM role defines who can assume it (trust policy) and what they can do once assumed (permissions policy). You will lock the trust policy to your specific GitHub repository and the main branch.
- Create a file called trust-policy.json in your project directory with the following content:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::[[AWS_ACCOUNT_ID=\"your 12-digit AWS account ID\"]]:oidc-provider/token.actions.githubusercontent.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": "repo:[[GITHUB_USERNAME=\"your GitHub username\"]]/[[REPO_NAME=\"your repository name\"]]:ref:refs/heads/main"
}
}
}
]
}
What does this trust policy do?
The Condition block is the critical security boundary. It uses two claims from the GitHub OIDC token:
- aud (audience) must equal sts.amazonaws.com to confirm the token is intended for AWS.
- sub (subject) must match your exact repository and branch. This means only workflows running on main in your specific repo can assume this role.
- Save trust-policy.json.
- Create the IAM role using the trust policy by running this command:
aws iam create-role --role-name GitHubActionsEKSDeployRole --assume-role-policy-document file://trust-policy.json
What does this command do?
This creates an IAM role named GitHubActionsEKSDeployRole with your trust policy attached. The --assume-role-policy-document flag specifies who is allowed to assume the role.
Next, you need to define what this role is allowed to do. The permissions policy grants access to push images to ECR and describe your EKS cluster.
- Create a file called permissions-policy.json with the following content:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"ecr:PutImage",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload"
],
"Resource": "arn:aws:ecr:[[AWS_REGION=\"your AWS region e.g. us-east-1\"]]:[[AWS_ACCOUNT_ID=\"your 12-digit AWS account ID\"]]:repository/[[ECR_REPO=\"your ECR repository name\"]]"
},
{
"Effect": "Allow",
"Action": [
"eks:DescribeCluster"
],
"Resource": "arn:aws:eks:[[AWS_REGION=\"your AWS region e.g. us-east-1\"]]:[[AWS_ACCOUNT_ID=\"your 12-digit AWS account ID\"]]:cluster/[[EKS_CLUSTER_NAME=\"your EKS cluster name\"]]"
}
]
}
What does this permissions policy grant?
- ecr:GetAuthorizationToken on * allows the role to get a login token for any ECR registry in the account. This action cannot be scoped to a single repository.
- The second statement scopes push-related actions (PutImage, InitiateLayerUpload, etc.) to your specific ECR repository.
- eks:DescribeCluster allows the workflow to call aws eks update-kubeconfig, which is needed to authenticate kubectl against your cluster.
- Save permissions-policy.json.
- Attach the permissions policy to the role by running:
aws iam put-role-policy \
--role-name GitHubActionsEKSDeployRole \
--policy-name GitHubActionsEKSPermissions \
--policy-document file://permissions-policy.json
- Verify the role was created correctly by running:
aws iam get-role --role-name GitHubActionsEKSDeployRole
- You should see the role details in the output, including the trust policy showing your repository in the Condition block.
Seeing "MalformedPolicyDocument" error?
- Check that your JSON files have no trailing commas or syntax errors. Run cat trust-policy.json | python3 -m json.tool to validate the JSON.
- Make sure the Federated principal ARN uses your actual 12-digit AWS account ID, not the placeholder text.
- Confirm the OIDC provider URL in the principal matches exactly: token.actions.githubusercontent.com (no https:// prefix in the ARN).
Help me fix my trust policy or permissions policy JSON.
Grant the role access to your EKS cluster
IAM authentication gets the workflow into AWS, but Kubernetes has its own authorization layer. You need to map the IAM role to a Kubernetes RBAC group so that kubectl commands work when the workflow assumes this role.
- Map the IAM role to the system:masters Kubernetes group by running:
eksctl create iamidentitymapping \
--cluster [[EKS_CLUSTER_NAME="your EKS cluster name"]] \
--region [[AWS_REGION="your AWS region e.g. us-east-1"]] \
--arn arn:aws:iam::[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]:role/GitHubActionsEKSDeployRole \
--group system:masters \
--username github-actions
What does this command do?
This adds an entry to your cluster's aws-auth ConfigMap. When the GitHub Actions workflow assumes GitHubActionsEKSDeployRole and connects to the cluster, Kubernetes recognizes it as part of system:masters and grants full cluster admin permissions.
The --username github-actions flag sets the username that appears in Kubernetes audit logs, making it easy to trace workflow actions.
- Confirm the mapping exists by running:
eksctl get iamidentitymapping --cluster [[EKS_CLUSTER_NAME="your EKS cluster name"]] --region=[[AWS_REGION="your AWS region e.g. us-east-1"]]
- You should see GitHubActionsEKSDeployRole listed in the output with group system:masters.
Command not found for eksctl?
If eksctl is not installed, you can edit the aws-auth ConfigMap directly using kubectl edit configmap aws-auth -n kube-system. Add your role ARN under the mapRoles section with the groups and username fields.
Help me map my IAM role to my EKS cluster without eksctl.
✔️ Awesome, I've got everything!
Great. Double check you have saved both trust-policy.json and permissions-policy.json.
ⓧ I'd like to double check the full code
Here is the complete trust-policy.json:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::YOUR_ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": "repo:YOUR_GITHUB_USERNAME/YOUR_REPO_NAME:ref:refs/heads/main"
}
}
}
]
}
Here is the complete permissions-policy.json:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"ecr:PutImage",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload"
],
"Resource": "arn:aws:ecr:YOUR_REGION:YOUR_ACCOUNT_ID:repository/YOUR_ECR_REPO"
},
{
"Effect": "Allow",
"Action": [
"eks:DescribeCluster"
],
"Resource": "arn:aws:eks:YOUR_REGION:YOUR_ACCOUNT_ID:cluster/YOUR_CLUSTER_NAME"
}
]
}
Your AWS account now trusts GitHub Actions, and your EKS cluster recognizes the deployment role. Next up, you will write the CI/CD workflow that uses this trust to build, push, and deploy automatically.
Write the GitHub Actions CI/CD Workflow
Your AWS account now trusts GitHub Actions through OIDC, and the IAM role is ready to be assumed. The next step is to write the workflow that actually uses that trust.
A GitHub Actions workflow is a YAML file that lives inside your repository. Every time you push code to main, GitHub reads this file and executes the steps you define: authenticate, build, push, and deploy.
In this step, get ready to:
- Create the workflow file structure with triggers, permissions, and environment variables.
- Add steps for OIDC auth, Docker build/push, and Kubernetes deployment with rollback.
- Configure repository variables and secrets in GitHub so the workflow can reference them.
Create the workflow file structure
GitHub Actions looks for workflow files in a specific directory inside your repository. You need to create .github/workflows/ and place your YAML file inside it.
- In your terminal, make sure you are inside your repository's root folder (where your Dockerfile lives).
- Create the workflows directory by running this command:
mkdir -p .github/workflows
- Confirm the directory was created by running:
ls .github/workflows
You should see an empty directory with no output errors.
- Create and open the workflow file .github/workflows/deploy.yml in your code editor.
- Paste in the following workflow header that defines when the workflow runs, what permissions it needs, and which environment variables it uses:
name: Deploy to Amazon EKS
on:
push:
branches:
- main
permissions:
id-token: write
contents: read
env:
AWS_REGION: ${{ vars.AWS_REGION }}
ECR_REPOSITORY: ${{ vars.ECR_REPOSITORY }}
EKS_CLUSTER_NAME: ${{ vars.EKS_CLUSTER_NAME }}
DEPLOYMENT_NAME: ${{ vars.DEPLOYMENT_NAME }}
jobs:
deploy:
name: Build, Push, and Deploy
runs-on: ubuntu-latest
steps:
What does this code do?
This sets up the skeleton of your CI/CD pipeline.
- on: push: branches: - main triggers the workflow every time you push to the main branch.
- permissions: id-token: write allows the workflow to request an OIDC token from GitHub. contents: read lets it check out your code.
- The env block pulls values from your repository's Actions variables (which you will configure in substep 3).
- runs-on: ubuntu-latest tells GitHub to run this job on a Linux virtual machine.
- Save .github/workflows/deploy.yml.
File not saving in the right place?
Make sure the file is at the path .github/workflows/deploy.yml relative to your repository root. If you see .github next to your Dockerfile and trust-policy.json, you are in the right spot.
Help me verify my deploy.yml file path
Add the build, push, and deploy steps
The steps: key is where the actual work happens. You will add steps that authenticate to AWS, build your Docker image, push it to ECR, and deploy it to your EKS cluster.
- In .github/workflows/deploy.yml, place your cursor at the end of the steps: line.
- Add the following authentication steps below steps: (indented with 6 spaces):
- name: Checkout code
uses: actions/checkout@v6
- name: Configure AWS credentials via OIDC
uses: aws-actions/configure-aws-credentials@v6
with:
role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
role-session-name: github-actions-deploy
aws-region: ${{ env.AWS_REGION }}
- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v2
What does this code do?
These three steps handle all authentication before any real work begins.
- actions/checkout@v6 clones your repository code onto the runner so Docker can build from your Dockerfile.
- aws-actions/configure-aws-credentials@v6 uses OIDC to assume the GitHubActionsEKSDeployRole you created in Step 1. No long-lived keys needed.
- aws-actions/amazon-ecr-login@v2 authenticates Docker to your Amazon ECR registry so it can push images. The id: login-ecr lets later steps reference the registry URL from its output.
- Below the ECR login step, add the Docker build and push step:
- name: Build and push Docker image
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
IMAGE_TAG: ${{ github.sha }}
run: |
docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
echo "image=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG" >> $GITHUB_OUTPUT
What does this code do?
This step builds your Docker image and pushes it to ECR, tagged with the unique commit SHA.
- ECR_REGISTRY is dynamically set from the ECR login step's output. No hardcoded registry URLs.
- IMAGE_TAG: ${{ github.sha }} tags each image with the exact commit that produced it. This makes every deployment traceable back to source code.
- The echo line writes the full image URI to the step output for reference.
- Below the build step, add the deployment and verification steps:
- name: Update kubeconfig
run: |
aws eks update-kubeconfig --region $AWS_REGION --name $EKS_CLUSTER_NAME
- name: Deploy to EKS
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
IMAGE_TAG: ${{ github.sha }}
run: |
kubectl set image deployment/$DEPLOYMENT_NAME \
$DEPLOYMENT_NAME=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
- name: Verify rollout
run: |
kubectl rollout status deployment/$DEPLOYMENT_NAME --timeout=120s
- name: Rollback on failure
if: failure()
run: |
echo "Deployment failed. Rolling back..."
kubectl rollout undo deployment/$DEPLOYMENT_NAME
kubectl rollout status deployment/$DEPLOYMENT_NAME --timeout=120s
What does this code do?
These four steps handle the actual Kubernetes deployment and its safety net.
- aws eks update-kubeconfig configures kubectl on the runner to talk to your Amazon EKS cluster.
- kubectl set image updates the Deployment's container image to the freshly pushed ECR image.
- kubectl rollout status watches the rollout for up to 120 seconds. If new pods become healthy, it exits successfully.
- if: failure() triggers the rollback step only when a previous step fails. It runs kubectl rollout undo to revert to the last working version automatically.
- Save .github/workflows/deploy.yml.
YAML indentation errors?
YAML is sensitive to spaces. Each step must be indented with exactly 6 spaces (under steps:). The run: | block content must be indented 10 spaces from the left. If your editor uses tabs, switch to spaces.
Help me fix YAML indentation in my workflow
✔️ Awesome, I've got everything!
Great. Make sure you have saved .github/workflows/deploy.yml with all the steps in place.
ⓧ I'd like to double check the full code
Here is the complete deploy.yml file. Compare your version against this:
name: Deploy to Amazon EKS
on:
push:
branches:
- main
permissions:
id-token: write
contents: read
env:
AWS_REGION: ${{ vars.AWS_REGION }}
ECR_REPOSITORY: ${{ vars.ECR_REPOSITORY }}
EKS_CLUSTER_NAME: ${{ vars.EKS_CLUSTER_NAME }}
DEPLOYMENT_NAME: ${{ vars.DEPLOYMENT_NAME }}
jobs:
deploy:
name: Build, Push, and Deploy
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Configure AWS credentials via OIDC
uses: aws-actions/configure-aws-credentials@v6
with:
role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
role-session-name: github-actions-deploy
aws-region: ${{ env.AWS_REGION }}
- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v2
- name: Build and push Docker image
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
IMAGE_TAG: ${{ github.sha }}
run: |
docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
echo "image=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG" >> $GITHUB_OUTPUT
- name: Update kubeconfig
run: |
aws eks update-kubeconfig --region $AWS_REGION --name $EKS_CLUSTER_NAME
- name: Deploy to EKS
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
IMAGE_TAG: ${{ github.sha }}
run: |
kubectl set image deployment/$DEPLOYMENT_NAME \
$DEPLOYMENT_NAME=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
- name: Verify rollout
run: |
kubectl rollout status deployment/$DEPLOYMENT_NAME --timeout=120s
- name: Rollback on failure
if: failure()
run: |
echo "Deployment failed. Rolling back..."
kubectl rollout undo deployment/$DEPLOYMENT_NAME
kubectl rollout status deployment/$DEPLOYMENT_NAME --timeout=120s
Configure GitHub repository secrets and variables
Your workflow references vars.AWS_REGION, vars.ECR_REPOSITORY, and other values that do not exist yet. You need to add them in your GitHub repository settings so the workflow can resolve them at runtime.
Why variables instead of hardcoding?
Storing configuration as repository variables keeps your workflow file generic. You can reuse the same YAML in another project by changing the variable values, without editing the workflow itself.
The role ARN goes in Secrets (not Variables) because it contains your AWS account ID, which is mildly sensitive.
- Open your GitHub repository in your browser.
- Click the Settings tab at the top of the repository page.
- In the left sidebar under Security, click Secrets and variables.
- Click Actions.
You are now on the Actions secrets and variables page. You will add four variables first, then one secret.
- Click the Variables tab.
- Click New repository variable.
- Enter AWS_REGION as the name, and your AWS region (for example us-east-1) as the value.
- Click Add variable.
- Repeat the process to add three more variables:
- ECR_REPOSITORY set to your ECR repository name (the same one you push images to from previous projects).
- EKS_CLUSTER_NAME set to your EKS cluster name.
- DEPLOYMENT_NAME set to the name of your Kubernetes Deployment resource.
Now add the secret for your IAM role ARN.
- Switch to the Secrets tab.
- Click New repository secret.
- Enter AWS_ROLE_ARN as the name.
- Paste the full ARN of your GitHubActionsEKSDeployRole as the value (it looks like arn:aws:iam::123456789012:role/GitHubActionsEKSDeployRole).
- Click Add secret.
You should now see four variables on the Variables tab and one secret on the Secrets tab.
Not sure what values to use?
Run aws eks list-clusters to find your cluster name. Run aws ecr describe-repositories to find your ECR repository name. The deployment name is whatever you used in your Kubernetes Deployment manifest from the previous project.
Help me find my variable values
Your workflow file is complete and your repository is configured with the variables it needs. Next up, you will push this code to main and watch the entire pipeline run end-to-end.
Deploy and Verify the Pipeline
Your OIDC trust is configured, the IAM role is scoped to your repository, and the workflow file is committed. Everything is wired up. Now it's time to trigger the pipeline and see all the pieces work together.
In this step, you'll push a code change to main and watch GitHub Actions authenticate to AWS, build your Docker image, push it to ECR, and deploy it to your EKS cluster. You'll then verify the deployment end-to-end using kubectl and your application's LoadBalancer URL.
In this step, get ready to:
- Trigger the CI/CD pipeline by pushing a code change to main.
- Verify the full deployment end-to-end from OIDC auth through to live application access.
Trigger the pipeline with a code change
The workflow triggers on any push to main. To see the full pipeline in action, you need to make a visible change to your application so you can confirm the deployment actually updated what's running on the cluster.
- Open your application code in your editor.
- Make a small visible change to your app. For example, update a health check response message or change a heading in your UI.
- Save the file.
Why a visible change?
A visible change gives you a concrete signal that the new image is actually running on the cluster. If the text changes when you load the app, the full pipeline worked. If it doesn't, something failed along the way.
- Stage, commit, and push your change to main by running these commands:
git add .
git commit -m "Update health check message"
git push origin main
- Open your GitHub repository in your browser.
- Click the Actions tab at the top of your repository.
You should see a workflow run in progress titled with your commit message. Click on it to watch each step execute in real time.
What should I see?
The workflow run page shows each step as a collapsible section. Steps execute sequentially. You'll see green checkmarks appear as each step completes successfully.
The full run typically takes 2-4 minutes. Most of that time is the Docker build and push step.
Don't see a workflow run?
Make sure you pushed to the main branch specifically. The workflow only triggers on pushes to main.
Confirm your workflow file is at exactly .github/workflows/deploy.yml in your repository. The .github folder must be at the root of the repo.
help me figure out why my GitHub Actions workflow isn't triggering on push to main
Verify the deployment end-to-end
Once the workflow run shows all green checkmarks, it's time to verify the deployment from multiple angles. You want to confirm that the OIDC auth worked, the image landed in ECR, and your cluster is running the new code.
- In the workflow run page, expand each step and confirm it passed:
- Confirm new pods are running with the updated image by running this in your terminal:
kubectl get pods
You should see your pods with a recent start time (the AGE column will show seconds or minutes).
- Verify the image tag matches the commit SHA from your workflow run:
kubectl describe deployment
What am I looking for?
In the output, look for the Image field under Containers. The tag after the colon should be a long alphanumeric string. That's the commit SHA.
Compare it to the commit SHA shown in your GitHub workflow run page. They should match exactly. This proves the pipeline deployed the image it built from your specific commit.
- Access your application via the LoadBalancer URL to confirm the change is live.
- Find your LoadBalancer URL by running:
kubectl get svc
Copy the EXTERNAL-IP value and open it in your browser. You should see your updated change (the new health check message or heading) reflected in the live application.
Workflow failed at one of the steps?
If the OIDC step failed with a message about sts:AssumeRoleWithWebIdentity, your trust policy's sub claim does not match. Verify the format is exactly repo:OWNER/REPO:ref:refs/heads/main with your actual GitHub username and repository name.
If the ECR login or push step failed, confirm that your permissions policy grants the ECR actions to the correct repository ARN in the correct region.
If the deploy or rollout step failed, confirm the IAM identity mapping is in place. The role needs to be mapped to a Kubernetes RBAC group for kubectl commands to work.
help me debug my GitHub Actions EKS deployment workflow that failed
Your CI/CD pipeline is working end-to-end. Every push to main now automatically builds, pushes, and deploys without any manual intervention or stored credentials. Next up, you'll test what happens when a deployment goes wrong and confirm the automatic rollback works as expected.
Test Rollback and Troubleshoot Failures
Your CI/CD pipeline is live. Every push to main triggers an automated build, push, and deploy to your EKS cluster.
But how do you know the safety net actually works? In production, a failed deployment that cannot roll back means downtime for your users. In this step, you will deliberately break your deployment to prove the rollback mechanism catches it, then learn to diagnose the most common failure modes.
In this step, get ready to:
- Trigger a deliberate deployment failure and verify rollback runs automatically.
- Diagnose the most common CI/CD failure modes (OIDC, ECR, rollout, permissions).
- Fix the break and confirm a clean end-to-end deployment.
Trigger a deliberate deployment failure
The Rollback on failure step in your workflow uses if: failure() to detect when any previous step fails. To prove this works, you need a change that builds and pushes successfully but crashes at runtime inside the cluster.
- Open your Dockerfile in your code editor.
- Find the final CMD instruction (the last line of your Dockerfile).
- Change it to the following invalid command that will crash immediately on startup:
CMD ["node", "-e", "process.exit(1)"]
Why this crashes the deployment
This CMD tells the container to run a Node.js one-liner that exits immediately with code 1. The Docker image builds fine because the syntax is valid. But when Kubernetes tries to start the container, it crashes instantly.
Kubernetes will keep restarting the container, putting pods into CrashLoopBackOff. The kubectl rollout status step watches for 120 seconds and exits with a non-zero code when pods never become ready.
- Save your Dockerfile.
- Commit and push the change to main by running these commands:
git add Dockerfile
git commit -m "test: break deployment to verify rollback"
git push origin main
- In your GitHub repository, click the Actions tab.
- Click on the running workflow to watch it execute.
The workflow will proceed through checkout, OIDC auth, ECR login, and Docker build/push. All of these succeed because the image is technically valid. Once it reaches the Verify rollout step, it will wait up to 120 seconds for pods to become ready.
- Wait for the Verify rollout step to timeout and fail (this takes up to 2 minutes).
- Confirm that the Rollback on failure step runs immediately after the timeout.
What should I see?
In the workflow run, the Verify rollout step shows a red X. The Rollback on failure step runs and shows the message "Deployment failed. Rolling back..." followed by a successful rollout status for the reverted version.
- Verify the previous working version is restored by running:
kubectl get pods
- Confirm your pods show Running status (not CrashLoopBackOff).
- Access your application via the LoadBalancer URL to confirm it still serves the previous working version.
Rollback step didn't run?
If you do not see the Rollback on failure step execute, check that your deploy.yml has if: failure() on the rollback step. Without this condition, GitHub Actions skips the step when a previous step fails.
Also verify the Verify rollout step does not have continue-on-error: true set. That would mask the failure and prevent if: failure() from triggering.
Help me debug why my rollback step isn't executing in GitHub Actions
Diagnose common CI/CD failures
You just tested one failure mode (crashing container). In real projects, you will encounter other categories of failure. Here are the four most common issues and how to resolve each.
OIDC trust errors
If the Configure AWS credentials via OIDC step fails with a message about not being authorized to perform sts:AssumeRoleWithWebIdentity, the trust policy sub claim does not match the workflow's token.
- Check that your trust-policy.json uses the exact format repo:OWNER/REPO:ref:refs/heads/main.
- Verify the GitHub username and repository name are spelled correctly (case-sensitive).
- Confirm you pushed to the main branch, not a feature branch.
Image pull errors
If pods show ImagePullBackOff status, the EKS nodes cannot pull the image from ECR.
- Verify the EKS node IAM role has the AmazonEC2ContainerRegistryReadOnly policy attached.
- Confirm the ECR repository is in the same region as your EKS cluster.
- Check that the image tag in the deployment matches an actual tag in your ECR repository.
Rollout timeout with CrashLoopBackOff
If kubectl rollout status times out and pods show CrashLoopBackOff, the container is starting but crashing repeatedly.
- Check the container logs by running:
kubectl logs deployment/$DEPLOYMENT_NAME
- Inspect the pod events for more detail by running:
kubectl describe pod -l app=$DEPLOYMENT_NAME
- Look for missing environment variables, failed health checks, or application errors in the output.
Permission denied on kubectl
If the Deploy to EKS or Verify rollout steps fail with an unauthorized error, the IAM role is not mapped to a Kubernetes RBAC group.
- Verify the identity mapping exists by running:
eksctl get iamidentitymapping --cluster [[EKS_CLUSTER="your-cluster-name"]] --region [[AWS_REGION="your-region"]]
- Look for your GitHubActionsEKSDeployRole in the output. If it is missing, re-run the eksctl create iamidentitymapping command from Step 1.
Bookmark this reference
These four failure modes cover the vast majority of CI/CD pipeline issues with EKS deployments. When your RunLayer app or any future project hits a red workflow run, start by checking which step failed and matching it to one of these categories.
Fix and redeploy
Now that you have confirmed the rollback mechanism works, restore your application to a healthy state and prove the pipeline works end-to-end one more time.
- Open your Dockerfile and revert the CMD line back to its original value (the command your app uses to start, such as CMD ["npm", "start"] or your project's actual start command).
- Save the file.
- Commit and push the fix to main by running:
git add Dockerfile
git commit -m "fix: restore working CMD after rollback test"
git push origin main
- Return to the Actions tab in your GitHub repository.
- Watch the new workflow run. Every step should pass with a green checkmark.
What should I see?
All steps pass: checkout, OIDC auth, ECR login, Docker build/push, kubeconfig update, deploy, and verify rollout. The Rollback on failure step is skipped (shown as a grey circle) because no failure occurred.
- Confirm healthy pods are running by executing:
kubectl get pods
- Verify the image tag matches the latest commit SHA by running:
kubectl describe deployment $DEPLOYMENT_NAME
- Look for the Image field in the output. It should end with the commit SHA from your latest push.
- Access your application via the LoadBalancer URL to confirm the fix is live.
Workflow still failing after your fix?
Double-check that you reverted the CMD line to exactly what it was before your test. A typo in the start command will cause the same CrashLoopBackOff behaviour.
If the build step fails, run docker build . locally to check for syntax errors in your Dockerfile before pushing again.
My pipeline is still failing after I reverted my Dockerfile change. Help me troubleshoot.
Secret mission
Add a Manual Approval Gate with GitHub Environments
Your pipeline deploys on every push to main with no human check. In production, teams add approval gates so a reviewer must sign off before code reaches live infrastructure. Can you add a manual approval requirement using GitHub Environments?
Clean Up Your Resources
Clean Up Your Resources
Decide whether to keep your resources running, pause them to come back later, or delete them entirely. The EKS cluster control plane costs approximately $0.10/hr even when no workloads are running, and EC2 nodes in your node group incur additional charges based on instance type.
Cost warning
Your EKS cluster control plane costs approximately $0.10/hr ($2.40/day) whether or not you have workloads running. EC2 node instances add further charges based on their instance type. If you are not continuing to the next project immediately, choose Pause or Delete to stop billing.
Resources you used:
- EKS cluster (control plane and node group).
- ECR repository images (tagged with commit SHAs).
- IAM OIDC identity provider for GitHub Actions.
- IAM Role: GitHubActionsEKSDeployRole (with inline permissions policy and trust policy).
- GitHub Environment: production (with required reviewer protection rule).
- GitHub Actions variables and secrets (AWS_REGION, ECR_REPOSITORY, EKS_CLUSTER_NAME, DEPLOYMENT_NAME, AWS_ROLE_ARN).
- .github/workflows/deploy.yml, trust-policy.json, and permissions-policy.json files in your repository.
Keep everything running
No action needed. Choose this if you are actively building on this pipeline or continuing to the next project in the series.
- Your CI/CD pipeline will continue deploying on every push to main (with the approval gate from the Secret Mission if you completed it).
- EKS control plane costs continue at approximately $0.10/hr.
- EC2 node instances continue billing based on instance type.
- ECR storage costs are minimal (pennies for a few images).
Pause - I'll come back to this later
Scale your node group to zero nodes to stop EC2 instance billing. The EKS control plane still costs $0.10/hr, but your nodes stop incurring charges.
- Scale your node group to zero by running:
eksctl scale nodegroup --cluster [[EKS_CLUSTER="your-cluster-name"]] --name [[NODEGROUP_NAME="your-nodegroup-name"]] --nodes 0
- Your deployments will have no running pods, but the cluster, IAM role, OIDC provider, and GitHub workflow all remain intact.
- When you are ready to resume, scale the node group back up (e.g. --nodes 2) and push a new commit to trigger a fresh deploy.
Delete - I don't want to use this again
Remove all project resources and start fresh. Follow these steps in order to avoid dependency errors.
- Delete the EKS cluster (this also removes the node group) by running:
eksctl delete cluster --name [[EKS_CLUSTER="your-cluster-name"]] --region [[AWS_REGION="your-region"]]
- List the inline policies on the IAM role by running:
aws iam list-role-policies --role-name GitHubActionsEKSDeployRole
- Delete each inline policy returned by running (replace the policy name with what the previous command returned):
aws iam delete-role-policy --role-name GitHubActionsEKSDeployRole --policy-name [[POLICY_NAME="your-inline-policy-name"]]
- Delete the IAM role by running:
aws iam delete-role --role-name GitHubActionsEKSDeployRole
- Delete the OIDC identity provider by running:
aws iam delete-open-id-connect-provider --open-id-connect-provider-arn [[OIDC_PROVIDER_ARN="arn:aws:iam::YOUR_ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com"]]
- Delete ECR images by running:
aws ecr batch-delete-image --repository-name [[ECR_REPO="your-ecr-repo-name"]] --image-ids imageTag=latest
- Delete the GitHub Environment: In your repository, go to Settings > Environments > click production > click Delete environment.
- Delete repository-level variables: In Settings > Secrets and variables > Actions > Variables tab, delete AWS_REGION, ECR_REPOSITORY, EKS_CLUSTER_NAME, and DEPLOYMENT_NAME.
- Delete the workflow and policy files from your repository by running:
rm -rf .github/workflows/deploy.yml trust-policy.json permissions-policy.json
- Commit and push the deletion to update your remote repository.
Nice Work!
Nice Work!
Nice work! You've built a production-grade CI/CD pipeline that securely deploys your containerized application to Amazon EKS on every push to main, with zero long-lived AWS credentials stored in GitHub.
You've learned how to:
- Set up OIDC federation between GitHub Actions and AWS, replacing long-lived access keys with short-lived credentials that expire after each workflow run.
- Written a complete GitHub Actions workflow that builds a Docker image, pushes it to Amazon ECR tagged with the commit SHA, and deploys it to your EKS cluster automatically on every push to main.
- Verified automated rollout and tested automatic rollback by triggering a deliberate deployment failure and confirming the previous working version was restored.
- Secret Mission: Added a manual approval gate using GitHub Environments so production deployments require a reviewer before proceeding.
Ready to quiz yourself?