Deploy a Python Scraper to AWS EKS

Provision EKS with Terraform, containerize a scraper, and monitor with Grafana.

Introduction

30 Second Summary

Think about how many apps and websites you use every day. Behind every one of them is a team managing servers, pushing updates, and monitoring for problems around the clock.

In this project, you will provision AWS infrastructure with Terraform, containerize a Python web scraper with Docker, and deploy it to a Kubernetes cluster on AWS EKS. You will also set up monitoring with Prometheus and Grafana to observe your workloads in real time.

What You'll Build

A production-style cloud engineering pipeline that provisions infrastructure, deploys a containerized application as a scheduled job, and monitors everything with industry-standard observability tools.

By the end of this project, you'll have:

  • A fully provisioned EKS cluster and ECR repository created entirely through Terraform.
  • A containerized Python web scraper running as a Kubernetes CronJob on EKS.
  • A Prometheus and Grafana monitoring stack deployed via Helm, visualizing cluster and workload metrics.
  • Secret Mission: Add a custom Prometheus metric to your scraper and build a Grafana dashboard panel that visualizes scrape results over time.

Are there any prerequisites?

You will need an AWS account with admin-level IAM permissions. Basic familiarity with Docker, Kubernetes, and Terraform concepts is helpful for getting the most out of this project.

Before We Start

Before you can provision infrastructure and deploy containers, you need the right toolkit on your machine.

This step installs five CLI tools via Homebrew, sets up Docker Desktop for building container images, and connects your terminal to your AWS account.

In this step, get ready to:
  • Install AWS CLI, Terraform, kubectl, Helm, and Python 3 via Homebrew.
  • Install and start Docker Desktop for Mac.
  • Configure AWS CLI with your credentials and verify access.
Install CLI tools via Homebrew

You need five command-line tools to manage infrastructure, build containers, and interact with your Kubernetes cluster.

  • Open Terminal by pressing Cmd+Space (macOS) and typing Terminal, then pressing Enter.
  • Add the HashiCorp tap (required for Terraform) by running this command:
brew tap hashicorp/tap
  • Install all five CLI tools by running this command:
brew install awscli hashicorp/tap/terraform kubernetes-cli helm python@3.13

What does this command do?

  • awscli installs the AWS CLI for managing AWS resources from your terminal.
  • hashicorp/tap/terraform installs Terraform from HashiCorp's official tap for Infrastructure as Code.
  • kubernetes-cli installs kubectl, the command-line tool for talking to Kubernetes clusters.
  • helm installs the Helm package manager for deploying pre-built Kubernetes applications.
  • python@3.13 installs Python 3.13 for writing the scraper script.

What is Infrastructure as Code?

Infrastructure as Code (IaC) means defining your cloud resources in configuration files instead of clicking through a console.

Terraform reads those files and creates (or destroys) the matching resources automatically. This makes your infrastructure repeatable and version-controlled.

Homebrew will download and install each tool. This may take a few minutes depending on your internet connection.

  • Verify all five tools installed successfully by running these version checks:
aws --version
terraform --version
kubectl version --client
helm version
python3.13 --version

✔️ I see version numbers for all five tools

All your CLI tools are installed and ready. Move on to the next substep.

ⓧ One or more commands say 'command not found'

If a specific tool failed to install, re-run the full install command. Homebrew will skip tools that are already installed and only install the missing ones.

  • Re-run the install by running this command:
brew install awscli hashicorp/tap/terraform kubernetes-cli helm python@3.13

After reinstalling, run the version check commands again to confirm everything works.

Seeing 'permission denied' errors?

Homebrew sometimes needs ownership of its directories reset. Try closing your terminal, reopening it, and running the install command again.

help me fix Homebrew permission errors on macOS

Install and start Docker Desktop for Mac

Docker Desktop provides the Docker engine needed to build and run container images on macOS. On macOS, containers run inside a lightweight Linux virtual machine that Docker Desktop manages for you.

  • Open your browser and go to the Docker Desktop download page:
  • docker.com/products/docker-desktop.
  • Click the download button for your Mac type (Apple chip or Intel chip).
  • Double-click the downloaded Docker.dmg file to open the installer.
  • Drag the Docker icon to the Applications folder.
  • Open Docker Desktop by double-clicking Docker.app in your Applications folder.
  • Accept the Docker Subscription Service Agreement when prompted.
  • Wait for Docker Desktop to finish starting. You will see the Docker whale icon appear in your menu bar.

Why Docker Desktop instead of Docker Engine?

On macOS, containers need a Linux kernel to run. Docker Desktop manages a lightweight Linux VM for you, handles networking, and provides a graphical dashboard to inspect running containers.

  • Verify Docker is running by switching back to your Terminal and running this command:
docker --version

✔️ I see a Docker version number

Docker is installed and running. Move on to the next substep.

ⓧ Command not found or 'Cannot connect to Docker daemon'

  • Make sure Docker Desktop is open and fully started. The whale icon in the menu bar should be still, not animating.
  • If it is still not recognized, close your Terminal window and open a new one so it picks up the new Docker path.

Docker Desktop won't start?

macOS may block the app the first time. Open System Settings, then Privacy & Security. Look for a message about Docker being blocked and click Open Anyway.

help me troubleshoot Docker Desktop not starting on macOS

Configure AWS CLI with credentials

The AWS CLI needs your credentials to create resources in your AWS account. You will provide your Access Key ID, Secret Access Key, default region, and output format.

  • Configure your AWS credentials by running this command:
aws configure

What does aws configure do?

This command stores your credentials locally in ~/.aws/credentials and your preferences in ~/.aws/config.

Every AWS CLI and Terraform command you run afterwards uses these stored values automatically.

The CLI will prompt you for four values. Enter them one at a time:

  • Enter your AWS Access Key ID when prompted.
  • Enter your AWS Secret Access Key when prompted.
  • Enter us-east-1 for the default region (or your preferred region).
  • Enter json for the default output format.

Where do I find my access keys?

In the AWS console, click your username in the top-right corner, then Security credentials. Scroll to Access keys and create a new key pair if you do not have one.

  • Verify your credentials work by running this command:
aws sts get-caller-identity

You should see a JSON response containing your Account, UserId, and Arn. This confirms your terminal is authenticated with AWS.

✔️ I see my AWS account details in JSON

Your AWS CLI is configured and authenticated. You are ready to start building infrastructure.

ⓧ I see an error about invalid credentials

  • Double-check that you copied the Access Key ID and Secret Access Key without extra spaces.
  • Make sure the access key is Active (not deactivated) in the AWS console under Security credentials.
  • Re-run the configure command and enter both values again carefully:
aws configure

After re-entering your credentials, run the verification command again.

Getting 'SignatureDoesNotMatch' or 'InvalidClientTokenId'?

These errors mean the credentials are wrong or expired. Generate a fresh access key in the AWS console and configure again with the new values.

help me fix AWS CLI credential errors

Your toolkit is ready. Next up, you will write the Python scraper and package it into a Docker image.

Write the Python Scraper and Dockerfile

Your development environment is ready to go. Now it's time to write the actual application that will run inside your Kubernetes cluster.

In this step, you'll create a Python script that fetches the top stories from Hacker News using their public API. Then you'll package that script into a Docker container so it can run anywhere, whether that's your laptop or a cloud cluster.

In this step, get ready to:
  • Write a Python scraper that fetches top Hacker News stories.
  • Containerize the scraper with a Dockerfile.
  • Build and test the Docker image locally.
Create the project directory and scraper script
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir hn-scraper
cd hn-scraper
  • Confirm the folder exists by running:
ls ~/Desktop | grep hn-scraper

You should see hn-scraper in the output.

The Hacker News API is a free, public API that returns story data in JSON format. Your scraper will call this API to fetch the top 5 story titles and print them with a timestamp.

  • Create a file called scraper.py in your hn-scraper folder and add the following code:
import requests
from datetime import datetime

def get_top_stories(n=5):
    url = 'https://hacker-news.firebaseio.com/v0/topstories.json'
    response = requests.get(url)
    story_ids = response.json()[:n]
    stories = []
    for sid in story_ids:
        story_url = f'https://hacker-news.firebaseio.com/v0/item/{sid}.json'
        story = requests.get(story_url).json()
        stories.append(story.get('title', 'No title'))
    return stories

What does this code do?

The get_top_stories function hits the Hacker News API to fetch the list of top story IDs. It then loops through the first 5 IDs and fetches each story's title individually.

The API at https://hacker-news.firebaseio.com/v0/topstories.json returns up to 500 story IDs sorted by ranking. Your scraper slices the first 5 and fetches their details one by one.

  • Now add the main execution block below the function in the same file. This is the code that runs when you execute the script directly:
if __name__ == '__main__':
    timestamp = datetime.utcnow().isoformat()
    print(f'[{timestamp}] Fetching top Hacker News stories...')
    stories = get_top_stories()
    for i, title in enumerate(stories, 1):
        print(f'{i}. {title}')
    print('Done.')

What does the main block do?

The if __name__ == '__main__': guard ensures this code only runs when you execute the script directly. It won't run if another module imports the get_top_stories function.

It grabs a UTC timestamp, calls the function, and prints each story title numbered 1 through 5. This gives you a quick log of what the scraper found and when it ran.

  • Save scraper.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

import requests
from datetime import datetime

def get_top_stories(n=5):
    url = 'https://hacker-news.firebaseio.com/v0/topstories.json'
    response = requests.get(url)
    story_ids = response.json()[:n]
    stories = []
    for sid in story_ids:
        story_url = f'https://hacker-news.firebaseio.com/v0/item/{sid}.json'
        story = requests.get(story_url).json()
        stories.append(story.get('title', 'No title'))
    return stories

if __name__ == '__main__':
    timestamp = datetime.utcnow().isoformat()
    print(f'[{timestamp}] Fetching top Hacker News stories...')
    stories = get_top_stories()
    for i, title in enumerate(stories, 1):
        print(f'{i}. {title}')
    print('Done.')
Create the requirements file and Dockerfile

To run the scraper inside a container, Docker needs to know which Python packages to install. You specify these in a requirements.txt file that the Dockerfile references during the build.

  • Create a file called requirements.txt in your hn-scraper folder with the following content:
requests
  • Create a file called Dockerfile (no file extension) in the same hn-scraper folder:
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY scraper.py .
ENTRYPOINT ["python", "scraper.py"]

What does this Dockerfile do?

Each line in the Dockerfile is a build instruction. FROM python:3.11-slim starts from a lightweight Python image. WORKDIR /app sets the working directory inside the container.

The COPY and RUN instructions install dependencies first, then copy the script. This ordering means Docker caches the dependency layer. If you change scraper.py later, Docker only rebuilds from that layer forward.

The ENTRYPOINT tells Docker what command to run when the container starts.

  • Save both requirements.txt and Dockerfile.
  • Confirm all three files exist by running:
ls

You should see Dockerfile, requirements.txt, and scraper.py in the output.

Missing a file?

Make sure you created all files inside the hn-scraper folder on your Desktop. Run pwd to confirm your terminal is in ~/Desktop/hn-scraper.

If the Dockerfile is missing, check that it has no file extension. It should be named exactly Dockerfile with a capital D and no .txt or other suffix.

Help me find my project files in the terminal.

✔️ Awesome, I've got everything!

Great. Double check you've saved both files before moving on.

ⓧ I'd like to double check the full code

requests
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY scraper.py .
ENTRYPOINT ["python", "scraper.py"]
Build and test the Docker image locally

Before pushing anything to the cloud, you want to confirm the scraper works inside a container on your own machine. This catches dependency issues and script errors early.

  • Build your Docker image by running this command:
docker build -t hn-scraper .

What does this command do?

The -t hn-scraper flag tags (names) your image so you can reference it easily. The . at the end tells Docker to use the current directory as the build context, which is where it looks for the Dockerfile.

You should see Docker pulling the base image and installing the requests package. The final line will show a success message with your image ID.

Seeing a 'Cannot connect to the Docker daemon' error?

This means Docker Desktop isn't running. Open Docker Desktop from your applications and wait for the whale icon in your menu bar to stop animating.

If you see a permissions error, make sure you're running the command from inside the hn-scraper folder where your Dockerfile lives.

Help me fix my Docker build error.

  • Run your containerized scraper by running this command:
docker run hn-scraper

You should see output with a timestamp, followed by 5 numbered Hacker News story titles, and a final Done. line.

Seeing a 'requests' ModuleNotFoundError?

This means the requirements.txt wasn't picked up during the build. Open your Dockerfile and make sure the COPY requirements.txt . line comes before the RUN pip install line.

After fixing, rebuild the image with docker build -t hn-scraper . and run it again.

Help me fix my scraper container error.

Your scraper is working inside a container. Next, you'll provision the cloud infrastructure where this container will run on a schedule.

Provision EKS Infrastructure with Terraform

Your scraper works locally, but it needs a permanent home in the cloud where it can run on a schedule without your laptop being open.

In this step, you will write Terraform configuration to provision a full AWS environment: a VPC with networking, an EKS cluster to run containers, and an ECR registry to store your Docker image.

In this step, get ready to:
  • Write Terraform configuration files that define your infrastructure.
  • Provision the VPC, EKS cluster, and ECR repository with a single command.
  • Connect kubectl to your new EKS cluster and verify the nodes are running.
Write the Terraform configuration

Terraform uses declarative configuration files to describe your desired infrastructure. You will create four files inside a terraform/ directory: one for provider settings, one for variables, one for the actual resources, and one for outputs.

  • Create a terraform directory inside your hn-scraper project folder by running this command:
mkdir terraform
  • Confirm the directory was created by running:
ls

You should see terraform listed alongside your existing Dockerfile, scraper.py, and requirements.txt.

  • Create the provider configuration file by running:
touch terraform/providers.tf
  • Open terraform/providers.tf in VS Code and paste the following code:
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
  required_version = ">= 1.3.0"
}

provider "aws" {
  region = var.region
}

What does this file do?

The terraform block declares the AWS provider and pins it to version 5.x. This ensures Terraform downloads the correct plugin to manage AWS resources.

The provider block tells AWS which region to create resources in. The actual region value comes from a variable you will define next.

  • Save providers.tf.
  • Create the variables file by running:
touch terraform/variables.tf
  • Open terraform/variables.tf and paste the following code:
variable "region" {
  default = "us-east-1"
}

variable "cluster_name" {
  default = "hn-scraper-cluster"
}

variable "ecr_repo_name" {
  default = "hn-scraper"
}

What does this file do?

Variables let you reuse values across your configuration without hardcoding them. Here you define three: the AWS region, a name for your EKS cluster, and a name for your ECR image repository.

  • Save variables.tf.
  • Create the main infrastructure file by running:
touch terraform/main.tf
  • Open terraform/main.tf and paste the following code:
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "hn-scraper-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["us-east-1a", "us-east-1b"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24"]

  enable_nat_gateway = true
  single_nat_gateway = true

  tags = {
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
  }
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = var.cluster_name
  cluster_version = "1.29"

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  cluster_endpoint_public_access = true

  eks_managed_node_groups = {
    default = {
      instance_types = ["t3.small"]
      desired_size   = 2
      min_size       = 1
      max_size       = 3
    }
  }
}

resource "aws_ecr_repository" "scraper" {
  name                 = var.ecr_repo_name
  image_tag_mutability = "MUTABLE"
  force_delete         = true
}

What does this file do?

This file defines three resources using community Terraform modules and an AWS resource block:

  • The vpc module creates a Virtual Private Cloud with two public subnets, two private subnets, and a NAT gateway so nodes in private subnets can pull container images from the internet.
  • The eks module creates a Kubernetes cluster running version 1.29 with a managed node group of 2 t3.small instances. These are the worker machines that will run your scraper pods.
  • The aws_ecr_repository resource creates a private Docker image registry where you will push your scraper image.
  • Save main.tf.
  • Create the outputs file by running:
touch terraform/outputs.tf
  • Open terraform/outputs.tf and paste the following code:
output "cluster_name" {
  value = module.eks.cluster_name
}

output "ecr_repository_url" {
  value = aws_ecr_repository.scraper.repository_url
}

output "region" {
  value = var.region
}

What does this file do?

Outputs expose values from your infrastructure after Terraform creates it. You will use cluster_name and region to configure kubectl, and ecr_repository_url to tag and push your Docker image in the next step.

  • Save outputs.tf.

You won't see any output from these files yet. The next substep runs terraform init to validate all four files together.

Files not showing in your editor?

Make sure you created the files inside the terraform/ subdirectory, not in the root hn-scraper/ folder. Run ls terraform/ to confirm you see all four files: providers.tf, variables.tf, main.tf, and outputs.tf.

Help me debug this

✔️ Awesome, I've got everything!

Great. Double check you have saved all four files before moving on.

ⓧ I'd like to double check the full code

Here are all four Terraform files for comparison:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
  required_version = ">= 1.3.0"
}

provider "aws" {
  region = var.region
}
variable "region" {
  default = "us-east-1"
}

variable "cluster_name" {
  default = "hn-scraper-cluster"
}

variable "ecr_repo_name" {
  default = "hn-scraper"
}
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "hn-scraper-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["us-east-1a", "us-east-1b"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24"]

  enable_nat_gateway = true
  single_nat_gateway = true

  tags = {
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
  }
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = var.cluster_name
  cluster_version = "1.29"

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  cluster_endpoint_public_access = true

  eks_managed_node_groups = {
    default = {
      instance_types = ["t3.small"]
      desired_size   = 2
      min_size       = 1
      max_size       = 3
    }
  }
}

resource "aws_ecr_repository" "scraper" {
  name                 = var.ecr_repo_name
  image_tag_mutability = "MUTABLE"
  force_delete         = true
}
output "cluster_name" {
  value = module.eks.cluster_name
}

output "ecr_repository_url" {
  value = aws_ecr_repository.scraper.repository_url
}

output "region" {
  value = var.region
}
Initialize and apply Terraform

With all configuration files saved, you can now ask Terraform to download the required providers and modules, then create the actual AWS resources.

  • Navigate into the terraform directory by running:
cd terraform
  • Initialize Terraform by running:
terraform init
  • You should see output confirming initialization is complete, along with lines showing the AWS provider and VPC/EKS modules being downloaded.

What does terraform init do?

Terraform reads your .tf files, identifies which providers and modules you referenced, then downloads them into a hidden .terraform/ directory. It also creates a lock file (.terraform.lock.hcl) that pins the exact versions for reproducibility.

Seeing an error during init?

Check that all four .tf files are saved inside the terraform/ directory and that you ran the command from inside that directory. A common cause is a syntax typo in one of the files. Terraform's error message will point to the file and line number.

Help me troubleshoot this error

  • Provision all the AWS resources by running:
terraform apply
  • Terraform will show you an execution plan listing all the resources it will create. Type yes when prompted and press Enter to confirm.

What is Terraform creating?

This takes approximately 10 minutes. While you wait, here is what Terraform is provisioning:

  • A VPC with public and private subnets across two availability zones.
  • A NAT gateway so private subnet resources can reach the internet.
  • An EKS control plane (the Kubernetes API server managed by AWS).
  • A managed node group with 2 EC2 instances (t3.small) that will run your workloads.
  • An ECR repository to store your scraper Docker image.

This will take about 10 minutes

EKS cluster provisioning is the longest wait in this project. Feel free to stretch, grab a drink, or read ahead while Terraform works.

  • When the apply finishes, you should see a summary message showing resources added, along with your outputs printed below it.

The exact resource count will vary (typically 50+ resources for the full VPC and EKS setup).

Apply failed with an error?

If you see an error about insufficient permissions, verify your AWS CLI credentials have admin-level access. If the error mentions a resource limit, check that your AWS account allows at least 2 running EC2 instances in us-east-1.

If Terraform partially created resources before failing, run terraform apply again. Terraform is idempotent and will pick up where it left off.

Help me fix this failure

Connect kubectl to your EKS cluster

Your cluster is running, but kubectl on your machine does not know about it yet. You need to update your local kubeconfig file so kubectl can authenticate with the new cluster.

  • Configure kubectl to connect to your EKS cluster by running:
aws eks --region $(terraform output -raw region) update-kubeconfig --name $(terraform output -raw cluster_name)

What does this command do?

This command uses your Terraform outputs (region and cluster_name) to tell the AWS CLI to fetch the EKS cluster's connection details. It writes them to your ~/.kube/config file so kubectl knows how to reach the cluster.

  • Verify your nodes are running by running:
kubectl get nodes
  • You should see 2 nodes listed with Ready in the STATUS column. The output looks similar to this:
NAME                             STATUS   ROLES    AGE   VERSION
ip-10-0-1-xxx.ec2.internal       Ready    <none>   5m    v1.29.x
ip-10-0-2-xxx.ec2.internal       Ready    <none>   5m    v1.29.x

Both nodes show Ready. This confirms your EKS cluster is healthy and your local kubectl is connected.

Nodes not showing or not Ready?

If you see zero nodes, wait a few minutes. The managed node group can take additional time to register after the cluster is created. Run kubectl get nodes again after 2-3 minutes.

If you see an authentication error, make sure you ran aws eks update-kubeconfig with the correct cluster name. Run terraform output cluster_name to verify.

Help me troubleshoot my cluster connection

Your infrastructure is live. Next up, you will push your Docker image to ECR and deploy the scraper as a Kubernetes CronJob on this cluster.

Push Image to ECR and Deploy to EKS

Your infrastructure is live and your Docker image is built. Now it is time to get that image into the cloud and schedule it to run automatically.

In this step, you will push your scraper image to ECR (your private container registry on AWS), then deploy it as a Kubernetes CronJob that runs the scraper every 5 minutes.

In this step, get ready to:
  • Authenticate Docker to ECR and push your scraper image.
  • Write a Kubernetes CronJob manifest that schedules the scraper.
  • Deploy the CronJob to EKS and verify it runs successfully.
Authenticate Docker to ECR and push the image

Before Docker can push images to your private ECR repository, it needs an authentication token. The token is valid for 12 hours.

  • First, grab the ECR repository URL from your Terraform outputs by running:
cd ~/Desktop/hn-scraper/terraform
terraform output ecr_repository_url

What does this command do?

This reads the ecr_repository_url output you defined in outputs.tf. The URL looks something like 123456789012.dkr.ecr.us-east-1.amazonaws.com/hn-scraper. You will use this in the next commands.

  • Copy the URL that the command prints (without the surrounding quotes). You will need it for the next three commands.
  • Authenticate Docker to your ECR registry by running:
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin [[ECR_REPO_HOST="your-account-id.dkr.ecr.us-east-1.amazonaws.com"]]

What does this command do?

The aws ecr get-login-password command fetches a temporary authentication token from AWS. The pipe sends that token directly to docker login as the password, with AWS as the username.

The host portion is the first part of your ECR repository URL (everything before /hn-scraper).

  • You should see Login Succeeded in your terminal.

Login failed or access denied?

Make sure the host you passed to --password-stdin matches the account and region of your ECR repository. Double-check by re-running terraform output ecr_repository_url and using the portion before /hn-scraper.

Also confirm Docker Desktop is running. The docker login command requires the Docker daemon.

Help me troubleshoot my ECR Docker login failure.

  • Tag your local image with the ECR repository URL by running:
docker tag hn-scraper:latest [[ECR_REPO_URL="your-account-id.dkr.ecr.us-east-1.amazonaws.com/hn-scraper"]]:latest
  • Push the image to ECR by running:
docker push [[ECR_REPO_URL="your-account-id.dkr.ecr.us-east-1.amazonaws.com/hn-scraper"]]:latest
  • You should see Docker pushing each layer and finishing with a digest and size summary.

What just happened?

Your scraper image now lives in a private AWS registry. Your EKS cluster nodes can pull this image directly from ECR because they share the same AWS account and have the necessary IAM permissions.

Push denied or timeout?

If you see a denied or authorization error, your login token may have expired. Re-run the aws ecr get-login-password command to get a fresh token.

If the push hangs, confirm your internet connection is stable. Large layers can take a minute on slower connections.

Help me fix my ECR push error.

Write the Kubernetes CronJob manifest

A CronJob in Kubernetes creates Jobs on a repeating schedule. You will write a manifest that tells Kubernetes to run your scraper container every 5 minutes.

  • Create the k8s directory inside your project folder by running:
mkdir -p ~/Desktop/hn-scraper/k8s
  • Create and open the CronJob manifest file by running:
code ~/Desktop/hn-scraper/k8s/cronjob.yaml
  • Paste the following YAML into cronjob.yaml:
apiVersion: batch/v1
kind: CronJob
metadata:
  name: hn-scraper
  namespace: default
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: hn-scraper
              image: [[ECR_REPO_URL="your-account-id.dkr.ecr.us-east-1.amazonaws.com/hn-scraper"]]:latest
              imagePullPolicy: Always

What does this manifest do?

  • schedule: "*/5 * * * *" tells Kubernetes to create a new Job every 5 minutes.
  • jobTemplate defines the Pod spec for each Job. It pulls your ECR image and runs the scraper.
  • restartPolicy: OnFailure means if the container exits with an error, Kubernetes retries it.
  • imagePullPolicy: Always ensures the cluster always pulls the latest image, even if one is cached locally.
  • Save cronjob.yaml.

Not sure what to put for the image URL?

Run terraform output ecr_repository_url from the terraform directory again and use the full output followed by :latest. It should look like 123456789012.dkr.ecr.us-east-1.amazonaws.com/hn-scraper:latest.

Help me figure out what image URL to use in my CronJob manifest.

✔️ Awesome, I've got everything!

Great. Make sure you have saved cronjob.yaml before moving on.

ⓧ I'd like to double check the full code

Here is what your cronjob.yaml should look like (with your own ECR URL in place of the placeholder):

apiVersion: batch/v1
kind: CronJob
metadata:
  name: hn-scraper
  namespace: default
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: hn-scraper
              image: <ECR_REPO_URL>:latest
              imagePullPolicy: Always
Apply the manifest and verify the CronJob

Now deploy the CronJob to your EKS cluster and confirm it schedules and runs the scraper successfully.

  • Apply the CronJob manifest by running:
kubectl apply -f ~/Desktop/hn-scraper/k8s/cronjob.yaml
  • You should see cronjob.batch/hn-scraper created in the output.
  • Verify the CronJob is registered and scheduled by running:
kubectl get cronjob hn-scraper
  • You should see a table with columns like NAME, SCHEDULE, SUSPEND, ACTIVE, LAST SCHEDULE, and AGE. The SCHEDULE column should show */5 * * * *.

What does this output mean?

SUSPEND should be False (meaning the CronJob is active). LAST SCHEDULE shows <none> initially because the first Job has not triggered yet. It will fire within 5 minutes.

  • Watch for a Job to be created by running:
kubectl get jobs --watch
  • Wait up to 5 minutes. When you see a Job appear with 1/1 under COMPLETIONS, the scraper finished successfully. Press Ctrl+C to stop watching.
  • Find the pod created by the completed Job and view its logs by running:
kubectl get pods --selector=job-name=$(kubectl get jobs -o jsonpath='{.items[0].metadata.name}') -o jsonpath='{.items[0].metadata.name}' | xargs kubectl logs
  • You should see the scraper's output: a timestamp line, 5 numbered story titles from Hacker News, and Done. at the end.

What just happened?

Your EKS cluster pulled the scraper image from ECR, ran it inside a Pod, and the Pod printed the top 5 Hacker News stories. The CronJob will repeat this every 5 minutes automatically.

No Job appearing after 5 minutes?

Run kubectl describe cronjob hn-scraper and check the Events section for errors. A common issue is an invalid schedule format or a typo in the manifest.

If the Job was created but the Pod is stuck in ImagePullBackOff, the cluster cannot pull from ECR. Run kubectl describe pod <pod-name> to see the full error. Confirm your ECR image URL is correct and that the EKS node IAM role has ECR pull permissions (the default managed node group usually has these).

Help me debug why my CronJob is not running.

Your scraper is now running on autopilot in the cloud. Next up, you will deploy a full monitoring stack so you can observe your cluster and workloads in real time.

Deploy Prometheus and Grafana Monitoring

Your scraper is running on a schedule inside your EKS cluster. But how do you know if it's actually succeeding? How do you spot resource bottlenecks before they cause failures?

This is where observability comes in. In this step, you'll deploy a full Prometheus + Grafana monitoring stack using a single Helm chart. You'll get metrics collection, pre-built dashboards, and visibility into every pod on your cluster.

In this step, get ready to:
  • Install the kube-prometheus-stack Helm chart with EKS-tuned settings.
  • Access Grafana dashboards via port-forwarding.
  • Find your scraper CronJob pods in the monitoring dashboards.
Add the Helm repository and install the monitoring stack

The kube-prometheus-stack Helm chart bundles Prometheus, Grafana, Alertmanager, kube-state-metrics, and node-exporter into a single install. This is how production teams deploy monitoring.

  • Add the prometheus-community Helm repository by running this command:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
  • Update your local chart index by running:
helm repo update

What is a Helm repository?

A Helm repository is like a package registry for Kubernetes applications. Adding the prometheus-community repo gives Helm access to all their charts, including kube-prometheus-stack.

On EKS, the Kubernetes control plane (etcd, controller-manager, scheduler) is fully managed by AWS. Those internal components don't expose metrics to your cluster. You need to tell the monitoring stack to skip scraping them.

  • Install the kube-prometheus-stack chart into a monitoring namespace with EKS-tuned values by running:
helm install kube-prom-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set kubeEtcd.enabled=false \
  --set kubeControllerManager.enabled=false \
  --set kubeScheduler.enabled=false

What do these flags do?

The --namespace monitoring flag installs all monitoring resources into a dedicated namespace, keeping them separate from your application workloads.

The three --set flags disable scrapers for etcd, kube-controller-manager, and kube-scheduler. On EKS, AWS manages these components and does not expose their metrics endpoints. Disabling them prevents noisy error logs and false alerts.

  • Wait for the installation to complete. This takes 1-2 minutes.
  • Verify that the monitoring pods are running by running:
kubectl get pods -n monitoring

You should see several pods with names starting with kube-prom-stack, including Prometheus, Grafana, and node-exporter pods. All should show a STATUS of Running.

Pods stuck in Pending or CrashLoopBackOff?

If pods are stuck in Pending, your nodes may not have enough resources. Check node capacity with kubectl describe nodes and look at the Allocatable vs Allocated sections.

If you see CrashLoopBackOff, check the pod logs with kubectl logs <pod-name> -n monitoring.

Help me troubleshoot my monitoring pods

Port-forward to Grafana and explore dashboards

Grafana is now running inside your cluster, but it's not exposed to the internet. Port-forwarding creates a secure tunnel from your local machine to the Grafana service.

  • Start a port-forward to the Grafana service by running:
kubectl port-forward -n monitoring svc/kube-prom-stack-grafana 3000:80

What does this command do?

This forwards port 3000 on your local machine to port 80 on the Grafana service inside the monitoring namespace. As long as this command is running, you can access Grafana in your browser.

  • Open a new terminal tab (keep the port-forward running in the first one).
  • Retrieve the auto-generated Grafana admin password by running:
kubectl get secret -n monitoring kube-prom-stack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode; echo
  • Copy the output. This is your Grafana admin password.
  • Open your browser and navigate to http://localhost:3000.
  • Log in with username admin and the password you just copied.

Grafana generates a random password

The kube-prometheus-stack chart auto-generates a random Grafana admin password on first install and stores it in a Kubernetes secret. The command above decodes that secret so you can log in.

  • Once logged in, click the hamburger menu (three horizontal lines) in the top-left corner.
  • Click Dashboards.
  • You should see a list of pre-built Kubernetes dashboards provided by the chart.

Browse through the dashboards. Look for ones like Kubernetes / Compute Resources / Cluster to see overall cluster resource usage.

  • Open the Kubernetes / Compute Resources / Cluster dashboard to verify Prometheus is collecting metrics from your nodes and pods.

You should see CPU and memory usage panels populated with data from your two nodes.

Seeing 'No data' in the panels?

Prometheus needs a few minutes after installation to start scraping and storing metrics. Wait 2-3 minutes, then refresh the dashboard.

If panels still show no data, verify Prometheus targets are healthy. Click the hamburger menu, go to Connections then Data sources, and confirm the Prometheus data source shows a green checkmark.

Help me check if Prometheus is scraping correctly

Find your scraper CronJob pods in the dashboards

The pre-built dashboards can filter by namespace and workload. Let's find your scraper pods to confirm they're visible to the monitoring stack.

  • In the Grafana sidebar, click Dashboards again.
  • Find and open the Kubernetes / Compute Resources / Namespace (Pods) dashboard.
  • In the namespace dropdown at the top of the dashboard, select default.

You should see resource usage data for pods in the default namespace. If your CronJob has fired recently, you'll see the scraper pod's CPU and memory consumption.

Why filter to the default namespace?

Your hn-scraper CronJob is deployed in the default namespace (as defined in your cronjob.yaml). Filtering the dashboard to this namespace isolates your scraper's resource usage from the monitoring stack's own pods in the monitoring namespace.

CronJob pods are short-lived

Your scraper runs every 5 minutes and completes in seconds. You may need to wait for the next scheduled run to see a pod appear. Check the schedule with kubectl get cronjobs to see the last scheduled time.

Your cluster now has full observability. Prometheus collects metrics from every node and pod, and Grafana gives you pre-built dashboards to visualize everything. In the Secret Mission, you'll take this further by adding custom metrics to your scraper itself.

Secret mission

Add Custom Metrics to Your Scraper

Your scraper runs on a schedule, but you have no visibility into whether it succeeds or how long it takes. In this bonus challenge, you will instrument the Python scraper with custom Prometheus metrics, wire them into the monitoring stack via a ServiceMonitor, and build a Grafana dashboard panel that visualizes scrape performance over time.

Clean Up Your Resources

Clean Up Your Resources

This project provisions AWS resources that cost approximately $0.19/hour (~$4.50/day) while running. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Cost warning

Your EKS cluster costs $0.10/hour, the NAT gateway costs ~$0.045/hour, and the two t3.small nodes add ~$0.042/hour. These charges accumulate 24/7 whether or not the scraper is running.

EKS billing is all-or-nothing. There is no way to pause the cluster without destroying it. Strongly consider deleting everything immediately after completing the project.

Resources you used:

  • EKS cluster (hn-scraper-cluster) with 2x t3.small managed nodes.
  • ECR repository (hn-scraper).
  • VPC (hn-scraper-vpc) with public/private subnets and NAT gateway.
  • kube-prometheus-stack Helm release in the monitoring namespace.
  • Kubernetes Deployment, Service, ServiceMonitor, and CronJob resources.
  • Local project files (hn-scraper directory on your Desktop).

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with Kubernetes and monitoring. Be aware costs accumulate at ~$0.19/hour.

  • Your EKS cluster, nodes, and all Kubernetes workloads remain running.
  • Your local project files in hn-scraper stay on your Desktop.
  • Docker Desktop, AWS CLI, Terraform, kubectl, and Helm remain installed on your machine.

Pause - I'll come back to this later

EKS does not support pausing a cluster. The only way to stop costs is to destroy the AWS resources. You can keep your local Terraform and Kubernetes files so you can re-provision later.

  • Stop any running port-forward sessions in your terminal (press Ctrl+C).
  • Navigate to your Terraform directory and destroy all cloud resources by running:
  • Type yes when prompted to confirm.
cd ~/Desktop/hn-scraper/terraform
terraform destroy

Your local files remain intact. When you want to resume, run terraform apply again to re-provision everything.

Delete - I don't want to use this again

Remove all project resources and start fresh. This destroys all AWS infrastructure and removes local files.

  • Stop any running port-forward sessions in your terminal (press Ctrl+C).
  • Navigate to your Terraform directory and destroy all AWS resources by running:
cd ~/Desktop/hn-scraper/terraform
terraform destroy
  • Type yes when prompted to confirm. This removes the EKS cluster, ECR repository, VPC, NAT gateway, and all associated resources.
  • Remove your local project files by running:
cd ~/Desktop
rm -rf hn-scraper
  • Optionally uninstall the CLI tools you installed via Homebrew by running:
brew uninstall awscli hashicorp/tap/terraform kubernetes-cli helm python@3.13
  • Optionally uninstall Docker Desktop by opening the app, clicking the Troubleshoot icon (top-right), then selecting Uninstall.

That's a Wrap!

That's a Wrap!

Nice work! You just built a complete cloud engineering pipeline from scratch.

You provisioned AWS infrastructure with Terraform, containerized a Python scraper, deployed it to Kubernetes, and set up production monitoring. This is the exact workflow used by cloud and DevOps teams every day.

You've learned how to:

  • Provision production-grade AWS infrastructure (VPC, EKS, ECR) entirely through Terraform configuration files.
  • Containerize a Python web scraper with Docker and deploy it as a Kubernetes CronJob on EKS.
  • Deploy a full Prometheus + Grafana observability stack via Helm with pre-built Kubernetes dashboards.
  • Secret Mission: Instrument the scraper with custom Prometheus metrics and build a Grafana dashboard panel to visualize scrape success over time.

Ready to quiz yourself?