Production DevSecOps Pipeline on GKE

Introduction

30 Second Summary

Every time you download an app update or visit a website, someone's pipeline built, scanned, and deployed that code. The difference between amateur and production-grade delivery is automated security gates that catch vulnerabilities before they reach users.

In this project, you will build a complete DevSecOps platform on Google Kubernetes Engine that covers the entire software delivery lifecycle from infrastructure provisioning to runtime security monitoring.

What You'll Build

A production-grade DevSecOps platform on GKE featuring a Python Flask microservice, Jenkins CI/CD with automated security scanning, ArgoCD GitOps deployment, Istio service mesh, and Prometheus/Grafana observability.

By the end of this project, you'll have:

• A GKE cluster provisioned entirely through OpenTofu infrastructure as code, with all tooling bootstrapped via Ansible playbooks.
• A Jenkins CI/CD pipeline with security gates (SonarQube SAST, Trivy container scanning, OWASP ZAP DAST) that block vulnerable deployments from reaching production.
• A fully observable, self-healing, GitOps-driven Kubernetes deployment with ArgoCD auto-sync, Istio service mesh, Kyverno policies, Falco runtime detection, and Prometheus + Grafana dashboards.
• Secret Mission: Create a Grafana dashboard that correlates Falco security alerts with application metrics to detect attack patterns in real-time.

Are there any prerequisites?

This project assumes some DevSecOps experience and a GCP account with billing enabled. You should already have Docker, kubectl, Helm, and Git installed locally.

GCP costs for this project are approximately $1.50-$2.00 per day using Spot VMs. The GKE free tier covers the cluster management fee.

Provision GKE Infrastructure with OpenTofu

Your local tools are ready. Now you need somewhere to deploy your DevSecOps platform.

In this step, you will define your entire GKE infrastructure as code using OpenTofu. You will create a VPC network, a GKE cluster with spot nodes for cost savings, an Artifact Registry for container images, and IAM service accounts with least-privilege access.

In this step, get ready to:

• Write OpenTofu configuration files for VPC, GKE cluster, Artifact Registry, and IAM.
• Run tofu init, tofu plan, and tofu apply to provision the infrastructure.
• Configure kubectl to connect to the new GKE cluster.

Write OpenTofu configuration for GKE infrastructure

Infrastructure as Code means you describe your cloud resources in configuration files. OpenTofu reads these files and creates the resources in your GCP project.

• Create a project directory and an infra folder inside it by running these commands:

cd ~/Desktop
mkdir -p devsecops-pipeline/infra
cd devsecops-pipeline/infra

• Confirm the folder exists by running:

pwd

You should see ~/Desktop/devsecops-pipeline/infra in the output.

• Create the variables.tf file that defines all input variables for your infrastructure by running:

cat > variables.tf << 'EOF'
variable "project_id" {
  description = "GCP project ID"
  type        = string
}

variable "region" {
  description = "GCP region"
  type        = string
  default     = "us-central1"
}

variable "zone" {
  description = "GCP zone"
  type        = string
  default     = "us-central1-a"
}

variable "cluster_name" {
  description = "GKE cluster name"
  type        = string
  default     = "devsecops-cluster"
}

variable "node_count" {
  description = "Number of nodes in the node pool"
  type        = number
  default     = 3
}
EOF

What does this file do?

Variables separate the values that change per environment from the infrastructure logic. The project_id variable has no default because every learner's GCP project is different. The other variables have sensible defaults for this project.

• Create the main.tf file with the provider and VPC configuration by running:

cat > main.tf << 'EOF'
terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 7.35.0"
    }
  }
}

provider "google" {
  project = var.project_id
  region  = var.region
}

# VPC network for the GKE cluster
resource "google_compute_network" "vpc" {
  name                    = "devsecops-vpc"
  auto_create_subnetworks = false
}

# Subnet with secondary ranges for pods and services
resource "google_compute_subnetwork" "subnet" {
  name          = "devsecops-subnet"
  ip_cidr_range = "10.0.0.0/24"
  region        = var.region
  network       = google_compute_network.vpc.id

  secondary_ip_range {
    range_name    = "pods"
    ip_cidr_range = "10.1.0.0/16"
  }

  secondary_ip_range {
    range_name    = "services"
    ip_cidr_range = "10.2.0.0/20"
  }
}

# GKE cluster with spot nodes for cost savings
resource "google_container_cluster" "primary" {
  name     = var.cluster_name
  location = var.zone

  # Remove default node pool and use a custom one
  remove_default_node_pool = true
  initial_node_count       = 1

  network    = google_compute_network.vpc.name
  subnetwork = google_compute_subnetwork.subnet.name

  ip_allocation_policy {
    cluster_secondary_range_name  = "pods"
    services_secondary_range_name = "services"
  }

  # Enable workload identity for secure pod-to-GCP access
  workload_identity_config {
    workload_pool = "${var.project_id}.svc.id.goog"
  }
}

# Custom node pool with spot VMs
resource "google_container_node_pool" "primary_nodes" {
  name       = "devsecops-node-pool"
  location   = var.zone
  cluster    = google_container_cluster.primary.name
  node_count = var.node_count

  node_config {
    machine_type = "e2-standard-2"
    spot         = true

    # Use the dedicated service account
    service_account = google_service_account.gke_sa.email
    oauth_scopes = [
      "https://www.googleapis.com/auth/cloud-platform"
    ]

    labels = {
      environment = "devsecops"
    }
  }
}

# Artifact Registry for container images
resource "google_artifact_registry_repository" "registry" {
  location      = var.region
  repository_id = "devsecops-registry"
  format        = "DOCKER"
  description   = "Docker registry for DevSecOps microservices"
}

# Service account for GKE nodes
resource "google_service_account" "gke_sa" {
  account_id   = "devsecops-gke-sa"
  display_name = "DevSecOps GKE Service Account"
}

# Allow GKE nodes to pull and push images to Artifact Registry
resource "google_project_iam_member" "registry_writer" {
  project = var.project_id
  role    = "roles/artifactregistry.writer"
  member  = "serviceAccount:${google_service_account.gke_sa.email}"
}

# Allow GKE nodes basic node access
resource "google_project_iam_member" "node_access" {
  project = var.project_id
  role    = "roles/container.nodeServiceAccount"
  member  = "serviceAccount:${google_service_account.gke_sa.email}"
}
EOF

What does this configuration create?

This single file defines your entire infrastructure. The VPC and subnet create an isolated network with dedicated IP ranges for pods and services.

The GKE cluster uses spot VMs (60-80% cheaper than regular instances) and Workload Identity for secure API access. The Artifact Registry stores your Docker images. The IAM service account gives nodes just enough permissions to pull images and operate.

Why spot VMs?

Spot VMs are 60-80% cheaper than regular instances. For a learning project, the occasional preemption is acceptable. It also teaches real production concepts like graceful termination and Pod Disruption Budgets.

• Create the outputs.tf file that exposes key values after provisioning by running:

cat > outputs.tf << 'EOF'
output "cluster_endpoint" {
  description = "GKE cluster endpoint"
  value       = google_container_cluster.primary.endpoint
  sensitive   = true
}

output "cluster_name" {
  description = "GKE cluster name"
  value       = google_container_cluster.primary.name
}

output "artifact_registry_url" {
  description = "Artifact Registry URL for pushing images"
  value       = "${var.region}-docker.pkg.dev/${var.project_id}/${google_artifact_registry_repository.registry.repository_id}"
}
EOF

What are outputs for?

Outputs expose values you need after provisioning. The cluster_endpoint is marked sensitive because it exposes the cluster's API address. The artifact_registry_url gives you the full path for pushing Docker images later.

• Verify all three files exist by running:

ls

You should see main.tf, variables.tf, and outputs.tf in the output.

Don't see all three files?

Make sure you ran each cat > filename << 'EOF' command completely. If a file is missing, re-run its creation command. Check your current directory is ~/Desktop/devsecops-pipeline/infra with pwd.

help me find my missing OpenTofu files

✔️ Awesome, I've got everything!

Great. All three configuration files are in place.

ⓧ I'd like to double check the full code

Here is the complete variables.tf:

variable "project_id" {
  description = "GCP project ID"
  type        = string
}

variable "region" {
  description = "GCP region"
  type        = string
  default     = "us-central1"
}

variable "zone" {
  description = "GCP zone"
  type        = string
  default     = "us-central1-a"
}

variable "cluster_name" {
  description = "GKE cluster name"
  type        = string
  default     = "devsecops-cluster"
}

variable "node_count" {
  description = "Number of nodes in the node pool"
  type        = number
  default     = 3
}

Here is the complete main.tf:

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 7.35.0"
    }
  }
}

provider "google" {
  project = var.project_id
  region  = var.region
}

# VPC network for the GKE cluster
resource "google_compute_network" "vpc" {
  name                    = "devsecops-vpc"
  auto_create_subnetworks = false
}

# Subnet with secondary ranges for pods and services
resource "google_compute_subnetwork" "subnet" {
  name          = "devsecops-subnet"
  ip_cidr_range = "10.0.0.0/24"
  region        = var.region
  network       = google_compute_network.vpc.id

  secondary_ip_range {
    range_name    = "pods"
    ip_cidr_range = "10.1.0.0/16"
  }

  secondary_ip_range {
    range_name    = "services"
    ip_cidr_range = "10.2.0.0/20"
  }
}

# GKE cluster with spot nodes for cost savings
resource "google_container_cluster" "primary" {
  name     = var.cluster_name
  location = var.zone

  # Remove default node pool and use a custom one
  remove_default_node_pool = true
  initial_node_count       = 1

  network    = google_compute_network.vpc.name
  subnetwork = google_compute_subnetwork.subnet.name

  ip_allocation_policy {
    cluster_secondary_range_name  = "pods"
    services_secondary_range_name = "services"
  }

  # Enable workload identity for secure pod-to-GCP access
  workload_identity_config {
    workload_pool = "${var.project_id}.svc.id.goog"
  }
}

# Custom node pool with spot VMs
resource "google_container_node_pool" "primary_nodes" {
  name       = "devsecops-node-pool"
  location   = var.zone
  cluster    = google_container_cluster.primary.name
  node_count = var.node_count

  node_config {
    machine_type = "e2-standard-2"
    spot         = true

    # Use the dedicated service account
    service_account = google_service_account.gke_sa.email
    oauth_scopes = [
      "https://www.googleapis.com/auth/cloud-platform"
    ]

    labels = {
      environment = "devsecops"
    }
  }
}

# Artifact Registry for container images
resource "google_artifact_registry_repository" "registry" {
  location      = var.region
  repository_id = "devsecops-registry"
  format        = "DOCKER"
  description   = "Docker registry for DevSecOps microservices"
}

# Service account for GKE nodes
resource "google_service_account" "gke_sa" {
  account_id   = "devsecops-gke-sa"
  display_name = "DevSecOps GKE Service Account"
}

# Allow GKE nodes to pull and push images to Artifact Registry
resource "google_project_iam_member" "registry_writer" {
  project = var.project_id
  role    = "roles/artifactregistry.writer"
  member  = "serviceAccount:${google_service_account.gke_sa.email}"
}

# Allow GKE nodes basic node access
resource "google_project_iam_member" "node_access" {
  project = var.project_id
  role    = "roles/container.nodeServiceAccount"
  member  = "serviceAccount:${google_service_account.gke_sa.email}"
}

Here is the complete outputs.tf:

output "cluster_endpoint" {
  description = "GKE cluster endpoint"
  value       = google_container_cluster.primary.endpoint
  sensitive   = true
}

output "cluster_name" {
  description = "GKE cluster name"
  value       = google_container_cluster.primary.name
}

output "artifact_registry_url" {
  description = "Artifact Registry URL for pushing images"
  value       = "${var.region}-docker.pkg.dev/${var.project_id}/${google_artifact_registry_repository.registry.repository_id}"
}

Run tofu init, tofu plan, and tofu apply

With configuration files in place, you follow the OpenTofu workflow: initialize, plan, then apply. This three-step process gives you a chance to review exactly what will be created before spending money.

• Set your GCP project ID as an environment variable so OpenTofu can use it by running:

export TF_VAR_project_id=[[PROJECT_ID="your-gcp-project-id"]]

What is TF_VAR_project_id?

OpenTofu automatically reads environment variables prefixed with TF_VAR_ and maps them to input variables. Setting TF_VAR_project_id feeds the value into the project_id variable defined in variables.tf.

• Initialize the OpenTofu working directory by running:

tofu init

You should see a message ending with OpenTofu has been successfully initialized! confirming the Google provider was downloaded.

Seeing a provider error?

If you see an error about the provider not being found, check your internet connection. The tofu init command downloads the hashicorp/google provider from the OpenTofu registry. Make sure you are inside the infra directory where your .tf files live.

help me fix my tofu init error

• Preview the infrastructure changes by running:

tofu plan

You should see a plan showing resources to be created. Look for the line that says Plan: X to add, 0 to change, 0 to destroy where X is the total number of resources.

Why run plan before apply?

The plan command shows you exactly what will be created, modified, or destroyed without making any changes. In production, this is a critical review step. It catches misconfigurations before they cost money or cause outages.

• Provision the infrastructure by running:

tofu apply

What will happen next?

OpenTofu will display the same plan you just reviewed, then ask for confirmation. Type yes and press Enter to proceed. This step takes 5-10 minutes because GKE cluster creation involves provisioning VMs and configuring networking.

• When prompted with Enter a value:, type yes and press Enter.

Wait for the apply to complete. You should see a final message like Apply complete! Resources: X added, 0 changed, 0 destroyed. confirming all resources were created.

What is the GKE free tier?

GCP provides a $74.40/month credit covering the management fee for one zonal cluster per billing account. The compute nodes (your spot VMs) still cost money, but the cluster control plane itself is effectively free.

Apply failed with a permissions error?

Ensure your gcloud CLI is authenticated to the correct project. Run gcloud config get-value project and confirm it matches the value you set in TF_VAR_project_id. You also need the Owner or Editor role on the project to create GKE clusters and IAM resources.

help me fix my tofu apply permissions error

Configure kubectl to connect to the new GKE cluster

Your cluster is running in GCP, but your local kubectl does not know about it yet. You need to fetch credentials so kubectl can communicate with the cluster's API server.

• Configure kubectl to connect to your new cluster by running:

gcloud container clusters get-credentials devsecops-cluster --location=us-central1-a

What does this command do?

This command updates your local ~/.kube/config file with the cluster's endpoint and authentication credentials. After this, every kubectl command you run targets your new GKE cluster.

• Verify the connection by listing your cluster nodes:

kubectl get nodes

You should see 3 nodes with a Ready status. These are your e2-standard-2 spot VMs running in GKE.

No nodes showing or connection refused?

If you see a connection error, ensure the tofu apply completed successfully. The cluster takes a few minutes to become fully ready after the apply finishes. Wait 2-3 minutes and try kubectl get nodes again.

help me connect kubectl to my GKE cluster

Your GKE infrastructure is live. Next up, you will bootstrap the cluster with all the tools your DevSecOps platform needs using Ansible.

Bootstrap Cluster with Ansible

Your GKE cluster is running and kubectl is connected. You now have raw infrastructure with no tooling installed on it.

In this step, you will write a single Ansible playbook that installs every tool your DevSecOps platform needs. Instead of running eight separate helm install commands manually, Ansible lets you declare the entire desired state of the cluster and apply it in one command.

In this step, get ready to:

• Write an Ansible playbook that installs all Helm charts into the GKE cluster.
• Run the playbook to configure the entire cluster in one command.
• Verify all pods are running across namespaces.

Write the Ansible playbook

Ansible uses an inventory file to know where to run tasks and a playbook file to declare what those tasks are. Since the Helm commands target your already-connected kubectl context, you will run Ansible against localhost.

• Create an ansible directory at the same level as your infra directory by running:

mkdir -p ansible
cd ansible

• Create the inventory file by running:

touch inventory.yml

• Confirm the file exists by running:

ls

You should see inventory.yml in the output.

• Open inventory.yml in your editor and paste the following content:

# Ansible inventory - run tasks on localhost using the local kubectl context
all:
  hosts:
    localhost:
      ansible_connection: local

What does this inventory do?

The inventory tells Ansible to run all tasks on your local machine using ansible_connection: local. This works because kubectl and helm are already configured to talk to your GKE cluster.

• Create the playbook file by running:

touch playbook.yml

• Open playbook.yml and paste the following Helm repository tasks:

# Ansible playbook to bootstrap the GKE cluster with all DevSecOps tooling
---
- name: Bootstrap DevSecOps cluster
  hosts: localhost
  gather_facts: false

  tasks:
    # --- Add all Helm repositories ---
    - name: Add Jenkins Helm repo
      kubernetes.core.helm_repository:
        name: jenkins
        repo_url: https://charts.jenkins.io

    - name: Add SonarQube Helm repo
      kubernetes.core.helm_repository:
        name: sonarqube
        repo_url: https://SonarSource.github.io/helm-chart-sonarqube

    - name: Add ArgoCD Helm repo
      kubernetes.core.helm_repository:
        name: argo
        repo_url: https://argoproj.github.io/argo-helm

    - name: Add Prometheus Community Helm repo
      kubernetes.core.helm_repository:
        name: prometheus-community
        repo_url: https://prometheus-community.github.io/helm-charts

    - name: Add Kyverno Helm repo
      kubernetes.core.helm_repository:
        name: kyverno
        repo_url: https://kyverno.github.io/kyverno/helm

    - name: Add Falco Helm repo
      kubernetes.core.helm_repository:
        name: falcosecurity
        repo_url: https://falcosecurity.github.io/charts

    - name: Add Istio Helm repo
      kubernetes.core.helm_repository:
        name: istio
        repo_url: https://istio-release.storage.googleapis.com/charts

    - name: Add NGINX Ingress Helm repo
      kubernetes.core.helm_repository:
        name: ingress-nginx
        repo_url: https://kubernetes.github.io/ingress-nginx

What does this code do?

Each kubernetes.core.helm_repository task registers a Helm chart repository on your machine. This is equivalent to running helm repo add manually, but Ansible makes it idempotent and repeatable.

• Now add the Helm install tasks below the repository section in the same file. Paste these tasks after the last helm_repository entry:

    # --- Install Helm charts ---
    - name: Install Jenkins
      kubernetes.core.helm:
        name: jenkins
        chart_ref: jenkins/jenkins
        release_namespace: jenkins
        create_namespace: true
        wait: true
        timeout: 10m0s

    - name: Install SonarQube Community Build
      kubernetes.core.helm:
        name: sonarqube
        chart_ref: sonarqube/sonarqube
        release_namespace: sonarqube
        create_namespace: true
        wait: true
        timeout: 10m0s
        values:
          community:
            enabled: true

    - name: Install ArgoCD
      kubernetes.core.helm:
        name: argocd
        chart_ref: argo/argo-cd
        chart_version: "9.5.16"
        release_namespace: argocd
        create_namespace: true
        wait: true
        timeout: 10m0s

    - name: Install kube-prometheus-stack
      kubernetes.core.helm:
        name: kube-prometheus-stack
        chart_ref: prometheus-community/kube-prometheus-stack
        release_namespace: monitoring
        create_namespace: true
        wait: true
        timeout: 10m0s

    - name: Install Kyverno
      kubernetes.core.helm:
        name: kyverno
        chart_ref: kyverno/kyverno
        release_namespace: kyverno
        create_namespace: true
        wait: true
        timeout: 10m0s

    - name: Install Falco
      kubernetes.core.helm:
        name: falco
        chart_ref: falcosecurity/falco
        release_namespace: falco
        create_namespace: true
        wait: true
        timeout: 10m0s
        values:
          tty: true
          falcosidekick:
            enabled: true
            webui:
              enabled: true

    - name: Install Istio Base
      kubernetes.core.helm:
        name: istio-base
        chart_ref: istio/base
        release_namespace: istio-system
        create_namespace: true
        wait: true
        timeout: 10m0s

    - name: Install Istiod
      kubernetes.core.helm:
        name: istiod
        chart_ref: istio/istiod
        release_namespace: istio-system
        wait: true
        timeout: 10m0s
        values:
          global:
            platform: gke

    - name: Install NGINX Ingress Controller
      kubernetes.core.helm:
        name: ingress-nginx
        chart_ref: ingress-nginx/ingress-nginx
        release_namespace: ingress-nginx
        create_namespace: true
        wait: true
        timeout: 10m0s

What does this code do?

Each kubernetes.core.helm task installs a Helm chart into a dedicated namespace. Key things to note:

• ArgoCD is pinned to chart version 9.5.16 which bundles ArgoCD v3.4.3.
• SonarQube uses community.enabled: true to install the Community Build edition.
• Falco enables Falcosidekick and its web UI for viewing alerts.
• Istio is installed in two stages. First istio/base (CRDs), then istio/istiod (the control plane) with global.platform: gke for GKE-specific optimizations.
• Every install uses wait: true so Ansible blocks until pods are actually running before moving to the next chart.

• Save both inventory.yml and playbook.yml.

✔️ Awesome, I've got everything!

Great. Double check you have saved both files in the ansible/ directory.

ⓧ I'd like to double check the full code

Here is the complete ansible/inventory.yml:

# Ansible inventory - run tasks on localhost using the local kubectl context
all:
  hosts:
    localhost:
      ansible_connection: local

Here is the complete ansible/playbook.yml:

# Ansible playbook to bootstrap the GKE cluster with all DevSecOps tooling
---
- name: Bootstrap DevSecOps cluster
  hosts: localhost
  gather_facts: false

  tasks:
    # --- Add all Helm repositories ---
    - name: Add Jenkins Helm repo
      kubernetes.core.helm_repository:
        name: jenkins
        repo_url: https://charts.jenkins.io

    - name: Add SonarQube Helm repo
      kubernetes.core.helm_repository:
        name: sonarqube
        repo_url: https://SonarSource.github.io/helm-chart-sonarqube

    - name: Add ArgoCD Helm repo
      kubernetes.core.helm_repository:
        name: argo
        repo_url: https://argoproj.github.io/argo-helm

    - name: Add Prometheus Community Helm repo
      kubernetes.core.helm_repository:
        name: prometheus-community
        repo_url: https://prometheus-community.github.io/helm-charts

    - name: Add Kyverno Helm repo
      kubernetes.core.helm_repository:
        name: kyverno
        repo_url: https://kyverno.github.io/kyverno/helm

    - name: Add Falco Helm repo
      kubernetes.core.helm_repository:
        name: falcosecurity
        repo_url: https://falcosecurity.github.io/charts

    - name: Add Istio Helm repo
      kubernetes.core.helm_repository:
        name: istio
        repo_url: https://istio-release.storage.googleapis.com/charts

    - name: Add NGINX Ingress Helm repo
      kubernetes.core.helm_repository:
        name: ingress-nginx
        repo_url: https://kubernetes.github.io/ingress-nginx

    # --- Install Helm charts ---
    - name: Install Jenkins
      kubernetes.core.helm:
        name: jenkins
        chart_ref: jenkins/jenkins
        release_namespace: jenkins
        create_namespace: true
        wait: true
        timeout: 10m0s

    - name: Install SonarQube Community Build
      kubernetes.core.helm:
        name: sonarqube
        chart_ref: sonarqube/sonarqube
        release_namespace: sonarqube
        create_namespace: true
        wait: true
        timeout: 10m0s
        values:
          community:
            enabled: true

    - name: Install ArgoCD
      kubernetes.core.helm:
        name: argocd
        chart_ref: argo/argo-cd
        chart_version: "9.5.16"
        release_namespace: argocd
        create_namespace: true
        wait: true
        timeout: 10m0s

    - name: Install kube-prometheus-stack
      kubernetes.core.helm:
        name: kube-prometheus-stack
        chart_ref: prometheus-community/kube-prometheus-stack
        release_namespace: monitoring
        create_namespace: true
        wait: true
        timeout: 10m0s

    - name: Install Kyverno
      kubernetes.core.helm:
        name: kyverno
        chart_ref: kyverno/kyverno
        release_namespace: kyverno
        create_namespace: true
        wait: true
        timeout: 10m0s

    - name: Install Falco
      kubernetes.core.helm:
        name: falco
        chart_ref: falcosecurity/falco
        release_namespace: falco
        create_namespace: true
        wait: true
        timeout: 10m0s
        values:
          tty: true
          falcosidekick:
            enabled: true
            webui:
              enabled: true

    - name: Install Istio Base
      kubernetes.core.helm:
        name: istio-base
        chart_ref: istio/base
        release_namespace: istio-system
        create_namespace: true
        wait: true
        timeout: 10m0s

    - name: Install Istiod
      kubernetes.core.helm:
        name: istiod
        chart_ref: istio/istiod
        release_namespace: istio-system
        wait: true
        timeout: 10m0s
        values:
          global:
            platform: gke

    - name: Install NGINX Ingress Controller
      kubernetes.core.helm:
        name: ingress-nginx
        chart_ref: ingress-nginx/ingress-nginx
        release_namespace: ingress-nginx
        create_namespace: true
        wait: true
        timeout: 10m0s

Why Ansible instead of a shell script?

A shell script with eight sequential helm install commands would work once but fails on reruns. Ansible tasks are idempotent. If a chart is already installed, Ansible skips it. If a run fails midway, you rerun the same command and it picks up where it left off.

This matters in production where cluster bootstrapping is often repeated across environments (dev, staging, prod) and must be version-controlled.

Run the playbook to configure the cluster

With both files saved, a single command bootstraps the entire cluster. This will take several minutes as each Helm chart installs and waits for pods to become ready.

• Run the playbook by executing this command from the ansible/ directory:

ansible-playbook -i inventory.yml playbook.yml

What should I see?

Ansible prints each task name as it runs. You will see changed next to each Helm install task on the first run. The full run typically takes 10-15 minutes because some charts (SonarQube, Jenkins) need time to pull images and start.

At the end, Ansible prints a PLAY RECAP line showing ok and changed counts. Zero failed tasks means success.

Seeing failures during the playbook run?

If a task fails with a timeout, your cluster nodes may be pulling large images. Rerun the same ansible-playbook command. Ansible skips tasks that already succeeded and retries only the failed ones.

If you see an error about the kubernetes.core collection not being found, install it by running:

ansible-galaxy collection install kubernetes.core

Help me troubleshoot my Ansible playbook failures

Verify all pods are running across namespaces

The playbook's wait: true flag confirms each chart's pods started. Now you will verify the full picture across every namespace.

• Check all pods across all namespaces by running:

kubectl get pods --all-namespaces

You should see pods in the Running or Completed state across these namespaces: jenkins, sonarqube, argocd, monitoring, kyverno, falco, istio-system, and ingress-nginx.

What should healthy output look like?

Every row should show 1/1 or 2/2 in the READY column and Running in the STATUS column. Some pods (like Helm hook jobs) show Completed which is normal.

If any pod shows Pending for more than a few minutes, your nodes may be out of resources. Run kubectl describe pod <pod-name> -n <namespace> to check for scheduling issues.

• Verify Helm releases are all deployed by running:

helm list --all-namespaces

You should see all releases with STATUS: deployed in the output.

Pods stuck in CrashLoopBackOff?

SonarQube needs approximately 3-4GB of RAM. If it keeps restarting, check that your 3x e2-standard-2 nodes (24GB total) have enough free memory after all other charts are running.

Run kubectl top nodes to check resource usage. If nodes are saturated, you can temporarily scale down kube-prometheus-stack resource requests in the playbook values.

Help me diagnose pod issues on my GKE cluster

Your cluster is fully bootstrapped. Every tool in the DevSecOps platform is installed and running. Next up, you will build the Flask microservice that flows through this entire pipeline.

Build the Flask Microservice

Your cluster is running and all the security and observability tooling is deployed. Now you need an application to push through the pipeline.

In this step, you will build a Python Flask microservice with health checks and Prometheus instrumentation, containerize it with security best practices, and push it to Artifact Registry.

In this step, get ready to:

• Create a Flask app with health, API, and metrics endpoints.
• Write a secure Dockerfile with multi-stage build and distroless base.
• Build and push the container image to GCP Artifact Registry.

Create the Flask application with Prometheus instrumentation

Every production microservice needs at least two things: a health endpoint for Kubernetes liveness probes, and a metrics endpoint for Prometheus to scrape.

• Create the app/ directory inside your project folder by running this command:

mkdir -p app

• Create the requirements file by running:

cat > app/requirements.txt << 'EOF'
flask
prometheus-flask-instrumentator
EOF

What are these dependencies?

The flask package is a lightweight Python web framework. The prometheus-flask-instrumentator package automatically instruments your Flask routes and exposes a /metrics endpoint that Prometheus can scrape.

• Create the Flask application file by running:

cat > app/app.py << 'EOF'
from flask import Flask, jsonify
from prometheus_flask_instrumentator import Instrumentator

# Create the Flask application
app = Flask(__name__)

# Instrument the app and expose /metrics for Prometheus
Instrumentator().instrument(app).expose(app)

@app.route('/health')
def health():
    """Health check endpoint for Kubernetes probes."""
    return jsonify({"status": "healthy"})

@app.route('/api/items')
def get_items():
    """API endpoint returning a list of items."""
    items = [
        {"id": 1, "name": "item-1"},
        {"id": 2, "name": "item-2"},
        {"id": 3, "name": "item-3"}
    ]
    return jsonify(items)

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)
EOF

What does this code do?

• Instrumentator().instrument(app).expose(app) hooks into every Flask route to measure request duration, then exposes those metrics at /metrics for Prometheus to scrape.
• /health returns a JSON status. Kubernetes will hit this endpoint with liveness and readiness probes to determine if the pod is alive.
• /api/items is a sample API endpoint that returns mock data. This gives the pipeline a real route to test against with OWASP ZAP later.

• Verify the files exist by running:

ls app/

You should see app.py and requirements.txt in the output.

Don't see the files?

Make sure you ran the mkdir -p app command first. If you see a "No such file or directory" error, the parent directory was not created.

help me debug why my app directory was not created

Write a secure Dockerfile

A production container should run as a non-root user and include only the runtime dependencies. Multi-stage builds keep the final image small by separating the build environment from the runtime.

• Create the Dockerfile by running:

cat > app/Dockerfile << 'EOF'
# Stage 1: Build - install Python dependencies
FROM python:3.12-slim AS builder

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt

# Stage 2: Runtime - use distroless base for minimal attack surface
FROM gcr.io/distroless/python3-debian12

# Copy installed packages from the build stage
COPY --from=builder /install /usr/local

# Copy application code
WORKDIR /app
COPY app.py .

# Run as non-root user (distroless default is nonroot UID 65534)
USER nonroot

# Expose the application port
EXPOSE 5000

# Start the Flask app
ENTRYPOINT ["python3", "app.py"]
EOF

What makes this Dockerfile secure?

• Multi-stage build: The first stage (builder) installs pip packages. The second stage copies only the installed packages, leaving behind pip, compilers, and other build tools that could be exploited.
• Distroless base: gcr.io/distroless/python3-debian12 has no shell, no package manager, and no unnecessary system utilities. This drastically reduces the attack surface.
• Non-root user: The USER nonroot directive ensures the container process cannot modify system files even if compromised.

Seeing a build error later?

If docker build fails with a permissions error or missing module, check that the COPY --from=builder path matches where pip installed packages. The --prefix=/install flag in the builder stage writes packages to /install, which the runtime stage copies to /usr/local.

help me fix my multi-stage Docker build

Build and push the container image to Artifact Registry

Your Artifact Registry repository (devsecops-registry) was created by OpenTofu in Step 2. Now you will authenticate Docker to push images there.

• Configure Docker to authenticate with your Artifact Registry region by running:

gcloud auth configure-docker [[GCP_REGION="us-central1"]]-docker.pkg.dev

What does this command do?

This registers gcloud as a Docker credential helper for your Artifact Registry hostname. Docker will automatically use your GCP credentials when pushing or pulling images from that registry.

• Build the Docker image by running:

docker build -t [[GCP_REGION="us-central1"]]-docker.pkg.dev/[[PROJECT_ID="your-gcp-project-id"]]/devsecops-registry/flask-app:latest app/

• Push the image to Artifact Registry by running:

docker push [[GCP_REGION="us-central1"]]-docker.pkg.dev/[[PROJECT_ID="your-gcp-project-id"]]/devsecops-registry/flask-app:latest

You should see layer uploads progressing and a final digest hash confirming the push succeeded.

Push denied or authentication error?

• Verify you ran gcloud auth configure-docker with the correct region hostname matching your Artifact Registry location.
• Confirm your service account has the Artifact Registry Writer role. This was set up by OpenTofu in Step 2.
• If you see a "not found" error, double-check that the repository name is devsecops-registry and matches what OpenTofu created.

help me fix Docker push authentication to GCP Artifact Registry

• Create the SonarQube project configuration file in the project root by running:

cat > sonar-project.properties << 'EOF'
# SonarQube project configuration
sonar.projectKey=flask-app
sonar.projectName=Flask Microservice
sonar.sources=app/
EOF

Why create this file now?

The sonar-project.properties file tells SonarQube where to find your source code and what to call the project. Jenkins will use this file in the SAST scanning stage you set up in Step 6.

• Verify the image exists in Artifact Registry by running:

gcloud artifacts docker images list [[GCP_REGION="us-central1"]]-docker.pkg.dev/[[PROJECT_ID="your-gcp-project-id"]]/devsecops-registry

You should see flask-app with the latest tag listed in the output.

✔️ Awesome, I've got everything!

Great. Make sure you have saved all your files and confirmed the image appears in Artifact Registry.

ⓧ I'd like to double check the full code

Here is the full content for each file created in this step:

app/requirements.txt

flask
prometheus-flask-instrumentator

app/app.py

from flask import Flask, jsonify
from prometheus_flask_instrumentator import Instrumentator

# Create the Flask application
app = Flask(__name__)

# Instrument the app and expose /metrics for Prometheus
Instrumentator().instrument(app).expose(app)

@app.route('/health')
def health():
    """Health check endpoint for Kubernetes probes."""
    return jsonify({"status": "healthy"})

@app.route('/api/items')
def get_items():
    """API endpoint returning a list of items."""
    items = [
        {"id": 1, "name": "item-1"},
        {"id": 2, "name": "item-2"},
        {"id": 3, "name": "item-3"}
    ]
    return jsonify(items)

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

app/Dockerfile

# Stage 1: Build - install Python dependencies
FROM python:3.12-slim AS builder

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt

# Stage 2: Runtime - use distroless base for minimal attack surface
FROM gcr.io/distroless/python3-debian12

# Copy installed packages from the build stage
COPY --from=builder /install /usr/local

# Copy application code
WORKDIR /app
COPY app.py .

# Run as non-root user (distroless default is nonroot UID 65534)
USER nonroot

# Expose the application port
EXPOSE 5000

# Start the Flask app
ENTRYPOINT ["python3", "app.py"]

sonar-project.properties

# SonarQube project configuration
sonar.projectKey=flask-app
sonar.projectName=Flask Microservice
sonar.sources=app/

Your microservice is containerized and stored in Artifact Registry. Next up, you will wire it into a Jenkins CI/CD pipeline with automated security gates.

Set Up Jenkins CI/CD Pipeline

Your cluster is fully bootstrapped with all the DevSecOps tooling running. Now you need to wire them together into an automated pipeline.

In this step, you will configure Jenkins to orchestrate the entire build-test-deploy lifecycle. Every code change will pass through security gates before it can reach production.

In this step, get ready to:

• Access the Jenkins UI and configure credentials for GCP, SonarQube, and Artifact Registry.
• Create a declarative Jenkinsfile with Build, SAST, Container Scan, Push, and Deploy stages.
• Run the pipeline and verify all security gates pass.

Access the Jenkins UI and configure credentials

Jenkins is running inside your cluster as a pod. To access its web UI from your local machine, you need to forward a local port to the Jenkins service.

• Retrieve the Jenkins admin password by running this command:

kubectl exec --namespace jenkins -it svc/jenkins -c jenkins -- /bin/cat /run/secrets/additional/chart-admin-password && echo

What does this command do?

The Jenkins Helm chart stores the admin password inside the running container at a secrets path. This command reads it directly from the pod and prints it to your terminal.

Copy the output. You will paste it into the login page shortly.

• Start a port-forward to the Jenkins service by running:

kubectl --namespace jenkins port-forward svc/jenkins 8080:8080

What does this command do?

This tunnels your local port 8080 to port 8080 on the Jenkins service inside the cluster. The tunnel stays active as long as the command runs. Leave this terminal open.

• Open a browser and navigate to http://localhost:8080.
• Log in with username admin and the password you copied earlier.

You should see the Jenkins dashboard with the main menu on the left side.

Can't reach Jenkins?

• Make sure the port-forward command is still running in your other terminal. If the pod restarted, run the port-forward command again.
• Confirm the Jenkins pod is healthy by running kubectl get pods -n jenkins. The pod should show Running status.

Help me troubleshoot Jenkins access on my GKE cluster.

Now you will add three sets of credentials that the pipeline needs to authenticate with external services.

• Click Manage Jenkins in the left sidebar.
• Click Credentials.
• Click (global) under the Stores scoped to Jenkins section.
• Click Add Credentials.

Add the GCP service account key first.

• Set Kind to Secret file.
• Upload your GCP service account JSON key file.
• Set ID to gcp-sa-key.
• Click Create.

Next, add the SonarQube authentication token.

• Click Add Credentials again.
• Set Kind to Secret text.
• Paste the SonarQube token you generated from the SonarQube UI into the Secret field.
• Set ID to sonarqube-token.
• Click Create.

How do I generate a SonarQube token?

In a separate terminal, port-forward the SonarQube service with kubectl port-forward svc/sonarqube-sonarqube 9000:9000 -n sonarqube. Open http://localhost:9000, log in with admin/admin (you will be prompted to change the password on first login), then navigate to My Account > Security > Generate Tokens.

Finally, add the Artifact Registry credential.

• Click Add Credentials again.
• Set Kind to Secret file.
• Upload the same GCP service account JSON key file (it has Artifact Registry Writer permissions).
• Set ID to artifact-registry-key.
• Click Create.

You should now see three credentials listed: gcp-sa-key, sonarqube-token, and artifact-registry-key.

Create the Jenkinsfile with security gate stages

A Jenkinsfile defines your entire pipeline as code. Each stage acts as a gate. If any security scan fails, the pipeline stops and the vulnerable image never reaches production.

• Create a file called Jenkinsfile in your project root directory by running:

touch Jenkinsfile

• Open Jenkinsfile in your editor and add the pipeline definition:

pipeline {
    agent any

    environment {
        PROJECT_ID = credentials('gcp-sa-key')
        SONAR_TOKEN = credentials('sonarqube-token')
        REGISTRY = 'us-central1-docker.pkg.dev'
        IMAGE_NAME = "${REGISTRY}/${env.GCP_PROJECT}/devsecops-registry/flask-app"
        IMAGE_TAG = "${env.BUILD_NUMBER}"
    }

    stages {
        // Stage 1: Build the Docker image
        stage('Build') {
            steps {
                sh 'docker build -t ${IMAGE_NAME}:${IMAGE_TAG} ./app'
            }
        }

        // Stage 2: Static Application Security Testing with SonarQube
        stage('SAST - SonarQube') {
            steps {
                withSonarQubeEnv('SonarQube') {
                    sh '''
                        sonar-scanner \
                          -Dsonar.projectKey=flask-app \
                          -Dsonar.sources=app/ \
                          -Dsonar.host.url=http://sonarqube-sonarqube.sonarqube.svc.cluster.local:9000 \
                          -Dsonar.token=${SONAR_TOKEN}
                    '''
                }
            }
            post {
                always {
                    // Wait for SonarQube quality gate result
                    waitForQualityGate abortPipeline: true
                }
            }
        }

        // Stage 3: Container image scan with Trivy
        stage('Container Scan - Trivy') {
            steps {
                sh '''
                    trivy image \
                      --exit-code 1 \
                      --severity CRITICAL,HIGH \
                      ${IMAGE_NAME}:${IMAGE_TAG}
                '''
            }
        }

        // Stage 4: Push image to Artifact Registry
        stage('Push Image') {
            steps {
                sh '''
                    cat ${PROJECT_ID} | docker login -u _json_key --password-stdin https://${REGISTRY}
                    docker push ${IMAGE_NAME}:${IMAGE_TAG}
                    docker tag ${IMAGE_NAME}:${IMAGE_TAG} ${IMAGE_NAME}:latest
                    docker push ${IMAGE_NAME}:latest
                '''
            }
        }

        // Stage 5: Deploy to Kubernetes
        stage('Deploy to K8s') {
            steps {
                sh '''
                    kubectl set image deployment/flask-app \
                      flask-app=${IMAGE_NAME}:${IMAGE_TAG} \
                      -n default
                '''
            }
        }
    }

    post {
        failure {
            echo 'Pipeline failed - security gate blocked deployment'
        }
        success {
            echo 'All security gates passed - deployment complete'
        }
    }
}

What does this pipeline do?

• Build: builds the Docker image from the app directory.
• SAST - SonarQube: runs static analysis on the Python source code. The waitForQualityGate step aborts the pipeline if SonarQube detects critical or high-severity issues.
• Container Scan - Trivy: scans the built image for vulnerabilities. The --exit-code 1 flag causes Trivy to fail the stage when CRITICAL or HIGH vulnerabilities are found.
• Push Image: authenticates to Artifact Registry using the GCP service account key and pushes the tagged image.
• Deploy to K8s: updates the running deployment with the new image tag using kubectl set image.

Why Jenkins over GitHub Actions?

Jenkins runs identically whether it is on your local cluster, in a data center, or in the cloud. Enterprise DevSecOps teams choose Jenkins because the Jenkinsfile is portable across environments and Jenkins has native integrations with SonarQube, Trivy, and Kubernetes.

GitHub Actions ties your pipeline to GitHub's infrastructure. Jenkins gives you full control over the execution environment.

• Save the Jenkinsfile.

✔️ Awesome, I've got everything!

Great. Make sure you have saved the file before moving on.

ⓧ I'd like to double check the full code

pipeline {
    agent any

    environment {
        PROJECT_ID = credentials('gcp-sa-key')
        SONAR_TOKEN = credentials('sonarqube-token')
        REGISTRY = 'us-central1-docker.pkg.dev'
        IMAGE_NAME = "${REGISTRY}/${env.GCP_PROJECT}/devsecops-registry/flask-app"
        IMAGE_TAG = "${env.BUILD_NUMBER}"
    }

    stages {
        // Stage 1: Build the Docker image
        stage('Build') {
            steps {
                sh 'docker build -t ${IMAGE_NAME}:${IMAGE_TAG} ./app'
            }
        }

        // Stage 2: Static Application Security Testing with SonarQube
        stage('SAST - SonarQube') {
            steps {
                withSonarQubeEnv('SonarQube') {
                    sh '''
                        sonar-scanner \
                          -Dsonar.projectKey=flask-app \
                          -Dsonar.sources=app/ \
                          -Dsonar.host.url=http://sonarqube-sonarqube.sonarqube.svc.cluster.local:9000 \
                          -Dsonar.token=${SONAR_TOKEN}
                    '''
                }
            }
            post {
                always {
                    // Wait for SonarQube quality gate result
                    waitForQualityGate abortPipeline: true
                }
            }
        }

        // Stage 3: Container image scan with Trivy
        stage('Container Scan - Trivy') {
            steps {
                sh '''
                    trivy image \
                      --exit-code 1 \
                      --severity CRITICAL,HIGH \
                      ${IMAGE_NAME}:${IMAGE_TAG}
                '''
            }
        }

        // Stage 4: Push image to Artifact Registry
        stage('Push Image') {
            steps {
                sh '''
                    cat ${PROJECT_ID} | docker login -u _json_key --password-stdin https://${REGISTRY}
                    docker push ${IMAGE_NAME}:${IMAGE_TAG}
                    docker tag ${IMAGE_NAME}:${IMAGE_TAG} ${IMAGE_NAME}:latest
                    docker push ${IMAGE_NAME}:latest
                '''
            }
        }

        // Stage 5: Deploy to Kubernetes
        stage('Deploy to K8s') {
            steps {
                sh '''
                    kubectl set image deployment/flask-app \
                      flask-app=${IMAGE_NAME}:${IMAGE_TAG} \
                      -n default
                '''
            }
        }
    }

    post {
        failure {
            echo 'Pipeline failed - security gate blocked deployment'
        }
        success {
            echo 'All security gates passed - deployment complete'
        }
    }
}

Run the pipeline and verify all security gates pass

With the Jenkinsfile saved and credentials configured, you can now create and trigger the pipeline job in the Jenkins UI.

• Return to the Jenkins dashboard in your browser.
• Click New Item in the left sidebar.
• Enter devsecops-pipeline as the item name.
• Select Pipeline as the project type.
• Click OK.

Now configure the pipeline to read from your Jenkinsfile.

• Scroll down to the Pipeline section.
• Change the Definition dropdown to Pipeline script from SCM.
• Set SCM to Git.
• Enter your Git repository URL in the Repository URL field.
• Set Branch Specifier to */main.
• Leave Script Path as Jenkinsfile.
• Click Save.

• Click Build Now in the left sidebar to trigger the pipeline.

The pipeline will execute each stage sequentially. You can watch the progress by clicking on the build number in the Build History section, then clicking Console Output.

What should I see?

Each stage should show a green checkmark in the pipeline visualization. The Console Output will show SonarQube analysis results, Trivy scan output with vulnerability counts, and the Docker push confirmation.

If any stage fails (a red X), the pipeline halts. This means a security gate caught an issue and blocked the deployment from reaching production.

Pipeline build failed?

• If the SAST stage fails, check that the SonarQube server is reachable from inside the cluster. Verify the pod is running with kubectl get pods -n sonarqube.
• If the Container Scan stage fails with vulnerabilities, this means Trivy found CRITICAL or HIGH severity issues in your base image. Review the Trivy output and consider updating the base image in your Dockerfile.
• If the Push Image stage fails, confirm the gcp-sa-key credential has Artifact Registry Writer permissions.

Help me debug my Jenkins pipeline failure on GKE.

Your CI/CD pipeline is now enforcing security at every stage. Next up, you will configure SonarQube quality gates and add OWASP ZAP dynamic scanning to catch vulnerabilities that static analysis misses.

Integrate Security Scanning

Your Jenkins pipeline already builds and deploys the Flask app, but without automated security gates, vulnerable code can slip into production undetected.

In this step, you'll configure three layers of security scanning that act as automated gatekeepers. Each gate targets a different attack surface: source code vulnerabilities (SAST), container image flaws (container scanning), and live application weaknesses (DAST).

In this step, get ready to:

• Configure SonarQube with a quality gate that blocks critical and high-severity issues.
• Update the Trivy container scan stage with a severity gate that fails the pipeline on CRITICAL or HIGH findings.
• Add an OWASP ZAP DAST stage that scans the live application for vulnerabilities.

Configure the SonarQube quality gate

A quality gate defines the conditions your code must meet before it can proceed through the pipeline. If any condition fails, the gate blocks the build.

• Access the SonarQube UI by port-forwarding to the SonarQube service:

kubectl port-forward svc/sonarqube-sonarqube 9000:9000 -n sonarqube

• Open http://localhost:9000 in your browser.
• Log in with the default credentials: username admin and password admin.
• Click Quality Gates in the top navigation bar.
• Click Create to create a new custom quality gate.
• Name it DevSecOps Gate.

Now add conditions that block the pipeline when critical or high-severity issues are detected.

• Below the Conditions section, click Unlock editing.
• Click Add Condition.
• In Where? select On Overall Code.
• In Quality Gate fails when, select the Security Rating metric.
• Set the value to A (the gate fails when Security Rating is worse than A).
• Repeat the process to add a Reliability Rating condition, also set to fail when worse than A.
• Click the Projects tab and associate the flask-app project with this gate.

What does this quality gate do?

The Security Rating and Reliability Rating conditions ensure the gate fails whenever SonarQube detects issues with Critical or High severity. A rating of A means zero Critical or High issues exist in the scanned code.

When the Jenkins pipeline calls the SonarQube scanner, it checks the gate status after analysis. If the gate is in a failed state, the pipeline stage fails and halts the build.

Next, update the sonar-project.properties file to match the project configuration in the SonarQube UI.

• Update sonar-project.properties in the root of your project repository with this content:

# SonarQube project configuration
sonar.projectKey=flask-app
sonar.projectName=Flask Microservice
sonar.sources=app/

# Quality gate will block the pipeline on critical/high issues
sonar.qualitygate.wait=true

What does this configuration do?

The sonar.projectKey links this properties file to the flask-app project you created in SonarQube. The sonar.sources tells the scanner which directory to analyze.

Setting sonar.qualitygate.wait=true makes the scanner wait for the quality gate computation to finish and return a failure exit code if the gate does not pass. This is what makes the Jenkins SAST stage actually block on findings.

• Commit and push the updated sonar-project.properties to your repository.

Quality gate not failing as expected?

Make sure you associated the flask-app project with the custom DevSecOps Gate quality gate. The default Sonar way gate may not have the strict conditions you need.

Help me debug my quality gate configuration

Configure the Trivy container scan severity gate

The Trivy stage already exists in your Jenkinsfile from the previous step. Now you'll configure it to use the --severity and --exit-code flags so the pipeline halts when CRITICAL or HIGH vulnerabilities are found in the container image.

• In your Jenkinsfile, find the Container Scan stage and update the trivy command to include the severity gate:

stage('Container Scan') {
    steps {
        // Scan the built image for CRITICAL and HIGH vulnerabilities
        // --exit-code 1 causes the pipeline to fail if any are found
        sh "trivy image --severity CRITICAL,HIGH --exit-code 1 --format table ${IMAGE_NAME}:${BUILD_NUMBER}"
    }
}

What do these flags do?

The --severity CRITICAL,HIGH flag tells Trivy to only report vulnerabilities at those two severity levels. Lower-severity findings are filtered out of the results.

The --exit-code 1 flag changes Trivy's default behavior. Without it, Trivy always exits with code 0 even when vulnerabilities are found. With this flag, any CRITICAL or HIGH finding causes a non-zero exit, which Jenkins interprets as a stage failure.

• Save the Jenkinsfile and commit the change.

Trivy scan timing out?

The first run downloads the vulnerability database, which can take a few minutes. Subsequent runs use the cached database and complete much faster.

Help me troubleshoot Trivy in my Jenkins pipeline

Add the OWASP ZAP DAST stage

SAST and container scanning catch issues in source code and images. DAST (Dynamic Application Security Testing) finds vulnerabilities that only appear at runtime, like missing security headers or injectable endpoints.

• In your Jenkinsfile, add a new DAST stage after the Deploy to K8s stage. This runs the OWASP ZAP baseline scan against the deployed Flask application:

stage('DAST') {
    steps {
        // Run OWASP ZAP baseline scan against the deployed app
        // The baseline scan spiders for 1 minute then runs passive scanning
        sh "docker run -t ghcr.io/zaproxy/zaproxy:stable zap-baseline.py -t http://flask-app-service.default.svc.cluster.local:5000"
    }
}

What does the ZAP baseline scan do?

The ZAP baseline scan runs a spider against the target URL for 1 minute, then performs passive scanning on the responses it collects. It is designed for CI/CD pipelines and is safe to run against production applications because it only observes traffic patterns without actively injecting payloads.

The scan detects issues like missing Content-Security-Policy headers, insecure cookie flags, information disclosure in error pages, and other configuration weaknesses that only manifest at runtime.

• Save and commit the Jenkinsfile with the new DAST stage.

Your final Jenkinsfile should now have these stages in order: Build, SAST, Container Scan, Push Image, Deploy to K8s, DAST.

• Trigger a new pipeline run in Jenkins by clicking Build Now on your pipeline job.
• Watch the pipeline stages complete. All three security gates (SAST, Container Scan, DAST) should pass with a green status.

What should I see?

In the Jenkins pipeline view, every stage should show a green checkmark. The SAST stage passes because your Flask app has no Critical or High severity issues. The Container Scan stage passes because the distroless base image has minimal vulnerabilities. The DAST stage passes because the ZAP baseline scan found no critical runtime issues.

DAST stage failing?

Make sure the Deploy to K8s stage completes before DAST runs. The ZAP scan needs a live application to target. Check that the service URL is reachable from inside the Jenkins pod by running a curl test against it.

Help me debug my OWASP ZAP DAST stage

✔️ Awesome, I've got everything!

Great. Double check you've saved and committed all files to your repository.

ⓧ I'd like to double check the full Jenkinsfile

Your Jenkinsfile should include all these stages:

pipeline {
    agent any

    environment {
        IMAGE_NAME = "us-central1-docker.pkg.dev/${PROJECT_ID}/devsecops-registry/flask-app"
    }

    stages {
        stage('Build') {
            steps {
                // Build the Docker image
                sh "docker build -t ${IMAGE_NAME}:${BUILD_NUMBER} ./app"
            }
        }

        stage('SAST') {
            steps {
                // Run SonarQube analysis with quality gate check
                withSonarQubeEnv('SonarQube') {
                    sh "sonar-scanner"
                }
                // Wait for quality gate result - fails pipeline if gate fails
                waitForQualityGate abortPipeline: true
            }
        }

        stage('Container Scan') {
            steps {
                // Scan the built image for CRITICAL and HIGH vulnerabilities
                // --exit-code 1 causes the pipeline to fail if any are found
                sh "trivy image --severity CRITICAL,HIGH --exit-code 1 --format table ${IMAGE_NAME}:${BUILD_NUMBER}"
            }
        }

        stage('Push Image') {
            steps {
                // Push the scanned image to Artifact Registry
                sh "docker push ${IMAGE_NAME}:${BUILD_NUMBER}"
            }
        }

        stage('Deploy to K8s') {
            steps {
                // Apply Kubernetes manifests to deploy the app
                sh "kubectl apply -f k8s/"
                sh "kubectl set image deployment/flask-app flask-app=${IMAGE_NAME}:${BUILD_NUMBER}"
            }
        }

        stage('DAST') {
            steps {
                // Run OWASP ZAP baseline scan against the deployed app
                // The baseline scan spiders for 1 minute then runs passive scanning
                sh "docker run -t ghcr.io/zaproxy/zaproxy:stable zap-baseline.py -t http://flask-app-service.default.svc.cluster.local:5000"
            }
        }
    }
}

Your pipeline now enforces three layers of automated security scanning. No code reaches production without passing SAST analysis, container vulnerability scanning, and runtime DAST checks. Next up, you'll set up ArgoCD to manage deployments through GitOps.

Deploy with ArgoCD GitOps

Your Jenkins pipeline now scans code for vulnerabilities and blocks bad builds. But how do your approved images actually reach the cluster?

In this step, you will define your application's desired state as Kubernetes manifests stored in Git, then let ArgoCD continuously sync that state to your cluster. This is the GitOps model: Git becomes the single source of truth, and the cluster self-corrects whenever someone introduces drift.

In this step, get ready to:

• Create Kubernetes manifests for the Flask app (Deployment, Service, HPA, PDB).
• Configure an ArgoCD Application with auto-sync and self-heal.
• Demonstrate drift detection by manually changing replicas and watching ArgoCD self-correct.

Create Kubernetes manifests for the Flask app

ArgoCD watches a Git repository for manifest changes. You need a dedicated directory containing all the Kubernetes resources your Flask app requires: a Deployment to run pods, a Service to expose them, an HPA for auto-scaling, and a PDB for availability guarantees.

• Create the k8s/ directory in your project root by running this command:

mkdir -p k8s

• Create the Deployment manifest by running:

cat > k8s/deployment.yaml << 'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flask-app
  namespace: flask-app
  labels:
    app: flask-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: flask-app
  template:
    metadata:
      labels:
        app: flask-app
    spec:
      containers:
      - name: flask-app
        image: us-central1-docker.pkg.dev/[[PROJECT_ID="your-gcp-project-id"]]/devsecops-registry/flask-app:latest
        ports:
        - containerPort: 5000
        # Liveness probe restarts unhealthy containers
        livenessProbe:
          httpGet:
            path: /health
            port: 5000
          initialDelaySeconds: 10
          periodSeconds: 15
        # Readiness probe controls traffic routing
        readinessProbe:
          httpGet:
            path: /health
            port: 5000
          initialDelaySeconds: 5
          periodSeconds: 10
        # Resource requests and limits for scheduling and HPA
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
EOF

What does this manifest do?

The Deployment defines 2 replicas of your Flask app running in the flask-app namespace.

The livenessProbe hits /health every 15 seconds. If it fails, Kubernetes restarts the container. The readinessProbe does the same but controls whether traffic is routed to the pod.

Resource requests guarantee the pod gets scheduled on a node with enough capacity. Limits prevent a single pod from consuming the entire node.

• Create the Service manifest by running:

cat > k8s/service.yaml << 'EOF'
apiVersion: v1
kind: Service
metadata:
  name: flask-app
  namespace: flask-app
  labels:
    app: flask-app
spec:
  selector:
    app: flask-app
  ports:
  - port: 5000
    targetPort: 5000
    protocol: TCP
  type: ClusterIP
EOF

What does this Service do?

The Service creates a stable internal DNS name (flask-app.flask-app.svc.cluster.local) that other services use to reach your Flask pods on port 5000. The ClusterIP type keeps it internal to the cluster. External access will come through the NGINX Ingress later.

• Create the HorizontalPodAutoscaler manifest by running:

cat > k8s/hpa.yaml << 'EOF'
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: flask-app-hpa
  namespace: flask-app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: flask-app
  # Minimum and maximum replica boundaries
  minReplicas: 2
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        # Scale up when average CPU exceeds 70%
        averageUtilization: 70
EOF

What does the HPA do?

The HorizontalPodAutoscaler watches the average CPU utilization across all flask-app pods. When it exceeds 70%, Kubernetes adds more replicas (up to 5). When load drops, it scales back down to 2.

• Create the PodDisruptionBudget manifest by running:

cat > k8s/pdb.yaml << 'EOF'
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: flask-app-pdb
  namespace: flask-app
spec:
  # At least 1 pod must remain available during voluntary disruptions
  minAvailable: 1
  selector:
    matchLabels:
      app: flask-app
EOF

What does the PDB do?

A PodDisruptionBudget protects your app during voluntary disruptions like node upgrades or spot VM preemptions. Kubernetes will refuse to evict a pod if doing so would leave fewer than 1 pod available.

• Confirm all four manifests exist by running:

ls k8s/

You should see deployment.yaml, hpa.yaml, pdb.yaml, and service.yaml listed.

• Commit and push these manifests to your Git repository by running:

git add k8s/
git commit -m "Add Kubernetes manifests for Flask app"
git push origin main

Push failing with authentication errors?

Make sure your Git remote URL uses HTTPS with a personal access token, or SSH with a key that has been added to your account. Run git remote -v to check your remote URL.

help me fix Git push authentication errors

Configure the ArgoCD Application with auto-sync

With your manifests in Git, you need to tell ArgoCD where to find them and where to deploy the resulting resources. You do this by creating an Application resource in the argocd namespace.

• Create the ArgoCD Application manifest by running:

cat > k8s/argocd-app.yaml << 'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: flask-app
  namespace: argocd
spec:
  # Project controls which repos and destinations are allowed
  project: default
  source:
    repoURL: [[REPO_URL="https://github.com/YOUR_USERNAME/your-devsecops-repo.git"]]
    targetRevision: HEAD
    path: k8s
  destination:
    server: https://kubernetes.default.svc
    namespace: flask-app
  syncPolicy:
    automated:
      # Prune removes resources from the cluster that are no longer in Git
      prune: true
      # Self-heal reverts manual cluster changes back to the Git state
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
EOF

What does this Application resource do?

The source block tells ArgoCD to watch the k8s/ path in your Git repository at the HEAD revision.

The destination block says deploy everything to the flask-app namespace on the local cluster.

The syncPolicy.automated section enables two critical behaviors. prune: true deletes cluster resources that no longer exist in Git. selfHeal: true reverts any manual changes made directly on the cluster back to what Git declares.

Why GitOps instead of kubectl apply?

With kubectl apply, deployments happen from a developer's machine or a CI runner. There is no record of who deployed what, and cluster state can drift silently from what is in source control.

GitOps flips this model. The cluster continuously pulls its desired state from Git. Every change is a commit with an author, a timestamp, and a diff you can review. Rollbacks become git revert.

• Create the flask-app namespace so ArgoCD can deploy into it:

kubectl create namespace flask-app --dry-run=client -o yaml | kubectl apply -f -

• Apply the ArgoCD Application resource by running:

kubectl apply -f k8s/argocd-app.yaml

• Watch ArgoCD sync your application by running:

kubectl get applications -n argocd

You should see flask-app with a SYNC STATUS of Synced and HEALTH STATUS of Healthy after a minute or two.

• Verify the Flask app pods are running in the target namespace:

kubectl get pods -n flask-app

You should see 2 pods with the flask-app prefix in Running status.

Application stuck in Progressing or OutOfSync?

Run kubectl describe application flask-app -n argocd to see detailed sync status and error messages. Common issues include incorrect repository URLs, private repos without configured credentials, or image pull errors from Artifact Registry.

help me debug my ArgoCD application sync failure

Demonstrate drift detection and self-healing

The real power of GitOps shows when someone makes an unauthorized change directly on the cluster. With selfHeal: true, ArgoCD will detect the drift and revert it automatically.

• Manually scale the deployment to 4 replicas to simulate an unauthorized change:

kubectl scale deployment flask-app --replicas=4 -n flask-app

• Immediately check the replica count to see the manual change:

kubectl get deployment flask-app -n flask-app

You should briefly see 4/4 in the READY column.

• Wait about 30 seconds for ArgoCD to detect the drift, then check again:

sleep 30 && kubectl get deployment flask-app -n flask-app

ArgoCD has reverted the replica count back to 2, matching what your deployment.yaml in Git declares. The cluster self-corrected without any human intervention.

How does self-heal work?

ArgoCD polls the cluster state every 3 minutes by default (configurable). When it detects that a resource's live state differs from the desired state in Git, and selfHeal is enabled, it triggers an automatic sync to bring the cluster back in line.

This means operational mistakes, accidental kubectl edits, and even malicious changes get reverted automatically. The only way to make a lasting change is through a Git commit.

Replicas not reverting to 2?

Make sure your ArgoCD Application has selfHeal: true in the syncPolicy.automated section. Run kubectl get application flask-app -n argocd -o yaml | grep -A 3 automated to verify. If selfHeal is missing or set to false, update your k8s/argocd-app.yaml and re-apply it.

help me fix ArgoCD self-heal not reverting my manual changes

Your application is now deployed and managed entirely through Git. Any change to the manifests in the k8s/ directory triggers an automatic sync, and any manual cluster modifications are reverted. Next up, you will layer Istio service mesh on top of this deployment to get mTLS encryption between services.

Configure Istio Service Mesh

In the previous step, you deployed your Flask app via ArgoCD with auto-sync and drift detection.

Your application is deployed and accessible, but all traffic between services flows as plain text. In a production environment, any pod in the cluster could intercept or spoof traffic to your Flask app. Istio solves this by injecting a sidecar proxy into each pod that encrypts all communication with mutual TLS.

In this step, get ready to:

• Enable Istio sidecar injection on the Flask app namespace.
• Enforce strict mTLS so all traffic is encrypted.
• Configure retry policies and circuit breaking for resilient networking.

Enable Istio sidecar injection on the application namespace

Istio's sidecar proxy (Envoy) intercepts all inbound and outbound traffic for a pod. To inject these sidecars automatically, you label the namespace with istio-injection=enabled. Any pods created in that namespace will then get a sidecar container alongside the application container.

• Label the flask-app namespace for Istio sidecar injection by running this command:

kubectl label namespace flask-app istio-injection=enabled --overwrite

• Restart the existing pods so they pick up the sidecar by running:

kubectl rollout restart deployment -n flask-app

• Verify the sidecar injection by checking that pods show 2/2 in the READY column by running:

kubectl get pods -n flask-app

You should see each pod with 2/2 containers ready. The second container is the istio-proxy sidecar.

What does 2/2 mean?

The READY column shows running containers / total containers. Before Istio injection, your pods had 1/1 (just the Flask app). Now they have 2/2 because Istio added the Envoy sidecar proxy alongside your application container.

Pods still showing 1/1?

Make sure the namespace label was applied correctly by running kubectl get namespace flask-app --show-labels. Look for istio-injection=enabled in the output.

If the label is there but pods still show 1/1, the rollout restart may not have completed. Wait 30 seconds and check again.

Help me troubleshoot why my Istio sidecar injection isn't working on GKE.

Configure strict mTLS PeerAuthentication policy

By default, Istio uses PERMISSIVE mode which accepts both plain text and mTLS traffic. For production security, you need STRICT mode which rejects any unencrypted connections to your service.

• Create the PeerAuthentication manifest by running:

cat <<EOF > k8s/istio-peer-auth.yaml
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: flask-app
spec:
  mtls:
    mode: STRICT
EOF

What does this manifest do?

The PeerAuthentication resource tells Istio how to handle incoming connections. Setting mode: STRICT means every connection to any pod in the flask-app namespace MUST present a valid mTLS certificate. Plain text traffic is blocked.

Because there is no selector field, the policy applies to all workloads in the namespace.

• Apply the PeerAuthentication policy by running:

kubectl apply -f k8s/istio-peer-auth.yaml

You should see peerauthentication.security.istio.io/default created in the output.

Seeing a resource not found error?

If you get an error about the PeerAuthentication resource not being recognized, Istio's CRDs may not be installed. Verify Istio is running with kubectl get pods -n istio-system. The istiod pod must be in Running state.

Help me fix my Istio PeerAuthentication error on GKE.

Set up VirtualService with retry policies and circuit breaking

Network failures are inevitable in distributed systems. Istio lets you configure retry policies and circuit breaking at the mesh level, so your application code does not need to handle transient failures itself.

• Create the VirtualService manifest with retry policies by running:

cat <<EOF > k8s/istio-virtual-service.yaml
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: flask-app
  namespace: flask-app
spec:
  hosts:
  - flask-app
  http:
  - route:
    - destination:
        host: flask-app
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: gateway-error,connect-failure,refused-stream
    timeout: 10s
EOF

What does this VirtualService do?

The VirtualService defines traffic routing rules for the flask-app service. The retry policy says: if a request fails, retry up to 3 times with a 2-second timeout per attempt. Retries trigger on gateway errors, connection failures, and refused streams.

The overall timeout: 10s sets a maximum time for the entire request (including retries). If all retries exhaust within 10 seconds, the client receives an error.

• Create the DestinationRule manifest with circuit breaking by running:

cat <<EOF > k8s/istio-destination-rule.yaml
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: flask-app
  namespace: flask-app
spec:
  host: flask-app
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
EOF

What does circuit breaking do?

The outlierDetection section implements circuit breaking. If a pod returns 5 consecutive 5xx errors, Istio ejects it from the load balancing pool for 30 seconds. This prevents one unhealthy pod from degrading the entire service.

The maxEjectionPercent: 50 ensures at least half of your pods remain in the pool at all times. This prevents a cascade where all pods get ejected simultaneously.

• Apply both networking manifests by running:

kubectl apply -f k8s/istio-virtual-service.yaml
kubectl apply -f k8s/istio-destination-rule.yaml

• Verify all Istio resources are applied in the flask-app namespace by running:

kubectl get peerauthentication,virtualservice,destinationrule -n flask-app

You should see all three resources listed: the PeerAuthentication named default, the VirtualService named flask-app, and the DestinationRule named flask-app.

Resources not showing up?

Double-check that you specified namespace: flask-app in each manifest's metadata section. If you accidentally applied to a different namespace, delete and reapply.

Help me debug my Istio VirtualService or DestinationRule configuration.

✔️ Awesome, I've got everything!

Great. Double check you've committed all three YAML files to your Git repo so ArgoCD can sync them.

ⓧ I'd like to double check the full code

Here are the three Istio manifest files:

k8s/istio-peer-auth.yaml

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: flask-app
spec:
  mtls:
    mode: STRICT

k8s/istio-virtual-service.yaml

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: flask-app
  namespace: flask-app
spec:
  hosts:
  - flask-app
  http:
  - route:
    - destination:
        host: flask-app
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: gateway-error,connect-failure,refused-stream
    timeout: 10s

k8s/istio-destination-rule.yaml

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: flask-app
  namespace: flask-app
spec:
  host: flask-app
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

Your Flask app is now protected by Istio's service mesh. All traffic is encrypted with mTLS, failed requests are automatically retried, and unhealthy pods get ejected from the pool. Next up, you'll configure NGINX Ingress to expose your application to external traffic.

Set Up NGINX Ingress

Your Istio service mesh handles internal pod-to-pod traffic with mutual TLS. But there is no way for users outside the cluster to reach your Flask app yet.

In this step, you'll create a Kubernetes Ingress resource that routes external HTTP traffic through the NGINX Ingress Controller (already running from your Ansible bootstrap). You'll also add rate limiting annotations to protect the app from abuse.

In this step, get ready to:

• Retrieve the NGINX Ingress Controller's external LoadBalancer IP.
• Create an Ingress resource with rate limiting annotations.
• Verify external access to the Flask app through the LoadBalancer IP.

Get the NGINX Ingress Controller LoadBalancer IP

On GKE, the NGINX Ingress Controller's Service of type LoadBalancer automatically receives an external IP from Google Cloud. You need this IP to access your app from outside the cluster.

• Retrieve the external IP assigned to the NGINX Ingress Controller by running:

kubectl get service ingress-nginx-controller --namespace=ingress-nginx

What does this command do?

This queries the Kubernetes Service named ingress-nginx-controller in the ingress-nginx namespace. The EXTERNAL-IP column shows the public IP that GKE provisioned for the LoadBalancer.

• Look at the EXTERNAL-IP column in the output. Copy this IP address. You'll use it to verify external access shortly.

EXTERNAL-IP shows pending?

GKE can take 1-2 minutes to provision a LoadBalancer IP. Re-run the command after waiting. If it stays pending, check that the GKE cluster's nodes have external network access and that the ingress-nginx namespace has running pods.

Help me debug why my NGINX Ingress LoadBalancer IP is stuck in pending state.

Create the Ingress resource with rate limiting

An Ingress resource defines routing rules that tell the NGINX Ingress Controller how to forward external traffic to your backend services. You'll add a rate limiting annotation to cap requests at 10 per second per IP, protecting your Flask app from being overwhelmed.

• Create the file k8s/ingress.yaml with the following content:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: flask-app-ingress
  namespace: flask-app
  annotations:
    # Rate limit: max 10 requests per second per client IP
    nginx.ingress.kubernetes.io/limit-rps: "10"
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: flask-app
            port:
              number: 5000

What does this manifest do?

The ingressClassName: nginx tells Kubernetes to route this Ingress through your NGINX Ingress Controller. The nginx.ingress.kubernetes.io/limit-rps: "10" annotation caps each client IP to 10 requests per second per controller replica. Exceeding this limit returns a 503 status code.

The routing rule forwards all traffic hitting / (and any subpath) to the flask-app Service on port 5000.

• Apply the Ingress resource by running:

kubectl apply -f k8s/ingress.yaml

• Verify the Ingress was created by running:

kubectl get ingress -n flask-app

• You should see flask-app-ingress listed with the LoadBalancer's external IP in the ADDRESS column. It may take 30-60 seconds for the address to populate.

ADDRESS column is empty?

The NGINX Ingress Controller needs a few moments to reconcile the new Ingress resource. Wait 30-60 seconds and re-run the kubectl get ingress command. Ensure ingressClassName in your manifest is set to nginx (not a different class name).

Why is my Ingress ADDRESS not populating?

Verify external access through the LoadBalancer IP

Now that the Ingress is routing traffic, you can confirm that external requests reach your Flask app through the LoadBalancer IP.

• Test the health endpoint by running this command (replace the IP with your actual LoadBalancer IP from earlier):

curl http://[[INGRESS_IP="your NGINX Ingress LoadBalancer external IP"]]/health

• You should see a JSON response indicating the app is healthy, such as {"status": "healthy"}.
• Test the API endpoint by running:

curl http://[[INGRESS_IP="your NGINX Ingress LoadBalancer external IP"]]/api/items

• You should see a JSON response with items data from your Flask app.

What is rate limiting doing here?

The limit-rps annotation protects your Flask microservice from being overwhelmed by too many requests from a single source. If a client exceeds 10 requests per second, the NGINX Ingress Controller returns a 503 response instead of forwarding the request to your pods.

The default burst multiplier is 5, so the actual burst allowance is 50 requests before rate limiting kicks in. This gives legitimate traffic room to spike briefly without being blocked.

Getting a connection refused or timeout?

Ensure you're using the correct external IP from the kubectl get service ingress-nginx-controller --namespace=ingress-nginx output. Also verify your Flask app pods are running with kubectl get pods -n flask-app. All pods should show Running status with the Istio sidecar container ready.

Help me debug why I can't reach my Flask app through the NGINX Ingress LoadBalancer IP.

✔️ Awesome, I've got everything!

Great. Double check you've saved k8s/ingress.yaml and that your curl commands returned successful responses.

ⓧ I'd like to double check the full code

Here is the complete k8s/ingress.yaml file:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: flask-app-ingress
  namespace: flask-app
  annotations:
    # Rate limit: max 10 requests per second per client IP
    nginx.ingress.kubernetes.io/limit-rps: "10"
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: flask-app
            port:
              number: 5000

Your Flask app is now accessible from outside the cluster with rate limiting protection in place. Next up, you'll enforce security policies with Kyverno to ensure only compliant workloads can run in your cluster.

Enforce Policies with Kyverno

Your cluster now has traffic flowing securely through Istio and NGINX Ingress. The next challenge is preventing bad configurations from ever reaching production.

In this step, you will deploy Kyverno ClusterPolicies that act as admission controllers. They intercept every resource creation request and block anything that violates your security standards before it touches the cluster.

In this step, get ready to:

• Deploy three Kyverno ClusterPolicies covering privileged containers, resource limits, and image registries.
• Test policy enforcement by deploying a non-compliant pod and confirming Kyverno blocks it.

Deploy a policy to disallow privileged containers

A privileged container has full access to the host's kernel and devices. Allowing one is equivalent to giving root access to the underlying node. This policy blocks any pod that sets privileged: true in its security context.

• Create the policy file by running:

cat <<EOF > k8s/kyverno-disallow-privileged.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-privileged-containers
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: disallow-privileged
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Privileged containers are not allowed."
        pattern:
          spec:
            containers:
              - =(securityContext):
                  =(privileged): "false"
EOF

What does this policy do?

The validationFailureAction: Enforce setting means Kyverno rejects non-compliant pods outright instead of just logging them.

The pattern uses an equality anchor =(securityContext) which means "if this field exists, it must match." If a container sets privileged: true, the pod is denied at admission time.

• Apply the policy to your cluster by running:

kubectl apply -f k8s/kyverno-disallow-privileged.yaml

• Verify the policy is active by running:

kubectl get clusterpolicy disallow-privileged-containers

You should see the policy listed with READY showing True.

Policy not showing as ready?

Make sure the Kyverno controller pods are running in the kyverno namespace. Check with kubectl get pods -n kyverno. If pods are in CrashLoopBackOff, the cluster may need more memory allocated to those pods.

Help me troubleshoot why my Kyverno ClusterPolicy is not becoming ready on GKE.

Deploy a policy to require resource limits

Without resource limits, a single misbehaving container can consume all available CPU or memory on a node, starving other workloads. This policy ensures every container specifies CPU and memory limits.

• Create the resource limits policy by running:

cat <<EOF > k8s/kyverno-require-limits.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: validate-resources
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "CPU and memory resource limits are required for all containers."
        pattern:
          spec:
            containers:
              - resources:
                  limits:
                    memory: "?*"
                    cpu: "?*"
EOF

What does the pattern syntax mean?

The "?*" pattern means "at least one character must be present." The ? matches exactly one alphanumeric character and * matches zero or more. Together they require that the field exists and is non-empty.

This ensures containers cannot be deployed without explicitly setting a memory and CPU limit value.

• Apply the policy by running:

kubectl apply -f k8s/kyverno-require-limits.yaml

• Confirm the policy is enforcing by running:

kubectl get clusterpolicy require-resource-limits

You should see READY as True for this policy as well.

Deploy a policy to restrict image registries

Pulling images from unknown registries introduces supply chain risk. This policy ensures all container images originate from your private Artifact Registry, blocking anything pulled from Docker Hub or other public sources.

• Create the image registry restriction policy by running:

cat <<EOF > k8s/kyverno-restrict-registries.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: validate-registries
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Images must come from the Artifact Registry."
        pattern:
          spec:
            containers:
              - image: "[[AR_REGION="us-central1"]]-docker.pkg.dev/*"
EOF

Why restrict to Artifact Registry only?

Your CI/CD pipeline already pushes scanned, signed images to Artifact Registry. By restricting the cluster to only pull from that registry, you close the loop on supply chain security. No developer can bypass the pipeline by deploying an unscanned image from a public registry.

• Apply the policy by running:

kubectl apply -f k8s/kyverno-restrict-registries.yaml

• Verify all three policies are now active by running:

kubectl get clusterpolicy

You should see all three policies listed with READY showing True.

Existing pods are not affected

Kyverno Enforce mode only blocks new pod creations and updates. Your existing Flask app pods continue running because they were deployed before this policy. If you redeploy them through ArgoCD, they will pass because they already pull from Artifact Registry with proper resource limits.

Test policy enforcement with a non-compliant pod

The real proof that policies work is seeing them reject a bad deployment. You will attempt to create a privileged container and confirm Kyverno blocks it at the admission webhook level.

• Attempt to deploy a privileged pod by running:

kubectl run privileged-test --image=nginx --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"privileged-test","image":"nginx","securityContext":{"privileged":true}}]}}'

You should see an error message from Kyverno indicating the pod was blocked. The output will contain a message about the disallow-privileged-containers policy denying the request.

What does this prove?

Kyverno's admission webhook intercepted the pod creation request before it reached the Kubernetes scheduler. The pod never existed in the cluster.

This is the power of policy as code: security is enforced automatically at the API level, not through manual reviews.

• Confirm no pod was created by running:

kubectl get pod privileged-test

You should see a not found message confirming the pod was never created.

Pod was created instead of blocked?

If the pod was created successfully, your disallow-privileged-containers policy may not be in Enforce mode. Run kubectl get clusterpolicy disallow-privileged-containers -o yaml and check that validationFailureAction is set to Enforce, not Audit. Delete the test pod with kubectl delete pod privileged-test and reapply the corrected policy.

Help me debug why Kyverno is not blocking privileged pods despite having an Enforce policy deployed.

✔️ Awesome, I've got everything!

Great. Double check you have saved all three policy files in your k8s/ directory.

ⓧ I'd like to double check the full code

Here are all three Kyverno policy files for reference.

k8s/kyverno-disallow-privileged.yaml:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-privileged-containers
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: disallow-privileged
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Privileged containers are not allowed."
        pattern:
          spec:
            containers:
              - =(securityContext):
                  =(privileged): "false"

k8s/kyverno-require-limits.yaml:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: validate-resources
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "CPU and memory resource limits are required for all containers."
        pattern:
          spec:
            containers:
              - resources:
                  limits:
                    memory: "?*"
                    cpu: "?*"

k8s/kyverno-restrict-registries.yaml:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: validate-registries
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Images must come from the Artifact Registry."
        pattern:
          spec:
            containers:
              - image: "[[AR_REGION="us-central1"]]-docker.pkg.dev/*"

Your cluster now has automated policy enforcement blocking privileged containers, containers without resource limits, and images from untrusted registries. Next up, you will configure Falco to detect runtime threats that slip past admission control.

Runtime Security with Falco

Your cluster now has policies enforcing security standards at admission time. But what happens after a pod is already running?

Policy engines like Kyverno block non-compliant deployments at the gate. Falco picks up where policies leave off by monitoring runtime behavior inside containers using eBPF. If an attacker gains a shell inside your pod, Falco detects it in real time.

In this step, get ready to:

• Verify Falco is running with the modern_ebpf driver.
• Create a custom Falco rule to detect shell execution inside containers.
• Trigger and verify Falco alerts in the Falcosidekick UI.

Verify Falco is running on your cluster

Falco was installed in Step 3 via Ansible. It uses the modern_ebpf driver, which is built into the Falco binary and requires no kernel module installation on GKE.

• Confirm all Falco pods are running by executing this command:

kubectl get pods -n falco

• Wait for all pods to reach Ready status by running:

kubectl wait pods --for=condition=Ready --all -n falco --timeout=120s

You should see Falco pods (one per node) plus Falcosidekick and Falcosidekick UI pods in a Running state.

• Verify Falco is using the modern_ebpf driver by checking the logs:

kubectl logs -l app.kubernetes.io/name=falco -n falco -c falco | head -20

You should see a log line confirming Falco started with the modern_ebpf engine.

What is the modern_ebpf driver?

Falco intercepts system calls to detect suspicious behavior. The modern_ebpf driver is compiled directly into the Falco binary, so it works on any Linux kernel 5.8+ without installing separate kernel modules. On GKE, this means zero extra setup.

Falco pods not reaching Ready?

Check pod events with kubectl describe pod -n falco for scheduling or resource issues. If the node lacks eBPF support (unlikely on GKE), the logs will mention a driver failure.

Help me debug Falco pods not starting on my GKE cluster.

Create a custom Falco rule to detect shell execution

Falco ships with built-in rules, but writing your own teaches you how runtime detection works. You will create a rule that fires whenever someone spawns a shell (/bin/sh or /bin/bash) inside a container. This is one of the most common indicators of a container breakout or unauthorized access.

• Create the custom rules file by running:

cat > k8s/falco-custom-rules.yaml << 'EOF'
customRules:
  custom-rules.yaml: |-
    - rule: Shell Execution in Container
      desc: Detect shell execution inside a container via execve syscall
      condition: >
        (evt.type in (execve, execveat)) and
        container.id != host and
        (proc.name = sh or proc.name = bash)
      output: >
        Shell executed in container
        (user=%user.name container_id=%container.id
        container_name=%container.name
        shell=%proc.name parent=%proc.pname
        cmdline=%proc.cmdline
        image=%container.image.repository
        k8s_ns=%k8s.ns.name
        k8s_pod=%k8s.pod.name)
      priority: WARNING
      tags: [container, shell, mitre_execution]
EOF

What does this rule do?

The condition field is the detection logic. It checks three things simultaneously:

• The event type is execve or execveat (a process is being spawned).
• The event happens inside a container (not on the host).
• The process name is sh or bash (a shell is being invoked).

The output field defines what information is included in the alert. Falco interpolates runtime fields like %user.name, %container.name, and %k8s.pod.name so you know exactly which pod was compromised.

• Deploy the custom rule to Falco by upgrading the Helm release with the new values file:

helm upgrade --namespace falco falco falcosecurity/falco \
  --set tty=true \
  --set falcosidekick.enabled=true \
  --set falcosidekick.webui.enabled=true \
  -f k8s/falco-custom-rules.yaml

How does Helm load custom rules?

The customRules key in the values file tells the Falco Helm chart to create a ConfigMap containing your rule definitions. Falco mounts these files under /etc/falco/rules.d and loads them after the default rules, so your custom rules supplement the built-in detection.

• Wait for all Falco pods to restart and reach Ready state:

kubectl wait pods --for=condition=Ready --all -n falco --timeout=120s

• Verify your custom rule was loaded by checking the Falco logs:

kubectl logs -l app.kubernetes.io/name=falco -n falco -c falco | grep "Shell Execution"

You should see a log line confirming the rule was loaded from /etc/falco/rules.d/custom-rules.yaml.

Rule not loading?

Check that the YAML indentation in k8s/falco-custom-rules.yaml is correct. The content under custom-rules.yaml: |- must use consistent 4-space indentation for each rule field. Also verify all Falco pods restarted with kubectl get pods -n falco.

Help me fix my Falco custom rule not loading after Helm upgrade.

Trigger Falco alerts and verify in Falcosidekick UI

With your rule deployed, you can now simulate an attacker gaining shell access to a running container. You will use kubectl exec to open a shell in one of your Flask app pods.

• Get the name of a running Flask app pod:

kubectl get pods -n flask-app -l app=flask-app

• Exec into the pod to trigger the Falco alert:

kubectl exec -it $(kubectl get pods -n flask-app -l app=flask-app -o name | head -1) -n flask-app -- /bin/sh -c "echo 'Falco test' && whoami"

What just happened?

Running kubectl exec spawns a new shell process inside the container. This triggers the execve syscall for /bin/sh, which matches your custom rule's condition. Falco captured the event and forwarded it to Falcosidekick.

• Verify the alert appeared in Falco's logs:

kubectl logs -l app.kubernetes.io/name=falco -n falco -c falco | grep Warning

You should see a Warning-level log entry containing "Shell executed in container" with details about the pod, container, and process.

• Open the Falcosidekick UI to view alerts visually by running:

kubectl -n falco port-forward svc/falco-falcosidekick-ui 2802

• Open http://localhost:2802 in your browser.
• Click **Events** in the sidebar to see the alert stream.
• Look for an event with rule name Shell Execution in Container and priority WARNING.

The event detail panel shows the container ID, pod name, namespace, user, shell binary, and parent process. This is the forensic context a security team needs to investigate the incident.

Why runtime security matters beyond policies

Kyverno prevents misconfigured deployments. Falco detects attacks against correctly configured workloads. An attacker exploiting an application vulnerability (like a remote code execution bug in your Flask app) would pass all admission policies, but the moment they spawn a shell, Falco catches it.

In production, you would route these alerts to a SIEM or Slack channel for immediate incident response.

No events in Falcosidekick UI?

Verify Falcosidekick is running with kubectl get pods -n falco. If Falcosidekick restarted after Falco, events prior to restart are lost. Run the kubectl exec command again to generate a fresh alert. Also confirm the port-forward is active and you are accessing port 2802.

Help me troubleshoot missing events in my Falcosidekick UI.

Your cluster now has real-time threat detection running alongside preventive policies. Next, you will set up Prometheus and Grafana to monitor both application performance and security events on unified dashboards.

Monitor with Prometheus and Grafana

Your cluster is secured, self-healing, and deployed via GitOps. But without visibility into what your application is actually doing, you are flying blind.

In this step, you will connect Prometheus to your Flask app's /metrics endpoint and build a custom Grafana dashboard that shows the RED metrics (Request rate, Error rate, Duration) in real-time. You will also configure an alert that fires automatically when errors spike.

In this step, get ready to:

• Access Grafana and explore pre-built Kubernetes dashboards.
• Create a ServiceMonitor so Prometheus scrapes your Flask app.
• Build a custom Grafana dashboard showing RED metrics with PromQL queries.
• Configure a PrometheusRule alert for high error rates.

Access Grafana and create a ServiceMonitor for your Flask app

The kube-prometheus-stack Helm chart ships with dozens of pre-built dashboards covering node health, pod resources, and cluster-wide metrics. Your Flask app's custom metrics are not visible yet because Prometheus does not know where to find them.

A ServiceMonitor is a custom resource that tells Prometheus which Kubernetes Services to scrape, on which port, and at which path. You will access Grafana first, then create a ServiceMonitor to connect Prometheus to your Flask app.

• Retrieve the Grafana admin password by running this command:

kubectl get secret --namespace monitoring kube-prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

What does this command do?

The kube-prometheus-stack chart stores the Grafana admin password in a Kubernetes Secret. This command extracts and decodes it. The default password is prom-operator unless you overrode it during installation.

• Copy the password that appears in your terminal output.
• Start a port-forward to access Grafana locally by running:

kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n monitoring

• Open http://localhost:3000 in your browser.
• Log in with username admin and the password you copied earlier.
• Click the hamburger menu (three lines) in the top left, then select Dashboards.
• Browse the General folder. You should see dashboards like Kubernetes / Compute Resources / Cluster and Node Exporter / Nodes.
• Open Kubernetes / Compute Resources / Namespace (Pods) and select your application namespace from the dropdown.

What are these dashboards showing?

These pre-built dashboards use metrics collected by kube-state-metrics and node-exporter (both installed automatically with kube-prometheus-stack). They show CPU, memory, network, and disk usage across every node and pod in your cluster.

Your Flask app's custom metrics are not here yet. The ServiceMonitor you create next tells Prometheus where to find them.

Cannot access Grafana?

If port 3000 is already in use, change the local port: use 3001:80 instead in the port-forward command and visit http://localhost:3001.

If the password does not work, the secret name may differ. Run kubectl get secrets -n monitoring and look for a secret ending in -grafana.

Help me troubleshoot Grafana access via port-forward

Now create the ServiceMonitor so Prometheus discovers your Flask app.

• Create the ServiceMonitor manifest by adding a new file k8s/service-monitor.yaml:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: flask-app
  namespace: default
  labels:
    # This label must match kube-prometheus-stack's serviceMonitorSelector
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app: flask-app
  endpoints:
    - port: http
      path: /metrics
      interval: 15s

What does this manifest do?

The selector.matchLabels field tells the ServiceMonitor to find any Kubernetes Service with the label app: flask-app. That matches the Service you created in k8s/service.yaml.

The endpoints block configures Prometheus to scrape the /metrics path on the named port http every 15 seconds. The release: kube-prometheus-stack label ensures the Prometheus Operator picks up this ServiceMonitor.

• Apply the ServiceMonitor by running:

kubectl apply -f k8s/service-monitor.yaml

• Verify Prometheus discovered the target by starting a port-forward:

kubectl port-forward svc/kube-prometheus-stack-prometheus 9090:9090 -n monitoring

• Open http://localhost:9090/targets in your browser.
• Look for a target group named serviceMonitor/default/flask-app with a status of UP.

Target not showing UP?

If the target does not appear, check that your Flask app Service has the label app: flask-app by running kubectl get svc flask-app -o yaml.

If the target appears but shows DOWN, the port name in your Service may not match. The ServiceMonitor expects a port named http. Check your k8s/service.yaml to confirm the port name.

Help me debug my ServiceMonitor target discovery

Create a custom Grafana dashboard with RED metrics

The RED method (Rate, Errors, Duration) is the standard for monitoring request-driven services. You will build three panels using PromQL queries against the metrics that prometheus-flask-instrumentator exposes automatically.

• Return to Grafana at http://localhost:3000.
• Click the + icon in the top right, then select New dashboard.
• Click Add visualization.
• Select Prometheus as the data source.

Add the first panel for request rate.

• In the query editor, enter the following PromQL query:

sum(rate(flask_http_request_duration_seconds_count[5m])) by (method, status)

• Set the panel title to Request Rate in the panel options on the right.
• Click Apply to save the panel.

What does this query measure?

The rate() function calculates the per-second increase of the counter over a 5-minute window. Wrapping it in sum() by (method, status) groups results by HTTP method and status code so you can see GET vs POST and 200 vs 500 separately.

Next, add the error rate panel.

• Click Add visualization again and select Prometheus as the data source.
• Enter this PromQL query for the error rate:

sum(rate(flask_http_request_duration_seconds_count{status=~"5.."}[5m])) / sum(rate(flask_http_request_duration_seconds_count[5m])) * 100

• Set the panel title to Error Rate (%).
• In the panel options, set the unit to Percent (0-100).
• Click Apply.

Finally, add the latency panel.

• Click Add visualization and select Prometheus.
• Enter this PromQL query for p95 latency:

histogram_quantile(0.95, sum(rate(flask_http_request_duration_seconds_bucket[5m])) by (le))

• Set the panel title to P95 Latency (seconds).
• In the panel options, set the unit to seconds (s).
• Click Apply.

What is a histogram quantile?

Prometheus histograms store request durations in pre-defined buckets. The histogram_quantile(0.95, ...) function calculates the 95th percentile latency from those buckets. This means 95% of requests complete faster than this value.

• Click the save icon (disk) in the top right of the dashboard.
• Name the dashboard Flask App RED Metrics and click Save.

You should see all three panels rendering data. If you recently deployed the app, give it a minute for metrics to populate.

Panels showing No data?

If panels show no data, your ServiceMonitor may not be scraping yet. Return to http://localhost:9090/targets and confirm the flask-app target is UP.

Generate some traffic to produce metrics. Open a second terminal and run a few requests against your app's Ingress IP or use port-forward to hit the Flask app directly.

Help me debug empty Grafana panels

Configure a PrometheusRule alert for high error rate

A PrometheusRule is a Kubernetes custom resource that defines alerting and recording rules. When the condition is met, Prometheus fires the alert to Alertmanager, which can route notifications to Slack, email, or PagerDuty.

• Create the PrometheusRule manifest by adding k8s/prometheus-rule.yaml:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: flask-app-alerts
  namespace: default
  labels:
    # Must match kube-prometheus-stack's ruleSelector
    release: kube-prometheus-stack
spec:
  groups:
    - name: flask-app
      rules:
        - alert: FlaskHighErrorRate
          # Fires when 5xx error rate exceeds 5% for 5 minutes
          expr: |
            sum(rate(flask_http_request_duration_seconds_count{status=~"5.."}[5m]))
            /
            sum(rate(flask_http_request_duration_seconds_count[5m]))
            * 100 > 5
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Flask app error rate above 5%"
            description: "The Flask application has an HTTP 5xx error rate of {{ $value }}% for the last 5 minutes."

How does this alert work?

The expr field contains the same error rate PromQL you used in the Grafana panel. The for: 5m clause means the condition must be true for 5 consecutive minutes before the alert fires. This prevents transient spikes from triggering false alarms.

The release: kube-prometheus-stack label ensures the Prometheus Operator picks up this rule and loads it into the running Prometheus instance.

• Apply the PrometheusRule by running:

kubectl apply -f k8s/prometheus-rule.yaml

• Verify the rule loaded by opening http://localhost:9090/alerts in your browser (with the Prometheus port-forward still active).
• You should see FlaskHighErrorRate listed under the flask-app group with a status of inactive (meaning the condition is not currently met).

Alert not appearing in Prometheus?

Make sure the release: kube-prometheus-stack label is present in your PrometheusRule metadata. The Prometheus Operator only loads rules that match its ruleSelector.

Run kubectl get prometheusrules -n default to confirm the resource was created. If it exists but is not loaded, check the Prometheus Operator logs with kubectl logs -n monitoring -l app.kubernetes.io/name=prometheus-operator.

Help me troubleshoot my PrometheusRule not loading

Your platform now has full observability. Prometheus scrapes your Flask app every 15 seconds, Grafana visualizes the RED metrics, and an alert fires automatically if errors spike above 5%. Next up, you will configure auto-scaling and self-healing so the platform responds to load automatically.

Configure Self-Healing and Auto-Scaling

Your DevSecOps platform now has security scanning, GitOps deployment, policy enforcement, runtime detection, and full observability. The final piece is resilience. A production system needs to handle traffic spikes and recover from failures without human intervention.

In this step, you'll verify the self-healing mechanisms already deployed through ArgoCD. You'll confirm the HPA scales pods based on CPU load, the PDB protects availability during disruptions, and the health probes restart unhealthy containers. Then you'll run a real load test and watch auto-scaling happen in real-time.

In this step, get ready to:

• Verify HPA and PDB are actively protecting your deployment.
• Confirm liveness and readiness probes restart unhealthy containers.
• Load test the application and watch auto-scaling in real-time.

Verify the HPA and PDB configurations

The HorizontalPodAutoscaler watches your pods' CPU usage and adjusts the replica count to match demand. The PodDisruptionBudget guarantees that at least one pod stays running during voluntary disruptions like node drains or cluster upgrades.

• Check the current state of your HPA by running:

kubectl get hpa -n flask-app

What should I see?

You should see a table with the HPA name, its reference to the flask-app Deployment, the current/target CPU percentage, MINPODS of 2, MAXPODS of 5, and the current REPLICAS count. If the TARGET column shows <unknown>, wait a minute for the metrics server to populate data.

• Get a detailed view of the HPA including its scaling conditions by running:

kubectl describe hpa -n flask-app

What does this output tell you?

The describe output shows the scaling conditions. Look for AbleToScale: True and ScalingActive: True in the Conditions section. These confirm the HPA can fetch metrics and calculate the desired replica count.

You should also see the target CPU utilization of 70% and the min/max replica bounds (2 to 5).

• Now verify the PDB is active by running:

kubectl get pdb -n flask-app

You should see the PDB with MIN AVAILABLE set to 1, ALLOWED DISRUPTIONS showing at least 1, and the current number of healthy pods.

Why use a PodDisruptionBudget?

When Kubernetes needs to evict pods (during node upgrades, scaling down, or spot instance preemptions), it respects PDB constraints. With minAvailable: 1, Kubernetes guarantees at least one flask-app pod stays running at all times. This prevents downtime during cluster maintenance.

On spot VMs (like your e2-standard-2 nodes), preemptions can happen at any time. The PDB ensures graceful handling of these events.

HPA showing unknown target?

If the TARGET column shows <unknown> for more than 2 minutes, the metrics server may not be scraping your pods correctly. Verify that kube-prometheus-stack is running in the monitoring namespace and that your deployment has resource requests defined.

Why is my HPA showing unknown metrics even though Prometheus is running?

Verify liveness and readiness probes

Kubernetes uses health probes to determine if a container is alive and ready to serve traffic. A failing liveness probe triggers a container restart. A failing readiness probe removes the pod from the Service's endpoints until it recovers.

• Examine the probe configuration on your running pods by running:

kubectl describe deployment flask-app -n flask-app | grep -A 5 "Liveness\|Readiness"

What do these probe settings mean?

The liveness probe hits /health every 10 seconds (periodSeconds: 10) after an initial 10-second delay (initialDelaySeconds: 10). If this endpoint fails 3 consecutive times, Kubernetes kills and restarts the container.

The readiness probe also checks /health but with a 5-second initial delay and 5-second interval. It controls whether the pod receives traffic from the Service.

• Verify probes are working by checking the restart count on your pods:

kubectl get pods -n flask-app -o wide

The RESTARTS column should show 0 for healthy pods. This confirms the probes are passing and no unnecessary restarts are occurring.

• Test that liveness probes actually restart unhealthy containers by exec-ing into a pod and killing the Flask process:

# Get a pod name
POD=$(kubectl get pods -n flask-app -o jsonpath='{.items[0].metadata.name}')

# Kill the Flask process inside the container
kubectl exec -n flask-app $POD -- kill 1

• Watch the pod restart by running:

kubectl get pods -n flask-app -w

Within 10-15 seconds, you should see the pod's STATUS change and the RESTARTS count increment by 1. Press Ctrl+C to stop watching once you see the restart.

Why this proves self-healing works

You just simulated a process crash inside the container. Kubernetes detected the failed liveness probe, terminated the unhealthy container, and started a fresh one automatically. In production, this same mechanism recovers from memory leaks, deadlocks, and application crashes without any human intervention.

Pod not restarting after kill?

If the pod was evicted instead of restarted, Kyverno may be blocking the new pod. Check kubectl get events -n flask-app for policy violations. Also verify the container uses PID 1 for its main process. Distroless images run the app as PID 1 by default.

Help me debug why my pod isn't restarting after I killed the process

Load test and watch HPA scale up in real-time

The ultimate test of auto-scaling is generating real load and watching Kubernetes respond. You'll use a simple curl loop against your Ingress IP to drive CPU usage above the 70% target, triggering the HPA to add more replicas.

• Get your NGINX Ingress external IP by running:

kubectl get svc -n ingress-nginx ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Copy the IP address from the output. You'll use this as the target for your load test.

• Open a second terminal window. Start the HPA watch in this second terminal so you can observe scaling in real-time:

kubectl get hpa -n flask-app --watch

• Switch back to your first terminal. Start the load test by running a curl loop against your Ingress IP:

# Replace INGRESS_IP with the IP from the earlier command
while true; do curl -s -o /dev/null http://[[INGRESS_IP="your Ingress external IP"]]/api/items; done

What does this load test do?

This loop sends continuous HTTP requests to your Flask app's /api/items endpoint as fast as possible. Each request consumes CPU in your pods, pushing the average CPU utilization above the 70% target. The HPA controller checks metrics every 15 seconds and will scale up when it detects sustained high usage.

• Watch the second terminal. Within 1-2 minutes, you should see the REPLICAS column increase from 2 to 3, then potentially to 4 or 5 as CPU pressure builds.
• Once you see the replica count increase (confirming auto-scaling works), press Ctrl+C in your first terminal to stop the load test.
• Press Ctrl+C in the second terminal to stop watching the HPA.

After stopping the load, Kubernetes will scale back down to 2 replicas after the stabilization window (approximately 5 minutes by default).

Why does scale-down take longer than scale-up?

The HPA has a default downscale stabilization window of 300 seconds (5 minutes). This prevents flapping where pods get rapidly added and removed during bursty traffic. The scale-up window is 0 seconds by default, meaning Kubernetes responds to load increases immediately.

HPA not scaling up?

If replicas stay at 2 after a few minutes of load, check that your deployment has CPU resource requests defined. The HPA calculates utilization as a percentage of the request value. Without requests, it cannot compute utilization.

Also verify the metrics API is responding by running kubectl top pods -n flask-app. If this shows actual CPU values, the metrics pipeline is working.

Help me debug why my HPA isn't scaling up under load

Your DevSecOps platform is now fully production-grade. It provisions infrastructure as code, enforces security at every stage, deploys via GitOps, detects threats at runtime, monitors everything with custom dashboards, and heals itself under load. That is enterprise-grade operations in a single project.

Secret mission

Create a Security Correlation Dashboard

Your DevSecOps platform already detects threats and monitors performance separately. But what if you could see them together? In this secret mission, you'll build an advanced Grafana dashboard that correlates Falco security alerts with application metrics to detect attack patterns in real-time.

Clean Up Your Resources

Clean Up Your Resources

This project uses paid GCP resources that cost approximately $1.50-2/day. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Cost warning

Your GKE cluster with 3x e2-standard-2 spot nodes is actively billing while running. Every hour the cluster stays up costs money against your GCP budget.

If you are done with the project, destroy your infrastructure immediately to stop all charges. There is no pause option for GKE compute nodes.

Resources you used:

• GKE cluster (3x e2-standard-2 spot nodes) with all Helm releases (Jenkins, SonarQube, ArgoCD, kube-prometheus-stack, Kyverno, Falco, Istio, NGINX Ingress).
• VPC network provisioned by OpenTofu.
• Artifact Registry repository (devsecops-registry) storing your container images.
• IAM service accounts with Artifact Registry writer and GKE node access roles.
• Local project files (infra/, app/, k8s/, ansible/, sonar-project.properties, Jenkinsfile).

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with the platform.

• Monitor your spend in the GCP Console under Billing. Your daily cost is approximately $1.50-2/day.
• Your OpenTofu state file in infra/ tracks all provisioned resources.
• The gcloud CLI and OpenTofu remain installed on your machine.

Pause - I'll come back to this later

There is no pause option for GKE. You must destroy the cluster to stop billing. Your OpenTofu state file and configuration files preserve everything you need to recreate the infrastructure later.

• Follow the steps in the Delete tab to destroy cloud resources and stop billing.
• Keep your local project files (infra/, app/, k8s/, ansible/). These contain everything needed to rebuild.
• When you are ready to resume, run tofu apply from the infra/ directory to recreate all GCP resources. Then re-run your Ansible playbook to reinstall all Helm charts.

Delete - I don't want to use this again

Remove all project resources and stop all billing permanently.

Destroy all GCP resources by navigating to the infra/ directory and running the destroy command:

cd infra
tofu destroy

What does this command do?

OpenTofu will show you a plan of every resource it will destroy (GKE cluster, VPC, Artifact Registry, IAM service accounts). It prompts you to type yes to confirm. All Helm releases running on the cluster are destroyed along with it.

• After the destroy completes, verify in the GCP Console that no resources remain. Navigate to Kubernetes Engine, Artifact Registry, and IAM to confirm everything is gone.
• Delete local project files by running:

cd ~/Desktop
rm -rf devsecops-pipeline

Uninstalling gcloud CLI

If you no longer need the gcloud CLI, Google provides documented uninstall instructions. Find your installation directory by running:

gcloud info --format='value(installation.sdk_root)'

Delete both that directory and your config directory (typically ~/.config/gcloud). See the official guide at cloud.google.com/sdk/docs/uninstall-cloud-sdk for the full steps.

That's a Wrap!

That's a Wrap!

Nice work! You have built a production-grade DevSecOps platform on GKE that most engineers take months to assemble.

You've learned how to:

• Provisioned a complete GKE cluster with OpenTofu infrastructure as code, then automated all tooling installation across namespaces with Ansible playbooks.
• Built a Jenkins CI/CD pipeline with automated security gates covering SAST via SonarQube, container scanning via Trivy, and DAST via OWASP ZAP that block vulnerable deployments.
• Deployed a self-healing, GitOps-driven platform with ArgoCD auto-sync, Istio service mesh with mTLS, Kyverno policy enforcement, Falco runtime detection, and full Prometheus + Grafana observability.
• Correlated Falco security alerts with application performance metrics in a custom Grafana dashboard for real-time attack pattern detection.

Ready to quiz yourself?