AIOps Self-Healing Kubernetes Platform

Build an ML-powered platform that detects anomalies and auto-scales pods.

Introduction

30 Second Summary

When a website crashes at 2 AM, someone has to wake up, diagnose the problem, and fix it manually. What if your infrastructure could detect the problem and fix itself before anyone even notices?

In this project, you will build a fully local AIOps platform that monitors a microservice, detects unusual traffic patterns using machine learning, and automatically scales your application to handle the load.

What You'll Build

A Python-based anomaly detection engine that integrates with Kubernetes, Prometheus, and Terraform to create a self-healing infrastructure platform running entirely on your local machine.

By the end of this project, you'll have:

  • A local Kind Kubernetes cluster running a containerized Python Flask microservice with full observability (Prometheus + Grafana).
  • Terraform Infrastructure as Code provisioning AWS-compatible resources against a Floci local cloud emulator at zero cost.
  • A Python AIOps anomaly detector using Isolation Forest that monitors metrics and auto-scales Kubernetes pods when anomalies are detected.
  • Secret Mission: Automate the entire platform setup with an Ansible playbook that deploys everything in a single command.

Are there any prerequisites?

This is an advanced project. You need Docker Desktop installed and running, along with Python 3.9+, Terraform, kubectl, Helm, Git, and Kind.

If you have completed DevOps or cloud infrastructure projects before and are comfortable with containers and the terminal, you are ready to go.

Start Local Infrastructure

Your AIOps platform needs two foundational services running locally before you can build anything on top of them.

The first is Floci, a local AWS emulator that lets you provision cloud resources (S3 buckets, DynamoDB tables) without connecting to a real AWS account. The second is a Kind cluster, which runs a full Kubernetes environment inside Docker containers on your machine.

In this step, get ready to:
  • Start Floci local AWS emulator via Docker Compose.
  • Create a Kind Kubernetes cluster with port mappings for accessing services.
  • Verify both services are healthy and responding to commands.
Start Floci local AWS emulator

Floci emulates AWS services locally on port 4566. You define it in a Docker Compose file so it starts with a single command.

  • Create the Floci configuration by adding this to a new file called docker-compose.yml in your aiops-platform project folder:
version: "3.8"

services:
  floci:
    image: floci/floci:latest
    ports:
      - "4566:4566"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - FLOCI_DEFAULT_REGION=us-east-1
      - FLOCI_STORAGE_MODE=persistent
      - FLOCI_STORAGE_PERSISTENT_PATH=/data
    volumes:
      - floci-data:/data

volumes:
  floci-data:

What does this configuration do?

  • image: floci/floci:latest pulls the Floci emulator image that provides S3, DynamoDB, and 47+ other AWS services locally.
  • ports: "4566:4566" exposes Floci on port 4566, the same port real AWS SDKs and CLIs can target.
  • FLOCI_STORAGE_MODE=persistent ensures your provisioned resources survive container restarts.
  • The named volume floci-data stores Floci's persistent data on your machine.
  • Save docker-compose.yml.
  • Start the Floci container in detached mode by running this command:
docker compose up -d
  • Verify the container is running by running:
docker compose ps
  • You should see the Floci container listed with a status of Up and port 4566 mapped.

Container not starting?

  • Make sure Docker Desktop is running. Floci runs as a Docker container and needs the Docker daemon active.
  • Check that port 4566 is not already in use by another service on your machine.
  • Run docker compose logs to see any error output from the container.

Why won't my Floci container start?

✔️ Awesome, I've got everything!

Great. Make sure you have saved docker-compose.yml and that docker compose ps shows the container running.

ⓧ I'd like to double check the full code

version: "3.8"

services:
  floci:
    image: floci/floci:latest
    ports:
      - "4566:4566"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - FLOCI_DEFAULT_REGION=us-east-1
      - FLOCI_STORAGE_MODE=persistent
      - FLOCI_STORAGE_PERSISTENT_PATH=/data
    volumes:
      - floci-data:/data

volumes:
  floci-data:
Create a Kind cluster with port mappings

Kind runs Kubernetes nodes as Docker containers. You need a cluster configuration that maps container ports to your host so you can access services later (Prometheus on port 30000 and Grafana on port 30001).

  • Create the cluster configuration by adding this to a new file called kind-config.yaml in your aiops-platform project folder:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30000
    hostPort: 30000
    protocol: TCP
  - containerPort: 30001
    hostPort: 30001
    protocol: TCP

What does this configuration do?

  • kind: Cluster tells Kind this is a cluster configuration file.
  • apiVersion: kind.x-k8s.io/v1alpha4 specifies the Kind config schema version.
  • extraPortMappings maps ports inside the Kind node container to your host machine. Port 30000 will expose Prometheus and port 30001 will expose Grafana later.
  • Save kind-config.yaml.
  • Create the Kind cluster by running this command:
kind create cluster --name aiops-platform --config kind-config.yaml

This takes 1-2 minutes. Kind downloads a node image, creates a Docker container acting as a Kubernetes node, and configures kubectl to point at the new cluster.

  • Verify your cluster is ready by running:
kubectl get nodes
  • You should see a node named aiops-platform-control-plane with a status of Ready.

Cluster creation failed?

  • Confirm Docker Desktop is running and has sufficient resources allocated (at least 4 GB RAM recommended).
  • If you see a message about an existing cluster, delete it first with kind delete cluster --name aiops-platform and try again.
  • On Windows, make sure Docker Desktop is set to Linux containers.

Why is my Kind cluster not creating successfully?

✔️ Awesome, I've got everything!

Great. Make sure you have saved kind-config.yaml and that kubectl get nodes shows a Ready node.

ⓧ I'd like to double check the full code

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30000
    hostPort: 30000
    protocol: TCP
  - containerPort: 30001
    hostPort: 30001
    protocol: TCP
Verify both services are healthy

Both Floci and Kind are running, but you should confirm they respond correctly to real commands before building on top of them.

  • Test that Floci responds to AWS S3 commands by running:
AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test aws --endpoint-url=http://localhost:4566 --region us-east-1 s3 ls

What does this command do?

This sends a real AWS CLI s3 ls command to Floci on localhost:4566 instead of AWS. The dummy credentials (test/test) work because Floci accepts any credentials.

An empty response (no output) is correct at this point. It means Floci is responding but has no buckets yet. You will create a bucket with Terraform in the next step.

  • Confirm the Kind cluster node is healthy by running:
kubectl get nodes
  • You should see output like this:
NAME                            STATUS   ROLES           AGE   VERSION
aiops-platform-control-plane   Ready    control-plane   ...   ...

Floci returning an error?

  • Make sure the Floci container is still running. Check with docker compose ps.
  • If you see a connection refused error, wait 10-15 seconds for Floci to fully initialize, then try again.
  • Verify you have the AWS CLI installed by running aws --version.

Help me debug my Floci or Kind connectivity issue.

Your local infrastructure is up and responding. Next up, you will use Terraform to provision AWS resources (an S3 bucket and DynamoDB table) on Floci.

Provision Infrastructure with Terraform

Your local infrastructure is alive. Floci is emulating AWS on port 4566, and your Kind cluster has a Ready node.

Now you need somewhere to store model artifacts for the AIOps engine you'll build later. Instead of creating cloud resources by hand, you'll use Terraform to define your infrastructure as code. Terraform will talk to Floci the same way it would talk to real AWS, so every resource you provision is reproducible with a single command.

In this step, get ready to:
  • Write a Terraform configuration that points the AWS provider at Floci.
  • Provision an S3 bucket and DynamoDB table on your local emulator.
  • Verify both resources exist using AWS CLI commands.
Write the Terraform configuration

Terraform uses .tf files to declare what infrastructure should exist. You'll create a single file that tells the AWS provider to send all API calls to Floci on localhost:4566 instead of real AWS endpoints.

  • Create the Terraform directory and file by running these commands:
mkdir terraform
cd terraform
touch main.tf
  • Confirm the file exists by running:
ls

You should see main.tf in the output.

  • Open terraform/main.tf in your code editor and paste the provider configuration:
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

provider "aws" {
  access_key                  = "test"
  secret_key                  = "test"
  region                      = "us-east-1"
  s3_use_path_style           = true
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true

  endpoints {
    s3       = "http://localhost:4566"
    dynamodb = "http://localhost:4566"
    sts      = "http://localhost:4566"
  }
}

What does this code do?

The terraform block pins the AWS provider to version ~> 6.0, which allows any 6.x release.

The provider "aws" block uses dummy credentials ("test") because Floci accepts any credentials. The endpoints block redirects all S3, DynamoDB, and STS API calls to http://localhost:4566 where Floci is listening.

The skip_credentials_validation and related flags prevent Terraform from verifying credentials against real AWS, which would fail since you're running locally.

  • Below the provider block, paste the resource definitions and outputs:
resource "aws_s3_bucket" "model_artifacts" {
  bucket = "aiops-model-artifacts"
}

resource "aws_dynamodb_table" "terraform_state_lock" {
  name         = "terraform-state-lock"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

output "bucket_name" {
  value = aws_s3_bucket.model_artifacts.id
}

output "dynamodb_table_name" {
  value = aws_dynamodb_table.terraform_state_lock.name
}

What does this code do?

  • aws_s3_bucket "model_artifacts" creates an S3 bucket called aiops-model-artifacts. This is where you'll store trained ML model files later.
  • aws_dynamodb_table "terraform_state_lock" creates a DynamoDB table for Terraform state locking. In a team environment, this prevents two people from modifying the same infrastructure at once.
  • The output blocks print the resource names after provisioning, making it easy to confirm what was created.
  • Save terraform/main.tf.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

provider "aws" {
  access_key                  = "test"
  secret_key                  = "test"
  region                      = "us-east-1"
  s3_use_path_style           = true
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true

  endpoints {
    s3       = "http://localhost:4566"
    dynamodb = "http://localhost:4566"
    sts      = "http://localhost:4566"
  }
}

resource "aws_s3_bucket" "model_artifacts" {
  bucket = "aiops-model-artifacts"
}

resource "aws_dynamodb_table" "terraform_state_lock" {
  name         = "terraform-state-lock"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

output "bucket_name" {
  value = aws_s3_bucket.model_artifacts.id
}

output "dynamodb_table_name" {
  value = aws_dynamodb_table.terraform_state_lock.name
}
Initialize and apply the Terraform configuration

Terraform needs to download the AWS provider plugin before it can create any resources. The init command handles this, and apply sends the resource definitions to Floci.

  • Make sure you are inside the terraform/ directory, then initialize Terraform by running:
terraform init

You should see a message that says Terraform has been successfully initialized! at the end of the output.

Seeing a provider error?

If you see an error about failing to query available provider packages, check your internet connection. Terraform downloads the AWS provider plugin from the Terraform Registry during init.

If you see an error about the working directory not containing configuration files, make sure you ran the command from inside the terraform/ directory where main.tf lives.

Help me debug my terraform init error

  • Apply the Terraform configuration to provision your resources by running:
terraform apply

What should I see?

Terraform shows you a plan listing the resources it will create (1 S3 bucket, 1 DynamoDB table). It then asks you to confirm by typing yes and pressing Enter.

  • Type yes when Terraform asks to perform the actions, then press Enter.

You should see output ending with Apply complete! Resources: 2 added, 0 changed, 0 destroyed. followed by your two outputs showing the bucket name and table name.

Apply failed with a connection error?

If you see a connection refused error, Floci might have stopped. Switch to a new terminal tab and run docker compose up -d from your project root to restart it.

If you see an error about the S3 bucket already existing, a previous apply may have partially completed. Run terraform apply again and Terraform will reconcile the state.

Help me fix my terraform apply error

Verify resources exist on Floci

Terraform reported success, but it's good practice to verify resources independently. You'll use the AWS CLI pointed at Floci to confirm both resources exist.

  • List S3 buckets on Floci by running:
aws --endpoint-url http://localhost:4566 s3 ls

You should see aiops-model-artifacts in the output.

  • List DynamoDB tables on Floci by running:
aws --endpoint-url http://localhost:4566 dynamodb list-tables

You should see terraform-state-lock listed under TableNames in the JSON response.

Why verify outside of Terraform?

Terraform tracks state in a local file, so its outputs reflect what it believes exists. Querying Floci directly with the AWS CLI proves the resources are actually there, not just recorded in Terraform's state file.

Not seeing your resources?

Make sure you're using --endpoint-url http://localhost:4566 in your AWS CLI command. Without it, the CLI tries to reach real AWS and won't find your local resources.

If you get a connection error, verify Floci is still running with docker ps. You should see a container using the floci/floci:latest image.

Help me troubleshoot my AWS CLI verification

Your Infrastructure as Code is working. You now have an S3 bucket for model artifacts and a DynamoDB table for state locking, all running locally on Floci at zero cost. Next up, you'll build a Python Flask microservice and deploy it into your Kind cluster.

Dockerize and Deploy Flask App to Kind

Your infrastructure layer is in place. Floci is emulating AWS services locally, and Terraform has provisioned an S3 bucket and DynamoDB table against it.

Now you need something to actually monitor. In this step, you will build a Python Flask microservice that exposes Prometheus metrics, containerize it with Docker, and deploy it to your Kind cluster.

In this step, get ready to:
  • Write a Flask app that tracks request count, latency, and error rate as Prometheus metrics.
  • Build a Docker image and load it into the Kind cluster.
  • Deploy the app to Kubernetes and verify the pod is running and serving metrics.
Write the Flask app with Prometheus metrics

The AIOps anomaly detector needs real metrics to analyze. You will write a Flask app that instruments every request with counters and histograms that Prometheus can scrape.

  • Create the app/ directory inside your aiops-platform project folder by running this command:
mkdir app
  • Create a new file called app.py inside the app/ directory.
  • Add the imports and Prometheus metric definitions by pasting this code:
from flask import Flask, jsonify
from prometheus_client import Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST
import time
import random

app = Flask(__name__)

# Define Prometheus metrics for tracking request patterns
REQUEST_COUNT = Counter(
    "flask_app_requests_total",
    "Total number of requests",
    ["method", "endpoint", "status"]
)

REQUEST_LATENCY = Histogram(
    "flask_app_request_latency_seconds",
    "Request latency in seconds",
    ["endpoint"]
)

ERROR_COUNT = Counter(
    "flask_app_errors_total",
    "Total number of errors",
    ["endpoint"]
)

What does this code do?

This sets up three Prometheus metrics that the anomaly detector will analyze later:

  • REQUEST_COUNT is a Counter that tracks the total number of requests, labeled by HTTP method, endpoint, and status code.
  • REQUEST_LATENCY is a Histogram that records how long each request takes in seconds.
  • ERROR_COUNT is a Counter that tracks errors per endpoint.
  • Below the metric definitions, add the health-check route:

@app.route("/")
def home():
    start_time = time.time()
    REQUEST_COUNT.labels(method="GET", endpoint="/", status="200").inc()
    latency = time.time() - start_time
    REQUEST_LATENCY.labels(endpoint="/").observe(latency)
    return jsonify({"status": "healthy", "service": "aiops-demo"})

What does this route do?

The / route is a health check. Every time it is hit, it increments the request counter and records how long the response took. This gives Prometheus a baseline of normal behavior to compare against.

  • Below the home() function, add the data endpoint, the metrics endpoint, and the main block:

@app.route("/api/data")
def get_data():
    start_time = time.time()
    # Simulate variable latency
    time.sleep(random.uniform(0.01, 0.1))
    if random.random() < 0.05:
        ERROR_COUNT.labels(endpoint="/api/data").inc()
        REQUEST_COUNT.labels(method="GET", endpoint="/api/data", status="500").inc()
        latency = time.time() - start_time
        REQUEST_LATENCY.labels(endpoint="/api/data").observe(latency)
        return jsonify({"error": "simulated failure"}), 500
    REQUEST_COUNT.labels(method="GET", endpoint="/api/data", status="200").inc()
    latency = time.time() - start_time
    REQUEST_LATENCY.labels(endpoint="/api/data").observe(latency)
    return jsonify({"data": "sample response", "timestamp": time.time()})


@app.route("/metrics")
def metrics():
    return generate_latest(), 200, {"Content-Type": CONTENT_TYPE_LATEST}


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

What does this code do?

  • get_data() simulates a real API endpoint with variable latency (10-100ms) and a 5% error rate. This gives the anomaly detector interesting patterns to analyze.
  • metrics() exposes all collected Prometheus metrics at /metrics in the format Prometheus expects to scrape.
  • The if __name__ block runs the Flask server on port 5000, listening on all interfaces so it is reachable inside a container.
  • Save app/app.py.

Seeing a syntax error?

  • Make sure the three code chunks are pasted in order with no overlap. The decorator @app.route must be directly above each function definition.
  • Check that the from prometheus_client import line includes all four names: Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST.
  • Help me debug my app.py file

✔️ Awesome, I've got everything!

Great. Double check you have saved app/app.py before moving on.

ⓧ I'd like to double check the full code

from flask import Flask, jsonify
from prometheus_client import Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST
import time
import random

app = Flask(__name__)

REQUEST_COUNT = Counter(
    "flask_app_requests_total",
    "Total number of requests",
    ["method", "endpoint", "status"]
)

REQUEST_LATENCY = Histogram(
    "flask_app_request_latency_seconds",
    "Request latency in seconds",
    ["endpoint"]
)

ERROR_COUNT = Counter(
    "flask_app_errors_total",
    "Total number of errors",
    ["endpoint"]
)


@app.route("/")
def home():
    start_time = time.time()
    REQUEST_COUNT.labels(method="GET", endpoint="/", status="200").inc()
    latency = time.time() - start_time
    REQUEST_LATENCY.labels(endpoint="/").observe(latency)
    return jsonify({"status": "healthy", "service": "aiops-demo"})


@app.route("/api/data")
def get_data():
    start_time = time.time()
    # Simulate variable latency
    time.sleep(random.uniform(0.01, 0.1))
    if random.random() < 0.05:
        ERROR_COUNT.labels(endpoint="/api/data").inc()
        REQUEST_COUNT.labels(method="GET", endpoint="/api/data", status="500").inc()
        latency = time.time() - start_time
        REQUEST_LATENCY.labels(endpoint="/api/data").observe(latency)
        return jsonify({"error": "simulated failure"}), 500
    REQUEST_COUNT.labels(method="GET", endpoint="/api/data", status="200").inc()
    latency = time.time() - start_time
    REQUEST_LATENCY.labels(endpoint="/api/data").observe(latency)
    return jsonify({"data": "sample response", "timestamp": time.time()})


@app.route("/metrics")
def metrics():
    return generate_latest(), 200, {"Content-Type": CONTENT_TYPE_LATEST}


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)
  • Create requirements.txt inside the app/ directory with the Flask and Prometheus client dependencies:
flask==3.1.3
prometheus-client==0.25.0

Why pin versions?

Pinning exact versions ensures your Docker build is reproducible. If a new Flask or prometheus-client release introduces a breaking change, your image still builds correctly.

  • Save app/requirements.txt.
  • Create a Dockerfile inside the app/ directory with the following content:
FROM python:3.11-slim

WORKDIR /app

# Install dependencies first for Docker layer caching
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY app.py .

EXPOSE 5000

CMD ["python", "app.py"]

What does this Dockerfile do?

  • FROM python:3.11-slim starts from a lightweight Python base image.
  • COPY requirements.txt and RUN pip install are done before copying the app code. This means Docker caches the dependency layer and only reinstalls when requirements change.
  • EXPOSE 5000 documents which port the container listens on.
  • Save app/Dockerfile.
  • Verify all three files exist by running:
ls app/

You should see Dockerfile, app.py, and requirements.txt in the output.

Build the Docker image and load it into Kind

Kind runs Kubernetes nodes as Docker containers. It cannot pull images from your local Docker daemon by default. You need to explicitly load your image into the cluster.

  • Build the Docker image by running this command:
docker build -t aiops-demo:latest app/

What does this command do?

The -t aiops-demo:latest flag tags the image with a name and version. The app/ argument tells Docker where to find the Dockerfile.

  • You should see output ending with a line like Successfully tagged aiops-demo:latest or a final build ID.

Build failing?

  • Make sure Docker Desktop is running. The Docker daemon must be active for builds to work.
  • Check that app/Dockerfile exists. The build context path app/ must contain the Dockerfile.
  • Help me fix my Docker build
  • Load the image into your Kind cluster by running:
kind load docker-image aiops-demo:latest --name aiops-platform

Why load the image manually?

Kind clusters run inside Docker containers and have their own image store, separate from your host's Docker daemon. The kind load docker-image command copies the image from your local Docker into the Kind node so Kubernetes can use it without pulling from a registry.

  • You should see output indicating the image was loaded successfully.

Image load hanging or failing?

  • Confirm your Kind cluster is still running with kind get clusters. You should see aiops-platform in the output.
  • If the load is slow, give it a minute. Large images take time to transfer into the cluster node.
  • Help me load my Docker image into Kind
Deploy to Kubernetes and verify the pod is running

With the image loaded, you can now create the Kubernetes resources: a Deployment to run the pods, a Service to expose them internally, and a ServiceMonitor so Prometheus knows where to scrape metrics.

  • Create the k8s/ directory by running:
mkdir k8s
  • Create a file called deployment.yaml inside k8s/ and paste the Deployment manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: aiops-demo
  labels:
    app: aiops-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: aiops-demo
  template:
    metadata:
      labels:
        app: aiops-demo
    spec:
      containers:
      - name: aiops-demo
        image: aiops-demo:latest
        ports:
        - containerPort: 5000
        resources:
          requests:
            cpu: 50m
            memory: 64Mi
          limits:
            cpu: 200m
            memory: 128Mi

What does this Deployment define?

  • replicas: 2 starts two copies of the Flask app for redundancy. The anomaly detector will scale this number up and down later.
  • image: aiops-demo:latest references the image you just loaded into Kind.
  • The resources block sets CPU and memory limits so each pod does not consume unbounded resources on your machine.
  • Below the Deployment, add the Service and ServiceMonitor separated by --- document separators:
---
apiVersion: v1
kind: Service
metadata:
  name: aiops-demo
  labels:
    app: aiops-demo
spec:
  selector:
    app: aiops-demo
  ports:
  - port: 80
    targetPort: 5000
    protocol: TCP
  type: ClusterIP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: aiops-demo
  labels:
    release: monitoring
spec:
  selector:
    matchLabels:
      app: aiops-demo
  endpoints:
  - port: "80"
    path: /metrics
    interval: 15s

What do the Service and ServiceMonitor do?

  • The Service exposes your pods on port 80 internally, forwarding traffic to container port 5000. Other pods in the cluster can reach your app at aiops-demo:80.
  • The ServiceMonitor tells Prometheus to scrape the /metrics path every 15 seconds. The release: monitoring label is how the kube-prometheus-stack discovers which ServiceMonitors to use.
  • Save k8s/deployment.yaml.

YAML indentation issues?

  • YAML is whitespace-sensitive. Use spaces only (no tabs). Each level of nesting is 2 spaces.
  • Make sure the --- separators between documents start at column 0 with no leading spaces.
  • Help me fix my deployment.yaml

✔️ Awesome, I've got everything!

Great. Make sure you have saved the file before applying it to the cluster.

ⓧ I'd like to double check the full code

apiVersion: apps/v1
kind: Deployment
metadata:
  name: aiops-demo
  labels:
    app: aiops-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: aiops-demo
  template:
    metadata:
      labels:
        app: aiops-demo
    spec:
      containers:
      - name: aiops-demo
        image: aiops-demo:latest
        ports:
        - containerPort: 5000
        resources:
          requests:
            cpu: 50m
            memory: 64Mi
          limits:
            cpu: 200m
            memory: 128Mi
---
apiVersion: v1
kind: Service
metadata:
  name: aiops-demo
  labels:
    app: aiops-demo
spec:
  selector:
    app: aiops-demo
  ports:
  - port: 80
    targetPort: 5000
    protocol: TCP
  type: ClusterIP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: aiops-demo
  labels:
    release: monitoring
spec:
  selector:
    matchLabels:
      app: aiops-demo
  endpoints:
  - port: "80"
    path: /metrics
    interval: 15s
  • Apply the manifest to your cluster by running:
kubectl apply -f k8s/deployment.yaml

What does this command do?

kubectl apply sends all three resource definitions (Deployment, Service, ServiceMonitor) to the Kubernetes API server. The cluster then schedules pods, creates the network endpoint, and registers the scrape target.

You should see output confirming each resource was created:

deployment.apps/aiops-demo created
service/aiops-demo created
servicemonitor.monitoring.coreos.com/aiops-demo created
  • Verify the pods are running by executing:
kubectl get pods

You should see two pods with aiops-demo in their names, both showing Running in the STATUS column. It may take 15-30 seconds for them to reach Running state.

Pods stuck in ImagePullBackOff or ErrImagePull?

  • This means Kubernetes is trying to pull the image from a remote registry instead of using the local copy. Re-run kind load docker-image aiops-demo:latest --name aiops-platform to ensure the image is in the cluster.
  • Check that the image name in deployment.yaml matches exactly: aiops-demo:latest.
  • Help me fix my pod image pull issue
  • Port-forward the service to test the metrics endpoint by running:
kubectl port-forward svc/aiops-demo 8080:80
  • Open a second terminal window and curl the metrics endpoint:
curl http://localhost:8080/metrics

You should see Prometheus-formatted metrics output including lines like flask_app_requests_total and flask_app_request_latency_seconds.

  • Press Ctrl+C in the first terminal to stop the port-forward.

Curl returning connection refused?

  • Make sure the port-forward command is still running in the first terminal. It must stay active while you curl from the second terminal.
  • Verify the pods are in Running state with kubectl get pods. If they are still starting up, wait a few seconds and try again.
  • Help me debug my port-forward

Your microservice is live on Kubernetes and already producing metrics. Next up, you will deploy Prometheus and Grafana to start collecting and visualizing those metrics in real time.

Deploy Observability Stack

Your Flask app is running and exposing metrics, but those metrics need a system that collects, stores, and visualizes them.

In this step, you will install the kube-prometheus-stack Helm chart into your cluster. This single chart deploys Prometheus (for metrics collection), Grafana (for dashboards), and the Prometheus Operator (for managing scrape targets via ServiceMonitors).

In this step, get ready to:
  • Install the kube-prometheus-stack Helm chart with ServiceMonitor discovery enabled.
  • Port-forward Prometheus and Grafana to access them from your browser.
  • Confirm that Prometheus is scraping your Flask app metrics and Grafana can display them.
Install kube-prometheus-stack via Helm

The kube-prometheus-stack Helm chart bundles Prometheus, Grafana, Alertmanager, and the Prometheus Operator into one install command. A critical setting for this project is serviceMonitorSelectorNilUsesHelmValues: false. This tells Prometheus to discover all ServiceMonitors across the cluster, not just ones labelled with the Helm release.

  • Add the prometheus-community Helm chart repository by running this command:
  • Add the prometheus-community Helm chart repository by running this command:
  • Add the prometheus-community Helm chart repository by running this command:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
  • Update your local Helm repo cache by running:
helm repo update
  • Install the chart with the release name monitoring into the monitoring namespace by running:
helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
  --set grafana.adminPassword=prom-operator

What does this command do?

  • helm install monitoring creates a Helm release named monitoring.
  • --namespace monitoring --create-namespace places all observability resources in a dedicated namespace (creating it if it does not exist).
  • serviceMonitorSelectorNilUsesHelmValues=false tells Prometheus to scrape ALL ServiceMonitors in the cluster, including the one you created for your Flask app in the previous step.
  • grafana.adminPassword=prom-operator sets the Grafana admin password to a known value so you can log in.
  • Wait for all pods in the monitoring namespace to reach Running status by running:
kubectl get pods -n monitoring --watch

You should see several pods come up including Prometheus, Grafana, the Prometheus Operator, kube-state-metrics, and node-exporter. Wait until all pods show Running or Completed status, then press Ctrl+C to stop watching.

Pods stuck in Pending or CrashLoopBackOff?

Check that your Kind cluster has enough resources. Run kubectl describe pod <pod-name> -n monitoring to see the events and reason for the failure.

If pods are stuck in Pending due to insufficient CPU or memory, try restarting Docker Desktop with more resources allocated.

help me debug pods stuck in Pending in my monitoring namespace

Port-forward Prometheus and Grafana

The observability services run inside the cluster with ClusterIP services. To access them from your browser, you need to forward local ports to the cluster services.

  • First, find the service names in the monitoring namespace by running:
kubectl get svc -n monitoring

You should see a list of services. Look for the Prometheus service (containing prometheus in the name) and the Grafana service (containing grafana in the name).

  • Start a port-forward for Prometheus on port 9090 by running this command in a new terminal tab:
kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090

Why a new terminal tab?

Port-forwarding is a blocking process. It keeps the connection open as long as the terminal is running. You need separate terminal tabs for each port-forward so they can run simultaneously.

  • Open another new terminal tab and start a port-forward for Grafana on port 3000 by running:
kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80
  • Open your browser and navigate to http://localhost:9090 to access the Prometheus UI.
  • Click **Status** in the top navigation bar, then click **Targets**.
  • Look for a target group that contains aiops-demo in its name. The state should show UP.

What does the Targets page show?

The Targets page lists every scrape endpoint Prometheus is monitoring. Each target shows the endpoint URL, its current state (UP or DOWN), and when it was last scraped. Seeing your Flask app as UP confirms that the ServiceMonitor you created in the previous step is working correctly.

Don't see aiops-demo in Targets?

Verify your ServiceMonitor has the label release: monitoring by running kubectl get servicemonitor aiops-demo -o yaml. The label must match the Helm release name so Prometheus discovers it.

Also confirm that serviceMonitorSelectorNilUsesHelmValues was set to false during installation. If not, uninstall and reinstall with the correct flag.

help me debug why Prometheus is not scraping my ServiceMonitor

Verify metrics in Grafana

Now that Prometheus is scraping your Flask app, you can visualize the metrics in Grafana.

  • Open your browser and navigate to http://localhost:3000 to access Grafana.
  • Log in with username admin and password prom-operator.
  • Click the **Explore** icon in the left sidebar (compass icon).
  • In the metric browser at the top, type flask_app_requests_total and press **Enter** or click **Run query**.

You should see data points appearing in the graph. This confirms that Grafana is connected to Prometheus and your Flask app metrics are flowing through the entire observability pipeline.

  • Try querying two more metrics to confirm the full set is available: flask_app_request_latency_seconds_bucket and flask_app_errors_total.

What are these three metrics?

  • flask_app_requests_total is a Counter tracking every request by method, endpoint, and status code.
  • flask_app_request_latency_seconds_bucket is a Histogram tracking response time distribution. The AIOps engine will use this to detect latency anomalies.
  • flask_app_errors_total is a Counter tracking errors by endpoint. Spikes here indicate something is wrong with the service.

No data showing in Grafana?

Make sure the Prometheus data source is selected in the Explore view (it should be the default). If no data source appears, Grafana may not have finished initializing. Give it another minute and refresh the page.

Also confirm your Flask app pods are still running by checking kubectl get pods in the default namespace. The app must be receiving at least some traffic for counters to have non-zero values.

help me troubleshoot why Grafana shows no metric data

Your observability stack is live. Prometheus is collecting metrics every 15 seconds, and Grafana can visualize them on demand. Next up, you will build the AIOps anomaly detector that queries these metrics and automatically scales your deployment when it detects unusual patterns.

Build the AIOps Anomaly Detector

Your observability stack is live. Prometheus is scraping metrics from your Flask app every 15 seconds, and Grafana is rendering them into dashboards.

Now you will build the brain of the platform. The AIOps anomaly detector watches Prometheus metrics in real time, learns what "normal" traffic looks like, and automatically scales your deployment when something unusual happens. This closes the loop from detection to remediation with zero human intervention.

In this step, get ready to:
  • Write a Python anomaly detector using Isolation Forest that queries Prometheus metrics.
  • Add Kubernetes auto-remediation logic that scales pods when anomalies are detected.
  • Run a load generator to trigger the self-healing loop and verify auto-scaling.
Write the anomaly detection engine

The anomaly detector needs to collect metrics from Prometheus, build a statistical model of normal behaviour, and flag deviations. You will use Isolation Forest from scikit-learn. It is an unsupervised ML algorithm that isolates outliers without needing labelled training data.

  • Create the aiops directory and its requirements file by running these commands:
mkdir aiops
cd aiops
  • Create the requirements.txt file inside aiops with the following content:
scikit-learn==1.9.0
kubernetes==36.0.2
requests==2.32.3
numpy==2.2.6

What are these dependencies?

  • scikit-learn provides the Isolation Forest algorithm for anomaly detection.
  • kubernetes is the official Python client for the Kubernetes API, used to scale deployments.
  • requests handles HTTP calls to the Prometheus query API.
  • numpy converts metrics into arrays for model training.
  • Confirm the file exists by running:
ls

You should see requirements.txt in the output.

  • Now create anomaly_detector.py inside the aiops folder and add the imports and configuration constants:
import time
import numpy as np
import requests
from sklearn.ensemble import IsolationForest
from kubernetes import client, config

# Configuration constants
PROMETHEUS_URL = "http://localhost:9090"
DEPLOYMENT_NAME = "aiops-demo"
NAMESPACE = "default"
MIN_REPLICAS = 2
MAX_REPLICAS = 6
COOLDOWN_SECONDS = 60
CHECK_INTERVAL = 10
BASELINE_SAMPLES = 30

What do these constants control?

  • PROMETHEUS_URL points to the port-forwarded Prometheus instance from the previous step.
  • MIN_REPLICAS and MAX_REPLICAS define the scaling boundaries so the detector never over-provisions or under-provisions.
  • COOLDOWN_SECONDS prevents scaling loops by enforcing a wait between scale operations.
  • BASELINE_SAMPLES is the number of metric snapshots collected before training the model. At 10-second intervals, that is about 5 minutes of baseline data.
  • Below the constants, add the query_prometheus function that sends PromQL queries to the Prometheus HTTP API:
def query_prometheus(query):
    """Query Prometheus and return the result."""
    response = requests.get(
        f"{PROMETHEUS_URL}/api/v1/query",
        params={"query": query}
    )
    data = response.json()
    if data["status"] == "success" and data["data"]["result"]:
        return float(data["data"]["result"][0]["value"][1])
    return 0.0

What does this function do?

It hits the /api/v1/query endpoint on Prometheus with a PromQL expression. If the query returns results, it extracts the numeric value from the first time series. If the metric has no data yet, it returns 0.0 as a safe fallback.

  • Below query_prometheus, add the collect_metrics function that gathers the three signals the model monitors:
def collect_metrics():
    """Collect current metrics from Prometheus."""
    rps = query_prometheus(
        'rate(flask_app_requests_total[1m])'
    )
    latency_p95 = query_prometheus(
        'histogram_quantile(0.95, rate(flask_app_request_latency_seconds_bucket[1m]))'
    )
    error_rate = query_prometheus(
        'rate(flask_app_errors_total[1m])'
    )
    return [rps, latency_p95, error_rate]

What metrics are we tracking?

  • Requests per second (RPS): the rate of incoming traffic over the last minute.
  • P95 latency: the 95th percentile response time. A spike here means the service is struggling.
  • Error rate: how many errors are occurring per second. A surge indicates failures under load.

These three dimensions give the Isolation Forest model a multi-signal view of service health.

  • Save anomaly_detector.py.

You won't see a change yet. The next section adds the remediation logic and the main loop that ties everything together.

Seeing an import error when testing?

You have not installed the dependencies yet. That comes after the full file is written. If you want to confirm syntax, ensure you have matching parentheses and quotes in each function.

help me fix import errors in my anomaly_detector.py

Add Kubernetes auto-remediation logic

Detection alone is not enough. The detector needs to take action when it spots anomalies. You will add functions that use the Kubernetes Python client to scale the deployment up when traffic spikes, and scale it back down when things normalise.

  • In anomaly_detector.py, below collect_metrics, add the scale_deployment function:
def scale_deployment(replicas):
    """Scale the deployment to the specified number of replicas."""
    config.load_kube_config()
    apps_v1 = client.AppsV1Api()
    body = {"spec": {"replicas": replicas}}
    apps_v1.patch_namespaced_deployment_scale(
        name=DEPLOYMENT_NAME,
        namespace=NAMESPACE,
        body=body
    )
    print(f"[REMEDIATION] Scaled {DEPLOYMENT_NAME} to {replicas} replicas")

What does this function do?

It loads your local kubeconfig (which points at the Kind cluster), then uses the AppsV1 API to patch the deployment's scale subresource. This changes the replica count without touching any other deployment configuration.

  • Below scale_deployment, add the get_current_replicas function:
def get_current_replicas():
    """Get current replica count."""
    config.load_kube_config()
    apps_v1 = client.AppsV1Api()
    deployment = apps_v1.read_namespaced_deployment(
        name=DEPLOYMENT_NAME,
        namespace=NAMESPACE
    )
    return deployment.spec.replicas

Why read replicas before scaling?

The detector needs to know the current state before deciding how many replicas to add. Without this check, it could attempt to scale beyond MAX_REPLICAS or scale down below MIN_REPLICAS.

  • Now add the main function that orchestrates baseline collection, model training, and the monitoring loop:
def main():
    print("[AIOps Engine] Starting anomaly detection...")
    print(f"[AIOps Engine] Collecting {BASELINE_SAMPLES} baseline samples...")

    # Collect baseline data for model training
    baseline_data = []
    for i in range(BASELINE_SAMPLES):
        metrics = collect_metrics()
        baseline_data.append(metrics)
        print(f"  Baseline sample {i+1}/{BASELINE_SAMPLES}: RPS={metrics[0]:.2f}, P95={metrics[1]:.4f}s, Errors={metrics[2]:.4f}")
        time.sleep(CHECK_INTERVAL)

    # Train Isolation Forest model on baseline
    X_train = np.array(baseline_data)
    model = IsolationForest(
        contamination=0.1,
        random_state=42,
        n_estimators=100
    )
    model.fit(X_train)
    print("[AIOps Engine] Isolation Forest model trained on baseline data")
    print("[AIOps Engine] Monitoring for anomalies...")

    last_scale_time = 0

    # Monitor loop - detect anomalies and remediate
    while True:
        metrics = collect_metrics()
        prediction = model.predict([metrics])[0]

        if prediction == -1:
            print(f"[ANOMALY DETECTED] RPS={metrics[0]:.2f}, P95={metrics[1]:.4f}s, Errors={metrics[2]:.4f}")

            current_time = time.time()
            if current_time - last_scale_time > COOLDOWN_SECONDS:
                current_replicas = get_current_replicas()
                new_replicas = min(current_replicas + 2, MAX_REPLICAS)
                if new_replicas > current_replicas:
                    scale_deployment(new_replicas)
                    last_scale_time = current_time
                else:
                    print(f"[AIOps Engine] Already at max replicas ({MAX_REPLICAS})")
            else:
                remaining = COOLDOWN_SECONDS - (current_time - last_scale_time)
                print(f"[AIOps Engine] Cooldown active, {remaining:.0f}s remaining")
        else:
            print(f"[NORMAL] RPS={metrics[0]:.2f}, P95={metrics[1]:.4f}s, Errors={metrics[2]:.4f}")
            # Scale down if above minimum
            current_replicas = get_current_replicas()
            if current_replicas > MIN_REPLICAS:
                current_time = time.time()
                if current_time - last_scale_time > COOLDOWN_SECONDS:
                    new_replicas = max(current_replicas - 1, MIN_REPLICAS)
                    scale_deployment(new_replicas)
                    last_scale_time = current_time

        time.sleep(CHECK_INTERVAL)


if __name__ == "__main__":
    main()

How does the self-healing loop work?

The engine collects 30 baseline samples under normal traffic. It trains an Isolation Forest model with contamination=0.1, telling the algorithm to expect about 10% of data points to be anomalous.

In the monitoring loop, model.predict() returns -1 for anomalies and 1 for normal data. On anomaly, it scales up by 2 replicas (capped at 6). On normal, it scales down by 1 (floored at 2). The cooldown prevents thrashing between scale events.

  • Save anomaly_detector.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved anomaly_detector.py with all functions in place.

ⓧ I'd like to double check the full code

import time
import numpy as np
import requests
from sklearn.ensemble import IsolationForest
from kubernetes import client, config

PROMETHEUS_URL = "http://localhost:9090"
DEPLOYMENT_NAME = "aiops-demo"
NAMESPACE = "default"
MIN_REPLICAS = 2
MAX_REPLICAS = 6
COOLDOWN_SECONDS = 60
CHECK_INTERVAL = 10
BASELINE_SAMPLES = 30


def query_prometheus(query):
    response = requests.get(
        f"{PROMETHEUS_URL}/api/v1/query",
        params={"query": query}
    )
    data = response.json()
    if data["status"] == "success" and data["data"]["result"]:
        return float(data["data"]["result"][0]["value"][1])
    return 0.0


def collect_metrics():
    rps = query_prometheus('rate(flask_app_requests_total[1m])')
    latency_p95 = query_prometheus('histogram_quantile(0.95, rate(flask_app_request_latency_seconds_bucket[1m]))')
    error_rate = query_prometheus('rate(flask_app_errors_total[1m])')
    return [rps, latency_p95, error_rate]


def scale_deployment(replicas):
    config.load_kube_config()
    apps_v1 = client.AppsV1Api()
    body = {"spec": {"replicas": replicas}}
    apps_v1.patch_namespaced_deployment_scale(
        name=DEPLOYMENT_NAME,
        namespace=NAMESPACE,
        body=body
    )
    print(f"[REMEDIATION] Scaled {DEPLOYMENT_NAME} to {replicas} replicas")


def get_current_replicas():
    config.load_kube_config()
    apps_v1 = client.AppsV1Api()
    deployment = apps_v1.read_namespaced_deployment(
        name=DEPLOYMENT_NAME,
        namespace=NAMESPACE
    )
    return deployment.spec.replicas


def main():
    print("[AIOps Engine] Starting anomaly detection...")
    print(f"[AIOps Engine] Collecting {BASELINE_SAMPLES} baseline samples...")
    baseline_data = []
    for i in range(BASELINE_SAMPLES):
        metrics = collect_metrics()
        baseline_data.append(metrics)
        print(f"  Baseline sample {i+1}/{BASELINE_SAMPLES}: RPS={metrics[0]:.2f}, P95={metrics[1]:.4f}s, Errors={metrics[2]:.4f}")
        time.sleep(CHECK_INTERVAL)
    X_train = np.array(baseline_data)
    model = IsolationForest(contamination=0.1, random_state=42, n_estimators=100)
    model.fit(X_train)
    print("[AIOps Engine] Isolation Forest model trained on baseline data")
    print("[AIOps Engine] Monitoring for anomalies...")
    last_scale_time = 0
    while True:
        metrics = collect_metrics()
        prediction = model.predict([metrics])[0]
        if prediction == -1:
            print(f"[ANOMALY DETECTED] RPS={metrics[0]:.2f}, P95={metrics[1]:.4f}s, Errors={metrics[2]:.4f}")
            current_time = time.time()
            if current_time - last_scale_time > COOLDOWN_SECONDS:
                current_replicas = get_current_replicas()
                new_replicas = min(current_replicas + 2, MAX_REPLICAS)
                if new_replicas > current_replicas:
                    scale_deployment(new_replicas)
                    last_scale_time = current_time
                else:
                    print(f"[AIOps Engine] Already at max replicas ({MAX_REPLICAS})")
            else:
                remaining = COOLDOWN_SECONDS - (current_time - last_scale_time)
                print(f"[AIOps Engine] Cooldown active, {remaining:.0f}s remaining")
        else:
            print(f"[NORMAL] RPS={metrics[0]:.2f}, P95={metrics[1]:.4f}s, Errors={metrics[2]:.4f}")
            current_replicas = get_current_replicas()
            if current_replicas > MIN_REPLICAS:
                current_time = time.time()
                if current_time - last_scale_time > COOLDOWN_SECONDS:
                    new_replicas = max(current_replicas - 1, MIN_REPLICAS)
                    scale_deployment(new_replicas)
                    last_scale_time = current_time
        time.sleep(CHECK_INTERVAL)


if __name__ == "__main__":
    main()
Run the load generator to trigger self-healing

You need a way to simulate a traffic spike so the anomaly detector sees something unusual. A load generator script sends concurrent requests to your Flask service, ramping from normal traffic to a 10x spike and back.

  • Create load_generator.py inside the aiops folder with the following content:
import requests
import time
import concurrent.futures
import random

# Target the Flask service via Kind port mapping
TARGET_URL = "http://localhost:30000/api/data"
NORMAL_RPS = 5
SPIKE_RPS = 50
SPIKE_DURATION = 30
NORMAL_DURATION = 60


def send_request():
    """Send a single request to the target."""
    try:
        response = requests.get(TARGET_URL, timeout=5)
        return response.status_code
    except requests.exceptions.RequestException:
        return 0


def generate_load(rps, duration, label):
    """Generate load at specified RPS for a duration."""
    print(f"[Load Generator] {label}: {rps} RPS for {duration}s")
    end_time = time.time() + duration
    request_count = 0
    with concurrent.futures.ThreadPoolExecutor(max_workers=20) as executor:
        while time.time() < end_time:
            batch_size = max(1, rps // 10)
            futures = [executor.submit(send_request) for _ in range(batch_size)]
            concurrent.futures.wait(futures)
            request_count += batch_size
            time.sleep(0.1)
    print(f"[Load Generator] {label} complete: {request_count} requests sent")


def main():
    print("[Load Generator] Starting traffic simulation...")
    print("[Load Generator] Phase 1: Normal traffic")
    generate_load(NORMAL_RPS, NORMAL_DURATION, "Normal traffic")
    print("[Load Generator] Phase 2: Traffic spike (simulating incident)")
    generate_load(SPIKE_RPS, SPIKE_DURATION, "SPIKE")
    print("[Load Generator] Phase 3: Return to normal")
    generate_load(NORMAL_RPS, NORMAL_DURATION, "Recovery")
    print("[Load Generator] Simulation complete")


if __name__ == "__main__":
    main()

What does the load generator simulate?

  • Phase 1: 60 seconds of normal traffic at 5 requests per second. This generates baseline metrics.
  • Phase 2: A 30-second traffic spike at 50 requests per second. This is the anomaly that triggers auto-scaling.
  • Phase 3: 60 seconds of normal traffic again. The detector should scale pods back down.
  • Save load_generator.py.

✔️ Awesome, I've got everything!

Great. Make sure both anomaly_detector.py and load_generator.py are saved in the aiops folder.

ⓧ I'd like to double check the full code

import requests
import time
import concurrent.futures
import random

TARGET_URL = "http://localhost:30000/api/data"
NORMAL_RPS = 5
SPIKE_RPS = 50
SPIKE_DURATION = 30
NORMAL_DURATION = 60


def send_request():
    try:
        response = requests.get(TARGET_URL, timeout=5)
        return response.status_code
    except requests.exceptions.RequestException:
        return 0


def generate_load(rps, duration, label):
    print(f"[Load Generator] {label}: {rps} RPS for {duration}s")
    end_time = time.time() + duration
    request_count = 0
    with concurrent.futures.ThreadPoolExecutor(max_workers=20) as executor:
        while time.time() < end_time:
            batch_size = max(1, rps // 10)
            futures = [executor.submit(send_request) for _ in range(batch_size)]
            concurrent.futures.wait(futures)
            request_count += batch_size
            time.sleep(0.1)
    print(f"[Load Generator] {label} complete: {request_count} requests sent")


def main():
    print("[Load Generator] Starting traffic simulation...")
    print("[Load Generator] Phase 1: Normal traffic")
    generate_load(NORMAL_RPS, NORMAL_DURATION, "Normal traffic")
    print("[Load Generator] Phase 2: Traffic spike (simulating incident)")
    generate_load(SPIKE_RPS, SPIKE_DURATION, "SPIKE")
    print("[Load Generator] Phase 3: Return to normal")
    generate_load(NORMAL_RPS, NORMAL_DURATION, "Recovery")
    print("[Load Generator] Simulation complete")


if __name__ == "__main__":
    main()
  • Install the AIOps dependencies by running:
pip install -r requirements.txt

Install failing with dependency conflicts?

If you have conflicting packages in your environment, create a virtual environment first with python -m venv venv and activate it before re-running the install.

help me fix dependency conflicts when installing scikit-learn and kubernetes packages

Now you will run the detector and the load generator together. You need two terminal windows for this.

  • Make sure Prometheus is still port-forwarded. If not, restart it by running:
kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090 &
  • Start the anomaly detector by running:
python anomaly_detector.py

You should see baseline collection begin. The engine prints each sample as it collects metrics every 10 seconds. This takes about 5 minutes.

What should I see during baseline collection?

The terminal prints numbered samples like:

[AIOps Engine] Starting anomaly detection...
[AIOps Engine] Collecting 30 baseline samples...
  Baseline sample 1/30: RPS=0.00, P95=0.0000s, Errors=0.0000
  Baseline sample 2/30: RPS=0.00, P95=0.0000s, Errors=0.0000

The values may be zero or very low if no traffic is hitting the service yet. That is fine. The model learns whatever pattern it sees during baseline.

  • While the detector runs, open a second terminal window. Navigate back to the aiops directory and start the load generator by running:
python load_generator.py

Watch the first terminal. After the model finishes training, the monitoring loop begins. When the load generator hits Phase 2 (the spike), you should see [ANOMALY DETECTED] messages followed by a [REMEDIATION] scale-up message.

  • Verify the auto-scaling worked by running in a third terminal:
kubectl get pods -l app=aiops-demo

You should see more than 2 pods. The detector scaled the deployment up in response to the traffic spike.

  • Wait for Phase 3 (normal traffic) to complete. Check pods again by running:
kubectl get pods -l app=aiops-demo

After the cooldown expires and normal traffic resumes, the detector scales pods back down toward MIN_REPLICAS (2).

Not seeing ANOMALY DETECTED messages?

  • Make sure the load generator is targeting the correct URL. The Flask service must be exposed via a NodePort on port 30000. Check with kubectl get svc aiops-demo.
  • Confirm Prometheus is scraping metrics. Open http://localhost:9090/targets in your browser and verify the Flask app target shows as UP.
  • If the baseline had zero traffic, the model's threshold may be too sensitive or too loose. Try running the load generator during baseline collection so the model learns real traffic patterns.

help me debug why my anomaly detector isn't detecting the traffic spike

Your self-healing platform is fully operational. The detector monitors, decides, and acts. Next, you will tackle the Secret Mission to automate the entire stack setup with Ansible.

Secret mission

Ansible Playbook for Full Automation

You have built the entire AIOps platform piece by piece. Now automate the whole thing. Write an Ansible playbook that bootstraps the full stack (Floci, Kind cluster, Terraform, Docker build, app deployment, observability) in a single command. Then write a teardown playbook that destroys everything cleanly.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally using Docker containers and Kind. There are no cloud costs or ongoing charges. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Kind cluster aiops-platform (runs as Docker containers, no cost).
  • Floci container (local AWS emulator via Docker Compose, no cost).
  • Docker images (aiops-demo:latest and floci/floci:latest, local storage only).
  • Project folder aiops-platform on your Desktop.

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with the platform.

  • Your Kind cluster continues running as Docker containers in the background.
  • Floci stays available on port 4566 for further Terraform or AWS CLI testing.
  • The anomaly detector and load generator scripts can be re-run at any time.

Pause - I'll come back to this later

Stop running containers to free up memory and CPU. Your cluster state is preserved inside the stopped containers.

  • Stop the Kind cluster node by running:
  • Stop the Floci container by running:
docker stop aiops-platform-control-plane
docker compose stop

To resume later, start the containers again:

docker start aiops-platform-control-plane
docker compose start

Delete - I don't want to use this again

Remove all project resources and start fresh. This cannot be undone.

  • Delete the Kind cluster by running:
kind delete cluster --name aiops-platform
  • Stop and remove the Floci container and its volume by running:
docker compose down -v
  • Remove the Docker images by running:
docker rmi aiops-demo:latest floci/floci:latest
  • Delete the project folder from your Desktop by running:

macOS

cd ~/Desktop
rm -rf aiops-platform

Windows

cd ~\Desktop
Remove-Item -Recurse -Force aiops-platform

Nice Work!

Nice Work!

You just built a fully local AIOps platform that detects anomalies in real-time and self-heals by auto-scaling pods. This is the same pattern that modern Platform Engineering teams use to manage production infrastructure at scale.

You've learned how to:

  • Provision AWS-compatible infrastructure using Terraform against a Floci local cloud emulator, deploying a Kind cluster with a containerized Flask microservice at zero cloud cost.
  • Deploy a full observability stack (Prometheus + Grafana) using Helm and wire custom metrics from a Python Flask app into real-time dashboards.
  • Build a Python AIOps anomaly detector using Isolation Forest that monitors Prometheus metrics and auto-scales Kubernetes pods via the Kubernetes API when traffic spikes are detected.
  • Secret Mission: Automated the entire platform setup and teardown with an Ansible playbook, deploying the full stack in a single command.

Ready to quiz yourself?