Diagnose Bottlenecks with Zabbix

Use Zabbix to diagnose CPU, memory, disk, and network incidents locally.

Introduction

30 Second Summary

When a service slows down, everyone feels the delay before anyone knows why. Several parts of the system can look suspicious at the same time.

In this project, you will build a free local operations assessment lab in Zabbix for three simulated servers. A hidden incident tests whether you can distinguish CPU, memory, disk, or network pressure from synchronized evidence.

What You'll Build

You'll show a completed incident investigation where Zabbix dashboard evidence explains why app-01 slowed down.

By the end of this project, you'll have:

  • A centralized monitoring view where web-01, app-01, and db-01 report six synchronized metrics from one reusable HTTP template.
  • An incident investigation dashboard that compares CPU, memory, disk, network, and service latency on one timeline.
  • A completed incident report that connects the high-latency symptom to evidence, recovery, and a problem management decision.
  • Secret Mission: Diagnose a second hidden incident on db-01 and add a targeted trigger that exposes its bottleneck earlier next time.

Are there any prerequisites?

You'll need a 64-bit Windows 10 build 19043 or later, or Windows 11, with administrator rights, hardware virtualization, internet access, at least 6 GB of available RAM, and Visual Studio Code. The guided setup covers WSL 2, Podman Desktop, and Compose.

Before We Start

This step locks in your commitment to an evidence-based monitoring assessment before any hands-on work begins. Candidates will use synchronized signals to distinguish CPU, memory, disk, and network bottlenecks that share the same latency symptom.

Set Up the Container Environment

The monitoring lab needs isolated Linux containers. The free-only requirement makes Podman Desktop the safer default because it is free and open source.

Windows Subsystem for Linux version 2 provides the Linux environment behind the lab. You will use Windows PowerShell to verify that foundation before installing Podman Desktop 1.29.3 with Compose.

In this step, get ready to:
  • Confirm that Windows meets the operating-system, memory, virtualization, and WSL requirements.
  • Install Podman Desktop 1.29.3 with a running Podman machine.
  • Install Compose and confirm that Podman can invoke it from Windows PowerShell.
Check the Windows prerequisites

Podman Desktop needs a supported Windows host with enough memory for its Linux machine. Checking these requirements now prevents installation failures once the container engine starts.

  • Press the Windows key to open Windows search.
  • Type Windows PowerShell into the search field.
  • Choose the administrator option to start Windows PowerShell with elevated rights.
  • Approve the Windows permission prompt if it appears.
  • Check the Windows hardware details by running this command:
systeminfo

What does this check show?

The systeminfo command reports the Windows edition, build, architecture, physical memory, and virtualization support. These values determine whether the PC can host the Podman machine.

The system report is dense, so take each requirement one line at a time.

  • Confirm that the operating-system architecture is 64-bit.
  • Confirm that the PC runs Windows 10 build 19043 or later, or Windows 11.
  • Confirm that at least 6 GB of physical memory is available.
  • Confirm that the virtualization requirements show hardware virtualization support.
  • Use Windows search to confirm that Visual Studio Code is available.
  • Use Windows search to confirm that Microsoft Edge is available.
  • Open the Podman Desktop downloads page in Microsoft Edge to confirm that the browser and internet connection work.

You should see the official Windows download options. This confirms that the installer source is reachable.

Does the PC miss a requirement?

Close unused applications if less than 6 GB of memory is available. Use your PC manufacturer's instructions to enable hardware virtualization if the report shows that it is unavailable.

Pause the project if the Windows build or architecture is unsupported. Help me interpret my Windows system report.

WSL provides the Linux layer that Podman uses on Windows. The next check confirms whether version 2 is ready.

  • Check the current WSL status by running this command:
wsl --status

What does the WSL check prove?

The command asks Windows for the current WSL configuration. A result showing WSL version 2 confirms that Podman can use the required machine provider.

✔️ WSL 2 is ready

Good work. The Linux layer required by the Podman machine is installed and responding.

ⓧ WSL 2 is unavailable

Windows needs to update WSL before it can provide the Linux environment for Podman.

  • Update WSL and install its core components by running these commands:
wsl --update
wsl --install --no-distribution

What do these commands change?

The first command updates WSL. The second installs WSL without adding a separate Linux distribution because Podman creates its own machine.

  • Restart Windows when the installation finishes.
  • Return to Windows PowerShell as Administrator using the same search steps.
  • Verify the updated WSL status by running this command:
wsl --status

What confirms success?

The updated status should show WSL version 2. That result confirms that Windows is ready to host the Podman machine.

WSL still unavailable?

Confirm that you restarted Windows after the installation. Check that Windows PowerShell is running with administrator rights before retrying the commands.

If the status still fails, help me troubleshoot my WSL setup.

Install Podman Desktop and create its machine

Podman Desktop supplies the container engine and manages the Linux machine that runs it. This project uses version 1.29.3 so every learner starts from the same application release.

  • Check whether Podman Desktop is already installed by running this command:
winget list --id RedHat.Podman-Desktop --exact

What does this version check do?

The command searches the installed Windows packages for the exact Podman Desktop package identifier. Its output shows the installed version or confirms that the package is absent.

✔️ I see version 1.29.3

Podman Desktop 1.29.3 is already installed. Continue to onboarding below.

ⓧ I see an older version

The older installation needs the current Windows installer before you create the Podman machine.

  • Open the official Podman Desktop downloads page in Microsoft Edge.
  • Select the x64 or arm64 Windows installer that matches the architecture from your system report.
  • Run the downloaded installer to update Podman Desktop to version 1.29.3.
  • Confirm the updated version by running this command:
winget list --id RedHat.Podman-Desktop --exact

What confirms the update?

The package listing should now show Podman Desktop version 1.29.3. This confirms that the update completed before onboarding begins.

ⓧ Podman Desktop is not listed

Podman Desktop needs to be installed before Windows can create the local container environment. The package download can take a few minutes, so a quiet terminal during this stage is normal.

  • Install Podman Desktop by running this command:
winget install -e --id RedHat.Podman-Desktop

What does this command install?

The command asks WinGet to install the exact Podman Desktop package from its registered source. The installer adds the desktop application that manages Podman on Windows.

  • Confirm that the terminal reports a successful installation.

Installation command not working?

Use the official Windows downloads page if WinGet is unavailable. Select the installer that matches the architecture from your system report.

If the installer stops, help me troubleshoot my Podman Desktop installation.

Onboarding connects Podman Desktop to WSL and creates the machine that runs Linux containers. Windows may request permission during installation, so approve the prompt if it appears.

  • Press the Windows key to open Windows search.
  • Type Podman Desktop into the search field.
  • Press Enter to start Podman Desktop.
  • Select Start Onboarding.
  • Keep WSL 2 as the machine provider.
  • Install Podman when onboarding prompts you.
  • Create the Podman machine when onboarding reaches the machine setup.
  • Approve the Windows permission prompt if it appears.

Machine creation can take a few minutes while Podman downloads its Linux components. The machine is ready when Podman Desktop shows it as running.

  • Select Settings in the Podman Desktop sidebar.
  • Select Resources.
  • Confirm that the Podman machine is shown as running.

That is the largest setup hurdle complete. Your Windows PC now has a running Linux container engine ready for the monitoring lab.

Podman machine not running?

Confirm that WSL version 2 is available. Restart Podman Desktop after resolving any Windows permission prompt.

If the machine still does not start, help me diagnose the Podman machine.

Install and verify Compose

Compose turns a multi-container definition into one coordinated environment. Podman forwards Compose commands to an external provider that connects through the local Podman socket.

  • Return to Settings in Podman Desktop.
  • Select Resources.
  • Find the Compose tile.
  • Select Setup in the Compose tile.
  • Follow the installation prompts shown by Podman Desktop.

The Compose provider may take a few minutes to download. Keep Podman Desktop open until the setup reports that Compose is available.

  • Return to Windows PowerShell from earlier.

Before you run the final check, ask yourself whether you expect a Compose version or an error.

  • Verify the Compose connection by running this command:
podman compose version

What does the final check prove?

Podman forwards this request to the installed Compose provider through its local socket. A printed Compose version proves that the provider is installed and reachable.

You should see a Compose version instead of an error. This is the visible proof that the container environment is ready.

  • Record the printed Podman version here: your printed Podman version.
  • Record the printed Compose version here: your printed Compose version.

Compose version not appearing?

Confirm that the Podman machine still shows as running. Return to the Compose tile if setup did not finish.

Restart Podman Desktop after setup if Windows PowerShell still cannot reach Compose. Help me troubleshoot the Compose verification.

Your free container foundation is ready. Next, you will launch the Zabbix stack and see all seven lab containers running.

Launch and Monitor the Lab

Your container environment is ready. The next challenge is proving that the database, simulated servers, Zabbix interface, and monitoring checks can communicate from one local lab.

You will launch the seven services first. You will then use one HTTP response per server to collect synchronized evidence for CPU, memory, disk, network, service latency, and incident state.

In this step, get ready to:
  • Create the three lab files inside a Visual Studio Code project folder.
  • Launch the seven containers and verify healthy simulator metrics.
  • Configure the reusable Zabbix template for all three simulated servers.
Create the lab files

The project has three files with separate responsibilities. The Compose file defines the local services. The simulator publishes metrics. The report gives you a structured place for the investigation evidence.

  • Press the Windows key to open Windows search.
  • Type Visual Studio Code into the search field.
  • Press Enter to open Visual Studio Code.
  • Use the Visual Studio Code folder picker to create zabbix-performance-lab on your Desktop.
  • Select zabbix-performance-lab as the current Visual Studio Code workspace.

The Explorer sidebar now represents the lab folder. Every relative path in the Compose configuration starts from this location.

  • Create a folder named simulator inside zabbix-performance-lab using the Explorer sidebar.
  • Create compose.yaml inside zabbix-performance-lab using the Explorer sidebar.
  • Create server.py inside the simulator folder using the Explorer sidebar.
  • Create incident-report.md inside zabbix-performance-lab using the Explorer sidebar.
  • Copy each complete file from the matching reference below into its Visual Studio Code tab.

✔️ Awesome, I've got everything!

Great. Save all three files before you launch the lab.

ⓧ I'd like to double check the full code

Check compose.yaml
name: zabbix-performance-lab

services:
  postgres:
    image: docker.io/library/postgres:18.6-alpine3.24
    container_name: zabbix-performance-db
    environment:
      POSTGRES_USER: zabbix
      POSTGRES_PASSWORD: zabbix-lab-password
      POSTGRES_DB: zabbix
    volumes:
      - postgres-data:/var/lib/postgresql
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U zabbix -d zabbix"]
      interval: 5s
      timeout: 3s
      retries: 20

  zabbix-server:
    image: docker.io/zabbix/zabbix-server-pgsql:alpine-7.4.14
    container_name: zabbix-performance-server
    environment:
      DB_SERVER_HOST: postgres
      POSTGRES_USER: zabbix
      POSTGRES_PASSWORD: zabbix-lab-password
      POSTGRES_DB: zabbix
    depends_on:
      postgres:
        condition: service_healthy

  zabbix-web:
    image: docker.io/zabbix/zabbix-web-nginx-pgsql:alpine-7.4.14
    container_name: zabbix-performance-web
    environment:
      ZBX_SERVER_HOST: zabbix-server
      DB_SERVER_HOST: postgres
      POSTGRES_USER: zabbix
      POSTGRES_PASSWORD: zabbix-lab-password
      POSTGRES_DB: zabbix
      PHP_TZ: UTC
    ports:
      - "127.0.0.1:8080:8080"
    depends_on:
      - zabbix-server

  sink:
    image: docker.io/library/python:3.14.7-alpine3.24
    container_name: zabbix-performance-sink
    command: ["python", "/app/server.py"]
    environment:
      MODE: sink
      SERVER_NAME: sink
    volumes:
      - type: bind
        source: ./simulator/server.py
        target: /app/server.py
        read_only: true

  web-01:
    image: docker.io/library/python:3.14.7-alpine3.24
    container_name: web-01
    command: ["python", "/app/server.py"]
    environment:
      SERVER_NAME: web-01
      NETWORK_SINK_HOST: sink
      INCIDENT_SECONDS: "300"
    ports:
      - "127.0.0.1:9101:8080"
    volumes:
      - type: bind
        source: ./simulator/server.py
        target: /app/server.py
        read_only: true
    depends_on:
      - sink

  app-01:
    image: docker.io/library/python:3.14.7-alpine3.24
    container_name: app-01
    command: ["python", "/app/server.py"]
    environment:
      SERVER_NAME: app-01
      NETWORK_SINK_HOST: sink
      INCIDENT_SECONDS: "300"
    ports:
      - "127.0.0.1:9102:8080"
    volumes:
      - type: bind
        source: ./simulator/server.py
        target: /app/server.py
        read_only: true
    depends_on:
      - sink

  db-01:
    image: docker.io/library/python:3.14.7-alpine3.24
    container_name: db-01
    command: ["python", "/app/server.py"]
    environment:
      SERVER_NAME: db-01
      NETWORK_SINK_HOST: sink
      INCIDENT_SECONDS: "300"
    ports:
      - "127.0.0.1:9103:8080"
    volumes:
      - type: bind
        source: ./simulator/server.py
        target: /app/server.py
        read_only: true
    depends_on:
      - sink

networks:
  default:
    name: zabbix-performance-network

volumes:
  postgres-data:
    name: zabbix-performance-data

What does this file define?

The services section defines PostgreSQL, the Zabbix server, the web interface, the network sink, and three simulator containers.

The final sections give the local network and persistent database volume stable names. Those names make the lab easy to verify and clean up.

Check incident-report.md
# Performance Incident Report

Incident ID:
Candidate:
Date and time:
Affected host:
Detection source: Zabbix

## Impact

Observed service symptom:
Start time:
Recovery time:
User or service impact:

## Evidence

| Signal | Healthy baseline | Incident peak | Timing relationship | Primary cause? |
| --- | ---: | ---: | --- | --- |
| CPU utilization | | | | Yes / No |
| Memory utilization | | | | Yes / No |
| Disk write throughput | | | | Yes / No |
| Network throughput | | | | Yes / No |
| Service latency | | | | Symptom |

## Diagnosis

Primary bottleneck: CPU / memory / disk / network
Reasoning:
Why the other three resources were ruled out:

## Response

Acknowledgment comment:
Containment or remediation action:
Recovery evidence:

## Problem Management Decision

Classification: Incident only / Open a problem investigation
Evidence of recurrence currently available:
Evidence still needed to prove recurrence:
Recommended preventive action:

## Final Summary

Write a concise technical summary connecting the symptom, evidence, root cause, remediation, and prevention recommendation.

How will the report help?

The report separates impact, evidence, diagnosis, response, and problem-management decisions. This structure keeps your final conclusion tied to measurements.

Check simulator/server.py
import http.client
import json
import os
import secrets
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

SERVER_NAME = os.getenv("SERVER_NAME", "simulated-server")
MODE = os.getenv("MODE", "server")
NETWORK_SINK_HOST = os.getenv("NETWORK_SINK_HOST", "sink")
NETWORK_SINK_PORT = 8080
INCIDENT_SECONDS = int(os.getenv("INCIDENT_SECONDS", "300"))
MEGABYTE = 1024 * 1024
MEMORY_CAPACITY_BYTES = 128 * MEGABYTE
MEMORY_LOAD_MB = 96
DISK_CHUNK = b"d" * (256 * 1024)
NETWORK_CHUNK = b"n" * (256 * 1024)
INCIDENT_TYPES = ("cpu", "memory", "disk", "network")


class LabState:
    def __init__(self):
        self.lock = threading.Lock()
        self.stop_event = None
        self.incident_id = None
        self.incident_active = False
        self.memory_load_bytes = 0
        self.disk_bytes_total = 0
        self.network_bytes_total = 0
        self.cpu_percent = 0.0
        self.disk_write_mbps = 0.0
        self.network_mbps = 0.0
        self.last_sample_time = time.monotonic()
        self.last_process_time = time.process_time()
        self.last_disk_bytes = 0
        self.last_network_bytes = 0
        threading.Thread(target=self.sample_metrics, daemon=True).start()

    def sample_metrics(self):
        while True:
            time.sleep(1)
            now = time.monotonic()
            process_now = time.process_time()
            with self.lock:
                elapsed = max(now - self.last_sample_time, 0.001)
                self.cpu_percent = min(
                    100.0,
                    max(0.0, (process_now - self.last_process_time) / elapsed * 100.0),
                )
                self.disk_write_mbps = (
                    self.disk_bytes_total - self.last_disk_bytes
                ) / elapsed / MEGABYTE
                self.network_mbps = (
                    self.network_bytes_total - self.last_network_bytes
                ) * 8 / elapsed / 1_000_000
                self.last_sample_time = now
                self.last_process_time = process_now
                self.last_disk_bytes = self.disk_bytes_total
                self.last_network_bytes = self.network_bytes_total

    def start_incident(self):
        self.stop_incident()
        stop_event = threading.Event()
        incident_type = secrets.choice(INCIDENT_TYPES)
        incident_id = secrets.token_hex(4)
        with self.lock:
            self.stop_event = stop_event
            self.incident_id = incident_id
            self.incident_active = True

        workers = {
            "cpu": self.cpu_worker,
            "memory": self.memory_worker,
            "disk": self.disk_worker,
            "network": self.network_worker,
        }
        threading.Thread(
            target=workers[incident_type], args=(stop_event,), daemon=True
        ).start()
        threading.Thread(
            target=self.auto_stop, args=(stop_event,), daemon=True
        ).start()
        return incident_id

    def auto_stop(self, stop_event):
        if not stop_event.wait(INCIDENT_SECONDS):
            self.stop_incident(stop_event)

    def stop_incident(self, expected_event=None):
        with self.lock:
            if expected_event is not None and self.stop_event is not expected_event:
                return False
            event = self.stop_event
            was_active = self.incident_active
            self.stop_event = None
            self.incident_id = None
            self.incident_active = False
            self.memory_load_bytes = 0
        if event is not None:
            event.set()
        return was_active

    def cpu_worker(self, stop_event):
        value = 1
        while not stop_event.is_set():
            for _ in range(10000):
                value = (value * value + 1) % 10000019

    def memory_worker(self, stop_event):
        blocks = []
        try:
            for _ in range(MEMORY_LOAD_MB):
                if stop_event.is_set():
                    break
                blocks.append(bytearray(MEGABYTE))
                with self.lock:
                    if self.stop_event is stop_event:
                        self.memory_load_bytes = len(blocks) * MEGABYTE
                stop_event.wait(0.02)
            stop_event.wait()
        finally:
            blocks.clear()
            with self.lock:
                if self.stop_event is stop_event:
                    self.memory_load_bytes = 0

    def disk_worker(self, stop_event):
        path = "/tmp/performance-lab.bin"
        try:
            with open(path, "wb", buffering=0) as output:
                while not stop_event.is_set():
                    written = output.write(DISK_CHUNK)
                    os.fsync(output.fileno())
                    if output.tell() >= 16 * MEGABYTE:
                        output.seek(0)
                    with self.lock:
                        self.disk_bytes_total += written
                    stop_event.wait(0.05)
        finally:
            try:
                os.remove(path)
            except FileNotFoundError:
                pass

    def network_worker(self, stop_event):
        connection = None
        try:
            while not stop_event.is_set():
                try:
                    if connection is None:
                        connection = http.client.HTTPConnection(
                            NETWORK_SINK_HOST, NETWORK_SINK_PORT, timeout=2
                        )
                    connection.request(
                        "POST",
                        "/sink",
                        body=NETWORK_CHUNK,
                        headers={"Content-Type": "application/octet-stream"},
                    )
                    response = connection.getresponse()
                    response.read()
                    if response.status != 204:
                        raise RuntimeError("Network sink returned an unexpected status")
                    with self.lock:
                        self.network_bytes_total += len(NETWORK_CHUNK)
                    stop_event.wait(0.02)
                except (OSError, http.client.HTTPException):
                    if connection is not None:
                        connection.close()
                    connection = None
                    stop_event.wait(0.2)
        finally:
            if connection is not None:
                connection.close()

    def snapshot(self):
        with self.lock:
            memory_percent = min(
                95.0,
                15.0 + (self.memory_load_bytes / MEMORY_CAPACITY_BYTES * 100.0),
            )
            active = self.incident_active
            return {
                "server": SERVER_NAME,
                "cpu_percent": round(self.cpu_percent, 2),
                "memory_percent": round(memory_percent, 2),
                "disk_write_mbps": round(self.disk_write_mbps, 2),
                "network_mbps": round(self.network_mbps, 2),
                "service_latency_ms": 450 if active else 25,
                "incident_active": 1 if active else 0,
            }

    def status(self):
        with self.lock:
            return {
                "server": SERVER_NAME,
                "incident_id": self.incident_id,
                "active": self.incident_active,
            }


STATE = None if MODE == "sink" else LabState()


class RequestHandler(BaseHTTPRequestHandler):
    def send_json(self, status_code, payload):
        body = json.dumps(payload).encode("utf-8")
        self.send_response(status_code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def do_GET(self):
        if MODE == "sink" and self.path == "/health":
            self.send_json(200, {"status": "ok"})
            return
        if MODE != "sink" and self.path == "/metrics":
            self.send_json(200, STATE.snapshot())
            return
        if MODE != "sink" and self.path == "/health":
            metrics = STATE.snapshot()
            time.sleep(metrics["service_latency_ms"] / 1000)
            self.send_json(200, {"status": "ok", "server": SERVER_NAME})
            return
        self.send_json(404, {"error": "not found"})

    def do_POST(self):
        if MODE == "sink" and self.path == "/sink":
            length = int(self.headers.get("Content-Length", "0"))
            if length:
                self.rfile.read(length)
            self.send_response(204)
            self.send_header("Content-Length", "0")
            self.end_headers()
            return
        if MODE != "sink" and self.path == "/incident/start":
            incident_id = STATE.start_incident()
            self.send_json(
                200,
                {
                    "status": "started",
                    "incident_id": incident_id,
                    "target": SERVER_NAME,
                    "duration_seconds": INCIDENT_SECONDS,
                },
            )
            return
        if MODE != "sink" and self.path == "/incident/stop":
            STATE.stop_incident()
            self.send_json(200, {"status": "stopped", "target": SERVER_NAME})
            return
        if MODE != "sink" and self.path == "/incident/status":
            self.send_json(200, STATE.status())
            return
        self.send_json(404, {"error": "not found"})

    def log_message(self, format, *args):
        return


if __name__ == "__main__":
    server = ThreadingHTTPServer(("0.0.0.0", 8080), RequestHandler)
    print(f"{SERVER_NAME} listening on port 8080 in {MODE} mode", flush=True)
    server.serve_forever()

How does the simulator work?

The simulator records CPU time, allocated memory, written bytes, and transferred bytes. Its metrics endpoint returns those measurements as one JSON document.

The incident endpoints start or stop one hidden load type. The response exposes an incident ID without exposing the selected bottleneck.

  • Save compose.yaml.
  • Save simulator/server.py.
  • Save incident-report.md.
  • Confirm the Explorer sidebar shows all three files in their specified locations.

Your lab definition is complete. The three files now provide the exact services, telemetry behavior, and evidence structure needed for the assessment.

Files in the wrong location?

Check that server.py sits inside the simulator folder. The bind mount in compose.yaml depends on that relative path.

Make sure Windows has not added a hidden .txt extension to any filename.

Still unsure? Help me check my Zabbix lab file structure.

Launch the local stack

The Compose definition turns the three files into a working monitoring environment. Its dependency rules wait for PostgreSQL readiness before the Zabbix server starts.

  • Open a PowerShell terminal in Visual Studio Code for the zabbix-performance-lab workspace.

The first launch downloads four container images. Expect this run to take several minutes while Podman stores those images locally.

  • Create the network, volume, and seven containers by running:
podman compose --file compose.yaml up --detach

What does this command create?

Compose creates zabbix-performance-network for service discovery. It creates zabbix-performance-data for persistent PostgreSQL data.

The detached mode leaves the seven services running after the command returns. Your terminal remains available for verification commands.

Before you inspect the result, how many running services do you expect the complete lab to need?

  • List the running containers by running:
podman ps

What does the container list prove?

The list confirms which containers are running. It also shows the host port mappings for the Zabbix frontend and three simulators.

You should see zabbix-performance-db, zabbix-performance-server, zabbix-performance-web, zabbix-performance-sink, web-01, app-01, and db-01 with a running status.

  • Press the Windows key to open Windows search.
  • Type Microsoft Edge into the search field.
  • Press Enter to open Microsoft Edge.
  • Enter http://localhost:8080 in the address bar.
  • Enter Admin in the user name field.
  • Enter zabbix in the password field.
  • Select Sign in.

You should reach the Zabbix interface. That screen proves the database, server, and web frontend completed their startup path.

  • Retrieve the current app-01 metrics from the Visual Studio Code PowerShell terminal by running:
Invoke-RestMethod -Uri http://127.0.0.1:9102/metrics

What does this response show?

PowerShell converts the simulator's JSON response into a readable object. Each property represents one synchronized sample from app-01.

The incident-state field separates a healthy baseline from an active load test. A value of 0 means the simulator is currently healthy.

You should see app-01 as the server. You should also see CPU, memory, disk, network, service-latency, and incident-state fields. The incident_active value should be 0.

Lab not responding?

If fewer than seven containers are running, compare compose.yaml with the full reference. YAML indentation controls which settings belong to each service.

If the sign-in page is still loading, give the database and Zabbix server another minute to finish their first initialization.

Need help with the failed check? Help me troubleshoot my local Zabbix containers.

Configure synchronized monitoring

A master item fetches the complete simulator payload once. Six dependent items reuse that payload, so every metric comes from the same sample.

Each dependent item uses JSONPath preprocessing to select one property. This bulk-collection pattern reduces repeated requests while preserving aligned evidence.

  • Return to the signed-in Zabbix interface.
  • Select Data collection.
  • Select Templates.
  • Select Create template.
  • Enter Performance Simulator by HTTP as the template name.
  • Select Templates as the template group.
  • Save the template.

The template now gives you one reusable configuration surface. Changes made here can flow to all three hosts.

  • Open the template's Items list.
  • Select Create item.
  • Enter Simulator metrics as the item name.
  • Select HTTP agent as the item type.
  • Enter sim.metrics as the key.
  • Enter http://{HOST.HOST}:8080/metrics as the URL.
  • Set the request type to GET.
  • Set the required status code to 200.

Why use the host name macro?

The {HOST.HOST} macro resolves to the current Zabbix host name. Linking the template to app-01 therefore sends the request to the simulator service with that same name.

HTTP agent checks run from the Zabbix server. The simulated hosts do not need agent interfaces.

  • Set the update interval to 5s.
  • Set the type of information to Text.
  • Save the master item.

The Items list should now show Simulator metrics with key sim.metrics.

Dependent Item Settings

  • CPU utilization uses key sim.cpu.pct. Its information type is Numeric (float). Its units are %. Its JSONPath is $.cpu_percent.
  • Memory utilization uses key sim.memory.pct. Its information type is Numeric (float). Its units are %. Its JSONPath is $.memory_percent.
  • Disk write throughput uses key sim.disk.write.mbps. Its information type is Numeric (float). Its units are MBps. Its JSONPath is $.disk_write_mbps.
  • Network throughput uses key sim.network.mbps. Its information type is Numeric (float). Its units are Mbps. Its JSONPath is $.network_mbps.
  • Service latency uses key sim.latency.ms. Its information type is Numeric (float). Its units are ms. Its JSONPath is $.service_latency_ms.
  • Incident active uses key sim.incident.active. Its information type is Numeric (unsigned). It has no units. Its JSONPath is $.incident_active.
  • Select Create item in the template's Items list.
  • Enter the first item name from the reference.
  • Select Dependent item as the item type.
  • Select Simulator metrics as the master item.
  • Enter the matching key from the reference.
  • Set the matching information type from the reference.
  • Enter the matching units when the reference provides them.
  • Add the matching JSONPath value as a preprocessing step.
  • Save the dependent item.
  • Repeat the dependent-item workflow for the remaining five reference entries.

The template's Items list should now contain the master item plus six dependent items. Each dependent item should show Simulator metrics as its master.

  • Open the template's Triggers list.
  • Select Create trigger.
  • Enter Service latency high on {HOST.NAME} as the trigger name.
  • Enter last(/Performance Simulator by HTTP/sim.latency.ms)>300 as the expression.
  • Select Average as the severity.
  • Save the trigger.

Why does the trigger only name latency?

The trigger reports the service symptom once the latest latency value exceeds 300 milliseconds. It leaves the underlying resource unnamed.

That ambiguity forces the investigation to correlate CPU, memory, disk, and network evidence before choosing a root cause.

  • Select Data collection.
  • Select Hosts.
  • Select Create host.
  • Enter web-01 as the host name.
  • Place the host in a group named Performance Lab.
  • Link Performance Simulator by HTTP in the host's Templates field.
  • Leave the host without an agent interface.
  • Save the host.

The first host now has the full monitoring design through one template link. The HTTP check uses its host name to reach the matching simulator.

  • Create app-01 in the Performance Lab group using the same template link.
  • Create db-01 in the Performance Lab group using the same template link.

Zabbix now has three monitored hosts with no agent interfaces. The shared template supplies every item and the latency trigger.

Before you check the collected values, do you expect every host to show the same six metric categories?

  • Select Monitoring.
  • Select Latest data.
  • Filter the results by host group Performance Lab.
  • Allow up to 60 seconds for the first values to arrive.

You should see recent CPU utilization, memory utilization, disk write throughput, network throughput, service latency, and incident active values for web-01, app-01, and db-01.

That is the monitoring path working end to end. One payload per host now feeds six synchronized diagnostic signals in Zabbix.

Latest data still empty?

Wait the full 60 seconds before changing the configuration. Zabbix may need that time to refresh newly created hosts and items.

Check that each host name exactly matches its simulator service name. The {HOST.HOST} macro depends on web-01, app-01, and db-01.

Still missing values? Help me troubleshoot my Zabbix HTTP and dependent items.

Your lab is live and collecting synchronized evidence from all three servers. Next, you can use those signals to investigate a hidden performance incident without being handed the cause.

Investigate and Resolve the Incident

Your reusable Zabbix template now collects six synchronized metrics from all three simulated servers.

A latency alert proves that a service is degraded. Metric correlation identifies which resource changed during the same time window.

You will investigate a hidden incident on app-01. Your evidence will guide the recovery response.

In this step, get ready to:
  • Build an incident dashboard that aligns resource metrics with service latency.
  • Diagnose one hidden CPU, memory, disk, or network bottleneck from synchronized evidence.
  • Acknowledge the problem, restore service, and complete the incident record.
Build the investigation dashboard

An incident dashboard aligns the service symptom with its possible resource causes. The shared timeline makes your diagnosis testable.

  • Return to your signed-in Zabbix session in Microsoft Edge.
  • Click Create dashboard.
  • Enter Performance Incident Console as the dashboard name.
  • Click Add widget.
  • Select Graph as the widget type.
  • Choose Item list as the data set.
  • Add the app-01 CPU utilization, Memory utilization, Disk write throughput, Network throughput, and Service latency items.
  • Click Apply.

You should see five lines sharing the same time axis. Each line represents one candidate cause or the service-latency symptom.

  • Click Add widget again.
  • Select Problems as the widget type.
  • Filter the widget to the Performance Lab host group.
  • Click Apply.
  • Click Save changes.

Your dashboard now combines performance evidence with the current problem state. This is the single view you will use during the investigation.

  • Set the dashboard time period to the most recent interval.
  • Observe the five healthy lines before generating load.
  • Switch back to incident-report.md in the Visual Studio Code window from earlier.
  • Enter the current healthy values in the Healthy baseline column.

You should see low resource activity on app-01. Service latency should sit close to its healthy baseline.

Trigger and diagnose the hidden incident

A defensible diagnosis depends on timing. The primary resource should rise during the same interval as service latency.

  • Switch back to the Windows PowerShell terminal from earlier.

Before you start the incident, consider whether the alert will identify a specific resource.

  • Start the hidden incident on app-01 by running this command:
Invoke-RestMethod -Method Post -Uri http://127.0.0.1:9102/incident/start

What does this command do?

The command sends a POST request to the simulator's incident endpoint. This starts one hidden load worker on app-01.

The response contains status, incident_id, target, and duration_seconds. The response omits the selected resource.

Incident request failed?

Return to Podman Desktop and confirm that app-01 is still running. Retry the request from the PowerShell terminal connected to your lab.

Still stuck? Help me diagnose why the incident start request cannot reach app-01.

  • Complete the report header with the returned incident ID, current time, affected host, candidate name, and detection source.
  • Use only the dashboard and Problems page as evidence during the diagnosis.
  • Refresh the Performance Incident Console dashboard.

You should see service latency climb while one resource line rises during the same interval. The other three resource lines should remain closer to their baselines.

  • Select Monitoring in the Zabbix navigation.
  • Select Problems.
  • Refresh the Problems page.

You should see Service latency high on app-01 in the problem list. The alert confirms service impact without identifying the resource bottleneck.

  • Align the graph time window with the start of the latency increase.
  • Compare CPU utilization with its healthy baseline.
  • Compare Memory utilization with its healthy baseline.
  • Compare Disk write throughput with its healthy baseline.
  • Compare Network throughput with its healthy baseline.
  • Select exactly one primary classification from CPU, memory, disk, or network.
  • Complete every row in the Evidence table with the baseline, peak, timing relationship, and primary-cause decision.
  • Complete the Diagnosis section with your classification, reasoning, root cause, and three rule-outs.

Your diagnosis should name the single resource whose spike overlaps the latency increase. The three rule-outs should cite their stable or unrelated timelines.

Recover service and complete the record

Acknowledgment records that someone owns the investigation. Recovery evidence proves that the chosen response restored the affected service.

  • Return to the Problems page from earlier.
  • Click Update for the active problem.
  • Enter a concise investigation comment in the Message field.
  • Select Acknowledge.
  • Click Update to apply the response.

The active event now has your investigation comment and acknowledgment recorded.

Before you stop the incident, consider how the latency line and primary resource line should respond.

  • Stop the hidden incident on app-01 by running this command:
Invoke-RestMethod -Method Post -Uri http://127.0.0.1:9102/incident/stop

What does this command do?

The command sends a POST request to the simulator's stop endpoint. The simulator ends the active resource load.

The response contains a stopped status plus the app-01 target. Zabbix then receives healthy values during its next checks.

Incident will not stop?

Confirm that app-01 remains running in Podman Desktop. Retry the stop request from the same PowerShell terminal.

Need help? Help me restore the app-01 simulator after the stop request failed.

  • Refresh the incident dashboard after the next polling cycle.

You should see service latency return toward its healthy baseline. The resource that supported your diagnosis should fall toward its baseline during the same period.

  • Return to the Problems page from earlier.
  • Refresh the Problems page after the healthy values arrive.

You should see the high-latency problem return to OK. This confirms that the service recovered after the load stopped.

  • Switch back to incident-report.md.
  • Complete the Impact section with the symptom, start time, recovery time, and service impact.
  • Complete the Response section with the acknowledgment comment, remediation action, and recovery evidence.
  • Review the report timeline for a clear progression from detection to recovery.
  • Complete the Problem Management Decision section with either Incident only or Open a problem investigation.
  • Review the decision for specific recurrence evidence that would justify a problem investigation.
  • Write the Final Summary to connect the symptom, evidence, root cause, remediation, and prevention recommendation.
  • Save incident-report.md.

Well done. You turned a symptom-only alert into an evidence-backed diagnosis.

Secret mission

Detect a Recurring Bottleneck

A second blind incident tests whether your first diagnosis was repeatable. You will turn fresh evidence from db-01 into a targeted trigger. Then you will decide whether the pattern deserves problem-management follow-up.

Clean Up Your Resources

Clean Up Your Resources

Your local Zabbix lab uses free tools, so keeping it does not create cloud charges. Decide whether to keep the lab running, pause its containers, or remove its local resources.

Resources you used:

  • Seven local Podman containers named zabbix-performance-db, zabbix-performance-server, zabbix-performance-web, zabbix-performance-sink, web-01, app-01, and db-01.
  • One named local network called zabbix-performance-network.
  • One named persistent volume called zabbix-performance-data.
  • Four cached container images named docker.io/zabbix/zabbix-server-pgsql:alpine-7.4.14, docker.io/zabbix/zabbix-web-nginx-pgsql:alpine-7.4.14, docker.io/library/postgres:18.6-alpine3.24, and docker.io/library/python:3.14.7-alpine3.24.
  • One local project folder named zabbix-performance-lab containing compose.yaml, simulator\server.py, and incident-report.md.

Keep everything running

No action is needed. Choose this option if you want to run more incidents or review your monitoring configuration.

  • The seven containers remain active.
  • The Zabbix web interface remains available at http://localhost:8080.
  • The zabbix-performance-data volume retains your dashboard, hosts, items, triggers, and incident history.
  • The zabbix-performance-lab folder retains your simulator code and completed incident report.

Pause - I'll come back to this later

Pausing frees the memory used by the running containers. Your monitoring configuration, project files, network, volume, and cached images remain available for later.

  • Return to the VS Code PowerShell terminal from earlier.
  • Stop the seven lab containers by running this command:
podman stop zabbix-performance-web zabbix-performance-server zabbix-performance-db zabbix-performance-sink web-01 app-01 db-01

Containers Do Not Stop?

This command stops each named container. The network, volume, cached images, and project files stay in place.

  • Confirm that the Podman machine from earlier is running.
  • Run the stop command again after the machine is ready.

Help me troubleshoot a Podman container that will not stop.

  • Switch back to the VS Code PowerShell terminal in zabbix-performance-lab when you want to resume.
  • Restart the complete lab by running this command:
podman compose --file compose.yaml up --detach

Lab Does Not Restart?

This command starts the services defined in compose.yaml. The existing volume restores your Zabbix configuration and monitoring history.

  • Confirm that the terminal is still inside the zabbix-performance-lab folder.
  • Confirm that the Podman machine is running.

Help me troubleshoot why my Zabbix Compose lab will not restart.

Delete - I don't want to use this again

Deleting the lab permanently removes the Zabbix database, monitoring configuration, and local project files. Your completed incident-report.md file remains useful only if you keep a separate copy.

  • Return to the VS Code PowerShell terminal from earlier.
  • Remove the seven lab containers by running this command:
podman rm --force zabbix-performance-web zabbix-performance-server zabbix-performance-db zabbix-performance-sink web-01 app-01 db-01

Containers Remain?

This command force-removes all seven lab containers. Removing them releases the local network and persistent volume for the next cleanup actions.

  • Confirm that the Podman machine is running.
  • Run the removal command again if one of the containers was still stopping.

Help me troubleshoot a Podman container that cannot be removed.

  • Remove the named lab network by running this command:
podman network rm zabbix-performance-network

Network Removal Fails?

This command deletes the network that connected the Zabbix services and simulators. The network can be removed after its containers are gone.

  • Confirm that all seven lab containers were removed.
  • Run the network removal command again after removing any remaining container.

Help me troubleshoot why the Podman network cannot be removed.

  • Delete the persistent Zabbix data by running this command:
podman volume rm zabbix-performance-data

Volume Removal Fails?

This command deletes the volume that stores the Zabbix database. Your hosts, dashboard, triggers, configuration, and monitoring history are removed with it.

  • Confirm that the database container was removed.
  • Run the volume removal command again after removing any remaining lab container.

Help me troubleshoot why the Podman volume cannot be removed.

  • Free the cached image space by running this optional command:
podman rmi docker.io/zabbix/zabbix-server-pgsql:alpine-7.4.14 docker.io/zabbix/zabbix-web-nginx-pgsql:alpine-7.4.14 docker.io/library/postgres:18.6-alpine3.24 docker.io/library/python:3.14.7-alpine3.24

Images Remain?

This command removes the four cached images downloaded for the lab. Keeping them has no cloud cost, but removing them frees local disk space.

  • Confirm that all seven lab containers were removed.
  • Run the image removal command again after removing any remaining container that uses an image.

Help me troubleshoot a Podman image that remains in use.

  • Use File Explorer to delete the zabbix-performance-lab folder from the location where you created it.

The local project files are removed after you delete the folder. Any cached images remain available only if you skipped the optional image cleanup.

Nice Work!

Nice Work!

Strong finish! Your local Zabbix lab now supports an evidence-based response from degraded-service detection through recovery.

You've learned how to:

  • Build centralized monitoring for web-01, app-01, and db-01. A reusable HTTP template uses one JSON payload to supply six synchronized metrics.
  • Create a performance dashboard that compares CPU, memory, disk, network, and service latency across one timeline. Use a symptom-focused alert to detect degraded service without exposing the underlying bottleneck.
  • Correlate synchronized evidence to produce a defensible bottleneck diagnosis. Acknowledge the incident in Zabbix. Document the remediation, recovery evidence, and problem management decision in incident-report.md.
  • Secret Mission: Diagnosed a second hidden incident on db-01. Added a targeted preventive trigger for the affected resource. Compared both incidents to recommend preventive action.

Ready to quiz yourself?