Diagnose a Failing Checkout Service

Build a local AI agent that diagnoses checkout failures from live telemetry.

Introduction

30 Second Summary

A checkout can fail while its error counter only tells you that customers are stuck. Finding the cause means following the evidence left by each request.

In this project, you will build Atlas, a local AI agent that investigates a failing checkout service. Atlas uses qwen3:4b through Ollama to query Prometheus metrics plus Loki logs before reporting an inventory timeout with evidence.

What You'll Build

Ask Atlas why a deliberately broken checkout is failing to watch it call live telemetry tools before returning the inventory timeout with evidence.

By the end of this project, you'll have:

  • Switch between healthy traffic and a repeatable checkout incident while each run reports its successful requests or errors.
  • Follow checkout activity across metrics, logs, and traces inside Grafana using Tempo for trace inspection.
  • Run the Atlas CLI to see each tool call before receiving an incident summary with a root cause, supporting evidence, and a recommended next action.
  • Secret Mission: clear the incident data and prove Atlas reports a healthy state without inventing a root cause.

Are there any prerequisites?

You need Windows 10 22H2 or newer with Docker Desktop running Linux containers.

The guide assumes Git plus Visual Studio Code are already installed. Ollama setup plus the repository dependencies are covered.

Keep internet access available for the container images plus the 2.5 GB model download.

Before We Start

Before the hands-on work begins you will commit to the local incident investigator you are building. Atlas investigates a local checkout-service incident by collecting real Prometheus metrics plus Loki log evidence before naming a root cause.

Prepare Ollama and the Repository

Atlas needs a local model runtime before it can investigate telemetry. Ollama provides that runtime on your Windows host.

The later Atlas container must reach Ollama through Docker Desktop. A clean Git repository in Visual Studio Code gives every service a predictable home.

In this step, get ready to:
  • Verify Docker Desktop, Docker Compose, Git, Visual Studio Code, and Ollama from Windows PowerShell.
  • Configure Ollama for container access before testing the qwen3:4b model.
  • Create the Atlas Git repository with its folders and ignore rules.
Prepare the local toolchain

Atlas spans your Windows host plus several Linux containers. Verifying each command now keeps missing tools from interrupting the incident lab.

  • Press the Windows key to open Windows search.
  • Type Docker Desktop into Windows search.
  • Press Enter to open Docker Desktop.
  • Wait for Docker Desktop to report that its engine is running.
  • Confirm that Docker Desktop is using Linux containers.
  • Use Docker Desktop's container-mode control to switch modes if Windows containers are active.
  • Press the Windows key to return to Windows search.
  • Type Windows PowerShell into Windows search.
  • Press Enter to open Windows PowerShell.
  • Check the preinstalled command-line tools by running:
docker --version
docker compose version
git --version
code --version

What do these checks prove?

  • The first result confirms that PowerShell can reach the Docker command-line interface.
  • The second result confirms that Docker Compose is available.
  • The third result confirms that Git can manage the repository.
  • The final result confirms that PowerShell can launch Visual Studio Code.

You should see version information after every command. This proves the core tools are reachable from the same PowerShell session.

Does a tool check fail?

  • Restart Docker Desktop if either Docker command fails.
  • Close PowerShell if Git or Visual Studio Code was installed while that window was open.
  • Open a fresh PowerShell window through Windows search before repeating the checks.

Still stuck? Help me identify which local tool PowerShell cannot reach.

Ollama runs the model that Atlas uses for its investigation. Check whether the Ollama command is already available before installing anything.

  • Check whether Ollama is installed by running:
ollama --version

What does this check prove?

The command prints the Ollama CLI version when the installation is available to your current PowerShell session.

✔️ Ollama prints a version

Ollama is installed. Your current PowerShell session can reach its command-line interface.

ⓧ Ollama is not installed

The Windows installer sets up Ollama without requiring Administrator rights. Install it before continuing with the container-access configuration.

  • Download OllamaSetup.exe from the official Ollama Windows page.
  • Run OllamaSetup.exe from your Downloads folder.
  • Complete the installation window.
  • Close the current PowerShell window.
  • Open a fresh PowerShell window through Windows search.
  • Confirm the installation by running:
ollama --version

What does this result confirm?

A printed version confirms that Windows can now find the Ollama command.

Still cannot find Ollama?

Confirm that the installer finished before you opened the new PowerShell window.

If the command still fails, help me troubleshoot the Ollama installation.

Why change the bind address?

Ollama binds to its local loopback address by default. The Windows user variable OLLAMA_HOST changes where it listens.

The value 0.0.0.0:11434 lets a later Docker container connect to Ollama. Keep this configuration on a trusted development machine because it exposes Ollama beyond its default local-only binding.

  • Use Ollama's taskbar icon to quit the application.
  • Press the Windows key to open Windows search.
  • Search for environment variables.
  • Open the Windows environment-variable editor for your user account.

You should see a section containing the variables assigned to your Windows user. This is where Ollama reads its host configuration when it starts.

  • Create a Windows user variable named OLLAMA_HOST with the value 0.0.0.0:11434.
  • Save the environment variable.

The user-variable list should now show OLLAMA_HOST with the value 0.0.0.0:11434.

  • Restart Ollama through Windows search.
  • Switch back to the PowerShell window from earlier.
  • Confirm that Ollama responds after the restart by running:
ollama --version

What does the restart check prove?

A version response confirms that the Ollama command remains reachable after the environment change.

That configuration is in place. The future Atlas container now has a host address it can use to reach Ollama.

Download and test qwen3:4b

Atlas uses the qwen3:4b model for its tool-calling loop. A short conversation confirms that Ollama can load the model before Docker enters the picture.

The first run downloads a 2.5 GB model, so allow extra time while the progress indicators update.

  • Download the model before starting its interactive chat by running:
ollama run qwen3:4b

What does this command do?

Ollama downloads the model when it is missing from your machine. It opens an interactive chat after the model is ready.

  • Send this short test prompt when the interactive chat is ready:
In one short sentence, say you are ready to investigate checkout incidents.

What does this test prove?

A relevant response confirms that Ollama loaded the model. It also confirms that the interactive chat can accept prompts.

You should receive a short response about investigating checkout incidents. The local model is now generating answers on your machine.

  • Close the interactive chat by entering:
/bye

What happens to the model?

The command closes the chat session. The downloaded model stays in Ollama's local model store.

Before you check the model list, do you expect the downloaded model tag to appear there?

  • Confirm that Ollama stored the model by running:
ollama ls

What does the model list show?

The list contains every model available to Ollama without another download.

You should see qwen3:4b in the model list. Your local runtime is ready for the Atlas container.

Is the model missing?

  • Confirm that the download reached completion before the interactive prompt appeared.
  • Run the model command from the earlier command block again if the download was interrupted.
  • Check that the command uses the exact model tag qwen3:4b.

If the model still does not appear, help me diagnose the missing model download.

Create the Atlas repository

The repository separates the checkout API from the Atlas CLI. A third folder holds the observability configuration that connects them to the telemetry stack.

  • Move to your Desktop by running this command in PowerShell:
cd ~/Desktop

What does this command do?

The command moves PowerShell to your Desktop. The repository you create next will be easy to find.

  • Create the atlas-incident-investigator folder before moving into it by running:
New-Item -Path "atlas-incident-investigator" -ItemType "Directory"
Set-Location -Path "atlas-incident-investigator"

What do these commands do?

  • The first command creates the atlas-incident-investigator folder on your Desktop.
  • The second command moves PowerShell into that folder.
  • Initialize the repository by running:
git init

What does this command do?

Git creates the repository metadata used to track changes inside atlas-incident-investigator.

You should see a message confirming that Git initialized a repository. The folder can now track every file you add.

  • Open the current repository in Visual Studio Code by running:
code .

What does this command do?

The dot represents the current folder. Visual Studio Code opens atlas-incident-investigator as the active repository.

You should see atlas-incident-investigator at the top of the Explorer sidebar. That confirms the editor opened the correct folder.

  • Switch back to the PowerShell window from earlier.
  • Create the repository scaffold by running:
New-Item -Path "." -Name "app" -ItemType "Directory"
New-Item -Path "." -Name "atlas" -ItemType "Directory"
New-Item -Path "." -Name "observability" -ItemType "Directory"
New-Item -Path ".gitignore" -ItemType "File"

How is the repository organized?

  • The app folder holds the checkout API source.
  • The atlas folder holds the Atlas CLI source.
  • The observability folder holds the Prometheus configuration.
  • The .gitignore file keeps generated Python files plus local environment values out of Git.

Visual Studio Code should update its Explorer sidebar with the three folders plus .gitignore.

  • Select .gitignore in the Visual Studio Code Explorer sidebar.
  • Add the ignore rules by copying this exact content into the file:
__pycache__/
*.pyc
.env

What do these rules protect?

  • The __pycache__/ rule excludes generated Python cache directories.
  • The *.pyc rule excludes compiled Python files.
  • The .env rule excludes local environment values from Git.
  • Save .gitignore.
  • Confirm that the Explorer sidebar lists app, atlas, observability, plus .gitignore.

Is the scaffold incomplete?

  • Confirm that PowerShell was inside atlas-incident-investigator when you created the folders.
  • Refresh the Visual Studio Code Explorer sidebar if the folders exist in PowerShell output but are missing from the editor.
  • Check that the ignore file is named .gitignore without an extra file extension.

Need help? Help me repair my Atlas repository scaffold.

✔️ Awesome, I've got everything!

Your repository scaffold is complete. Make sure .gitignore is saved before continuing.

ⓧ I'd like to double check the full code

Compare your .gitignore file with this complete version:

__pycache__/
*.pyc
.env
  • Confirm that app exists for the checkout API source, Dockerfile, requirements, and traffic script.
  • Confirm that atlas exists for the Atlas CLI source, Dockerfile, and requirements.
  • Confirm that observability exists for the Prometheus configuration.

Before you run the final checks, which result do you expect to confirm that the local model is ready?

  • Switch back to the PowerShell window from earlier.
  • Verify the complete local toolchain by running:
docker --version
docker compose version
git --version
code --version
ollama ls

What does the final check cover?

  • The first two results confirm that Docker Desktop plus Docker Compose respond from PowerShell.
  • The next two results confirm that Git plus the Visual Studio Code command-line interface respond.
  • The final result confirms that Ollama can find the downloaded model.

Every command should print successfully. The final list should include qwen3:4b.

You have the complete foundation in place. Ollama can run the model while your new Git repository is ready for the checkout service.

Does a final check fail?

  • Confirm that Docker Desktop still reports its Linux container engine as running.
  • Open a fresh PowerShell window if a recently installed command is missing.
  • Repeat the earlier model download if qwen3:4b is absent from the Ollama model list.

Still blocked? Help me diagnose the failed Atlas setup check.

Your local model is ready. Next, you'll launch the healthy checkout API that Atlas eventually investigates.

Launch the Checkout API

Your Atlas repository is ready. Now it needs a working service that produces evidence for every investigation.

A visible healthy response gives you a baseline before failures add complexity. In this step, you will build a containerized Python checkout API that exposes a labeled Prometheus counter.

In this step, get ready to:
  • Build checkout routes with deterministic fault control.
  • Add a repeatable traffic generator plus the container files.
  • Launch the service with Docker Compose and prove healthy checkouts work.
Build the checkout routes

The checkout service needs a healthy path plus a controlled failure path. Its counter records each outcome so later investigations can measure the incident.

  • In the Visual Studio Code Explorer from earlier, add a file named app.py inside the existing app folder.
  • Add the imports plus shared service state by copying this first section into app/app.py:
import json
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from itertools import count
from urllib.parse import urlparse

from prometheus_client import CONTENT_TYPE_LATEST, Counter, generate_latest

CHECKOUT_REQUESTS = Counter(
    "checkout_requests",
    "Checkout requests grouped by outcome",
    ["outcome"],
)
CHECKOUT_REQUESTS.labels(outcome="success")
CHECKOUT_REQUESTS.labels(outcome="error")

FAULT_LOCK = threading.Lock()
FAULT_ENABLED = False
ORDER_IDS = count(1001)


def set_fault(enabled: bool) -> None:
    global FAULT_ENABLED
    with FAULT_LOCK:
        FAULT_ENABLED = enabled


def fault_is_enabled() -> bool:
    with FAULT_LOCK:
        return FAULT_ENABLED

What does this code do?

  • The counter separates requests with the outcome label. This gives successful requests and failed requests their own series.
  • The two initial label calls create both series at zero. The error series therefore exists before the first failure.
  • The lock protects the shared fault state when the threaded server handles concurrent requests.
  • The counter from count(1001) gives each checkout a predictable order identifier.
  • Append the response helper plus the healthy service routes to app/app.py by copying this section below the existing code:


class CheckoutHandler(BaseHTTPRequestHandler):
    def send_json(self, status_code: int, payload: dict) -> None:
        body = json.dumps(payload).encode("utf-8")
        self.send_response(status_code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def do_GET(self) -> None:
        path = urlparse(self.path).path

        if path == "/health":
            self.send_json(200, {"status": "ok"})
            return

        if path == "/metrics":
            body = generate_latest()
            self.send_response(200)
            self.send_header("Content-Type", CONTENT_TYPE_LATEST)
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)
            return

How do these routes help?

  • The send_json() helper returns a status code plus a JSON body with the correct response headers.
  • The /health route gives Docker clients a simple proof that the process is responding.
  • The /metrics route publishes the counter in the Prometheus exposition format.
  • Append the fault controls plus the failed checkout path to app/app.py by copying this section next:


        if path == "/fault/on":
            set_fault(True)
            self.send_json(200, {"fault": "inventory-timeout", "enabled": True})
            return

        if path == "/fault/off":
            set_fault(False)
            self.send_json(200, {"fault": "inventory-timeout", "enabled": False})
            return

        if path == "/checkout":
            order_id = f"order-{next(ORDER_IDS)}"

            if fault_is_enabled():
                time.sleep(0.15)
                CHECKOUT_REQUESTS.labels(outcome="error").inc()
                self.send_json(
                    500,
                    {
                        "status": "failed",
                        "order_id": order_id,
                        "message": "checkout unavailable",
                    },
                )
                return

How does fault injection work?

  • The /fault/on route activates the inventory timeout simulation.
  • The /fault/off route restores healthy checkout behavior.
  • A faulted checkout pauses briefly before incrementing the error series. It then returns an HTTP status code of 500.
  • Finish the successful checkout path plus the server startup by copying this final section into app/app.py:


            CHECKOUT_REQUESTS.labels(outcome="success").inc()
            self.send_json(
                200,
                {"status": "confirmed", "order_id": order_id},
            )
            return

        self.send_json(404, {"status": "not-found"})

    def log_message(self, format: str, *args) -> None:
        return


if __name__ == "__main__":
    server = ThreadingHTTPServer(("0.0.0.0", 8000), CheckoutHandler)
    print("checkout-api listening on port 8000", flush=True)
    server.serve_forever()

What completes the service?

  • A healthy checkout increments the success series. Its response confirms the order with a unique identifier.
  • Unknown routes receive an HTTP status code of 404.
  • The threaded HTTP server listens on every container interface at port 8000.
  • Save app/app.py.
  • Confirm app.py appears beneath the app folder in the Explorer sidebar.
Package the service and traffic generator

Manual requests make incident results inconsistent. The traffic generator produces the same healthy baseline or injected failure whenever you run it.

  • Add a file named traffic.py inside the existing app folder.
  • Add the request helper by copying this first section into app/traffic.py:
import json
import sys
import time
from urllib.error import HTTPError
from urllib.request import urlopen

BASE_URL = "http://localhost:8000"


def request(path: str) -> int:
    try:
        with urlopen(f"{BASE_URL}{path}", timeout=5) as response:
            response.read()
            return response.status
    except HTTPError as error:
        error.read()
        return error.code

What does the request helper do?

  • The base URL points to the checkout service from inside its own container.
  • The helper returns the response status for successful requests.
  • The error branch also returns the response status. This lets the script count expected fault responses without stopping.
  • Complete the traffic modes by copying this section below the existing code in app/traffic.py:


def run(mode: str) -> None:
    if mode not in {"baseline", "incident", "healthy"}:
        raise SystemExit("Use: python traffic.py baseline|incident|healthy")

    fault_path = "/fault/on" if mode == "incident" else "/fault/off"
    request(fault_path)
    total = 20 if mode == "incident" else 10
    success = 0
    errors = 0

    for _ in range(total):
        status = request("/checkout")
        if status == 200:
            success += 1
        else:
            errors += 1

    time.sleep(3)
    print(
        json.dumps(
            {
                "mode": mode,
                "requests": total,
                "success": success,
                "errors": errors,
            }
        )
    )


if __name__ == "__main__":
    run(sys.argv[1] if len(sys.argv) > 1 else "baseline")

How are the traffic modes different?

  • The baseline mode disables the fault before sending ten requests.
  • The incident mode enables the fault before sending twenty requests.
  • The healthy mode disables the fault before sending ten requests.
  • The final JSON summary reports the selected mode plus the request totals.
  • Save app/traffic.py.
  • Confirm traffic.py appears beneath the app folder in the Explorer sidebar.
  • Add a file named requirements.txt inside the existing app folder.
  • Define the pinned Python packages by copying this content into app/requirements.txt:
opentelemetry-api==1.44.0
opentelemetry-sdk==1.44.0
opentelemetry-exporter-otlp-proto-http==1.44.0
prometheus-client==0.26.0

Why pin these packages?

Pinned versions make every container build install the same dependency set. The Prometheus client powers the counter and metrics route in this version of the API.

The OpenTelemetry packages support the live logs plus traces added later. Keeping the package set in one file makes the container reproducible.

  • Save app/requirements.txt.
  • Confirm requirements.txt appears beneath the app folder in the Explorer sidebar.
  • Add a file named Dockerfile inside the existing app folder.
  • Define the checkout image by copying this content into app/Dockerfile:
FROM python:3.13-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py traffic.py ./

EXPOSE 8000
CMD ["python", "app.py"]

How is the image assembled?

  • The image starts with the slim Python 3.13 runtime.
  • The dependency file is copied before the source files. Docker can reuse the installation layer when only application code changes.
  • The image exposes port 8000 before starting app.py.
  • Save app/Dockerfile.
  • Confirm Dockerfile appears beneath the app folder in the Explorer sidebar.
Run a healthy baseline

Docker Compose gives the checkout service a repeatable build plus a stable port mapping. The first run creates the image before starting the API in the background.

  • In the top level of the existing atlas-incident-investigator repository, add a file named compose.yaml.
  • Define the checkout service by copying this content into compose.yaml:
services:
  checkout-api:
    build:
      context: ./app
    ports:
      - "8000:8000"

What does this configuration do?

  • The build context points Docker at the existing app folder.
  • The port mapping makes the container service available at http://localhost:8000 on your computer.
  • Save compose.yaml.
  • Switch back to the PowerShell terminal in the existing atlas-incident-investigator repository.
  • Build the checkout image plus start the service by running:
docker compose up --build -d checkout-api

What does this command do?

Docker Compose builds the image from app/Dockerfile before starting checkout-api in detached mode. The service keeps running after PowerShell returns to the prompt.

The first build can take longer while Docker downloads the Python image plus installs the packages. Once it finishes, you will see the checkout service listed as started.

Checkout service did not start?

  • Confirm Docker Desktop is still running with Linux containers.
  • Check that compose.yaml sits at the top level of atlas-incident-investigator.
  • Check that Dockerfile uses a capital D inside the app folder.

Still blocked? Help me troubleshoot why the checkout-api service failed to build or start.

Before you generate traffic, how many successful requests do you expect the baseline mode to report?

  • Generate the healthy baseline inside the running checkout container by running:
docker compose exec checkout-api python traffic.py baseline

What does this baseline prove?

The command runs the traffic script inside the active service container. Baseline mode first disables the fault.

The script then sends ten checkout requests. Its final summary measures the service through real HTTP responses.

You should see baseline as the mode. The summary should report 10 successes plus 0 errors.

  • Open http://localhost:8000/health in your browser.

You should see a JSON response with status set to ok. This proves the containerized HTTP server is reachable.

  • Open http://localhost:8000/metrics in your browser.

Find the checkout_requests_total series. You should see a positive success outcome plus an error outcome at zero.

That is your first observability signal working. The service records healthy traffic before any incident exists.

Before the final check, what status do you expect from one more checkout while the fault is disabled?

  • Open http://localhost:8000/checkout in your browser.

You should see a JSON response with status set to confirmed. You should also see an order identifier that starts with order-.

Baseline traffic did not succeed?

  • Confirm the build command completed before running the traffic script.
  • Check that traffic.py uses http://localhost:8000 as its base URL.
  • Check that the checkout handler increments the success counter before returning the confirmed response.

Need another pair of eyes? Help me debug why the checkout baseline does not report ten successful requests.

✔️ Awesome, I've got everything!

Great work. Your checkout service is running at http://localhost:8000 with healthy traffic plus both counter outcomes initialized.

ⓧ I'd like to double check the full code

Compare each file with these complete references. Save any corrections before rebuilding the service.

import json
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from itertools import count
from urllib.parse import urlparse

from prometheus_client import CONTENT_TYPE_LATEST, Counter, generate_latest

CHECKOUT_REQUESTS = Counter(
    "checkout_requests",
    "Checkout requests grouped by outcome",
    ["outcome"],
)
CHECKOUT_REQUESTS.labels(outcome="success")
CHECKOUT_REQUESTS.labels(outcome="error")

FAULT_LOCK = threading.Lock()
FAULT_ENABLED = False
ORDER_IDS = count(1001)


def set_fault(enabled: bool) -> None:
    global FAULT_ENABLED
    with FAULT_LOCK:
        FAULT_ENABLED = enabled


def fault_is_enabled() -> bool:
    with FAULT_LOCK:
        return FAULT_ENABLED


class CheckoutHandler(BaseHTTPRequestHandler):
    def send_json(self, status_code: int, payload: dict) -> None:
        body = json.dumps(payload).encode("utf-8")
        self.send_response(status_code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def do_GET(self) -> None:
        path = urlparse(self.path).path

        if path == "/health":
            self.send_json(200, {"status": "ok"})
            return

        if path == "/metrics":
            body = generate_latest()
            self.send_response(200)
            self.send_header("Content-Type", CONTENT_TYPE_LATEST)
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)
            return

        if path == "/fault/on":
            set_fault(True)
            self.send_json(200, {"fault": "inventory-timeout", "enabled": True})
            return

        if path == "/fault/off":
            set_fault(False)
            self.send_json(200, {"fault": "inventory-timeout", "enabled": False})
            return

        if path == "/checkout":
            order_id = f"order-{next(ORDER_IDS)}"

            if fault_is_enabled():
                time.sleep(0.15)
                CHECKOUT_REQUESTS.labels(outcome="error").inc()
                self.send_json(
                    500,
                    {
                        "status": "failed",
                        "order_id": order_id,
                        "message": "checkout unavailable",
                    },
                )
                return

            CHECKOUT_REQUESTS.labels(outcome="success").inc()
            self.send_json(
                200,
                {"status": "confirmed", "order_id": order_id},
            )
            return

        self.send_json(404, {"status": "not-found"})

    def log_message(self, format: str, *args) -> None:
        return


if __name__ == "__main__":
    server = ThreadingHTTPServer(("0.0.0.0", 8000), CheckoutHandler)
    print("checkout-api listening on port 8000", flush=True)
    server.serve_forever()

What to compare in the API

Confirm the five routes are inside do_GET(). Confirm both counter label series are initialized before the server starts.

import json
import sys
import time
from urllib.error import HTTPError
from urllib.request import urlopen

BASE_URL = "http://localhost:8000"


def request(path: str) -> int:
    try:
        with urlopen(f"{BASE_URL}{path}", timeout=5) as response:
            response.read()
            return response.status
    except HTTPError as error:
        error.read()
        return error.code


def run(mode: str) -> None:
    if mode not in {"baseline", "incident", "healthy"}:
        raise SystemExit("Use: python traffic.py baseline|incident|healthy")

    fault_path = "/fault/on" if mode == "incident" else "/fault/off"
    request(fault_path)
    total = 20 if mode == "incident" else 10
    success = 0
    errors = 0

    for _ in range(total):
        status = request("/checkout")
        if status == 200:
            success += 1
        else:
            errors += 1

    time.sleep(3)
    print(
        json.dumps(
            {
                "mode": mode,
                "requests": total,
                "success": success,
                "errors": errors,
            }
        )
    )


if __name__ == "__main__":
    run(sys.argv[1] if len(sys.argv) > 1 else "baseline")

What to compare in the traffic script

Confirm the script recognizes all three modes. Confirm the incident total is twenty while the healthy totals are ten.

opentelemetry-api==1.44.0
opentelemetry-sdk==1.44.0
opentelemetry-exporter-otlp-proto-http==1.44.0
prometheus-client==0.26.0

What to compare in the requirements

Confirm all four package versions match exactly. A changed pin can produce a different container environment.

FROM python:3.13-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py traffic.py ./

EXPOSE 8000
CMD ["python", "app.py"]

What to compare in the image

Confirm both Python files are copied into /app. Confirm the final command starts app.py.

services:
  checkout-api:
    build:
      context: ./app
    ports:
      - "8000:8000"

What to compare in Compose

Confirm the service name is checkout-api. Confirm host port 8000 maps to container port 8000.

Your checkout API now returns confirmed orders plus records their outcomes. Next, you will send its metrics, logs, and traces into a local observability stack.

Wire Live Telemetry into Grafana

Your checkout API now produces confirmed orders. That response proves the request worked while it was inside the application.

Operational evidence needs to survive outside the process. In this step, you will connect the API to Grafana OpenTelemetry LGTM so Prometheus, Loki, and Tempo can capture each successful checkout.

In this step, get ready to:
  • Instrument checkout requests with OpenTelemetry spans and logs.
  • Add the local Grafana observability backend to Docker Compose.
  • Find the baseline checkout metric, log entry, and trace in Grafana Explore.
Instrument checkout requests with telemetry

OpenTelemetry gives each request a shared service identity. The OTLP/HTTP exporters send logs and traces to the backend configured through the container environment.

  • In app/app.py, select everything from import json through the line immediately above CHECKOUT_REQUESTS = Counter(.
  • Replace the selected region by pasting this telemetry setup:
import json
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from itertools import count
from urllib.parse import urlparse

from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk._logs.export import SimpleLogRecordProcessor
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.trace.status import Status, StatusCode
from prometheus_client import CONTENT_TYPE_LATEST, Counter, generate_latest

RESOURCE = Resource.create({SERVICE_NAME: "checkout-api"})

tracer_provider = TracerProvider(resource=RESOURCE)
tracer_provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter()))
tracer = tracer_provider.get_tracer("checkout-api")

logger_provider = LoggerProvider(resource=RESOURCE)
logger_provider.add_log_record_processor(
    SimpleLogRecordProcessor(OTLPLogExporter())
)
otel_logger = logger_provider.get_logger("checkout-api")

What does this telemetry setup do?

  • RESOURCE labels the telemetry with the service name checkout-api.
  • TracerProvider creates spans that record what happens during a checkout request.
  • LoggerProvider creates structured log records with the same service identity.
  • SimpleSpanProcessor and SimpleLogRecordProcessor export each signal immediately for this local lab.
  • In app/app.py, find the end of fault_is_enabled().
  • Add this logging helper immediately below that function:
def emit_log(severity: str, body: str, attributes: dict) -> None:
    print(
        json.dumps(
            {"severity": severity, "body": body, **attributes},
            sort_keys=True,
        ),
        flush=True,
    )
    otel_logger.emit(
        severity_text=severity,
        body=body,
        attributes=attributes,
    )

How does the logging helper work?

  • The JSON print keeps a readable copy of every event in the container output.
  • otel_logger.emit() sends the same event to the OpenTelemetry log exporter.
  • The attributes carry searchable values such as the order ID and checkout outcome.

The checkout route now needs to wrap each request in a span. The replacement is split across two chunks for readability, so paste both chunks before saving the file.

  • In app/app.py, find the route that begins with if path == "/checkout":.
  • Replace the current checkout route with this first chunk:
        if path == "/checkout":
            order_id = f"order-{next(ORDER_IDS)}"
            with tracer.start_as_current_span("checkout.request") as span:
                span.set_attribute("checkout.order_id", order_id)

                if fault_is_enabled():
                    time.sleep(0.15)
                    CHECKOUT_REQUESTS.labels(outcome="error").inc()
                    span.set_attribute("checkout.outcome", "error")
                    span.set_attribute("checkout.dependency", "inventory")
                    span.set_status(
                        Status(StatusCode.ERROR, "inventory dependency timeout")
                    )
                    emit_log(
                        "ERROR",
                        "inventory dependency timeout",
                        {
                            "order_id": order_id,
                            "dependency": "inventory",
                            "outcome": "error",
                        },
                    )
                    self.send_json(
                        500,
                        {
                            "status": "failed",
                            "order_id": order_id,
                            "message": "checkout unavailable",
                        },
                    )

How does the failure path record evidence?

  • checkout.request creates one trace span for the request.
  • The span stores the order ID before the route checks the fault state.
  • A faulted request records an error outcome and identifies the inventory dependency.
  • The error log carries the same order ID, dependency, and outcome for later searches.
  • Complete the replacement by pasting this success path directly below the first chunk:
                    return

                CHECKOUT_REQUESTS.labels(outcome="success").inc()
                span.set_attribute("checkout.outcome", "success")
                emit_log(
                    "INFO",
                    "checkout completed",
                    {"order_id": order_id, "outcome": "success"},
                )
                self.send_json(
                    200,
                    {"status": "confirmed", "order_id": order_id},
                )
                return

How does the success path record evidence?

  • The existing success counter still increments for every confirmed checkout.
  • The span records success as the checkout outcome.
  • The structured log records checkout completed with the order ID.
  • The request still returns its confirmed order response after emitting the telemetry.
  • Save app/app.py.
  • Rebuild the checkout service to confirm the telemetry code loads by running:
docker compose up --build -d checkout-api

What does this check prove?

Docker rebuilds the image with the updated Python file. A running checkout container proves the new imports and telemetry providers initialized successfully.

  • Return to your browser tab for http://localhost:8000/health.
  • Refresh the page to verify the rebuilt service still responds.

You should see the existing healthy JSON response with status set to ok. The request confirms that the instrumented application still starts correctly.

Does the checkout service stop during startup?

  • Check the terminal output for a Python indentation error near the replaced checkout route.
  • Confirm both checkout chunks sit inside do_GET() with the indentation shown.
  • Confirm the telemetry setup appears before CHECKOUT_REQUESTS.

Still stuck? Help me debug the instrumented checkout service.

Add the local observability backend

The exporters need a destination outside the checkout container. The LGTM container provides the collector plus local stores for metrics, logs, and traces.

  • In the Visual Studio Code Explorer, return to the observability folder from earlier.
  • Use the file creation control in the Explorer to create prometheus.yaml inside observability.
  • Add the Prometheus scrape configuration by pasting:
global:
  scrape_interval: 2s
  evaluation_interval: 2s

scrape_configs:
  - job_name: checkout-api
    static_configs:
      - targets:
          - checkout-api:8000

What does this scrape configuration do?

  • scrape_interval tells Prometheus to collect the API metrics every two seconds.
  • evaluation_interval sets the same interval for evaluating Prometheus rules.
  • checkout-api:8000 reaches the API through the Docker Compose network.
  • Save observability/prometheus.yaml.
  • Confirm prometheus.yaml appears beneath the observability folder in the Explorer.
  • In compose.yaml, select the current file contents.
  • Replace the selected contents with the complete checkout and LGTM configuration:
services:
  checkout-api:
    build:
      context: ./app
    ports:
      - "8000:8000"
    environment:
      OTEL_EXPORTER_OTLP_ENDPOINT: http://lgtm:4318
      OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
    depends_on:
      - lgtm

  lgtm:
    image: grafana/otel-lgtm:0.31.0
    ports:
      - "3000:3000"
      - "4318:4318"
    volumes:
      - lgtm-data:/data
      - ./observability/prometheus.yaml:/otel-lgtm/prometheus.yaml:ro

volumes:
  lgtm-data:

How does the stack connect?

  • OTEL_EXPORTER_OTLP_ENDPOINT directs the API exporters to the LGTM container on port 4318.
  • OTEL_EXPORTER_OTLP_PROTOCOL selects protobuf over HTTP for the exported signals.
  • The LGTM service publishes Grafana on port 3000.
  • lgtm-data preserves the local observability data across ordinary container restarts.
  • Save compose.yaml.

The first launch can take several minutes while Docker downloads the LGTM image. A quiet terminal during that download is normal.

  • Build and start the complete stack by running:
docker compose up --build -d

What does this command start?

Docker Compose builds the updated checkout image before starting both services in the background. The dependency rule starts LGTM as part of the same Compose project.

The command should finish with both services started. Your local metrics, logs, and traces now have somewhere to land.

Does the LGTM service fail to start?

  • Confirm Docker Desktop is still running with Linux containers.
  • Check that observability/prometheus.yaml is saved beneath the repository folder.
  • Check that ports 3000 and 4318 are available on your computer.

Need another pair of eyes? Help me troubleshoot the LGTM Compose service.

Prove all three signals arrive

A single checkout now leaves three kinds of evidence. The counter shows volume, the log records the event, and the trace follows the request.

Before you generate traffic, which signal do you expect to show a numeric series, which one should contain a message, and which one should show a request span?

  • Generate a fresh set of successful checkouts by running:
docker compose exec checkout-api python traffic.py baseline

What should this traffic produce?

The baseline mode disables the fault before sending ten checkout requests. It waits three seconds so Prometheus has time to complete another scrape.

You should see JSON reporting baseline mode with ten successful requests and zero errors. That confirms the instrumented path handled every request.

  • Open http://localhost:3000 in your browser.
  • Enter admin in the username field if Grafana asks you to sign in.
  • Enter admin in the password field.
  • Complete the sign-in form to reach Grafana.
  • Select Explore from the Grafana navigation.
  • Choose the Prometheus data source.
  • Enter checkout_requests_total in the query editor.
  • Submit the Prometheus query from the editor.

You should see labeled checkout counter series. The success series has a value above zero because the baseline requests completed.

  • Switch the Explore data source to Loki.
  • Enter this query in the LogQL editor:
{service_name="checkout-api"} |= "checkout completed"

What does this log query match?

The stream selector limits the search to logs from the checkout API. The line filter keeps events containing the successful checkout message.

  • Submit the LogQL query from the editor.

You should see recent log entries containing checkout completed. Each entry includes an order ID and a successful outcome.

  • Switch the Explore data source to Tempo.
  • Use the trace search controls to filter by the span name checkout.request.
  • Submit the trace search.
  • Open one matching checkout trace.

You should see a checkout.request span from the checkout-api service. Its attributes include the checkout order ID and successful outcome.

What does each signal contribute?

  • Prometheus shows how many checkout requests reached each outcome.
  • Loki preserves the event message and its searchable attributes.
  • Tempo shows the request span plus the attributes recorded during execution.

Missing one of the signals?

  • Wait for another two-second Prometheus scrape before repeating the metric query.
  • Confirm the Loki query uses service_name with an underscore.
  • Confirm the Tempo span filter uses checkout.request exactly.

Still missing telemetry? Help me trace the signal from checkout-api to Grafana.

✔️ Awesome, I've got everything!

Great work. Save every file and keep both Compose services running for the incident test.

ⓧ I'd like to double check the full code

Compare each repository file with the complete versions below.

__pycache__/
*.pyc
.env

This file keeps Python cache files and local environment files out of Git.

FROM python:3.13-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py traffic.py ./

EXPOSE 8000
CMD ["python", "app.py"]

This image installs the API dependencies before copying and starting the application files.

opentelemetry-api==1.44.0
opentelemetry-sdk==1.44.0
opentelemetry-exporter-otlp-proto-http==1.44.0
prometheus-client==0.26.0

These pinned packages provide the OpenTelemetry exporters and Prometheus counter.

import json
import sys
import time
from urllib.error import HTTPError
from urllib.request import urlopen

BASE_URL = "http://localhost:8000"


def request(path: str) -> int:
    try:
        with urlopen(f"{BASE_URL}{path}", timeout=5) as response:
            response.read()
            return response.status
    except HTTPError as error:
        error.read()
        return error.code


def run(mode: str) -> None:
    if mode not in {"baseline", "incident", "healthy"}:
        raise SystemExit("Use: python traffic.py baseline|incident|healthy")

    fault_path = "/fault/on" if mode == "incident" else "/fault/off"
    request(fault_path)
    total = 20 if mode == "incident" else 10
    success = 0
    errors = 0

    for _ in range(total):
        status = request("/checkout")
        if status == 200:
            success += 1
        else:
            errors += 1

    time.sleep(3)
    print(
        json.dumps(
            {
                "mode": mode,
                "requests": total,
                "success": success,
                "errors": errors,
            }
        )
    )


if __name__ == "__main__":
    run(sys.argv[1] if len(sys.argv) > 1 else "baseline")

The traffic generator keeps baseline, incident, and healthy runs repeatable.

import json
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from itertools import count
from urllib.parse import urlparse

from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk._logs.export import SimpleLogRecordProcessor
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.trace.status import Status, StatusCode
from prometheus_client import CONTENT_TYPE_LATEST, Counter, generate_latest

RESOURCE = Resource.create({SERVICE_NAME: "checkout-api"})

tracer_provider = TracerProvider(resource=RESOURCE)
tracer_provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter()))
tracer = tracer_provider.get_tracer("checkout-api")

logger_provider = LoggerProvider(resource=RESOURCE)
logger_provider.add_log_record_processor(
    SimpleLogRecordProcessor(OTLPLogExporter())
)
otel_logger = logger_provider.get_logger("checkout-api")

CHECKOUT_REQUESTS = Counter(
    "checkout_requests",
    "Checkout requests grouped by outcome",
    ["outcome"],
)
CHECKOUT_REQUESTS.labels(outcome="success")
CHECKOUT_REQUESTS.labels(outcome="error")

FAULT_LOCK = threading.Lock()
FAULT_ENABLED = False
ORDER_IDS = count(1001)


def set_fault(enabled: bool) -> None:
    global FAULT_ENABLED
    with FAULT_LOCK:
        FAULT_ENABLED = enabled


def fault_is_enabled() -> bool:
    with FAULT_LOCK:
        return FAULT_ENABLED


def emit_log(severity: str, body: str, attributes: dict) -> None:
    print(
        json.dumps(
            {"severity": severity, "body": body, **attributes},
            sort_keys=True,
        ),
        flush=True,
    )
    otel_logger.emit(
        severity_text=severity,
        body=body,
        attributes=attributes,
    )


class CheckoutHandler(BaseHTTPRequestHandler):
    def send_json(self, status_code: int, payload: dict) -> None:
        body = json.dumps(payload).encode("utf-8")
        self.send_response(status_code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def do_GET(self) -> None:
        path = urlparse(self.path).path

        if path == "/health":
            self.send_json(200, {"status": "ok"})
            return

        if path == "/metrics":
            body = generate_latest()
            self.send_response(200)
            self.send_header("Content-Type", CONTENT_TYPE_LATEST)
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)
            return

        if path == "/fault/on":
            set_fault(True)
            self.send_json(200, {"fault": "inventory-timeout", "enabled": True})
            return

        if path == "/fault/off":
            set_fault(False)
            self.send_json(200, {"fault": "inventory-timeout", "enabled": False})
            return

        if path == "/checkout":
            order_id = f"order-{next(ORDER_IDS)}"
            with tracer.start_as_current_span("checkout.request") as span:
                span.set_attribute("checkout.order_id", order_id)

                if fault_is_enabled():
                    time.sleep(0.15)
                    CHECKOUT_REQUESTS.labels(outcome="error").inc()
                    span.set_attribute("checkout.outcome", "error")
                    span.set_attribute("checkout.dependency", "inventory")
                    span.set_status(
                        Status(StatusCode.ERROR, "inventory dependency timeout")
                    )
                    emit_log(
                        "ERROR",
                        "inventory dependency timeout",
                        {
                            "order_id": order_id,
                            "dependency": "inventory",
                            "outcome": "error",
                        },
                    )
                    self.send_json(
                        500,
                        {
                            "status": "failed",
                            "order_id": order_id,
                            "message": "checkout unavailable",
                        },
                    )
                    return

                CHECKOUT_REQUESTS.labels(outcome="success").inc()
                span.set_attribute("checkout.outcome", "success")
                emit_log(
                    "INFO",
                    "checkout completed",
                    {"order_id": order_id, "outcome": "success"},
                )
                self.send_json(
                    200,
                    {"status": "confirmed", "order_id": order_id},
                )
                return

        self.send_json(404, {"status": "not-found"})

    def log_message(self, format: str, *args) -> None:
        return


if __name__ == "__main__":
    server = ThreadingHTTPServer(("0.0.0.0", 8000), CheckoutHandler)
    print("checkout-api listening on port 8000", flush=True)
    server.serve_forever()

This complete API emits a metric, structured log, and trace span for each checkout.

services:
  checkout-api:
    build:
      context: ./app
    ports:
      - "8000:8000"
    environment:
      OTEL_EXPORTER_OTLP_ENDPOINT: http://lgtm:4318
      OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
    depends_on:
      - lgtm

  lgtm:
    image: grafana/otel-lgtm:0.31.0
    ports:
      - "3000:3000"
      - "4318:4318"
    volumes:
      - lgtm-data:/data
      - ./observability/prometheus.yaml:/otel-lgtm/prometheus.yaml:ro

volumes:
  lgtm-data:

This Compose file connects the API exporters to the LGTM backend and preserves its data.

global:
  scrape_interval: 2s
  evaluation_interval: 2s

scrape_configs:
  - job_name: checkout-api
    static_configs:
      - targets:
          - checkout-api:8000

This Prometheus configuration scrapes the checkout API every two seconds.

Your checkout API now leaves evidence in all three observability systems. Next, you will trigger the inventory fault and discover why an error counter can expose a symptom without revealing its cause.

Trigger the Checkout Incident

Your healthy baseline proved that the checkout service can confirm orders. Now you'll use its deterministic fault mode to create a repeatable incident.

Prometheus can count failed checkout requests. This step tests whether that count can explain why the service is failing.

In this step, get ready to:
  • Generate 20 failed checkout requests with incident mode.
  • Evaluate the error counter with PromQL to test how much the metric reveals.
Trigger the inventory timeout

Fault injection gives you a controlled way to reproduce a failure. The traffic script activates the inventory timeout before sending 20 checkout requests.

  • Return to the PowerShell terminal in Visual Studio Code.
  • Use the existing traffic script's incident mode by running:
docker compose exec checkout-api python traffic.py incident

What does incident mode do?

  • Docker Compose runs the existing traffic script inside the active checkout API container.
  • The incident mode enables the fault through /fault/on.
  • The script sends 20 checkout requests through the faulted path.
  • The script waits three seconds so Prometheus has time to scrape the new counter value.

You should see JSON with mode set to incident. The output should show requests as 20.

The success count should be 0. The errors count should be 20.

You now have a reproducible incident. Every checkout in this run followed the faulted path.

Did the incident run fail?

  • Confirm Docker Desktop is running if the checkout service cannot be reached.
  • Retry the incident command after the checkout service shows as running.
  • Check that incident is the final argument if the output still reports successful requests.

Still stuck? Help me troubleshoot the Atlas incident traffic command.

Measure the error spike

Grafana Explore lets you evaluate the new error data through the Prometheus query editor. The five-minute query measures how far the error counter has risen.

  • Return to the Grafana browser tab at http://localhost:3000.
  • Select Explore from the left side menu.
  • Select Prometheus from the data source picker in the upper left.
  • Select Code in the Prometheus query editor.
  • Paste this expression into the query editor:
sum(increase(checkout_requests_total{outcome="error"}[5m]))

What does this query measure?

  • The five-minute range examines the error-labeled counter over the previous five minutes.
  • The increase function measures how far that counter rose during the range.
  • The outer sum combines the matching series into one incident value.

Before you evaluate the query, do you think one positive counter value can identify the failed dependency?

  • Click Run query in the upper right.

You should see a result greater than zero. That result confirms that checkout errors increased during the five-minute window.

The metric proves that a checkout error spike occurred. It does not identify the failed dependency.

  • Keep the Loki error logs unopened so the next investigation begins with this unresolved gap.

No positive result?

  • Confirm Prometheus is selected as the data source if no result appears.
  • Run the incident command from the previous substep again if the five-minute window has expired.
  • Compare the expression with the provided query if Grafana cannot evaluate it.

Need another set of eyes? Help me troubleshoot the PromQL error query in Grafana Explore.

Your error spike is measurable, but its cause remains unresolved. Next, you'll give Atlas the tools to gather the missing evidence.

Build Atlas and Diagnose the Failure

Your incident now has a measurable symptom. The Prometheus metric proves that checkout errors increased, but it cannot identify the failed dependency.

Atlas closes that evidence gap with an Ollama tool-calling agent loop packaged through Docker Compose. It queries Prometheus through Grafana before searching Loki for the root cause.

In this step, get ready to:
  • Package Atlas as a profiled command-line service.
  • Build the telemetry tools and Ollama agent loop.
  • Run Atlas against the live incident evidence.
Package Atlas as a Compose service

Atlas needs its own container so it can reach Grafana over the Compose network. A profile keeps this one-off investigation tool separate from the services that stay running.

  • Select the atlas folder from earlier in the Visual Studio Code Explorer.
  • Create requirements.txt inside the atlas folder.
  • Pin the Ollama Python package by copying this line into atlas/requirements.txt:
ollama==0.6.2

Why Pin the Package?

The 0.6.2 pin gives Atlas a consistent Ollama client interface. The agent relies on that interface to receive tool calls from qwen3:4b.

  • Save atlas/requirements.txt.
  • Confirm requirements.txt appears inside the atlas folder in the Explorer.

Package File in the Wrong Folder?

Check that requirements.txt is nested under atlas. Move it there if the file appears beside compose.yaml.

Still stuck? Help me place the Atlas requirements file correctly.

✔️ Awesome, I've got everything!

Great. Your Atlas dependency is pinned in atlas/requirements.txt.

ⓧ I'd like to double check the full code

ollama==0.6.2
  • Create a file named Dockerfile inside the atlas folder.
  • Define the Atlas container by copying this code into atlas/Dockerfile:
FROM python:3.13-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY atlas.py .

ENTRYPOINT ["python", "atlas.py"]

What Does This Container Do?

  • The base image provides the Python runtime used by Atlas.
  • The dependency layer installs the pinned Ollama package from requirements.txt.
  • The entry point sends every argument after the service name to atlas.py.
  • Save atlas/Dockerfile.
  • Confirm Dockerfile appears beside requirements.txt inside the atlas folder.

Dockerfile Missing from Atlas?

Check the filename has no extension. Visual Studio Code should show Dockerfile inside the atlas folder.

Need a hand? Help me check my Atlas Dockerfile location and contents.

✔️ Awesome, I've got everything!

Your Atlas container definition is ready for the agent code.

ⓧ I'd like to double check the full code

FROM python:3.13-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY atlas.py .

ENTRYPOINT ["python", "atlas.py"]
  • Return to compose.yaml in the editor.
  • Find the top-level volumes: line near the bottom of the file.
  • Insert the Atlas service directly above volumes: by copying this block:
  atlas-cli:
    build:
      context: ./atlas
    profiles: ["tools"]
    environment:
      OLLAMA_HOST: http://host.docker.internal:11434
      OLLAMA_MODEL: qwen3:4b
      GRAFANA_URL: http://lgtm:3000
      GRAFANA_USER: admin
      GRAFANA_PASSWORD: admin
    depends_on:
      - lgtm

How Does the Atlas Service Connect?

  • The tools profile keeps atlas-cli out of a normal stack startup.
  • The Ollama host URL reaches the Windows service from inside the container.
  • The Grafana URL reaches the observability container over the Compose network.
  • The local Grafana credentials authorize Atlas to use the data source proxy endpoints.
  • Save compose.yaml.
  • Confirm atlas-cli is aligned with checkout-api under services.

Atlas Service Indented Incorrectly?

Make sure atlas-cli has the same indentation as checkout-api. Keep volumes aligned with services.

Need help checking the YAML structure? Help me fix the indentation in my Compose file.

✔️ Awesome, I've got everything!

Your Compose project now includes the profiled Atlas service.

ⓧ I'd like to double check the full code

services:
  checkout-api:
    build:
      context: ./app
    ports:
      - "8000:8000"
    environment:
      OTEL_EXPORTER_OTLP_ENDPOINT: http://lgtm:4318
      OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
    depends_on:
      - lgtm

  lgtm:
    image: grafana/otel-lgtm:0.31.0
    ports:
      - "3000:3000"
      - "4318:4318"
    volumes:
      - lgtm-data:/data
      - ./observability/prometheus.yaml:/otel-lgtm/prometheus.yaml:ro

  atlas-cli:
    build:
      context: ./atlas
    profiles: ["tools"]
    environment:
      OLLAMA_HOST: http://host.docker.internal:11434
      OLLAMA_MODEL: qwen3:4b
      GRAFANA_URL: http://lgtm:3000
      GRAFANA_USER: admin
      GRAFANA_PASSWORD: admin
    depends_on:
      - lgtm

volumes:
  lgtm-data:
Build the telemetry tools and agent loop

The agent needs two callable tools. One tool checks whether errors exist, while the second searches logs only when the metric justifies a deeper investigation.

  • Create atlas.py inside the atlas folder.
  • Add the imports and environment settings by copying this code into atlas/atlas.py:
import base64
import json
import os
import sys
from urllib.parse import urlencode
from urllib.request import Request, urlopen

from ollama import Client

OLLAMA_HOST = os.environ.get(
    "OLLAMA_HOST", "http://host.docker.internal:11434"
)
OLLAMA_MODEL = os.environ.get("OLLAMA_MODEL", "qwen3:4b")
GRAFANA_URL = os.environ.get("GRAFANA_URL", "http://lgtm:3000")
GRAFANA_USER = os.environ.get("GRAFANA_USER", "admin")
GRAFANA_PASSWORD = os.environ.get("GRAFANA_PASSWORD", "admin")

How Does Atlas Find Its Services?

  • The URL helpers prepare authenticated requests to Grafana.
  • The Ollama client connects to the model runtime on your Windows host.
  • The environment settings match the values passed into the Atlas container.
  • Save atlas/atlas.py.
  • Confirm the file currently ends with the GRAFANA_PASSWORD setting.

Seeing Import or Quote Problems?

Check that every imported name matches the snippet. Confirm each environment value has matching quotation marks.

Want help comparing the setup block? Help me check the imports and environment settings in atlas.py.

  • Define the investigation rules by copying this block below GRAFANA_PASSWORD:
SYSTEM_PROMPT = """
You are Atlas, a local incident investigator running in a tool-calling loop.
Never diagnose from the user's wording alone.

Required investigation:
1. Call query_metrics with exactly:
   sum(increase(checkout_requests_total{outcome="error"}[5m]))
2. If the metric result is greater than zero, call query_logs with exactly:
   {service_name="checkout-api"} |= "inventory dependency timeout"
3. If the metric is zero or empty, report that there is no current evidence of
   checkout failures and do not invent a root cause.
4. If errors and matching logs exist, return these headings:
   Incident Summary, Root Cause, Evidence, Next Action.
5. Treat tool output as evidence. Do not claim facts absent from that output.
""".strip()

How Does the Prompt Ground Atlas?

  • The first rule forces Atlas to measure the current error count.
  • The second rule permits a log search only after the metric shows errors.
  • The healthy-state rule prevents an unsupported root cause.
  • The final rules require evidence and a consistent response structure.
  • Save atlas/atlas.py.
  • Confirm the prompt ends with """.strip().

Prompt String Not Closing?

Check for three quotation marks at both ends of SYSTEM_PROMPT. Keep the PromQL and LogQL expressions exactly as shown.

Need help finding a mismatch? Help me check the Atlas system prompt for missing quotes or changed queries.

  • Add the Grafana request helper below SYSTEM_PROMPT by copying this function:
def grafana_get(path: str, parameters: dict) -> dict:
    query_string = urlencode(parameters)
    url = f"{GRAFANA_URL}{path}?{query_string}"
    credentials = base64.b64encode(
        f"{GRAFANA_USER}:{GRAFANA_PASSWORD}".encode("utf-8")
    ).decode("ascii")
    request = Request(
        url,
        headers={
            "Accept": "application/json",
            "Authorization": f"Basic {credentials}",
        },
    )
    with urlopen(request, timeout=15) as response:
        return json.loads(response.read().decode("utf-8"))

What Does the Grafana Helper Do?

  • The helper encodes each tool's query as URL parameters.
  • The credentials become a Basic authentication header for the local Grafana API.
  • The response body becomes a Python dictionary that each tool can inspect.
  • Save atlas/atlas.py.
  • Confirm grafana_get ends by returning decoded JSON.

Helper Structure Looks Uneven?

Keep the request headers inside the Request call. Keep the final return inside the response context.

Need help with the nesting? Help me check the indentation in grafana_get.

  • Add the Prometheus tool below grafana_get by copying this function:
def query_metrics(query: str) -> str:
    """Run an instant PromQL query against the local Prometheus data source.

    Args:
        query: The complete PromQL expression to evaluate.

    Returns:
        A JSON string containing the Prometheus result rows.
    """
    try:
        payload = grafana_get(
            "/api/datasources/proxy/uid/prometheus/api/v1/query",
            {"query": query},
        )
        return json.dumps(payload.get("data", {}).get("result", []), indent=2)
    except Exception as error:
        return json.dumps({"error": str(error)})

What Does the Metric Tool Return?

The function sends an instant PromQL query through Grafana's Prometheus proxy. It returns only the result rows so the model receives focused evidence.

A caught request problem becomes JSON too. That keeps failures visible to the agent loop.

  • Save atlas/atlas.py.
  • Confirm query_metrics includes the Prometheus proxy path.

Metric Tool Nested Incorrectly?

Keep the success return inside the try block. Keep the error return inside the matching except block.

Need help checking the function? Help me compare query_metrics with the required Atlas code.

  • Add the Loki tool below query_metrics by copying this function:
def query_logs(query: str, since: str = "10m") -> str:
    """Run a LogQL range query against the local Loki data source.

    Args:
        query: The complete LogQL expression to evaluate.
        since: How far back Loki should search, such as 10m.

    Returns:
        A JSON string containing matching Loki streams and log entries.
    """
    try:
        payload = grafana_get(
            "/api/datasources/proxy/uid/loki/loki/api/v1/query_range",
            {
                "query": query,
                "since": since,
                "limit": "20",
                "direction": "backward",
            },
        )
        return json.dumps(payload.get("data", {}).get("result", []), indent=2)
    except Exception as error:
        return json.dumps({"error": str(error)})

What Does the Log Tool Return?

The function searches a recent Loki time range through Grafana. Its result preserves matching streams and log entries as JSON evidence.

The result limit keeps the local investigation concise. Backward ordering returns the newest matching entries first.

  • Save atlas/atlas.py.
  • Confirm query_logs includes the Loki range-query path.

Log Tool Missing Parameters?

Check that the parameter dictionary contains query, since, limit, and direction as separate entries.

Need help checking the tool? Help me compare query_logs with the required Atlas code.

  • Start the agent loop below query_logs by copying this function section:
def run_agent(prompt: str) -> None:
    client = Client(host=OLLAMA_HOST)
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": prompt},
    ]
    available_tools = {
        "query_metrics": query_metrics,
        "query_logs": query_logs,
    }

    for _ in range(6):
        response = client.chat(
            model=OLLAMA_MODEL,
            messages=messages,
            tools=list(available_tools.values()),
            think=True,
        )
        messages.append(response.message)

        if not response.message.tool_calls:
            print("\nAtlas diagnosis:\n")
            print(response.message.content or "No final response was produced.")
            return

How Does the Agent Start?

  • The client connects to the Ollama host from the container environment.
  • The message history begins with the investigation rules and your question.
  • The tool map gives the model access to the two telemetry functions.
  • The loop asks the model for the next action across a maximum of six turns.
  • Save atlas/atlas.py.
  • Confirm the current function ends with return inside the no-tool-call branch.
  • Complete run_agent by adding this code directly below that return line:
        for tool_call in response.message.tool_calls:
            name = tool_call.function.name
            arguments = tool_call.function.arguments
            function = available_tools.get(name)
            print(f"\nTool call: {name}")
            print(json.dumps(arguments, indent=2))

            if function is None:
                result = json.dumps({"error": f"Unknown tool: {name}"})
            else:
                result = function(**arguments)

            print("Tool result:")
            print(result)
            messages.append(
                {
                    "role": "tool",
                    "tool_name": name,
                    "content": result,
                }
            )

    raise SystemExit("Atlas stopped after six tool-calling turns.")

How Does Atlas Execute a Tool Call?

  • Atlas prints the selected tool name and its arguments before execution.
  • The mapped Python function runs with the model's arguments.
  • Atlas prints the tool result so you can inspect the evidence yourself.
  • The tool message returns that evidence to the model for the next turn.
  • Save atlas/atlas.py.
  • Confirm the raise SystemExit line aligns with the outer for loop.

Agent Loop Indentation Looks Wrong?

Keep the tool-call loop inside the six-turn loop. Keep the final exit outside both loops.

Need help mapping the indentation? Help me fix the structure of run_agent.

  • Add the command-line entry point at the bottom of atlas.py by copying this code:
if __name__ == "__main__":
    user_prompt = " ".join(sys.argv[1:]).strip()
    if not user_prompt:
        raise SystemExit(
            'Use: python atlas.py "Why are checkouts failing?"'
        )
    run_agent(user_prompt)

How Does Atlas Receive Your Question?

The entry point joins the command-line arguments into one prompt. It stops with usage guidance when no question is provided.

A valid prompt passes directly into run_agent and starts the investigation loop.

  • Save atlas/atlas.py.
  • Confirm run_agent(user_prompt) is the final line in the file.

✔️ Awesome, I've got everything!

Atlas now has two telemetry tools and a complete Ollama agent loop.

ⓧ I'd like to double check the full code

import base64
import json
import os
import sys
from urllib.parse import urlencode
from urllib.request import Request, urlopen

from ollama import Client

OLLAMA_HOST = os.environ.get(
    "OLLAMA_HOST", "http://host.docker.internal:11434"
)
OLLAMA_MODEL = os.environ.get("OLLAMA_MODEL", "qwen3:4b")
GRAFANA_URL = os.environ.get("GRAFANA_URL", "http://lgtm:3000")
GRAFANA_USER = os.environ.get("GRAFANA_USER", "admin")
GRAFANA_PASSWORD = os.environ.get("GRAFANA_PASSWORD", "admin")

SYSTEM_PROMPT = """
You are Atlas, a local incident investigator running in a tool-calling loop.
Never diagnose from the user's wording alone.

Required investigation:
1. Call query_metrics with exactly:
   sum(increase(checkout_requests_total{outcome="error"}[5m]))
2. If the metric result is greater than zero, call query_logs with exactly:
   {service_name="checkout-api"} |= "inventory dependency timeout"
3. If the metric is zero or empty, report that there is no current evidence of
   checkout failures and do not invent a root cause.
4. If errors and matching logs exist, return these headings:
   Incident Summary, Root Cause, Evidence, Next Action.
5. Treat tool output as evidence. Do not claim facts absent from that output.
""".strip()


def grafana_get(path: str, parameters: dict) -> dict:
    query_string = urlencode(parameters)
    url = f"{GRAFANA_URL}{path}?{query_string}"
    credentials = base64.b64encode(
        f"{GRAFANA_USER}:{GRAFANA_PASSWORD}".encode("utf-8")
    ).decode("ascii")
    request = Request(
        url,
        headers={
            "Accept": "application/json",
            "Authorization": f"Basic {credentials}",
        },
    )
    with urlopen(request, timeout=15) as response:
        return json.loads(response.read().decode("utf-8"))


def query_metrics(query: str) -> str:
    """Run an instant PromQL query against the local Prometheus data source.

    Args:
        query: The complete PromQL expression to evaluate.

    Returns:
        A JSON string containing the Prometheus result rows.
    """
    try:
        payload = grafana_get(
            "/api/datasources/proxy/uid/prometheus/api/v1/query",
            {"query": query},
        )
        return json.dumps(payload.get("data", {}).get("result", []), indent=2)
    except Exception as error:
        return json.dumps({"error": str(error)})


def query_logs(query: str, since: str = "10m") -> str:
    """Run a LogQL range query against the local Loki data source.

    Args:
        query: The complete LogQL expression to evaluate.
        since: How far back Loki should search, such as 10m.

    Returns:
        A JSON string containing matching Loki streams and log entries.
    """
    try:
        payload = grafana_get(
            "/api/datasources/proxy/uid/loki/loki/api/v1/query_range",
            {
                "query": query,
                "since": since,
                "limit": "20",
                "direction": "backward",
            },
        )
        return json.dumps(payload.get("data", {}).get("result", []), indent=2)
    except Exception as error:
        return json.dumps({"error": str(error)})


def run_agent(prompt: str) -> None:
    client = Client(host=OLLAMA_HOST)
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": prompt},
    ]
    available_tools = {
        "query_metrics": query_metrics,
        "query_logs": query_logs,
    }

    for _ in range(6):
        response = client.chat(
            model=OLLAMA_MODEL,
            messages=messages,
            tools=list(available_tools.values()),
            think=True,
        )
        messages.append(response.message)

        if not response.message.tool_calls:
            print("\nAtlas diagnosis:\n")
            print(response.message.content or "No final response was produced.")
            return

        for tool_call in response.message.tool_calls:
            name = tool_call.function.name
            arguments = tool_call.function.arguments
            function = available_tools.get(name)
            print(f"\nTool call: {name}")
            print(json.dumps(arguments, indent=2))

            if function is None:
                result = json.dumps({"error": f"Unknown tool: {name}"})
            else:
                result = function(**arguments)

            print("Tool result:")
            print(result)
            messages.append(
                {
                    "role": "tool",
                    "tool_name": name,
                    "content": result,
                }
            )

    raise SystemExit("Atlas stopped after six tool-calling turns.")


if __name__ == "__main__":
    user_prompt = " ".join(sys.argv[1:]).strip()
    if not user_prompt:
        raise SystemExit(
            'Use: python atlas.py "Why are checkouts failing?"'
        )
    run_agent(user_prompt)
Run the evidence-backed investigation

The first Atlas build can take a few minutes while Docker prepares the Python image and installs the Ollama package. Later builds reuse cached layers.

  • Build the profiled Atlas image by running this command in the PowerShell terminal from earlier:
docker compose build atlas-cli

What Does This Command Do?

Docker Compose reads the atlas-cli build context from compose.yaml. It creates an image containing the Ollama package and your completed atlas.py file.

The terminal returns to the PowerShell prompt when the image build finishes successfully.

Atlas Image Failing to Build?

Confirm Docker Desktop is still running. Check that requirements.txt, Dockerfile, and atlas.py all appear inside the atlas folder.

Still blocked? Help me troubleshoot the atlas-cli image build.

Before you start Atlas, which source do you think it will query first to decide whether the incident is real?

  • Start the investigation by running this command:
docker compose run --rm atlas-cli "Why are checkouts failing?"

What Does This Command Do?

  • Docker Compose starts a one-off atlas-cli container.
  • The quoted question becomes the user prompt passed to atlas.py.
  • The --rm flag removes the one-off container after Atlas exits.

You should first see a query_metrics tool call using sum(increase(checkout_requests_total{outcome="error"}[5m])). Its result contains a nonzero error value.

Atlas then prints a query_logs tool call using {service_name="checkout-api"} |= "inventory dependency timeout".

The final diagnosis contains Incident Summary, Root Cause, Evidence, and Next Action headings. The root cause names the inventory dependency timeout.

Atlas Missing Evidence?

  • Repeat the incident traffic mode from the previous step if the recent metric result is empty or zero.
  • Confirm Ollama is still running with the container-access host setting if Atlas cannot reach the model.
  • Confirm the Grafana OpenTelemetry LGTM service remains running if either telemetry tool returns a connection problem.

Need help tracing the failed turn? Help me diagnose why Atlas did not call both telemetry tools.

That is the full evidence chain working. Atlas now moves from an error spike to the inventory timeout without guessing from the question alone.

Secret mission

Prove Atlas Stays Quiet When Healthy

The failure diagnosis worked because Atlas found current evidence. Now clear the incident history and challenge Atlas with a leading question to prove it stays quiet when the checkout service is healthy.

Clean Up Your Resources

The local software path has no metered API cost. Docker Desktop has separate commercial licensing terms for larger organizations.

Nice Work!

Nice Work!

Outstanding work! You built Atlas as a local incident investigator that follows real telemetry from a failing checkout service.

You've learned how to:

  • Built a containerized Python checkout service with deterministic fault injection. Emitted Prometheus metrics for request outcomes. Sent OpenTelemetry logs and traces over OTLP/HTTP.
  • Ran Grafana OpenTelemetry LGTM through Docker Compose as one local observability stack. Used PromQL to prove the error spike. Used LogQL to identify the inventory timeout.
  • Built an Ollama tool-calling loop for qwen3:4b. Printed each metric query before Atlas returned an evidence-backed diagnosis. Printed each log query before Atlas returned the root cause.
  • Completed the optional Secret Mission by clearing the old telemetry. Proved Atlas reports a healthy state without calling query_logs or inventing a root cause.

Ready to quiz yourself?