Offline AI Observability Assistant

Build an offline AI pipeline that diagnoses server issues using local LLMs.

Introduction

30 Second Summary

When a server slows down at 2 AM, the on-call engineer needs answers fast. The difference between a 5-minute fix and an all-night firefight often comes down to how quickly someone can connect the dots between CPU spikes, error logs, and recent changes.

In this project, you will build an offline AI assistant that diagnoses infrastructure problems by combining live metrics, logs, and local LLM inference into a single pipeline that works with zero internet dependency.

What You'll Build

You'll create an AI-powered diagnosis system that fetches server metrics from Prometheus, retrieves error logs from Loki, and sends everything to a locally-running Llama 3.2 model via Ollama for root cause analysis.

By the end of this project, you'll have:

  • A Docker Compose stack running Prometheus, Loki, Grafana, and a simulated metrics exporter with zero internet dependency at runtime.
  • A production-structured FastAPI backend that orchestrates the full question-to-diagnosis flow using Ollama.
  • A working end-to-end pipeline where you ask "Why is vm-04 slow?" and receive an AI-generated diagnosis with severity rating and recommended fix.
  • Secret Mission: Add structured JSON output mode that returns machine-readable diagnosis reports with severity, confidence, and recommended actions.

Are there any prerequisites?

You should have basic Linux command line familiarity and an understanding of what servers, CPU, and memory are. You also need Ollama installed with the llama3.2 model already pulled.

Step 1 will walk you through installing Docker and Docker Compose on your Ubuntu server.

Set Up Docker and the Monitoring Stack

Your Docker installation and Ollama with llama3.2:3b are ready. Now you need the other half of the diagnosis pipeline: live infrastructure data.

In this step, you'll launch a full monitoring stack using Docker Compose. Prometheus will collect metrics, Loki will store logs, and Grafana will give you a visual dashboard to inspect both.

In this step, get ready to:
  • Create the project directory structure for all services.
  • Define Prometheus, Loki, Grafana, and a fake metrics exporter in a Docker Compose file.
  • Write configuration files and launch the full stack.
Create the project directory structure

The monitoring stack needs separate configuration directories for each service. You'll create folders for Prometheus, Loki, the fake exporter, and the backend that you'll build in a later step.

  • Return to your project directory and create the subdirectories by running these commands:
mkdir -p prometheus loki fake-exporter backend
  • Create a placeholder file that the fake exporter service will mount:
touch fake-exporter/metrics.py
  • Confirm the directories exist by running:
ls

You should see prometheus, loki, fake-exporter, and backend listed in the output.

Define the monitoring services in Docker Compose

Docker Compose lets you declare your entire multi-container stack in a single YAML file. One command will pull images, create networks, and start everything.

  • Create a file called docker-compose.yml in your project root by running:
nano docker-compose.yml
  • Paste the following content into the file:
services:
  prometheus:
    image: prom/prometheus:v3.11.3
    container_name: ntc-prometheus
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    networks:
      - ntc-internal
    restart: unless-stopped

  loki:
    image: grafana/loki:3.7.0
    container_name: ntc-loki
    volumes:
      - ./loki/loki-config.yml:/etc/loki/local-config.yaml
      - loki_data:/loki
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - ntc-internal
    restart: unless-stopped

  grafana:
    image: grafana/grafana:13.0.0
    container_name: ntc-grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=ntcadmin
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
    ports:
      - "3000:3000"
    networks:
      - ntc-internal
    restart: unless-stopped

  fake-exporter:
    image: prom/prometheus:v3.11.3
    container_name: ntc-fake-exporter
    volumes:
      - ./fake-exporter/metrics.py:/metrics.py
    entrypoint: []
    command: >
      /bin/sh -c "echo 'Fake exporter placeholder - replace with Python exporter' && sleep infinity"
    networks:
      - ntc-internal
    restart: unless-stopped

networks:
  ntc-internal:
    driver: bridge
    internal: false

volumes:
  prometheus_data:
  loki_data:
  grafana_data:
  • Save the file and exit nano by pressing Ctrl+O, then Enter, then Ctrl+X.

What does this file do?

This Docker Compose file defines four services that form the observability backbone:

  • prometheus scrapes metrics from targets every 15 seconds and stores them as time-series data. You'll query it with PromQL later.
  • loki is a log aggregation engine. It receives and indexes log entries so you can search them with LogQL.
  • grafana provides a web dashboard for visualizing both metrics and logs. The admin password is set to ntcadmin.
  • fake-exporter is a placeholder that simulates a node exporter. In a real environment, this would be a metrics agent running on target servers.

All services share the ntc-internal network, so they can communicate using service names as hostnames.

Why a bridge network instead of an internal-only network?

The ntc-internal network uses internal: false so that containers can pull images and you can access Grafana, Prometheus, and Loki through your browser via the published ports.

In a production NTC deployment, you would set internal: true and expose only a reverse proxy. For this development setup, direct access makes testing easier.

Write Prometheus and Loki configuration files, then launch the stack

Each service needs its own configuration. Prometheus needs to know what to scrape, and Loki needs to know where to store data.

  • Create the Prometheus configuration file by running:
nano prometheus/prometheus.yml
  • Paste the following content:
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["fake-exporter:9100"]
        labels:
          instance: "vm-04"
  • Save and exit nano (Ctrl+O, Enter, Ctrl+X).

What does this configuration do?

The scrape_interval: 15s tells Prometheus to collect data from all targets every 15 seconds.

Two scrape jobs are defined. The prometheus job monitors Prometheus itself. The node job targets the fake exporter container and labels it as vm-04. This label is how your AI assistant will identify the server when a NOC engineer asks about it.

  • Create the Loki configuration file by running:
nano loki/loki-config.yml
  • Paste the following content:
auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: "2020-10-24"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  allow_structured_metadata: true
  • Save and exit nano (Ctrl+O, Enter, Ctrl+X).

What does this Loki configuration do?

Loki is configured to store log chunks on the local filesystem inside the container. The inmemory ring store means Loki runs as a single instance with no external coordination service needed.

The tsdb store with schema v13 is Loki's current storage engine. It indexes logs by label and timestamp so that LogQL queries are fast.

  • Launch the entire monitoring stack in detached mode by running:
docker compose up -d

What does this command do?

The -d flag starts all containers in the background. Docker Compose will pull the images (Prometheus, Loki, Grafana) from Docker Hub, create the ntc-internal network, create named volumes, and start each container.

This may take a few minutes on the first run while images download.

  • Verify all containers are running by running:
docker compose ps

You should see all four containers listed with a status of Up. Each service (ntc-prometheus, ntc-loki, ntc-grafana, ntc-fake-exporter) should show its published port.

  • Confirm Prometheus is accessible by running:
curl -s http://localhost:9090/-/ready

You should see a response indicating Prometheus is ready to serve traffic.

  • Confirm Grafana is accessible by running:
curl -s -o /dev/null -w "%{http_code}" http://localhost:3000

You should see 200 returned, confirming Grafana is serving its web interface on port 3000.

Containers not starting?

  • If a container shows Exited instead of Up, check its logs with docker compose logs <service-name> to see the error.
  • If port 9090, 3100, or 3000 is already in use, stop the conflicting process or change the host port in docker-compose.yml.
  • If you see a file-not-found error for the Prometheus or Loki config, double check that prometheus/prometheus.yml and loki/loki-config.yml exist in the correct subdirectories.

Why aren't my Docker Compose containers starting?

✔️ Awesome, I've got everything!

Great. Double check you've saved all three files before moving on.

ⓧ I'd like to double check the full code

Here is the complete docker-compose.yml:

services:
  prometheus:
    image: prom/prometheus:v3.11.3
    container_name: ntc-prometheus
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    networks:
      - ntc-internal
    restart: unless-stopped

  loki:
    image: grafana/loki:3.7.0
    container_name: ntc-loki
    volumes:
      - ./loki/loki-config.yml:/etc/loki/local-config.yaml
      - loki_data:/loki
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - ntc-internal
    restart: unless-stopped

  grafana:
    image: grafana/grafana:13.0.0
    container_name: ntc-grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=ntcadmin
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
    ports:
      - "3000:3000"
    networks:
      - ntc-internal
    restart: unless-stopped

  fake-exporter:
    image: prom/prometheus:v3.11.3
    container_name: ntc-fake-exporter
    volumes:
      - ./fake-exporter/metrics.py:/metrics.py
    entrypoint: []
    command: >
      /bin/sh -c "echo 'Fake exporter placeholder - replace with Python exporter' && sleep infinity"
    networks:
      - ntc-internal
    restart: unless-stopped

networks:
  ntc-internal:
    driver: bridge
    internal: false

volumes:
  prometheus_data:
  loki_data:
  grafana_data:

Here is the complete prometheus/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["fake-exporter:9100"]
        labels:
          instance: "vm-04"

Here is the complete loki/loki-config.yml:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: "2020-10-24"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  allow_structured_metadata: true

Your monitoring stack is live. Prometheus is scraping, Loki is ready to receive logs, and Grafana is waiting for you at port 3000. Next up, you'll build the FastAPI backend that ties all of this data to the LLM.

Build the FastAPI Backend Structure

Your monitoring stack is live and collecting data. Now you need a backend API that can receive an engineer's question, pull metrics from Prometheus and logs from Loki, and pass everything to the LLM for diagnosis.

FastAPI is a Python web framework built for async operations. It handles multiple HTTP calls concurrently, which is exactly what you need when your API has to query Prometheus, Loki, and Ollama in sequence. In this step, you will scaffold the backend, set up configuration management, and create the diagnosis endpoint.

In this step, get ready to:
  • Create the backend project structure with pinned dependencies and a Dockerfile.
  • Implement environment-based configuration and the FastAPI application.
  • Build the diagnosis API endpoint and wire the backend into Docker Compose.
Create the backend folder structure, dependencies, and Dockerfile

A production-structured Python backend separates concerns into folders: routes handle HTTP, services hold business logic, config manages environment variables, and prompts store LLM templates.

  • Create the backend folder structure by running these commands:
mkdir -p backend/app/routes backend/app/services backend/app/config backend/app/prompts
  • Create the empty __init__.py files that tell Python each folder is an importable package:
touch backend/app/__init__.py backend/app/routes/__init__.py backend/app/services/__init__.py backend/app/config/__init__.py backend/app/prompts/__init__.py

Why all the __init__.py files?

Python uses __init__.py files to mark directories as packages. Without them, import app.routes.diagnose would fail because Python wouldn't recognize the folder structure as importable code.

  • Write the requirements.txt file inside the backend/ folder with these pinned dependencies:
fastapi==0.136.3
uvicorn==0.34.0
httpx==0.28.1
pydantic==2.11.3
pydantic-settings==2.9.1

What do these packages do?

  • fastapi is the web framework for building async APIs.
  • uvicorn is the ASGI server that runs your FastAPI app.
  • httpx is an async HTTP client for making outbound requests to Prometheus, Loki, and Ollama.
  • pydantic handles request/response data validation.
  • pydantic-settings reads configuration from environment variables.
  • Save backend/requirements.txt.
  • Now create the Dockerfile that packages your backend into a container. Write this inside the backend/ folder:
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app/ ./app/

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

What does this Dockerfile do?

  • FROM python:3.11-slim starts from a lightweight Python image.
  • It copies requirements.txt first and installs dependencies. This layer gets cached, so rebuilds are fast when only your code changes.
  • The CMD starts Uvicorn serving your FastAPI app on port 8000.
  • Save backend/Dockerfile.
  • Confirm both files exist by running:
ls backend/

You should see Dockerfile, requirements.txt, and the app/ directory listed.

Don't see the files?

Make sure you ran the mkdir -p and touch commands from your project root directory (the same folder where docker-compose.yml lives). Run ls to check you're in the right place.

help me find my project root directory

Implement configuration and the FastAPI application

Your backend needs to know where Prometheus, Loki, and Ollama are running. Rather than hardcoding URLs, you will read them from environment variables so the same code works in Docker and during local development.

  • Create backend/app/config/settings.py with the following configuration class:
from pydantic_settings import BaseSettings


class Settings(BaseSettings):
    prometheus_url: str = "http://prometheus:9090"
    loki_url: str = "http://loki:3100"
    ollama_url: str = "http://host.docker.internal:11434"
    ollama_model: str = "llama3.2:3b"
    ollama_timeout: int = 120

    class Config:
        env_file = ".env"


settings = Settings()

What does this configuration do?

Pydantic's BaseSettings automatically reads environment variables. If an env var PROMETHEUS_URL exists, it overrides the default http://prometheus:9090. This is how the Docker Compose environment variables will feed into your app.

The ollama_url defaults to http://host.docker.internal:11434 because Ollama runs on your host machine, not inside Docker. The special hostname host.docker.internal bridges from inside a container back to the host.

  • Save backend/app/config/settings.py.
  • Now create backend/app/main.py with the FastAPI application, CORS middleware, and the health check endpoint:
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from app.routes.diagnose import router as diagnose_router

app = FastAPI(
    title="NTC AI Observability Assistant",
    description="Offline AI-powered infrastructure diagnosis",
    version="1.0.0",
)

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

app.include_router(diagnose_router, prefix="/api")


@app.get("/health")
async def health_check():
    return {"status": "healthy"}

What does this code do?

  • The FastAPI() call creates your application with metadata that appears in the auto-generated docs.
  • CORS middleware allows requests from any origin. This is needed if you later add a web frontend.
  • include_router mounts the diagnosis routes at the /api prefix, keeping the URL namespace organized.
  • The /health endpoint gives Docker a quick way to confirm the app is alive.
  • Save backend/app/main.py.
Implement the diagnosis endpoint and add the backend to Docker Compose

The diagnosis endpoint is where an engineer's question enters the system. It accepts a question and optional server name, then delegates to a diagnosis service that orchestrates the full pipeline.

  • Create backend/app/routes/diagnose.py with the POST endpoint:
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel
from typing import Optional
from app.services.diagnosis_service import run_diagnosis

router = APIRouter()


class DiagnoseRequest(BaseModel):
    question: str
    server: Optional[str] = "vm-04"
    format: Optional[str] = "text"


class DiagnoseResponse(BaseModel):
    question: str
    server: str
    diagnosis: str
    metrics_context: str
    logs_context: str


@router.post("/diagnose", response_model=DiagnoseResponse)
async def diagnose_infrastructure(request: DiagnoseRequest):
    try:
        result = await run_diagnosis(
            question=request.question,
            server=request.server,
            output_format=request.format,
        )
        return result
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

What does this endpoint do?

  • DiagnoseRequest defines what the API accepts: a question (required), a server name (defaults to vm-04), and an output format.
  • DiagnoseResponse defines what the API returns: the original question, the AI diagnosis, plus the raw metrics and logs context for transparency.
  • The handler calls run_diagnosis() which will orchestrate the full pipeline once you build it in the next steps.
  • Save backend/app/routes/diagnose.py.

The route imports run_diagnosis from a service module that does not exist yet. You need a placeholder so the app can start without crashing.

  • Create backend/app/services/diagnosis_service.py with this temporary stub:
# Placeholder - full implementation comes in Step 4
async def run_diagnosis(question: str, server: str, output_format: str = "text") -> dict:
    return {
        "question": question,
        "server": server,
        "diagnosis": "Diagnosis service not yet implemented",
        "metrics_context": "No metrics fetched yet",
        "logs_context": "No logs fetched yet",
    }

Why a placeholder?

Python raises a ModuleNotFoundError if an import target does not exist. This stub lets FastAPI start up and serve the docs page while you build the real diagnosis logic in later steps.

  • Save backend/app/services/diagnosis_service.py.

Now you need to add the backend as a service in your Docker Compose file so it runs alongside the monitoring stack.

  • Open docker-compose.yml and find the fake-exporter service block. Add the following backend service below it (before the networks: section):
  backend:
    build: ./backend
    container_name: ntc-backend
    environment:
      - PROMETHEUS_URL=http://prometheus:9090
      - LOKI_URL=http://loki:3100
      - OLLAMA_URL=http://host.docker.internal:11434
    ports:
      - "8000:8000"
    networks:
      - ntc-internal
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: unless-stopped
    depends_on:
      - prometheus
      - loki

What does this service configuration do?

  • build: ./backend tells Docker Compose to build the image from your backend/Dockerfile.
  • The environment variables override the defaults in settings.py, pointing the backend at the other containers by their service names.
  • extra_hosts maps host.docker.internal to the host machine's gateway IP. This is how the backend container reaches Ollama running natively on your server.
  • depends_on ensures Prometheus and Loki start before the backend.
  • Save docker-compose.yml.
  • Rebuild and start all services by running:
docker compose up -d --build
  • Wait about 30 seconds for the backend to install dependencies and start. Check all containers are running:
docker compose ps

You should see ntc-backend listed alongside the other services with a status of Up.

  • Verify the FastAPI docs page is accessible by running:
curl http://localhost:8000/docs

You should see HTML content for the Swagger UI documentation page. This confirms your FastAPI backend is running and serving requests.

Backend container keeps restarting?

  • Check the container logs with docker compose logs backend to see the actual error message.
  • If you see a ModuleNotFoundError, make sure you created the stub diagnosis_service.py file with the run_diagnosis function.
  • If you see an indentation or syntax error, double check that your Python files match the code above exactly.

help me debug why my backend container won't start

✔️ Awesome, I've got everything!

Great. Double check you have saved all your files, then continue to the next step.

ⓧ I'd like to double check the full code

Here is the complete code for each file at the end of this step.

fastapi==0.136.3
uvicorn==0.34.0
httpx==0.28.1
pydantic==2.11.3
pydantic-settings==2.9.1
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app/ ./app/

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
from pydantic_settings import BaseSettings


class Settings(BaseSettings):
    prometheus_url: str = "http://prometheus:9090"
    loki_url: str = "http://loki:3100"
    ollama_url: str = "http://host.docker.internal:11434"
    ollama_model: str = "llama3.2:3b"
    ollama_timeout: int = 120

    class Config:
        env_file = ".env"


settings = Settings()
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from app.routes.diagnose import router as diagnose_router

app = FastAPI(
    title="NTC AI Observability Assistant",
    description="Offline AI-powered infrastructure diagnosis",
    version="1.0.0",
)

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

app.include_router(diagnose_router, prefix="/api")


@app.get("/health")
async def health_check():
    return {"status": "healthy"}
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel
from typing import Optional
from app.services.diagnosis_service import run_diagnosis

router = APIRouter()


class DiagnoseRequest(BaseModel):
    question: str
    server: Optional[str] = "vm-04"
    format: Optional[str] = "text"


class DiagnoseResponse(BaseModel):
    question: str
    server: str
    diagnosis: str
    metrics_context: str
    logs_context: str


@router.post("/diagnose", response_model=DiagnoseResponse)
async def diagnose_infrastructure(request: DiagnoseRequest):
    try:
        result = await run_diagnosis(
            question=request.question,
            server=request.server,
            output_format=request.format,
        )
        return result
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))
services:
  prometheus:
    image: prom/prometheus:v3.11.3
    container_name: ntc-prometheus
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    networks:
      - ntc-internal
    restart: unless-stopped

  loki:
    image: grafana/loki:3.7.0
    container_name: ntc-loki
    volumes:
      - ./loki/loki-config.yml:/etc/loki/local-config.yaml
      - loki_data:/loki
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - ntc-internal
    restart: unless-stopped

  grafana:
    image: grafana/grafana:13.0.0
    container_name: ntc-grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=ntcadmin
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
    ports:
      - "3000:3000"
    networks:
      - ntc-internal
    restart: unless-stopped

  fake-exporter:
    image: prom/prometheus:v3.11.3
    container_name: ntc-fake-exporter
    volumes:
      - ./fake-exporter/metrics.py:/metrics.py
    entrypoint: []
    command: >
      /bin/sh -c "echo 'Fake exporter placeholder - replace with Python exporter' && sleep infinity"
    networks:
      - ntc-internal
    restart: unless-stopped

  backend:
    build: ./backend
    container_name: ntc-backend
    environment:
      - PROMETHEUS_URL=http://prometheus:9090
      - LOKI_URL=http://loki:3100
      - OLLAMA_URL=http://host.docker.internal:11434
    ports:
      - "8000:8000"
    networks:
      - ntc-internal
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: unless-stopped
    depends_on:
      - prometheus
      - loki

networks:
  ntc-internal:
    driver: bridge
    internal: false

volumes:
  prometheus_data:
  loki_data:
  grafana_data:

Your backend is live and serving API documentation. Next up, you will build the service modules that fetch real metrics from Prometheus and logs from Loki to feed into your diagnosis pipeline.

Integrate Prometheus and Loki Data Fetching

Your FastAPI backend has a solid foundation with routes and configuration. The missing piece is the data layer. Right now, the /api/diagnose endpoint references a diagnosis service that does not yet exist.

In this step, you will build the service modules that pull live metrics from Prometheus and recent logs from Loki. These services give the LLM real infrastructure data to reason about instead of guessing.

In this step, get ready to:
  • Build the Prometheus service module that fetches CPU, memory, and disk metrics via PromQL.
  • Build the Loki service module that fetches recent error and warning logs via LogQL.
  • Create a context builder that combines metrics and logs into a structured string for the LLM.
Implement the Prometheus service module

Prometheus exposes an HTTP API at /api/v1/query that accepts PromQL queries and returns instant metric values. Your service module will send three PromQL queries (CPU, memory, disk) and format the results as human-readable strings.

  • Create a new file at backend/app/services/prometheus_service.py and add the following code:
import httpx
from app.config.settings import settings


async def fetch_cpu_usage(server: str) -> str:
    query = f'100 - (avg by(instance) (rate(node_cpu_seconds_total{{instance="{server}",mode="idle"}}[5m])) * 100)'
    result = await _query_prometheus(query)
    if result:
        return f"CPU Usage: {result}%"
    return "CPU Usage: No data available"


async def fetch_memory_usage(server: str) -> str:
    query = f'(1 - (node_memory_MemAvailable_bytes{{instance="{server}"}} / node_memory_MemTotal_bytes{{instance="{server}"}})) * 100'
    result = await _query_prometheus(query)
    if result:
        return f"Memory Usage: {result}%"
    return "Memory Usage: No data available"


async def fetch_disk_usage(server: str) -> str:
    query = f'(1 - (node_filesystem_avail_bytes{{instance="{server}",mountpoint="/"}} / node_filesystem_size_bytes{{instance="{server}",mountpoint="/"}})) * 100'
    result = await _query_prometheus(query)
    if result:
        return f"Disk Usage: {result}%"
    return "Disk Usage: No data available"


async def _query_prometheus(query: str) -> str:
    url = f"{settings.prometheus_url}/api/v1/query"
    async with httpx.AsyncClient(timeout=10.0) as client:
        try:
            response = await client.get(url, params={"query": query})
            response.raise_for_status()
            data = response.json()
            if data["status"] == "success" and data["data"]["result"]:
                value = data["data"]["result"][0]["value"][1]
                return f"{float(value):.2f}"
        except (httpx.HTTPError, KeyError, IndexError, ValueError):
            pass
    return ""

What does this code do?

This module has two layers. The three public functions (fetch_cpu_usage, fetch_memory_usage, fetch_disk_usage) each build a PromQL query string targeting a specific metric for the given server name.

The private _query_prometheus helper makes the actual HTTP GET request to /api/v1/query. It parses the JSON response, extracts the numeric value, and formats it to two decimal places. If Prometheus returns no data (as it will with our fake exporter), the functions return a "No data available" message instead of crashing.

Why does this use httpx instead of requests?

The httpx library supports Python's async/await natively. Since FastAPI handlers are async, using httpx means your backend can fetch CPU, memory, and disk metrics concurrently without blocking the event loop.

  • Save backend/app/services/prometheus_service.py.

Seeing a syntax error with the curly braces?

The double curly braces {{ and }} are Python f-string escapes that produce literal { and } characters in the output. PromQL uses braces for label selectors like {instance="vm-04"}. Make sure every opening {{ has a matching }}.

help me fix the PromQL f-string syntax in my prometheus_service.py

Implement the Loki service module

Loki works differently from Prometheus. Instead of instant queries, you query a time range and get back streams of log lines. The /loki/api/v1/query_range endpoint accepts LogQL, start/end timestamps in nanoseconds, and returns matching log entries.

  • Create a new file at backend/app/services/loki_service.py and add the following code:
import httpx
import time
from app.config.settings import settings


async def fetch_recent_logs(server: str, minutes: int = 30) -> str:
    end_ns = str(int(time.time() * 1e9))
    start_ns = str(int((time.time() - minutes * 60) * 1e9))
    query = f'{{job="syslog", instance="{server}"}} |= "error" or "ERROR" or "warning" or "WARN"'

    url = f"{settings.loki_url}/loki/api/v1/query_range"
    params = {
        "query": query,
        "start": start_ns,
        "end": end_ns,
        "limit": 50,
        "direction": "backward",
    }

    async with httpx.AsyncClient(timeout=10.0) as client:
        try:
            response = await client.get(url, params=params)
            response.raise_for_status()
            data = response.json()
            if data["status"] == "success":
                logs = _parse_log_streams(data["data"]["result"])
                if logs:
                    return "\n".join(logs[-20:])
        except (httpx.HTTPError, KeyError, IndexError):
            pass
    return "No recent error/warning logs found"


def _parse_log_streams(results: list) -> list:
    log_lines = []
    for stream in results:
        for value in stream.get("values", []):
            log_lines.append(value[1])
    return log_lines

What does this code do?

The fetch_recent_logs function builds a LogQL query that filters for error and warning messages from a specific server over the last 30 minutes. Loki expects timestamps in nanoseconds, so the function multiplies time.time() by 1e9.

The _parse_log_streams helper iterates through Loki's response format (an array of streams, each containing an array of [timestamp, log_line] pairs) and extracts just the log text. It returns the 20 most recent matching lines.

Since our stack has no log shipper pushing data to Loki yet, this will return "No recent error/warning logs found". The LLM prompt in the next step handles missing data gracefully.

  • Save backend/app/services/loki_service.py.

Confused by the nanosecond timestamps?

Loki requires start and end times in Unix nanoseconds. Python's time.time() returns seconds, so multiplying by 1e9 converts to nanoseconds. If you use seconds directly, Loki interprets them as dates in 1970 and returns nothing.

help me understand Loki's timestamp format

Build the context builder that combines metrics and logs

The context builder is the glue between your data-fetching services and the LLM prompt. It calls all three Prometheus fetchers and the Loki fetcher, then formats everything into a structured string.

  • Create a new file at backend/app/services/context_builder.py and add the following code:
from app.services.prometheus_service import (
    fetch_cpu_usage,
    fetch_memory_usage,
    fetch_disk_usage,
)
from app.services.loki_service import fetch_recent_logs


async def build_context(server: str) -> tuple:
    cpu = await fetch_cpu_usage(server)
    memory = await fetch_memory_usage(server)
    disk = await fetch_disk_usage(server)

    metrics_context = f"""Server: {server}
{cpu}
{memory}
{disk}"""

    logs_context = await fetch_recent_logs(server)

    return metrics_context, logs_context

What does this code do?

The build_context function orchestrates the data-gathering phase. It calls each metric fetcher and the log fetcher, then assembles a metrics_context string with the server name and all three metric readings.

It returns a tuple of (metrics_context, logs_context) that the diagnosis service will inject into the LLM prompt in the next step.

  • Save backend/app/services/context_builder.py.
  • Verify your backend/app/services/ folder now contains three new files by running:
ls backend/app/services/

You should see __init__.py, context_builder.py, loki_service.py, and prometheus_service.py listed.

Missing a file?

Double check you created the files inside backend/app/services/ and not in the project root or another directory. The import paths in context_builder.py rely on the exact folder structure.

help me fix my project directory structure

✔️ Awesome, I've got everything!

Great. Make sure all three files are saved before continuing.

ⓧ I'd like to double check the full code

Here is the full code for each service file.

import httpx
from app.config.settings import settings


async def fetch_cpu_usage(server: str) -> str:
    query = f'100 - (avg by(instance) (rate(node_cpu_seconds_total{{instance="{server}",mode="idle"}}[5m])) * 100)'
    result = await _query_prometheus(query)
    if result:
        return f"CPU Usage: {result}%"
    return "CPU Usage: No data available"


async def fetch_memory_usage(server: str) -> str:
    query = f'(1 - (node_memory_MemAvailable_bytes{{instance="{server}"}} / node_memory_MemTotal_bytes{{instance="{server}"}})) * 100'
    result = await _query_prometheus(query)
    if result:
        return f"Memory Usage: {result}%"
    return "Memory Usage: No data available"


async def fetch_disk_usage(server: str) -> str:
    query = f'(1 - (node_filesystem_avail_bytes{{instance="{server}",mountpoint="/"}} / node_filesystem_size_bytes{{instance="{server}",mountpoint="/"}})) * 100'
    result = await _query_prometheus(query)
    if result:
        return f"Disk Usage: {result}%"
    return "Disk Usage: No data available"


async def _query_prometheus(query: str) -> str:
    url = f"{settings.prometheus_url}/api/v1/query"
    async with httpx.AsyncClient(timeout=10.0) as client:
        try:
            response = await client.get(url, params={"query": query})
            response.raise_for_status()
            data = response.json()
            if data["status"] == "success" and data["data"]["result"]:
                value = data["data"]["result"][0]["value"][1]
                return f"{float(value):.2f}"
        except (httpx.HTTPError, KeyError, IndexError, ValueError):
            pass
    return ""
import httpx
import time
from app.config.settings import settings


async def fetch_recent_logs(server: str, minutes: int = 30) -> str:
    end_ns = str(int(time.time() * 1e9))
    start_ns = str(int((time.time() - minutes * 60) * 1e9))
    query = f'{{job="syslog", instance="{server}"}} |= "error" or "ERROR" or "warning" or "WARN"'

    url = f"{settings.loki_url}/loki/api/v1/query_range"
    params = {
        "query": query,
        "start": start_ns,
        "end": end_ns,
        "limit": 50,
        "direction": "backward",
    }

    async with httpx.AsyncClient(timeout=10.0) as client:
        try:
            response = await client.get(url, params=params)
            response.raise_for_status()
            data = response.json()
            if data["status"] == "success":
                logs = _parse_log_streams(data["data"]["result"])
                if logs:
                    return "\n".join(logs[-20:])
        except (httpx.HTTPError, KeyError, IndexError):
            pass
    return "No recent error/warning logs found"


def _parse_log_streams(results: list) -> list:
    log_lines = []
    for stream in results:
        for value in stream.get("values", []):
            log_lines.append(value[1])
    return log_lines
from app.services.prometheus_service import (
    fetch_cpu_usage,
    fetch_memory_usage,
    fetch_disk_usage,
)
from app.services.loki_service import fetch_recent_logs


async def build_context(server: str) -> tuple:
    cpu = await fetch_cpu_usage(server)
    memory = await fetch_memory_usage(server)
    disk = await fetch_disk_usage(server)

    metrics_context = f"""Server: {server}
{cpu}
{memory}
{disk}"""

    logs_context = await fetch_recent_logs(server)

    return metrics_context, logs_context

Your data layer is complete. The services know how to pull metrics and logs from the monitoring stack and package them into structured context. Next up, you will build the prompt engineering template and wire everything to the local LLM for AI-powered diagnosis.

Build the Prompt Engineering Pipeline and Wire Ollama

Your backend can now fetch live metrics from Prometheus and logs from Loki. But raw numbers and log lines alone don't answer an engineer's question like "Why is vm-04 slow?"

In this step, you'll build the bridge between observability data and human-readable answers. You'll design a prompt template that feeds metrics and logs to a local LLM, then wire Ollama into the pipeline so the system returns a structured root cause analysis.

In this step, get ready to:
  • Design a structured diagnosis prompt template for infrastructure analysis.
  • Connect to Ollama's local LLM for offline AI inference.
  • Wire the full pipeline from question to AI-generated diagnosis.
Create the diagnosis prompt template

Prompt engineering is where you shape how the LLM interprets data and formats its response. A well-structured prompt constrains the model to only reference the provided evidence and output a consistent structure that NOC engineers can act on quickly.

  • Create the empty __init__.py file to make prompts a Python package by running:
touch backend/app/prompts/__init__.py
  • Create the file backend/app/prompts/diagnosis.py and add the build_diagnosis_prompt function by pasting the following code:
def build_diagnosis_prompt(
    question: str, server: str, metrics_context: str, logs_context: str
) -> list:
    system_message = """You are an expert infrastructure engineer at NTC Pakistan. Your role is to diagnose server issues based on metrics and logs provided to you.

Rules:
- Only use the data provided. Do not hallucinate or invent data.
- If data is missing or shows no issues, say so clearly.
- Provide your analysis in this structure: Root Cause, Evidence, Severity (Critical/High/Medium/Low), Recommended Fix.
- Be concise and actionable. NOC engineers need quick answers."""

    user_message = f"""NOC Engineer Question: {question}
Server: {server}

=== CURRENT METRICS ===
{metrics_context}

=== RECENT ERROR/WARNING LOGS (last 30 minutes) ===
{logs_context}

Based on the metrics and logs above, provide your diagnosis."""

    return [
        {"role": "system", "content": system_message},
        {"role": "user", "content": user_message},
    ]

What does this prompt do?

The prompt uses a two-message structure that Ollama's chat endpoint expects.

  • The system message sets the model's role as an NTC infrastructure expert and constrains it to only use provided data.
  • The user message injects the engineer's question, live metrics, and recent logs into clearly labeled sections.
  • The structured output format (Root Cause, Evidence, Severity, Recommended Fix) ensures consistent responses that engineers can act on.
  • Below the function, add the JSON schema definition that will be used later for structured output mode:
DIAGNOSIS_JSON_SCHEMA = {
    "type": "object",
    "properties": {
        "root_cause": {"type": "string"},
        "evidence": {"type": "string"},
        "severity": {
            "type": "string",
            "enum": ["Critical", "High", "Medium", "Low"],
        },
        "confidence_percent": {"type": "integer"},
        "recommended_actions": {
            "type": "array",
            "items": {"type": "string"},
        },
    },
    "required": [
        "root_cause",
        "evidence",
        "severity",
        "confidence_percent",
        "recommended_actions",
    ],
}

What is this JSON schema for?

Ollama supports a format field that accepts a JSON schema object to enforce structured output from the LLM. This schema defines the exact shape of a machine-readable diagnosis response.

You won't use it on the main path yet, but the diagnosis service will pass it to Ollama when a caller requests JSON format.

  • Save backend/app/prompts/diagnosis.py.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

def build_diagnosis_prompt(
    question: str, server: str, metrics_context: str, logs_context: str
) -> list:
    system_message = """You are an expert infrastructure engineer at NTC Pakistan. Your role is to diagnose server issues based on metrics and logs provided to you.

Rules:
- Only use the data provided. Do not hallucinate or invent data.
- If data is missing or shows no issues, say so clearly.
- Provide your analysis in this structure: Root Cause, Evidence, Severity (Critical/High/Medium/Low), Recommended Fix.
- Be concise and actionable. NOC engineers need quick answers."""

    user_message = f"""NOC Engineer Question: {question}
Server: {server}

=== CURRENT METRICS ===
{metrics_context}

=== RECENT ERROR/WARNING LOGS (last 30 minutes) ===
{logs_context}

Based on the metrics and logs above, provide your diagnosis."""

    return [
        {"role": "system", "content": system_message},
        {"role": "user", "content": user_message},
    ]


DIAGNOSIS_JSON_SCHEMA = {
    "type": "object",
    "properties": {
        "root_cause": {"type": "string"},
        "evidence": {"type": "string"},
        "severity": {
            "type": "string",
            "enum": ["Critical", "High", "Medium", "Low"],
        },
        "confidence_percent": {"type": "integer"},
        "recommended_actions": {
            "type": "array",
            "items": {"type": "string"},
        },
    },
    "required": [
        "root_cause",
        "evidence",
        "severity",
        "confidence_percent",
        "recommended_actions",
    ],
}
Implement the Ollama service

The Ollama service is the module that sends the constructed prompt to your locally-running Llama 3.2 model and returns its response. It posts to Ollama's /api/chat endpoint with streaming disabled so you get a single complete response.

  • Create the file backend/app/services/ollama_service.py and paste the following code:
import httpx
from app.config.settings import settings


async def generate_diagnosis(prompt_messages: list, json_schema: dict = None) -> str:
    url = f"{settings.ollama_url}/api/chat"
    payload = {
        "model": settings.ollama_model,
        "messages": prompt_messages,
        "stream": False,
        "options": {
            "temperature": 0.3,
        },
    }

    if json_schema:
        payload["format"] = json_schema

    async with httpx.AsyncClient(timeout=settings.ollama_timeout) as client:
        try:
            response = await client.post(url, json=payload)
            response.raise_for_status()
            data = response.json()
            return data["message"]["content"]
        except httpx.TimeoutException:
            return "Error: LLM inference timed out. The model may be loading. Please try again."
        except httpx.HTTPError as e:
            return f"Error: Could not reach Ollama at {settings.ollama_url}. Ensure Ollama is running. Details: {str(e)}"

What does this service do?

  • It builds a payload targeting the llama3.2:3b model with stream: false (wait for the full answer) and temperature: 0.3 (low randomness for factual, consistent answers).
  • If a json_schema is provided, it passes the schema in the format field to enforce structured JSON output from the model.
  • Timeouts and HTTP errors return human-readable error strings instead of crashing the API.

Why temperature 0.3?

Temperature controls how creative or random the LLM's output is. For infrastructure diagnosis you want factual, reproducible answers. A low temperature like 0.3 keeps the model focused on the evidence provided rather than speculating.

  • Save backend/app/services/ollama_service.py.

Worried about the 120-second timeout?

The first request to Ollama after a restart can take longer because the model loads into memory. The 120-second timeout (set in settings.py) accounts for this cold start. Subsequent requests are much faster.

help me understand Ollama cold start times and how to pre-warm the model

Wire the diagnosis service together

The diagnosis service is the orchestrator. It receives the engineer's question, fetches context from Prometheus and Loki, formats the prompt, sends it to Ollama, and returns the complete result.

  • Create the file backend/app/services/diagnosis_service.py and paste the following code:
from app.services.context_builder import build_context
from app.services.ollama_service import generate_diagnosis
from app.prompts.diagnosis import build_diagnosis_prompt, DIAGNOSIS_JSON_SCHEMA


async def run_diagnosis(
    question: str, server: str, output_format: str = "text"
) -> dict:
    metrics_context, logs_context = await build_context(server)

    prompt_messages = build_diagnosis_prompt(
        question=question,
        server=server,
        metrics_context=metrics_context,
        logs_context=logs_context,
    )

    json_schema = DIAGNOSIS_JSON_SCHEMA if output_format == "json" else None
    diagnosis = await generate_diagnosis(prompt_messages, json_schema=json_schema)

    return {
        "question": question,
        "server": server,
        "diagnosis": diagnosis,
        "metrics_context": metrics_context,
        "logs_context": logs_context,
    }

What does this orchestrate?

  • build_context fetches CPU, memory, disk metrics from Prometheus and recent error logs from Loki.
  • build_diagnosis_prompt formats the fetched data into the system/user message pair the LLM expects.
  • generate_diagnosis sends the prompt to Ollama and returns the AI's response.
  • The returned dict includes both the diagnosis AND the raw context, so the caller can see exactly what data the LLM was given.
  • Save backend/app/services/diagnosis_service.py.
  • Rebuild and restart the backend container to pick up your new files by running:
docker compose up -d --build backend
  • Wait for the container to start, then run the full end-to-end test by running:
curl -X POST http://localhost:8000/api/diagnose -H "Content-Type: application/json" -d '{"question": "Why is vm-04 experiencing high CPU?", "server": "vm-04"}'

You should see a JSON response containing a diagnosis field with a natural-language analysis from the LLM. It will reference the metrics and logs context (likely noting "No data available" for metrics and "No recent error/warning logs found" since the fake exporter doesn't produce real node_exporter metrics).

Why does it say no data available?

The fake exporter is a placeholder. It doesn't expose real node_exporter metrics, so Prometheus has no CPU/memory/disk data to return. The LLM is instructed to acknowledge missing data honestly.

The important thing is that the full pipeline works end-to-end. The model receives the context, understands the question, and produces a structured diagnosis. In a real deployment with actual node exporters, the metrics would be populated.

Getting a timeout or connection error?

  • Make sure Ollama is running on your host machine. Check with ollama list to confirm the llama3.2:3b model is available.
  • The first request can be slow (30-60 seconds) as the model loads into memory. Wait and try again.
  • If you see "Could not reach Ollama", verify the Docker container can reach your host. The extra_hosts setting in docker-compose.yml maps host.docker.internal to the host gateway.

help me debug why my backend can't reach Ollama inside Docker

✔️ Awesome, I've got everything!

Great. Make sure all three new files are saved.

ⓧ I'd like to double check the full code

def build_diagnosis_prompt(
    question: str, server: str, metrics_context: str, logs_context: str
) -> list:
    system_message = """You are an expert infrastructure engineer at NTC Pakistan. Your role is to diagnose server issues based on metrics and logs provided to you.

Rules:
- Only use the data provided. Do not hallucinate or invent data.
- If data is missing or shows no issues, say so clearly.
- Provide your analysis in this structure: Root Cause, Evidence, Severity (Critical/High/Medium/Low), Recommended Fix.
- Be concise and actionable. NOC engineers need quick answers."""

    user_message = f"""NOC Engineer Question: {question}
Server: {server}

=== CURRENT METRICS ===
{metrics_context}

=== RECENT ERROR/WARNING LOGS (last 30 minutes) ===
{logs_context}

Based on the metrics and logs above, provide your diagnosis."""

    return [
        {"role": "system", "content": system_message},
        {"role": "user", "content": user_message},
    ]


DIAGNOSIS_JSON_SCHEMA = {
    "type": "object",
    "properties": {
        "root_cause": {"type": "string"},
        "evidence": {"type": "string"},
        "severity": {
            "type": "string",
            "enum": ["Critical", "High", "Medium", "Low"],
        },
        "confidence_percent": {"type": "integer"},
        "recommended_actions": {
            "type": "array",
            "items": {"type": "string"},
        },
    },
    "required": [
        "root_cause",
        "evidence",
        "severity",
        "confidence_percent",
        "recommended_actions",
    ],
}
import httpx
from app.config.settings import settings


async def generate_diagnosis(prompt_messages: list, json_schema: dict = None) -> str:
    url = f"{settings.ollama_url}/api/chat"
    payload = {
        "model": settings.ollama_model,
        "messages": prompt_messages,
        "stream": False,
        "options": {
            "temperature": 0.3,
        },
    }

    if json_schema:
        payload["format"] = json_schema

    async with httpx.AsyncClient(timeout=settings.ollama_timeout) as client:
        try:
            response = await client.post(url, json=payload)
            response.raise_for_status()
            data = response.json()
            return data["message"]["content"]
        except httpx.TimeoutException:
            return "Error: LLM inference timed out. The model may be loading. Please try again."
        except httpx.HTTPError as e:
            return f"Error: Could not reach Ollama at {settings.ollama_url}. Ensure Ollama is running. Details: {str(e)}"
from app.services.context_builder import build_context
from app.services.ollama_service import generate_diagnosis
from app.prompts.diagnosis import build_diagnosis_prompt, DIAGNOSIS_JSON_SCHEMA


async def run_diagnosis(
    question: str, server: str, output_format: str = "text"
) -> dict:
    metrics_context, logs_context = await build_context(server)

    prompt_messages = build_diagnosis_prompt(
        question=question,
        server=server,
        metrics_context=metrics_context,
        logs_context=logs_context,
    )

    json_schema = DIAGNOSIS_JSON_SCHEMA if output_format == "json" else None
    diagnosis = await generate_diagnosis(prompt_messages, json_schema=json_schema)

    return {
        "question": question,
        "server": server,
        "diagnosis": diagnosis,
        "metrics_context": metrics_context,
        "logs_context": logs_context,
    }

Your AI diagnosis pipeline is fully wired. A question goes in, metrics and logs are fetched, the prompt is constructed, and the local LLM returns a structured diagnosis. Next up, you'll prove this entire system works with zero internet connectivity.

Test End-to-End and Verify Offline Operation

Your AI observability pipeline is fully built. Every service is running, and the diagnosis endpoint already returns LLM-generated analysis.

The whole point of this project is that it works without internet. In this step, you will cut off outbound network access, run real diagnosis queries, and prove that the entire system operates in air-gapped mode.

In this step, get ready to:
  • Block outbound internet traffic and confirm the diagnosis pipeline still works.
  • Test three different diagnosis scenarios (high CPU, memory leak, disk full).
  • Reconnect internet and review the Grafana dashboard to confirm metric collection.
Block outbound internet traffic

An air-gapped system means zero reliance on external servers. The strongest way to prove this is to physically block all outbound internet traffic and confirm nothing breaks.

  • Block all outbound traffic on your external network interface by running this command:
sudo iptables -A OUTPUT -o eth0 -j DROP

What does this command do?

This adds a firewall rule that drops all packets leaving through eth0 (your external network interface). Local traffic between Docker containers and the host still flows through internal interfaces (docker0, loopback), so your stack keeps working.

  • Confirm internet is blocked by running:
curl -s --max-time 5 https://google.com || echo "Internet is blocked"

You should see the message "Internet is blocked" printed in your terminal.

  • Now test the diagnosis endpoint while offline by running:
curl -X POST http://localhost:8000/api/diagnose \
  -H "Content-Type: application/json" \
  -d '{"question": "Why is vm-04 experiencing high CPU?", "server": "vm-04"}'

The response should contain a natural-language diagnosis from the LLM. It may take 10-30 seconds for the model to generate its answer.

What should I see?

The JSON response includes a diagnosis field with the LLM's analysis. Since the fake exporter does not produce real metrics, the LLM will acknowledge that data is unavailable or missing. This is correct behaviour. The important thing is that the response is generated locally without any internet dependency.

Getting a timeout or connection error?

Make sure Ollama is still running on your host. The iptables rule blocks external traffic, but Ollama on localhost:11434 uses the loopback interface which is unaffected.

Run ollama list to confirm Ollama is responsive. If the LLM takes over 120 seconds, it may have timed out. Try the request again as the model warms up on first use.

If your network interface is not named eth0 (check with ip link show), adjust the iptables command to match your actual interface name.

Help me debug my offline diagnosis request.

Test three diagnosis scenarios

A single test proves connectivity. Multiple tests with different questions prove the LLM can reason about different infrastructure problems. Each query should return a distinct diagnosis based on the question asked.

  • Test a high CPU scenario by running:
curl -X POST http://localhost:8000/api/diagnose \
  -H "Content-Type: application/json" \
  -d '{"question": "Why is vm-04 experiencing high CPU?", "server": "vm-04"}'
  • Test a memory leak scenario by running:
curl -X POST http://localhost:8000/api/diagnose \
  -H "Content-Type: application/json" \
  -d '{"question": "Is there a memory leak on vm-04?", "server": "vm-04"}'
  • Test a disk full scenario by running:
curl -X POST http://localhost:8000/api/diagnose \
  -H "Content-Type: application/json" \
  -d '{"question": "Is vm-04 running out of disk space?", "server": "vm-04"}'

Why do the diagnoses differ?

Even though the underlying metrics data is the same (or unavailable from the fake exporter), the LLM tailors its response to each question. It focuses on CPU-related causes for the first query, memory patterns for the second, and filesystem issues for the third.

This demonstrates how the prompt engineering template you built in Step 4 guides the model to reason specifically about whatever the NOC engineer asks.

Each response should arrive within 10-30 seconds. All three should complete successfully with no network errors.

Seeing an error about Ollama being unreachable?

If the response contains the text "Could not reach Ollama", the iptables rule may be blocking Docker's access to the host. Verify your interface name is correct. Run ip route and check which interface has the default route. That is the one to block.

Help me figure out why Ollama is unreachable in offline mode.

Reconnect internet and review Grafana

Now that you have confirmed the pipeline works offline, restore your internet connection and verify that the monitoring stack has been collecting metrics the entire time.

  • Remove the firewall rule and restore internet access by running:
sudo iptables -D OUTPUT -o eth0 -j DROP
  • Confirm internet is restored by running:
curl -s --max-time 5 https://google.com > /dev/null && echo "Internet restored"

You should see "Internet restored" in your terminal output.

  • Open your browser and navigate to http://localhost:3000 to access Grafana.
  • Log in with username admin and password ntcadmin.
  • Click the hamburger menu (three horizontal lines) in the top left.
  • Select **Connections**, then **Data sources**.
  • Click **Add data source** and select **Prometheus**.
  • Enter http://prometheus:9090 in the **Prometheus server URL** field.
  • Click **Save & test** at the bottom of the page.

You should see a green success message confirming that Grafana can reach Prometheus.

  • Click **Explore** in the left sidebar to query metrics directly.
  • In the query editor, type up and click **Run query**.

You should see a result showing which scrape targets are up. The prometheus job should show a value of 1, confirming that Prometheus has been actively scraping metrics while you tested offline.

What does this prove?

The entire observability stack continued collecting and storing metrics even while outbound internet was blocked. Prometheus scraped its own metrics internally, Loki stayed ready to receive logs, and Grafana kept serving dashboards. No component needed external access to function.

Grafana not loading at port 3000?

Run docker compose ps and confirm the ntc-grafana container is healthy. If it restarted during the offline test, give it a few seconds to come back up.

Help me access the Grafana dashboard.

You have just proven that your AI observability assistant works completely offline. Every component communicates locally, the LLM generates diagnoses without touching the internet, and the monitoring stack keeps collecting data regardless of network state. This is a production-ready, air-gapped architecture.

Secret mission

Add Structured JSON Output Mode

Your diagnosis endpoint returns plain text. That works for humans reading it on screen, but what about automated alerting systems that need to parse severity and recommended actions programmatically? Add a structured JSON output mode that returns machine-readable diagnosis reports.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on your local machine using Docker containers and a locally-hosted LLM. There are no cloud services or ongoing costs to worry about.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Docker containers (ntc-prometheus, ntc-loki, ntc-grafana, ntc-fake-exporter, ntc-backend).
  • Docker volumes (prometheus_data, loki_data, grafana_data).
  • Docker images (Prometheus, Loki, Grafana, Python, and the built backend image).
  • Project files on disk (docker-compose.yml, backend/, prometheus/, loki/, fake-exporter/).
  • Docker Engine and Docker Compose plugin (installed in Step 1).

Keep everything running

No action needed. Choose this if you are still actively building on this project or plan to extend it soon.

  • Your Docker containers continue running in the background, scraping metrics and ready to accept diagnosis requests.
  • Ollama remains available on your host for inference at any time.
  • Docker Engine stays installed and configured on your system.

Pause - I'll come back to this later

Stop the running containers to free up CPU and memory, but keep all files and images so you can restart quickly.

  • Stop all project containers by running this command from your project directory:
docker compose down

This stops and removes the containers and network, but keeps your Docker volumes and images intact.

  • When you are ready to resume, run this from the same directory:
docker compose up -d

Everything restarts with your previous data preserved in the volumes.

Delete - I don't want to use this again

Remove all project containers, volumes, images, and files completely.

  • Stop all containers and remove volumes and images by running this from your project directory:
docker compose down -v --rmi all
  • Delete the project directory:
cd ~
rm -rf ~/Desktop/ntc-observability
  • If you want to remove Docker Engine entirely from your system, run:
sudo apt purge docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras
  • Remove leftover Docker data:
sudo rm -rf /var/lib/docker
sudo rm -rf /var/lib/containerd

Nice Work!

Nice Work!

You just built a fully offline AI observability assistant that diagnoses infrastructure issues using live metrics, logs, and local LLM inference. No cloud APIs, no internet dependency, no data leaving your servers.

You've learned how to:

  • Deploy a complete monitoring stack (Prometheus, Loki, Grafana) using Docker Compose and wire it into a FastAPI backend with modular service architecture.
  • Fetch real-time infrastructure data using PromQL and LogQL, then feed it into a structured prompt engineering template designed for infrastructure diagnosis.
  • Connect to a local Llama 3.2 model via Ollama for completely air-gapped AI inference and prove the entire system works with zero internet dependency.
  • Secret Mission: Added a structured JSON output mode that returns machine-readable diagnosis reports with severity, confidence, and recommended actions.

Ready to quiz yourself?