AI-Powered Log Analysis & Incident Response
Build an autonomous SRE system that diagnoses Kubernetes errors with AI.
Introduction
30 Second Summary
When something breaks in a production system at 3am, someone gets paged. They dig through logs, diagnose the problem, and write up an incident report with a suggested fix.
In this project, you will build an autonomous SRE system that watches your Kubernetes cluster for errors, diagnoses them with a local AI model, and automatically creates incident reports with remediation steps.
What You'll Build
A complete incident response pipeline that connects Fluent Bit log collection, a FastAPI middleware service, Ollama LLM analysis, Redis caching, and automated GitHub Issue creation into a single workflow running on your local machine.
By the end of this project, you'll have:
- A FastAPI service that receives real-time error logs from your Kubernetes cluster and filters them by severity.
- An LLM-powered diagnosis engine that categorizes errors, identifies root causes, and suggests kubectl remediation commands.
- A full end-to-end pipeline where Fluent Bit catches errors, Ollama diagnoses them, Redis caches results, and a GitHub Issue appears automatically.
- Secret Mission: Add Slack webhook alerting for real-time notifications alongside your GitHub Issues.
Are there any prerequisites?
This project assumes basic familiarity with Python, Kubernetes concepts (pods, deployments), and REST APIs. You should also have Docker Desktop with Kubernetes enabled on your machine.
Install Ollama and Pull the Model
Your autonomous SRE system needs a local LLM to analyze error logs and suggest remediation commands. Ollama lets you run large language models directly on your machine with no cloud API costs or network latency.
In this step, you'll install Ollama, download a lightweight model optimized for structured output, and confirm the API is ready to receive requests from the rest of your pipeline.
In this step, get ready to:
- Install Ollama and verify the CLI.
- Pull the qwen3:1.7b model and pre-load it into memory.
- Confirm the API responds on localhost:11434.
Install Ollama on Ubuntu
Ollama runs as a system service on Linux, exposing a local HTTP API that your FastAPI middleware will call later.
- Install Ollama by running the official install script:
curl -fsSL https://ollama.com/install.sh | shWhat does this command do?
This downloads and runs Ollama's official installer. It installs the ollama binary to /usr/bin/ollama, creates a systemd service so Ollama starts automatically, and sets up the ollama user and group for the service.
- Verify the installation succeeded by checking the version:
ollama -v- You should see a version number printed in your terminal (e.g. ollama version is 0.x.x).
Command not found?
If your terminal says ollama: command not found, try opening a new terminal window. The installer adds Ollama to your PATH, but existing sessions may not pick up the change.
If the install script itself failed, make sure curl is installed (sudo apt install curl) and that you have internet connectivity.
help me troubleshoot the Ollama installation on Ubuntu
Pull the qwen3:1.7b model
The qwen3:1.7b model is a 1.4GB download that balances speed and quality for log analysis tasks. It generates structured JSON output, which your SRE pipeline needs for parsing diagnoses.
- Download the model by running:
ollama pull qwen3:1.7bWhy qwen3:1.7b?
At 1.4GB (Q4_K_M quantization), this model loads fast on CPU and generates responses in seconds. For log analysis (pattern matching and remediation lookup), a 1.7B parameter model is sufficient.
Larger models would make the demo painfully slow without a dedicated GPU. You can always swap in a bigger model later if you have the hardware.
- Once the download finishes, run a warm-up query to pre-load the model into RAM:
ollama run qwen3:1.7b "What is a Kubernetes pod?"Why a warm-up query?
The first time you query a model after pulling it, Ollama loads it into RAM. This initial load takes 10-30 seconds. Subsequent calls are fast because the model stays loaded in memory for 5 minutes by default.
Running this warm-up now means your pipeline won't time out waiting for the first real error to be analyzed.
- You should see a text response about Kubernetes pods after 10-30 seconds of loading. The model may output a thinking section wrapped in <think> tags followed by its answer.
- Type /bye and press Enter to exit the interactive session.
Model download stuck or slow?
The download is 1.4GB. On a slow connection this can take several minutes. If it stalls, cancel with Ctrl+C and retry the pull command.
If you see a memory error during the warm-up query, close other memory-intensive applications. The model needs about 2GB of free RAM to load.
help me troubleshoot Ollama model download or loading issues
Verify the Ollama API is serving
Your FastAPI service will communicate with Ollama over HTTP. You need to confirm the API endpoint is reachable and responds to chat requests.
- Send a test request to the /api/chat endpoint by running:
curl http://localhost:11434/api/chat -d '{
"model": "qwen3:1.7b",
"messages": [{"role": "user", "content": "Say hello in one word."}],
"stream": false
}'What does this command do?
This sends a POST request to Ollama's chat API at http://localhost:11434/api/chat. The stream: false parameter tells Ollama to return a single complete JSON response instead of streaming tokens one by one.
This is the same format your FastAPI middleware will use when calling Ollama to analyze error logs.
- You should see a JSON response containing a message field with the model's reply. The response should arrive in under 2 seconds since the model is already loaded from the warm-up query.
Connection refused?
If you see Connection refused, the Ollama service may not be running. Start it manually with:
sudo systemctl start ollamaThen retry the curl command. If the service fails to start, check the logs with sudo journalctl -u ollama for details.
help me troubleshoot Ollama API connectivity issues
Ollama is installed, your model is loaded, and the API is confirmed serving. Next up, you'll set up the rest of the project environment including Docker, Redis, and your Python virtual environment.
Set Up the Project Environment
Ollama is installed and your AI model is ready to analyze logs. Now you need the infrastructure around it.
Your autonomous SRE system depends on several services working together. Kubernetes provides the cluster that generates logs. Redis caches repeated diagnoses. GitHub's API receives the incident reports your system creates automatically.
In this step, get ready to:
- Verify Docker Desktop and Kubernetes are running.
- Create a Python virtual environment and start a Redis container.
- Generate a GitHub Personal Access Token for automated issue creation.
Verify Docker Desktop and Kubernetes are running
Docker Desktop includes a single-node Kubernetes cluster you can enable in its settings. Your SRE pipeline runs entirely on this local cluster.
- Open Docker Desktop. If it is not running, press Cmd+Space (macOS) or the Windows key (Windows/Linux) to open your search bar, type Docker Desktop, and press Enter.
- Click the gear icon to open Settings.
- Select Kubernetes in the left sidebar.
- Check the box next to Enable Kubernetes if it is not already enabled, then click Apply & Restart.
Stop Rancher Desktop if it is running
Docker Desktop and Rancher Desktop cannot run simultaneously. If you have Rancher Desktop open, quit it completely before proceeding. Both tools manage the same Kubernetes context and will conflict.
- Verify your kubectl context points to the Docker Desktop cluster by running this command:
kubectl cluster-infoYou should see output showing the Kubernetes control plane running at https://127.0.0.1:6443 (or a similar local address).
Seeing 'connection refused' or wrong context?
If kubectl cannot connect, Docker Desktop's Kubernetes may still be starting up. Wait 1-2 minutes after enabling it, then try again.
If the output mentions a different cluster (like minikube or Rancher), switch your context by running kubectl config use-context docker-desktop and then re-run kubectl cluster-info.
help me fix my kubectl context so it points to Docker Desktop
Create the Python environment and start Redis
Your FastAPI service needs a dedicated Python environment to keep its dependencies isolated. Redis provides the caching layer that prevents repeated LLM calls for the same error.
- Create the project directory and move into it by running these commands:
mkdir ~/autonomous-sre
cd ~/autonomous-sre- Create a Python virtual environment by running:
python3 -m venv venv- Activate the virtual environment by running:
source venv/bin/activateYour terminal prompt should now show (venv) at the beginning, confirming the virtual environment is active.
- Install the project dependencies by running:
pip install fastapi uvicorn redis httpx ollamaWhat are these dependencies?
- fastapi is the web framework that receives log events from Fluent Bit.
- uvicorn is the ASGI server that runs your FastAPI application.
- redis is the Python client for communicating with your Redis cache.
- httpx is an HTTP client for calling the GitHub API.
- ollama is the official Python library for communicating with your local Ollama instance.
Now start Redis. First, check if port 6379 is already in use.
- Check for port conflicts by running:
lsof -i :6379If you see output listing a process, another service is using port 6379. Stop it before continuing.
- Start the Redis container by running:
docker run -d --rm --name sre-redis -p 6379:6379 redisWhat does this command do?
- -d runs the container in the background.
- --rm automatically removes the container when it stops, leaving zero data behind.
- --name sre-redis gives the container a memorable name so you can stop it easily later.
- -p 6379:6379 maps the container's Redis port to your host so your Python code can connect at localhost:6379.
- Verify Redis is running by running:
docker exec sre-redis redis-cli pingYou should see PONG in the output. This confirms Redis is accepting connections.
Seeing 'port is already allocated'?
If Docker reports the port is in use, find the conflicting process with lsof -i :6379 and stop it. A common cause is a previously running Redis instance from another project.
help me resolve a port 6379 conflict when starting my Redis container
Create a GitHub Personal Access Token
Your SRE system automatically opens GitHub Issues when it diagnoses an error. To do this programmatically, it needs a Personal Access Token with permission to create issues in a specific repository.
- First, create a test repository for the incident reports. Go to github.com/new in your browser.
- Enter sre-incident-test as the repository name.
- Leave it as Public (or Private, either works).
- Click Create repository.
Now create the fine-grained token scoped to this repository.
- Navigate to github.com/settings/personal-access-tokens/new in your browser.
- Enter a descriptive name like autonomous-sre-token in the Token name field.
- Set an expiration (30 days is fine for a project token).
- Under Repository access, select Only select repositories and choose your sre-incident-test repository.
- Expand the Permissions section, then expand Repository permissions.
- Find Issues and set it to Read and write.
- Click Generate token.
- Copy the token immediately and save it somewhere safe (a password manager or a local text file). GitHub only shows it once.
Why a fine-grained token?
Fine-grained tokens let you scope access to a single repository with only the permissions your application needs. Classic tokens grant broad access to all your repositories, which is riskier for a project like this.
The Issues read and write permission is all your SRE system needs. It can create new issues and search existing ones to avoid duplicates.
Don't see the fine-grained option?
Make sure you are on the correct page. The URL should be github.com/settings/personal-access-tokens/new. If you ended up on the classic token page, look for the Fine-grained tokens tab or navigate directly using the URL.
help me find where to create a fine-grained GitHub Personal Access Token
Your environment is fully set up. You have Kubernetes running, Python dependencies installed, Redis caching ready, and GitHub access configured. Next up, you will build the FastAPI service that receives and processes log events.
Build the FastAPI Log Receiver
Your environment is ready with Python, Redis, and Ollama all running. Now you need the middleware that sits between your log collector and the AI layer.
In this step, you will build a FastAPI service that receives log events from Fluent Bit, filters out noise, and keeps only the errors worth investigating.
In this step, get ready to:
- Create a FastAPI application with a health check endpoint.
- Define Pydantic models that validate incoming Fluent Bit log payloads.
- Build a POST /logs endpoint that filters for ERROR and CRITICAL severity entries.
Create the FastAPI application and health endpoint
The FastAPI service is the entry point for all log events flowing from your Kubernetes cluster. It needs to bind to 0.0.0.0 so pods can reach it via host.docker.internal.
- Make sure your virtual environment is active. If it is not, activate it and navigate to your project directory by running:
cd ~/autonomous-sre
source venv/bin/activate- Create the file main.py inside your ~/autonomous-sre/ folder by running:
touch main.py- Confirm the file exists by running:
lsYou should see main.py listed alongside your venv folder.
- Open main.py in your editor and add the following code to set up the application with a health check endpoint:
from fastapi import FastAPI
import uvicorn
# Initialize the FastAPI application
app = FastAPI()
# Health check endpoint for connectivity testing
@app.get("/health")
def health_check():
return {"status": "ok"}
if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=8000)What does this code do?
app = FastAPI() creates the web application that will receive log events.
The /health endpoint returns a simple JSON response. You will use it later to verify that Kubernetes pods can reach this service.
The uvicorn.run() call at the bottom binds the server to 0.0.0.0:8000, making it accessible from Docker containers and Kubernetes pods on your machine.
- Save main.py.
- Start the server by running:
python main.py- Open a second terminal tab, activate your virtual environment, and test the health endpoint by running:
cd ~/autonomous-sre
source venv/bin/activate
curl http://localhost:8000/healthYou should see {"status":"ok"} printed in your terminal.
Not seeing a response?
- Check that the first terminal shows the server running on http://0.0.0.0:8000. If you see a port conflict error, another process is already using port 8000.
- Make sure you activated the virtual environment (source venv/bin/activate) before running python main.py.
- help me debug why my FastAPI server is not starting
- Stop the server in the first terminal with Ctrl+C. You will restart it after finishing the full file.
Add Pydantic models for request validation
Fluent Bit sends logs as a JSON array. Each record contains a timestamp, the log message, and optional Kubernetes metadata. Pydantic models let FastAPI validate this structure automatically and reject malformed payloads before they reach your processing logic.
- In main.py, find the line from fastapi import FastAPI and add the following imports directly below it:
from pydantic import BaseModel
from typing import Optional- Now add the model definitions between the app = FastAPI() line and the @app.get("/health") decorator:
# Pydantic model for a single Fluent Bit log record
class LogRecord(BaseModel):
date: float
log: str
kubernetes: Optional[dict] = None
# Fluent Bit sends logs as a batched JSON array
FluentBitPayload = list[LogRecord]What does this code do?
LogRecord defines the shape of a single log entry from Fluent Bit. The date field is a Unix timestamp. The log field holds the actual log message text. The kubernetes field is optional metadata that Fluent Bit enriches after collecting the log.
FluentBitPayload is a type alias for a list of LogRecord objects. When you use this as a function parameter type hint, FastAPI expects the request body to be a JSON array and validates every item against LogRecord.
You will not see a change yet. The next chunk uses these models in the log receiver endpoint.
Implement the POST /logs endpoint with severity filtering
Not every log matters. Your SRE system only cares about errors. This endpoint accepts the full batch from Fluent Bit but only processes entries containing ERROR or CRITICAL in the log message.
- In main.py, add the following endpoint between the health_check function and the if __name__ == "__main__": block:
# Receive log events from Fluent Bit and filter for errors
@app.post("/logs")
def receive_logs(payload: FluentBitPayload):
# Filter for ERROR and CRITICAL severity entries only
error_logs = []
for record in payload:
log_content = record.log.upper()
if "ERROR" in log_content or "CRITICAL" in log_content:
error_logs.append(record)
print(f"Received {len(payload)} logs, {len(error_logs)} are errors")
return {"status": "ok", "processed": len(error_logs)}What does this code do?
The payload: FluentBitPayload parameter tells FastAPI to expect a JSON array and validate each item as a LogRecord. If the body is malformed, FastAPI returns a 422 error automatically.
The loop checks each record's log field (case-insensitively) for ERROR or CRITICAL keywords. Only matching entries get collected for further processing.
The print() statement gives you visibility into how many logs arrive versus how many are actual errors. You will see this output in the terminal when Fluent Bit starts sending data.
- Save main.py.
- Start the server again by running:
python main.py- In your second terminal tab, test the endpoint with a sample payload by running:
curl -X POST http://localhost:8000/logs \
-H "Content-Type: application/json" \
-d '[{"date": 1700000000.0, "log": "ERROR: connection refused to database"}, {"date": 1700000001.0, "log": "INFO: health check passed"}, {"date": 1700000002.0, "log": "CRITICAL: out of memory"}]'You should see {"status":"ok","processed":2} returned in your second terminal.
In the server terminal, the output reads Received 3 logs, 2 are errors. Only the ERROR and CRITICAL entries were counted.
Seeing a 422 validation error?
- A 422 means the JSON body does not match the Pydantic model. Check that each object in your array has a date field (a number) and a log field (a string).
- Make sure the body is a JSON array ([...]), not a single object ({...}).
- help me fix a 422 validation error on my /logs endpoint
- Stop the server with Ctrl+C.
✔️ Awesome, I've got everything!
Great. Double check you have saved main.py before moving on.
ⓧ I'd like to double check the full code
Here is the complete main.py at this point:
from fastapi import FastAPI
from pydantic import BaseModel
from typing import Optional
import uvicorn
# Initialize the FastAPI application
app = FastAPI()
# Pydantic model for a single Fluent Bit log record
class LogRecord(BaseModel):
date: float
log: str
kubernetes: Optional[dict] = None
# Fluent Bit sends logs as a batched JSON array
FluentBitPayload = list[LogRecord]
# Health check endpoint for connectivity testing
@app.get("/health")
def health_check():
return {"status": "ok"}
# Receive log events from Fluent Bit and filter for errors
@app.post("/logs")
def receive_logs(payload: FluentBitPayload):
# Filter for ERROR and CRITICAL severity entries only
error_logs = []
for record in payload:
log_content = record.log.upper()
if "ERROR" in log_content or "CRITICAL" in log_content:
error_logs.append(record)
print(f"Received {len(payload)} logs, {len(error_logs)} are errors")
return {"status": "ok", "processed": len(error_logs)}
if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=8000)Your log receiver is ready to accept events and sort signal from noise. Next up, you will connect it to Ollama so errors get diagnosed automatically by AI.
Add the LLM Intelligence Layer
Your FastAPI service can receive and filter error logs. Now it needs a brain.
In this step, you will connect your log receiver to Ollama's local LLM so every ERROR or CRITICAL log gets an automated diagnosis. You will also build a rate limiter to prevent runaway inference from overwhelming your CPU.
In this step, get ready to:
- Build the Ollama integration using the official Python library.
- Engineer a system prompt that returns structured SRE diagnoses.
- Implement a token-bucket rate limiter that caps LLM calls to 5 per minute.
Build the Ollama integration
The ollama Python library gives you a direct interface to the Ollama API running on your machine. You installed it in Step 2 as part of your virtual environment dependencies.
- Make sure your virtual environment is active. If not, activate it by running:
cd ~/autonomous-sre
source venv/bin/activate- Open main.py in your editor.
- Find the import section at the top of the file. It currently looks like this:
from fastapi import FastAPI
from pydantic import BaseModel
from typing import Optional
import uvicorn- Add three new imports below the existing ones so the import section looks like this:
from fastapi import FastAPI
from pydantic import BaseModel
from typing import Optional
import uvicorn
import ollama
import time
import jsonWhat do these imports do?
- ollama provides the chat() function that talks to your local Ollama server.
- time is used by the rate limiter to track token refill intervals.
- json parses the structured JSON output from the LLM.
Engineer the system prompt and parse the response
The LLM needs precise instructions to return structured data. You will write a system prompt that tells the model to act as an SRE and respond with a JSON object containing exactly four fields.
- In main.py, find the line that says FluentBitPayload = list[LogRecord]. Add the following system prompt constant below it:
# System prompt instructing the LLM to act as an SRE
SRE_SYSTEM_PROMPT = """You are an expert Site Reliability Engineer. Analyze the following Kubernetes error log and respond with a JSON object containing exactly these fields:
- error_category: A short category name (e.g., CrashLoopBackOff, OOMKilled, ImagePullError, ConfigError, NetworkError)
- root_cause: A one-sentence explanation of the most likely root cause
- severity: One of "critical", "high", "medium", or "low"
- kubectl_fix: A single kubectl command that would help diagnose or fix the issue
Respond ONLY with valid JSON. No explanation outside the JSON object."""What does this prompt do?
The prompt constrains the LLM to return only a JSON object with four specific fields. By listing concrete examples for error_category and specifying exact allowed values for severity, you reduce hallucination and make the output parseable by code.
The format='json' parameter you will pass to Ollama further enforces valid JSON output at the token generation level.
- Now add the analyze_with_llm function directly below the SRE_SYSTEM_PROMPT constant:
def analyze_with_llm(log_message: str) -> dict:
"""Send a log message to Ollama for SRE diagnosis."""
# Call Ollama with structured JSON output
response = ollama.chat(
model='qwen3:1.7b',
messages=[
{"role": "system", "content": SRE_SYSTEM_PROMPT},
{"role": "user", "content": f"Analyze this error log:\n{log_message}"}
],
stream=False,
format='json'
)
# Parse the LLM response into structured fields
try:
result = json.loads(response['message']['content'])
return {
"error_category": result.get("error_category", "unknown"),
"root_cause": result.get("root_cause", "Unable to determine"),
"severity": result.get("severity", "medium"),
"kubectl_fix": result.get("kubectl_fix", "kubectl get events")
}
except (json.JSONDecodeError, KeyError):
return {
"error_category": "parse_error",
"root_cause": "Failed to parse LLM response",
"severity": "medium",
"kubectl_fix": "kubectl get events"
}What does this function do?
- ollama.chat() sends the log message to qwen3:1.7b with stream=False so you get a single complete response instead of streamed chunks.
- format='json' tells Ollama to constrain the model output to valid JSON at the token level.
- The response content is accessed via response['message']['content'] which contains the raw text the model generated.
- The try/except block handles cases where the model returns malformed JSON, falling back to safe defaults.
- Save main.py.
Seeing an import error for ollama?
Make sure your virtual environment is active (you should see (venv) in your terminal prompt). If not, run source venv/bin/activate from the ~/autonomous-sre directory.
Help me debug this
Implement the token-bucket rate limiter
The qwen3:1.7b model runs on your CPU. Each inference call takes a few seconds. If a burst of errors hits your service simultaneously, queuing unlimited LLM calls would exhaust your machine's resources.
A token-bucket rate limiter solves this by capping LLM calls to 5 per minute. Each call consumes one token. Tokens refill gradually over time.
- In main.py, find the line FluentBitPayload = list[LogRecord]. Add the following class between that line and your SRE_SYSTEM_PROMPT constant:
# Token-bucket rate limiter to protect LLM from burst traffic
class TokenBucketRateLimiter:
def __init__(self, max_tokens: int = 5, refill_rate: float = 5/60):
self.max_tokens = max_tokens
self.tokens = max_tokens
self.refill_rate = refill_rate # tokens per second
self.last_refill = time.time()
def _refill(self):
now = time.time()
elapsed = now - self.last_refill
self.tokens = min(self.max_tokens, self.tokens + elapsed * self.refill_rate)
self.last_refill = now
def consume(self) -> bool:
self._refill()
if self.tokens >= 1:
self.tokens -= 1
return True
return False
# Create a global rate limiter instance
rate_limiter = TokenBucketRateLimiter()How does the token bucket work?
The bucket starts with 5 tokens. Each LLM call consumes 1 token. Tokens refill at a rate of 5 per 60 seconds (5/60 tokens per second).
If a burst of 5 errors arrives at once, the first 5 get analyzed immediately. After that, the limiter returns a fallback response until enough time passes for tokens to refill. This protects your CPU from being overwhelmed.
Why a simple token bucket instead of a library?
For a single-process service like this, an in-memory token bucket is simpler and more transparent than adding another dependency. You can see exactly how the rate limiting works in about 20 lines of code.
- Now integrate the rate limiter into your analyze_with_llm function. Find the line that says:
# Call Ollama with structured JSON output
response = ollama.chat(- Add this rate limiter check directly above that comment so the top of your function body looks like this:
def analyze_with_llm(log_message: str) -> dict:
"""Send a log message to Ollama for SRE diagnosis."""
# Check rate limiter before calling LLM
if not rate_limiter.consume():
return {
"error_category": "rate_limited",
"root_cause": "LLM rate limit exceeded, try again later",
"severity": "low",
"kubectl_fix": "none"
}
# Call Ollama with structured JSON output
response = ollama.chat(What happens when the rate limit is hit?
Instead of blocking or throwing an error, the function returns a safe fallback dictionary. The caller still gets a valid response shape, so your API never crashes. The rate_limited category makes it obvious in downstream systems that the diagnosis was skipped.
- Finally, update your POST /logs endpoint to call analyze_with_llm() for each filtered error. Find your existing receive_logs function and replace it with this:
@app.post("/logs")
def receive_logs(payload: FluentBitPayload):
results = []
for record in payload:
# Check if log contains ERROR or CRITICAL severity
log_upper = record.log.upper()
if "ERROR" in log_upper or "CRITICAL" in log_upper:
# Analyze the error with the LLM
diagnosis = analyze_with_llm(record.log)
results.append({
"log": record.log,
"diagnosis": diagnosis
})
return {"processed": len(results), "results": results}What changed in the endpoint?
Previously, the endpoint only filtered and acknowledged error logs. Now each filtered log gets passed to analyze_with_llm() for AI diagnosis. The response includes both the original log and the structured diagnosis so you can verify the pipeline end to end.
- Save main.py.
- Test the updated service by running:
uvicorn main:app --host 0.0.0.0 --port 8000- In a second terminal (with your venv active), send a test error log by running:
curl -X POST http://localhost:8000/logs \
-H "Content-Type: application/json" \
-d '[{"date": 1234567890, "log": "ERROR: CrashLoopBackOff for pod nginx-deployment-abc123"}]'You should see a JSON response containing a diagnosis object with error_category, root_cause, severity, and kubectl_fix fields filled in by the LLM.
The first call may take a few seconds while the model processes the request.
Getting a connection refused error from Ollama?
Make sure Ollama is still running. Check with ollama -v. If it is not running, start it with ollama serve in a separate terminal.
If the first request times out, the model might have been unloaded from memory (it unloads after 5 minutes of inactivity). Send the request again and the second call will be faster.
Help me debug this connection error
✔️ Awesome, I've got everything!
Great. Double check you have saved main.py and that your test curl returns a diagnosis.
ⓧ I'd like to double check the full code
Here is the complete main.py at this point:
from fastapi import FastAPI
from pydantic import BaseModel
from typing import Optional
import uvicorn
import ollama
import time
import json
app = FastAPI()
# Pydantic models for Fluent Bit payload
class LogRecord(BaseModel):
date: float
log: str
kubernetes: Optional[dict] = None
FluentBitPayload = list[LogRecord]
# Token-bucket rate limiter to protect LLM from burst traffic
class TokenBucketRateLimiter:
def __init__(self, max_tokens: int = 5, refill_rate: float = 5/60):
self.max_tokens = max_tokens
self.tokens = max_tokens
self.refill_rate = refill_rate # tokens per second
self.last_refill = time.time()
def _refill(self):
now = time.time()
elapsed = now - self.last_refill
self.tokens = min(self.max_tokens, self.tokens + elapsed * self.refill_rate)
self.last_refill = now
def consume(self) -> bool:
self._refill()
if self.tokens >= 1:
self.tokens -= 1
return True
return False
# Create a global rate limiter instance
rate_limiter = TokenBucketRateLimiter()
# System prompt instructing the LLM to act as an SRE
SRE_SYSTEM_PROMPT = """You are an expert Site Reliability Engineer. Analyze the following Kubernetes error log and respond with a JSON object containing exactly these fields:
- error_category: A short category name (e.g., CrashLoopBackOff, OOMKilled, ImagePullError, ConfigError, NetworkError)
- root_cause: A one-sentence explanation of the most likely root cause
- severity: One of "critical", "high", "medium", or "low"
- kubectl_fix: A single kubectl command that would help diagnose or fix the issue
Respond ONLY with valid JSON. No explanation outside the JSON object."""
def analyze_with_llm(log_message: str) -> dict:
"""Send a log message to Ollama for SRE diagnosis."""
# Check rate limiter before calling LLM
if not rate_limiter.consume():
return {
"error_category": "rate_limited",
"root_cause": "LLM rate limit exceeded, try again later",
"severity": "low",
"kubectl_fix": "none"
}
# Call Ollama with structured JSON output
response = ollama.chat(
model='qwen3:1.7b',
messages=[
{"role": "system", "content": SRE_SYSTEM_PROMPT},
{"role": "user", "content": f"Analyze this error log:\n{log_message}"}
],
stream=False,
format='json'
)
# Parse the LLM response into structured fields
try:
result = json.loads(response['message']['content'])
return {
"error_category": result.get("error_category", "unknown"),
"root_cause": result.get("root_cause", "Unable to determine"),
"severity": result.get("severity", "medium"),
"kubectl_fix": result.get("kubectl_fix", "kubectl get events")
}
except (json.JSONDecodeError, KeyError):
return {
"error_category": "parse_error",
"root_cause": "Failed to parse LLM response",
"severity": "medium",
"kubectl_fix": "kubectl get events"
}
@app.get("/health")
def health_check():
return {"status": "ok"}
@app.post("/logs")
def receive_logs(payload: FluentBitPayload):
results = []
for record in payload:
# Check if log contains ERROR or CRITICAL severity
log_upper = record.log.upper()
if "ERROR" in log_upper or "CRITICAL" in log_upper:
# Analyze the error with the LLM
diagnosis = analyze_with_llm(record.log)
results.append({
"log": record.log,
"diagnosis": diagnosis
})
return {"processed": len(results), "results": results}
if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=8000)Your service now has an AI brain that diagnoses errors in real time. Next up, you will add Redis caching so repeated identical errors get instant responses without waiting for LLM inference.
Wire Up Redis Caching
Your LLM analysis layer is working, but every single error log triggers a full inference call to Ollama.
In production, the same error often fires repeatedly. Without caching, your system wastes seconds re-diagnosing an identical problem it already solved. In this step, you will add a Redis-backed caching layer that stores LLM diagnoses and serves them instantly on repeat encounters.
In this step, get ready to:
- Create error signature hashing to fingerprint each log message.
- Store and retrieve LLM diagnoses in Redis with a 10-minute TTL.
- Update the log endpoint to check cache before calling the LLM.
Implement error signature hashing
To use Redis as a lookup table, you need a consistent key for each unique error. Hashing the log message content with MD5 gives you a fixed-length fingerprint that maps one-to-one to the original error text.
- In main.py, add the following imports and Redis client initialization at the top of the file, below your existing imports:
import redis
import hashlib
import json
# Connect to local Redis instance
redis_client = redis.Redis(host='localhost', port=6379, decode_responses=True)- Below the Redis client initialization, add the hashing function:
def get_error_signature(log_message: str) -> str:
"""Create a unique hash from the log message for cache key lookup."""
return hashlib.md5(log_message.encode()).hexdigest()What does this code do?
The Redis client connects to the container you started in Step 2. The decode_responses=True flag tells redis-py to return strings instead of raw bytes, so you can work with the data directly.
The get_error_signature() function takes a log message, encodes it to bytes, and runs it through MD5 to produce a consistent 32-character hex string. Two identical log messages always produce the same hash, making it a perfect cache key.
- Save main.py.
- Confirm Redis is still reachable by running this in your terminal:
docker exec sre-redis redis-cli pingYou should see PONG in the output, confirming the Redis container is alive and accepting connections.
Seeing 'Error: No such container'?
Your Redis container may have stopped. Because you started it with the --rm flag, it auto-deleted when it stopped. Restart it with the same command from Step 2.
help me restart my Redis container for the SRE project
Store LLM diagnosis results in Redis with a 10-minute TTL
Now you need two helper functions: one to look up a cached diagnosis, and one to store a new diagnosis after the LLM produces it. The 10-minute TTL ensures stale diagnoses expire automatically.
- In main.py, add these two functions below get_error_signature():
def get_cached_diagnosis(signature: str) -> dict | None:
"""Check Redis for a cached diagnosis matching this error signature."""
cached = redis_client.get(f"diagnosis:{signature}")
if cached:
return json.loads(cached)
return None
def cache_diagnosis(signature: str, diagnosis: dict):
"""Store a diagnosis in Redis with a 10-minute TTL."""
redis_client.set(f"diagnosis:{signature}", json.dumps(diagnosis), ex=600)What does this code do?
The key format diagnosis:{signature} namespaces your cache entries so they do not collide with other Redis data. The get_cached_diagnosis() function returns the parsed dict if a match exists, or None if the key has expired or never existed.
The ex=600 parameter sets a 600-second (10-minute) expiry. After 10 minutes, Redis deletes the key automatically. This means the system re-diagnoses errors periodically in case conditions have changed.
- Save main.py.
Seeing a syntax error on the type hint?
The dict | None union syntax requires Python 3.10 or higher. If you are running an older version, replace it with Optional[dict] and add from typing import Optional at the top of your file.
help me fix my type hint syntax error
Add cache-hit logging to the log endpoint
The cache functions are ready, but the POST /logs endpoint still calls the LLM on every request. You need to wire in the cache check so the endpoint skips the LLM entirely when it finds a cached result.
- In main.py, find the section inside your POST /logs endpoint where you call analyze_with_llm() for each filtered ERROR/CRITICAL log entry.
- Replace the direct LLM call with the following cache-check-before-LLM logic:
# Compute a fingerprint for this error message
signature = get_error_signature(log_message)
# Check cache first to avoid redundant LLM calls
cached = get_cached_diagnosis(signature)
if cached:
print(f"[CACHE HIT] Skipping LLM for signature: {signature[:8]}...")
results.append(cached)
continue
# Cache miss - call the LLM for a fresh diagnosis
diagnosis = analyze_with_llm(log_message)
# Store the result for future lookups
cache_diagnosis(signature, diagnosis)
results.append(diagnosis)What does this code do?
For each error log entry, the endpoint now computes a hash, checks Redis, and only calls the LLM on a cache miss. The print() statement logs cache hits to your terminal so you can visually confirm the caching is working during testing.
After a successful LLM call, the diagnosis is stored in Redis. The next time the same error arrives within 10 minutes, the response comes from Redis in milliseconds instead of waiting seconds for LLM inference.
- Save main.py.
- Make sure your virtual environment is activated and start the FastAPI service by running:
cd ~/autonomous-sre && source venv/bin/activate && uvicorn main:app --host 0.0.0.0 --port 8000- In a separate terminal, send the same test log twice to see the cache in action:
curl -X POST http://localhost:8000/logs -H "Content-Type: application/json" -d '[{"date": "2026-05-13T10:00:00Z", "log": "{\"level\": \"ERROR\", \"message\": \"CrashLoopBackOff for pod nginx-abc123\"}"}]'- Run the same curl command again.
On the second request, you should see [CACHE HIT] Skipping LLM for signature: in your FastAPI terminal output. The response also returns much faster because it skips the LLM entirely.
Not seeing a cache hit on the second request?
Make sure your Redis container is still running. Run docker ps and look for sre-redis in the list. If it is missing, restart it.
Also verify the second curl request has the exact same JSON body. Even a single character difference produces a different hash, causing a cache miss.
help me debug why my Redis cache is not being hit
✔️ Awesome, I've got everything!
Great. Double check you have saved your file before moving on.
ⓧ I'd like to double check the full code
Here is the complete main.py as it should look at the end of this step:
from fastapi import FastAPI
from pydantic import BaseModel
import uvicorn
import ollama
import time
import redis
import hashlib
import json
app = FastAPI()
# Connect to local Redis instance
redis_client = redis.Redis(host='localhost', port=6379, decode_responses=True)
# Token bucket rate limiter to protect LLM from burst traffic
class TokenBucketRateLimiter:
def __init__(self, max_tokens: int = 5, refill_rate: float = 5/60):
self.max_tokens = max_tokens
self.refill_rate = refill_rate # tokens per second
self.tokens = max_tokens
self.last_refill = time.time()
def consume(self) -> bool:
"""Try to consume a token. Returns True if allowed, False if rate limited."""
now = time.time()
elapsed = now - self.last_refill
self.tokens = min(self.max_tokens, self.tokens + elapsed * self.refill_rate)
self.last_refill = now
if self.tokens >= 1:
self.tokens -= 1
return True
return False
rate_limiter = TokenBucketRateLimiter()
# System prompt for SRE diagnosis
SRE_SYSTEM_PROMPT = """You are an expert Site Reliability Engineer. Analyze the following Kubernetes error log and respond in JSON format with these fields:
- error_category: a short category name (e.g., CrashLoopBackOff, OOMKilled, ImagePullBackOff)
- root_cause: a one-sentence explanation of the likely root cause
- severity: one of "low", "medium", "high", "critical"
- kubectl_fix: a single kubectl command that would help diagnose or fix the issue
Respond ONLY with valid JSON, no other text."""
def get_error_signature(log_message: str) -> str:
"""Create a unique hash from the log message for cache key lookup."""
return hashlib.md5(log_message.encode()).hexdigest()
def get_cached_diagnosis(signature: str) -> dict | None:
"""Check Redis for a cached diagnosis matching this error signature."""
cached = redis_client.get(f"diagnosis:{signature}")
if cached:
return json.loads(cached)
return None
def cache_diagnosis(signature: str, diagnosis: dict):
"""Store a diagnosis in Redis with a 10-minute TTL."""
redis_client.set(f"diagnosis:{signature}", json.dumps(diagnosis), ex=600)
def analyze_with_llm(log_message: str) -> dict:
"""Send a log message to Ollama for AI-powered diagnosis."""
if not rate_limiter.consume():
return {
"error_category": "rate_limited",
"root_cause": "Too many LLM requests. Try again later.",
"severity": "low",
"kubectl_fix": "N/A"
}
response = ollama.chat(
model='qwen3:1.7b',
messages=[
{"role": "system", "content": SRE_SYSTEM_PROMPT},
{"role": "user", "content": f"Analyze this error log:\n{log_message}"}
],
stream=False,
format='json'
)
try:
result = json.loads(response['message']['content'])
return {
"error_category": result.get("error_category", "unknown"),
"root_cause": result.get("root_cause", "Unable to determine"),
"severity": result.get("severity", "medium"),
"kubectl_fix": result.get("kubectl_fix", "kubectl describe pod <pod-name>")
}
except (json.JSONDecodeError, KeyError):
return {
"error_category": "parse_error",
"root_cause": "LLM response could not be parsed",
"severity": "medium",
"kubectl_fix": "kubectl logs <pod-name>"
}
@app.post("/logs")
async def receive_logs(logs: list):
"""Receive log events from Fluent Bit and process ERROR/CRITICAL entries."""
results = []
for entry in logs:
# Extract the log content from the Fluent Bit payload
log_data = entry.get("log", "")
# Parse JSON log content if it's a string
if isinstance(log_data, str):
try:
log_data = json.loads(log_data)
except json.JSONDecodeError:
continue
# Filter for ERROR and CRITICAL severity only
level = log_data.get("level", "").upper()
if level not in ("ERROR", "CRITICAL"):
continue
log_message = log_data.get("message", "")
if not log_message:
continue
# Compute a fingerprint for this error message
signature = get_error_signature(log_message)
# Check cache first to avoid redundant LLM calls
cached = get_cached_diagnosis(signature)
if cached:
print(f"[CACHE HIT] Skipping LLM for signature: {signature[:8]}...")
results.append(cached)
continue
# Cache miss - call the LLM for a fresh diagnosis
diagnosis = analyze_with_llm(log_message)
# Store the result for future lookups
cache_diagnosis(signature, diagnosis)
results.append(diagnosis)
return {"processed": len(results), "results": results}
@app.get("/health")
async def health_check():
"""Health check endpoint for connectivity testing."""
return {"status": "healthy"}
if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=8000)Your caching layer is live. Repeated errors now get instant responses from Redis, freeing up Ollama for genuinely new problems. Next up, you will automate the creation of GitHub Issues so every diagnosed error gets a structured incident ticket without human intervention.
Automate GitHub Issue Creation
Your caching layer ensures repeated errors get instant responses without burning LLM compute. Now it's time to close the loop by turning diagnoses into actionable incident reports.
In this step, you'll build the automated incident response piece. When the LLM produces a diagnosis, your system will create a GitHub Issue containing the root cause, severity, and a suggested kubectl fix. It will also check for duplicates so your repository doesn't fill up with repeated tickets for the same error.
In this step, get ready to:
- Build an httpx integration that creates GitHub Issues via the REST API.
- Structure issue bodies with diagnosis details and remediation commands.
- Add deduplication logic that prevents duplicate issues for the same error.
Set up GitHub API configuration
Your FastAPI service needs to authenticate with the GitHub API using the Personal Access Token you created in Step 2. Reading credentials from environment variables keeps them out of your source code.
- In main.py, find the imports at the top of the file. Add os and httpx to your existing imports, plus the GitHub configuration lines below them:
import os
import httpx
# GitHub configuration from environment variables
GITHUB_TOKEN = os.environ["GITHUB_TOKEN"]
GITHUB_REPO = os.environ["GITHUB_REPO"]What does this code do?
os.environ["GITHUB_TOKEN"] reads your Personal Access Token from the shell environment. Using square-bracket access means the app will crash immediately if the variable is missing, which is better than a silent failure later.
GITHUB_REPO expects the format owner/repo (e.g. yourname/sre-incident-test). The code splits this string to construct API URLs.
- Save main.py.
Keep your token out of code
You'll export these environment variables in the terminal before starting the FastAPI server. Never hardcode tokens directly in your Python files.
Build the issue creation function
The GitHub REST API accepts a POST request with a title, body, and labels to create an issue. Your function will format the LLM diagnosis into a structured markdown report that an on-call engineer can act on immediately.
- In main.py, add the following function after your analyze_with_llm() function:
async def create_github_issue(log_message: str, diagnosis: dict, signature: str):
"""Create a GitHub Issue with structured diagnosis report."""
owner, repo = GITHUB_REPO.split("/")
# Structure issue body with diagnosis fields and raw log context
body = f"""## Automated Incident Report
**Error Category:** {diagnosis.get('error_category', 'Unknown')}
**Severity:** {diagnosis.get('severity', 'Unknown')}
**Root Cause:** {diagnosis.get('root_cause', 'Unknown')}
### Proposed Fix
```bash
{diagnosis.get('kubectl_fix', 'No fix suggested')}
```
### Raw Log Context
```
{log_message}
```
---
*Signature: {signature}*
"""
# Create the issue title with category and signature prefix
title = f"[{diagnosis.get('error_category', 'Error')}] Incident {signature[:8]}"
async with httpx.AsyncClient() as client:
response = await client.post(
f"https://api.github.com/repos/{owner}/{repo}/issues",
headers={
"Authorization": f"Bearer {GITHUB_TOKEN}",
"Accept": "application/vnd.github+json"
},
json={
"title": title,
"body": body,
"labels": [diagnosis.get('severity', 'unknown')]
}
)
response.raise_for_status()
print(f"Created GitHub Issue #{response.json()['number']}: {title}")What does this code do?
The function builds a markdown-formatted issue body containing every field the LLM returned: error category, severity, root cause, and the proposed kubectl fix. It also includes the raw log text so the engineer can see the original error.
The title uses the format [ErrorCategory] Incident abc12345 where the last part is the first 8 characters of the error signature hash. This makes issues searchable and helps the deduplication logic you'll add next.
httpx.AsyncClient() sends the POST request asynchronously so the FastAPI event loop isn't blocked while waiting for GitHub's response. The Authorization: Bearer header authenticates with your fine-grained PAT.
- Save main.py.
Seeing a syntax error in the f-string?
Triple-quoted f-strings with curly braces can be tricky. Make sure your opening f""" is on the same line as body = and the closing """ is on its own line. Any literal curly braces in markdown (like in code fences) should not conflict since they don't contain Python expressions.
Help me fix an f-string syntax error in my create_github_issue function
Add issue deduplication
If the same error fires every 30 seconds, you don't want 100 identical issues in your repository. Deduplication checks whether an open issue already exists for this error signature before creating a new one.
- In main.py, add the following function above create_github_issue():
async def find_existing_issue(signature: str) -> int | None:
"""Search open issues for an existing report with this error signature."""
owner, repo = GITHUB_REPO.split("/")
async with httpx.AsyncClient() as client:
response = await client.get(
f"https://api.github.com/repos/{owner}/{repo}/issues",
params={"state": "open"},
headers={
"Authorization": f"Bearer {GITHUB_TOKEN}",
"Accept": "application/vnd.github+json"
}
)
response.raise_for_status()
# Search issue titles for the error signature
for issue in response.json():
if signature[:8] in issue["title"]:
return issue["number"]
return NoneWhat does this code do?
This function calls the GitHub API's list issues endpoint with state=open to fetch only unresolved issues. It then searches each issue's title for the first 8 characters of the error signature.
If a matching title is found, the function returns that issue's number (so the caller knows a duplicate exists). If no match is found, it returns None, signaling that a new issue should be created.
Now you need to wire both functions into your existing POST /logs endpoint. The logic is: after getting a diagnosis, check for duplicates first, then create an issue only if none exists.
- In your receive_logs function, find the section where you call cache_diagnosis(signature, diagnosis). After the cache logic, add the following code before the final return statement:
# Skip GitHub integration if LLM failed
if "error" in diagnosis:
continue
# Check for existing open issue before creating a new one
existing = await find_existing_issue(signature)
if existing:
print(f"Issue already exists: #{existing}, skipping creation")
continue
# Create a new GitHub Issue
await create_github_issue(log_message, diagnosis, signature)What's happening here?
First, you skip the GitHub integration entirely if the LLM returned an error (rate-limited or failed to parse). No point creating an issue with broken data.
Then find_existing_issue() checks GitHub for open issues with this signature. If one exists, you log it and skip. Otherwise, create_github_issue() fires and creates a new structured incident report.
- Save main.py.
Unsure where to place the new code?
The new lines go inside your for record in records: loop, after the block that handles cache misses and stores the result. They should be at the same indentation level as the signature = get_error_signature(log_message) line above them.
Help me figure out where to add the GitHub issue logic in my receive_logs function
✔️ Awesome, I've got everything!
Great. Double check you've saved main.py with all three additions: the environment variable config, the find_existing_issue() function, the create_github_issue() function, and the updated receive_logs endpoint.
ⓧ I'd like to double check the full code
Here is the full main.py at this point in the project. Compare your file against it:
import os
import json
import time
import hashlib
from fastapi import FastAPI
import redis
import httpx
from ollama import chat
app = FastAPI()
# Redis client for caching
redis_client = redis.Redis(host='localhost', port=6379, decode_responses=True)
# GitHub configuration from environment variables
GITHUB_TOKEN = os.environ["GITHUB_TOKEN"]
GITHUB_REPO = os.environ["GITHUB_REPO"]
class TokenBucketRateLimiter:
"""Rate limiter using token bucket algorithm."""
def __init__(self, max_tokens: int, refill_rate: float):
self.max_tokens = max_tokens
self.tokens = max_tokens
self.refill_rate = refill_rate
self.last_refill = time.time()
def allow(self) -> bool:
"""Check if a request is allowed and consume a token."""
now = time.time()
elapsed = now - self.last_refill
self.tokens = min(self.max_tokens, self.tokens + elapsed * self.refill_rate)
self.last_refill = now
if self.tokens >= 1:
self.tokens -= 1
return True
return False
# Allow 5 LLM calls per minute
rate_limiter = TokenBucketRateLimiter(max_tokens=5, refill_rate=5/60)
def get_error_signature(log_message: str) -> str:
"""Generate a hash signature from the log message for deduplication."""
return hashlib.sha256(log_message.encode()).hexdigest()
def get_cached_diagnosis(signature: str) -> dict | None:
"""Check Redis for a cached diagnosis."""
cached = redis_client.get(f"diagnosis:{signature}")
if cached:
return json.loads(cached)
return None
def cache_diagnosis(signature: str, diagnosis: dict):
"""Store diagnosis in Redis with a 10-minute TTL."""
redis_client.set(f"diagnosis:{signature}", json.dumps(diagnosis), ex=600)
def analyze_with_llm(log_message: str) -> dict:
"""Send log message to Ollama for SRE diagnosis."""
if not rate_limiter.allow():
print("Rate limit exceeded, skipping LLM call")
return {"error": "rate_limited"}
system_prompt = """You are an SRE AI assistant. Analyze the following Kubernetes error log and provide a structured JSON diagnosis with these fields:
- error_category: a short category name (e.g., CrashLoopBackOff, OOMKilled, ImagePullBackOff)
- root_cause: a brief explanation of why this error occurred
- severity: one of critical, high, medium, low
- kubectl_fix: a single kubectl command that would help remediate this issue"""
response = chat(
model='qwen3:1.7b',
messages=[
{'role': 'system', 'content': system_prompt},
{'role': 'user', 'content': f"Analyze this error log:\n{log_message}"}
],
stream=False,
format='json'
)
try:
return json.loads(response.message.content)
except json.JSONDecodeError:
return {"error": "Failed to parse LLM response", "raw": response.message.content}
async def find_existing_issue(signature: str) -> int | None:
"""Search open issues for an existing report with this error signature."""
owner, repo = GITHUB_REPO.split("/")
async with httpx.AsyncClient() as client:
response = await client.get(
f"https://api.github.com/repos/{owner}/{repo}/issues",
params={"state": "open"},
headers={
"Authorization": f"Bearer {GITHUB_TOKEN}",
"Accept": "application/vnd.github+json"
}
)
response.raise_for_status()
# Search issue titles for the error signature
for issue in response.json():
if signature[:8] in issue["title"]:
return issue["number"]
return None
async def create_github_issue(log_message: str, diagnosis: dict, signature: str):
"""Create a GitHub Issue with structured diagnosis report."""
owner, repo = GITHUB_REPO.split("/")
# Structure issue body with diagnosis fields and raw log context
body = f"""## Automated Incident Report
**Error Category:** {diagnosis.get('error_category', 'Unknown')}
**Severity:** {diagnosis.get('severity', 'Unknown')}
**Root Cause:** {diagnosis.get('root_cause', 'Unknown')}
### Proposed Fix
```bash
{diagnosis.get('kubectl_fix', 'No fix suggested')}
```
### Raw Log Context
```
{log_message}
```
---
*Signature: {signature}*
"""
# Create the issue title with category and signature prefix
title = f"[{diagnosis.get('error_category', 'Error')}] Incident {signature[:8]}"
async with httpx.AsyncClient() as client:
response = await client.post(
f"https://api.github.com/repos/{owner}/{repo}/issues",
headers={
"Authorization": f"Bearer {GITHUB_TOKEN}",
"Accept": "application/vnd.github+json"
},
json={
"title": title,
"body": body,
"labels": [diagnosis.get('severity', 'unknown')]
}
)
response.raise_for_status()
print(f"Created GitHub Issue #{response.json()['number']}: {title}")
@app.post("/logs")
async def receive_logs(records: list):
"""Receive log events from Fluent Bit and process errors."""
for record in records:
log_message = record.get("log", record.get("message", ""))
level = record.get("level", "").upper()
# Only process ERROR and CRITICAL entries
if level not in ("ERROR", "CRITICAL"):
continue
# Compute error signature for deduplication
signature = get_error_signature(log_message)
# Check cache first
diagnosis = get_cached_diagnosis(signature)
if diagnosis:
print(f"Cache hit for signature {signature[:8]}")
else:
# Call LLM on cache miss
diagnosis = analyze_with_llm(log_message)
if "error" not in diagnosis:
cache_diagnosis(signature, diagnosis)
# Skip GitHub integration if LLM failed
if "error" in diagnosis:
continue
# Check for existing open issue before creating a new one
existing = await find_existing_issue(signature)
if existing:
print(f"Issue already exists: #{existing}, skipping creation")
continue
# Create a new GitHub Issue
await create_github_issue(log_message, diagnosis, signature)
return {"status": "processed"}
@app.get("/health")
async def health():
"""Health check endpoint."""
return {"status": "healthy"}Your incident response automation is complete. When a diagnosis comes through, it checks for duplicates and creates a clean, actionable GitHub Issue. Next up, you'll deploy a deliberately crashy app to your Kubernetes cluster so you have real error logs to feed the pipeline.
Deploy a Crashy App to Kubernetes
Your pipeline can receive logs, diagnose them with AI, cache results, and open GitHub Issues. Now you need something that actually produces errors for the pipeline to consume.
In this step, you will deploy a deliberately broken application to your Kubernetes cluster. This "crashy app" outputs JSON-formatted error logs to stdout on a timer, giving Fluent Bit a steady stream of errors to forward to your FastAPI service.
In this step, get ready to:
- Create a Kubernetes Deployment manifest that runs an Alpine container with a log-emitting loop.
- Deploy the crashy app and verify its pod is running.
- Confirm the log format matches what Fluent Bit expects to parse.
Create the crashy app Deployment manifest
Kubernetes Deployments are defined in YAML manifests. Your crashy app needs a single container that runs an infinite loop, printing a JSON error log every 30 seconds.
- Create a file called crashy-app.yaml inside your ~/autonomous-sre/ folder by running this command:
touch ~/autonomous-sre/crashy-app.yaml- Open crashy-app.yaml in your editor and paste the following manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
name: crashy-app
spec:
replicas: 1
selector:
matchLabels:
app: crashy-app
template:
metadata:
labels:
app: crashy-app
spec:
containers:
- name: crashy
image: alpine
command: ["/bin/sh", "-c"]
args:
- |
while true; do
echo "{\"level\":\"ERROR\",\"message\":\"simulated pod crash: OOMKilled\",\"timestamp\":\"$(date -Iseconds)\"}"
sleep 30
doneWhat does this manifest do?
This Deployment runs a single Alpine container that loops forever. Every 30 seconds it prints a JSON object to stdout with three fields.
- level is set to ERROR so Fluent Bit's grep filter will match it.
- message simulates an OOMKilled crash, a common Kubernetes failure mode.
- timestamp uses ISO 8601 format via date -Iseconds so logs are time-ordered.
- Save crashy-app.yaml.
YAML indentation errors?
YAML is whitespace-sensitive. Make sure you are using spaces (not tabs) and that each indentation level is exactly 2 spaces. The args: block must be indented under command: at the same level.
help me fix YAML indentation in my Kubernetes manifest
Deploy the crashy app to your cluster
- Apply the manifest to your Kubernetes cluster by running this command:
kubectl apply -f ~/autonomous-sre/crashy-app.yaml- Verify the pod is running by checking the pod status:
kubectl get podsYou should see a pod named crashy-app- followed by a random suffix, with a status of Running and 1/1 in the READY column.
Pod stuck in ContainerCreating or ImagePullBackOff?
If the pod shows ContainerCreating, wait 30 seconds and run kubectl get pods again. Kubernetes is pulling the Alpine image.
If you see ImagePullBackOff, check that Docker Desktop is running and that Kubernetes is enabled in Docker Desktop settings under Settings then Kubernetes.
help me debug my crashy-app pod that won't start
Confirm the log format matches what Fluent Bit will parse
Fluent Bit's grep filter needs a level field in each log line to filter by severity. You need to confirm that the crashy app's output contains valid JSON with that field.
- View the crashy app's logs by running this command:
kubectl logs -l app=crashy-appYou should see one or more lines of JSON output that look like this:
{"level":"ERROR","message":"simulated pod crash: OOMKilled","timestamp":"2026-05-13T10:30:00+00:00"}Why does the format matter?
Fluent Bit's kubernetes metadata filter uses Merge_Log On to parse JSON log lines into structured fields. Once parsed, the grep filter can match on the level field to keep only ERROR and CRITICAL entries.
If the log output were plain text instead of JSON, Fluent Bit would not be able to filter by severity. The structured format is what makes selective forwarding possible.
No logs showing yet?
The loop sleeps for 30 seconds between log lines. If the pod just started, wait 30 seconds and run the kubectl logs command again.
✔️ Awesome, I've got everything!
Great. Double check you have saved crashy-app.yaml and that your pod shows Running status.
ⓧ I'd like to double check the full code
Here is the complete crashy-app.yaml manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
name: crashy-app
spec:
replicas: 1
selector:
matchLabels:
app: crashy-app
template:
metadata:
labels:
app: crashy-app
spec:
containers:
- name: crashy
image: alpine
command: ["/bin/sh", "-c"]
args:
- |
while true; do
echo "{\"level\":\"ERROR\",\"message\":\"simulated pod crash: OOMKilled\",\"timestamp\":\"$(date -Iseconds)\"}"
sleep 30
doneYour crashy app is live and steadily producing error logs. Next up, you will verify that pods in your cluster can reach your FastAPI service on the host machine so Fluent Bit can forward logs to it.
Verify Pod-to-Host Connectivity
Your crashy app is generating error logs inside Kubernetes pods, and your FastAPI service is ready to receive them on your host machine.
There is a networking boundary between your Kubernetes pods and your host. Pods live in their own virtual network, so they cannot just call localhost to reach your machine. Before deploying Fluent Bit, you need to confirm that pods can actually reach your FastAPI service.
In this step, get ready to:
- Start the FastAPI service on your host machine.
- Test pod-to-host connectivity from inside a Kubernetes pod.
- Record the working host address for the Fluent Bit configuration.
Start the FastAPI service and test connectivity
Docker Desktop on Linux provides a special DNS name, host.docker.internal, that resolves to your host machine from inside containers. You will test whether pods can use this name to reach your FastAPI service.
- Start the FastAPI service by running these commands in your terminal:
cd ~/autonomous-sre
source venv/bin/activate
export GITHUB_TOKEN=your-github-pat-here
export GITHUB_REPO="[[GITHUB_REPO="your-username/sre-incident-test"]]"
uvicorn main:app --host 0.0.0.0 --port 8000What do these commands do?
- export GITHUB_TOKEN and export GITHUB_REPO set the environment variables your FastAPI app needs to create GitHub Issues.
- --host 0.0.0.0 binds the server to all network interfaces. Without this, uvicorn only listens on 127.0.0.1 and Kubernetes pods would not be able to reach it.
- Replace your-github-pat-here with the Personal Access Token you created earlier. Do not wrap it in quotes.
You should see output like Uvicorn running on http://0.0.0.0:8000 in your terminal. This means the service is running and ready to accept connections.
Seeing 'Address already in use'?
Another process is already using port 8000. Find it with lsof -i :8000 and stop it, or choose a different port.
help me fix a port conflict on port 8000
- Open a second terminal window. The FastAPI server is blocking your first terminal, so you need a fresh one for the next command.
- Test connectivity from inside a Kubernetes pod by running:
kubectl run test-curl --rm -it --image=curlimages/curl --restart=Never -- curl http://host.docker.internal:8000/healthWhat does this command do?
This spins up a temporary pod running the curlimages/curl image, sends a request to your FastAPI /health endpoint via host.docker.internal, then automatically deletes the pod when done.
The --rm flag ensures the pod is cleaned up. The --restart=Never flag tells Kubernetes not to restart the pod after the curl command finishes.
✔️ I see a JSON response from /health
Your pod successfully reached your host machine. The host.docker.internal DNS name is working. Skip ahead to the "Record the working host address" substep below.
ⓧ I see 'Could not resolve host' or connection refused
This is a rare edge case on some Docker Desktop Linux versions where host.docker.internal does not resolve from Kubernetes pods. Continue to the next substep for the fallback approach.
Fall back to the docker0 bridge IP
If host.docker.internal did not resolve, you can use the docker0 bridge IP instead. This is the network interface Docker creates to connect containers to your host.
- Find your docker0 bridge IP by running:
ip addr show docker0Look for the inet line in the output. The IP address before the / is your bridge IP. It is typically 172.17.0.1.
- Test connectivity using the bridge IP by running (replace the IP if yours differs):
kubectl run test-curl --rm -it --image=curlimages/curl --restart=Never -- curl http://[[DOCKER0_IP="your docker0 bridge IP, e.g. 172.17.0.1"]]:8000/healthYou should now see a JSON response from your /health endpoint. This confirms the bridge IP works as the route from pods to your host.
Still seeing connection refused?
- Make sure your FastAPI server is still running in the other terminal. Check for the Uvicorn running on http://0.0.0.0:8000 message.
- Confirm the server is bound to 0.0.0.0 and not 127.0.0.1. A server on localhost only accepts connections from the same machine.
- Check that no firewall rule is blocking port 8000. Try sudo ufw status to see active rules.
help me debug pod-to-host connectivity
Record the working host address
You now have a confirmed route from Kubernetes pods to your FastAPI service. In the next step, you will configure Fluent Bit to send logs to this address.
- Note down whichever address worked for you:
- host.docker.internal if the first test succeeded.
- The docker0 bridge IP (e.g. 172.17.0.1) if you used the fallback.
Keep your FastAPI server running
Leave the uvicorn process running in its terminal. You will need it for the Fluent Bit deployment and the end-to-end test in the upcoming steps.
The networking bridge is confirmed. Next up, you will deploy Fluent Bit with a custom configuration that sends error logs to this address automatically.
Deploy Fluent Bit with Custom Configuration
Your FastAPI service is running and reachable from inside the cluster. Now you need something to actually send logs to it.
Fluent Bit is a lightweight log processor that runs as a DaemonSet on every Kubernetes node. It tails container log files, enriches them with Kubernetes metadata, and forwards matching records to any destination you configure. In this step, you will deploy Fluent Bit via Helm with a custom pipeline that filters for ERROR and CRITICAL logs, then ships them to your FastAPI endpoint over HTTP.
In this step, get ready to:
- Add the Fluent Bit Helm chart repository and create a custom values file.
- Deploy Fluent Bit to your Kubernetes cluster with your custom configuration.
- Verify the DaemonSet is running and forwarding logs to your FastAPI service.
Add the Helm repository and create the values file
Helm charts are pre-packaged Kubernetes applications. The Fluent Bit team maintains an official chart that deploys a DaemonSet with sensible defaults. You will override the default filters and outputs to match your pipeline.
- Add the Fluent Bit Helm chart repository and update your local cache by running these commands:
helm repo add fluent https://fluent.github.io/helm-charts
helm repo updateWhat do these commands do?
The first command registers the Fluent Bit Helm repository under the name fluent. The second command downloads the latest chart index so Helm knows which versions are available.
- Confirm the repository was added by running:
helm repo listYou should see an entry named fluent pointing to https://fluent.github.io/helm-charts in the output.
Now you need a values file that tells the chart how to configure Fluent Bit's pipeline. The chart uses classic .conf format inside its config block, not the newer YAML pipeline format.
- Create the custom values file by running:
cd ~/autonomous-sre- Create a file called fluent-bit-values.yaml inside your autonomous-sre folder and paste the following configuration:
# Custom Fluent Bit Helm values
# Overrides filters and outputs for the SRE pipeline
config:
# Keep the default inputs (tail + systemd) from the chart
# Override filters: kubernetes metadata first, then grep for severity
filters: |
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
[FILTER]
Name grep
Match kube.*
Logical_Op or
Regex log ERROR
Regex log CRITICAL
# Override outputs: send filtered logs to FastAPI via HTTP
outputs: |
[OUTPUT]
Name http
Match *
Host [[HOST_ADDRESS="host.docker.internal or your docker0 bridge IP"]]
Port 8000
URI /logs
Format jsonWhat does this configuration do?
The config.filters block defines two filters that run in order. The kubernetes filter MUST come first because it parses the raw JSON from container logs (via Merge_Log On) and extracts the log field. Without this step, the grep filter would have nothing to match against.
The grep filter uses Logical_Op or with two Regex entries. This keeps any record where the log field contains either ERROR or CRITICAL. All other records are dropped before reaching the output.
The config.outputs block replaces the default Elasticsearch output with an HTTP output. It posts JSON payloads to your FastAPI /logs endpoint on port 8000.
Why classic .conf format inside a YAML file?
The Fluent Bit Helm chart's config section expects classic [FILTER] and [OUTPUT] blocks as multi-line strings. The chart template injects these strings directly into the Fluent Bit ConfigMap. Using the newer YAML pipeline format here would cause a silent misconfiguration.
- Save fluent-bit-values.yaml.
- Confirm the file exists by running:
ls ~/autonomous-sre/fluent-bit-values.yamlYou should see the file path printed in the output.
File not found?
Make sure you saved the file inside ~/autonomous-sre/ and that the filename is exactly fluent-bit-values.yaml. Double-check for typos like fluentbit-values.yaml (missing hyphen).
help me find my fluent-bit-values.yaml file
Deploy Fluent Bit with Helm
With the values file ready, you can deploy Fluent Bit to your cluster. The helm upgrade --install command is idempotent. It creates a new release if one does not exist, or updates an existing one.
- Deploy Fluent Bit by running:
helm upgrade --install fluent-bit fluent/fluent-bit -f fluent-bit-values.yamlWhat does this command do?
This tells Helm to install (or upgrade) a release named fluent-bit using the fluent/fluent-bit chart. The -f fluent-bit-values.yaml flag applies your custom filter and output overrides on top of the chart defaults.
Helm creates a DaemonSet that runs one Fluent Bit pod on every node in your cluster. On a single-node Docker Desktop cluster, that means one pod.
You should see output confirming the release status as deployed.
Seeing a chart not found error?
Make sure you ran helm repo update after adding the repository. Also verify your internet connection is active since Helm needs to download the chart.
help me troubleshoot my Fluent Bit Helm install
Verify the deployment
Fluent Bit needs a few seconds to pull its image and start collecting logs. You should confirm the pods are running and that the HTTP output is attempting to reach your FastAPI service.
- Check that the Fluent Bit pods are running by running:
kubectl get pods -l app.kubernetes.io/name=fluent-bitYou should see one pod with a status of Running. If the status shows ContainerCreating, wait 30 seconds and try again.
- Check the Fluent Bit logs to confirm the pipeline is active by running:
kubectl logs -l app.kubernetes.io/name=fluent-bit --tail=30Look for lines indicating the HTTP output is attempting connections. If your FastAPI service is running on the host, you should see successful POST attempts in the Fluent Bit logs.
What should I see?
Healthy Fluent Bit logs will show the input plugin loading, the kubernetes filter initializing, and the HTTP output plugin connecting. You might also see retry messages if FastAPI is not running yet. That is fine for now.
Pod stuck in CrashLoopBackOff?
A CrashLoopBackOff usually means the config has a syntax error. Check the pod logs for a parsing error message. Common causes include incorrect indentation in the fluent-bit-values.yaml file or mixing YAML pipeline format with classic .conf format.
If you see an error about the grep filter, verify that you only use Regex entries (not Exclude) when Logical_Op is set. Combining both causes a configuration error.
help me fix my Fluent Bit CrashLoopBackOff
✔️ Awesome, I've got everything!
Great. Double check you have saved fluent-bit-values.yaml and your Fluent Bit pod shows Running status.
ⓧ I'd like to double check the full code
Here is the complete fluent-bit-values.yaml file:
# Custom Fluent Bit Helm values
# Overrides filters and outputs for the SRE pipeline
config:
# Keep the default inputs (tail + systemd) from the chart
# Override filters: kubernetes metadata first, then grep for severity
filters: |
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
[FILTER]
Name grep
Match kube.*
Logical_Op or
Regex log ERROR
Regex log CRITICAL
# Override outputs: send filtered logs to FastAPI via HTTP
outputs: |
[OUTPUT]
Name http
Match *
Host host.docker.internal
Port 8000
URI /logs
Format jsonFluent Bit is now running in your cluster, filtering for error-level logs, and forwarding them to your FastAPI endpoint. In the next step, you will run the full end-to-end integration test and watch the entire pipeline fire.
Run the End-to-End Integration Test
Every piece of the pipeline is now in place. Fluent Bit is tailing container logs, your FastAPI service is ready to receive them, Ollama is loaded, and Redis is standing by to cache results.
In this step, you will fire the full pipeline end-to-end and confirm every link in the chain works: log collection, severity filtering, LLM diagnosis, caching, and automated GitHub Issue creation.
In this step, get ready to:
- Confirm the FastAPI service receives logs from Fluent Bit in real time.
- Watch the full pipeline fire and verify a GitHub Issue appears in your test repository.
- Confirm the Redis cache eliminates redundant LLM calls on repeated errors.
Confirm the FastAPI service is receiving logs
Your FastAPI service should already be running from Step 8. If you stopped it, you need to start it again so Fluent Bit has somewhere to send its filtered logs.
- Open a terminal in your ~/autonomous-sre/ directory.
- If your FastAPI server is not already running, activate the virtual environment and start it by running:
source venv/bin/activate
uvicorn main:app --host 0.0.0.0 --port 8000- Watch the terminal output. Within 30 seconds (the crashy app's error interval), you should see incoming POST requests logged by Uvicorn.
What should I see?
Your terminal should show lines indicating Fluent Bit is hitting the /logs endpoint. You will see Uvicorn access log lines showing POST /logs requests arriving, along with any processing messages your application prints.
Not seeing any incoming requests?
Make sure your FastAPI server is bound to 0.0.0.0:8000 (not 127.0.0.1). Kubernetes pods cannot reach 127.0.0.1 on the host.
Check that the Fluent Bit pods are still running with kubectl get pods. If a pod restarted, check its logs for connection errors pointing to the wrong host or port.
Help me debug this
Watch the full pipeline fire
The crashy app emits a JSON error log every 30 seconds. Once Fluent Bit forwards that error to your FastAPI service, the entire chain fires in sequence.
- If you want to trigger an immediate error instead of waiting, delete the crashy-app pod so the Deployment recreates it by running:
kubectl delete pod -l app=crashy-appWhy delete the pod?
Deleting a managed pod forces the Deployment controller to spin up a replacement immediately. The new pod starts its error loop from scratch, producing a fresh log entry within seconds rather than waiting for the next 30-second cycle.
- Switch back to the terminal running your FastAPI service and watch the output.
You should see the following sequence in your FastAPI logs as each error event arrives:
- A log line showing the incoming event was filtered as ERROR or CRITICAL severity.
- A cache-miss message indicating Redis does not have a stored diagnosis for this error signature.
- An LLM call to Ollama (this may take a few seconds on CPU inference).
- A message confirming the diagnosis was cached in Redis.
- A message confirming a GitHub Issue was created in your test repository.
- Open your test repository on GitHub in a browser.
- Click the Issues tab.
- Verify a new issue has appeared with a structured body containing the error category, root cause, severity, proposed kubectl fix, and raw log context.
What does the GitHub Issue look like?
The issue title contains the error category and a short hash of the error signature. The body is structured in markdown sections: root cause analysis, severity assessment, proposed remediation command, and the original log context for reference.
This is the kind of structured incident report an SRE team would use to triage production issues without needing a human to read raw logs first.
GitHub Issue not appearing?
Check your FastAPI terminal for HTTP error responses from the GitHub API. A 401 status means your Personal Access Token is invalid or expired. A 404 means the repository name or owner in your environment variables does not match your actual repository.
Confirm your token has Issues: Read and write permission scoped to the correct repository.
Help me debug the GitHub API integration
Verify Redis cache hit on repeated errors
The crashy app produces the same error message every 30 seconds. Once the first error is diagnosed and cached, every subsequent identical error should be served from Redis without calling Ollama.
- Wait for the next error cycle (about 30 seconds) or delete the pod again to trigger a fresh log immediately:
kubectl delete pod -l app=crashy-app- Watch your FastAPI terminal output for the second error event.
This time, you should see a cache-hit message instead of an LLM call. The response is near-instant because the diagnosis is served directly from Redis.
- Confirm no duplicate GitHub Issue was created. Your deduplication logic searches open issues for the same error signature before creating a new one.
- Verify the cached entry directly in Redis by running this in a separate terminal:
docker exec sre-redis redis-cli KEYS 'diagnosis:*'- You should see at least one key matching the diagnosis: prefix.
- Check the TTL of that key by running:
docker exec sre-redis redis-cli TTL $(docker exec sre-redis redis-cli KEYS 'diagnosis:*' | head -1)- You should see a number less than or equal to 600 (the 10-minute TTL configured in your caching layer).
Why does the cache matter?
LLM inference on CPU can take several seconds per call. In a production cluster where the same error fires hundreds of times per minute, caching the diagnosis means only the first occurrence triggers expensive compute. Every subsequent occurrence gets an instant response from Redis.
The 600-second TTL balances freshness with efficiency. If the underlying cause changes, the cache expires and a new diagnosis is generated automatically.
Not seeing a cache hit?
Make sure your error signature hashing is deterministic. If the hash includes a timestamp or request ID, each error generates a unique key and never hits the cache.
Verify the Redis container is still running with docker ps. Because you started it with --rm, if it stopped for any reason the container is gone and you need to restart it.
Help me debug my Redis caching layer
Your autonomous SRE pipeline is fully operational. Errors appear, AI diagnoses them, results are cached, and incident tickets open themselves. Next up, you will extend this system with real-time Slack alerting in the Secret Mission.
Secret mission
Add Slack Webhook Alerting
Your pipeline creates GitHub Issues, but those require someone to check their inbox. Real SRE teams get instant alerts in Slack the moment something breaks. Extend your system to send formatted Slack notifications alongside the GitHub Issues, so your team sees errors the second they happen.
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely locally on your machine, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.
Resources you used:
- Ollama system service with the qwen3:1.7b model (stored at /usr/share/ollama/.ollama/models).
- Redis Docker container (sre-redis, ephemeral with --rm flag).
- Fluent Bit Helm release in your Kubernetes cluster.
- Crashy app Deployment in your Kubernetes cluster.
- GitHub Personal Access Token (fine-grained).
- Python virtual environment with project dependencies.
Keep everything running
No action needed. Choose this if you are still actively building or plan to continue experimenting with the pipeline right away.
- Ollama continues running as a systemd service. The model stays loaded in RAM for 5 minutes after each request, then unloads automatically.
- Redis container stays running until you stop it or restart Docker.
- Fluent Bit and the crashy app remain deployed in your cluster. They consume minimal resources.
- Your GitHub PAT remains active until you delete it or it expires.
Pause - I'll come back to this later
Shut down running processes to free up memory and CPU. Your files and configurations stay intact so you can resume later.
- Stop the Ollama service by running:
- Stop the Redis container by running:
sudo systemctl stop ollamadocker stop sre-redis- Scale the crashy app to zero replicas so it stops generating logs:
kubectl scale deployment crashy-app --replicas=0When you are ready to resume, start Ollama again with sudo systemctl start ollama, run a new Redis container, and scale the crashy app back up.
Delete - I don't want to use this again
Remove all project resources completely. Nothing will remain on your machine after these steps.
Remove Ollama entirely
- Stop and disable the Ollama service, then remove all Ollama files by running these commands:
sudo systemctl stop ollama
sudo systemctl disable ollama
sudo rm /etc/systemd/system/ollama.service
sudo rm -r $(which ollama | tr 'bin' 'lib')
sudo rm $(which ollama)
sudo userdel ollama
sudo groupdel ollama
sudo rm -r /usr/share/ollamaKeep Ollama but remove the model
If you want to keep Ollama installed for other projects but free up disk space, run this instead of the full uninstall:
ollama rm qwen3:1.7bStop the Redis container
- Stop the Redis container by running:
docker stop sre-redisBecause you started Redis with the --rm flag, the container is automatically removed when it stops. No data persists.
Remove Fluent Bit and the crashy app from Kubernetes
- Uninstall the Fluent Bit Helm release and delete the crashy app Deployment by running:
helm uninstall fluent-bit
kubectl delete deployment crashy-appDelete the Python virtual environment and project files
- Remove the entire project directory by running:
rm -rf ~/Desktop/autonomous-sreRevoke your GitHub Personal Access Token
- Go to GitHub and click your profile picture in the top right corner.
- Click Settings.
- In the left sidebar, click Developer settings.
- Under Personal access tokens, click Fine-grained tokens.
- Find the token you created for this project and click Delete to the right of it.
That's a Wrap!
That's a Wrap!
Nice work. You just built an autonomous SRE system that detects, diagnoses, and responds to Kubernetes incidents without human intervention.
You've learned how to:
- Build an AI-powered log analysis pipeline using FastAPI, Ollama, and Redis that automatically diagnoses Kubernetes errors with a local LLM and caches results to eliminate redundant inference.
- Deploy Fluent Bit as a Kubernetes DaemonSet with a custom pipeline that collects container logs, filters for errors, and forwards them via HTTP in real time.
- Automate incident response by creating GitHub Issues with structured diagnosis reports, severity labels, and proposed kubectl remediation commands.
- Secret Mission: Added Slack webhook alerting for real-time notifications alongside GitHub Issues.
Ready to quiz yourself?