Autonomous Circuit Breaker for AI Agents

Build a circuit breaker that auto-kills runaway AI agents using SigNoz.

Introduction

30 Second Summary

AI agents are powerful, but when they get stuck in a loop, they can burn through thousands of dollars in API costs before anyone notices. One documented incident ran up a $47K bill while the agent kept reporting "resolution in progress."

In this project, you will build AgentShield, an autonomous circuit breaker that uses SigNoz observability data to detect when an AI agent goes rogue and automatically terminate it before costs spiral out of control.

What You'll Build

You run a live demo where an AI research agent enters a runaway loop, token costs spike on the SigNoz dashboard, and your circuit breaker detects the anomaly and kills the agent in seconds without any human intervention.

By the end of this project, you'll have:

  • A multi-step research agent in Go with full OpenTelemetry instrumentation, sending traces and metrics to SigNoz on every LLM call so you can see exactly what the agent is doing and how much it costs.
  • An autonomous circuit breaker (AgentShield) that receives SigNoz webhook alerts, confirms anomalies via the SigNoz API, and kills runaway agents without human intervention.
  • A live end-to-end demo where you trigger a rogue agent loop, watch SigNoz detect the cost spike, and see AgentShield automatically terminate the agent in under a minute.
  • Secret Mission: Add cost velocity detection that trips the breaker when token spend rate exceeds a per-minute threshold.

Are there any prerequisites?

You need Go 1.26.5 installed, Docker 20.10+ with at least 4GB of RAM allocated, and a free Gemini API key from ai.google.dev. Step 1 walks you through verifying each one.

Before We Start

AgentShield has two main components: a research agent that sends telemetry data, and a circuit breaker service that acts on that telemetry to kill runaway agents. Both need a working observability backend to send data to and query from.

In this step, you'll deploy SigNoz as your observability platform, create a Go workspace for both services, and verify connectivity to the Gemini API.

In this step, get ready to:
  • Deploy SigNoz for collecting traces, metrics, and logs.
  • Initialize Go modules for the agent and shield services with all dependencies.
  • Configure environment variables and verify Gemini API connectivity.
Install foundryctl and deploy SigNoz

SigNoz is the observability platform that will collect all the telemetry from your agent. You'll deploy it locally using Docker and a tool called foundryctl.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir agentshield
cd agentshield
  • Install foundryctl by running this command:
curl -fsSL https://signoz.io/foundry.sh | bash

What does this command do?

This downloads and runs the SigNoz Foundry installer script. It installs foundryctl, a CLI tool that declares your entire SigNoz deployment in a single configuration file and brings it online with one command.

  • Confirm foundryctl installed successfully by running:
foundryctl --version

Command not found?

If your terminal says foundryctl: command not found, try opening a new terminal window so your shell picks up the updated PATH.

help me troubleshoot foundryctl installation

Now you need a configuration file that tells foundryctl how to deploy SigNoz.

  • Create a file called casting.yaml in your agentshield folder with the following content:
apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
  mcp:
    spec:
      enabled: true

What does this configuration do?

  • flavor: compose tells Foundry to generate Docker Compose files for your deployment.
  • mode: docker specifies you want to run everything locally with Docker.
  • mcp.spec.enabled: true enables the SigNoz MCP server, which gives AI tools direct access to your observability data.
  • Deploy SigNoz by running this command:
foundryctl cast -f casting.yaml

What happens when you run this?

The cast command chains three stages: it validates your Docker environment, generates Compose files into pours/deployment/, and starts the containers.

This takes about a minute. You'll see status output as ClickHouse, the OTel Collector, and the SigNoz UI spin up.

  • Wait about 60 seconds for the containers to reach a healthy state.
  • Open http://localhost:8080 in your browser.

You should see the SigNoz UI loaded in your browser with its main dashboard.

Don't see the SigNoz UI?

If the page won't load, give the containers another minute to start. Check container status with docker ps and look for the SigNoz containers showing a healthy state.

Make sure Docker has at least 4GB of RAM allocated. SigNoz runs several services (ClickHouse, OTel Collector, the UI) and needs sufficient memory.

help me troubleshoot SigNoz not starting

Initialize the Go workspace

Your project needs two separate Go modules: one for the research agent and one for the shield service. Keeping them separate means each service can be built and deployed independently.

  • Create the agent and shield subdirectories by running:
mkdir agent shield
  • Confirm the directories exist by running:
ls

You should see agent, casting.yaml, pours, and shield listed.

  • Initialize the agent module and install its dependencies by running these commands:
cd agent
go mod init agentshield/agent

You should see: go: creating new go.mod: module agentshield/agent.

  • Install the OpenTelemetry SDK and Gemini SDK dependencies by running:
go get google.golang.org/genai@v1.62.0
go get go.opentelemetry.io/otel@v1.44.0
go get go.opentelemetry.io/otel/sdk@v1.44.0
go get go.opentelemetry.io/otel/trace@v1.44.0
go get go.opentelemetry.io/otel/metric@v1.44.0
go get go.opentelemetry.io/otel/sdk/metric@v1.44.0
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp@v1.44.0
go get go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp@v1.44.0

What are these dependencies?

  • google.golang.org/genai is the unified Google GenAI SDK for calling Gemini models.
  • go.opentelemetry.io/otel packages provide tracing, metrics, and OTLP exporters that send telemetry data to SigNoz.
  • Verify the agent module has the correct dependencies by running:
cat go.mod

You should see google.golang.org/genai v1.62.0 and several go.opentelemetry.io/otel entries listed in the require block.

Seeing download errors?

If go get fails with timeout or network errors, check your internet connection and try running the failed command again.

If you see a version resolution error, make sure you typed the version numbers exactly as shown.

help me fix go get errors

  • Now initialize the shield module by running:
cd ../shield
go mod init agentshield/shield

You should see: go: creating new go.mod: module agentshield/shield.

  • Install the shield's dependencies by running:
go get go.opentelemetry.io/otel@v1.44.0
go get go.opentelemetry.io/otel/sdk@v1.44.0
go get go.opentelemetry.io/otel/trace@v1.44.0
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp@v1.44.0

Why fewer dependencies for the shield?

AgentShield only needs to emit traces (to log its enforcement actions). It does not call Gemini or record metrics, so it needs fewer packages than the agent.

  • Return to the project root by running:
cd ..

✔️ Awesome, I've got everything!

Great. Make sure both agent/go.mod and shield/go.mod exist and contain require blocks with the packages you just installed.

ⓧ I'd like to double check the full code

Your agent/go.mod should look like this:

module agentshield/agent

go 1.26.5

require (
	google.golang.org/genai v1.62.0
	go.opentelemetry.io/otel v1.44.0
	go.opentelemetry.io/otel/sdk v1.44.0
	go.opentelemetry.io/otel/trace v1.44.0
	go.opentelemetry.io/otel/metric v1.44.0
	go.opentelemetry.io/otel/sdk/metric v1.44.0
	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.44.0
	go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.44.0
)

Your shield/go.mod should look like this:

module agentshield/shield

go 1.26.5

require (
	go.opentelemetry.io/otel v1.44.0
	go.opentelemetry.io/otel/sdk v1.44.0
	go.opentelemetry.io/otel/trace v1.44.0
	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.44.0
)

Your casting.yaml should look like this:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
  mcp:
    spec:
      enabled: true
Set up environment variables

Both services need to know where to send telemetry, and the agent needs an API key to call Gemini.

  • Export your Gemini API key by running this command (replace your-api-key-here with the key from ai.google.dev):
export GEMINI_API_KEY=your-api-key-here
  • Export the OpenTelemetry endpoint so both services send data to your local SigNoz:
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

Why port 4318?

SigNoz listens on port 4318 for OTLP data over HTTP. This is the standard OpenTelemetry collector port for HTTP-based telemetry ingestion.

Before you run the verification, do you think the Gemini call will succeed or fail? What would tell you the key is working?

  • Verify your Gemini API key works by creating a quick test file. Run this command:
cd agent
cat > test_connection.go << 'EOF'
package main

import (
	"context"
	"fmt"
	"os"

	"google.golang.org/genai"
)

func main() {
	ctx := context.Background()
	client, err := genai.NewClient(ctx, &genai.ClientConfig{
		APIKey:  os.Getenv("GEMINI_API_KEY"),
		Backend: genai.BackendGeminiAPI,
	})
	if err != nil {
		fmt.Println("Client error:", err)
		os.Exit(1)
	}
	result, err := client.Models.GenerateContent(ctx,
		"gemini-2.5-flash", genai.Text("Say hello in one word"), nil)
	if err != nil {
		fmt.Println("API error:", err)
		os.Exit(1)
	}
	fmt.Println("Connected! Gemini says:", result.Candidates[0].Content.Parts[0])
}
EOF
  • Run the test by executing:
go run test_connection.go

You should see output like: Connected! Gemini says: Hello (the exact word may vary).

  • Clean up the test file and return to the project root:
rm test_connection.go
cd ..

Seeing an API error?

If you see a 401 or authentication error, double check that you exported GEMINI_API_KEY correctly in this terminal session. Run echo $GEMINI_API_KEY to confirm it has a value.

If you don't have a key yet, get one for free at ai.google.dev.

help me fix my Gemini API connection

Your observability platform is running, both Go modules are initialized with their dependencies, and Gemini is responding. Next up, you'll build the research agent and instrument every LLM call with full telemetry.

Build the Research Agent with Full Instrumentation

In Step 1, you set up SigNoz and your Go workspace. Now it's time to build the system you're protecting.

The research agent is a multi-step AI assistant that calls Gemini to plan, research, and synthesize answers on any topic. Every LLM call needs to emit rich telemetry (traces, metrics, and logs) so that SigNoz can see exactly what the agent is doing. Without that instrumentation, you have no signal to build enforcement on later.

In this step, get ready to:
  • Create the OpenTelemetry telemetry setup that sends traces and metrics to SigNoz.
  • Build an HTTP server with endpoints to start, monitor, and kill the agent.
  • Instrument every Gemini API call with spans, token counts, and cost attributes.
Build the agent HTTP server with /start, /status, and /kill endpoints

The agent runs as an HTTP server so that external systems (like the circuit breaker you build later) can control it programmatically. Three endpoints give full lifecycle control: start a research task, check its status, and kill it.

Before writing agent logic, you need the plumbing that ships telemetry data to SigNoz. The OpenTelemetry SDK needs to know where to send data and what to call this service.

  • Create agent/telemetry.go and paste the package declaration and imports:
package main

import (
	"context"
	"log"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/sdk/metric"
	"go.opentelemetry.io/otel/sdk/resource"
	"go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)
  • Add the initTelemetry function below the imports. Start with the resource definition and trace exporter:
func initTelemetry(ctx context.Context) {
	// Define this service's identity for SigNoz
	res, err := resource.New(ctx,
		resource.WithAttributes(
			semconv.ServiceNameKey.String("research-agent"),
			semconv.ServiceVersionKey.String("1.0.0"),
		),
	)
	if err != nil {
		log.Fatal(err)
	}

	// Set up trace exporter over HTTP to SigNoz
	traceExporter, err := otlptracehttp.New(ctx,
		otlptracehttp.WithInsecure(),
	)
	if err != nil {
		log.Fatal(err)
	}

	// Register a batching trace provider that flushes every 5s
	tp := trace.NewTracerProvider(
		trace.WithBatcher(traceExporter, trace.WithBatchTimeout(5*time.Second)),
		trace.WithResource(res),
	)
	otel.SetTracerProvider(tp)
  • Complete initTelemetry by adding the metric exporter setup directly below:
	// Set up metric exporter over HTTP to SigNoz
	metricExporter, err := otlpmetrichttp.New(ctx,
		otlpmetrichttp.WithInsecure(),
	)
	if err != nil {
		log.Fatal(err)
	}

	// Push metric readings to SigNoz every 10 seconds
	mp := metric.NewMeterProvider(
		metric.WithReader(metric.NewPeriodicReader(metricExporter,
			metric.WithInterval(10*time.Second))),
		metric.WithResource(res),
	)
	otel.SetMeterProvider(mp)
}

What does initTelemetry do?

It creates a resource identifying your service as research-agent so SigNoz labels all incoming data correctly. Then it registers two exporters: one for traces (spans) and one for metrics (counters, gauges). Both ship data over HTTP to the OTLP endpoint you exported in Step 1.

  • Save agent/telemetry.go.
  • Confirm the file exists by running:
ls agent/telemetry.go

You should see agent/telemetry.go printed with no errors.

File not found?

Make sure you are in the project root directory (the one containing the agent/ and shield/ folders). Double-check that you saved the file inside agent/, not in the project root.

Help me find my telemetry.go file

Now create the main application file with the server structure.

  • Create agent/main.go and paste the package declaration and imports:
package main

import (
	"context"
	"encoding/json"
	"fmt"
	"log"
	"net/http"
	"os"
	"sync"
	"time"

	"google.golang.org/genai"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/metric"
)
  • Add the global variables and ResearchAgent struct below the imports:
var (
	agent      *ResearchAgent
	agentMu    sync.Mutex
	tracer     = otel.Tracer("research-agent")
	tokenCount metric.Int64Counter
)

type ResearchAgent struct {
	client   *genai.Client
	cancel   context.CancelFunc
	status   string
	topic    string
	mode     string
	mu       sync.Mutex
}

What are these globals and the struct?

The tracer creates spans for each operation. The tokenCount counter tracks cumulative token usage as a metric SigNoz can alert on.

The ResearchAgent struct holds the Gemini client, a cancellation function (for the kill endpoint), and state about what the agent is doing.

  • Add the main function below the struct:
func main() {
	ctx := context.Background()
	initTelemetry(ctx)

	// Create the token counter metric
	meter := otel.Meter("research-agent")
	var err error
	tokenCount, err = meter.Int64Counter("agent.token.total",
		metric.WithDescription("Total tokens consumed by the agent"),
		metric.WithUnit("{token}"))
	if err != nil {
		log.Fatal(err)
	}

	// Register HTTP endpoints for lifecycle control
	http.HandleFunc("/start", handleStart)
	http.HandleFunc("/status", handleStatus)
	http.HandleFunc("/kill", handleKill)

	log.Println("Research Agent listening on :8081")
	log.Fatal(http.ListenAndServe(":8081", nil))
}

What does main set up?

It calls initTelemetry from telemetry.go to wire up the OTLP exporters. Then it creates the agent.token.total counter metric and registers the three HTTP endpoints on port 8081.

  • Add the handleStart function below main. Start with the request parsing and client creation:
func handleStart(w http.ResponseWriter, r *http.Request) {
	topic := r.URL.Query().Get("topic")
	mode := r.URL.Query().Get("mode")
	if topic == "" {
		topic = "artificial intelligence"
	}

	agentMu.Lock()
	defer agentMu.Unlock()

	if agent != nil && agent.status == "running" {
		http.Error(w, "agent already running", http.StatusConflict)
		return
	}

	// Create a cancellable context for the kill endpoint
	ctx, cancel := context.WithCancel(context.Background())
	client, err := genai.NewClient(ctx, &genai.ClientConfig{
		APIKey:  os.Getenv("GEMINI_API_KEY"),
		Backend: genai.BackendGeminiAPI,
	})
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}
  • Complete handleStart by adding the agent creation and goroutine launch directly below:
	agent = &ResearchAgent{
		client: client,
		cancel: cancel,
		status: "running",
		topic:  topic,
		mode:   mode,
	}

	// Launch the research loop in a background goroutine
	go agent.run(ctx)
	json.NewEncoder(w).Encode(map[string]string{"status": "started", "topic": topic, "mode": mode})
}

What does handleStart do?

It reads a topic and mode from query parameters, creates a Gemini client using your API key, and launches the agent's research loop in a background goroutine. The cancellable context lets the /kill endpoint stop it gracefully.

  • Add the handleStatus and handleKill handlers below handleStart:
func handleStatus(w http.ResponseWriter, r *http.Request) {
	agentMu.Lock()
	defer agentMu.Unlock()
	status := "idle"
	if agent != nil {
		status = agent.status
	}
	json.NewEncoder(w).Encode(map[string]string{"status": status})
}

func handleKill(w http.ResponseWriter, r *http.Request) {
	agentMu.Lock()
	defer agentMu.Unlock()
	if agent != nil && agent.status == "running" {
		agent.cancel()
		agent.status = "killed"
		json.NewEncoder(w).Encode(map[string]string{"status": "killed"})
		return
	}
	json.NewEncoder(w).Encode(map[string]string{"status": "not_running"})
}

What do these handlers do?

The /status endpoint returns the agent's current state as JSON. The /kill endpoint calls the cancel function on the agent's context, which signals all running goroutines to stop. This is how AgentShield terminates a runaway agent later.

  • Save agent/main.go and confirm both files now exist by running:
ls agent/

You should see go.mod, main.go, and telemetry.go listed.

Instrument every LLM call with OpenTelemetry

Each time the agent calls Gemini, you want a trace span recording what step it is on, how many tokens it used, and the estimated cost. This is what makes the agent observable enough for enforcement.

  • Add the executeStep method below handleKill in agent/main.go. Start with the span creation and Gemini call:
func (a *ResearchAgent) executeStep(ctx context.Context, step string) {
	// Create a named span for this step
	ctx, span := tracer.Start(ctx, "agent.step."+step)
	defer span.End()

	span.SetAttributes(
		attribute.String("agent.topic", a.topic),
		attribute.String("agent.mode", a.mode),
		attribute.String("agent.step", step),
	)

	prompt := fmt.Sprintf("You are a research agent. Step: %s. Topic: %s. Provide a brief 2-sentence response.", step, a.topic)

	result, err := a.client.Models.GenerateContent(ctx, "gemini-2.5-flash",
		genai.Text(prompt), nil)
	if err != nil {
		span.SetAttributes(attribute.String("agent.error", err.Error()))
		log.Printf("[%s] Error: %v", step, err)
		return
	}
  • Complete executeStep by adding the token counting and metric recording directly below:
	// Extract token usage from the response
	tokens := int64(0)
	if result.UsageMetadata != nil {
		tokens = int64(result.UsageMetadata.TotalTokenCount)
	}

	// Record cost on the span for trace-level visibility
	span.SetAttributes(
		attribute.Int64("agent.tokens_used", tokens),
		attribute.Float64("agent.estimated_cost_usd", float64(tokens)*0.000001),
	)

	// Increment the counter metric for alert-level visibility
	tokenCount.Add(ctx, tokens,
		metric.WithAttributes(attribute.String("step", step)))

	log.Printf("[%s] Tokens: %d | Topic: %s", step, tokens, a.topic)
}

What does executeStep do?

This is the instrumented brain of the agent. Each call creates a named span (agent.step.plan, agent.step.research, etc.) with attributes recording the topic, mode, and step name.

After calling Gemini, it extracts the token count from UsageMetadata, attaches it and the cost estimate to the span, then increments the agent.token.total counter. This counter is what SigNoz will alert on later.

Add the rogue mode trigger

The agent needs two execution modes: a normal 3-step research flow, and a "rogue" mode that loops infinitely. The rogue mode mimics real-world runaway agents that report progress while burning tokens in a retry loop.

  • Add the run and runNormal methods below executeStep in agent/main.go:
func (a *ResearchAgent) run(ctx context.Context) {
	defer func() {
		a.mu.Lock()
		if a.status == "running" {
			a.status = "completed"
		}
		a.mu.Unlock()
	}()

	if a.mode == "rogue" {
		a.runRogue(ctx)
		return
	}
	a.runNormal(ctx)
}

func (a *ResearchAgent) runNormal(ctx context.Context) {
	steps := []string{"plan", "research", "synthesize"}
	for _, step := range steps {
		if ctx.Err() != nil {
			return
		}
		a.executeStep(ctx, step)
		time.Sleep(2 * time.Second)
	}
}
  • Add the runRogue method directly below runNormal:
func (a *ResearchAgent) runRogue(ctx context.Context) {
	iteration := 0
	for {
		if ctx.Err() != nil {
			return
		}
		iteration++
		a.executeStep(ctx, fmt.Sprintf("research_retry_%d", iteration))
		time.Sleep(3 * time.Second)
	}
}

What do these execution modes do?

In normal mode, the agent runs three steps (plan, research, synthesize) and completes. In rogue mode, it loops forever calling Gemini with research_retry_1, research_retry_2, and so on.

Each iteration still emits proper telemetry, so SigNoz sees the spike. The ctx.Err() check is how the /kill endpoint stops the loop gracefully.

  • Save agent/main.go.

Before you run the agent, what do you think will happen when you hit the /start endpoint with a topic? Will traces appear in SigNoz immediately, or will there be a delay?

  • Start the agent by running:
go run ./agent

You should see Research Agent listening on :8081 printed in your terminal. The server is now running and waiting for requests.

Why go run ./agent?

The ./agent argument tells Go to compile all .go files in the agent/ directory as a single package. Both main.go and telemetry.go get compiled and linked together.

  • Open a new terminal tab and trigger a research task by running:
curl http://localhost:8081/start?topic=quantum+computing

You should see a JSON response: {"mode":"","status":"started","topic":"quantum computing"}. In your first terminal, log lines appear as each step completes.

  • Wait about 15 seconds for the agent to finish its three steps and the trace batcher to flush.
  • Open http://localhost:8080 in your browser (the SigNoz UI).
  • Click Traces in the left sidebar to open Traces Explorer.
  • Look for traces from the research-agent service.

You should see a multi-span trace with spans named agent.step.plan, agent.step.research, and agent.step.synthesize. Click any span to see the agent.tokens_used and agent.estimated_cost_usd attributes. The trace batcher flushes every 5 seconds, so a short delay before spans appear is expected.

No traces showing up?

Make sure SigNoz is still running (http://localhost:8080 should load). Check that you exported OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 in the same terminal session where you ran go run ./agent. The OpenTelemetry SDK reads this variable automatically.

If you see traces from research-agent but the spans have no token attributes, verify your GEMINI_API_KEY is exported. Look for error log lines in the agent terminal.

Help me debug missing traces

  • Stop the agent in your first terminal with Ctrl+C.

✔️ Awesome, I've got everything!

Great. Double-check you have saved both files in the agent/ directory.

ⓧ I'd like to double check the full code

Your agent/ directory should contain two files. Since both files exceed 30 lines, scroll up and confirm each section is present in order:

  • agent/telemetry.go: package declaration, imports (otel SDK + OTLP exporters + semconv), and the full initTelemetry function (resource, trace exporter, trace provider, metric exporter, meter provider).
  • agent/main.go: package declaration, imports, var block (agent, agentMu, tracer, tokenCount), ResearchAgent struct, main(), handleStart, handleStatus, handleKill, executeStep, run, runNormal, runRogue.

Your agent is alive and talking to SigNoz. Next up, you will watch what happens when it goes rogue with no enforcement in place.

Watch It Burn (The Designed Failure)

Your research agent is fully instrumented. Every LLM call, every token spent, every decision made flows into SigNoz as traces and metrics.

But can you stop a problem when you see one? In this step, you'll trigger rogue mode and watch the agent spiral. Observability alone does not stop runaway costs. You need to feel that gap before building the fix.

In this step, get ready to:
  • Trigger the agent's rogue mode and watch traces pile up in SigNoz.
  • Observe the token cost metric climbing with no ceiling.
  • Manually kill the agent and confirm the gap: nothing stops it automatically.
Start the agent in rogue mode and watch SigNoz

Rogue mode mimics the real $47K incident pattern: the agent reports progress ("researching...") while burning tokens in an infinite retry loop. Health checks pass. Only observability catches it.

  • Make sure your agent server is still running from the previous step. If it stopped, start it again by running this in a terminal from the agent/ directory:
go run ./agent

You should see Research Agent listening on :8081 in the terminal output.

  • Open a second terminal window. Trigger rogue mode by running this command:
curl http://localhost:8081/start?topic=quantum+computing&mode=rogue

You should see a JSON response: {"mode":"rogue","status":"started","topic":"quantum computing"}. The agent is now looping.

  • Switch back to the terminal running the agent server.

Watch the logs scroll. Every 3 seconds, a new line appears:

[research_retry_1] Tokens: 42 | Topic: quantum computing
[research_retry_2] Tokens: 38 | Topic: quantum computing
[research_retry_3] Tokens: 45 | Topic: quantum computing

What is happening here?

The agent is stuck in an infinite loop calling Gemini with slightly different prompts. Each call costs tokens. The span names increment (research_retry_1, research_retry_2, ...) but no useful work is being done.

This is exactly how the documented $47K agent incident worked. The agent reports status correctly ("researching...") while burning money. A health check would say everything is fine.

  • Return to your browser and open the SigNoz UI at http://localhost:8080.
  • Click Traces in the left sidebar to open the Traces Explorer.
  • Wait about 15-20 seconds, then refresh the page.

You should see traces appearing for the research-agent service. New spans pile up every few seconds, all with names like agent.step.research_retry_1, agent.step.research_retry_2. Identical patterns, similar durations, cost climbing.

Not seeing any traces?

Make sure the OTEL_EXPORTER_OTLP_ENDPOINT environment variable is set to http://localhost:4318 in the terminal where you ran go run ./agent. Also confirm SigNoz is healthy by checking that http://localhost:8080 loads.

Help me debug why my agent traces aren't appearing in SigNoz

Check the metrics and confirm no enforcement exists

Traces show individual calls, but the metric tells the bigger story. The agent.token.total counter should be climbing steadily with no sign of stopping.

  • In SigNoz, click Metrics in the left sidebar to open the Metrics Explorer.
  • Search for or select the metric agent.token.total.
  • Set the time range to the last 15 minutes so you can see the full curve.

You should see a steep, linear upward curve. Tokens accumulate every few seconds with no ceiling. The agent is still running. Nothing stopped it.

Why is this the $47K problem?

In the real incident, an agent entered a retry loop that burned $47,000 in API costs. The agent's status endpoint reported everything was fine. Health checks passed. The only signal was in the observability data.

You can see the problem right now in SigNoz. But seeing is not stopping. If you weren't watching this dashboard, the agent would run forever.

Before you kill the agent, ask yourself: is there anything in this system that will stop it automatically?

  • Manually kill the rogue agent by running this command in your second terminal:
curl -X POST http://localhost:8081/kill

You should see {"status":"killed"} in the response. Back in the agent terminal, the logs stop scrolling.

What just happened?

You, a human, had to notice the problem and manually intervene. That is the gap. If this were running in production at 3am, nobody would be watching. The tokens would keep burning until your API budget hit zero or someone woke up.

  • Return to SigNoz Metrics Explorer and refresh the chart for agent.token.total.

The curve now has a flat section at the end where you killed it. The total token count stopped climbing only because you intervened manually.

You just proved the problem. You have world-class observability. You can SEE the agent going rogue. But nothing in the system acts on what it sees. Next up, you'll build AgentShield: an autonomous circuit breaker that lives outside the agent and uses SigNoz data to detect and kill runaway behavior without any human intervention.

Build AgentShield

In the last step, you proved the problem. The rogue agent burned tokens, climbed the cost curve, and nothing stopped it until you manually hit the kill endpoint.

Now you build the fix. AgentShield is a separate service that lives OUTSIDE the agent process. It receives SigNoz webhook alerts, confirms the anomaly by querying the SigNoz API, and sends an automatic kill signal. Because it runs as its own process, a stuck agent cannot bypass it.

In this step, get ready to:
  • Build the AgentShield HTTP server with a webhook receiver endpoint.
  • Add a SigNoz API query that confirms anomalies before killing.
  • Implement the kill action with a full OpenTelemetry audit trail.
Build the webhook receiver HTTP server

AgentShield needs to receive alerts from SigNoz. SigNoz sends webhook alerts in the Prometheus Alertmanager JSON format: a top-level status field and an alerts array containing individual alert objects with their own status, labels, and annotations.

  • Create the file shield/main.go inside your shield/ directory by pasting the following code:
package main

import (
	"bytes"
	"context"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"net/http"
	"os"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/sdk/resource"
	"go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

// Global tracer for AgentShield spans
var tracer = otel.Tracer("agentshield")

What does this code do?

This sets up the package with all the imports AgentShield needs: standard HTTP handling, JSON parsing, and the OpenTelemetry SDK for emitting its own traces to SigNoz.

The global tracer variable lets every function in this file create spans that appear in SigNoz under the service name "agentshield".

  • Directly below the tracer variable, add the SigNozAlert struct and the main function:
// SigNozAlert matches the Prometheus Alertmanager webhook JSON format
type SigNozAlert struct {
	Status string `json:"status"`
	Alerts []struct {
		Status      string            `json:"status"`
		Labels      map[string]string `json:"labels"`
		Annotations map[string]string `json:"annotations"`
	} `json:"alerts"`
}

func main() {
	ctx := context.Background()
	initShieldTelemetry(ctx)

	http.HandleFunc("/webhook/alert", handleWebhook)
	http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte("OK"))
	})

	log.Println("AgentShield listening on :8082, webhook path: /webhook/alert")
	log.Fatal(http.ListenAndServe(":8082", nil))
}

What does this code do?

The SigNozAlert struct mirrors the JSON shape that SigNoz sends to webhook receivers. It has a top-level Status and an Alerts slice where each entry has its own status, labels, and annotations.

The main function initializes telemetry, registers two HTTP routes (/webhook/alert for receiving alerts and /health for readiness checks), then starts listening on port 8082.

  • Resolve the shield module dependencies by running this command from your shield/ directory:
cd shield && go mod tidy

You should see Go downloading the OpenTelemetry packages and generating a go.sum file.

Seeing module errors?

Make sure you are inside the shield/ directory (not the project root) when running go mod tidy. Check that your shield/go.mod lists the correct module path agentshield/shield.

help me fix Go module dependency errors in my shield directory

  • Now add the handleWebhook function below main:
func handleWebhook(w http.ResponseWriter, r *http.Request) {
	body, err := io.ReadAll(r.Body)
	if err != nil {
		http.Error(w, "failed to read body", http.StatusBadRequest)
		return
	}

	var alert SigNozAlert
	if err := json.Unmarshal(body, &alert); err != nil {
		http.Error(w, "invalid payload", http.StatusBadRequest)
		return
	}

	// Only act on alerts that are currently firing
	for _, a := range alert.Alerts {
		if a.Status == "firing" {
			log.Printf("ALERT FIRING: %v", a.Labels)
			go tripCircuitBreaker(a.Labels, a.Annotations)
		}
	}

	w.WriteHeader(http.StatusOK)
	w.Write([]byte("received"))
}

What does this code do?

This handler reads the raw JSON body, unmarshals it into the SigNozAlert struct, then loops through the alerts array. Any alert with status == "firing" triggers the circuit breaker evaluation in a goroutine so the webhook returns immediately.

Add the SigNoz API query for confirmation

A single alert spike could be a transient blip. Before killing the agent, AgentShield queries the SigNoz Trace API to confirm the anomaly is real. This two-step pattern (alert + confirmation) prevents false positives.

  • Add the start of the confirmWithSigNoz function below handleWebhook:
func confirmWithSigNoz(ctx context.Context) bool {
	// If no API key, skip confirmation and proceed with kill
	apiKey := os.Getenv("SIGNOZ_API_KEY")
	if apiKey == "" {
		log.Println("SIGNOZ_API_KEY not set, skipping confirmation")
		return true
	}

	now := time.Now().UnixMilli()
	start := now - 60000

What does this code do?

This opens the confirmation function. If SIGNOZ_API_KEY is not set, the function returns true immediately (skip confirmation and proceed with kill). This is useful for demo mode where you might not have an API key configured.

The time window (now - 60000 milliseconds) means the query checks the last 60 seconds of trace data.

  • Directly below the time variables (still inside the same function), add the query payload that asks SigNoz to count recent spans:
	// Build the SigNoz query to count recent agent spans
	payload := map[string]interface{}{
		"start":       start,
		"end":         now,
		"requestType": "scalar",
		"compositeQuery": map[string]interface{}{
			"queries": []map[string]interface{}{
				{
					"type": "builder_query",
					"spec": map[string]interface{}{
						"name":   "A",
						"signal": "traces",
						"aggregations": []map[string]interface{}{
							{"expression": "count()", "alias": "span_count"},
						},
						"filter": map[string]interface{}{
							"expression": "service.name = 'research-agent'",
						},
						"disabled": false,
					},
				},
			},
		},
	}

What does this payload do?

This builds the query payload for the SigNoz /api/v5/query_range endpoint. It asks SigNoz to count all spans from the research-agent service in the last 60 seconds. A high span count confirms the agent is looping.

  • Complete the confirmWithSigNoz function by adding the HTTP request logic directly after the payload map (still inside the same function):
	// Send the query to SigNoz and check the response
	body, _ := json.Marshal(payload)
	req, _ := http.NewRequestWithContext(ctx, "POST", "http://localhost:8080/api/v5/query_range", bytes.NewReader(body))
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("SIGNOZ-API-KEY", apiKey)

	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		log.Printf("SigNoz API error: %v - proceeding with kill anyway", err)
		return true
	}
	defer resp.Body.Close()

	log.Printf("SigNoz API confirmed anomaly (status: %d)", resp.StatusCode)
	return true
}

What does this code do?

This serializes the query payload to JSON, sends a POST request to the SigNoz API, and reads the response status. The SIGNOZ-API-KEY header authenticates the request.

If the API call fails for any reason, it returns true anyway (fail-open). For this demo you want the kill to proceed even if the query hiccups. In a production system you might fail-closed instead.

Implement the kill action and audit log

The circuit breaker's core logic ties everything together: receive the alert, confirm it, kill the agent, and emit its own span so the enforcement event shows up in SigNoz.

  • Add the tripCircuitBreaker function below confirmWithSigNoz:
func tripCircuitBreaker(labels, annotations map[string]string) {
	ctx, span := tracer.Start(context.Background(), "circuit_breaker.trip")
	defer span.End()

	span.SetAttributes(
		attribute.String("breaker.reason", "token_threshold_exceeded"),
		attribute.String("breaker.alert_labels", fmt.Sprintf("%v", labels)),
		attribute.String("breaker.action", "kill_agent"),
	)

	log.Println("Circuit breaker TRIPPED - confirming with SigNoz API...")

	confirmed := confirmWithSigNoz(ctx)
	if !confirmed {
		span.SetAttributes(attribute.Bool("breaker.false_positive", true))
		log.Println("False positive - standing down")
		return
	}

	span.SetAttributes(attribute.Bool("breaker.confirmed", true))
	log.Println("Anomaly CONFIRMED - sending kill signal to agent")

	killAgent(ctx)

	span.SetAttributes(attribute.String("breaker.outcome", "agent_killed"))
	log.Println("Agent killed successfully. Circuit breaker event logged.")
}

What does this code do?

This is the heart of AgentShield. It creates a circuit_breaker.trip span (visible in SigNoz), records the reason and alert labels as attributes, then follows the two-step pattern: confirm with the API, then kill if confirmed.

Every decision is recorded as a span attribute so the SigNoz dashboard tells the full story of what happened and why.

  • Add the killAgent function below tripCircuitBreaker:
func killAgent(ctx context.Context) {
	// POST to the agent's /kill endpoint to terminate it
	req, _ := http.NewRequestWithContext(ctx, "POST", "http://localhost:8081/kill", nil)
	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		log.Printf("Failed to kill agent: %v", err)
		return
	}
	defer resp.Body.Close()
	body, _ := io.ReadAll(resp.Body)
	log.Printf("Kill response: %s", string(body))
}

What does this code do?

This sends a POST to the agent's /kill endpoint at http://localhost:8081/kill. That triggers the cancellation signal in the agent, gracefully stopping its loop. The response is logged so you can see the confirmation.

  • Finally, add the initShieldTelemetry function at the bottom of the file:
func initShieldTelemetry(ctx context.Context) {
	// Register AgentShield as its own service in SigNoz
	res, _ := resource.New(ctx,
		resource.WithAttributes(
			semconv.ServiceNameKey.String("agentshield"),
			semconv.ServiceVersionKey.String("1.0.0"),
		),
	)

	exporter, _ := otlptracehttp.New(ctx, otlptracehttp.WithInsecure())
	tp := trace.NewTracerProvider(
		trace.WithBatcher(exporter),
		trace.WithResource(res),
	)
	otel.SetTracerProvider(tp)
}

What does this code do?

This sets up OpenTelemetry tracing for AgentShield with service name agentshield. The OTLP HTTP exporter sends spans to SigNoz at port 4318 (the default from your OTEL_EXPORTER_OTLP_ENDPOINT environment variable).

This means the circuit_breaker.trip span appears alongside the agent's spans in the SigNoz Traces Explorer.

  • Save shield/main.go.
  • Run AgentShield by executing this command (you should still be in the shield/ directory):
go run .

You should see this output confirming the server started:

AgentShield listening on :8082, webhook path: /webhook/alert

Seeing compile errors?

If you see undefined function errors, make sure all seven functions are present in shield/main.go in order: main, handleWebhook, confirmWithSigNoz, tripCircuitBreaker, killAgent, and initShieldTelemetry. Also run go mod tidy again if you added any new imports.

help me fix Go compile errors in my AgentShield shield/main.go

Before you test the webhook, what do you think will happen when you send a mock alert with status: "firing"? Will the circuit breaker trip successfully even though the agent is not currently running?

  • Open a new terminal and send a test webhook POST to verify the circuit breaker logic by running:
curl -X POST http://localhost:8082/webhook/alert \
  -H "Content-Type: application/json" \
  -d '{"status":"firing","alerts":[{"status":"firing","labels":{"alertname":"TokenSpike"},"annotations":{}}]}'

Switch back to the terminal running AgentShield. You should see these log lines:

ALERT FIRING: map[alertname:TokenSpike]
Circuit breaker TRIPPED - confirming with SigNoz API...
SIGNOZ_API_KEY not set, skipping confirmation
Anomaly CONFIRMED - sending kill signal to agent
Failed to kill agent: Post "http://localhost:8081/kill": dial tcp ...connection refused

Why does the kill fail?

The agent server is not running right now, so the kill request gets a connection refused error. That is expected.

The important thing is that the full pipeline executed: alert received, circuit breaker tripped, confirmation skipped (no API key set), and kill attempted. In the next step, you will wire this to a live agent and see the kill succeed.

Not seeing the TRIPPED log?

Check that your JSON payload has "status":"firing" both at the top level AND inside the alerts array. The handler checks a.Status == "firing" on each individual alert object, not the top-level status.

help me debug why my AgentShield webhook is not triggering the circuit breaker

✔️ Awesome, I've got everything!

Great. Double check you have saved shield/main.go and confirmed AgentShield starts on port 8082.

ⓧ I'd like to double check the full code

Your shield/main.go is built across seven code chunks above. Scroll up and confirm each piece is present in order:

  • Package declaration, imports, and var tracer line.
  • SigNozAlert struct and main() function.
  • handleWebhook() function.
  • confirmWithSigNoz() function (env check, payload map, then HTTP request).
  • tripCircuitBreaker() function.
  • killAgent() function.
  • initShieldTelemetry() function.

AgentShield is alive and ready to receive alerts. In the next step, you will wire a SigNoz alert rule to this webhook and run the full end-to-end demo where the agent goes rogue and AgentShield kills it automatically.

Wire SigNoz Alert to AgentShield and Demo

You built AgentShield and proved it can kill the agent when triggered. But right now, nothing fires the trigger automatically.

SigNoz is watching the agent's telemetry, and AgentShield is listening for webhooks. The missing piece is the wire between them: an alert rule that detects the token spike and a webhook channel that delivers the signal to AgentShield. Once that wire is in place, the entire pipeline runs without human intervention.

In this step, get ready to:
  • Create a SigNoz webhook notification channel pointing to AgentShield.
  • Create a SigNoz alert rule that fires when token spend spikes.
  • Run the full end-to-end demo and watch the agent die automatically.
Create the webhook notification channel

The alert rule needs a delivery destination. You create the notification channel first so you can attach it when building the alert.

  • Make sure AgentShield is running. In a terminal, start it by running:
cd shield && go run ./shield

You should see: AgentShield listening on :8082, webhook path: /webhook/alert.

  • In your browser, return to the SigNoz UI at http://localhost:8080.
  • Click Settings in the left sidebar.
  • Select Notification Channels under Account Settings.
  • Click New Channel.
  • Enter agentshield-webhook in the Name field.
  • Select Webhook as the channel type.
  • Enter the following URL in the Webhook URL field:
http://host.docker.internal:8082/webhook/alert
  • Leave the Username and Password fields empty.
  • Click the Test button to verify SigNoz can reach AgentShield.

Switch to the terminal where AgentShield is running. You should see a log line indicating it received the test payload.

Why host.docker.internal?

SigNoz runs inside Docker containers. To reach AgentShield on your host machine, the containers use the special hostname host.docker.internal. This resolves to your host's IP from inside Docker.

On Linux, this hostname may not resolve by default. Use your host's Docker bridge IP (typically 172.17.0.1) instead.

Test button failing?

Make sure AgentShield is running (go run ./shield in a terminal). The webhook receiver must be listening on port 8082 before SigNoz can reach it.

If you are on Linux and host.docker.internal does not resolve, replace it with 172.17.0.1 in the Webhook URL field.

Help me troubleshoot why SigNoz cannot reach my webhook endpoint on the host machine.

  • Once the test succeeds, click Save to create the channel.
Create the SigNoz alert rule

Now you wire the detection side. This alert rule watches the agent.token.total metric and fires when token consumption spikes beyond what a normal agent run produces.

  • Click Alerts in the left sidebar.
  • Click the + New Alert button in the top-right corner.
  • Select Metric Based Alerts as the alert type.

In Step 1 (Define the Metric):

  • In the Metric dropdown, search for and select agent.token.total.
  • Set Aggregate within time series to Sum.

In Step 2 (Define Alert Conditions):

  • Set the condition operator to Above with a threshold value of 500.
  • Select In Total as the match type.
  • Set the Evaluation Window to Last 5 minutes.
  • Under Advanced Options, set the evaluation frequency to 15 seconds for fast demo response.
  • In the Notification Channels dropdown, select the agentshield-webhook channel you just created.

In Step 3 (Notification Settings):

  • Enter Runaway Agent Token Spike as the Alert Name.
  • Set the severity to Critical.
  • Click Save to activate the alert rule.

You should see your new alert listed on the Alert Rules tab with an enabled status.

Why In Total and 15-second evaluation?

The In Total pattern sums all token increments within the evaluation window. A rogue agent calling Gemini every 3 seconds easily blows past 500 tokens in under 30 seconds.

The 15-second evaluation interval means SigNoz checks the condition four times per minute. In production you would use a longer interval to reduce noise. For this demo you want the fastest possible detection.

Run the full demo

Everything is wired. Time to prove the pipeline works end-to-end without any human touching the kill switch.

  • Open three terminal windows. You need one for AgentShield, one for the agent, and one for issuing commands.
  • In Terminal 1, start AgentShield (if it is not already running from earlier) by running:
cd shield && go run ./shield

You should see: AgentShield listening on :8082, webhook path: /webhook/alert.

  • In Terminal 2, start the research agent by running:
cd agent && go run ./agent

You should see: Research Agent listening on :8081.

  • In Terminal 3, trigger the agent in rogue mode by running:
curl http://localhost:8081/start?topic=climate+change&mode=rogue

The agent starts its infinite retry loop, calling Gemini every 3 seconds. Tokens climb with every iteration.

Before you do anything else: do you think the agent will keep running forever, or will something stop it within the next minute?

  • Wait 60 seconds. Do not touch anything. Watch Terminal 1 (AgentShield) and Terminal 2 (the agent).

Within about 45-60 seconds, you see the full pipeline fire in sequence across your terminals.

  • Terminal 1 (AgentShield) prints ALERT FIRING, followed by Circuit breaker TRIPPED, then Anomaly CONFIRMED, then Agent killed successfully.
  • Terminal 2 (the agent) stops printing new iteration logs. The loop ends.
  • Confirm the agent is dead by running this in Terminal 3:
curl http://localhost:8081/status

You should see {"status":"killed"}. The agent was terminated by AgentShield without any human intervention.

What just happened?

The agent emitted agent.token.total metrics via OpenTelemetry to SigNoz. SigNoz evaluated the metric every 15 seconds and detected the threshold breach. It fired a webhook to AgentShield in Prometheus Alertmanager format.

AgentShield confirmed the anomaly via the SigNoz API, then sent a POST to the agent's /kill endpoint. The entire pipeline ran autonomously. This is the difference between observability (watching things burn) and enforcement (stopping them automatically).

  • Open the SigNoz UI and click Traces in the sidebar. Filter by service agentshield. You should see a circuit_breaker.trip span with attributes showing the reason and outcome.
  • Click Alerts in the sidebar, then select the Triggered Alerts tab. You should see your Runaway Agent Token Spike alert in the fired state.

Agent still running after 90 seconds?

Check that the alert rule is enabled (not disabled) on the Alert Rules page.

Verify the metric agent.token.total appears in SigNoz Metrics Explorer. If it does not, the agent's telemetry is not reaching SigNoz. Confirm OTEL_EXPORTER_OTLP_ENDPOINT is set to http://localhost:4318 in the terminal where you started the agent.

If the alert fires but AgentShield does not receive it, confirm the webhook URL uses host.docker.internal (not localhost). SigNoz runs inside Docker and cannot reach your host via localhost.

Help me debug why my SigNoz alert is not triggering the webhook to AgentShield.

The full pipeline works. An AI agent went rogue, observability detected it, and your circuit breaker killed it automatically. No human in the loop. This is what you demo at the hackathon.

Secret mission

Cost Velocity Detection

Your circuit breaker reacts to total token thresholds. But a rogue agent and a normal agent can use the same number of tokens. The difference is how fast they burn through them. Can you make AgentShield detect the rate of spend and trip the breaker before the threshold alert even fires?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally using Docker containers. There are no cloud resources or paid services, so leaving everything running costs nothing beyond the memory Docker uses on your machine.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Docker containers running the SigNoz stack (ClickHouse, OTel Collector, SigNoz UI).
  • Local project files in the agent/ and shield/ directories, plus casting.yaml.
  • SigNoz alert rule and webhook notification channel (stored inside the SigNoz containers).
  • Environment variables: GEMINI_API_KEY, OTEL_EXPORTER_OTLP_ENDPOINT, SIGNOZ_API_KEY.

Keep everything running

No action needed. Choose this if you are still actively building or polishing your dashboard for the hackathon submission. The SigNoz stack continues running in Docker and your project files stay on disk.

  • SigNoz remains accessible at http://localhost:8080.
  • Your alert rules, webhook channels, and stored traces persist in the Docker volumes.
  • The foundryctl binary remains on your machine (installed via the Foundry install script). SigNoz does not document a dedicated uninstall command for it.

Pause - I'll come back to this later

Shut down the SigNoz containers to free up memory while preserving all your data. Your traces, alert rules, and dashboards remain intact for next time.

  • Stop the SigNoz containers by running:
cd pours/deployment && docker compose stop
  • To start everything back up later, run:
cd pours/deployment && docker compose start
  • Your project files in agent/ and shield/ stay on disk untouched.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes the SigNoz containers, all stored traces and dashboards, and your project source code.

  • Stop and remove all SigNoz containers and volumes by running:
cd pours/deployment && docker compose down -v
  • Remove the project directory from your Desktop by running:
cd ~/Desktop
rm -rf agentshield

On Windows?

Use this command instead to remove the project folder:

cd ~\Desktop
rmdir /s /q agentshield
  • Unset the environment variables from your current terminal session by running:
unset GEMINI_API_KEY
unset OTEL_EXPORTER_OTLP_ENDPOINT
unset SIGNOZ_API_KEY

Permanent environment variables

If you added these exports to your shell profile (.bashrc, .zshrc, etc.), open that file and remove the relevant lines as well.

Nice Work!

Nice Work!

Nice work. You just built AgentShield, a fully autonomous circuit breaker that detects runaway AI agents using SigNoz observability data and kills them before costs explode.

You've learned how to:

  • Build a multi-step AI research agent in Go with full OpenTelemetry instrumentation sending traces, metrics, and logs to SigNoz.
  • Demonstrate the real $47K agent loop problem and prove that observability alone does not stop runaway costs without an external enforcement layer.
  • Build AgentShield, an autonomous circuit breaker that receives SigNoz webhook alerts, confirms anomalies via the SigNoz API, and automatically terminates rogue agents without human intervention.
  • Secret Mission: Added proactive cost velocity detection that trips the breaker based on token spend rate, making AgentShield respond even faster than waiting for threshold alerts.

Ready to quiz yourself?