Build an LLM Evaluation Engine in Go

Build a Go evaluation engine that scores LLM traces with gated PASS/FAIL decisions.

Introduction

30 Second Summary

When you deploy an LLM-powered feature, how do you know it's actually working well? A single bad response can erode user trust, but manually reviewing outputs doesn't scale past a handful of requests.

In this project, you will build a production-grade evaluation engine in Go that automatically judges LLM outputs for quality. You'll wire up a serverless pipeline on AWS that validates execution traces, runs deterministic quality gates, and uses an LLM judge to score responses across multiple iterations.

What You'll Build

You submit a raw LLM execution trace via HTTP, and within seconds the engine tells you whether that output is production-ready, should fail, or needs to be flagged for human review.

By the end of this project, you'll have:

  • A validated ingestion API that rejects malformed traces with detailed JSON Schema violation errors, so garbage data never enters your pipeline.
  • Five deterministic evaluation gates that instantly catch tool timeouts, poor retrieval quality, and budget overruns without spending a cent on LLM calls.
  • An LLM-as-a-judge system powered by Amazon Nova Pro that runs 3 scoring iterations per trace, computes variance, and flags unreliable judgments for human review instead of trusting a single roll of the dice.
  • A complete Kinesis-to-Lambda-to-DynamoDB data pipeline designed for 1M req/sec scale, tested at thousands of concurrent requests.
  • Secret Mission: A Bedrock Agent that queries your evaluation data in natural language to surface insights like pass rates by prompt version.

Are there any prerequisites?

This project assumes intermediate Go experience (structs, interfaces, goroutines) and a configured AWS CLI with credentials. You should also have basic familiarity with DynamoDB and Lambda concepts.

You'll need Go 1.24+ installed, an AWS account with Bedrock access enabled in us-east-1, and a WSL terminal (Windows) or native Linux/macOS terminal.

Before We Start

Production LLM systems generate thousands of execution traces every minute. Each trace contains tool calls, retrieval results, prompt versions, and a final response. Without a structured evaluation pipeline, you have no way to know whether your LLM is producing quality results or silently degrading.

In this step, you will verify your Go and AWS environment, scaffold the project directory, and install every dependency the evaluation engine needs. This gives you a solid foundation before you write any evaluation logic.

In this step, get ready to:
  • Verify your Go version is 1.24+ and confirm AWS CLI connectivity in us-east-1.
  • Initialize a Go module and install all AWS SDK and validation dependencies.
  • Create the project directory structure and install the Lambda packaging tool.
Verify your Go version and AWS credentials

The evaluation engine requires Go 1.24 or higher because the AWS SDK v2 depends on features introduced in that version. Your AWS CLI also needs to be pointing at us-east-1 where Amazon Bedrock is available.

  • Check your Go version by running this command:
go version

You should see go1.24 or higher in the output.

Seeing a version lower than 1.24?

Visit the official Go downloads page at https://go.dev/dl/ and install the latest version. After installing, close and reopen your terminal, then run go version again.

help me upgrade Go to version 1.24 or higher on my system

  • Confirm your AWS CLI is configured and pointing at the right region by running:
aws sts get-caller-identity

You should see a JSON response containing your Account ID, UserId, and Arn.

  • Verify your region is set to us-east-1 by running:
aws configure get region

The output should read us-east-1. If it shows a different region, run aws configure set region us-east-1 to fix it.

Getting 'Unable to locate credentials'?

Your AWS CLI is not configured yet. Run aws configure and enter your Access Key ID, Secret Access Key, and set the default region to us-east-1.

help me configure my AWS CLI credentials for us-east-1

Initialize the Go module and install dependencies

Every Go project starts with a module that tracks its dependencies. You will initialize one and then pull in the AWS SDK packages for Kinesis, DynamoDB, and Bedrock, plus the JSON Schema validator and Lambda runtime library.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir llm-eval-engine
cd llm-eval-engine
  • Initialize the Go module by running:
go mod init llm-eval-engine

You should see a message confirming go.mod was created.

  • Install all project dependencies by running these commands:
go get github.com/aws/aws-lambda-go@v1.47.0
go get github.com/aws/aws-sdk-go-v2@v1.30.0
go get github.com/aws/aws-sdk-go-v2/config@v1.27.0
go get github.com/aws/aws-sdk-go-v2/service/kinesis@v1.29.0
go get github.com/aws/aws-sdk-go-v2/service/dynamodb@v1.34.0
go get github.com/aws/aws-sdk-go-v2/service/bedrockruntime@v1.13.0
go get github.com/santhosh-tekuri/jsonschema/v6@v6.0.1

What do these packages do?

  • aws-lambda-go is the runtime library that lets your Go code run inside AWS Lambda.
  • aws-sdk-go-v2 packages provide the clients for Kinesis (event streaming), DynamoDB (storage), and Bedrock Runtime (LLM judge calls).
  • jsonschema/v6 validates incoming trace payloads against a JSON Schema (draft 2020-12) before they enter the pipeline.
  • Confirm the dependencies are recorded by checking your go.mod file:
cat go.mod

You should see all seven packages listed under the require block.

✔️ I see all seven packages listed

Your dependencies are locked in. On to the directory structure.

ⓧ Some packages are missing

Re-run the specific go get command for the missing package. Check for typos in the module path. Each path is case-sensitive.

help me figure out why go get is not adding a package to my go.mod

Create the project directory structure and install build-lambda-zip

The evaluation engine follows a standard Go project layout. The cmd/ directories hold entry points (the API server and the Lambda evaluator). The internal/ directories hold the core logic that only this project imports.

  • Create all project directories at once by running:
mkdir -p cmd/api cmd/evaluator internal/models internal/validator internal/evaluator schemas
  • Confirm the structure exists by running:
find . -type d

You should see all six directories listed alongside the root . directory.

What is each directory for?

  • cmd/api/ holds the HTTP ingestion server that accepts evaluation traces.
  • cmd/evaluator/ holds the Lambda function that processes traces through evaluation gates.
  • internal/models/ defines the Go structs for traces and evaluation decisions.
  • internal/validator/ handles JSON Schema validation of incoming payloads.
  • internal/evaluator/ contains the deterministic gates and LLM-as-a-judge logic.
  • schemas/ stores the JSON Schema definition files.
  • Install the Lambda packaging tool by running:
go install github.com/aws/aws-lambda-go/cmd/build-lambda-zip@latest
  • Verify it installed correctly by running:
build-lambda-zip --help

You should see usage information printed to the terminal. This tool packages your compiled Go binary into a zip file formatted for AWS Lambda.

Command not found for build-lambda-zip?

Your Go bin directory may not be in your PATH. Run export PATH=$PATH:$(go env GOPATH)/bin and try again. To make this permanent, add that line to your ~/.bashrc or ~/.zshrc file.

help me add my Go bin directory to PATH on WSL

Before you move on, take a moment. What do you think the full go.mod file should look like at this point?

  • Double check by running cat go.mod one more time.

✔️ Awesome, I've got everything!

Great. Make sure you have saved your progress and your terminal is still inside the llm-eval-engine folder.

ⓧ I'd like to double check the full code

Your go.mod should look like this (exact minor versions of indirect dependencies may vary):

module llm-eval-engine

go 1.24

require (
	github.com/aws/aws-lambda-go v1.47.0
	github.com/aws/aws-sdk-go-v2 v1.30.0
	github.com/aws/aws-sdk-go-v2/config v1.27.0
	github.com/aws/aws-sdk-go-v2/service/kinesis v1.29.0
	github.com/aws/aws-sdk-go-v2/service/dynamodb v1.34.0
	github.com/aws/aws-sdk-go-v2/service/bedrockruntime v1.13.0
	github.com/santhosh-tekuri/jsonschema/v6 v6.0.1
)

Your environment is ready and every dependency is locked in. Next up, you will build the evaluation data model and the ingestion API that validates incoming traces against a JSON Schema.

Build the Eval Data Model and Ingestion API

Your project structure and dependencies are ready. Now you need a way to accept LLM execution traces and reject garbage data before it enters the pipeline.

In this step, you will define the Go data model that represents an evaluation trace, write a JSON Schema that enforces structural correctness, and build an HTTP endpoint that validates incoming payloads against that schema.

In this step, get ready to:
  • Define the complete eval trace data model in Go structs.
  • Write a JSON Schema (draft 2020-12) that enforces required fields, types, and enum constraints.
  • Build an HTTP ingestion server that rejects invalid traces with detailed error messages.
Define the eval trace data model

Every LLM execution trace carries specific pieces of data: which prompt version was used, which model ran, what documents were retrieved, which tools were called, and how long it all took. You need Go structs that mirror this shape exactly.

  • Create the file internal/models/trace.go and add the retrieval and tool call structs by pasting this code:
package models

import "time"

// RetrievalChunk represents a single chunk returned from vector search
type RetrievalChunk struct {
	ChunkID    string  `json:"chunk_id"`
	Content    string  `json:"content"`
	Score      float64 `json:"score"`
}

// RetrievalSnapshot captures a full retrieval operation
type RetrievalSnapshot struct {
	VectorStoreID string           `json:"vector_store_id"`
	Query         string           `json:"query"`
	Chunks        []RetrievalChunk `json:"chunks"`
}

// ToolCallTrace records one tool invocation with timing
type ToolCallTrace struct {
	Name      string      `json:"name"`
	Input     interface{} `json:"input"`
	Output    interface{} `json:"output"`
	LatencyMs int64       `json:"latency_ms"`
	TimedOut  bool        `json:"timed_out"`
}

What does this code do?

Each struct maps to a distinct part of an LLM execution trace:

  • RetrievalChunk holds a single document fragment returned from a vector store search, along with its similarity score (0 to 1).
  • RetrievalSnapshot groups all chunks from one retrieval query so you can evaluate the full context window.
  • ToolCallTrace records a single tool invocation with its latency and whether it timed out. This powers the deterministic timeout gate later.
  • Continue in the same file, adding the execution step, token usage, and main trace structs directly below the code you just pasted:
// ExecutionStep tracks one step in the LLM pipeline
type ExecutionStep struct {
	StepIndex   int    `json:"step_index"`
	Description string `json:"description"`
	Status      string `json:"status"`
}

// TokenUsage records input and output token counts
type TokenUsage struct {
	InputTokens  int `json:"input_tokens"`
	OutputTokens int `json:"output_tokens"`
}

// EvalTrace is the top-level evaluation payload
type EvalTrace struct {
	TraceID           string              `json:"trace_id,omitempty"`
	PromptVersion     string              `json:"prompt_version"`
	ModelVersion      string              `json:"model_version"`
	RetrievalSnapshots []RetrievalSnapshot `json:"retrieval_snapshots"`
	ToolCallTraces    []ToolCallTrace     `json:"tool_call_traces"`
	ExecutionSteps    []ExecutionStep     `json:"execution_steps"`
	FinalResponse     string              `json:"final_response"`
	RunGroupID        string              `json:"run_group_id"`
	TokensUsed        TokenUsage          `json:"tokens_used"`
	TotalLatencyMs    int64               `json:"total_latency_ms"`
	Timestamp         time.Time           `json:"timestamp"`
}

What does this code do?

EvalTrace is the root struct that combines everything. It captures the prompt version, model version, retrieval context, tool calls, execution steps, and the final LLM response.

RunGroupID groups multiple evaluation runs of the same trace together. This enables the variance tracking you will build in Step 5.

  • Save internal/models/trace.go.
  • Verify the code compiles by running:
go build ./...

You should see no output, which means the build succeeded with no errors.

Seeing a compile error?

  • Check that the package declaration at the top of the file is package models (not package main).
  • Make sure import "time" is present. The Timestamp field uses time.Time.
  • Confirm you are running the command from the llm-eval-engine project root.

help me fix a Go compile error in my trace model

Write the JSON Schema for trace validation

The Go structs define the shape in code, but you also need a JSON Schema file on disk. The schema validator library compiles this file at startup and uses it to reject any incoming payload that violates the contract before it touches your data pipeline.

  • Create the file schemas/eval_trace.json and add the schema header plus retrieval snapshot definition:
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["prompt_version", "model_version", "retrieval_snapshots", "tool_call_traces", "execution_steps", "final_response", "run_group_id", "tokens_used", "total_latency_ms", "timestamp"],
  "properties": {
    "prompt_version": {"type": "string"},
    "model_version": {"type": "string"},
    "retrieval_snapshots": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["vector_store_id", "query", "chunks"],
        "properties": {
          "vector_store_id": {"type": "string"},
          "query": {"type": "string"},
          "chunks": {
            "type": "array",
            "items": {
              "type": "object",
              "required": ["chunk_id", "content", "score"],
              "properties": {
                "chunk_id": {"type": "string"},
                "content": {"type": "string"},
                "score": {"type": "number", "minimum": 0, "maximum": 1}
              }
            }
          }
        }
      }
    },

What does this section define?

  • The $schema field pins the schema to draft 2020-12, the latest stable version.
  • The required array ensures every trace includes all essential fields. Missing any one produces a validation error.
  • The score field enforces minimum: 0 and maximum: 1 so similarity scores always fall in a normalised range.
  • Continue in the same file, adding the tool call and execution step definitions directly below the last comma:
    "tool_call_traces": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["name", "latency_ms", "timed_out"],
        "properties": {
          "name": {"type": "string"},
          "latency_ms": {"type": "integer", "minimum": 0},
          "timed_out": {"type": "boolean"}
        }
      }
    },
    "execution_steps": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["step_index", "description", "status"],
        "properties": {
          "step_index": {"type": "integer"},
          "description": {"type": "string"},
          "status": {"type": "string", "enum": ["SUCCESS", "FAILURE", "SKIPPED"]}
        }
      }
    },

What do these constraints enforce?

  • tool_call_traces requires name, latency_ms, and timed_out on every tool call entry. The timeout gate in Step 4 relies on these values being present.
  • The status field uses an enum constraint to only allow SUCCESS, FAILURE, or SKIPPED. Any other value triggers a schema violation.
  • Complete the schema file by adding the remaining field definitions and closing braces:
    "final_response": {"type": "string"},
    "run_group_id": {"type": "string"},
    "tokens_used": {
      "type": "object",
      "required": ["input_tokens", "output_tokens"],
      "properties": {
        "input_tokens": {"type": "integer", "minimum": 0},
        "output_tokens": {"type": "integer", "minimum": 0}
      }
    },
    "total_latency_ms": {"type": "integer", "minimum": 0},
    "timestamp": {"type": "string", "format": "date-time"}
  }
}
  • Save schemas/eval_trace.json.
  • Verify the file is valid JSON by running:
python3 -c "import json; json.load(open('schemas/eval_trace.json')); print('valid JSON')"

You should see valid JSON printed. If you see a JSONDecodeError, check for missing commas or mismatched braces.

Seeing a JSONDecodeError?

  • The most common cause is a missing comma after the retrieval_snapshots closing brace or the execution_steps closing brace.
  • Check that the final closing } matches the opening { at the top of the file. You should have exactly two closing braces at the end: one for properties and one for the root object.

help me fix a JSON syntax error in my schema file

Build the HTTP ingestion server with schema validation

You have the data model and the schema on disk. Now you need an HTTP server that accepts POST requests, validates the body against the schema, and either rejects invalid traces with specific error details or accepts valid ones with a generated trace ID.

  • Install the UUID package that the server uses to generate trace IDs:
go get github.com/google/uuid
  • Create the file cmd/api/main.go and add the imports, schema loading, and server entry point:
package main

import (
	"encoding/json"
	"fmt"
	"io"
	"log"
	"net/http"
	"strings"

	"github.com/google/uuid"
	"github.com/santhosh-tekuri/jsonschema/v6"
	"llm-eval-engine/internal/models"
)

// schema holds the compiled JSON Schema for trace validation
var schema *jsonschema.Schema

// init compiles the schema at startup so every request uses the same instance
func init() {
	compiler := jsonschema.NewCompiler()
	var err error
	schema, err = compiler.Compile("schemas/eval_trace.json")
	if err != nil {
		log.Fatalf("failed to compile schema: %v", err)
	}
}

func main() {
	http.HandleFunc("/traces", handleTrace)
	log.Println("API server listening on :8080")
	log.Fatal(http.ListenAndServe(":8080", nil))
}

What does this code do?

  • The init() function runs before main() and compiles your eval_trace.json schema once. If the schema file has errors, the server crashes immediately with a clear message instead of failing silently on the first request.
  • The main() function registers the /traces route and starts listening on port 8080.
  • Add the request handler function directly above the func main() line in the same file:
func handleTrace(w http.ResponseWriter, r *http.Request) {
	if r.Method != http.MethodPost {
		http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
		return
	}
	body, err := io.ReadAll(r.Body)
	if err != nil {
		http.Error(w, "failed to read body", http.StatusBadRequest)
		return
	}
	// Parse into generic interface for schema validation
	var raw interface{}
	if err := json.Unmarshal(body, &raw); err != nil {
		http.Error(w, "invalid JSON", http.StatusBadRequest)
		return
	}
	// Validate against the compiled JSON Schema
	if err := schema.Validate(raw); err != nil {
		http.Error(w, fmt.Sprintf("schema validation failed: %v", err), http.StatusBadRequest)
		return
	}
	// Deserialize into typed struct for downstream use
	var trace models.EvalTrace
	if err := json.Unmarshal(body, &trace); err != nil {
		http.Error(w, "failed to parse trace", http.StatusBadRequest)
		return
	}
	trace.TraceID = strings.ReplaceAll(uuid.New().String(), "-", "")
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusAccepted)
	json.NewEncoder(w).Encode(map[string]string{"trace_id": trace.TraceID, "status": "accepted"})
}

What does this handler do?

The handler applies a three-stage validation pipeline:

  • First, it parses the raw body into a generic interface{}. This lets the schema validator inspect the structure without needing typed Go structs.
  • Second, schema.Validate(raw) checks the payload against every rule in your JSON Schema. If any field is missing, the wrong type, or outside the allowed enum, it returns a detailed error listing every violation.
  • Third, it unmarshals into the typed EvalTrace struct, generates a UUID trace ID, and returns 202 Accepted with the ID.
  • Save cmd/api/main.go.
  • Start the server by running:
go run cmd/api/main.go

You should see API server listening on :8080 printed to the terminal. The server is now running and blocking that terminal session.

Server crashes on startup?

  • If you see failed to compile schema, your schemas/eval_trace.json file has a syntax error. Run the python3 JSON check from the previous substep to find it.
  • If you see an import error for github.com/google/uuid, confirm you ran go get github.com/google/uuid from the project root.
  • If port 8080 is already in use, stop whatever process is using it or change the port in main.go.

help me fix my Go API server that crashes on startup

  • Open a second terminal tab. Before you test with curl, what do you think will happen if you send a payload missing a required field?

Test with a valid trace by running this curl command:

curl -s -X POST http://localhost:8080/traces \
  -H "Content-Type: application/json" \
  -d '{
    "prompt_version": "v1.0",
    "model_version": "gpt-4-0613",
    "retrieval_snapshots": [{"vector_store_id": "vs-1", "query": "test", "chunks": [{"chunk_id": "c1", "content": "hello", "score": 0.9}]}],
    "tool_call_traces": [{"name": "search", "input": {}, "output": {}, "latency_ms": 120, "timed_out": false}],
    "execution_steps": [{"step_index": 1, "description": "retrieve", "status": "SUCCESS"}],
    "final_response": "The answer is 42.",
    "run_group_id": "rg-001",
    "tokens_used": {"input_tokens": 500, "output_tokens": 100},
    "total_latency_ms": 1500,
    "timestamp": "2026-08-04T10:00:00Z"
  }'

You should see a JSON response with a trace_id and "status": "accepted".

  • Now test with a malformed trace. Send a payload missing the timestamp field:
curl -s -X POST http://localhost:8080/traces \
  -H "Content-Type: application/json" \
  -d '{"prompt_version": "v1.0", "model_version": "gpt-4"}'

You should see a 400 Bad Request response starting with schema validation failed followed by a list of the specific fields that are missing.

Why is this validation so important?

The schema gate rejects garbage at the door before it ever enters your Kinesis stream or DynamoDB table. Every trace that makes it past this point is guaranteed to have the fields your deterministic gates and LLM-as-a-judge depend on.

Without this gate, a missing latency_ms field on a tool call would silently break your timeout evaluator downstream. Failing early with a clear message saves hours of debugging at scale.

✔️ Awesome, I've got everything!

Great. Double check you have saved both internal/models/trace.go and cmd/api/main.go, and that your server is still running in the first terminal.

ⓧ I'd like to double check the full code

You built three files in this step. Scroll up and confirm each is present in order:

  • internal/models/trace.go with the RetrievalChunk, RetrievalSnapshot, ToolCallTrace, ExecutionStep, TokenUsage, and EvalTrace structs.
  • schemas/eval_trace.json with all three JSON chunks combined into one valid file (header through closing brace).
  • cmd/api/main.go with the imports, init(), handleTrace(), and main() functions.

Your ingestion endpoint is live and rejecting bad data. Next up, you will wire this server to Amazon Kinesis so that accepted traces flow into a durable stream for processing by your Lambda evaluator.

Implement Deterministic Evaluation Gates

Your traces arrive in DynamoDB but just sit there with status PENDING_EVALUATION. There is no quality judgment happening yet.

Deterministic gates are the fast, zero-variance evaluations that catch obvious failures: timed-out tools, poor retrieval quality, blown token budgets. They run in milliseconds and produce binary PASS/FAIL with no ambiguity. They also serve as a cost gate: if any deterministic check fails, you skip the expensive LLM-as-a-judge entirely.

In this step, get ready to:
  • Implement five gate functions that evaluate trace quality in milliseconds.
  • Build a composite evaluator that runs all gates and writes decisions to DynamoDB.
  • Configure thresholds via environment variables and redeploy the Lambda.
Add gate result models to the data layer

Each gate produces a structured result record containing its name, whether it passed, the evidence explaining why, and the threshold it used. You need these types defined before writing the gate functions themselves.

  • Open internal/models/trace.go and add the following two structs at the bottom of the file, below the existing EvalTrace struct:
type GateResult struct {
	GateName      string `json:"gate_name"`
	Passed        bool   `json:"passed"`
	Evidence      string `json:"evidence"`
	ThresholdUsed string `json:"threshold_used"`
}

type EvalDecision struct {
	TraceID        string       `json:"trace_id"`
	EvalTimestamp  string       `json:"eval_timestamp"`
	OverallStatus  string       `json:"overall_status"`
	GateResults    []GateResult `json:"gate_results"`
	PromptVersion  string       `json:"prompt_version"`
	ModelVersion   string       `json:"model_version"`
}

What do these structs represent?

GateResult captures what a single gate concluded: its name, pass/fail boolean, the specific evidence (like which tool timed out), and the threshold it checked against.

EvalDecision is the composite record that DynamoDB stores. It contains the overall PASS/FAIL plus all individual gate results for a full audit trail.

  • Save internal/models/trace.go.
Create the evaluation gates file with the first two gates

The gates live in a dedicated package so they can be tested independently of the Lambda handler. The first gate checks that the LLM actually produced a response. The second catches tools that exceeded their timeout threshold.

  • Create a new file at internal/evaluator/gates.go and add the following package declaration, imports, config struct, and first two gate functions:
package evaluator

import (
	"fmt"
	"llm-eval-engine/internal/models"
)

// GateConfig holds configurable thresholds for all gates
type GateConfig struct {
	ToolTimeoutMs     int64
	RetrievalMinScore float64
	MaxLatencyMs      int64
	MaxOutputTokens   int
}

// ResponseSchemaCompliance checks that the LLM produced a response
func ResponseSchemaCompliance(trace models.EvalTrace) models.GateResult {
	if trace.FinalResponse == "" {
		return models.GateResult{GateName: "RESPONSE_SCHEMA", Passed: false, Evidence: "final_response is empty", ThresholdUsed: "non-empty"}
	}
	return models.GateResult{GateName: "RESPONSE_SCHEMA", Passed: true, Evidence: "response present", ThresholdUsed: "non-empty"}
}

What does this code do?

GateConfig centralizes all threshold values so you can tune gates without changing logic. Each gate receives this config and checks the trace against the relevant threshold.

ResponseSchemaCompliance is the simplest gate. It fails if the LLM returned an empty string. A missing response means the model produced nothing useful.

  • Now add the ToolTimeoutGate function below ResponseSchemaCompliance in the same file:
// ToolTimeoutGate fails if any tool exceeded its timeout threshold
func ToolTimeoutGate(trace models.EvalTrace, cfg GateConfig) models.GateResult {
	for _, tool := range trace.ToolCallTraces {
		if tool.TimedOut || tool.LatencyMs > cfg.ToolTimeoutMs {
			return models.GateResult{
				GateName:      "TOOL_TIMEOUT",
				Passed:        false,
				Evidence:      fmt.Sprintf("tool '%s' timed out or exceeded %dms (actual: %dms)", tool.Name, cfg.ToolTimeoutMs, tool.LatencyMs),
				ThresholdUsed: fmt.Sprintf("%dms", cfg.ToolTimeoutMs),
			}
		}
	}
	return models.GateResult{GateName: "TOOL_TIMEOUT", Passed: true, Evidence: "all tools within timeout", ThresholdUsed: fmt.Sprintf("%dms", cfg.ToolTimeoutMs)}
}

Why does this gate check both fields?

The gate checks tool.TimedOut (a boolean the caller sets) and tool.LatencyMs > cfg.ToolTimeoutMs (the raw duration). This catches tools that were explicitly flagged as timed out AND tools that ran slow but were not flagged. The evidence string names the specific tool that failed.

  • Save internal/evaluator/gates.go.
  • Confirm the new package compiles by running:
go build ./...

You should see no output. No output from go build means everything compiled successfully.

Seeing a compilation error?

Check that internal/evaluator/gates.go uses package evaluator (not package main). Verify the import path is llm-eval-engine/internal/models and that the GateResult struct you added to trace.go is exported (starts with uppercase).

Why won't my evaluator package compile?

Add the retrieval quality, latency, and token budget gates

The remaining three gates check retrieval chunk quality, total execution latency, and output token usage. Each produces a detailed evidence string when it fails.

  • Add the RetrievalQualityGate function below ToolTimeoutGate in internal/evaluator/gates.go:
// RetrievalQualityGate fails if any retrieved chunk falls below the score threshold
func RetrievalQualityGate(trace models.EvalTrace, cfg GateConfig) models.GateResult {
	for _, snap := range trace.RetrievalSnapshots {
		for _, chunk := range snap.Chunks {
			if chunk.Score < cfg.RetrievalMinScore {
				return models.GateResult{
					GateName:      "RETRIEVAL_QUALITY",
					Passed:        false,
					Evidence:      fmt.Sprintf("chunk '%s' score %.3f below threshold %.3f", chunk.ChunkID, chunk.Score, cfg.RetrievalMinScore),
					ThresholdUsed: fmt.Sprintf("%.3f", cfg.RetrievalMinScore),
				}
			}
		}
	}
	return models.GateResult{GateName: "RETRIEVAL_QUALITY", Passed: true, Evidence: "all chunks above threshold", ThresholdUsed: fmt.Sprintf("%.3f", cfg.RetrievalMinScore)}
}

What does this gate catch?

If any single retrieval chunk has a similarity score below the configured minimum, the entire trace fails this gate. Low-score chunks mean the LLM was fed irrelevant context, making its response unreliable regardless of how well it reads.

  • Add the LatencySLAGate and TokenBudgetGate functions below RetrievalQualityGate:
// LatencySLAGate fails if total execution time exceeds the latency budget
func LatencySLAGate(trace models.EvalTrace, cfg GateConfig) models.GateResult {
	if trace.TotalLatencyMs > cfg.MaxLatencyMs {
		return models.GateResult{
			GateName:      "LATENCY_SLA",
			Passed:        false,
			Evidence:      fmt.Sprintf("total latency %dms exceeds SLA %dms", trace.TotalLatencyMs, cfg.MaxLatencyMs),
			ThresholdUsed: fmt.Sprintf("%dms", cfg.MaxLatencyMs),
		}
	}
	return models.GateResult{GateName: "LATENCY_SLA", Passed: true, Evidence: fmt.Sprintf("latency %dms within SLA", trace.TotalLatencyMs), ThresholdUsed: fmt.Sprintf("%dms", cfg.MaxLatencyMs)}
}

// TokenBudgetGate fails if output tokens exceed the configured limit
func TokenBudgetGate(trace models.EvalTrace, cfg GateConfig) models.GateResult {
	if trace.TokensUsed.OutputTokens > cfg.MaxOutputTokens {
		return models.GateResult{
			GateName:      "TOKEN_BUDGET",
			Passed:        false,
			Evidence:      fmt.Sprintf("output tokens %d exceeds limit %d", trace.TokensUsed.OutputTokens, cfg.MaxOutputTokens),
			ThresholdUsed: fmt.Sprintf("%d tokens", cfg.MaxOutputTokens),
		}
	}
	return models.GateResult{GateName: "TOKEN_BUDGET", Passed: true, Evidence: fmt.Sprintf("%d output tokens within budget", trace.TokensUsed.OutputTokens), ThresholdUsed: fmt.Sprintf("%d tokens", cfg.MaxOutputTokens)}
}

What do these gates protect against?

LatencySLAGate catches traces where the total execution blew past the SLA budget. A 10-second response time on a user-facing LLM call is unacceptable in production.

TokenBudgetGate catches responses that ran away with token generation. Excessive output tokens signal prompt drift or hallucination spirals, and they cost more money.

  • Save internal/evaluator/gates.go.
  • Verify all five gates compile by running:
go build ./...

No output means all five gate functions compiled cleanly.

Getting an unused import error?

Make sure the "fmt" import is present in your import block. Every gate function uses fmt.Sprintf for the evidence and threshold strings.

Help me fix my gates.go compilation error.

Build the composite gate evaluator

Individual gates are useful but the Lambda needs a single function that runs all five, collects results, and returns an overall verdict. If any gate fails, the composite verdict is FAIL.

  • Add the RunDeterministicGates function at the bottom of internal/evaluator/gates.go:
// RunDeterministicGates executes all five gates and returns results with overall verdict
func RunDeterministicGates(trace models.EvalTrace, cfg GateConfig) ([]models.GateResult, bool) {
	results := []models.GateResult{
		ResponseSchemaCompliance(trace),
		ToolTimeoutGate(trace, cfg),
		RetrievalQualityGate(trace, cfg),
		LatencySLAGate(trace, cfg),
		TokenBudgetGate(trace, cfg),
	}
	allPassed := true
	for _, r := range results {
		if !r.Passed {
			allPassed = false
			break
		}
	}
	return results, allPassed
}

Why return both results and the boolean?

The boolean gives a quick overall verdict (PASS or FAIL). The full results slice provides the audit trail: every gate's name, evidence, and threshold, even the ones that passed. This makes debugging failed traces straightforward because you can see exactly which gate tripped.

  • Save internal/evaluator/gates.go.

Why deterministic gates before LLM-as-a-judge?

This is the cost gate pattern. Deterministic checks run in microseconds and cost nothing. An LLM judge call costs money per invocation. If a trace already fails a deterministic gate (timed-out tool, empty response), there is no point paying for a judge to evaluate it further. You save ~$0.001-0.01 per failed trace at scale.

Update the Lambda handler with configurable thresholds

The gates are implemented but the Lambda does not call them yet. You need to wire the composite evaluator into the handler, read thresholds from environment variables, and write the decision record (with gate results) to DynamoDB.

  • In cmd/evaluator/main.go, find the import block at the top of the file.
  • Replace the entire import block with this updated version that adds os, strconv, and the evaluator package:
import (
	"context"
	"encoding/base64"
	"encoding/json"
	"log"
	"os"
	"strconv"
	"time"

	"github.com/aws/aws-lambda-go/events"
	"github.com/aws/aws-lambda-go/lambda"
	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/dynamodb"
	"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
	"llm-eval-engine/internal/evaluator"
	"llm-eval-engine/internal/models"
)
  • Add the following three helper functions between the const tableName line and the func init() block:
// getEnvInt64 reads an int64 from an environment variable with a fallback default
func getEnvInt64(key string, def int64) int64 {
	if v := os.Getenv(key); v != "" {
		if n, err := strconv.ParseInt(v, 10, 64); err == nil {
			return n
		}
	}
	return def
}

// getEnvInt reads an int from an environment variable with a fallback default
func getEnvInt(key string, def int) int {
	if v := os.Getenv(key); v != "" {
		if n, err := strconv.Atoi(v); err == nil {
			return n
		}
	}
	return def
}

// getEnvFloat reads a float64 from an environment variable with a fallback default
func getEnvFloat(key string, def float64) float64 {
	if v := os.Getenv(key); v != "" {
		if f, err := strconv.ParseFloat(v, 64); err == nil {
			return f
		}
	}
	return def
}

What do these helpers do?

Each helper reads a Lambda environment variable and parses it into the correct Go type. If the variable is unset or unparseable, it falls back to the default value. This lets you tune gate thresholds by updating Lambda env vars, no code redeploy needed.

  • Save cmd/evaluator/main.go.
  • Confirm compilation by running:
go build ./...

No output means the new imports and helpers integrated cleanly.

Seeing an import cycle or unused import?

Verify the evaluator import path is exactly llm-eval-engine/internal/evaluator. If you see an "imported and not used" error for the evaluator package, that is expected for now. You will reference it in the handler function next.

Help me resolve my import error in cmd/evaluator/main.go.

  • Now replace the existing handler function in cmd/evaluator/main.go with this updated version. Start by deleting the old func handler through its closing brace, then paste this first section:
func handler(ctx context.Context, event events.KinesisEvent) error {
	// Read gate thresholds from Lambda environment variables
	gateCfg := evaluator.GateConfig{
		ToolTimeoutMs:     getEnvInt64("TOOL_TIMEOUT_MS", 5000),
		RetrievalMinScore: getEnvFloat("RETRIEVAL_MIN_SCORE", 0.7),
		MaxLatencyMs:      getEnvInt64("MAX_LATENCY_MS", 10000),
		MaxOutputTokens:   getEnvInt("MAX_OUTPUT_TOKENS", 2048),
	}
	for _, record := range event.Records {
		data, err := base64.StdEncoding.DecodeString(record.Kinesis.Data)
		if err != nil {
			log.Printf("failed to decode record: %v", err)
			continue
		}
		var trace models.EvalTrace
		if err := json.Unmarshal(data, &trace); err != nil {
			log.Printf("failed to unmarshal trace: %v", err)
			continue
		}

What changed from the previous handler?

The gate configuration is now built from environment variables at the top of each invocation. The decode and unmarshal logic is unchanged from before. The key addition is that thresholds are not hardcoded. You can change them in the Lambda console without redeploying code.

  • Continue the handler by adding the gate evaluation and DynamoDB write logic immediately after the unmarshal block (still inside the for loop):
		// Run all deterministic gates and determine overall verdict
		gateResults, allPassed := evaluator.RunDeterministicGates(trace, gateCfg)
		status := "PASS"
		if !allPassed {
			status = "FAIL"
		}
		gateResultsJSON, _ := json.Marshal(gateResults)
		evalTimestamp := time.Now().UTC().Format(time.RFC3339Nano)
		_, err = dynamoClient.PutItem(ctx, &dynamodb.PutItemInput{
			TableName: aws.String(tableName),
			Item: map[string]types.AttributeValue{
				"trace_id":       &types.AttributeValueMemberS{Value: trace.TraceID},
				"eval_timestamp": &types.AttributeValueMemberS{Value: evalTimestamp},
				"status":         &types.AttributeValueMemberS{Value: status},
				"gate_results":   &types.AttributeValueMemberS{Value: string(gateResultsJSON)},
				"prompt_version": &types.AttributeValueMemberS{Value: trace.PromptVersion},
				"model_version":  &types.AttributeValueMemberS{Value: trace.ModelVersion},
			},
		})
		if err != nil {
			log.Printf("failed to write decision to DynamoDB: %v", err)
		}
	}
	return nil
}

What does this section do?

evaluator.RunDeterministicGates runs all five gates and returns the results plus a boolean. If any gate failed, status becomes FAIL. The full gate results are serialized to JSON and stored alongside the decision in DynamoDB, giving you a complete audit trail for every evaluation.

  • Save cmd/evaluator/main.go.
  • Run the build one more time to confirm everything compiles cleanly:
go build ./...

No output means the handler, gates, and models all compile together without errors.

✔️ Awesome, I've got everything!

Great. Confirm you have saved both internal/evaluator/gates.go and cmd/evaluator/main.go.

ⓧ I'd like to double check the full code

Scroll back through the chunks you added. Confirm you have in order:

  • internal/models/trace.go: the original structs plus GateResult and EvalDecision at the bottom.
  • internal/evaluator/gates.go: package declaration, imports, GateConfig, five gate functions (ResponseSchemaCompliance, ToolTimeoutGate, RetrievalQualityGate, LatencySLAGate, TokenBudgetGate), and RunDeterministicGates.
  • cmd/evaluator/main.go: updated imports (including os, strconv, internal/evaluator), three env helpers, and the new handler function that calls RunDeterministicGates and writes decisions.
Redeploy the Lambda with environment variables and verify

The code is ready locally. Now you need to build the Lambda binary, package it, deploy the new code, and set the threshold environment variables on the function.

  • Build the Lambda binary and package it by running these commands:
cd cmd/evaluator
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -tags lambda.norpc -o bootstrap main.go
zip function.zip bootstrap
  • Deploy the updated code to your Lambda function by running:
aws lambda update-function-code --function-name llm-eval-evaluator --zip-file fileb://function.zip

You should see JSON output containing the function ARN and a LastUpdateStatus field.

  • Set the gate threshold environment variables on the Lambda by running:
aws lambda update-function-configuration --function-name llm-eval-evaluator --environment '{"Variables":{"TOOL_TIMEOUT_MS":"5000","RETRIEVAL_MIN_SCORE":"0.7","MAX_LATENCY_MS":"10000","MAX_OUTPUT_TOKENS":"2048"}}'

What do these thresholds mean?

TOOL_TIMEOUT_MS=5000 means any tool call taking more than 5 seconds fails. RETRIEVAL_MIN_SCORE=0.7 means any retrieval chunk with a similarity score below 0.7 triggers a failure.

MAX_LATENCY_MS=10000 caps total execution at 10 seconds. MAX_OUTPUT_TOKENS=2048 flags responses that generated more than 2048 output tokens. You can change any of these later without redeploying code.

  • Navigate back to the project root by running:
cd ../..

Seeing a ResourceConflictException?

This happens if you ran update-function-configuration immediately after update-function-code. Lambda needs a few seconds between updates. Wait 10 seconds and retry the configuration command.

Help me fix my Lambda deployment error.

Now test that a trace with a timed-out tool produces a FAIL decision. Before you submit the test trace, what do you think the gate results will look like for a tool that took 8000ms when the threshold is 5000ms?

  • Start your local API server in one terminal by running:
go run cmd/api/main.go
  • Open a second terminal and submit a trace with a timed-out tool by running:
curl -X POST http://localhost:8080/traces -H "Content-Type: application/json" -d '{
  "prompt_version": "v2.1",
  "model_version": "gpt-4-0613",
  "retrieval_snapshots": [{
    "vector_store_id": "vs-001",
    "query": "test query",
    "chunks": [{"chunk_id": "c1", "content": "test", "score": 0.85}]
  }],
  "tool_call_traces": [{
    "name": "database_lookup",
    "input": {"query": "SELECT *"},
    "output": {"rows": 5},
    "latency_ms": 8000,
    "timed_out": true
  }],
  "execution_steps": [{"step_index": 1, "description": "lookup", "status": "SUCCESS"}],
  "final_response": "Here are the results.",
  "run_group_id": "rg-timeout-test",
  "tokens_used": {"input_tokens": 500, "output_tokens": 200},
  "total_latency_ms": 8500,
  "timestamp": "2026-08-04T12:00:00Z"
}'

You should see a 202 response with a trace_id. The trace flows through Kinesis to your Lambda, which now runs the gates.

  • Wait about 15 seconds for the Kinesis-to-Lambda pipeline to process, then check the DynamoDB table by running:
aws dynamodb scan --table-name llm-eval-decisions

You should see a record with "status": "FAIL" and a gate_results field containing JSON. Inside that JSON, look for TOOL_TIMEOUT with "passed": false and evidence mentioning database_lookup. That is your deterministic gate catching the timed-out tool.

Seeing PENDING_EVALUATION instead of FAIL?

The old Lambda code may still be running. Lambda deployments can take up to 30 seconds to propagate. Wait another 30 seconds and submit a new trace. If the issue persists, check the Lambda's CloudWatch logs for errors by running aws logs tail /aws/lambda/llm-eval-evaluator --since 5m.

Help me debug why my Lambda is not producing gate decisions.

Your evaluation engine now makes real quality decisions. Traces with structural failures get caught in milliseconds and never reach the expensive LLM judge. Next up, you will wire Amazon Nova Pro as an LLM-as-a-judge to evaluate the traces that pass all deterministic gates.

Wire Bedrock LLM-as-a-Judge with Variance Tracking

Your deterministic gates catch structural failures instantly. A timed-out tool, a missing response, retrieval chunks below threshold. But they cannot answer the question that matters most: did the LLM actually respond faithfully and completely?

For semantic quality, you need a judge model. You will invoke Amazon Nova Pro as an LLM-as-a-judge to score each trace on faithfulness and completeness. The catch is that LLM judges are themselves non-deterministic. Run the same evaluation three times and you might get different scores. By tracking variance across iterations, you distinguish reliable judgments from noisy ones and flag high-variance results for human review instead of trusting a single roll of the dice.

In this step, get ready to:
  • Build the Bedrock LLM-as-a-judge evaluator that constructs a scoring prompt and invokes Amazon Nova Pro.
  • Implement multi-iteration variance tracking that runs the judge 3 times and flags unreliable scores.
  • Wire a composite decision engine that combines deterministic gates with judge results for a final PASS/FAIL/REVIEW decision.
Add judge data types and build the scoring prompt

The judge needs its own data structures to store individual iteration scores and aggregated results. You will add these to your existing models file, then create a new file for the judge logic.

  • In internal/models/trace.go, add the following two structs after the existing GateResult struct:
type JudgeIteration struct {
	Faithfulness float64 `json:"faithfulness"`
	Completeness float64 `json:"completeness"`
	Reasoning    string  `json:"reasoning"`
}

type JudgeResult struct {
	Iterations         []JudgeIteration `json:"iterations"`
	MeanFaithfulness   float64          `json:"mean_faithfulness"`
	MeanCompleteness   float64          `json:"mean_completeness"`
	StddevFaithfulness float64          `json:"stddev_faithfulness"`
	StddevCompleteness float64          `json:"stddev_completeness"`
	HighVariance       bool             `json:"high_variance"`
}

What do these structs hold?

  • JudgeIteration stores a single judge call's scores (faithfulness 0-1, completeness 0-1) plus the model's reasoning explanation.
  • JudgeResult aggregates all iterations, computing mean and standard deviation per metric and flagging high variance.
  • Now find the existing EvalDecision struct in the same file. It currently looks like this:
type EvalDecision struct {
	TraceID        string       `json:"trace_id"`
	EvalTimestamp  string       `json:"eval_timestamp"`
	OverallStatus  string       `json:"overall_status"`
	GateResults    []GateResult `json:"gate_results"`
	PromptVersion  string       `json:"prompt_version"`
	ModelVersion   string       `json:"model_version"`
}
  • Add a JudgeResult field between GateResults and PromptVersion, so the struct now looks like this:
type EvalDecision struct {
	TraceID       string       `json:"trace_id"`
	EvalTimestamp string       `json:"eval_timestamp"`
	OverallStatus string       `json:"overall_status"`
	GateResults   []GateResult `json:"gate_results"`
	JudgeResult   *JudgeResult `json:"judge_result,omitempty"`
	PromptVersion string       `json:"prompt_version"`
	ModelVersion  string       `json:"model_version"`
}

Why a pointer for JudgeResult?

The *JudgeResult pointer with omitempty means the field is nil (and excluded from JSON) when deterministic gates fail and the judge never runs. This keeps your DynamoDB records clean for traces that get short-circuited.

  • Save internal/models/trace.go.
  • Verify the project compiles by running:
go build ./...

You should see no output, meaning the build succeeded.

Seeing a compile error?

  • Check that JudgeIteration and JudgeResult are placed outside any existing struct, not nested inside one.
  • Confirm backtick JSON tags are complete (opening and closing backticks on each line).
  • help me fix a compile error in my trace.go model file
  • Create a new file internal/evaluator/judge.go and add the package header, imports, constants, and Bedrock request types:
package evaluator

import (
	"context"
	"encoding/json"
	"fmt"
	"math"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/bedrockruntime"
	"llm-eval-engine/internal/models"
)

const novaProModelID = "amazon.nova-pro-v1:0"
const judgeIterations = 3
const varianceThreshold = 0.15

type bedrockRequest struct {
	Messages        []bedrockMessage       `json:"messages"`
	InferenceConfig bedrockInferenceConfig `json:"inferenceConfig"`
}

type bedrockMessage struct {
	Role    string           `json:"role"`
	Content []bedrockContent `json:"content"`
}

type bedrockContent struct {
	Type string `json:"type"`
	Text string `json:"text"`
}

What do these constants control?

  • novaProModelID is the exact Bedrock model identifier for Amazon Nova Pro.
  • judgeIterations = 3 means you run the judge three times per trace to measure variance.
  • varianceThreshold = 0.15 flags any judgment where the standard deviation exceeds 0.15 on the 0-1 scale as unreliable.
  • Below the bedrockContent struct, add the remaining types and the buildJudgePrompt function:
type bedrockInferenceConfig struct {
	Temperature float64 `json:"temperature"`
	MaxTokens   int     `json:"maxTokens"`
}

type judgeScoreResponse struct {
	Faithfulness float64 `json:"faithfulness"`
	Completeness float64 `json:"completeness"`
	Reasoning    string  `json:"reasoning"`
}

func buildJudgePrompt(trace models.EvalTrace) string {
	return fmt.Sprintf(`You are an LLM evaluation judge. Score the following LLM response on two dimensions.

Context provided to LLM (retrieved chunks):
%s

Tool outputs:
%s

Final response:
%s

Return ONLY valid JSON with this exact structure:
{"faithfulness": <0.0-1.0>, "completeness": <0.0-1.0>, "reasoning": "<brief explanation>"}

faithfulness: Does the response accurately reflect the retrieved context without hallucination?
completeness: Does the response fully address what was asked based on available context?`,
		fmt.Sprintf("%v", trace.RetrievalSnapshots),
		fmt.Sprintf("%v", trace.ToolCallTraces),
		trace.FinalResponse,
	)
}

What does buildJudgePrompt do?

This function assembles a structured prompt that feeds the judge model all the context it needs: the retrieved chunks, tool outputs, and the final LLM response. It asks Nova Pro to return a JSON object with two scores (0-1 each) and a reasoning explanation. The structured output format makes parsing reliable.

  • Save internal/evaluator/judge.go and verify the build:
go build ./...

No output means success. The file compiles but the functions are not called yet.

Implement the single-call invoker and multi-iteration variance tracker

Each judge evaluation calls Nova Pro once and parses the JSON response. The variance tracker wraps this in a loop, running three iterations and computing statistical measures to determine reliability.

  • In internal/evaluator/judge.go, add the invokeJudgeOnce function below buildJudgePrompt:
func invokeJudgeOnce(ctx context.Context, client *bedrockruntime.Client, prompt string) (models.JudgeIteration, error) {
	req := bedrockRequest{
		Messages: []bedrockMessage{
			{Role: "user", Content: []bedrockContent{{Type: "text", Text: prompt}}},
		},
		InferenceConfig: bedrockInferenceConfig{Temperature: 0.7, MaxTokens: 512},
	}
	body, _ := json.Marshal(req)
	resp, err := client.InvokeModel(ctx, &bedrockruntime.InvokeModelInput{
		ModelId:     aws.String(novaProModelID),
		Body:        body,
		ContentType: aws.String("application/json"),
	})
	if err != nil {
		return models.JudgeIteration{}, err
	}
	var outer struct {
		Output struct {
			Message struct {
				Content []struct {
					Text string `json:"text"`
				} `json:"content"`
			} `json:"message"`
		} `json:"output"`
	}
	if err := json.Unmarshal(resp.Body, &outer); err != nil {
		return models.JudgeIteration{}, err
	}
	text := outer.Output.Message.Content[0].Text
	var score judgeScoreResponse
	if err := json.Unmarshal([]byte(text), &score); err != nil {
		return models.JudgeIteration{}, fmt.Errorf("failed to parse judge score: %v, raw: %s", err, text)
	}
	return models.JudgeIteration{Faithfulness: score.Faithfulness, Completeness: score.Completeness, Reasoning: score.Reasoning}, nil
}

What does this function do?

  • Constructs the Nova Pro request with temperature: 0.7 for natural variation between iterations.
  • Calls InvokeModel with the model ID amazon.nova-pro-v1:0 and parses the nested response body.
  • The response structure (output.message.content[0].text) matches the official Nova Pro InvokeModel response format.
  • Now add the RunJudge function below invokeJudgeOnce. This is the variance tracker:
func RunJudge(ctx context.Context, client *bedrockruntime.Client, trace models.EvalTrace) models.JudgeResult {
	prompt := buildJudgePrompt(trace)
	var iterations []models.JudgeIteration
	for i := 0; i < judgeIterations; i++ {
		iter, err := invokeJudgeOnce(ctx, client, prompt)
		if err != nil {
			iter = models.JudgeIteration{Faithfulness: 0.5, Completeness: 0.5, Reasoning: fmt.Sprintf("judge error: %v", err)}
		}
		iterations = append(iterations, iter)
	}
	var sumF, sumC float64
	for _, it := range iterations {
		sumF += it.Faithfulness
		sumC += it.Completeness
	}
	n := float64(len(iterations))
	meanF := sumF / n
	meanC := sumC / n
	var varF, varC float64
	for _, it := range iterations {
		varF += math.Pow(it.Faithfulness-meanF, 2)
		varC += math.Pow(it.Completeness-meanC, 2)
	}
	stdF := math.Sqrt(varF / n)
	stdC := math.Sqrt(varC / n)
	highVariance := stdF > varianceThreshold || stdC > varianceThreshold
	return models.JudgeResult{
		Iterations:         iterations,
		MeanFaithfulness:   meanF,
		MeanCompleteness:   meanC,
		StddevFaithfulness: stdF,
		StddevCompleteness: stdC,
		HighVariance:       highVariance,
	}
}

How does variance tracking work?

The function runs the judge 3 times with the same prompt. Because temperature > 0, each call may produce slightly different scores. It computes the mean and population standard deviation for both faithfulness and completeness.

If either standard deviation exceeds 0.15, the judgment is flagged as HighVariance. This means the model was inconsistent and the result should go to human review rather than being trusted as an automated decision.

  • Save internal/evaluator/judge.go and verify the build:
go build ./...

Clean output confirms all the judge functions compile correctly.

Build failing on the bedrockruntime import?

  • Run go mod tidy to ensure the bedrockruntime module is downloaded.
  • Verify the import path is exactly github.com/aws/aws-sdk-go-v2/service/bedrockruntime.
  • help me fix the bedrockruntime import error in judge.go
Build the composite decision engine and redeploy

The composite engine is the decision-maker. It runs deterministic gates first as a cost gate (if they fail, skip the expensive Bedrock calls entirely), then runs the LLM judge, and produces a final PASS/FAIL/REVIEW decision.

  • Create a new file internal/evaluator/composite.go with the package header and threshold constants:
package evaluator

import (
	"context"

	"github.com/aws/aws-sdk-go-v2/service/bedrockruntime"
	"llm-eval-engine/internal/models"
)

const faithfulnessThreshold = 0.7
const completenessThreshold = 0.7
  • Below the constants, add the RunCompositeEvaluation function:
func RunCompositeEvaluation(ctx context.Context, bedrockClient *bedrockruntime.Client, trace models.EvalTrace, cfg GateConfig) models.EvalDecision {
	gateResults, allGatesPassed := RunDeterministicGates(trace, cfg)
	decision := models.EvalDecision{
		TraceID:       trace.TraceID,
		GateResults:   gateResults,
		PromptVersion: trace.PromptVersion,
		ModelVersion:  trace.ModelVersion,
	}
	if !allGatesPassed {
		decision.OverallStatus = "FAIL"
		return decision
	}
	judgeResult := RunJudge(ctx, bedrockClient, trace)
	decision.JudgeResult = &judgeResult
	if judgeResult.HighVariance {
		decision.OverallStatus = "REVIEW"
	} else if judgeResult.MeanFaithfulness >= faithfulnessThreshold && judgeResult.MeanCompleteness >= completenessThreshold {
		decision.OverallStatus = "PASS"
	} else if judgeResult.MeanFaithfulness < faithfulnessThreshold || judgeResult.MeanCompleteness < completenessThreshold {
		decision.OverallStatus = "FAIL"
	} else {
		decision.OverallStatus = "REVIEW"
	}
	return decision
}

What is the decision logic?

  • If any deterministic gate fails, immediately return FAIL. The judge never runs. This is the cost gate pattern.
  • If the judge reports high variance (stddev > 0.15), return REVIEW regardless of scores. The judgment is unreliable.
  • If variance is low and both scores exceed 0.7, return PASS.
  • If variance is low and either score is below 0.7, return FAIL.
  • Save internal/evaluator/composite.go and verify:
go build ./...

The build will fail because cmd/evaluator/main.go still uses the old handler logic. You will fix that next.

  • In cmd/evaluator/main.go, find the imports section. Add "github.com/aws/aws-sdk-go-v2/service/bedrockruntime" to the import block.
  • Find the variable declaration var dynamoClient *dynamodb.Client and replace it with a grouped declaration:
var (
	dynamoClient  *dynamodb.Client
	bedrockClient *bedrockruntime.Client
)
  • In the init() function, add this line after dynamoClient = dynamodb.NewFromConfig(cfg):
	bedrockClient = bedrockruntime.NewFromConfig(cfg)
  • Now replace the entire handler function body. Find the existing for _, record := range event.Records loop and replace the processing logic inside it with the composite evaluation. The updated handler:
func handler(ctx context.Context, event events.KinesisEvent) error {
	gateCfg := evaluator.GateConfig{
		ToolTimeoutMs:     getEnvInt64("TOOL_TIMEOUT_MS", 5000),
		RetrievalMinScore: getEnvFloat("RETRIEVAL_MIN_SCORE", 0.7),
		MaxLatencyMs:      getEnvInt64("MAX_LATENCY_MS", 10000),
		MaxOutputTokens:   getEnvInt("MAX_OUTPUT_TOKENS", 2048),
	}
	for _, record := range event.Records {
		data, err := base64.StdEncoding.DecodeString(record.Kinesis.Data)
		if err != nil {
			log.Printf("failed to decode record: %v", err)
			continue
		}
		var trace models.EvalTrace
		if err := json.Unmarshal(data, &trace); err != nil {
			log.Printf("failed to unmarshal trace: %v", err)
			continue
		}
		decision := evaluator.RunCompositeEvaluation(ctx, bedrockClient, trace, gateCfg)
		decision.EvalTimestamp = time.Now().UTC().Format(time.RFC3339Nano)
		decisionJSON, _ := json.Marshal(decision)
		judgeJSON := "{}"
		if decision.JudgeResult != nil {
			b, _ := json.Marshal(decision.JudgeResult)
			judgeJSON = string(b)
		}
		gateJSON, _ := json.Marshal(decision.GateResults)
  • Directly below that code, add the DynamoDB write with the new fields:
		_, err = dynamoClient.PutItem(ctx, &dynamodb.PutItemInput{
			TableName: aws.String(tableName),
			Item: map[string]types.AttributeValue{
				"trace_id":        &types.AttributeValueMemberS{Value: decision.TraceID},
				"eval_timestamp":  &types.AttributeValueMemberS{Value: decision.EvalTimestamp},
				"status":          &types.AttributeValueMemberS{Value: decision.OverallStatus},
				"gate_results":    &types.AttributeValueMemberS{Value: string(gateJSON)},
				"judge_result":    &types.AttributeValueMemberS{Value: judgeJSON},
				"full_decision":   &types.AttributeValueMemberS{Value: string(decisionJSON)},
				"prompt_version":  &types.AttributeValueMemberS{Value: decision.PromptVersion},
				"model_version":   &types.AttributeValueMemberS{Value: decision.ModelVersion},
			},
		})
		if err != nil {
			log.Printf("failed to write decision to DynamoDB: %v", err)
		}
	}
	return nil
}

What changed in the handler?

  • Instead of calling RunDeterministicGates directly, the handler now calls RunCompositeEvaluation which orchestrates both deterministic gates and the LLM judge.
  • The DynamoDB write now includes judge_result (iterations, scores, variance) and full_decision (complete audit trail).
  • The bedrockClient is passed through to enable Nova Pro invocations inside the composite function.
  • Save cmd/evaluator/main.go and verify the full project compiles:
go build ./...

Clean output means all files compile together correctly.

Seeing unused import or undefined errors?

  • If you see undefined: bedrockClient, confirm you added both the variable declaration and the initialization in init().
  • If you see an unused import for the old evaluator functions, make sure you removed the old RunDeterministicGates call from the handler and replaced it with RunCompositeEvaluation.
  • help me fix build errors in my updated evaluator main.go
  • Build and package the Lambda binary by running:
cd cmd/evaluator
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -tags lambda.norpc -o bootstrap main.go
  • Create the deployment zip:
zip function.zip bootstrap
cd ../..
  • Deploy the updated Lambda code:
aws lambda update-function-code --function-name llm-eval-evaluator --zip-file fileb://cmd/evaluator/function.zip

Before you test, think about what you expect to see. You will submit a trace that passes all deterministic gates. What decision should appear in DynamoDB?

  • Start the API server in a separate terminal:
go run cmd/api/main.go
  • Submit a trace that passes all deterministic gates by running this curl command:
curl -X POST http://localhost:8080/traces -H "Content-Type: application/json" -d '{
  "prompt_version": "v2.1",
  "model_version": "gpt-4-0125",
  "retrieval_snapshots": [{"vector_store_id": "vs-001", "query": "test", "chunks": [{"chunk_id": "c1", "content": "test content", "score": 0.92}]}],
  "tool_call_traces": [{"name": "search", "input": {}, "output": {}, "latency_ms": 200, "timed_out": false}],
  "execution_steps": [{"step_index": 1, "description": "retrieve", "status": "SUCCESS"}],
  "final_response": "Based on the test content, here is a comprehensive answer.",
  "run_group_id": "rg-judge-test",
  "tokens_used": {"input_tokens": 500, "output_tokens": 150},
  "total_latency_ms": 3000,
  "timestamp": "2026-08-04T10:00:00Z"
}'

You should get back a 202 response with a trace ID.

  • Wait 10-15 seconds for Kinesis to deliver the record to Lambda, then scan DynamoDB:
aws dynamodb scan --table-name llm-eval-decisions --max-items 1

You should see a record with 3 judge iterations (individual faithfulness and completeness scores), mean scores, standard deviation values, a variance flag, and a composite decision of PASS, FAIL, or REVIEW depending on how Nova Pro scored your trace.

What should I see?

The DynamoDB record will contain a judge_result field with the three iteration scores. If the scores are consistent (stddev < 0.15) and both means are above 0.7, the status is PASS. If scores vary widely, it will be REVIEW. The key insight is that you now have statistical confidence in the judgment, not just a single point estimate.

DynamoDB record still shows old format?

  • Confirm the update-function-code command succeeded. Check that function.zip contains the bootstrap binary (run unzip -l cmd/evaluator/function.zip).
  • If the record has no judge_result field but shows FAIL with gate results, the trace likely failed a deterministic gate before reaching the judge. Verify your curl payload has valid values within all thresholds.
  • Check Lambda logs with aws logs tail /aws/lambda/llm-eval-evaluator --since 5m for Bedrock permission errors.
  • help me debug why my Lambda is not producing judge results in DynamoDB

✔️ Awesome, I've got everything!

Great. Double check you have saved all files: internal/models/trace.go, internal/evaluator/judge.go, internal/evaluator/composite.go, and cmd/evaluator/main.go.

ⓧ I'd like to double check the full code

Your project now has four changed files. Scroll up and confirm each section is present in order:

  • internal/models/trace.go: JudgeIteration struct, JudgeResult struct, and updated EvalDecision with the *JudgeResult field.
  • internal/evaluator/judge.go: package header with imports, 3 constants, 5 type definitions, buildJudgePrompt, invokeJudgeOnce, RunJudge.
  • internal/evaluator/composite.go: package header with imports, 2 threshold constants, RunCompositeEvaluation function.
  • cmd/evaluator/main.go: bedrockruntime import, grouped var with both clients, bedrockClient initialization in init(), and updated handler calling RunCompositeEvaluation.

Your evaluation engine now has two layers of quality judgment. Deterministic gates fire instantly for structural failures. The LLM judge scores semantic quality across multiple iterations, and the variance tracker ensures you only trust consistent judgments. Next up, you will put it all together in the Secret Mission by connecting a Bedrock Agent to query your evaluation data with natural language.

Secret mission

Build a Bedrock Agent to Query Evaluation Decisions

You have a database full of evaluation decisions but no way to ask natural language questions about patterns. In this secret mission, you'll build a Bedrock Agent that queries your DynamoDB evaluation data and surfaces actionable insights like "which prompt version has the highest pass rate?" This requires understanding Bedrock Agent action groups, Lambda-backed action handlers, and designing a query interface over your evaluation data.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses on-demand AWS resources that incur costs while active.

Cost warning

The Kinesis Data Stream is the most expensive idle resource at approximately $0.04/hr. DynamoDB on-demand charges per request (pennies at test scale). Lambda is covered by the free tier for up to 1M requests/month.

Left running with no traffic, expect roughly $1/day in charges (mostly from Kinesis). If you are not actively using the pipeline, choose Pause or Delete below.

Resources you used:

  • Kinesis Data Stream (llm-eval-stream) in us-east-1.
  • DynamoDB table (llm-eval-decisions) with two GSIs in us-east-1.
  • Lambda function (llm-eval-evaluator) with Kinesis event source mapping.
  • Lambda function (llm-eval-agent-handler) for Bedrock Agent action group.
  • Bedrock Agent (llm-eval-analyst) with EvalQueryActions action group.
  • Local project files (llm-eval-engine) on your Desktop.

Keep everything running

No action needed. Choose this if you are still actively building or plan to extend the project soon.

  • Your Kinesis stream, DynamoDB table, Lambda functions, and Bedrock Agent remain active.
  • Expect approximately $1/day in idle charges, mostly from the Kinesis stream.
  • Your local project files stay on your Desktop for future development.

Pause - I'll come back to this later

Shut down the most expensive idle resource (Kinesis) while keeping your evaluation data and Lambda logic intact.

  • Delete the Kinesis stream by running:
aws kinesis delete-stream --stream-name llm-eval-stream
  • Your DynamoDB table and Lambda functions remain. DynamoDB only charges per request, so idle cost is effectively zero.
  • When you return, recreate the stream and rewire the event source mapping to resume.

What stays active?

Your DynamoDB table retains all evaluation decisions. Both Lambda functions remain deployed but cost nothing unless invoked. The Bedrock Agent stays available for querying your historical data.

Delete - I don't want to use this again

Remove all project resources and stop all billing. Follow these steps in order.

  • First, find and delete the Kinesis event source mapping on the evaluator Lambda by running:
aws lambda list-event-source-mappings --function-name llm-eval-evaluator
  • Copy the UUID value from the output, then delete the mapping by running:
aws lambda delete-event-source-mapping --uuid [[MAPPING_UUID="your event source mapping UUID from the output above"]]
  • Delete the Kinesis Data Stream by running:
aws kinesis delete-stream --stream-name llm-eval-stream
  • Delete both Lambda functions by running:
aws lambda delete-function --function-name llm-eval-evaluator
aws lambda delete-function --function-name llm-eval-agent-handler
  • Delete the DynamoDB table by running:
aws dynamodb delete-table --table-name llm-eval-decisions
  • Delete the Bedrock Agent. First, find your agent ID by running:
aws bedrock-agent list-agents --query "agentSummaries[?agentName=='llm-eval-analyst'].agentId" --output text
  • Copy the agent ID from the output, then delete the agent by running:
aws bedrock-agent delete-agent --agent-id [[AGENT_ID="your agent ID from the output above"]] --skip-resource-in-use-check
  • Finally, delete your local project files by running:
cd ~/Desktop
rm -rf llm-eval-engine

How do I know everything is deleted?

Run aws kinesis list-streams and confirm llm-eval-stream no longer appears. Run aws dynamodb list-tables and confirm llm-eval-decisions is gone.

Run aws lambda list-functions and confirm neither evaluator function appears. Run aws bedrock-agent list-agents and confirm llm-eval-analyst is gone.

Nice Work!

Nice Work!

You just built a production-grade LLM evaluation engine in Go that ingests execution traces, validates them against schema, runs deterministic quality gates, and invokes an LLM judge with variance tracking to produce trustworthy PASS/FAIL/REVIEW decisions.

You learned how to:

  • Build a JSON Schema validated ingestion API in Go that rejects malformed LLM execution traces before they enter your pipeline, and wire it to a Kinesis-to-Lambda-to-DynamoDB pipeline designed for 1M req/sec horizontal scale.
  • Implement five deterministic evaluation gates with configurable thresholds (tool timeout, retrieval quality, latency SLA, token budget, response schema) that produce instant PASS/FAIL decisions and act as a cost gate before expensive LLM judge calls.
  • Wire Amazon Nova Pro as an LLM-as-a-judge with 3-iteration variance tracking that distinguishes reliable scores from noisy ones, producing gated PASS/FAIL/REVIEW outcomes with full audit trails.
  • Secret Mission: Connected a Bedrock Agent to query your evaluation data in natural language, surfacing actionable insights like pass rates by prompt version and high-variance outliers.

Ready to quiz yourself?