Build an Observable DevSecOps Pipeline

Build a monitored Go container service with telemetry and security CI gates.

Introduction

30 Second Summary

A reassuring green light can hide work that is failing or taking too long. Hidden risks stay invisible until somebody checks the evidence.

In this project, you will build an observable service in Go with a policy-gated GitHub Actions pipeline. Prometheus measures runtime behavior before Trivy checks the container for fixed critical vulnerabilities.

What You'll Build

Picture your Docker Compose stack turning normal, slow, and failed requests into live evidence that determines whether your GitHub Actions build passes.

By the end of this project, you'll have:

  • A containerized service that you can drive through normal, slow, or intentional error paths from one command.
  • Queryable operational evidence that you can inspect as line-delimited JSON logs or analyze with PromQL for request rate, error ratio, and p95 latency.
  • A policy-gated workflow that accepts a build only after behavior, telemetry, and vulnerability checks pass. It preserves the evidence. It also cleans up every run.
  • Secret Mission: Add and trigger a Prometheus alert when more than 10 percent of recent requests fail.

Are there any prerequisites?

You need basic command-line familiarity, a GitHub account with repository creation permission, and a Linux workstation with Docker Engine, Docker Compose, Git, and Visual Studio Code installed. Go, Prometheus, and Trivy run in containers, so you do not install them locally.

Before We Start

Before hands-on work begins, commit to the evidence your delivery pipeline must produce. It will inspect service health, behavior, logs, metrics, and latency while blocking fixed critical container vulnerabilities.

Set Up the Repository

Each later pipeline check depends on a predictable workspace. A verified toolchain keeps environment failures separate from application failures.

The local toolchain consists of Docker Engine, Docker Compose, and Git. You will then prepare an empty Git workspace in Visual Studio Code.

In this step, get ready to:
  • Verify Docker Engine, Docker Compose, and Git from the Linux terminal.
  • Prepare the observable-container-pipeline workspace in Visual Studio Code.
  • Establish the Git repository on the main branch.
Verify the local toolchain

Docker Engine runs the project containers. Docker Compose coordinates those containers as one stack.

Git records each project change. These checks confirm that all three tools are available before any files are added.

  • Verify the three command-line tools by running these checks:
docker version
docker compose version
git --version

What Do These Checks Prove?

  • The docker version command asks Docker Engine for client and server component versions.
  • The docker compose version command prints Docker Compose version information.
  • The git --version command prints the installed Git suite version.

You should see version information from all three checks. This confirms that the container toolchain is ready.

Did One of the Checks Fail?

If Docker shows client details but cannot contact its server, confirm that Docker Engine is running. If another check fails, confirm that its installed command is available in your terminal.

Help me diagnose my local toolchain check.

Create the project workspace

The project needs one named folder to hold its service code and monitoring configuration. Visual Studio Code treats an opened folder as a workspace.

  • Open your Linux application search.
  • Search for Visual Studio Code.
  • Select Visual Studio Code from the results.

Visual Studio Code opens without a project workspace. The next selection gives the project a permanent location.

  • Click File in the top menu.
  • Select Open Folder....
  • Select New Folder.
  • Enter observable-container-pipeline as the folder name.
  • Select Select Folder.
  • Select Yes, I trust the authors if the Workspace Trust dialog appears.

Good start. You should see observable-container-pipeline at the top of the Explorer sidebar.

Why Skip Local Installs?

The project runs Go, Prometheus, and Trivy inside containers. Your workstation only needs the tools you already verified.

Initialize the Git repository

A Git repository stores project history inside a hidden .git directory. Initializing it with main gives the project the branch used later when you push to GitHub.

  • Click View in the Visual Studio Code top menu.
  • Select Terminal.
  • Initialize the open folder as a Git repository by running this command:
git init -b main

What Does This Command Do?

The command creates the hidden .git directory inside observable-container-pipeline. It sets main as the initial branch.

Git should report that it initialized an empty repository. The reported path ends with observable-container-pipeline/.git/.

Before you check, what branch and hidden directory do you expect to see? Your next command tests that prediction.

  • Confirm the active branch and hidden repository directory by running these checks:
git branch --show-current
ls -a

What Should You See?

The first line should show main. The directory listing should include .git.

Missing the Branch or Directory?

Confirm that the integrated terminal is inside the observable-container-pipeline folder. Check that the repository initialization completed before running the verification commands.

Help me verify my Git repository setup.

That's the foundation in place. Your workspace is open as a Git repository on the main branch.

Your repository is ready. Next up, you will run the containerized service and see what structured logs can reveal.

Run the Service and Inspect Logs

Your repository is initialized. It now needs a running service that demonstrates the behavior your delivery pipeline will eventually test.

You will package a small Go service inside a container. Docker Compose will run the same service without requiring Go on your workstation.

Structured JSON logs record each request. Those individual records make an aggregate error ratio or latency percentile awkward to calculate.

In this step, get ready to:
  • Build a Go service with health, normal, slow, and intentional error behavior.
  • Package the service as a non-root binary in a scratch container image.
  • Generate mixed traffic and inspect the structured request logs.
Build the Go service

The service needs enough behavior to produce meaningful evidence. Its work endpoint completes normally unless a query parameter asks for a slow response or an intentional failure.

  • Create a file named main.go inside the observable-container-pipeline folder using the new-file control in the Visual Studio Code sidebar.
  • Add the service foundation by pasting this at the top of main.go:
package main

import (
	"encoding/json"
	"fmt"
	"io"
	"log/slog"
	"net/http"
	"os"
	"time"
)

const baseURL = "http://127.0.0.1:8080"

var (
	logger = slog.New(slog.NewJSONHandler(os.Stdout, nil))
	httpClient = &http.Client{Timeout: 2 * time.Second}
)

type statusRecorder struct {
	http.ResponseWriter
	status int
}

What Does This Foundation Do?

  • The imports provide JSON encoding, structured logging, request handling, command output, and timing.
  • The baseURL value gives the service's built-in checks a consistent local address.
  • The httpClient timeout prevents an internal check from waiting indefinitely.
  • The statusRecorder stores the response status for each request log.
  • Add the response recording methods directly below statusRecorder by pasting this section:
func (w *statusRecorder) WriteHeader(code int) {
	if w.status != 0 {
		return
	}
	w.status = code
	w.ResponseWriter.WriteHeader(code)
}

func (w *statusRecorder) Write(payload []byte) (int, error) {
	if w.status == 0 {
		w.WriteHeader(http.StatusOK)
	}
	return w.ResponseWriter.Write(payload)
}

How Does Status Recording Work?

  • The recorder wraps the standard HTTP response writer so the service can capture the status code.
  • The first call to WriteHeader() stores the response status.
  • The Write() method records a successful status when a handler writes a body without setting one first.
  • Add the request logging middleware below the recorder methods by pasting this section:
func logRequest(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		started := time.Now()
		recorder := &statusRecorder{ResponseWriter: w}
		next.ServeHTTP(recorder, r)
		if recorder.status == 0 {
			recorder.status = http.StatusOK
		}
		duration := time.Since(started)
		fields := []any{"method", r.Method, "route", r.URL.Path, "status", recorder.status, "duration_ms", duration.Milliseconds()}
		if recorder.status >= http.StatusInternalServerError {
			logger.Error("request completed", fields...)
			return
		}
		logger.Info("request completed", fields...)
	})
}

What Does the Middleware Capture?

  • The logRequest() middleware measures the time spent processing each request.
  • Every completed request records its method, route, status, and duration in milliseconds.
  • Server errors use an error-level log entry so failed work stands out from successful work.
  • Add the JSON response helper and request handlers below logRequest() by pasting this section:
func writeJSON(w http.ResponseWriter, status int, payload map[string]string) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(status)
	_ = json.NewEncoder(w).Encode(payload)
}

func healthHandler(w http.ResponseWriter, _ *http.Request) {
	writeJSON(w, http.StatusOK, map[string]string{"status": "ok"})
}

func workHandler(w http.ResponseWriter, r *http.Request) {
	switch r.URL.Query().Get("mode") {
	case "slow":
		time.Sleep(400 * time.Millisecond)
		writeJSON(w, http.StatusOK, map[string]string{"result": "completed slowly"})
	case "error":
		writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "intentional failure"})
	default:
		writeJSON(w, http.StatusOK, map[string]string{"result": "completed"})
	}
}

How Does the Service Behave?

  • The writeJSON() helper gives every response a JSON body and an explicit status.
  • The healthHandler() reports whether the process can answer a request.
  • The workHandler() supports completed, slow, and intentional error responses.
  • The slow mode waits for 400 milliseconds so its duration differs clearly in the logs.
  • Add the internal request helper and health check below workHandler() by pasting this section:
func fetch(path string) (int, string, error) {
	response, err := httpClient.Get(baseURL + path)
	if err != nil {
		return 0, "", err
	}
	defer response.Body.Close()
	body, err := io.ReadAll(response.Body)
	if err != nil {
		return 0, "", err
	}
	return response.StatusCode, string(body), nil
}

func runHealthcheck() int {
	status, _, err := fetch("/health")
	if err != nil || status != http.StatusOK {
		return 1
	}
	return 0
}

How Does the Health Check Work?

  • The fetch() helper requests a service path and returns its status plus body.
  • The runHealthcheck() function succeeds only when /health returns a successful status.
  • The container uses this result to decide whether the service is healthy.
  • Add the mixed-traffic generator below runHealthcheck() by pasting this section:
func runTraffic() int {
	paths := []string{"/work", "/work", "/work", "/work", "/work?mode=slow", "/work?mode=slow", "/work?mode=error", "/work?mode=error"}
	for _, path := range paths {
		status, _, err := fetch(path)
		if err != nil {
			fmt.Printf("request %s failed: %v\n", path, err)
			return 1
		}
		fmt.Printf("request %s returned %d\n", path, status)
	}
	return 0
}

Why Generate Mixed Traffic?

  • The paths slice defines four normal requests, two slow requests, and two failed requests.
  • The loop prints every returned status so you can compare command output with the service logs.
  • The deliberate mix gives you a small workload with both latency and failure signals.
  • Finish main.go by adding the server and command selection logic below runTraffic():
func runServer() {
	mux := http.NewServeMux()
	mux.Handle("/health", http.HandlerFunc(healthHandler))
	mux.Handle("/work", logRequest(http.HandlerFunc(workHandler)))
	server := &http.Server{Addr: ":8080", Handler: mux, ReadHeaderTimeout: 5 * time.Second}
	logger.Info("server starting", "address", server.Addr)
	if err := server.ListenAndServe(); err != nil {
		logger.Error("server stopped", "error", err)
		os.Exit(1)
	}
}

func main() {
	if len(os.Args) > 1 {
		switch os.Args[1] {
		case "--healthcheck":
			os.Exit(runHealthcheck())
		case "--generate-traffic":
			os.Exit(runTraffic())
		default:
			fmt.Fprintf(os.Stderr, "unknown command: %s\n", os.Args[1])
			os.Exit(2)
		}
	}
	runServer()
}

How Does the Program Start?

  • The runServer() function exposes /health and /work on port 8080.
  • Only work requests pass through logRequest() so the logs focus on application behavior.
  • The main() function selects the health check, traffic generator, or server mode from the supplied argument.
  • Save main.go.
  • Confirm the Visual Studio Code file list shows main.go inside observable-container-pipeline.

Seeing a Problem in main.go?

Check that every pasted section appears once in the same order shown above. Confirm that each opening brace has a matching closing brace.

Use the full-code comparison below if a function landed inside the previous function by mistake.

Help me compare my main.go file

Package and start the service

A multi-stage container build compiles the binary in a Go builder image. The runtime stage contains only the finished application binary.

  • Create a file named go.mod inside observable-container-pipeline using the new-file control in the Visual Studio Code sidebar.
  • Define the Go module by pasting this into go.mod:
module example.com/observable-service

go 1.27.0

What Does go.mod Define?

The module name identifies the service's code. The Go directive selects language version 1.27.0 for the containerized build.

  • Save go.mod.
  • Confirm the Visual Studio Code file list now shows go.mod beside main.go.
  • Create a file named Dockerfile inside observable-container-pipeline using the new-file control in the Visual Studio Code sidebar.
  • Define the container build by pasting this into Dockerfile:
FROM golang:1.27.0-alpine3.24 AS build

WORKDIR /src
COPY go.mod main.go ./
RUN go mod tidy && go mod verify
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /app .

FROM scratch
COPY --from=build /app /app
EXPOSE 8080
USER 65532:65532
HEALTHCHECK --interval=2s --timeout=2s --retries=10 CMD ["/app", "--healthcheck"]
ENTRYPOINT ["/app"]

What Does the Container Build Do?

  • The builder stage downloads module data and compiles a Linux binary named /app.
  • The runtime stage copies that binary into a minimal scratch image.
  • Numeric user 65532:65532 runs the service without root privileges.
  • The health check calls the binary's built-in --healthcheck mode.
  • Save Dockerfile.
  • Confirm the Visual Studio Code file list now shows Dockerfile beside the Go files.
  • Create a file named compose.yaml inside observable-container-pipeline using the new-file control in the Visual Studio Code sidebar.
  • Define the application service by pasting this into compose.yaml:
services:
  app:
    build:
      context: .
    image: observable-service:local
    ports:
      - "8080:8080"

What Does the Compose File Do?

  • The app service builds from the files in observable-container-pipeline.
  • The resulting image uses the local name observable-service:local.
  • The port mapping makes the service available through local port 8080.
  • Save compose.yaml.
  • Confirm the Visual Studio Code file list shows all four project files.

✔️ Awesome, I've got everything!

Your four files are ready. Save each file before building the stack.

ⓧ I'd like to double check the full code

Compare each file with the complete versions below. Every line should match before you build the image.

module example.com/observable-service

go 1.27.0

package main

import (
	"encoding/json"
	"fmt"
	"io"
	"log/slog"
	"net/http"
	"os"
	"time"
)

const baseURL = "http://127.0.0.1:8080"

var (
	logger = slog.New(slog.NewJSONHandler(os.Stdout, nil))
	httpClient = &http.Client{Timeout: 2 * time.Second}
)

type statusRecorder struct {
	http.ResponseWriter
	status int
}

func (w *statusRecorder) WriteHeader(code int) {
	if w.status != 0 {
		return
	}
	w.status = code
	w.ResponseWriter.WriteHeader(code)
}

func (w *statusRecorder) Write(payload []byte) (int, error) {
	if w.status == 0 {
		w.WriteHeader(http.StatusOK)
	}
	return w.ResponseWriter.Write(payload)
}

func logRequest(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		started := time.Now()
		recorder := &statusRecorder{ResponseWriter: w}
		next.ServeHTTP(recorder, r)
		if recorder.status == 0 {
			recorder.status = http.StatusOK
		}
		duration := time.Since(started)
		fields := []any{"method", r.Method, "route", r.URL.Path, "status", recorder.status, "duration_ms", duration.Milliseconds()}
		if recorder.status >= http.StatusInternalServerError {
			logger.Error("request completed", fields...)
			return
		}
		logger.Info("request completed", fields...)
	})
}

func writeJSON(w http.ResponseWriter, status int, payload map[string]string) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(status)
	_ = json.NewEncoder(w).Encode(payload)
}

func healthHandler(w http.ResponseWriter, _ *http.Request) {
	writeJSON(w, http.StatusOK, map[string]string{"status": "ok"})
}

func workHandler(w http.ResponseWriter, r *http.Request) {
	switch r.URL.Query().Get("mode") {
	case "slow":
		time.Sleep(400 * time.Millisecond)
		writeJSON(w, http.StatusOK, map[string]string{"result": "completed slowly"})
	case "error":
		writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "intentional failure"})
	default:
		writeJSON(w, http.StatusOK, map[string]string{"result": "completed"})
	}
}

func fetch(path string) (int, string, error) {
	response, err := httpClient.Get(baseURL + path)
	if err != nil {
		return 0, "", err
	}
	defer response.Body.Close()
	body, err := io.ReadAll(response.Body)
	if err != nil {
		return 0, "", err
	}
	return response.StatusCode, string(body), nil
}

func runHealthcheck() int {
	status, _, err := fetch("/health")
	if err != nil || status != http.StatusOK {
		return 1
	}
	return 0
}

func runTraffic() int {
	paths := []string{"/work", "/work", "/work", "/work", "/work?mode=slow", "/work?mode=slow", "/work?mode=error", "/work?mode=error"}
	for _, path := range paths {
		status, _, err := fetch(path)
		if err != nil {
			fmt.Printf("request %s failed: %v\n", path, err)
			return 1
		}
		fmt.Printf("request %s returned %d\n", path, status)
	}
	return 0
}

func runServer() {
	mux := http.NewServeMux()
	mux.Handle("/health", http.HandlerFunc(healthHandler))
	mux.Handle("/work", logRequest(http.HandlerFunc(workHandler)))
	server := &http.Server{Addr: ":8080", Handler: mux, ReadHeaderTimeout: 5 * time.Second}
	logger.Info("server starting", "address", server.Addr)
	if err := server.ListenAndServe(); err != nil {
		logger.Error("server stopped", "error", err)
		os.Exit(1)
	}
}

func main() {
	if len(os.Args) > 1 {
		switch os.Args[1] {
		case "--healthcheck":
			os.Exit(runHealthcheck())
		case "--generate-traffic":
			os.Exit(runTraffic())
		default:
			fmt.Fprintf(os.Stderr, "unknown command: %s\n", os.Args[1])
			os.Exit(2)
		}
	}
	runServer()
}

FROM golang:1.27.0-alpine3.24 AS build

WORKDIR /src
COPY go.mod main.go ./
RUN go mod tidy && go mod verify
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /app .

FROM scratch
COPY --from=build /app /app
EXPOSE 8080
USER 65532:65532
HEALTHCHECK --interval=2s --timeout=2s --retries=10 CMD ["/app", "--healthcheck"]
ENTRYPOINT ["/app"]

services:
  app:
    build:
      context: .
    image: observable-service:local
    ports:
      - "8080:8080"

The first build can take a few minutes while Docker downloads the Go builder image. A quiet terminal during part of this work is normal.

  • Build the image and start the healthy application service by running:
docker compose up --build --wait

What Does This Command Do?

  • The --build flag compiles the Go service into observable-service:local before startup.
  • The --wait flag keeps the command active until the application is running and healthy.
  • The health check proves that the binary can answer /health inside the container.

That is the containerized service running. Docker Compose returns control after the application reaches its healthy state.

Did the Stack Fail to Start?

Check the build output for the file or line named near the failure. A Go compilation failure usually points to a missing brace, duplicated section, or pasted line in main.go.

A container build failure can also come from an unsaved go.mod or Dockerfile.

Help me diagnose my container build

Generate traffic and inspect logs

A healthy process proves that the server responds. Mixed traffic tests whether the service also records successful, slow, and failed work.

Before you generate the traffic, how many of the eight requests do you expect to return a successful status?

  • Generate the built-in traffic sequence by running:
docker compose exec app /app --generate-traffic

What Does This Command Do?

Docker Compose runs the application binary inside the existing app container. The --generate-traffic mode sends the eight predefined requests to the running server.

You will see six requests return 200. Two intentional error requests return 500.

Before you inspect the service logs, do you think the individual request records will reveal an aggregate error ratio or latency percentile?

  • Inspect the application logs by running:
docker compose logs app

What Does the Log Command Show?

The command displays output from the running app service. Each work request produces one structured record.

The records include method, route, status, and duration_ms fields.

You can find successful and failed requests one line at a time. The slow records also have larger duration_ms values.

The intended limitation is now visible. These logs do not calculate the failure ratio or show the request-duration distribution as a percentile.

Missing Request Logs?

Confirm that the traffic command printed eight returned statuses before reading the logs. If the traffic command could not reach the service, rerun the stack startup command and wait for its health check to complete.

Check that /work is wrapped with logRequest() inside runServer().

Help me find my missing request logs

You now have a healthy service with observable request events, including deliberate failures and slow responses. Next, Prometheus will turn those scattered events into metrics you can query.

Instrument the Service with Prometheus

Your container now handles normal work, slow work, and intentional failures. Its structured logs prove that each request occurred.

Operational questions need aggregate evidence across many requests. In this step, Prometheus turns the service activity into request counts and latency distributions that you can query.

In this step, get ready to:
  • Instrument the work endpoint with request and duration metrics.
  • Add Prometheus to the local Docker Compose stack.
  • Query traffic volume, failures, and p95 latency.
Instrument the work route

A counter records how many requests the service handles. A histogram groups request durations into buckets for percentile calculations.

Why Prometheus for this service?

Direct Prometheus scraping keeps the telemetry path visible. You can see how the service publishes metrics before a monitoring system stores them as time series.

A broader OpenTelemetry setup adds instrumentation, a Collector, and a storage backend. Prometheus keeps this project focused on metrics contracts and queries.

  • Replace the contents of go.mod with this module definition:
module example.com/observable-service

go 1.27.0

require github.com/prometheus/client_golang v1.24.1

What does this dependency add?

The Prometheus Go client supplies the counter, histogram, registry, and HTTP handler used by the service. Pinning v1.24.1 makes container builds use the same client release.

  • Save go.mod.
  • Add the Prometheus package import to the import block in main.go by copying this line:
	"github.com/prometheus/client_golang/prometheus"

What does this import provide?

The Prometheus package creates metric collectors. It also registers those collectors with the default registry.

  • Replace the existing var block in main.go with these collectors:
var (
	logger = slog.New(slog.NewJSONHandler(os.Stdout, nil))

	httpRequests = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Namespace: "demo",
			Subsystem: "http",
			Name:      "requests_total",
			Help:      "Total number of handled HTTP work requests.",
		},
		[]string{"route", "status"},
	)

	httpRequestDuration = prometheus.NewHistogramVec(
		prometheus.HistogramOpts{
			Namespace: "demo",
			Subsystem: "http",
			Name:      "request_duration_seconds",
			Help:      "Duration of handled HTTP work requests in seconds.",
			Buckets:   []float64{0.05, 0.1, 0.25, 0.5, 1},
		},
		[]string{"route"},
	)

	httpClient = &http.Client{Timeout: 2 * time.Second}
)

What do these collectors measure?

  • The httpRequests counter separates work requests by stable route and status labels.
  • The httpRequestDuration histogram records request durations in seconds across five buckets.
  • The labels stay low-cardinality because their possible values come from a small controlled set.
  • Add the init() function below the var block:
func init() {
	prometheus.MustRegister(httpRequests, httpRequestDuration)
	httpRequests.WithLabelValues("/work", "200")
	httpRequests.WithLabelValues("/work", "500")
	httpRequestDuration.WithLabelValues("/work")
}

Why initialize the label values?

The registration step makes both collectors available through the default Prometheus registry. Initializing the known label combinations exposes zero-valued series before traffic arrives.

  • Save main.go.
  • Rebuild the healthy app with the registered collectors by running:
docker compose up --build --wait

What does this rebuild prove?

Docker rebuilds the Go binary with the Prometheus dependency. Compose waits until the application health check succeeds.

The app should return to a healthy state. This confirms that the collectors compile and register successfully.

Does the rebuild fail?

Check that the Prometheus import appears inside the import block. Confirm that go.mod contains the dependency line exactly once.

Compare the collector names with httpRequests and httpRequestDuration. Go rejects a reference when its capitalization differs.

Help me diagnose the failed metrics build.

  • Add the strconv import to the standard-library section of main.go by copying this line:
	"strconv"

Why convert the status code?

Prometheus label values are strings. The service converts each numeric HTTP status before recording it as the status label.

  • Replace the existing logRequest function with the instrumented middleware:
func instrument(route string, next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		started := time.Now()
		recorder := &statusRecorder{ResponseWriter: w}
		next.ServeHTTP(recorder, r)

		if recorder.status == 0 {
			recorder.status = http.StatusOK
		}

		duration := time.Since(started)
		status := strconv.Itoa(recorder.status)
		httpRequests.WithLabelValues(route, status).Inc()
		httpRequestDuration.WithLabelValues(route).Observe(duration.Seconds())

		fields := []any{
			"method", r.Method,
			"route", route,
			"status", recorder.status,
			"duration_ms", duration.Milliseconds(),
		}
		if recorder.status >= http.StatusInternalServerError {
			logger.Error("request completed", fields...)
			return
		}
		logger.Info("request completed", fields...)
	})
}

How does instrumentation work?

  • The middleware measures elapsed time around the work handler.
  • The counter increments the series matching the route and response status.
  • The histogram observes the same request duration in seconds.
  • The structured log records the event with the same route and status context.
  • Add the Prometheus HTTP handler import below the client import:
	"github.com/prometheus/client_golang/prometheus/promhttp"

What does the handler expose?

The promhttp package converts registered collectors into the text format Prometheus scrapes. The handler serves that output over HTTP.

  • Replace the existing runServer function with this version:
func runServer() {
	mux := http.NewServeMux()
	mux.Handle("/health", http.HandlerFunc(healthHandler))
	mux.Handle("/work", instrument("/work", http.HandlerFunc(workHandler)))
	mux.Handle("/metrics", promhttp.Handler())

	server := &http.Server{
		Addr:              ":8080",
		Handler:           mux,
		ReadHeaderTimeout: 5 * time.Second,
	}

	logger.Info("server starting", "address", server.Addr)
	if err := server.ListenAndServe(); err != nil {
		logger.Error("server stopped", "error", err)
		os.Exit(1)
	}
}

What changed in the server?

The /work route now passes through instrument(). The /metrics route publishes the registered time series.

Health probes and metric scrapes bypass the work middleware. Their background traffic cannot dilute the work endpoint's error ratio.

  • Save main.go.
  • Rebuild the service with its metrics route by running:
docker compose up --build --wait

What should this start?

The command replaces the running app with the newly instrumented image. Compose waits for the same application health check to pass.

  • Visit http://localhost:8080/metrics in a browser.

You should see Prometheus text output containing demo_http_requests_total and demo_http_request_duration_seconds.

That is the service's observability contract working. The application now publishes count and duration evidence for every work request.

Cannot see the custom metrics?

Confirm that runServer() registers /metrics with promhttp.Handler(). Check that the browser uses port 8080.

Help me find why the metrics endpoint is missing.

A metrics contract also gives automation a concrete way to verify that telemetry exists. The service will carry that check into its later delivery workflow.

  • Add the strings import below the strconv import:
	"strings"

Why inspect response content?

A successful status code only proves that an endpoint responded. The strings package lets the smoke test confirm that each response contains its required contract value.

  • Add runSmokeTest() below runHealthcheck():
func runSmokeTest() int {
	checks := []struct {
		name     string
		path     string
		status   int
		contains string
	}{
		{name: "health", path: "/health", status: http.StatusOK, contains: `"status":"ok"`},
		{name: "normal work", path: "/work", status: http.StatusOK, contains: `"result":"completed"`},
		{name: "error contract", path: "/work?mode=error", status: http.StatusInternalServerError, contains: `"error":"intentional failure"`},
		{name: "metrics contract", path: "/metrics", status: http.StatusOK, contains: "demo_http_requests_total"},
	}

	failed := false
	for _, check := range checks {
		status, body, err := fetch(check.path)
		if err != nil || status != check.status || !strings.Contains(body, check.contains) {
			fmt.Printf("FAIL %s: status=%d error=%v\n", check.name, status, err)
			failed = true
			continue
		}
		fmt.Printf("PASS %s: status=%d\n", check.name, status)
	}

	if failed {
		return 1
	}
	return 0
}

What does the smoke test prove?

  • The health check confirms that the process can respond.
  • The normal work check confirms the successful behavior contract.
  • The error check confirms that the intentional failure remains predictable.
  • The metrics check confirms that request telemetry is exposed.
  • Replace the existing main function with this command switch:
func main() {
	if len(os.Args) > 1 {
		switch os.Args[1] {
		case "--healthcheck":
			os.Exit(runHealthcheck())
		case "--smoke-test":
			os.Exit(runSmokeTest())
		case "--generate-traffic":
			os.Exit(runTraffic())
		default:
			fmt.Fprintf(os.Stderr, "unknown command: %s\n", os.Args[1])
			os.Exit(2)
		}
	}

	runServer()
}

How is the contract exposed?

The new --smoke-test branch makes the contract available through the application binary. The server remains the default behavior when no command argument is present.

  • Save main.go.
  • Rebuild the application with its smoke-test command by running:
docker compose up --build --wait

What does this final app rebuild confirm?

The build confirms that the contract function compiles with the instrumented server. The health check confirms that the resulting binary still starts correctly.

You should see the app return to a healthy state. Its binary now carries health, traffic generation, and smoke-test commands.

Seeing an unused import or undefined function?

Confirm that strings appears in the import block. Check that runSmokeTest() appears before main().

Help me repair the smoke-test build.

Add Prometheus to the stack

The service now publishes metrics from memory. A Prometheus server must scrape those values on a schedule to retain queryable time series.

  • Select the observable-container-pipeline folder in the Visual Studio Code Explorer sidebar.
  • Click the New File control in the Explorer toolbar.
  • Name the file prometheus.yml.
  • Add the scrape configuration by copying this code:
global:
  scrape_interval: 5s
  evaluation_interval: 5s

scrape_configs:
  - job_name: "observable-service"
    static_configs:
      - targets: ["app:8080"]

What does this configuration do?

  • The global interval asks Prometheus to scrape targets every five seconds.
  • The job name identifies the service in queries.
  • The target uses the Compose service name app on port 8080.
  • Save prometheus.yml.
  • Confirm that the file appears beside compose.yaml in the Explorer sidebar.

Does the YAML show indentation errors?

Use spaces for every indentation level. Confirm that static_configs is nested under the observable-service job.

Help me check the Prometheus YAML structure.

Docker Compose gives Prometheus a stable service address plus storage that survives a normal stack shutdown. The app health dependency prevents scraping from starting before the service is ready.

  • Replace the contents of compose.yaml with the completed stack:
services:
  app:
    build:
      context: .
    image: observable-service:local
    ports:
      - "8080:8080"

  prometheus:
    image: prom/prometheus:3.14.0
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    ports:
      - "9090:9090"
    depends_on:
      app:
        condition: service_healthy

volumes:
  prometheus-data:

How does the completed stack fit together?

  • The Prometheus 3.14.0 container reads the local configuration through a read-only mount.
  • The prometheus-data volume stores scraped time series outside the container filesystem.
  • Port 9090 exposes the query interface to your browser.
  • The health dependency waits for the app before starting Prometheus.
  • Save compose.yaml.
  • Start the completed monitoring stack by running:
docker compose up --build --wait

What does Compose start?

Compose builds the application image before starting both services. It waits until the app is healthy and Prometheus is running.

You should see both services reach their ready state. The local stack now includes the application plus its metrics store.

Does Prometheus fail to start?

Check that prometheus.yml sits in the same folder as compose.yaml. Confirm that the mounted file path starts with ./prometheus.yml.

If port 9090 is already occupied, stop the local process using that port before retrying the stack.

Help me diagnose the Prometheus container startup.

Query service behavior

PromQL turns the stored series into operational answers. Rate functions measure recent counter growth while histogram quantiles estimate latency percentiles.

  • Generate a fresh mix of normal, slow, and failed work requests by running:
docker compose exec app /app --generate-traffic

What traffic does this create?

The generator sends four normal requests, two slow requests, and two intentional error requests. Every request updates the counter and histogram before producing its structured log.

You should see eight request results. The final output includes both 200 and 500 status codes.

  • Wait for two five-second scrape cycles.
  • Visit http://localhost:9090/query in your browser.

Before you run the first query, do you expect Prometheus to report the application target as reachable?

  • Check the application target by entering this expression:
up{job="observable-service"}

What does the up query ask?

The up series records whether the latest scrape succeeded. The job matcher limits the result to the observable service.

  • Run the expression using the control beside the query input.

You should see a result value of 1. This proves Prometheus can scrape the application.

  • Measure recent request traffic by status with this expression:
sum(rate(demo_http_requests_total[1m])) by (status)

What does the request-rate query ask?

The rate function calculates the counter's average per-second increase over one minute. The sum groups the resulting traffic by status label.

  • Run the expression using the control beside the query input.

You should see separate request-rate series for status 200 and status 500.

  • Calculate the recent 5xx error ratio with this expression:
sum(rate(demo_http_requests_total{status=~"5.."}[1m]))
/
sum(rate(demo_http_requests_total[1m]))

What does the error-ratio query ask?

The numerator measures recent 5xx traffic. Dividing it by the rate of all work requests produces the failed share of recent traffic.

  • Run the expression using the control beside the query input.

You should see a value above zero. The exact value can shift as the one-minute rate window moves.

Before the latency query, do you expect the slow requests to influence the upper end of the distribution?

  • Estimate p95 work latency with this expression:
histogram_quantile(0.95, sum(rate(demo_http_request_duration_seconds_bucket[1m])) by (le))

What does the p95 query ask?

The query calculates recent rates for every histogram bucket. It preserves the le boundary required to estimate the 95th percentile.

  • Run the expression using the control beside the query input.

You should see a latency series for the service. This converts the slow requests into a distribution-level signal.

You have closed the gap that the logs exposed. Prometheus can now answer how frequently requests arrive, how often they fail, and how slow the upper end becomes.

Do the queries return no data?

Wait for another scrape cycle before rerunning the expression. Rate queries need samples across time before they can calculate a result.

Run the traffic generator again if the one-minute window has moved past your requests. Confirm that the up query still returns 1.

Help me diagnose an empty Prometheus query.

Double-check the completed files before moving on.

✔️ Awesome, I've got everything!

Great. Save every file before you continue.

ⓧ I'd like to double check the full code

FROM golang:1.27.0-alpine3.24 AS build

WORKDIR /src
COPY go.mod main.go ./
RUN go mod tidy && go mod verify
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /app .

FROM scratch
COPY --from=build /app /app
EXPOSE 8080
USER 65532:65532
HEALTHCHECK --interval=2s --timeout=2s --retries=10 CMD ["/app", "--healthcheck"]
ENTRYPOINT ["/app"]

What should you compare?

Your Dockerfile remains unchanged. It still builds the Go binary in a builder stage before copying it into the non-root scratch image.

services:
  app:
    build:
      context: .
    image: observable-service:local
    ports:
      - "8080:8080"

  prometheus:
    image: prom/prometheus:3.14.0
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    ports:
      - "9090:9090"
    depends_on:
      app:
        condition: service_healthy

volumes:
  prometheus-data:

What should you compare?

Your compose.yaml should define the app and Prometheus services. It should also define the prometheus-data volume.

module example.com/observable-service

go 1.27.0

require github.com/prometheus/client_golang v1.24.1

What should you compare?

Your go.mod should pin the Prometheus Go client dependency at v1.24.1.

package main

import (
	"encoding/json"
	"fmt"
	"io"
	"log/slog"
	"net/http"
	"os"
	"strconv"
	"strings"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

const baseURL = "http://127.0.0.1:8080"

var (
	logger = slog.New(slog.NewJSONHandler(os.Stdout, nil))

	httpRequests = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Namespace: "demo",
			Subsystem: "http",
			Name:      "requests_total",
			Help:      "Total number of handled HTTP work requests.",
		},
		[]string{"route", "status"},
	)

	httpRequestDuration = prometheus.NewHistogramVec(
		prometheus.HistogramOpts{
			Namespace: "demo",
			Subsystem: "http",
			Name:      "request_duration_seconds",
			Help:      "Duration of handled HTTP work requests in seconds.",
			Buckets:   []float64{0.05, 0.1, 0.25, 0.5, 1},
		},
		[]string{"route"},
	)

	httpClient = &http.Client{Timeout: 2 * time.Second}
)

func init() {
	prometheus.MustRegister(httpRequests, httpRequestDuration)
	httpRequests.WithLabelValues("/work", "200")
	httpRequests.WithLabelValues("/work", "500")
	httpRequestDuration.WithLabelValues("/work")
}

type statusRecorder struct {
	http.ResponseWriter
	status int
}

func (w *statusRecorder) WriteHeader(code int) {
	if w.status != 0 {
		return
	}
	w.status = code
	w.ResponseWriter.WriteHeader(code)
}

func (w *statusRecorder) Write(payload []byte) (int, error) {
	if w.status == 0 {
		w.WriteHeader(http.StatusOK)
	}
	return w.ResponseWriter.Write(payload)
}

func instrument(route string, next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		started := time.Now()
		recorder := &statusRecorder{ResponseWriter: w}
		next.ServeHTTP(recorder, r)

		if recorder.status == 0 {
			recorder.status = http.StatusOK
		}

		duration := time.Since(started)
		status := strconv.Itoa(recorder.status)
		httpRequests.WithLabelValues(route, status).Inc()
		httpRequestDuration.WithLabelValues(route).Observe(duration.Seconds())

		fields := []any{
			"method", r.Method,
			"route", route,
			"status", recorder.status,
			"duration_ms", duration.Milliseconds(),
		}
		if recorder.status >= http.StatusInternalServerError {
			logger.Error("request completed", fields...)
			return
		}
		logger.Info("request completed", fields...)
	})
}

func writeJSON(w http.ResponseWriter, status int, payload map[string]string) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(status)
	_ = json.NewEncoder(w).Encode(payload)
}

func healthHandler(w http.ResponseWriter, _ *http.Request) {
	writeJSON(w, http.StatusOK, map[string]string{"status": "ok"})
}

func workHandler(w http.ResponseWriter, r *http.Request) {
	switch r.URL.Query().Get("mode") {
	case "slow":
		time.Sleep(400 * time.Millisecond)
		writeJSON(w, http.StatusOK, map[string]string{"result": "completed slowly"})
	case "error":
		writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "intentional failure"})
	default:
		writeJSON(w, http.StatusOK, map[string]string{"result": "completed"})
	}
}

func fetch(path string) (int, string, error) {
	response, err := httpClient.Get(baseURL + path)
	if err != nil {
		return 0, "", err
	}
	defer response.Body.Close()

	body, err := io.ReadAll(response.Body)
	if err != nil {
		return 0, "", err
	}
	return response.StatusCode, string(body), nil
}

func runHealthcheck() int {
	status, _, err := fetch("/health")
	if err != nil || status != http.StatusOK {
		return 1
	}
	return 0
}

func runSmokeTest() int {
	checks := []struct {
		name     string
		path     string
		status   int
		contains string
	}{
		{name: "health", path: "/health", status: http.StatusOK, contains: `"status":"ok"`},
		{name: "normal work", path: "/work", status: http.StatusOK, contains: `"result":"completed"`},
		{name: "error contract", path: "/work?mode=error", status: http.StatusInternalServerError, contains: `"error":"intentional failure"`},
		{name: "metrics contract", path: "/metrics", status: http.StatusOK, contains: "demo_http_requests_total"},
	}

	failed := false
	for _, check := range checks {
		status, body, err := fetch(check.path)
		if err != nil || status != check.status || !strings.Contains(body, check.contains) {
			fmt.Printf("FAIL %s: status=%d error=%v\n", check.name, status, err)
			failed = true
			continue
		}
		fmt.Printf("PASS %s: status=%d\n", check.name, status)
	}

	if failed {
		return 1
	}
	return 0
}

func runTraffic() int {
	paths := []string{
		"/work",
		"/work",
		"/work",
		"/work",
		"/work?mode=slow",
		"/work?mode=slow",
		"/work?mode=error",
		"/work?mode=error",
	}

	for _, path := range paths {
		status, _, err := fetch(path)
		if err != nil {
			fmt.Printf("request %s failed: %v\n", path, err)
			return 1
		}
		fmt.Printf("request %s returned %d\n", path, status)
	}
	return 0
}

func runServer() {
	mux := http.NewServeMux()
	mux.Handle("/health", http.HandlerFunc(healthHandler))
	mux.Handle("/work", instrument("/work", http.HandlerFunc(workHandler)))
	mux.Handle("/metrics", promhttp.Handler())

	server := &http.Server{
		Addr:              ":8080",
		Handler:           mux,
		ReadHeaderTimeout: 5 * time.Second,
	}

	logger.Info("server starting", "address", server.Addr)
	if err := server.ListenAndServe(); err != nil {
		logger.Error("server stopped", "error", err)
		os.Exit(1)
	}
}

func main() {
	if len(os.Args) > 1 {
		switch os.Args[1] {
		case "--healthcheck":
			os.Exit(runHealthcheck())
		case "--smoke-test":
			os.Exit(runSmokeTest())
		case "--generate-traffic":
			os.Exit(runTraffic())
		default:
			fmt.Fprintf(os.Stderr, "unknown command: %s\n", os.Args[1])
			os.Exit(2)
		}
	}

	runServer()
}

What should you compare?

Your main.go should register both collectors, instrument /work, expose /metrics, and provide the smoke-test command.

global:
  scrape_interval: 5s
  evaluation_interval: 5s

scrape_configs:
  - job_name: "observable-service"
    static_configs:
      - targets: ["app:8080"]

What should you compare?

Your prometheus.yml should scrape app:8080 every five seconds under the observable-service job.

Your local stack now turns request behavior into queryable evidence. Next, you can make the delivery pipeline enforce those application and telemetry contracts.

Create the Health-Only Pipeline

Your local Prometheus metrics now explain request traffic, failures, and latency. The next question is whether every change can prove that the service still starts in a clean environment.

A green GitHub Actions result can create broad confidence from a narrow test. This step builds that narrow baseline with Docker Compose so you can examine exactly what the result proves.

In this step, get ready to:
  • Define a least-privilege workflow that starts the stack and checks application health.
  • Publish the local repository to an empty GitHub repository.
  • Inspect the passing workflow run to identify the evidence behind its green status.
Create the health-only workflow

A workflow is a YAML file that tells GitHub Actions which events start a job. It also defines the runner and the commands that job executes.

  • In the VS Code Explorer sidebar, create a folder named .github using the new-folder control.

You'll see the .github folder in the repository file tree.

  • Create a folder named workflows inside .github using the same control.

You'll now see the path .github/workflows in the file tree.

  • Create ci.yml inside .github/workflows using the new-file control.
  • Fill ci.yml by pasting this workflow:
name: observable-container-ci

on: [push, pull_request]

permissions:
  contents: read

jobs:
  health-check:
    runs-on: ubuntu-24.04
    steps:
      - name: Check out repository
        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false

      - name: Start observable stack
        run: docker compose up --build --wait

      - name: Run health check
        run: docker compose exec -T app /app --healthcheck

      - name: Show service logs
        if: always()
        run: docker compose logs --no-color

      - name: Tear down stack
        if: always()
        run: docker compose down -v --remove-orphans

What Does This Workflow Do?

  • The on setting starts the workflow for pushes and pull requests.
  • The permissions setting gives the workflow read-only access to repository contents.
  • The ubuntu-24.04 runner provides the Linux environment that executes the job.
  • The pinned actions/checkout reference keeps the downloaded action tied to one immutable commit.
  • The health step runs /app --healthcheck inside the application container.
  • The two if: always() conditions preserve logs and remove the stack after every outcome.
  • Save .github/workflows/ci.yml.
  • Confirm the VS Code Explorer shows ci.yml inside .github/workflows.

Workflow File Missing?

Confirm that ci.yml sits directly inside .github/workflows. GitHub only discovers workflow files stored in that directory.

Check that the filename ends with .yml.

Help me check why GitHub cannot discover my workflow file.

✔️ Awesome, I've got everything!

Solid start. Your repository now contains a health-only workflow with restricted permissions and guaranteed cleanup.

ⓧ I'd like to double check the full code

name: observable-container-ci

on: [push, pull_request]

permissions:
  contents: read

jobs:
  health-check:
    runs-on: ubuntu-24.04
    steps:
      - name: Check out repository
        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false

      - name: Start observable stack
        run: docker compose up --build --wait

      - name: Run health check
        run: docker compose exec -T app /app --healthcheck

      - name: Show service logs
        if: always()
        run: docker compose logs --no-color

      - name: Tear down stack
        if: always()
        run: docker compose down -v --remove-orphans
Publish the repository to GitHub

The local Git repository needs a remote home before the workflow can run. An empty GitHub repository lets you push the existing history without merging generated starter files.

This step uses public visibility. Standard GitHub-hosted runners are free and unlimited for public repositories.

  • Open GitHub's new repository page in your browser.
  • Choose your GitHub account as the repository owner.
  • Enter observable-container-pipeline as the repository name.
  • Choose Public as the repository visibility.
  • Leave every repository initialization option unselected.
  • Click Create repository.

Good progress. Your empty remote repository is ready to receive the complete local project.

  • Copy the remote repository URL from the Quick Setup page.

The copied URL points your local repository at its new GitHub destination.

  • Switch back to Visual Studio Code.
  • Open an integrated terminal from the top menu.
  • Use your copied repository URL in place of REMOTE-URL in the third command.

Before you push, do you expect this deliberately narrow workflow to finish with a green result?

  • Publish the local repository by running these commands:
git add .
git commit -m "First commit"
git remote add origin REMOTE-URL
git push -u origin main

What Do These Git Commands Do?

  • The first command stages every project file for the commit.
  • The second command records the initial repository snapshot.
  • The third command connects the local repository to GitHub.
  • The fourth command publishes the main branch and starts the workflow.
  • Wait for the push command to finish.
  • Refresh the GitHub repository page to confirm the project files appear on main.

Push Blocked?

Confirm that you replaced REMOTE-URL with the URL copied from your repository.

If GitHub requests authentication, complete the sign-in flow for the account that owns the repository.

Help me troubleshoot my first GitHub push.

Inspect what the green run proves

The push triggers the workflow because ci.yml listens for the push event. The run now gives you a concrete result to examine.

Before you inspect the details, what evidence do you expect to find behind the green status?

  • Click the Actions tab under the repository name.
  • Select the observable-container-ci workflow from the left sidebar.
  • Open the newest workflow run from the run list.
  • Select the health-check job from the workflow graph.
  • Expand the Run health check step.
  • Review the completed step names in the job log.

You'll see a green health-check job with successful startup, health, log collection, and teardown steps. The application container started and answered its health check.

The passing result contains no evidence for /work behavior, the /metrics contract, or a vulnerability policy. That limited proof is the intended shortfall of this first pipeline.

Your first pipeline is green. Next, you'll make that status depend on behavior, telemetry, and security evidence.

Enforce Contracts and Security

Your green GitHub Actions workflow currently proves that the container starts. It leaves application behavior untested.

This step turns the workflow into an enforcement point. It verifies behavior before applying a vulnerability policy.

In this step, get ready to:
  • Replace the health-only check with behavior checks plus a Prometheus metrics contract.
  • Add a fixed critical vulnerability gate with Trivy.
  • Push the hardened workflow to GitHub.
Strengthen the workflow contract

A CI contract turns expected behavior into a repeatable test. Your built-in smoke test covers health behavior, normal work, the intentional error response, plus the metrics endpoint.

  • In .github/workflows/ci.yml, rename the health-check: job key to test-and-scan:.
  • Find the step named Run health check.
  • Replace that step with this contract check:
      - name: Run behavior and observability contracts
        run: docker compose exec -T app /app --smoke-test

What does this contract prove?

  • The health check confirms that the service responds successfully.
  • The normal work check confirms that /work returns its expected success response.
  • The error contract confirms that the intentional failure returns status 500.
  • The metrics contract confirms that /metrics exposes demo_http_requests_total.
  • Save .github/workflows/ci.yml.
  • Return to the Linux terminal in observable-container-pipeline.
  • Rebuild the completed Compose stack by running this command:
docker compose up --build --wait

What does this command do?

Docker Compose rebuilds the application image before starting the services. The wait flag keeps the command active until the application is healthy.

Before you run the smoke test, predict whether the service will satisfy all four contracts.

  • Run the behavior plus observability checks with this command:
docker compose exec -T app /app --smoke-test

How does the smoke test report results?

The command calls each service path from inside the running application container. Each satisfied contract produces a line beginning with PASS.

The intentional error contract succeeds when it receives status 500. That response proves the failure path behaves as designed.

You should see four passing checks for health, normal work, the error contract, plus the metrics contract. The workflow now tests evidence that the health-only version ignored.

Smoke test not passing?

Check that the application service is healthy before running the test. Rebuild the stack if your container still uses an older application image.

Compare the workflow command with docker compose exec -T app /app --smoke-test.

Help me diagnose a failing smoke-test contract.

Add the vulnerability policy gate

The next workflow step scans the built image for fixed critical vulnerabilities. A matching finding blocks delivery with a non-zero result.

The first run downloads Trivy's vulnerability database before scanning. This database changes as new findings become available.

  • In .github/workflows/ci.yml, locate the behavior plus observability contract step.
  • Insert this scan step immediately below it:
      - name: Block fixed critical vulnerabilities
        run: |
          docker run --rm \
            -v /var/run/docker.sock:/var/run/docker.sock \
            aquasec/trivy:0.74.0 image \
            --scanners vuln \
            --exit-code 1 \
            --severity CRITICAL \
            --ignore-unfixed \
            observable-service:local

How does this gate work?

  • The Docker socket mount gives the Trivy container access to observable-service:local.
  • The --scanners vuln option selects vulnerability scanning.
  • The --severity CRITICAL option limits the gate to critical findings.
  • The --ignore-unfixed option excludes findings that have no available fix.
  • The --exit-code 1 option blocks the job when a matching finding exists.
  • Save .github/workflows/ci.yml.

Before you scan the local image, predict whether the current vulnerability policy will pass or block it.

  • Test the same policy locally by running this command:
docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  aquasec/trivy:0.74.0 image \
  --scanners vuln \
  --exit-code 1 \
  --severity CRITICAL \
  --ignore-unfixed \
  observable-service:local

How should I read the result?

A successful exit means the image has no detected fixed critical vulnerability under this policy. A blocked scan displays the finding that requires attention.

Both outcomes prove that the gate is working. The result can change later when Trivy publishes a newer vulnerability database.

Scan cannot inspect the image?

Confirm that observable-service:local exists from the Compose build. Check that Docker Engine is running on your Linux workstation.

Check that the Docker socket mount matches /var/run/docker.sock:/var/run/docker.sock.

Help me troubleshoot the Trivy container scan.

Review the hardened workflow

The existing if: always() guards keep diagnostic logs available after a failure. They also ensure that the Compose stack is removed after every run.

Use these tabs to compare your completed workflow before pushing it.

✔️ Awesome, I've got everything!

Your workflow now checks behavior, telemetry, plus fixed critical vulnerabilities. The cleanup steps remain protected after earlier failures.

ⓧ I'd like to double check the full code

name: observable-container-ci

on: [push, pull_request]

permissions:
  contents: read

jobs:
  test-and-scan:
    runs-on: ubuntu-24.04
    steps:
      - name: Check out repository
        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false

      - name: Start observable stack
        run: docker compose up --build --wait

      - name: Run behavior and observability contracts
        run: docker compose exec -T app /app --smoke-test

      - name: Block fixed critical vulnerabilities
        run: |
          docker run --rm \
            -v /var/run/docker.sock:/var/run/docker.sock \
            aquasec/trivy:0.74.0 image \
            --scanners vuln \
            --exit-code 1 \
            --severity CRITICAL \
            --ignore-unfixed \
            observable-service:local

      - name: Show service logs
        if: always()
        run: docker compose logs --no-color

      - name: Tear down stack
        if: always()
        run: docker compose down -v --remove-orphans
  • Stage .github/workflows/ci.yml with the same Git method you used in the previous step.
  • Commit the staged change with the message Enforce contracts and security.
  • Push the current main branch to your existing GitHub remote.

Before you inspect the run, predict which workflow step would stop delivery if the image contained a fixed critical vulnerability.

  • Return to the GitHub repository page from the previous step.
  • Select the Actions tab.
  • Open the newest workflow run.

You should see successful checks for health, normal work, intentional error behavior, plus the metrics contract. Trivy either passes the image policy or blocks the job with the detected critical finding displayed.

The log collection step runs after either outcome. The teardown step also runs after either outcome.

That is the pipeline hardened: every green run now carries behavioral, observability, plus security evidence.

Secret mission

Fire an Error-Ratio Alert

Your metrics can already reveal failed requests. Turn that data into a visible Prometheus alert that fires when more than 10 percent of recent work requests fail.

Clean Up Your Resources

Clean Up Your Resources

Your local Docker Compose stack has no hosting charge. Decide whether to keep it active, pause it while preserving metrics, or delete its runtime state.

Cost warning

Standard GitHub Actions runners are free and unlimited for public repositories.

Private repositories consume included minutes first. GitHub charges per minute after that allowance is exhausted.

Resources you used:

  • Local application and Prometheus containers with their Compose network.
  • The named prometheus-data volume containing scraped metrics.
  • The local application image named observable-service:local.
  • The GitHub repository with its workflow runs on main.

Keep everything running

No action needed. Choose this if you want to keep testing queries or generating traffic.

  • Leave the Compose stack running so Prometheus continues scraping the service.
  • Keep the GitHub repository on main so the hardened workflow and alert rule remain available.

Pause - I'll come back to this later

Pausing removes the running containers and Compose network. The named prometheus-data volume preserves your collected metrics.

  • Check that the Linux terminal from earlier is still inside the observable-container-pipeline folder.
  • Pause the Compose stack by running this command:
docker compose down

What Does This Command Do?

The docker compose down command stops and removes the application container. It also stops and removes the Prometheus container.

The Compose network is removed. The prometheus-data volume remains available for your next start.

  • Confirm the terminal reports that the application container, Prometheus container, and Compose network were removed.

Compose Cannot Find the Project?

Make sure the terminal is inside the observable-container-pipeline folder containing compose.yaml.

If the command still fails, help me diagnose this Docker Compose cleanup problem.

Delete - I don't want to use this again

Deleting the named volume permanently removes the metrics you collected. Your source files and GitHub repository stay untouched by this local cleanup command.

  • Check that the Linux terminal from earlier is still inside the observable-container-pipeline folder.
  • Delete the local Compose runtime resources by running this command:
docker compose down -v --remove-orphans

What Gets Deleted?

  • The base command removes the application container, Prometheus container, and Compose network.
  • The -v flag removes the named prometheus-data volume.
  • The --remove-orphans flag removes containers for services that no longer appear in compose.yaml.
  • Confirm the terminal reports that the containers, network, and named volume were removed.
  • Remove the local application image named observable-service:local through the image-management tool you use with Docker.
  • Confirm your local image manager no longer lists observable-service:local.

Repository deletion is permanent, so keeping the remote copy is the safer default. Remove it only when you no longer need the project history.

  • Delete the GitHub repository separately using GitHub's repository deletion controls only if you no longer want the remote copy.

Cleanup Command Failing?

Run the command from the folder containing compose.yaml so Docker Compose can identify the project resources.

If cleanup still fails, help me diagnose the remaining Compose resources.

Nice Work!

Nice Work!

Excellent work! Your delivery pipeline now proves how the service behaves before applying a security policy.

Your repository also turns recent failures into a firing alert that you can see in Prometheus.

You've learned how to:

  • Build a containerized Go service in a non-root scratch image. Its endpoints produce normal responses. They also produce slow responses. They can return intentional errors. Its structured JSON logs capture each request.
  • Record request totals with Prometheus. Measure request durations with histograms. Use PromQL to calculate request rate. Use it to calculate error ratio. Use it to estimate p95 latency.
  • Enforce behavior contracts in GitHub Actions. Enforce telemetry contracts in the same workflow. Apply least-privilege permissions. Pin the checkout action to an immutable commit. Block fixed critical vulnerabilities with Trivy. Collect logs after every outcome. Tear down the stack after every outcome.
  • Complete the Secret Mission by defining HighErrorRatio. Trigger the alert when recent 5xx responses exceed 10 percent.

Ready to quiz yourself?