Zero-Cost Federal Platform Rehearsal Rig

Build a three-namespace Kubernetes rig mirroring a federal data platform.

Introduction

30 Second Summary

Your team is waiting on cloud provisioning it does not control. Weeks pass. The architecture decisions pile up untested. Meanwhile, the program needs proof that your delivery pipeline works before it will fund the next phase.

In this project, you will build a zero-cost rehearsal rig on your workstation that mirrors a federal data platform's three promotion zones, deploys an event-driven object store, a version-pinned database, and a pipeline host, then ships the whole thing through a private repo with the one governance document that prevents anyone from mistaking rehearsal results for production readiness.

What You'll Build

One command on a clean clone brings up a full three-zone Kubernetes environment where a file upload fires a notification event, a database accepts a migration, and a pipeline runner pulls from the public registry to prove outbound connectivity, while a single-page fidelity map tells the next person who reads it exactly what this rig cannot prove.

By the end of this project, you'll have:

  • A three-namespace Kubernetes cluster with Kustomize overlays that mirror your dev/int/prod promotion model, brought up from one command and delivered through a private GitHub repo with branch-per-story PRs, a lint workflow, and a tagged release tracked in Linear.
  • An object store (SeaweedFS) that fires the moment a file arrives, plus PostgreSQL 18 accepting migrations pinned to the exact version your production will run.
  • A self-managed GitLab CE instance with a runner executing a pipeline that proves the outbound network path, backed by four decision records justifying every choice and a fidelity map with six standing gaps, each owned and dated, that prevents rehearsal results from being quoted upward as readiness.
  • Secret Mission: a promotion pipeline that applies Kustomize overlays in sequence through GitLab stages, proving the promotion shape works end to end through the CI/CD system.

Are there any prerequisites?

You need a Windows machine with WSL2 capability, a free GitHub account, and a free Linear account. Familiarity with Kubernetes concepts and YAML at a principal-engineer level is expected.

Everything else (Docker Engine, kind, kubectl, Kustomize) is installed from scratch inside the project. No cloud accounts with a credit card are required. No data leaves your workstation except to GitHub and Linear on the free tier.

Before We Start

Before any tool is installed or any file is created, this is the moment to commit to what you are building and why it matters on your program today.

Design Phase and Delivery Scaffolding

A rehearsal rig built without design records is a personal experiment nobody else can evaluate. The four decision records justify the stack before anything runs. The topology diagram makes the shape visible before a single container starts.

This step establishes the fixed order: design first, then delivery scaffolding (the GitHub repo and Linear project that give every subsequent step a branch, a ticket, and a merge that closes it), then tools. By the end, every future step has a home.

In this step, get ready to:
  • Write four decision records and a topology diagram that justify the stack before it runs.
  • Create a private GitHub repository with a lint workflow and a Linear project with five stories.
  • Install Docker Engine, kind, kubectl, and Kustomize under WSL2 and verify each tool.
Write the decision records and topology diagram

Decision records lock in the reasoning behind your choices so the next person (or future you) can see what was considered and why. Each ADR follows a fixed structure: context, decision, alternatives, tradeoff, reversal trigger.

  • Create the directory structure for your documentation by running these commands:
mkdir -p docs/decisions
  • Create the first decision record by saving the following content into docs/decisions/adr-001-kubernetes-distribution.md:
# ADR-001: Kubernetes Distribution

## Status
Accepted

## Context
The rehearsal rig needs a local Kubernetes cluster mirroring three promotion zones (dev, int, prod).

## Decision
Use kind v0.32.0.

## Alternatives Considered
- k3s: requires systemd, adds complexity under WSL2.
- k3d: k3s-in-Docker, but kind has broader CI adoption and simpler multi-node config.

## Tradeoff Accepted
Three namespaces in one cluster, not three clusters. No per-zone admission policy. No cluster-scoped separation.

## Reversal Trigger
When admission policy must be tested per zone.

What does this record capture?

ADR-001 names the single biggest architectural shortcut in the rig: one cluster with three namespaces instead of three separate clusters. This tradeoff reduces resource consumption but means admission policies cannot differ per zone. The reversal trigger tells a future reader exactly when to abandon this choice.

  • Create the second decision record by saving the following content into docs/decisions/adr-002-object-store.md:
# ADR-002: Object Store

## Status
Accepted

## Context
The rig needs an object store with event notification capability.

## Decision
Use SeaweedFS (Apache 2.0, actively maintained).

## Alternatives Considered
- MinIO: archived April 2026, unmaintained since February 2026.

## Tradeoff Accepted
Neither free store carries the S3 notification interface (PutBucketNotificationConfiguration). Event shape (create/update/delete/rename) transfers. Interface does not. Consumer code requires adaptation at integration.

## Reversal Trigger
If a maintained fork restores the S3 notification interface.
  • Confirm both files exist by running:
ls docs/decisions/

You should see adr-001-kubernetes-distribution.md and adr-002-object-store.md listed.

  • Create the third decision record by saving the following content into docs/decisions/adr-003-fidelity-map-governance.md:
# ADR-003: Fidelity Map Governance

## Status
Accepted

## Context
Rehearsal results risk being cited as production readiness without explicit bounds.

## Decision
The fidelity map is a numbered deliverable with owner and review date, not an inline caveat.

## Alternatives Considered
- Inline caveats scattered through documentation.

## Tradeoff Accepted
More overhead to maintain. Far more durable as a governance artifact.

## Reversal Trigger
Never.
  • Create the fourth decision record by saving the following content into docs/decisions/adr-004-scope-exclusions.md:
# ADR-004: Scope Exclusions

## Status
Accepted

## Context
The rig's scope must be bounded explicitly.

## Decision
The rig deliberately will not attempt:
- Cross-account replication
- Encryption-key semantics
- Impact-level baselines
- Platform guardrails
- The authorization process (RMF Assess and Authorize, or successor construct)

## Alternatives Considered
- Attempt partial coverage of excluded items.

## Tradeoff Accepted
Narrower scope but honest about what is not proven.

## Reversal Trigger
Explicit program requirement with resourcing attached.
  • Confirm all four ADRs exist by running:
ls docs/decisions/

You should see all four files: adr-001-kubernetes-distribution.md, adr-002-object-store.md, adr-003-fidelity-map-governance.md, and adr-004-scope-exclusions.md.

Why four separate records instead of one design doc?

Each ADR is independently reviewable and independently reversible. When circumstances change (they will), the team reverses one decision without re-reading a 20-page design document. The reversal trigger on each record tells you exactly when to reopen it.

  • Create the topology diagram by saving the following content into docs/topology.md:
# Topology: Rehearsal Rig

## Components

```
Host Machine (WSL2) + Docker Engine
├── kind cluster: rehearsal-rig
│   ├── Namespace: dev
│   │   ├── SeaweedFS (master, volume, filer, S3 gateway)
│   │   └── PostgreSQL 18 (StatefulSet)
│   ├── Namespace: int
│   │   └── (receives promoted manifests)
│   └── Namespace: prod
│       └── (receives promoted manifests)
├── GitLab CE (Docker Compose, port 8929)
│   └── GitLab Runner (alpine, registered)
└── GitHub Actions
    └── lint workflow (validates YAML on PRs)
```

What does this diagram show?

The topology makes visible what lives where. The kind cluster holds the three namespaces that mirror promotion zones. SeaweedFS and PostgreSQL live inside the cluster as pods. GitLab CE runs on the host alongside the cluster via Docker Compose. GitHub Actions runs remotely, triggered by pull requests.

  • Confirm the topology file exists by running:
ls docs/topology.md

You should see docs/topology.md listed.

Files not appearing?

Make sure you are running commands from the project root directory. If you used a different editor to create the files, confirm they saved to the correct path relative to your working directory.

Help me troubleshoot missing files in my project directory.

Create the GitHub repository and Linear project

The design is written. Now it needs a home where every change is tracked, reviewed, and merged through a pull request that closes a ticket.

  • Open your browser and go to github.com/new.
  • Set the repository name to something descriptive (e.g. rehearsal-rig).
  • Select Private for visibility.
  • Check Add a README file.
  • Click Create repository.

Why private?

This is a federal delivery asset. Private keeps it off the public internet. On the free tier, private repos still get 2,000 GitHub Actions minutes per month, which is more than enough for a YAML lint workflow.

  • In your new repository on GitHub, click Add file then Create new file.
  • Type .github/workflows/lint.yml as the file path (GitHub creates the directories automatically).
  • Paste the following workflow content:
name: Lint YAML

on:
  pull_request:
  push:
    branches: [main]

jobs:
  lint-yaml:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install yamllint
        run: pip install yamllint
      - name: Lint manifests
        run: |
          if [ -d manifests ]; then yamllint manifests/; fi
      - name: Validate Compose file
        run: |
          if [ -f gitlab/docker-compose.yml ]; then
            docker compose -f gitlab/docker-compose.yml config --quiet
          fi

What does this workflow do?

The workflow triggers on every pull request and every push to main. It installs yamllint to check YAML syntax in the manifests/ directory. It validates the Docker Compose file using docker compose config --quiet, which exits non-zero if the file is malformed.

The conditional checks (if [ -d manifests ] and if [ -f gitlab/docker-compose.yml ]) allow the workflow to pass cleanly before those directories exist. Once you add manifests in later steps, any YAML error will fail the build.

  • Click Commit changes to commit directly to main.
  • Click the Actions tab in your repository.

You should see the "Lint YAML" workflow has run and shows a green checkmark. It passed because neither manifests/ nor gitlab/docker-compose.yml exist yet, so both conditional steps exit cleanly.

  • Add a note to the repository README. In the repo, click README.md, click the edit pencil, and add this line at the bottom:
## Note
Branch protection and code scanning are stretch goals requiring a paid GitHub plan. Stated, not assumed.
  • Commit the README change to main.

Now set up the delivery tracking. Open linear.app in your browser and sign in (or create a free account).

  • Create a new project named Rehearsal Rig v1.0.
  • Create one epic with the same name: Rehearsal Rig v1.0.
  • Under that epic, create five stories:
  • Story 1: Cluster foundation and object store.
  • Story 2: Database and migration.
  • Story 3: Pipeline host and runner.
  • Story 4: Fidelity map and documentation.
  • Story 5: Tagged release and completion gate.

What does each story represent?

Each story maps to a branch and a PR that closes it on merge. This is the delivery contract: no work ships without a ticket, a review, and a close. By the end of the project, all five stories are closed and the epic is done.

🙋‍♀️ Not sure where to create epics in Linear?

In Linear, projects are the closest equivalent to epics. Create a project, then add issues (stories) inside it. The free tier supports unlimited issues and projects.

Help me set up my Linear project for the rehearsal rig.

Install the toolchain and open the first story branch

The design is committed and the delivery scaffolding is in place. Now install the four tools the rig depends on: Docker Engine with the Compose plugin, kind, kubectl, and Kustomize.

Why not Docker Desktop?

Docker Desktop's Subscription Service Agreement (section 4.2) bars government entities from using it without a paid subscription. Docker Engine with the Compose plugin under WSL2 is unaffected by this restriction because it remains open source under Apache 2.0.

  • Open your WSL2 terminal.
  • Install Docker Engine by running this command:
curl -fsSL https://get.docker.com | sh
  • Add your user to the docker group so you can run Docker without sudo:
sudo usermod -aG docker $USER
  • Close your WSL2 terminal and reopen it for the group change to take effect.
  • Verify Docker Engine and the Compose plugin are working by running:
docker --version
docker compose version

You should see version numbers for both Docker Engine and the Compose plugin.

Permission denied when running docker?

If you see a permission error, your terminal session has not picked up the new group membership yet. Close the WSL2 window completely and reopen it. Run groups to confirm docker appears in the list.

Help me fix Docker permission errors in WSL2.

  • Install kind v0.32.0 by downloading the release binary:
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.32.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind
  • Verify the installed version by running:
kind version

✔️ I see kind v0.32.0

The output shows kind v0.32.0. You are on the correct version. Continue to the next install.

ⓧ I see an older version

An older version means a previously installed binary is taking priority in your PATH. Remove it and reinstall:

which kind
sudo rm $(which kind)
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.32.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind
kind version

Confirm the output now shows kind v0.32.0.

ⓧ Command not found

The binary did not land in your PATH. Check that /usr/local/bin is in your PATH by running:

echo $PATH

If /usr/local/bin is missing, add it to your shell profile (~/.bashrc or ~/.zshrc):

echo 'export PATH=/usr/local/bin:$PATH' >> ~/.bashrc
source ~/.bashrc
kind version

Confirm the output shows kind v0.32.0.

  • Install kubectl by running:
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv kubectl /usr/local/bin/kubectl
  • Verify kubectl is available by running:
kubectl version --client

You should see a client version output confirming kubectl is installed.

  • Install Kustomize v5.8.1 by downloading the release archive:
curl -LO https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.8.1/kustomize_v5.8.1_linux_amd64.tar.gz
tar xzf kustomize_v5.8.1_linux_amd64.tar.gz
sudo mv kustomize /usr/local/bin/kustomize
rm kustomize_v5.8.1_linux_amd64.tar.gz
  • Verify the installed version by running:
kustomize version

✔️ I see v5.8.1

The output shows v5.8.1. Kustomize is ready. Continue below.

ⓧ I see an older version

A previously installed Kustomize binary is shadowing the new one. Remove it and reinstall:

which kustomize
sudo rm $(which kustomize)
curl -LO https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.8.1/kustomize_v5.8.1_linux_amd64.tar.gz
tar xzf kustomize_v5.8.1_linux_amd64.tar.gz
sudo mv kustomize /usr/local/bin/kustomize
rm kustomize_v5.8.1_linux_amd64.tar.gz
kustomize version

Confirm the output now shows v5.8.1.

ⓧ Command not found

The binary did not land in your PATH. Confirm /usr/local/bin is on your PATH:

echo $PATH

If missing, add it the same way you did for kind (append to ~/.bashrc then source it). Then run kustomize version again.

All four tools are installed. Now clone your repo locally, commit the design docs, and open the first story branch.

  • Clone your GitHub repository into your WSL2 home by running:
cd ~/Desktop
git clone [[REPO_URL="https://github.com/YOUR_USERNAME/rehearsal-rig.git"]]
cd rehearsal-rig
  • Copy the docs/ directory you created earlier into the cloned repo (if you created it outside the repo). If you created the files directly inside the cloned repo, skip this step.
  • Commit and push the decision records and topology by running:
git add docs/
git commit -m "Add decision records and topology diagram"
git push origin main
  • Open the first story branch by running:
git checkout -b story-1-cluster-object-store

You are now on the story-1-cluster-object-store branch, ready to build in the next step.

Before you move on, predict: if you check the GitHub Actions tab now, what should you see for the lint workflow on main?

  • Open the Actions tab in your GitHub repository.

The "Lint YAML" workflow shows a green checkmark on both the workflow commit and the docs commit. Both passed because the conditional steps found nothing to lint and exited cleanly.

  • Run the final tool verification:
kind version && kustomize version

You should see kind v0.32.0 followed by v5.8.1. Both tools are at their pinned versions.

Lint workflow shows red?

If the workflow failed on the docs commit, check whether yamllint found issues in any .yml or .yaml file you accidentally placed in the repo root. The workflow's conditional only protects the manifests/ directory check. Files at the root with .yaml extension are not linted by this workflow, so the issue is likely an Actions syntax error in the workflow file itself. Click the failed run and read the error message.

Help me debug my failing GitHub Actions lint workflow.

Your design is committed, your delivery scaffolding is live, and your tools are verified. Next up, you will create the kind cluster and deploy SeaweedFS to prove that arrival events fire end to end.

Cluster Foundation and Object Store

Your decision records are committed, your delivery scaffolding is live, and your first story branch is open. Now the build starts.

The cluster and the object store belong to one story because the event-notification proof depends on both. Three namespaces establish the promotion shape (dev, int, prod). SeaweedFS proves arrival events via its filer metadata subscription. Along the way you will hit a designed gap: the S3 notification API you might expect is explicitly absent, and the rig must say so plainly.

In this step, get ready to:
  • Create the kind cluster with three namespaces and the manifest skeleton.
  • Deploy SeaweedFS and prove event notification fires on object arrival.
  • Record the first fidelity-map gap and merge the story PR.
Create the cluster and manifest skeleton

The kind cluster is a single Docker container running the Kubernetes control plane. You configure it with a YAML file that sets the cluster name and node roles.

  • Create the cluster configuration file by running:
# kind-config.yaml
# Defines a single control-plane node cluster named rehearsal-rig
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: rehearsal-rig
nodes:
- role: control-plane

What does this configuration do?

The kind: Cluster and apiVersion: kind.x-k8s.io/v1alpha4 fields tell kind how to parse the file.

The name: rehearsal-rig field sets the cluster context name so kubectl commands target this specific cluster. One control-plane node is all the rig needs. Three namespaces, not three clusters, is the deliberate tradeoff recorded in ADR-001.

  • Create the cluster by running:
kind create cluster --config kind-config.yaml --wait 5m

What should I see?

kind prints progress lines as it ensures the node image, prepares the node, writes configuration, starts the control-plane, installs CNI, and installs StorageClass. It finishes with a line setting the kubectl context.

  • Create the three promotion-zone namespaces by running:
kubectl create namespace dev
kubectl create namespace int
kubectl create namespace prod
  • Verify the namespaces exist by running:
kubectl get ns

You should see dev, int, and prod alongside the system namespaces (default, kube-system, kube-public, kube-node-lease, local-path-storage).

Cluster creation failing?

Ensure Docker Engine is running inside WSL2. Run docker info to confirm. If you see a connection error, start the Docker daemon first.

If kind reports the cluster already exists, delete it with kind delete cluster --name rehearsal-rig and re-run the create command.

help me troubleshoot kind cluster creation failures

  • Create the manifest directory skeleton by running:
mkdir -p manifests/base/seaweedfs
mkdir -p manifests/overlays/dev
mkdir -p manifests/overlays/int
mkdir -p manifests/overlays/prod
  • Confirm the directories exist by running:
find manifests -type d

You should see manifests/base/seaweedfs and the three overlay directories listed.

Why this directory layout?

Kustomize uses a base directory for shared manifests and overlay directories for per-zone differences. Each overlay sets only what differs (namespace, replica count, resource limits). This is the "promotion shape fixed once" principle.

Splitting manifests into their own repository is a later decision, named here rather than assumed.

Deploy SeaweedFS and prove event notification

SeaweedFS needs four components: a master (coordinates the cluster), a volume server (stores data), a filer (provides a file-system interface), and an S3 gateway (exposes the S3 API). Each gets its own Deployment and Service manifest.

  • Create the master deployment manifest by saving this file:
# Deployment and Service for the SeaweedFS master node
apiVersion: apps/v1
kind: Deployment
metadata:
  name: seaweedfs-master
spec:
  replicas: 1
  selector:
    matchLabels:
      app: seaweedfs-master
  template:
    metadata:
      labels:
        app: seaweedfs-master
    spec:
      containers:
      - name: master
        image: chrislusf/seaweedfs
        args: ["master"]
        ports:
        - containerPort: 9333
---
apiVersion: v1
kind: Service
metadata:
  name: seaweedfs-master
spec:
  selector:
    app: seaweedfs-master
  ports:
  - port: 9333
    targetPort: 9333

What does this manifest do?

The master coordinates volume placement across the cluster. Port 9333 is its HTTP/gRPC endpoint. The Service gives other pods a stable DNS name (seaweedfs-master) to reach it.

  • Create the volume server manifest by saving this file:
# Deployment and Service for the SeaweedFS volume server
apiVersion: apps/v1
kind: Deployment
metadata:
  name: seaweedfs-volume
spec:
  replicas: 1
  selector:
    matchLabels:
      app: seaweedfs-volume
  template:
    metadata:
      labels:
        app: seaweedfs-volume
    spec:
      containers:
      - name: volume
        image: chrislusf/seaweedfs
        args: ["volume", "-mserver=seaweedfs-master:9333", "-port=8080"]
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: seaweedfs-volume
spec:
  selector:
    app: seaweedfs-volume
  ports:
  - port: 8080
    targetPort: 8080
  • Create the filer manifest by saving this file:
# Deployment and Service for the SeaweedFS filer
apiVersion: apps/v1
kind: Deployment
metadata:
  name: seaweedfs-filer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: seaweedfs-filer
  template:
    metadata:
      labels:
        app: seaweedfs-filer
    spec:
      containers:
      - name: filer
        image: chrislusf/seaweedfs
        args: ["filer", "-master=seaweedfs-master:9333"]
        ports:
        - containerPort: 8888
        - containerPort: 18888
---
apiVersion: v1
kind: Service
metadata:
  name: seaweedfs-filer
spec:
  selector:
    app: seaweedfs-filer
  ports:
  - name: http
    port: 8888
    targetPort: 8888
  - name: grpc
    port: 18888
    targetPort: 18888

Why two ports on the filer?

Port 8888 serves HTTP requests. Port 18888 is the gRPC endpoint that weed filer.meta.tail connects to for streaming metadata change events. This gRPC stream is how you will prove event notification works.

  • Create the S3 gateway manifest by saving this file:
# Deployment and Service for the SeaweedFS S3 gateway
apiVersion: apps/v1
kind: Deployment
metadata:
  name: seaweedfs-s3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: seaweedfs-s3
  template:
    metadata:
      labels:
        app: seaweedfs-s3
    spec:
      containers:
      - name: s3
        image: chrislusf/seaweedfs
        args: ["s3", "-filer=seaweedfs-filer:8888"]
        ports:
        - containerPort: 8333
---
apiVersion: v1
kind: Service
metadata:
  name: seaweedfs-s3
spec:
  selector:
    app: seaweedfs-s3
  ports:
  - port: 8333
    targetPort: 8333
  • Create the base kustomization.yaml that ties these resources together:
# Lists all SeaweedFS resources for the base layer
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- master-deployment.yaml
- volume-deployment.yaml
- filer-deployment.yaml
- s3-gateway-deployment.yaml
  • Create the dev overlay that sets the namespace:
# Overlay for the dev namespace
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: dev
resources:
- ../../base/seaweedfs
  • Create the int and prod overlays (identical structure, different namespace). For manifests/overlays/int/kustomization.yaml:
# Overlay for the int namespace
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: int
resources:
- ../../base/seaweedfs
  • For manifests/overlays/prod/kustomization.yaml:
# Overlay for the prod namespace
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: prod
resources:
- ../../base/seaweedfs
  • Apply the dev overlay to deploy SeaweedFS into the dev namespace by running:
kustomize build manifests/overlays/dev | kubectl apply -f -
  • Wait for pods to become ready by running:
kubectl get pods -n dev -w

Wait until all four pods (master, volume, filer, s3) show 1/1 Running. Press Ctrl+C to stop watching.

Pods stuck in CrashLoopBackOff?

The volume server depends on the master being ready. If it crashes, wait 30 seconds and check again. kind pulls images from Docker Hub on first deploy, which can take a minute on slow connections.

Run kubectl logs -n dev <pod-name> to see the specific error from the failing container.

help me debug SeaweedFS pods that are not starting in my kind cluster

  • Port-forward the S3 gateway to your local machine by running:
kubectl port-forward -n dev svc/seaweedfs-s3 8333:8333 &
  • Upload a test object using the AWS CLI by running:
aws --endpoint-url http://localhost:8333 s3 mb s3://test-bucket
aws --endpoint-url http://localhost:8333 s3 cp /etc/hostname s3://test-bucket/hello.txt

You should see output confirming the bucket was created and the file was uploaded.

The object store works. Now the natural next step is to configure event notifications so you know when objects arrive. If this were AWS S3, you would call PutBucketNotificationConfiguration. Before you run this, do you think it will succeed against SeaweedFS?

  • Try configuring S3 event notifications by running:
aws --endpoint-url http://localhost:8333 s3api put-bucket-notification-configuration \
  --bucket test-bucket \
  --notification-configuration '{"TopicConfigurations":[]}'

You get an error. SeaweedFS explicitly does not implement PutBucketNotificationConfiguration or GetBucketNotificationConfiguration. This is the designed gap: the S3 notification interface does not transfer, even though the underlying event data does. You will record this in the fidelity map shortly.

Why this matters for the program

On production AWS, your consumer code would call PutBucketNotificationConfiguration to route events to SQS or Lambda. Here, the event shape (create, update, delete, rename with path and timestamp) exists but must be consumed through SeaweedFS's own gRPC interface. Consumer code will require adaptation at integration.

  • Open a second terminal. Port-forward the filer gRPC port by running:
kubectl port-forward -n dev svc/seaweedfs-filer 18888:18888 &
  • Subscribe to filer metadata events by running:
weed filer.meta.tail -filer=localhost:8888

What does this command do?

This connects to the filer's SubscribeMetadata gRPC stream and prints every file-system event as it happens. It is the SeaweedFS equivalent of receiving S3 event notifications.

  • In your first terminal, upload another test file by running:
echo "event-test" | aws --endpoint-url http://localhost:8333 s3 cp - s3://test-bucket/event-proof.txt

Switch to the terminal running weed filer.meta.tail. You should see a Create event with the path /buckets/test-bucket/event-proof.txt printed. This proves the filer notification subsystem fires on object arrival.

No event appearing?

Ensure the filer gRPC port-forward is still active (port 18888). If weed filer.meta.tail reports a connection error, restart the port-forward and re-run the tail command.

Also confirm you are uploading via the S3 gateway (port 8333), not directly to the filer.

help me troubleshoot SeaweedFS filer metadata events not appearing

Record the gap, push, and merge

The proof is done. The event shape transfers but the S3 notification interface does not. This gap belongs in the fidelity map as a governed, numbered entry with an owner and a review date.

  • Create docs/fidelity-map.md with the first gap entry:
# Fidelity Map

## Gap 1: SeaweedFS filer notification interface

SeaweedFS filer notification uses SubscribeMetadata gRPC, not S3
PutBucketNotificationConfiguration. Event shape (create/update/delete/rename
with path and timestamp) transfers. Interface does not. Consumer code will
require adaptation at integration.

- **Owner:** platform engineer
- **Review date:** [set at project start + 90 days]
- **Exit condition:** production object store confirmed with S3 notification
  API support

Why a fidelity map entry and not just a comment?

A comment gets buried. A numbered gap with an owner and a review date is a governance artifact. It travels with the repository. An entry leaves the map only with evidence and a date, never quietly.

  • Stop the port-forwards (bring the background jobs to the foreground and press Ctrl+C, or run kill %1 %2).
  • Stage and commit all new files by running:
git add kind-config.yaml manifests/ docs/fidelity-map.md
git commit -m "feat: cluster foundation and object store with event proof"
  • Push the branch by running:
git push origin HEAD
  • Open a pull request in GitHub targeting main.
  • Confirm the lint workflow passes on the PR.
  • Merge the PR, closing the Linear ticket for story 1.

Before you check the final state, predict: how many namespaces does kubectl get ns show now?

  • Verify the end state by running:
kubectl get ns
kubectl get pods -n dev

You should see dev, int, and prod in the namespace list, and four Running pods in the dev namespace. The PR is merged, the Linear ticket is closed, and the fidelity map records its first standing gap.

Lint workflow failing on the PR?

The workflow validates YAML in manifests/. Common issues: trailing whitespace, tabs instead of spaces, or a missing newline at end of file. Run yamllint manifests/ locally to see the exact error before pushing a fix.

help me fix yamllint errors in my Kubernetes manifests

✔️ Awesome, I've got everything!

Great. Confirm you have saved all manifest files, committed to your story branch, and merged the PR. Your Linear board should show story 1 closed.

ⓧ I'd like to double check the full code

Scroll up and confirm these files are present in your project, in this order:

  • kind-config.yaml (cluster config with name: rehearsal-rig and one control-plane node).
  • manifests/base/seaweedfs/master-deployment.yaml (Deployment + Service, port 9333).
  • manifests/base/seaweedfs/volume-deployment.yaml (Deployment + Service, port 8080).
  • manifests/base/seaweedfs/filer-deployment.yaml (Deployment + Service, ports 8888 and 18888).
  • manifests/base/seaweedfs/s3-gateway-deployment.yaml (Deployment + Service, port 8333).
  • manifests/base/seaweedfs/kustomization.yaml (lists all four deployment files).
  • manifests/overlays/dev/kustomization.yaml (namespace: dev, references ../../base/seaweedfs).
  • manifests/overlays/int/kustomization.yaml (namespace: int, references ../../base/seaweedfs).
  • manifests/overlays/prod/kustomization.yaml (namespace: prod, references ../../base/seaweedfs).
  • docs/fidelity-map.md (Gap 1 with owner and review date fields).

The cluster foundation is solid and the object store fires on arrival. Next up, you will deploy PostgreSQL for version-fixed migration testing and stand up GitLab CE to prove the outbound network path.

Database, Pipeline Host, and Runner Proof

Your cluster and object store are proven. Two more pieces need to stand on their own before the rig earns its keep.

PostgreSQL 18 must accept a version-fixed migration so the team can smoke-test schema tooling against the exact database version production will run. GitLab CE must execute a pipeline whose runner pulls from the public registry, proving the outbound network path that fails first on a locked-down workstation.

In this step, get ready to:
  • Deploy PostgreSQL 18 with least-privilege credentials and verify a schema migration.
  • Deploy GitLab CE and a registered runner that executes a trivial pipeline.
  • Record observations and close Linear stories 2 and 3.
Deploy PostgreSQL 18 and run a schema migration

You are on the story-2 branch. The database proof needs a StatefulSet pinned to postgres:18, a Secret enforcing least privilege, and a migration Job that exercises psql against the running pod.

  • Create the postgres manifests directory by running:
mkdir -p manifests/base/postgres
  • Create the Secret that defines two database users by saving this file as manifests/base/postgres/secret.yaml:
apiVersion: v1
kind: Secret
metadata:
  name: postgres-credentials
type: Opaque
stringData:
  # Migration user owns schema changes
  POSTGRES_USER: migration_user
  POSTGRES_PASSWORD: migrate-rehearsal-2026
  # Application user has read/write only
  APP_USER: app_user
  APP_PASSWORD: app-rehearsal-2026

What does this Secret do?

The Secret defines two users. migration_user owns DDL changes. app_user gets read/write only. No superuser is exposed to the pod.

This is least privilege: the migration user can alter the schema, but the application user that runs at runtime cannot.

  • Confirm the new directory and file exist by running:
ls manifests/base/postgres/

You should see secret.yaml listed.

  • Create the ConfigMap with minimal tuning values at manifests/base/postgres/configmap.yaml:
apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-config
data:
  # Minimal tuning for a rehearsal workload
  postgresql.conf: |
    shared_buffers = 128MB
    max_connections = 20
    log_statement = 'all'

Why tune a rehearsal database?

The point is not performance. log_statement = 'all' makes every query visible during debugging. The explicit ConfigMap also proves the mount path works, exactly as it would in a production StatefulSet.

  • Create the StatefulSet at manifests/base/postgres/statefulset.yaml:
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          # Version-pinned to match production
          image: postgres:18
          ports:
            - containerPort: 5432
          envFrom:
            - secretRef:
                name: postgres-credentials
          volumeMounts:
            - name: config
              mountPath: /etc/postgresql/postgresql.conf
              subPath: postgresql.conf
      volumes:
        - name: config
          configMap:
            name: postgres-config

What does this StatefulSet do?

The image tag postgres:18 pins the database to the same major version production will run. The migration smoke test only means something if it runs against the exact version.

The envFrom block injects credentials from the Secret. The volume mount places your tuned postgresql.conf inside the container.

  • Create a headless Service so the migration Job can reach the pod by DNS name. Save this as manifests/base/postgres/service.yaml:
apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  # Headless service for StatefulSet DNS
  clusterIP: None
  selector:
    app: postgres
  ports:
    - port: 5432
      targetPort: 5432
  • Create the Kustomize base file at manifests/base/postgres/kustomization.yaml:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - secret.yaml
  - configmap.yaml
  - service.yaml
  - statefulset.yaml
  • In manifests/overlays/dev/kustomization.yaml, find the resources: list. It currently looks like this:
resources:
  - ../../base/seaweedfs
  • Add the postgres base so the list now reads:
resources:
  - ../../base/seaweedfs
  - ../../base/postgres
  • Apply the updated dev overlay and wait for the postgres pod to become ready by running:
kustomize build manifests/overlays/dev | kubectl apply -f -
kubectl wait --for=condition=Ready pod -l app=postgres -n dev --timeout=90s

You should see pod/postgres-0 condition met confirming the database is running in the dev namespace.

Pod stuck in Pending or CrashLoopBackOff?

  • Run kubectl describe pod postgres-0 -n dev and check the Events section. A FailedMount event means the ConfigMap name in the StatefulSet does not match the ConfigMap metadata name.
  • If the pod is CrashLoopBackOff, check logs with kubectl logs postgres-0 -n dev. A password-related error means the Secret is missing or the key names do not match what the postgres image expects.
  • help me debug my postgres pod not starting in kind

Now apply the migration. The Job will connect to the running postgres pod via the headless Service and create the schema.

  • Create the migration Job at manifests/base/postgres/migration-job.yaml:
apiVersion: batch/v1
kind: Job
metadata:
  name: pg-migration
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: postgres:18
          env:
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-credentials
                  key: POSTGRES_PASSWORD
          command: ["/bin/sh", "-c"]
          args:
            - |
              psql -h postgres -U migration_user -d postgres -c "
              CREATE TABLE schema_migrations (
                version VARCHAR(255) PRIMARY KEY,
                applied_at TIMESTAMP DEFAULT NOW()
              );
              CREATE TABLE documents (
                id SERIAL PRIMARY KEY,
                title VARCHAR(255) NOT NULL,
                created_at TIMESTAMP DEFAULT NOW()
              );
              INSERT INTO schema_migrations (version)
                VALUES ('001_initial');"

What does this Job do?

The Job runs psql as the migration_user against the headless Service DNS name postgres. It creates a schema_migrations tracking table and one application table (documents), then records the version.

This proves that migration tooling can connect, authenticate with least-privilege credentials, and alter the schema on the pinned database version.

  • Add the migration Job to your kustomization base. Open manifests/base/postgres/kustomization.yaml and add - migration-job.yaml to the resources list.
  • Apply the updated overlay and wait for the Job to complete by running:
kustomize build manifests/overlays/dev | kubectl apply -f -
kubectl wait --for=condition=complete job/pg-migration -n dev --timeout=60s

You should see job.batch/pg-migration condition met.

Before you verify the migration result, what do you expect SELECT * FROM schema_migrations; to return?

  • Verify the migration by running a temporary psql pod as the application user:
kubectl run psql-check --rm -it --restart=Never -n dev \
  --image=postgres:18 \
  --env="PGPASSWORD=app-rehearsal-2026" \
  -- psql -h postgres -U app_user -d postgres \
  -c "SELECT * FROM schema_migrations;"

You should see one row: version 001_initial with an applied_at timestamp. That confirms the migration ran and the application user can read the result.

Query returns zero rows or connection refused?

  • If the Job completed but the table is empty, check the Job logs: kubectl logs job/pg-migration -n dev. Look for SQL syntax errors in the output.
  • If you see could not translate host name "postgres" to address, the headless Service is missing or its name does not match. Verify kubectl get svc -n dev shows a postgres entry.
  • help me debug my PostgreSQL migration Job in Kubernetes
  • Commit all postgres manifests, push the story-2 branch, open a PR, confirm the lint workflow passes, merge, and close the Linear ticket for story 2.
Deploy GitLab CE and prove the outbound network path

The runner must pull alpine:latest from the public registry to execute its job. That pull is the proof. If the network path is blocked, the pipeline fails immediately and you know before production does.

  • Switch to the story-3 branch by running:
git checkout -b story-3
  • Create the GitLab directory and the Compose file by running:
mkdir -p gitlab
  • Save the following as gitlab/docker-compose.yml:
services:
  gitlab:
    image: gitlab/gitlab-ce:latest
    hostname: gitlab.local
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://gitlab.local:8929'
        gitlab_rails['gitlab_shell_ssh_port'] = 2424
    ports:
      - "8929:8929"
      - "2424:22"
    shm_size: '256m'
    volumes:
      - gitlab-config:/etc/gitlab
      - gitlab-logs:/var/log/gitlab
      - gitlab-data:/var/opt/gitlab

  gitlab-runner:
    image: gitlab/gitlab-runner:alpine
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - runner-config:/etc/gitlab-runner
    depends_on:
      - gitlab

volumes:
  gitlab-config:
  gitlab-logs:
  gitlab-data:
  runner-config:

What does this Compose file do?

It brings up two containers. The gitlab service runs the full GitLab CE application on port 8929 with SSH on 2424. The GITLAB_OMNIBUS_CONFIG environment variable is evaluated before gitlab.rb on each start.

The gitlab-runner container mounts the Docker socket so it can spin up job containers on the host engine. shm_size: '256m' is required. Without it, GitLab's Prometheus metrics exhaust shared memory and the container crashes.

  • Bring up GitLab by running:
docker compose -f gitlab/docker-compose.yml up -d

This will take several minutes

GitLab CE performs a full omnibus reconfigure on first start. Expect 3 to 7 minutes before the web UI responds. This is the slowest operation in the entire rig. Do not restart the container. Watch progress with docker compose -f gitlab/docker-compose.yml logs -f gitlab and wait for the log line containing master process ready.

  • Poll until GitLab responds by running:
until curl -s http://localhost:8929/-/readiness | grep -q ok; do
  echo "Waiting for GitLab..."
  sleep 10
done
echo "GitLab is ready"

When you see GitLab is ready, the web UI is accepting connections.

Still waiting after 10 minutes?

  • Run docker compose -f gitlab/docker-compose.yml ps. If the gitlab container shows unhealthy or restarting, check available RAM. GitLab needs at least 4 GB free.
  • If the container exited, run docker compose -f gitlab/docker-compose.yml logs gitlab | tail -50 and look for memory or shm errors.
  • help me troubleshoot GitLab CE not becoming healthy in Docker Compose
  • Retrieve the initial root password by running:
docker compose -f gitlab/docker-compose.yml exec gitlab \
  cat /etc/gitlab/initial_root_password | grep Password:
  • Note the password. You will use it to log in and create a runner registration token.
  • Open http://localhost:8929 in your browser. Sign in as root with the password from the previous command.
  • Create a new project named test-pipeline.
  • Navigate to test-pipeline then Settings then CI/CD. Expand Runners and click New project runner.
  • Leave defaults and click Create runner. Copy the registration token displayed.
  • Register the runner against your GitLab instance by running (replace your-token-here with the token you copied):
docker compose -f gitlab/docker-compose.yml exec gitlab-runner \
  gitlab-runner register \
    --non-interactive \
    --url "http://gitlab:8929" \
    --token "your-token-here" \
    --executor "docker" \
    --docker-image alpine:latest \
    --docker-network-mode "gitlab_default"

Why use the internal hostname?

The runner container addresses the GitLab container by its Compose service name (gitlab) on port 8929. The --docker-network-mode flag ensures job containers also join the same Compose network, so they can clone from GitLab without host-level DNS tricks.

You should see Runner registered successfully.

  • Create the test pipeline file. Save this as gitlab/test-project/.gitlab-ci.yml:
# Proves the runner can pull from the public registry
stages:
  - test

pipeline-proof:
  stage: test
  image: alpine:latest
  script:
    - echo 'pipeline works'

What does this pipeline prove?

The job uses alpine:latest as its image. The runner must pull that image from the public Docker registry. If the outbound network path is blocked (a common first failure on hardened workstations), this job fails immediately with a pull error.

  • Push the .gitlab-ci.yml file into your GitLab test project. Use the GitLab web IDE or push via Git over SSH on port 2424.

Before you check the pipeline status, do you expect it to pass or fail? Think about what has to happen for the runner to pull alpine:latest.

  • Return to your test-pipeline project in the GitLab UI. Click Build then Pipelines.

You should see a pipeline with a green passed badge. Click into the job to see pipeline works in the output. The runner pulled from the public registry successfully.

Pipeline stuck in pending or failed?

  • If the pipeline stays pending, the runner is not registered or is offline. Run docker compose -f gitlab/docker-compose.yml exec gitlab-runner gitlab-runner list to verify.
  • If the job fails with an image pull error, the workstation cannot reach the public registry. Check DNS and firewall rules in WSL2.
  • help me debug my GitLab Runner not picking up jobs
Record observations and close stories

Every observation you record now becomes evidence in the readout. The time GitLab took to become healthy is the number the next person planning a bring-up will use.

  • Note two observations for later use in docs/readout.md: (1) the elapsed time from docker compose up to GitLab responding, and (2) whether runner registration succeeded on the first attempt.
  • Commit the gitlab/ directory and your observations note.
  • Push the story-3 branch, open a PR, confirm the lint workflow passes, merge, and close the Linear ticket for story 3.
  • Open the story-4 branch for the next step by running:
git checkout -b story-4

Confirm both Linear tickets (stories 2 and 3) now show as Done. Three of five stories are complete.

Lint workflow failing on your PR?

  • The workflow validates YAML files. Run yamllint gitlab/docker-compose.yml locally to catch syntax issues before pushing.
  • If the error is about the Compose file specifically, run docker compose -f gitlab/docker-compose.yml config to validate its structure.
  • help me fix YAML lint errors in my GitLab Compose file

Two independent proofs now stand. The database accepts migrations on the pinned version. The pipeline host pulls from the public registry. Next up, you will write the governance documents that prevent anyone from mistaking these results for production readiness.

Fidelity Map, Readout, and Documentation

The rig without its governance documents is a toy. A cluster that runs, a database that accepts migrations, a pipeline that passes green. None of it means anything to a delivery team unless someone has written down what the rig cannot prove.

This step produces the artifacts that justify the rig's cost. The fidelity map names six gaps with owners and review dates. The readout aims at two audiences who decide whether the team uses it. The runbook is tested by someone who did not build it, working from the document alone while the author says nothing.

In this step, get ready to:
  • Complete the fidelity map with six numbered gaps, each owned and dated.
  • Write the two-audience readout and three-failure runbook.
  • Produce the documentation source map and merge story 4.
Complete the fidelity map by hand

The fidelity map is the one artifact the AI does not draft. Its entire value comes from a person thinking about what the rig cannot do. You already recorded Gap 1 in Step 2. Now you add five more prescribed gaps plus confirm the discovered gap.

  • Open docs/fidelity-map.md in your editor on the story-4 branch.
  • Replace the existing content with the full six-gap fidelity map below:
# Fidelity Map — Rehearsal Rig v1.0

An entry leaves this map only with evidence and a date, never quietly.

| # | Gap | Owner | Review Date | Exit Condition |
|---|-----|-------|-------------|----------------|
| 1 | SeaweedFS filer notification is gRPC-only; S3 notification API absent (discovered during build) | Platform Engineer | [+90 days] | Production object store confirmed with S3 notification API support, or consumer adapted |
| 2 | Platform guardrails, proxies, and admission policy not present | Infrastructure Lead | [+90 days] | CSP-specific guardrails applied and tested in staging |
| 3 | Cross-account encryption-key semantics not tested | Security Engineer | [+90 days] | KMS cross-account replication validated without silent failure |
| 4 | Impact-level baselines (IL4/IL5) and control additions not addressed | Compliance Lead | [+90 days] | Controls mapped to target impact level with evidence |
| 5 | Real provisioning lead times not modeled | Delivery Manager | [+60 days] | Actual CSP provisioning SLA captured from program data |
| 6 | Authorization process (RMF A&A or successor phases) not represented | Security Engineer | [+90 days] | Authorization pathway documented with program office |

Why write this by hand?

An AI assistant can draft infrastructure manifests because correctness is verifiable: it either deploys or it does not. A fidelity map requires judgment about what the rig cannot prove. That judgment is the artifact's value.

Gap 3 is where cross-account encryption-key replication fails silently in production. Gap 5 is real provisioning lead time, which is the program's actual schedule risk. These are not things a tool discovers. They are things a person names because they have seen programs miss them.

  • Save docs/fidelity-map.md.
  • Confirm the file has six gaps by running:
grep -c "^|" docs/fidelity-map.md

You should see 8 (six gap rows plus the header row plus the separator row).

Seeing fewer than 8 lines?

Check that each gap is on its own table row. If you used a list format instead of a table, the grep pattern won't match. Either reformat as a table or adjust your check.

help me verify my fidelity map has six gaps with the correct fields

Write the two-audience readout and three-failure runbook

The readout targets two audiences who decide whether the team uses this rig. Engineering leadership needs the topology and its tradeoffs. The program and government stakeholder need one paragraph on what the team can do while blocked and one sentence on what the rig does not prove.

  • Create docs/readout.md with the following structure:
# Readout — Rehearsal Rig v1.0

Reviewed: YYYY-MM-DD

## For Engineering Leadership

Topology: kind cluster (rehearsal-rig) with three namespaces (dev, int, prod),
SeaweedFS with filer notification, PostgreSQL 18, GitLab CE with runner.

Three-namespace shortcut and its cost: no per-zone admission policy,
no cluster-scoped separation. This is stated, not hidden.

Two failure modes designed out:
1. A team sitting idle on provisioning it does not control.
2. A rehearsal result quoted upward as production readiness.

## For Program and Government Stakeholder

While blocked on production provisioning, the team can now validate
migration tooling, event-driven wiring, and pipeline mechanics against
the pinned versions and promotion shape.

The rig does not prove production readiness.

## Lessons Learned

- Slowest step: [name the step and elapsed time]
- Wrong assumption: [name what you expected vs what happened]
- Next iteration would change: [one concrete improvement]
- Clean-clone elapsed time: [to be filled in Step 5]

What makes this readout effective?

Each audience gets only what they need to decide. Engineering leadership sees the topology and its tradeoff. The government stakeholder sees a one-sentence boundary. The lessons learned section is filled with real observations, not aspirational notes.

  • Save docs/readout.md.
  • Fill in the Lessons Learned section with your actual observations from Steps 1-3 (which step ran longest, what surprised you).
  • Create the runbook directory and file by running:
mkdir -p docs/runbook
  • Create docs/runbook/runbook.md with the three-failure structure:
# Runbook — Rehearsal Rig v1.0

Reviewed: YYYY-MM-DD
Tested-by: [name/role of second reader, date tested]

## Failure 1: kind cluster won't start

**Symptom:** `kind create cluster` hangs or exits with error.
**Check:** Run `docker ps` to verify Docker Engine is running.
Run `kind get clusters` to check for stale cluster state.
**Fix:** If Docker is not running, start it with `sudo service docker start`.
If stale state exists, run `kind delete cluster --name rehearsal-rig`
then recreate.

## Failure 2: SeaweedFS events not firing

**Symptom:** `weed filer.meta.tail` shows no output after file upload.
**Check:** Verify the filer pod is running: `kubectl get pods -n dev`.
Verify port-forward is active on the filer gRPC port.
**Fix:** Restart the filer pod. Re-establish the port-forward.
Confirm upload targets the S3 gateway, not the filer directly.

## Failure 3: GitLab runner fails to register

**Symptom:** Runner container starts but jobs stay pending.
**Check:** Run `docker logs <runner-container>` for registration errors.
Verify GitLab is healthy: `curl http://gitlab.local:8929/-/health`.
**Fix:** Wait for GitLab reconfigure to complete (3-7 minutes on first start).
Re-register the runner with a fresh token from GitLab Admin > Runners.

Why symptom, check, fix?

Every runbook entry follows the same shape: what the person sees (symptom), what they verify to narrow the cause (check), and what they do to resolve it (fix). This structure means someone who did not build the rig can work through it without asking the author.

  • Save docs/runbook/runbook.md.
  • Verify both files exist by running:
ls docs/readout.md docs/runbook/runbook.md

Both files should appear in the output with no errors.

File not found?

Confirm you created the docs/runbook/ directory before writing the file. Run mkdir -p docs/runbook if it is missing.

help me find where my runbook file ended up

Write the source map, test the runbook, and merge

The source map closes the traceability loop. Every documentation page names the file that backs it. Tutorials and contributor files are deliberately cut.

  • Create docs/source-map.md with the following content:
# Documentation Source Map

Reviewed: YYYY-MM-DD

| Page | Source File | Reviewed |
|------|------------|----------|
| Architecture at a glance | docs/topology.md | YYYY-MM-DD |
| Quickstart | README.md | YYYY-MM-DD |
| Operations guide | docs/runbook/runbook.md | YYYY-MM-DD |
| Runbook | docs/runbook/runbook.md | YYYY-MM-DD |
| Decision index | docs/decisions/ | YYYY-MM-DD |
| Agent architecture | docs/topology.md | YYYY-MM-DD |
| Configuration reference | manifests/ | YYYY-MM-DD |
| Limits page (fidelity map) | docs/fidelity-map.md | YYYY-MM-DD |

Every claim traces to a file, a command, or a named source.
Tutorials and contributor files: deliberately cut.

Why no tutorials or contributor files?

Tutorials rot faster than reference docs because they encode a specific version's workflow. A contributor guide implies the repo is open for external contributions, which a rehearsal rig is not. Both are cut to keep the documentation set honest about what it maintains.

  • Save docs/source-map.md.
  • Add a reviewed-date field to every doc page that does not already have one (docs/topology.md, each ADR in docs/decisions/).
  • Verify the full documentation set by running:
ls docs/fidelity-map.md docs/readout.md docs/runbook/runbook.md docs/source-map.md docs/topology.md docs/decisions/

You should see all five doc files listed plus the four ADR files in docs/decisions/.

Now test the runbook. Hand it to a second person (a colleague, a friend, anyone who did not build this). They follow the document alone while you say nothing. Every question they ask is a defect in the document, not in them. Fix every defect before merging.

No second reader available right now?

Simulate the protocol: close the runbook, wait at least an hour (ideally 24 hours), then follow it cold on a fresh terminal without looking at anything else. Every moment you reach for memory instead of the document is a gap to fix.

  • After testing, fill in the tested-by field in docs/runbook/runbook.md with the name/role of the tester and the date.
  • Stage and commit all documentation files by running:
git add docs/
git commit -m "feat: complete fidelity map, readout, runbook, and source map"
  • Push the story-4 branch by running:
git push origin story-4
  • Open a pull request on GitHub from story-4 into main.
  • Confirm the lint workflow passes.
  • Merge the PR.
  • In Linear, close the ticket for story 4 (Fidelity map and documentation).

Before you check the final state, what do you expect to see when you list the docs directory now? How many files total?

  • Verify the merged state on main by running:
git checkout main && git pull
find docs/ -name '*.md' | wc -l

You should see at least 9 markdown files in docs/ (fidelity map, readout, runbook, source map, topology, and four ADRs).

  • Open the story-5 branch for the final step by running:
git checkout -b story-5

Lint workflow failing on the PR?

The lint workflow validates YAML files in manifests/ and the Docker Compose file. If it fails, check whether a markdown file accidentally landed in manifests/, or whether the Compose file has a syntax issue from a prior step.

help me debug my failing GitHub Actions lint workflow

Four stories done, one to go. Your rig has infrastructure, a pipeline, and now the governance documents that make it a team asset. Next up: the tagged release and the timed clean-clone proof that closes the loop.

Tagged Release and Completion Gate

Your documentation set is complete. The fidelity map, runbook, readout, and source map are committed with reviewed dates, and story 4 is closed.

A running rig is not a finished deliverable. Completion means every PR merged, every ticket closed, the release tagged, the clean-clone time recorded, and the fidelity map published before anyone else sees it.

In this step, get ready to:
  • Run a timed clean-clone proof that records the actual bring-up duration.
  • Tag and release v1.0.0 on GitHub.
  • Close the Linear project and confirm the completion definition.
Perform the timed clean-clone proof

The readout claims a clean-clone time. That number must come from measurement, not estimation. You will destroy the rig, clone fresh, and bring it back up from scratch.

  • First, write the bring-up script that automates the full rig creation. Create bring-up.sh in the repository root with the following content:
#!/bin/bash
# Single bring-up script for the rehearsal rig
START_TIME=$(date +%s)

# Create the kind cluster from config
kind create cluster --config kind-config.yaml

# Apply Kustomize overlays for all three zones
kustomize build manifests/overlays/dev | kubectl apply -f -
kustomize build manifests/overlays/int | kubectl apply -f -
kustomize build manifests/overlays/prod | kubectl apply -f -

# Start GitLab CE and runner
docker compose -f gitlab/docker-compose.yml up -d

# Record elapsed time
END_TIME=$(date +%s)
ELAPSED=$((END_TIME - START_TIME))
echo "Bring-up completed in ${ELAPSED} seconds."

What does this script do?

The script captures the wall-clock start time, then runs every infrastructure command in sequence: cluster creation, overlay application for all three namespaces, and the GitLab Compose stack.

At the end it prints the total elapsed seconds. This is the number that goes into the readout, proving feasibility with a measurement rather than a guess.

  • Make the script executable by running:
  • Commit the bring-up script to the story-5 branch and push by running:
chmod +x bring-up.sh
git add bring-up.sh
git commit -m "Add bring-up script for clean-clone proof"
git push origin story-5

The script is now in the remote repository, ready for the fresh clone.

  • Tear down the running rig by running these commands:
kind delete cluster --name rehearsal-rig
docker compose -f gitlab/docker-compose.yml down -v

Both the cluster and GitLab containers are now gone. Your machine is back to a bare state.

  • Clone the repository fresh into a temporary directory by running:
cd ~/Desktop
git clone [[YOUR_REPO_URL="https://github.com/your-username/rehearsal-rig.git"]] rehearsal-rig-clean
cd rehearsal-rig-clean
git checkout story-5

Before you run the bring-up script, predict: how long do you think a full bring-up will take? Which step will be the slowest?

  • Run the bring-up script and record the elapsed time:
./bring-up.sh

Watch the output. The script prints the total elapsed seconds when it finishes. The slowest step is typically GitLab reconfigure (3-7 minutes).

  • Verify the rig came up by running:
kubectl get ns

You should see dev, int, and prod namespaces listed alongside the system namespaces.

  • Update docs/readout.md with the timed results. Replace the [to be filled in Step 5] placeholder with: total elapsed time in seconds, the slowest step named (GitLab reconfigure), one assumption that was wrong, and what the next iteration would change.
  • Commit and push the updated readout:
git add docs/readout.md
git commit -m "Record clean-clone elapsed time in readout"
git push origin story-5

Script fails or hangs?

If bring-up.sh fails with a permission error, confirm you ran chmod +x bring-up.sh before committing. If the script hangs on GitLab, check that Docker has enough memory allocated (at least 4 GB for WSL2).

If kubectl get ns fails after bring-up, the kind cluster may not have finished starting. Wait 30 seconds and retry.

help me debug my bring-up script

Tag the release and create the GitHub Release

The clean-clone proof passed. The rig works from a fresh clone. Now tag this commit and publish the release so the repository has a clear v1.0.0 milestone.

  • Update the repository README.md with links to the key documents. Create or overwrite README.md in the repository root with the following content:
# Rehearsal Rig

Zero-cost rehearsal rig mirroring three promotion zones (dev, int, prod).

## Quickstart

Run `./bring-up.sh` from the repository root.

## Key Documents

- [Fidelity Map](docs/fidelity-map.md)
- [Readout](docs/readout.md)

## Notes

Branch protection and code scanning are stretch goals
requiring a paid GitHub plan.

Why link the fidelity map from the README?

The README is the first thing anyone sees when they open the repository. Linking the fidelity map here ensures no one uses the rig without knowing what it cannot prove.

  • Commit the README, push, open the PR for story 5, and merge it:
git add README.md
git commit -m "Add README with links to fidelity map and readout"
git push origin story-5
  • Open a pull request for the story-5 branch on GitHub.
  • Confirm the lint workflow passes, then merge the PR.
  • Close the Linear ticket for story 5.
  • Switch to main and pull the merged code, then tag the release by running:
git checkout main
git pull origin main
git tag v1.0.0
git push origin v1.0.0
  • Create the GitHub Release in the web UI:
  • Open your repository on GitHub and click Releases (to the right of the file list).
  • Click Draft a new release.
  • Select v1.0.0 from the Choose a tag dropdown.
  • Enter v1.0.0 as the release title.
  • In the description, write one paragraph summarizing what the rig contains and one sentence drawn from the fidelity map stating what it does not prove.
  • Click Publish release.

The GitHub releases page now shows v1.0.0 as the latest release.

Tag not appearing on GitHub?

If the tag does not show up in the Choose a tag dropdown, confirm you pushed it with git push origin v1.0.0. A common mistake is pushing the branch without the tag.

help me push my git tag to GitHub

Mark the Linear project Completed

Every story is closed. The epic is done. The final act is marking the Linear project itself as Completed.

  • Open your Linear project ("Rehearsal Rig v1.0").
  • Confirm all five stories show as closed.
  • Set the project status to Completed.

Walk through the completion definition one final time:

  • All five stories closed, epic done.
  • Release tagged v1.0.0 and published.
  • Documentation present with reviewed dates.
  • Clean-clone time recorded in the readout.
  • Fidelity map published and linked from the README.

The rig is now a team asset, not a personal experiment. A principal engineer can hand this repository to a delivery team today. One command brings up the rig, the fidelity map tells them what it cannot prove, and the decision records tell them why this shape rather than another.

Why enforce this definition?

Without a strict completion gate, "done" drifts to mean "it runs on my machine." A tagged release with a recorded clean-clone time and a published fidelity map means anyone can reproduce the result and anyone can see its boundaries. That is the difference between a personal experiment and a deliverable.

Secret mission

Add a Promotion Pipeline

The Kustomize overlays exist for all three zones, but nothing has proven the promotion shape works end to end through the CI/CD system. In this extension, you wire the GitLab Runner to your kind cluster and write a multi-stage pipeline that applies dev, then int, then prod in sequence. The same overlays the team will use in production are now driven by the same pipeline system.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

This project runs entirely locally or on free tiers. No metered services are involved. No data leaves your workstation except to GitHub and Linear.

Resources you used:

  • kind cluster (rehearsal-rig): runs inside Docker, zero cost, local only.
  • GitLab CE and Runner containers: run on host Docker Engine via gitlab/docker-compose.yml, zero cost, local only.
  • SeaweedFS and PostgreSQL 18: run as pods inside the kind cluster, zero cost.
  • GitHub repository: private, free tier (2,000 Actions minutes/month included).
  • Linear project: free tier.

Keep everything running

No action needed. Choose this if you are still actively building on the rig or plan to extend it with additional overlays, services, or pipeline stages.

  • The kind cluster and GitLab containers continue running until you stop Docker or restart your machine.
  • Your GitHub repository and Linear project remain accessible on their free tiers.
  • Docker Engine, kind, kubectl, and kustomize remain installed in your WSL2 environment.

Pause - I'll come back to this later

Shut down running processes to free up memory and CPU. Your files and configuration remain on disk for when you return.

  • Stop the GitLab CE and Runner containers by running:
docker compose -f gitlab/docker-compose.yml stop
  • Stop the kind cluster container by running:
docker stop rehearsal-rig-control-plane
  • To resume later, start the containers again:
docker start rehearsal-rig-control-plane
docker compose -f gitlab/docker-compose.yml start

Delete - I don't want to use this again

Remove all project resources from your machine and start fresh.

  • Delete the kind cluster and remove GitLab containers with their volumes by running:
kind delete cluster --name rehearsal-rig
docker compose -f gitlab/docker-compose.yml down -v
  • Confirm the cluster is gone by running:
kind get clusters
  • You should see no output, confirming the cluster is deleted.
  • Remove the local project folder (run this from the parent directory):
rm -rf rehearsal-rig
  • Delete the GitHub repository: go to your repository page, click Settings, scroll down to Danger Zone, and click Delete this repository.
  • Delete the Linear project: open your project, click the project name to open settings, and select Delete project.

Well Done!

Well Done!

Nice work! You just built a zero-cost rehearsal rig that mirrors a federal data platform's three promotion zones, complete with governance artifacts that bound every result it produces.

You've learned how to:

  • Design before building by writing four decision records and a topology diagram, then delivering through a private GitHub repo with branch-per-story PRs, a lint workflow, and a tagged v1.0.0 release tracked in Linear.
  • Stand up a full platform rehearsal with a three-namespace Kubernetes cluster, SeaweedFS filer notification, PostgreSQL 18 migrations, and a GitLab CE pipeline proving the outbound network path.
  • Govern a rehearsal rig honestly by publishing a fidelity map with six standing gaps, each owned and dated, so no result gets cited as production readiness.
  • Secret Mission: Built a promotion pipeline driving Kustomize overlays through three GitLab stages in sequence, proving the promotion shape works end to end through the CI/CD system.

Ready to quiz yourself?