Build an Agent Substrate AI Analyst

Build a stateful AI analyst with gateway-injected OpenRouter credentials.

Introduction

30 Second Summary

An analyst becomes more useful as it remembers the notes behind each conclusion. A restart can erase that context, while a saved access key can travel with data it was never meant to join.

In this project, you will run a stateful AI analyst with Agent Substrate on a local Kubernetes cluster. You will prove that its memory survives a full suspend and resume cycle while gateway-side credential injection keeps the raw OpenRouter key outside the analyst.

What You'll Build

You'll watch a gVisor-isolated actor return from suspension with its notes and conversation intact while its status confirms the raw API key never entered its environment.

By the end of this project, you'll have:

  • A default-deny gateway test where the first routed question returns HTTP 403. The same question succeeds after explicit authorization.
  • A markdown analyst brief generated from synthetic corporate notes. A saved evidence pack captures the supporting lifecycle and policy output.
  • A suspend-and-resume proof showing that notes and prior turns survive a full-memory snapshot. Actor status and logs confirm that OPENROUTER_API_KEY remains absent.
  • Secret Mission: Tag the suspended snapshot as analyst-seed. Clone analyst-twin from that saved state. Give each analyst a different follow-up question to make their histories diverge.

Are there any prerequisites?

You'll need an OpenRouter account plus administrator access to configure your container runtime.

Your machine must offer at least 8 GB of RAM, 4 CPUs, and about 20 GB of free disk space.

Before We Start

This first checkpoint commits you to a disposable local lab that uses only synthetic corporate content. Your stateful AI Analyst will prove a secure lifecycle: memory survives lifecycle changes while the raw OpenRouter credential remains at the gateway.

Set Up the Cross-Platform Lab

A reliable Agent Substrate lifecycle depends on enough local capacity for its control plane plus two isolated workers. Verifying the host first prevents the longer bootstrap from failing halfway through.

You will prepare a supported container runtime. You will also stage a temporary credential before creating the local Kubernetes lab.

In this step, get ready to:
  • Prepare a supported host with the required tools plus runtime capacity.
  • Store a temporary OpenRouter key with the required free-model configuration.
  • Pass preflight before bootstrapping the local Agent Substrate cluster.
Prepare the supported host

The local cluster gets its CPU plus memory from Docker Desktop, Colima, or Docker Engine. The runtime needs at least four CPUs plus 8 GB of memory.

Choose the tab for your operating system. Windows setup starts in Windows before all project work moves into Ubuntu on WSL2.

macOS

Intel Macs plus Apple Silicon Macs use the same project scripts. Match each runtime download to your Mac's architecture.

Docker Desktop

  • Install Docker Desktop for your Mac's architecture.
  • Press Cmd+Space to open macOS search.
  • Type Docker Desktop into the search bar.
  • Press Enter to open Docker Desktop.
  • Click Settings in Docker Desktop.
  • Select Resources.
  • Select Advanced.
  • Set the CPU limit to at least 4.
  • Set the memory limit to at least 8 GB.
Why Allocate This Capacity?

The runtime hosts the cluster control plane plus two gVisor workers. These limits keep worker startup from competing with the cluster services.

  • Save the Docker Desktop resource changes.

Docker Desktop applies the new limits to its virtual machine. The runtime is ready for the later preflight check.

Are the Resource Controls Missing?

Confirm that Docker Desktop is using its supported virtualization mode. Reopen the resource settings after Docker Desktop finishes starting.

Ask for help with the settings that appear on your Mac: Help me find the Docker Desktop CPU and memory controls on macOS.

Colima

  • Install Colima v0.10.3 for your Mac's architecture.
  • Press Cmd+Space to open macOS search.
  • Type Terminal into the search bar.
  • Press Enter to open Terminal.
  • Resize the Colima virtual machine by running these commands:
colima stop
colima start --cpu 4 --memory 8
What Do These Commands Do?

The first command stops the current Colima virtual machine. The second command starts it with four CPUs plus 8 GiB of memory.

The terminal reports that Colima has started. Its virtual machine now has the capacity required by the lab.

Does Colima Fail to Start?

Check that no other container runtime is occupying the resources Colima needs. Confirm that the Colima download matches your Mac's architecture.

Ask for help using the non-secret terminal output: Help me diagnose why Colima did not start with four CPUs and 8 GiB of memory.

  • Install Git with your preferred macOS package manager.
  • Install Bash with your preferred macOS package manager.
  • Confirm that curl is available for the preflight check.

Your Mac now has a container runtime with the required CPU plus memory limits. The version checks below complete the host setup.

Windows

Docker Desktop runs on Windows. The starter repository plus every project command stay inside the Ubuntu filesystem.

  • Install Docker Desktop on Windows.
  • Press the Windows key to open search.
  • Type Docker Desktop into the search bar.
  • Press Enter to open Docker Desktop.
  • Click Settings in Docker Desktop.
  • Select Resources.
  • Enable WSL 2 integration for Ubuntu.

WSL2 reads its global CPU plus memory limits from %UserProfile%\.wslconfig. The limits apply after WSL shuts down.

  • Press the Windows key to open search.
  • Type Notepad into the search bar.
  • Press Enter to open Notepad.
  • Create %UserProfile%\.wslconfig with this configuration:
[wsl2]
memory=8GB
processors=4

What Does This Configuration Do?

The [wsl2] section controls the WSL2 virtual machine. The settings expose 8 GB of memory plus four processors to Ubuntu.

  • Save %UserProfile%\.wslconfig in your Windows user folder.
  • Press the Windows key to open search.
  • Type PowerShell into the search bar.
  • Press Enter to open PowerShell.
  • Restart WSL by running this command:
wsl --shutdown

What Does This Command Do?

The command stops the WSL virtual machine. Ubuntu receives the new CPU plus memory limits when it starts again.

  • Press the Windows key to open search.
  • Type Ubuntu into the search bar.
  • Press Enter to open Ubuntu.
  • Install Git inside Ubuntu with its package manager.
  • Install Bash inside Ubuntu with its package manager.
  • Install curl inside Ubuntu with its package manager.
  • Keep the supplied starter repository under $HOME in Ubuntu.

Why Use the Ubuntu Filesystem?

Docker recommends keeping bind-mounted project source in the Linux filesystem. A path under /mnt/c can introduce slower file access plus permission differences.

Does Ubuntu Still Have the Old Limits?

Confirm that %UserProfile%\.wslconfig was saved in your Windows user folder. Run the shutdown command again after closing every Ubuntu window.

Ask for help with the non-secret configuration: Help me check why Ubuntu on WSL2 does not see four processors and 8 GB of memory.

Linux

Docker Engine uses the Linux host's resources directly. The host must expose at least four CPUs plus 8 GB of memory.

  • Install Docker Engine by following the instructions for your Linux distribution.
  • Install Git with your distribution's package manager.
  • Install Bash with your distribution's package manager.
  • Install curl with your distribution's package manager.

Docker group membership grants root-level privileges. Review that access before allowing your current user to run Docker without elevated commands.

Can Docker Not Start Containers?

Confirm that the Docker service is running. Recheck the documented non-root setup if Docker only works with elevated privileges.

Ask for help using the non-secret command output: Help me diagnose why my Linux user cannot use Docker for this local lab.

Why Use kind for This Lab?

The pinned repository's kind workflow creates the local registry. It also enables the feature gates required by Agent Substrate.

The workflow wires in gVisor plus parallel image pulls. Docker Desktop's built-in Kubernetes follows a different cluster path.

The repository requires Go 1.27.2. Check the toolchain selected by your Bash environment before continuing.

  • Check the installed Go version by running this command:
go version

What Does This Command Check?

The command prints the Go toolchain selected by your shell. The result must include go1.27.2 for this lab.

✔️ I see the required Go version

Your Go toolchain is ready for the pinned Agent Substrate build.

ⓧ I see an older Go version

Replace the older toolchain with Go 1.27.2 before continuing.

macOS

Windows

Linux

ⓧ Go is not found

Install Go 1.27.2 before continuing.

macOS

Windows

Linux

The kind cluster uses Kubernetes 1.37.0. A kubectl 1.37.1 client stays within the supported minor-version range.

  • Check the installed kubectl client by running this command:
kubectl version --client

What Does This Command Check?

The command prints the local kubectl client version without requiring a cluster connection. The result must include v1.37.1.

✔️ I see the required kubectl version

Your kubectl client is compatible with the local cluster created by this project.

ⓧ I see an older kubectl version

Upgrade the client to kubectl 1.37.1 before continuing.

macOS

Windows

Linux

ⓧ kubectl is not found

Install kubectl 1.37.1 before continuing.

macOS

Windows

Linux

Stage the temporary OpenRouter credential

The OpenRouter key eventually becomes a Kubernetes Secret used by the egress gateway. The actor receives a placeholder authorization header without receiving the raw key.

Credentials deserve a careful pause. Keep this temporary key only in the local .env file.

  • Visit the official OpenRouter Keys page.
  • Create a temporary OpenRouter key for this disposable lab.
  • Copy the temporary key to your clipboard.
  • Return to a Bash terminal in the supplied starter repository.
  • Create .env from .env.example by running this command:
cp .env.example .env

What Does This Command Do?

The command copies the supplied configuration template to .env. The new file keeps the credential local to this lab.

  • Open .env in your editor.
  • Replace its contents with this configuration:
OPENROUTER_API_KEY=your-api-key-here
OPENROUTER_MODEL=openrouter/free
OPENROUTER_FALLBACK_MODEL=poolside/laguna-s-2.1:free

What Does This Configuration Do?

  • The OPENROUTER_API_KEY line supplies the temporary credential to the bootstrap process.
  • The OPENROUTER_MODEL line selects the openrouter/free router.
  • The OPENROUTER_FALLBACK_MODEL line keeps poolside/laguna-s-2.1:free as the fallback model.
  • Replace your-api-key-here with the temporary key from your clipboard.
  • Save .env.
  • Close .env before taking any screenshots.

Will Preflight Reject the Credential File?

Confirm that .env sits beside .env.example. Check that the temporary key replaced the placeholder without extra spaces.

Keep both model values exactly as shown. Ask for help without sharing the key: Help me check my local credential configuration safely.

✔️ Awesome, I've got everything!

Your local .env now contains the temporary credential plus the two required free-model values.

ⓧ I'd like to double check the full code

Compare your file with this safe structure. Keep your real temporary key in place without displaying it.

OPENROUTER_API_KEY=your-api-key-here
OPENROUTER_MODEL=openrouter/free
OPENROUTER_FALLBACK_MODEL=poolside/laguna-s-2.1:free
Run preflight and bootstrap the cluster

Preflight checks the supported host before the longer build starts. It covers dependencies, runtime capacity, required ports, disk space, WSL2 placement, and the local credential placeholder.

  • Verify the local environment from the supplied starter repository by running this command:
./scripts/preflight.sh

What Does Preflight Check?

The script detects macOS, WSL2, or Linux. It verifies Bash, Git, Docker, Go, kubectl, curl, ports 5001 plus 8000, approximately 20 GB of free disk space, and the required runtime capacity.

On WSL2, it checks that the project uses the Linux filesystem. It also confirms that .env no longer contains the template placeholder.

Preflight completes successfully. Your host has passed every check required for bootstrap.

Does Preflight Stop?

Use the failed check to identify the missing dependency or capacity limit. Confirm that your container runtime is running before retrying.

On WSL2, move the supplied starter repository under $HOME if the reported path begins with /mnt/c.

Ask for help using only the non-secret failure output: Help me resolve my preflight failure.

Bootstrap is the slowest part of this setup because it builds the Analyst image plus the local cluster components. Continuous terminal output means the process is still progressing.

  • Build the complete local lab by running this command:
./scripts/bootstrap.sh

What Does Bootstrap Create?

  • The script clones Agent Substrate at commit f03327bd4aa88e3aea93f26bca0752990e1ee165.
  • The script runs hack/create-kind-cluster.sh to create the kind cluster plus local registry.
  • The script runs hack/install-ate-kind.sh --deploy-ate-system --credential-provider='{"name":"k8s.io"}' to install Agent Substrate with the bundled credential provider.
  • The script builds the supplied Analyst source into a digest-pinned scratch image.
  • The script creates the Analyst WorkerPool plus ActorTemplate.
  • The script creates the suspended analyst actor in atespace team-a.
  • The script stores the temporary key as token in Secret ns1/example-api without displaying its value.
  • The script creates the deny-by-default egress policy through ate-secret://k8s.io/default/ns1/example-api/token.

The bootstrap returns control to your terminal with the local registry plus Agent Substrate resources running. The analyst remains suspended until its first request.

Does Bootstrap Fail?

Confirm that preflight still passes. Check that no other process claimed ports 5001 or 8000 after preflight completed.

Keep Docker Desktop, Colima, or Docker Engine running throughout the build. A stopped runtime interrupts the cluster bootstrap.

Ask for help using only the non-secret terminal output: Help me diagnose my bootstrap failure.

Before you run the final checks, predict whether both workers are free and whether the suspended analyst has a worker assignment.

  • Check the worker states plus the suspended actor by running these commands:
kubectl ate get workers
kubectl ate get actor analyst -a team-a

What Do These Checks Prove?

The first command lists the workers managed by Agent Substrate. The second command inspects the analyst actor inside team-a.

A suspended actor has no assigned worker pod or worker IP. This creates a clean lifecycle starting point for the first request.

You see two workers in FREE state. You also see the suspended analyst with an empty worker assignment.

Are the Expected Workers Missing?

Review the end of the bootstrap output for the first resource that failed to become ready. Confirm that your kubectl context still points to the project-created kind cluster.

Ask for help with the non-secret worker plus actor output: Help me diagnose why I cannot see two free workers and a suspended analyst actor.

Your Lab Checkpoint

  • Preflight has verified Bash, Git, the Docker runtime, Go 1.27.2, kubectl 1.37.1, curl, the required ports, runtime capacity, disk space, and applicable WSL2 placement.
  • The kind cluster uses Agent Substrate commit f03327bd4aa88e3aea93f26bca0752990e1ee165 plus the project-created local registry.
  • The Analyst WorkerPool has two gVisor-isolated workers in FREE state.
  • The Analyst ActorTemplate uses the locally built digest-pinned scratch image.
  • The team-a/analyst actor is suspended without a worker assignment.
  • The ns1/example-api Secret stores the temporary key as token without exposing its value.
  • The analyst policy replaces its placeholder Authorization header through ate-secret://k8s.io/default/ns1/example-api/token.
  • Atespace team-a still lacks access to namespace ns1.

Your local lab now has a suspended actor plus an intentionally unauthorized secret path. Next, you will wake the analyst and watch the gateway enforce that default-deny boundary.

Hit the Default-Deny Wall

The local Agent Substrate cluster now holds a suspended analyst plus two free gVisor workers. The next test follows an OpenRouter-backed request from actor wake-up to the egress gateway.

Credential storage is separate from actor authorization. This test shows whether the credential provider alone grants actor access before any namespace grant exists.

In this step, get ready to:
  • Send the suspended analyst its first OpenRouter-backed question.
  • Capture the gateway's authorization result.
  • Inspect the configured credential-replacement rule.
Test the current authorization boundary

Request-triggered resume lets traffic wake a suspended actor. The first model call also tests whether the gateway can resolve the referenced credential for team-a.

Before you run this, do you expect a stored secret to be enough for the request to pass the gateway?

  • Return to the Bash terminal connected to the local cluster.
  • Send the exact operating-assumptions question by running this command:
./scripts/ask.sh analyst "Summarize the current operating assumptions"

What Does This Command Test?

  • The ./scripts/ask.sh helper sends the exact question to the analyst actor.
  • The request triggers the suspended actor to resume on an available worker.
  • The helper reports the gateway's HTTP status after the actor attempts its OpenRouter call.

You've reached the default-deny wall. The request woke the actor.

The gateway returned HTTP 403 because team-a cannot resolve the secret in ns1.

Seeing a Different Result?

  • Check that the current folder contains scripts/ask.sh.
  • Confirm that the terminal is still connected to the local cluster from the previous step.
  • Stop if the helper reports a successful response.

A successful response means the cluster no longer matches the default-deny starting state.

Help me diagnose the unexpected analyst response.

Inspect the credential-replacement rule

An egress policy describes how the gateway handles outbound traffic. Its replaceHeaders rule substitutes a secret-backed value outside the actor.

This check reads the policy structure. It does not display the value stored in the Kubernetes Secret.

Before you inspect the YAML, which three pieces should still connect the actor's placeholder header to the secret reference?

  • Inspect the analyst's current egress policy by running this command:
kubectl ate get egress-policy analyst -a team-a -o yaml

How Does the Policy Explain the Denial?

  • The replaceHeaders rule tells the gateway to substitute a configured header after credential resolution.
  • The Authorization field names the outbound header that carries the actor's placeholder before replacement.
  • The ate-secret://k8s.io/default/ns1/example-api/token reference points the provider at the required secret value.
  • Authorization for team-a does not include ns1. Credential resolution therefore stops at the gateway.

You'll see replaceHeaders, Authorization, plus ate-secret://k8s.io/default/ns1/example-api/token in the YAML. The configured replacement path remains intact.

The HTTP 403 comes from namespace authorization. Gateway-side replacement keeps the raw OpenRouter key outside the actor environment.

  • Leave the namespace grant unapplied.

Missing the Replacement Rule?

  • Confirm that the command targets actor analyst.
  • Confirm that the command targets atespace team-a.
  • Compare the credentialUri value with ate-secret://k8s.io/default/ns1/example-api/token.

Help me inspect the analyst egress policy.

The default-deny boundary is now proven. Next, you'll authorize team-a for ns1 before repeating the exact question.

Grant Gateway Secret Access

Your Agent Substrate analyst is awake after the first request. The earlier HTTP 403 proved that the credential provider enforces explicit namespace authorization.

The OpenRouter key already lives in a Kubernetes Secret. This step grants team-a access to ns1 without changing the actor configuration.

In this step, get ready to:
  • Apply the explicit atespace-to-namespace grant.
  • Restart the credential provider to load the new policy.
  • Confirm the repeated analyst request succeeds through gateway-side injection.
Authorize team-a for ns1

A credential grant maps an atespace to an allowed secret namespace. The provider can resolve a secret path only when the policy covers that namespace.

  • Return to the Bash terminal from earlier.
  • Apply the prepared grant from manifests/credential-grant.yaml by running this command:
kubectl apply -f manifests/credential-grant.yaml

What Does This Command Do?

  • The kubectl apply command updates the cluster from the prepared manifest.
  • The grant authorizes atespace team-a to resolve credential-provider secrets in namespace ns1.
  • The command does not display the value stored in ns1/example-api.

The terminal confirms that the resource from manifests/credential-grant.yaml was applied. The cluster now contains the explicit namespace grant.

Does the Grant Fail to Apply?

Confirm that your terminal is still inside the starter repository. The relative path must resolve to manifests/credential-grant.yaml.

Check that kubectl remains connected to the local Agent Substrate cluster.

Help me diagnose the credential grant.

Load the new namespace policy

The bundled provider loads its namespace policy at startup. Restarting its deployment makes the new team-a grant active.

  • Reload the provider by running these commands:
kubectl -n ate-system rollout restart deployment/k8s-credential-provider
kubectl rollout status deployment/k8s-credential-provider -n ate-system

What Do These Commands Do?

  • The first command restarts k8s-credential-provider inside ate-system.
  • The second command watches the replacement deployment until its rollout completes.
  • The gateway connection can briefly return 503 while the provider reconnects.

The status command confirms that the provider deployment completed its rollout. The restarted provider now uses the namespace grant.

Is the Provider Rollout Stuck?

Confirm that the Docker runtime from earlier is still running. The local cluster cannot replace the provider pod without it.

Check that the namespace is ate-system and the deployment is k8s-credential-provider.

Help me diagnose the provider rollout.

Prove gateway-side injection

The actor still sends its placeholder Authorization header. The gateway can now resolve ate-secret://k8s.io/default/ns1/example-api/token before forwarding the request.

Before you repeat the request, predict whether the gateway can resolve the credential now.

  • Send the exact operating-assumptions question again by running this command:
./scripts/ask.sh analyst "Summarize the current operating assumptions"

What Happens During This Request?

  • The helper sends Summarize the current operating assumptions to the running analyst.
  • The gateway fetches the credential referenced by the egress policy.
  • The gateway replaces the placeholder Authorization header before the request reaches OpenRouter.
  • The helper retries only the documented transient 503 reconnect response.

You should see HTTP 200 followed by the analyst answer. The same question that reached the deny wall now succeeds.

What Does This Prove?

The gateway fetched the secret only after the explicit namespace grant became active. The egress policy supplied the credential reference used for replacement.

The grant did not add OPENROUTER_API_KEY to the actor environment. Only the non-secret model configuration remains available to the actor.

You have closed the authorization gap. Your analyst can now call OpenRouter while the raw key stays outside its process memory.

Gateway-side credential injection is working. Next, you will give the analyst synthetic notes so its in-memory state becomes worth preserving.

Build Memory and Save Evidence

Your authorized Agent Substrate actor now reaches OpenRouter through gateway egress. The API key still stays outside its process.

A full-memory snapshot is persuasive when the actor carries useful volatile state. Synthetic notes plus derived findings create a baseline for the resume test.

In this step, get ready to:
  • Store three synthetic corporate notes in the analyst's working memory.
  • Generate two findings that combine those notes.
  • Create the two baseline evidence files.
Create a useful memory baseline

Free-model provider handling can vary. Synthetic notes keep confidential company information out of every request.

  • Add the product goal, operational risk, and customer constraint by running these commands:
./scripts/note.sh analyst "Product goal: Launch a self-service analytics pilot for 20 design partners by the end of the quarter."
./scripts/note.sh analyst "Operational risk: Supplier data can arrive up to 24 hours late during regional outages."
./scripts/note.sh analyst "Customer constraint: Weekly pilot summaries must exclude personal data and fit a 10-minute review."

What Do These Commands Store?

  • Each invocation writes one note into the running analyst's in-process memory.
  • The product note defines the outcome that the analyst must protect.
  • The risk note introduces a condition that can weaken the plan.
  • The customer note sets privacy plus review-time boundaries.
  • Read each helper response to confirm that the note was accepted.

Did a Note Fail to Save?

  • Return to the same project checkout used for bootstrap.
  • Confirm that every command uses analyst as the actor name.
  • Check that each synthetic note stays inside one pair of double quotes.

Help me fix the note helper.

Before you check, what note count do you expect the actor to report?

  • Query the actor's /status route by running this command:
./scripts/status.sh analyst

What Does the Status Prove?

The status helper reads the working memory held by the actor. A note count of 3 proves that all three synthetic facts are available for analysis.

You'll see the note count reported as 3. The actor is now carrying useful volatile state.

Is the Note Count Below Three?

  • Review the three note helper responses for the command that failed.
  • Run only the missing note command from above again.
  • Repeat the status check after the helper accepts the note.

Help me reconcile the note count.

Turn the notes into findings

A useful finding connects multiple constraints. Each successful helper call stores the prompt plus the response plus the selected model in process memory.

Before you ask, how do you expect the operational risk to change the product plan?

  • Send both analytical questions to the actor by running these commands:
./scripts/ask.sh analyst "How should the pilot plan balance the end-of-quarter goal, data-refresh risk, and weekly privacy constraint?"
./scripts/ask.sh analyst "What weekly decision process protects the launch goal when supplier data is delayed and summaries must exclude personal data?"

What Enters Conversation Memory?

  • The first question asks the analyst to balance all three notes in one plan.
  • The second question asks for a repeatable decision process under the same constraints.
  • Each completed turn retains its prompt plus response plus selected model information.

Each helper reports HTTP 200. Each answer should connect details from more than one synthetic note.

Did an Analytical Request Fail?

  • Confirm that both commands use the running analyst actor.
  • Run the same question again if the helper reports the documented transient 503 provider reconnect status.
  • Check that each question stays inside one pair of double quotes.

Help me diagnose the analytical request.

Render the brief and evidence pack

A Markdown brief captures the analyst's current knowledge plus findings. The evidence pack records the surrounding infrastructure state without saving secret values.

  • Render the analyst brief into evidence/analyst-brief.md by running this command:
./scripts/brief.sh analyst > evidence/analyst-brief.md

What Does the Brief Capture?

The redirection writes the rendered brief to evidence/analyst-brief.md. The file contains the three synthetic notes plus the conversation-derived findings.

  • Return to the text editor you used for .env.
  • Select evidence/analyst-brief.md in the project file tree.

You'll see the product goal plus the operational risk plus the customer constraint. You'll also see findings derived from the analytical turns.

Is the Brief Missing Content?

  • Confirm that the brief file sits inside the evidence folder.
  • Repeat the note count check if one of the three synthetic notes is missing.
  • Repeat the affected analytical question if one of the findings is missing.

Help me repair the analyst brief.

Before you create the evidence pack, which infrastructure states do you expect it to preserve?

  • Create evidence/first-half.txt by running this command:
./scripts/evidence-pack.sh

What Enters the Evidence Pack?

The helper gathers actor plus worker plus template plus egress-policy state. It excludes secret values.

The egress policy still records gateway-side authorization replacement through the Kubernetes credential provider. The actor environment still excludes OPENROUTER_API_KEY.

  • Select evidence/first-half.txt in the same project file tree.

You'll see sections for actor state plus worker state plus template state plus egress-policy state. You won't see the temporary OpenRouter key.

Is the Evidence Pack Missing?

  • Return to the same project checkout used for the earlier helper commands.
  • Confirm that scripts/evidence-pack.sh completed without an error.
  • Refresh the text editor's project file tree before checking the evidence folder again.

Help me create the evidence pack.

That's your baseline secured. Next, you'll suspend the analyst and test whether this useful working state survives the full lifecycle.

Prove Suspend and Resume

Your Agent Substrate actor now holds three synthetic notes plus the completed analytical history in process memory. The saved brief captures this baseline before a lifecycle change.

Suspension removes the worker assignment. That release proves the actor is no longer tied to a running worker.

Resume must restore useful memory from a full-memory snapshot. The actor must remain free of the raw API credential.

In this step, get ready to:
  • Prove suspension releases the analyst's worker assignment.
  • Confirm the restored memory matches the saved baseline.
  • Audit the resumed environment for secret isolation.
Suspend the analyst and capture the empty assignment

Suspending the analyst captures its process memory before releasing the active worker. The resulting actor record exposes that lifecycle change through empty worker fields.

  • Predict what will happen to the analyst's worker assignment after suspension.
  • Suspend the analyst by running this command:
kubectl ate suspend actor analyst -a team-a

What Does Suspension Do?

  • The command captures the analyst's process memory in a snapshot.
  • Agent Substrate releases the worker after the snapshot is ready.
  • The actor remains available for a later resume request.

The suspension command completes once Agent Substrate has preserved the actor state.

  • Inspect the suspended actor by running this command:
kubectl ate get actor analyst -a team-a

What Should the Actor Record Show?

  • The actor status identifies the analyst as suspended.
  • The WORKER POD field is empty.
  • The WORKER IP field is empty.

You can now see that the analyst has no active worker assignment. Its useful state survives inside the snapshot.

  • Record the suspended actor state in the existing evidence pack by running its helper again:
./scripts/evidence-pack.sh

What Does the Evidence Helper Capture?

  • The helper records the current actor state.
  • It captures the empty worker assignment while the analyst is suspended.
  • It records the template plus egress-policy state without printing secret values.

Your lifecycle evidence now includes the suspended actor plus its empty worker assignment. That is the first half of the suspend and resume proof.

Does the Analyst Still Have a Worker?

Confirm that the command targets actor analyst in atespace team-a.

Run the actor inspection command again after the suspension finishes. The worker fields become empty once the lifecycle transition completes.

Help me inspect the suspended analyst actor.

Resume the actor and compare its memory

A resume request assigns a worker to the suspended actor. Agent Substrate restores the snapshot into that worker before the analyst serves requests.

  • Resume the analyst from its snapshot by running this command:
kubectl ate resume actor analyst -a team-a

What Does Resume Restore?

  • The command requests a new worker assignment for the analyst.
  • Agent Substrate restores the captured process memory into that worker.
  • The restored memory contains the analyst's notes plus its conversation history.

The analyst returns to its running state with a worker assignment. The next checks reveal whether its useful memory survived.

  • Predict whether the note count plus conversation count will match the baseline.
  • Check the restored working-memory counts by running this command:
./scripts/status.sh analyst

What Does the Status Check Prove?

  • The note count shows whether all three synthetic notes returned.
  • The turn count shows whether the baseline conversation history returned.
  • The environment audit reports whether the raw API key is present.

You should see three notes. The conversation count should match the baseline from before suspension.

  • Render the resumed analyst's current brief by running this command:
./scripts/brief.sh analyst

What Does the Brief Comparison Prove?

  • The rendered brief reads the analyst's restored in-process state.
  • The three synthetic notes should match evidence/analyst-brief.md.
  • The baseline findings should also match the saved brief.

The resumed brief should contain the same notes plus the same baseline analysis. That matching content proves the snapshot restored more than an empty process.

  • Confirm the resumed analyst has a worker assignment by running this command:
kubectl ate get actor analyst -a team-a

What Changed After Resume?

  • The actor record now identifies the analyst as running.
  • The worker fields now contain an assignment.
  • The assigned worker can differ from the worker used before suspension.

That completes the lifecycle proof. The analyst released its worker before returning with the same useful memory.

Do the Restored Counts Look Wrong?

Confirm that every helper targets actor analyst. A different actor has a separate memory history.

Confirm that the resume command uses atespace team-a. Run the status helper again after the actor reaches its running state.

Help me compare the restored analyst state with my baseline.

Audit the resumed actor for secret isolation

The Kubernetes credential provider keeps the OpenRouter key outside the actor. Gateway-side credential injection supplies authorization only when an approved request leaves the actor.

The startup audit gives you direct evidence from the resumed process. It should expose non-secret model configuration without exposing the key.

  • Predict whether the startup audit will report the API key as present or absent.
  • Inspect the resumed actor's startup audit by running this command:
kubectl ate logs actor analyst -a team-a

What Does the Audit Prove?

  • The startup audit reports OPENROUTER_API_KEY present=false.
  • The status output exposes OPENROUTER_MODEL as non-secret configuration.
  • It also exposes OPENROUTER_FALLBACK_MODEL as non-secret configuration.
  • The existing atespace grant remains responsible for gateway authorization.

You should find OPENROUTER_API_KEY present=false in the actor logs. The status check should show only the non-secret model variable names.

Can't Find the Startup Audit?

Confirm that the logs command targets the resumed analyst actor in team-a.

Confirm that the actor has a worker assignment before inspecting its logs. The startup audit belongs to the resumed process.

Help me find the resumed actor's secret-isolation audit.

You have proved both sides of the design. Full-memory restoration preserves the analyst's working state while gateway-held authorization keeps the raw credential outside that state.

Secret mission

Clone and Fork the Analyst

Turn the analyst's suspended memory into a reusable seed for a second actor. Start both analysts from the same knowledge, then give them separate questions and prove that their histories diverge.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to keep the local lab available, pause its runtime, or delete it completely. Your local cluster has no cloud hosting cost.

Cost warning

The selected models openrouter/free and poolside/laguna-s-2.1:free are free.

Accounts that have purchased fewer than 10 credits are limited to 50 free-model requests per day. Accounts with at least 10 purchased credits receive a different allowance.

A different model selection can introduce charges. Removing the temporary key blocks further requests from this lab.

Resources you used:

  • A local kind cluster plus the project-created local registry container.
  • The Agent Substrate control plane plus an Analyst WorkerPool with two gVisor workers.
  • ActorTemplate analyst plus actors analyst and analyst-twin.
  • The full-memory snapshots plus snapshot tag analyst-seed.
  • The two egress policies plus CredentialGrant team-a-to-ns1.
  • The Kubernetes Secret ns1/example-api containing the temporary OpenRouter key.
  • The temporary OpenRouter key plus the local .env file.
  • The starter repository folder containing the generated manifests, actor binary, logs, and evidence files.
  • The evidence files evidence/analyst-brief.md, evidence/analyst-twin-brief.md, and evidence/first-half.txt.

Keep everything running

No cluster deletion is needed. Choose this option if you want to inspect the saved actor state again.

The original analyst actor is already suspended. Its snapshot remains available inside the cluster.

  • Suspend analyst-twin with the actor lifecycle workflow from Step 5.
  • Confirm that both actors are suspended with no worker assignment.
  • Open the OpenRouter Keys page.
  • Remove the temporary key you created for this project.
  • Delete .env from the starter repository folder with your editor file explorer.
  • Leave the local cluster plus its registry container running.

You should see both actors remain suspended with no worker assignment. Their snapshot-backed state stays available without occupying either worker.

Your cluster remains ready for offline lifecycle inspection. External model requests stay disabled until you stage a new key.

Pause - I'll come back to this later

Shut down the running processes to free host resources. Your cluster data plus repository files remain available for another session.

The original analyst actor is already suspended. Pause the twin before stopping the runtime.

  • Suspend analyst-twin with the actor lifecycle workflow from Step 5.
  • Confirm that both actors are suspended with no worker assignment.
  • Open the OpenRouter Keys page.
  • Remove the temporary key you created for this project.
  • Delete .env from the starter repository folder with your editor file explorer.
  • Stop the router port-forward in the terminal panel that is running it.
  • Select your operating system below to stop the container runtime.

macOS

  • Stop Docker Desktop if it powered this lab.
  • Stop Colima if it powered this lab.

The local cluster should become unavailable after the selected runtime stops. Its stored containers remain ready for the next runtime start.

Windows

  • Stop Docker Desktop on Windows.
  • Close the remaining Ubuntu terminal sessions in your WSL2 environment.

The local cluster should become unavailable after Docker Desktop stops. The starter repository remains stored in Ubuntu under $HOME.

Linux

  • Stop Docker Engine with your distribution's service manager.

The local cluster should become unavailable after Docker Engine stops. Its stored containers remain ready for the next service start.

To continue later, restart the same container runtime. Restart the router port-forward before sending another actor request.

Delete - I don't want to use this again

Remove all project resources if you want to start fresh. Your separately installed development tools remain available for future projects.

Deleting the lab permanently removes both actors plus their snapshot-derived histories. Your evidence contains no raw credential value.

  • Open the OpenRouter Keys page.
  • Remove the temporary key you created for this project.
  • Delete .env from the starter repository folder with your editor file explorer.
  • Return to the repository terminal from this project.
  • Remove the local cluster plus its project-created registry by running the supplied cleanup script:
./scripts/cleanup.sh

What does the cleanup remove?

The supplied script calls the pinned hack/delete-kind-cluster.sh workflow. That workflow deletes the local kind cluster.

Deleting the cluster removes both actors, their snapshots, the snapshot tag, both egress policies, the grant, the WorkerPool, the ActorTemplate, and the Kubernetes Secret.

The workflow removes kind-registry only when it carries the project label created-by=agent-substrate.

  • Confirm that the cleanup script exits without an error.
  • Confirm that the cluster's node containers are no longer running in your container runtime.
  • Confirm that kind-registry is no longer running in your container runtime.

That closes the local cluster cleanly. The actors plus their snapshot-derived state are no longer reachable.

Cleanup script failed?

  • Confirm that your container runtime is running.
  • Confirm that your terminal is still in the starter repository folder.
  • Retry the supplied cleanup script after the runtime is responsive.

Get help diagnosing the cleanup failure.

The starter repository folder is the remaining local resource. Deleting it also removes your generated manifests, actor binary, logs, and evidence files.

  • Select your operating system below to delete the starter repository folder.

macOS

  • Close the editor window for the starter repository.
  • Locate the starter repository folder in the parent directory where you cloned it.
  • Delete the starter repository folder with your file manager.

Confirm that the starter repository folder no longer appears in its parent directory. Your local project files are now removed.

Windows

  • Close the editor window for the starter repository.
  • Locate the starter repository folder under Ubuntu's $HOME directory.
  • Delete the starter repository folder with your file manager.

Confirm that the starter repository folder no longer appears under $HOME. Your local project files are now removed.

Linux

  • Close the editor window for the starter repository.
  • Locate the starter repository folder in the parent directory where you cloned it.
  • Delete the starter repository folder with your file manager.

Confirm that the starter repository folder no longer appears in its parent directory. Your local project files are now removed.

Nice Work!

Nice Work!

You did it! You built a stateful AI analyst on Agent Substrate. It preserves working memory across suspension without receiving the raw OpenRouter key.

You've learned how to:

  • Run a custom AI analyst as a gVisor-isolated actor on a local Kubernetes cluster with two workers.
  • Meet a planned HTTP 403 at the default-deny gateway. Resolve it with an explicit atespace-to-namespace grant that lets the gateway inject authorization.
  • Restore the analyst's complete state from a full-memory snapshot. The restored state includes three synthetic notes plus its baseline conversation history. Confirm OPENROUTER_API_KEY remains absent from the actor environment.
  • Secret Mission: Clone the tagged analyst-seed snapshot into analyst-twin. Fork the two analysts with separate follow-up questions. Their turn counts differ. Their final findings diverge.

Ready to quiz yourself?