Inspect Agent Substrate Actors

Run local Agent Substrate actors and prove state restoration and worker reuse.

Introduction

30 Second Summary

Most services keep a machine busy even when nobody is using them. Across many workloads, that idle capacity quickly piles up.

In this project, you will run a pinned Agent Substrate lab where traffic wakes a suspended Actor on demand. Repeatable assertions prove continuity across process-memory snapshots and DurableDir state.

What You'll Build

You can send one HTTP request that wakes a suspended counter inside a gVisor sandbox with both saved counter values intact.

By the end of this project, you'll have:

A lifecycle trace where portfolio-counter wakes from suspension and displays its assigned worker pod.
Exact persistence assertions proving that the process-memory counter and file-backed counter each continue by one after suspension.
A portfolio walkthrough that pairs an unchanged Kubernetes pod list with changing WorkerPool assignments and a control-plane diagram.
Secret Mission: Fill all three Counter Demo workers, prove a fourth Actor cannot obtain capacity, release one worker, and assert that the fourth Actor receives that exact pod.

Are there any prerequisites?

You need an Intel Mac running macOS 14 or later. You also need internet access, local administrator access, and enough Docker Desktop resources for a single-node development cluster.

Step 1 verifies every required tool and covers conditional setup before the core lab begins.

Before We Start

This opening checkpoint gives you a moment to define the local Counter Demo experiment before any infrastructure changes. Your evidence will cover request-driven actor resume, process-memory snapshot continuity, DurableDir state continuity, plus reusable WorkerPool capacity in a pinned Agent Substrate environment.

Prepare Your Intel Mac and Pin the Lab

The Agent Substrate project has pre-1.0 status. A moving source branch could change the commands or lifecycle behavior behind your experiment.

This preparation locks every later observation to one revision. You will also prepare the local toolchain before starting the 44-minute core learning clock.

In this step, get ready to:

Verify the required Intel Mac development toolchain.
Pin the official Agent Substrate source to the project revision.
Install the matching Substrate plugin and confirm every tool responds.

Verify the local toolchain

This lab requires an Intel Mac running macOS 14 or later. You also need local administrator access for the system-level installers.

Press Cmd+Space to open Spotlight.
Type Terminal and press Return to open Apple Terminal.

Command Line Tools for Xcode supplies the developer utilities that the source workflow depends on. Start by checking whether macOS already has an active developer directory.

Check the active developer directory by running this command:

xcode-select --print-path

What does this check do?

The command prints the directory used by Apple developer tools. A returned developer path confirms that Command Line Tools for Xcode is available.

✔️ I see a developer path

Your Apple developer tools are ready. Keep this Terminal window open for the remaining checks.

ⓧ The command reports an error

macOS opens an installation dialog after the next command. The installer may request your administrator approval.

Start the Command Line Tools for Xcode installer by running this command:

xcode-select --install

What does this command do?

The command asks macOS to install Apple developer tools. These tools include the Git client used to obtain the project source.

Complete the macOS installation dialog.
Confirm the active developer directory by running the check again:

xcode-select --print-path

What should you see?

You should now see an Apple developer directory. That path confirms the installation is active.

Still missing a developer path?

Restart Terminal after the installer finishes. Confirm that the installation dialog reached its completion state.

Ask for help with the exact message you see: Help me troubleshoot Command Line Tools for Xcode on my Intel Mac.

The local Docker Desktop engine hosts the development cluster. The pinned Intel release keeps the container environment reproducible.

Check the Docker Desktop release and engine connection by running these commands:

docker desktop version --short
docker version

What do these checks prove?

The first command reports the Docker Desktop release.
The second command reports client information plus server information. A server section confirms that Docker Desktop is running.

✔️ I see version 4.95.0

That is the pinned Docker Desktop release. The server information also proves that its local engine is accepting requests.

ⓧ I see another version

The architecture choice is easy to miss. Use the Intel installer for Docker Desktop 4.95.0 so the lab matches the pinned environment.

Open the Docker Desktop release notes.
Find release 4.95.0.
Select the Mac download for the Intel chip.
Double-click Docker.dmg to open the installer.
Drag the Docker icon into the Applications folder.
Double-click Docker.app in the Applications folder.
Review the Docker Subscription Service Agreement shown on first launch.
Verify the pinned release and running engine by repeating these commands:

docker desktop version --short
docker version

What should you see?

The first command should report 4.95.0. The second command should include server information from Docker Desktop.

Docker engine unavailable?

Use Spotlight to start Docker Desktop. Wait for the application to report that its engine is running before repeating the check.

Ask for help with the output from both commands: Help me diagnose why Docker Desktop is installed but its engine is unavailable.

ⓧ Docker is not installed

Docker Desktop installation requires administrator access. The application can run as your regular account after installation.

Open the Docker Desktop release notes.
Find release 4.95.0.
Select the Mac download for the Intel chip.
Double-click Docker.dmg to open the installer.
Drag the Docker icon into the Applications folder.
Double-click Docker.app in the Applications folder.
Review the Docker Subscription Service Agreement shown on first launch.
Verify the installation and running engine by repeating these commands:

docker desktop version --short
docker version

What should you see?

The first command should report 4.95.0. The second command should include client information plus server information.

Docker command still missing?

Close Terminal after Docker Desktop finishes its first launch. Open a fresh Terminal window so it receives the updated command path.

Ask for help with the missing command: Help me make the Docker command available in Apple Terminal.

The pinned source uses Go to build its command-line plugin. The Intel package provides the required darwin/amd64 toolchain.

Check the installed Go version by running this command:

go version

What does this check show?

The command reports the Go release plus its target operating system and architecture. This lab requires release 1.27.2 for darwin/amd64.

✔️ I see version 1.27.2

Your Go compiler matches the lab pin. Continue to the shared path configuration below the tabs.

ⓧ I see an older version

Open the official Go downloads page.
Download go1.27.2.darwin-amd64.pkg.
Open go1.27.2.darwin-amd64.pkg from your Downloads folder.
Complete the macOS installer prompts.
Confirm the replacement version in a fresh Terminal window by running:

go version

What should you see?

The output should report Go 1.27.2 for darwin/amd64.

ⓧ Command not found

Open the official Go downloads page.
Download go1.27.2.darwin-amd64.pkg.
Open go1.27.2.darwin-amd64.pkg from your Downloads folder.
Complete the macOS installer prompts.
Confirm the installation in a fresh Terminal window by running:

go version

What should you see?

The output should report Go 1.27.2 for darwin/amd64.

Go places installed tools in its workspace binary directory. Adding that directory to your shell path makes the Substrate plugin available through future Terminal sessions.

Add this line to ~/.zshrc with your text editor:

export PATH="$PATH:$(go env GOPATH)/bin"

What does this configuration do?

The line adds Go's binary installation directory to PATH. Terminal can then find tools installed with go install.

Apply the updated shell path and recheck Go by running these commands:

source ~/.zshrc
go version

What should you see?

The shell reloads your profile before reporting Go 1.27.2. The same path will expose the Substrate plugin later in this step.

Go version still incorrect?

Close every open Terminal window after the package installation. Open a fresh window before checking the version again.

Ask for help with the reported path and version: Help me fix my Go installation on an Intel Mac.

The kubectl client inspects the Kubernetes resources created later. Its Intel binary must match the pinned client release.

Check the installed kubectl client by running this command:

kubectl version --client

What does this check show?

The command reports the local kubectl client version. This check does not require a running cluster.

✔️ I see version 1.36.1

Your kubectl client matches the lab pin. Its location on PATH is already working.

ⓧ I see an older version

The install command asks for your Mac administrator password when it moves kubectl into /usr/local/bin. Terminal keeps the password hidden while you type.

Replace the older kubectl client with the verified Intel release by running these commands:

curl -LO "https://dl.k8s.io/release/v1.36.1/bin/darwin/amd64/kubectl"
curl -LO "https://dl.k8s.io/release/v1.36.1/bin/darwin/amd64/kubectl.sha256"
echo "$(cat kubectl.sha256)  kubectl" | shasum -a 256 --check
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
sudo chown root: /usr/local/bin/kubectl
rm kubectl.sha256
kubectl version --client

What do these commands do?

The first two commands download kubectl 1.36.1 plus its published checksum for darwin/amd64.
The checksum command verifies the binary before installation. A valid download prints kubectl: OK.
The remaining commands make kubectl executable. They place it in /usr/local/bin before reporting the installed client version.

Checksum or permission failure?

Delete the downloaded files after a checksum failure. Download both files again from the same release before retrying.

Ask for help with the exact output: Help me diagnose my kubectl installation failure.

ⓧ Command not found

The install command asks for your Mac administrator password when it moves kubectl into /usr/local/bin. Terminal keeps the password hidden while you type.

Install the verified Intel kubectl release by running these commands:

curl -LO "https://dl.k8s.io/release/v1.36.1/bin/darwin/amd64/kubectl"
curl -LO "https://dl.k8s.io/release/v1.36.1/bin/darwin/amd64/kubectl.sha256"
echo "$(cat kubectl.sha256)  kubectl" | shasum -a 256 --check
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
sudo chown root: /usr/local/bin/kubectl
rm kubectl.sha256
kubectl version --client

What do these commands do?

The first two commands download kubectl 1.36.1 plus its published checksum for darwin/amd64.
The checksum command verifies the binary before installation. A valid download prints kubectl: OK.
The remaining commands make kubectl executable. They place it in /usr/local/bin before reporting the installed client version.

kubectl still unavailable?

Open a fresh Terminal window after installation. Confirm that /usr/local/bin is included in your shell path.

Ask for help with the exact output: Help me make kubectl available in Apple Terminal.

Pin the Agent Substrate source

A Git commit identifies one exact source state. Pinning the checkout prevents later changes on the main branch from altering your experiment.

Move to your Desktop by running this command:

cd ~/Desktop

Why use the Desktop?

This location keeps the substrate repository easy to find. Every later repository command uses the same folder.

Clone the official source and list the resulting folder by running these commands:

git clone https://github.com/agent-substrate/substrate.git
ls

What do these commands do?

Git copies the official repository history into ~/Desktop/substrate. The directory listing confirms that the new substrate folder exists.

Repository clone failed?

Confirm that your Mac has internet access. Remove any incomplete substrate folder before repeating the clone.

Ask for help with the Git output: Help me troubleshoot cloning the Agent Substrate repository.

Check out the pinned project revision by running these commands:

cd substrate
git checkout ac5c5ee0a001b3c25008f0ba713742d8402e1dfd

Why pin this commit?

The checkout fixes the repository at ac5c5ee0a001b3c25008f0ba713742d8402e1dfd. The lab can now rely on the matching scripts plus resource definitions.

Before you check the revision, predict whether Git will print the pinned commit or a moving branch tip.

Print the current commit by running this command:

git rev-parse --verify HEAD

What does this verification prove?

Git resolves HEAD to the exact commit currently checked out. The printed value must match the lab revision character for character.

You should see ac5c5ee0a001b3c25008f0ba713742d8402e1dfd. Your source now matches the revision used throughout this project.

Commit value does not match?

Confirm that Terminal is inside ~/Desktop/substrate. Repeat the checkout command before printing HEAD again.

Ask for help with both Git commands: Help me pin my Agent Substrate checkout to the required commit.

Install the matching Substrate plugin

The repository builds kubectl-ate from the pinned source. It also manages the matching kind plus ko development dependencies.

Why use repository-managed dependencies?

The repository scripts select the development dependencies expected by this commit. Separate installations could introduce versions that the pinned workflow did not select.

Return to the pinned repository by running this command:

cd ~/Desktop/substrate

Why return to this folder?

The plugin installation uses a relative package path inside the repository. Running it from ~/Desktop/substrate ensures Go finds the matching source.

Build and install the matching plugin by running this command:

go install ./cmd/kubectl-ate

What does this command do?

Go compiles the kubectl-ate package from the pinned checkout. It installs the resulting executable into the Go binary directory added to your shell path.

Plugin installation failed?

Confirm that the current folder is ~/Desktop/substrate. Confirm that go version reports the pinned Go release.

Ask for help with the build output: Help me troubleshoot the kubectl-ate installation from the pinned repository.

Before you run the final check, predict whether every command will report its installed tool without an error.

Verify the complete local environment by running these commands:

go version
kubectl version --client
xcode-select --print-path
kubectl ate --version

What does the final check prove?

The Go command confirms release 1.27.2 on the Intel architecture.
The kubectl command confirms client release 1.36.1.
The Apple command confirms an active developer-tools path.
The plugin command confirms that the pinned Substrate CLI is available through kubectl.

That is the preparation complete. Your Intel Mac now has the pinned source plus every local command needed to run the lab.

One final check still failing?

Match the failing command to its setup section above. Open a fresh Terminal window after any installer or shell-path change.

Ask for help with the complete final output: Help me finish my Agent Substrate prerequisite verification.

Your pinned lab foundation is ready. Next, you will launch Agent Substrate and inspect the boundary between Kubernetes infrastructure and Substrate-native actor records.

Launch Substrate and Inspect the Control Planes

Your Intel Mac now has the pinned Agent Substrate checkout. Its required local tools are ready.

A Kubernetes-only view leaves the Actor lifecycle out of sight. You will launch the lab before comparing its infrastructure plane with its actor control plane.

In this step, get ready to:

Launch the local development cluster.
Deploy the Agent Substrate lab services.
Separate infrastructure resources from actor records across the two control planes.

Bring up the local foundation

The repository scripts turn the pinned source into a reproducible local kind environment. Running them from the repository keeps the deployment tied to the revision you checked out.

Create the single-node cluster from the pinned repository by running this command:

hack/create-kind-cluster.sh

What does this script create?

The script creates a single-node kind cluster. It also starts a project-created local container registry for development images.

That is the local foundation in place. The cluster is now connected to kubectl.

Cluster creation failed?

Confirm Docker Desktop is still running.
Confirm Terminal is still inside the pinned Agent Substrate repository.
Wait for Docker Desktop to finish starting before retrying the command.

Help me diagnose why the pinned Agent Substrate cluster script cannot create my local kind cluster.

The control-plane deployment can take several minutes on an Intel Mac. Ongoing terminal output means the local build is progressing.

Deploy the local Agent Substrate control plane by running this command:

hack/install-ate-kind.sh --deploy-ate-system --credential-provider='{"name":"k8s.io"}'

What does this deployment add?

The deployment creates Agent Substrate services in the ate-system namespace.
The deployment configures local snapshot storage for suspended actors.
The deployment brings up ate-api-server for control-plane operations.
The deployment brings up the atenet-router service for later routed requests.

When the command returns to your shell prompt without an error, the local control plane is ready for the demo workload.

Control plane deployment stopped?

Check that the cluster creation command completed successfully.
Confirm Docker Desktop has enough free resources for the local cluster.
Retry the deployment from the pinned repository after resolving the reported build failure.

Help me troubleshoot the local Agent Substrate control-plane deployment.

Deploy the Counter Demo

The Counter Demo adds a WorkerPool with three warm worker pods. Those pods stay ready before any actor receives an assignment.

Add the pinned Counter Demo to the local cluster by running this command:

hack/install-ate-kind.sh --deploy-demo-counter

What does the demo deploy?

The demo creates WorkerPool counter with 3 replicas in ate-demo-counter.
The demo creates the counter ActorTemplate for logical counter actors.
The template uses SNAPSHOT_FIDELITY_MEMORY so later snapshots can preserve process memory.
The template uses DurableDir for file-backed state.

The Counter Demo is now deployed. Its warm workers are ready for inspection.

Counter Demo deployment failed?

Confirm the Agent Substrate control-plane deployment finished without an error.
Confirm Terminal remains inside the pinned repository.
Retry the Counter Demo command after the control-plane services are available.

Help me find why the Counter Demo deployment did not complete.

Before you inspect the cluster, which Kubernetes resource do you think will reveal the configured worker capacity?

Inspect the WorkerPool, worker pods, and node-level agent by running these commands:

kubectl get workerpool counter -n ate-demo-counter
kubectl get pods -n ate-demo-counter
kubectl get daemonset -n ate-system

What should you see?

The WorkerPool output shows counter configured with three replicas.
The pod output shows three warm worker pods in ate-demo-counter.
The DaemonSet output shows atelet running at the node level.

Record the WorkerPool replica count as 3 in your evidence notes.
Identify the three pods in ate-demo-counter as warm workers in your evidence notes.
Identify atelet as the node-level DaemonSet in your evidence notes.

Missing workers or atelet?

Check the namespace spelling in each command.
Confirm the Counter Demo deployment command completed before repeating the worker checks.
Confirm the control-plane deployment completed before repeating the DaemonSet check.

Help me diagnose missing Counter Demo workers or an unavailable atelet DaemonSet.

Reveal the suspended actor

A logical actor can exist before it owns a worker. Creating one now gives you a clean lifecycle baseline for the routing experiment.

Create the portfolio counter actor from the demo template by running this command:

kubectl ate create actor portfolio-counter -a ate-demo-counter --template counter

What does actor creation do?

This command registers portfolio-counter in the ate-demo-counter atespace. The actor begins suspended without consuming a worker pod.

The actor record now exists. Its initial state is ready to inspect.

Actor creation failed?

Confirm the matching kubectl-ate plugin remains available on your PATH.
Confirm the Counter Demo deployment completed successfully.
Check that the actor name is portfolio-counter.

Help me troubleshoot why the counter actor could not be created.

Before you run this Kubernetes query, do you think it can return the new actor as a native resource?

Test the Kubernetes-only view by running this query:

kubectl get actor

Why is the actor missing here?

This Kubernetes-only query cannot return portfolio-counter as an actor row. That gap is the intended result.

Kubernetes maintains the WorkerPool, pods, namespace, and DaemonSet. Agent Substrate stores actor lifecycle records outside the Kubernetes API hot path.

Before you query the Substrate view, do you expect the new actor to have a worker assignment already?

Inspect the actor baseline and worker assignments through Agent Substrate by running these commands:

kubectl ate get actor portfolio-counter -a ate-demo-counter
kubectl ate get workers

What should you see?

The actor row places portfolio-counter in atespace ate-demo-counter.
Its template is counter.
Its state is ACTOR_STATE_SUSPENDED.
Its WORKER POD field is empty.
Its VERSION field contains the initial lifecycle value.
The worker table shows three Counter Demo rows with FREE status.

Record the actor's VERSION value here: your initial actor version.

Actor state or worker count looks different?

Confirm the actor name is portfolio-counter.
Confirm the atespace is ate-demo-counter.
Confirm no request has been routed to the actor yet.

Help me compare my actor row and worker table with the expected suspended baseline.

That is a strong baseline: the worker pool is ready while portfolio-counter remains suspended. Next, you will route a request that wakes the actor onto one of those free workers.

Route a Request and Wake the Actor

Your Counter Demo now has a suspended actor. Three warm workers are available.

The next goal is to wake the actor with an HTTP request. You will connect through the Agent Substrate router.

A routed request must identify the actor before traffic can reach it. The control plane must resume the actor into a warm worker before the router forwards the waiting request.

In this step, get ready to:

Expose the local router through a port-forward.
Send a targeted POST request that resumes portfolio-counter.
Confirm the actor's request-driven resume.

Expose the router in Terminal A

The router already runs inside the local cluster. A port-forward gives your Mac a temporary path to that service through localhost:8000.

The port-forward process must remain active while you send requests. You will keep it running in a dedicated terminal window.

Switch back to the Apple Terminal window from earlier.
Start the long-running port-forward by running this command:

kubectl port-forward -n ate-system svc/atenet-router 8000:80

What does this command do?

The -n ate-system option targets the namespace containing the router service.
The svc/atenet-router value selects the local Agent Substrate router.
The 8000:80 mapping sends traffic from port 8000 on your Mac to port 80 on the service.

Terminal A displays output confirming that traffic is being forwarded. Its command prompt stays occupied while the connection remains active.

Leave Terminal A running for the rest of this experiment.

Port-forward not staying active?

Confirm Docker Desktop is still running.
Confirm your local kind cluster is still connected to kubectl.
Check whether another local process is already using port 8000.

Help me diagnose why the Agent Substrate router port-forward stops or fails to start.

Press Cmd+Space to open macOS search.
Type Terminal in the search field.
Press Enter to open a second Terminal window.
Use the new window as Terminal B.

Send the routed POST request

The request carries the actor's address in the ate-target-actor header. The router uses that address to find portfolio-counter in the ate-demo-counter atespace.

Before you send the request, predict whether the actor can answer while it is still suspended.

Send the request from Terminal B by running this command:

curl -X POST -H "ate-target-actor: ate-demo-counter/portfolio-counter" http://localhost:8000

How does the waiting request move?

The router reads ate-demo-counter/portfolio-counter from the request header.
The API server selects one free Counter Demo worker for the actor.
Atelet and ateom restore the gVisor sandbox inside that worker.
The router opens the worker tunnel after the actor is ready. The original request then reaches the counter.

You will see one response line containing preserved memory count and preserved file counter. The value after hello from: identifies the actor IP.

There it is. Your suspended actor has answered through a warm worker.

Copy the complete response line from Terminal B into your notes.

Request not returning a counter response?

Confirm Terminal A still shows an active port-forward.
Check that the request header targets ate-demo-counter/portfolio-counter exactly.
Confirm the request uses http://localhost:8000.

Help me troubleshoot why the routed POST request is not returning the Counter Demo response.

Match the actor to its worker

The actor table records lifecycle state through the Agent Substrate control plane. The worker table links the logical actor to one physical pod.

Before you inspect the tables, predict whether the actor still has an empty worker assignment.

Inspect the resumed actor and the shared worker pool by running these commands:

kubectl ate get actor portfolio-counter -a ate-demo-counter
kubectl ate get workers

What do these tables prove?

The first command reads the actor record from the Agent Substrate control plane.
The STATE column shows the actor's current lifecycle state.
The WORKER POD column identifies the pod hosting the running actor.
The second command shows each worker as FREE or ASSIGNED.

You should see portfolio-counter in ACTOR_STATE_RUNNING. The WORKER POD column should contain a pod name.

Its VERSION should be higher than the creation value you recorded. This confirms that the lifecycle record changed during request-driven resume.

The worker table should show one ASSIGNED row for portfolio-counter. The other two Counter Demo workers should remain FREE.

Compare the current VERSION with the creation version from earlier.
Match the actor's WORKER POD value to the POD value in the ASSIGNED worker row.
Document the lifecycle sequence as ACTOR_STATE_SUSPENDED, ACTOR_STATE_RESUMING, and ACTOR_STATE_RUNNING.

Actor still suspended or unassigned?

Confirm the routed request returned a complete counter response before inspecting the actor.
Confirm Terminal A still has the router port-forward running.
Run the inspection commands again after the request finishes.

Help me determine why portfolio-counter remains suspended or has no assigned worker after the routed request.

Strong result. You have tied the routed response to a running actor with a specific worker assignment.

Next, you will suspend this actor. Exact counter assertions will test whether both state layers survive.

Prove Snapshot State Continuity

Your first routed request woke portfolio-counter from suspension. The actor now occupies a warm Counter Demo worker.

A normal process restart loses values held in process memory. This creates the central question for a stateful actor.

In this step, executable assertions test whether Agent Substrate restores that memory after suspension. A second assertion tests the actor's DurableDir state.

In this step, get ready to:

Capture both counters before suspension.
Observe the actor's suspended lifecycle boundary.
Prove exact counter continuity after request-driven resume.

Capture the pre-suspend baseline

Each Counter Demo response contains an in-memory count. It also contains a file-backed count.

Two shell functions extract those values from a routed response. The extracted numbers become the baseline for the final arithmetic checks.

Switch back to Terminal B from your routed request.
Define both parsers by running the block below:

memory_count() {
  printf '%s\n' "$1" | sed -E 's/.*preserved memory count: ([0-9]+).*/\1/'
}

file_count() {
  printf '%s\n' "$1" | sed -E 's/.*preserved file counter: ([0-9]+).*/\1/'
}

before=$(curl -X POST \
  -H "ate-target-actor: ate-demo-counter/portfolio-counter" \
  http://localhost:8000)
before_memory=$(memory_count "$before")
before_file=$(file_count "$before")
printf 'Before suspend: memory=%s file=%s\n' "$before_memory" "$before_file"

What does this baseline do?

The memory_count() function extracts the preserved memory count from a complete response.
The file_count() function extracts the preserved file counter from the same response format.
The before variable stores one fresh routed response before suspension.
The final line prints both parsed baseline values for comparison after resume.

Confirm that the Before suspend: line contains a numeric memory value.
Confirm that the same line contains a numeric file value.

Your two baseline numbers now capture the actor's state immediately before the lifecycle boundary.

Seeing an empty counter value?

Check that Terminal A still shows active forwarding for the router service.
Confirm that the routed response includes the preserved memory count field.
Confirm that the routed response includes the preserved file counter field.

Help me debug the counter parser output.

Suspend the actor and inspect the release

The Counter ActorTemplate uses SNAPSHOT_FIDELITY_MEMORY. Suspending the actor captures a process-memory snapshot plus its durable volume.

The suspension also releases the assigned worker. The actor record remains under the Substrate control plane.

Before you suspend the actor, which assignment field do you expect to clear when its worker returns to the pool?

Suspend portfolio-counter by running this block:

kubectl ate suspend actor portfolio-counter -a ate-demo-counter
kubectl ate get actor portfolio-counter -a ate-demo-counter | grep ACTOR_STATE_SUSPENDED
kubectl ate get workers

What does this lifecycle check prove?

The suspend operation moves the actor from ACTOR_STATE_RUNNING through ACTOR_STATE_SUSPENDING to ACTOR_STATE_SUSPENDED.
The actor query confirms that the lifecycle reached its suspended state.
The worker table shows that the released assignment has returned to the WorkerPool as free capacity.

Confirm that the actor's WORKER POD field is empty.
Record the actor's higher VERSION in your project notes.
Confirm that the worker table now contains three FREE rows.

The lifecycle boundary is now visible. The actor is suspended while all three long-running worker pods remain available in Kubernetes.

Actor still holding a worker?

Wait for the suspension operation to finish before checking the actor row.
Confirm that the actor query reports ACTOR_STATE_SUSPENDED.
Rerun the worker inspection command after the suspended state appears.

Help me inspect the suspended actor assignment.

Resume through traffic and assert continuity

The next routed request asks the control plane to restore the suspended actor into a free worker. The waiting request reaches the counter after the actor becomes ready.

The arithmetic checks compare the restored counters with the baseline. This turns state continuity into a repeatable test.

Before you run the assertion, do you expect the counters to continue from the baseline or restart from their initial values?

Trigger request-driven resume by running this assertion block:

after=$(curl -X POST \
  -H "ate-target-actor: ate-demo-counter/portfolio-counter" \
  http://localhost:8000)
after_memory=$(memory_count "$after")
after_file=$(file_count "$after")
printf 'After resume: memory=%s file=%s\n' "$after_memory" "$after_file"

test "$after_memory" -eq "$((before_memory + 1))" && \
test "$after_file" -eq "$((before_file + 1))" && \
echo "PASS: memory and durable-file counters survived suspend/resume"

kubectl ate get actor portfolio-counter -a ate-demo-counter | grep ACTOR_STATE_RUNNING

How does the assertion prove continuity?

The after variable stores the first response after request-driven resume.
The parser functions extract the restored memory value and durable-file value.
Each test requires its post-resume value to equal the corresponding baseline plus exactly one.
The final actor query confirms that portfolio-counter returned to ACTOR_STATE_RUNNING with a populated worker assignment.

Confirm that Terminal B prints PASS: memory and durable-file counters survived suspend/resume.
Confirm that the actor query reports ACTOR_STATE_RUNNING.

That is the core proof complete. Your actor restored its process state plus its DurableDir data across a real suspend and resume cycle.

Missing the PASS result?

Check that you ran the baseline block in the same Terminal B session.
Confirm that both baseline variables contain numeric values.
Check that Terminal A still has the router port-forward running at localhost:8000.

Help me debug the suspend and resume assertion.

✔️ Awesome, I've got everything!

Your Terminal B session now contains both parser functions plus the complete suspend and resume evidence.

ⓧ I'd like to double check the full code

memory_count() {
  printf '%s\n' "$1" | sed -E 's/.*preserved memory count: ([0-9]+).*/\1/'
}

file_count() {
  printf '%s\n' "$1" | sed -E 's/.*preserved file counter: ([0-9]+).*/\1/'
}

before=$(curl -X POST \
  -H "ate-target-actor: ate-demo-counter/portfolio-counter" \
  http://localhost:8000)
before_memory=$(memory_count "$before")
before_file=$(file_count "$before")
printf 'Before suspend: memory=%s file=%s\n' "$before_memory" "$before_file"

kubectl ate suspend actor portfolio-counter -a ate-demo-counter
kubectl ate get actor portfolio-counter -a ate-demo-counter | grep ACTOR_STATE_SUSPENDED
kubectl ate get workers

after=$(curl -X POST \
  -H "ate-target-actor: ate-demo-counter/portfolio-counter" \
  http://localhost:8000)
after_memory=$(memory_count "$after")
after_file=$(file_count "$after")
printf 'After resume: memory=%s file=%s\n' "$after_memory" "$after_file"

test "$after_memory" -eq "$((before_memory + 1))" && \
test "$after_file" -eq "$((before_file + 1))" && \
echo "PASS: memory and durable-file counters survived suspend/resume"

kubectl ate get actor portfolio-counter -a ate-demo-counter | grep ACTOR_STATE_RUNNING

What does the full check cover?

This reference includes the two parser functions plus both routed requests. It also includes the lifecycle inspections and exact arithmetic assertions.

Your persistence proof is repeatable. Next, you will release and reacquire shared worker capacity without changing the Kubernetes pod set.

Reuse the WorkerPool and Document the Architecture

Your counter has survived a full suspend and request-driven resume cycle. Both state layers now have executable proof.

A fixed pod set can hide changing actor assignments. This step makes that movement visible with Agent Substrate, a WorkerPool, and Kubernetes observations.

In this step, get ready to:

Activate a second counter actor on the shared WorkerPool.
Prove that one assignment can be released and reacquired without changing Kubernetes pods.
Document the architecture boundary with live lifecycle evidence.

Create and activate a second actor

A second running actor raises assigned capacity from one worker to two. The remaining free worker makes the upcoming release test easy to observe.

Create the second actor and capture baseline capacity by running this block:

kubectl ate create actor portfolio-counter-2 -a ate-demo-counter --template counter
curl -X POST \
  -H "ate-target-actor: ate-demo-counter/portfolio-counter-2" \
  http://localhost:8000

assigned_count() {
  kubectl ate get workers | \
    awk 'NR > 1 && $4 == "ASSIGNED" {n++} END {print n+0}'
}

assigned_before=$(assigned_count)
printf 'Assigned before release: %s\n' "$assigned_before"
kubectl get pods -n ate-demo-counter

What does this block prove?

The actor creation command adds portfolio-counter-2 to the existing atespace.
The routed POST wakes the new actor onto one free worker.
The assigned_count function counts worker rows whose status is ASSIGNED.
The final pod listing records the Kubernetes infrastructure before any capacity is released.

You should see a counter response from portfolio-counter-2. The assigned count should be 2 while kubectl still lists three worker pods.

That baseline is locked in: two logical actors now occupy two warm workers.

Second actor not running?

Return to Terminal A to confirm the router port-forward from earlier is still running.
Review the routed response for signs that the request did not reach the counter actor.
Review the worker table printed by the block to confirm that a free worker was available.

Help me diagnose why portfolio-counter-2 did not obtain a worker assignment.

Release and reacquire one worker

Worker multiplexing separates an actor assignment from the lifetime of its Kubernetes pod. Suspending an actor should free capacity while every warm worker pod remains in place.

Before you run this, predict whether the pod names will stay unchanged as the assignment count drops and rises.

Suspend the second actor and test both capacity changes by running this block:

kubectl ate suspend actor portfolio-counter-2 -a ate-demo-counter
assigned_after_release=$(assigned_count)
test "$assigned_after_release" -eq "$((assigned_before - 1))" && \
echo "PASS: one assignment released"

kubectl ate resume actor portfolio-counter-2 -a ate-demo-counter
assigned_after_resume=$(assigned_count)
test "$assigned_after_resume" -eq "$assigned_before" && \
echo "PASS: released capacity reacquired"

kubectl get pods -n ate-demo-counter
kubectl ate get workers

How do the assertions work?

The suspension snapshots portfolio-counter-2 before returning its worker to the pool.
The first arithmetic test requires the assigned count to fall by exactly one.
The explicit resume restores the actor onto available worker capacity.
The second arithmetic test requires the assigned count to return to its baseline.

You should see PASS: one assignment released followed by PASS: released capacity reacquired. The same three worker pod names should appear after both assignment changes.

You have now proved that actor capacity moves independently of Kubernetes pod creation.

Missing a capacity PASS result?

Review the printed worker table to confirm that portfolio-counter-2 reached a free worker during the baseline run.
Check the actor row to confirm that suspension finished before the first count was captured.
Check the actor row to confirm that the explicit resume returned it to ACTOR_STATE_RUNNING.

Help me debug which WorkerPool capacity assertion failed.

✔️ Awesome, I've got everything!

Your shell now contains the reusable assigned_count function. Both capacity assertions have passed.

ⓧ I'd like to double check the full code

kubectl ate create actor portfolio-counter-2 -a ate-demo-counter --template counter
curl -X POST \
  -H "ate-target-actor: ate-demo-counter/portfolio-counter-2" \
  http://localhost:8000

assigned_count() {
  kubectl ate get workers | \
    awk 'NR > 1 && $4 == "ASSIGNED" {n++} END {print n+0}'
}

assigned_before=$(assigned_count)
printf 'Assigned before release: %s\n' "$assigned_before"
kubectl get pods -n ate-demo-counter

kubectl ate suspend actor portfolio-counter-2 -a ate-demo-counter
assigned_after_release=$(assigned_count)
test "$assigned_after_release" -eq "$((assigned_before - 1))" && \
echo "PASS: one assignment released"

kubectl ate resume actor portfolio-counter-2 -a ate-demo-counter
assigned_after_resume=$(assigned_count)
test "$assigned_after_resume" -eq "$assigned_before" && \
echo "PASS: released capacity reacquired"

kubectl get pods -n ate-demo-counter
kubectl ate get workers

How to use this reference

This reference combines the complete second-actor setup with the release and reacquisition assertions. Compare it with the commands in your current Terminal session.

Document the control-plane boundary

The left side of the diagram shows infrastructure owned through Kubernetes. The right side shows actor metadata owned through the Agent Substrate API and PostgreSQL.

The request path connects both control planes through atelet, ateom, and gVisor. The snapshot store preserves process memory plus DurableDir state between assignments.

Capture the architecture boundary in your portfolio notes by copying this diagram:

Kubernetes API and etcd                 Agent Substrate API and PostgreSQL
-----------------------                 ---------------------------------
Namespace: ate-demo-counter             Atespace: ate-demo-counter
WorkerPool: counter, replicas: 3        ActorTemplate: counter
Worker Pods: warm, long-running         Actor records and lifecycle versions
DaemonSet: atelet                       Worker FREE/ASSIGNED records
Infrastructure scheduling              Snapshot and assignment metadata
           |                                          |
           +------------------+-----------------------+
                              |
Client -> atenet-router -> ate-api-server -> atelet -> ateom/gVisor actor
                              |                         |
                              +---- snapshot store <----+
                                   memory + DurableDir

Actor lifecycle:
CREATE -> SUSPENDED -> RESUMING -> RUNNING
RUNNING -> SUSPENDING -> SUSPENDED -> RESUMING -> RUNNING

Multiplexing:
Many logical actors -> three reusable warm WorkerPool pods

How to explain the diagram

Kubernetes maintains the namespace, WorkerPool, warm worker pods, and atelet DaemonSet.
Agent Substrate maintains actor records, lifecycle versions, worker assignments, and snapshot metadata.
The router waits for the control plane to assign or restore an actor before forwarding its request.
Suspension returns worker capacity to the pool while preserving the actor snapshot.

Complete your evidence checklist:

Add the pinned Agent Substrate revision and Intel Mac compatibility notes.
Add your existing WorkerPool, worker pod, atelet DaemonSet, and router observations.
Add your actor lifecycle, worker assignment, and version observations.
Add the exact counter values from the passing persistence assertion.
Add the assigned-capacity counts from release and reacquisition.
Add the architecture diagram explaining the two control planes.

Before this final check, predict the assigned total. Predict the free total too.

Confirm the final WorkerPool capacity picture by running these commands:

kubectl get pods -n ate-demo-counter
kubectl ate get workers

What does this final check show?

The first command confirms that the Counter Demo still has the same three warm pods. The second command shows which actors currently occupy those workers.

You should see three Counter Demo worker pods. Two workers should be ASSIGNED to portfolio-counter and portfolio-counter-2 while the third remains FREE.

Final worker totals look different?

Review the actor rows to confirm that both portfolio actors are running.
Return to the capacity block if portfolio-counter-2 remained suspended.
Compare the current pod names with the baseline list to confirm that no worker pod was replaced.

Help me reconcile my final actor and worker totals.

Strong finish. Your evidence now connects snapshot continuity to WorkerPool multiplexing across a stable Kubernetes pod set.

Secret mission

Oversubscribe the WorkerPool

Fill all three Counter Demo workers so a fourth actor cannot obtain capacity. Release one specific worker pod. Then prove the waiting actor receives that exact pod.

Clean Up Your Resources

Choose whether to keep this local Kubernetes lab available, pause its active processes, or remove its resources. The lab creates no cloud charges.

Cost warning

Docker Desktop 4.95.0 is free for personal use, education, non-commercial open source, and qualifying small businesses.

Other commercial use may require a paid subscription.

Resources you used:

A local kind cluster containing the single-node Kubernetes development environment.
A project-created local container registry used by the development deployment.
Agent Substrate control-plane resources in ate-system plus local snapshot storage.
The Counter Demo WorkerPool named counter plus its three warm worker pods in ate-demo-counter.
Four Actor records named portfolio-counter, portfolio-counter-2, pool-fill-3, and reuse-target plus their locally stored state.

Keep everything running

No action needed. Choose this if you want to continue testing actor lifecycles or worker reuse.

Docker Desktop stays active on your Intel Mac.
The kind cluster retains its control plane, Counter Demo, actor records, and snapshots.
The router remains available at http://localhost:8000 while the port-forward continues running in Terminal A.
Your three Counter Demo workers remain assigned to the running actors.

Pause - I'll come back to this later

Pausing frees local compute while preserving the cluster for another session. Your actor records and snapshots remain inside the local lab.

Switch back to Terminal B.
Reuse the actor suspension command from Step 5 for portfolio-counter-2.
Reuse the same suspension command for pool-fill-3.
Reuse the same suspension command for reuse-target.
Confirm that every worker is free by running this command:

kubectl ate get workers

What should I see?

Good work. The three Counter Demo worker rows should now show FREE because every actor is suspended.

Still seeing an assigned worker?

Check the assigned actor in the worker table. Suspend that actor with the same command pattern you used in Step 5.

Help me identify which Agent Substrate actor still holds a worker assignment.

Switch back to Terminal A.
Press Control-C to stop the router port-forward.
Close Docker Desktop from its macOS application menu.
Restart Docker Desktop before returning to the lab.

Delete - I don't want to use this again

Deleting the lab removes the cluster resources and their stored state. The pinned Agent Substrate source checkout remains on your Mac.

Switch back to Terminal A.
Press Control-C to stop the router port-forward.
Switch back to Terminal B.
Change into the root of your pinned Agent Substrate repository.
Delete the local lab by running this command:

./hack/delete-kind-cluster.sh

What does this cleanup remove?

The script deletes the local kind cluster. This removes the Agent Substrate control plane, Counter Demo, actor records, snapshots, WorkerPool, and worker pods.

The script also removes the local registry container when Agent Substrate created it.

Your pinned repository and installed development tools remain on your Mac.

Confirm that the cleanup command finishes without an error.
Return to Docker Desktop.
Confirm that the project cluster containers and project registry container are gone.

Cleanup complete. The project cluster and registry are now removed from your Mac.

Cleanup did not finish?

Keep Docker Desktop running while the cleanup script removes its containers.
Return to the root of the pinned Agent Substrate repository.
Run the cleanup command again.

Help me diagnose the local cleanup failure.

Nice Work!

That is a serious infrastructure milestone. Your local Agent Substrate laboratory now proves state restoration plus warm-worker reuse with executable assertions.

What you learned:

Mapped the two control planes in an architecture diagram that matches Kubernetes WorkerPool pods to Agent Substrate Actor assignments.
Proved state continuity across suspension plus request-driven resume. Exact arithmetic confirmed that process memory plus DurableDir data each continued by one.
Demonstrated WorkerPool multiplexing through capacity release plus reacquisition while all three worker pods stayed unchanged.
Secret Mission: Oversubscribed the three-worker pool with four actors. The final equality assertion proved reuse-target received the exact pod released by portfolio-counter.

Ready to quiz yourself?