Resumable Agents with Agent Substrate

Introduction

30 Second Summary

An AI assistant might work for a few seconds, then sit idle until someone needs it again. Across hundreds of assistants, those quiet pauses can tie up the computing power needed for new work.

In this project, you will run Atlas, a simulated platform-operations assistant, on a local Agent Substrate v0.5.0 actor platform. You will suspend Atlas to release its worker before a later request restores its interaction count.

What You'll Build

You'll send Atlas a request before suspending it. A later request wakes Atlas with its remembered interaction count continuing from where it stopped.

By the end of this project, you'll have:

• A local actor platform that serves Atlas through a gVisor sandbox. Its response displays memory, durable-file, and root filesystem counters.
• Clear terminal evidence showing Atlas move between ACTOR_STATE_SUSPENDED and ACTOR_STATE_RUNNING. The WORKER IPS field shows when Atlas holds worker capacity.
• An on-demand resume demo where Atlas restores its interaction count at 2 after releasing its idle worker.
• Secret Mission: Tag Atlas's suspended memory. Create atlas-assistant-clone from that snapshot. Record exactly which counters the clone restores.

Are there any prerequisites?

The project uses local resources, so you do not need a cloud account. Plan for administrator access, internet for the first build, and a recommended allocation of 4 CPU and 8 GB RAM.

Before We Start

Your confirmed Linux environment provides the consistent starting point for an honest lifecycle test of Atlas, a simulated platform-operations assistant that does not use a language model. Its counters make each state layer visible: process memory represents remembered interactions across suspend and resume, while the file and root-filesystem counters represent other state layers.

Set Up and Build the Local Platform

Your Atlas demo depends on actor lifecycle behavior from an exact Agent Substrate release. A local Kubernetes cluster built with kind keeps that behavior reproducible without a cloud account.

This step validates the Linux toolchain. You will pin Agent Substrate v0.5.0 before building its local gVisor counter platform.

In this step, get ready to:

• Validate the Linux prerequisite toolchain.
• Pin the repository to the exact Agent Substrate release commit.
• Build the local platform and verify the counter template.

Check and prepare the Linux toolchain

Each prerequisite owns one layer of the local build. Checking them first separates missing software from version mismatches.

What does the toolchain provide?

• Git downloads the source and locks your checkout to one commit.
• Go builds the command-line tools managed by the repository.
• Docker supplies the container runtime for the local cluster.
• The kubectl command connects to the cluster.
• The curl command sends requests to Atlas later in the project.

This tutorial recommends 4 CPU and 8 GB RAM for the local build. Upstream does not publish that pair as a minimum.

• Confirm your Linux machine can make 4 CPU available to the local container runtime.
• Confirm your Linux machine can make 8 GB RAM available to the local container runtime.

Before you run the checks, predict which prerequisite is most likely to need attention.

• Check every prerequisite by running these commands:

git --version
go version
docker version
docker info
kubectl version --client
curl --version

How do I read these checks?

• The Git line confirms that source-control commands are available.
• The Go line must report 1.27.0 or newer.
• The Docker version output confirms that the client is installed.
• The Docker information output confirms that the container runtime is reachable.
• The kubectl line must report client version 1.37.0.
• The curl line confirms that the HTTP client is available.

If you install Go from its Linux archive, PATH configuration can be fiddly. The Go binary path and the Go tools path serve different purposes.

✔️ All prerequisite checks pass

Your existing toolchain meets the project requirements. That gives the platform build a clean starting point.

• Confirm the Go output reports 1.27.0 or newer.
• Confirm the kubectl output reports 1.37.0.
• Confirm the Docker information check returns daemon details.
• Confirm your shell startup configuration includes /usr/local/go/bin when Go came from the Linux archive.
• Confirm your shell startup configuration includes $(go env GOPATH)/bin for Go-installed tools.

ⓧ Go or kubectl is outdated

An older tool can fail against the pinned source even when its command exists. Installing system software may request administrator approval.

• Install Go 1.27.2 from the official Go downloads page using the package that matches your Linux architecture.
• Install kubectl 1.37.0 by following the official Linux installation guide.
• Add /usr/local/go/bin to your shell PATH when Go came from the Linux archive.
• Add $(go env GOPATH)/bin to your shell PATH for Go-installed tools.
• Reload your shell so the updated PATH entries take effect.

ⓧ One or more commands are missing

A missing command means its prerequisite needs a fresh installation. The official guides below provide the distribution-specific steps.

• Install Git by following the official Git installation guide for your Linux distribution.
• Install Go 1.27.2 from the official Go downloads page using the amd64 or arm64 package that matches your machine.
• Install Docker Engine by following its distribution-specific Linux guide.
• Install kubectl 1.37.0 by following the official Linux installation guide.
• Install curl from its official download page using the route for your Linux distribution.
• Add /usr/local/go/bin to your shell PATH when Go came from the Linux archive.
• Add $(go env GOPATH)/bin to your shell PATH for Go-installed tools.
• Reload your shell so the new commands become available.

• Recheck the completed toolchain by running the same commands:

git --version
go version
docker version
docker info
kubectl version --client
curl --version

What confirms the toolchain is ready?

Each command now returns information instead of stopping at a missing command. The Docker information check also reaches the running container runtime.

The Go output reports 1.27.0 or newer. The kubectl output reports client version 1.37.0.

🙋‍♀️ Still seeing a prerequisite failure?

• Restart your shell if a newly installed command is still missing.
• Review both Go path entries if Go works but Go-installed tools remain unavailable.
• Return to the Docker installation guide if the client works but the information check cannot reach the runtime.

Help me diagnose my prerequisite check output. Share the non-sensitive command output in the NextWork community if the environment needs a human check.

Clone and pin the source

Agent Substrate is pre-1.0. Pinning the checkout prevents later source changes from altering the lifecycle behavior you test.

• Move to your Desktop by running this command:

cd ~/Desktop

What does this command do?

The command moves your terminal into the Desktop directory under your Linux home folder. The repository will be easy to find after cloning.

• Create the pinned substrate checkout by running these commands:

git clone https://github.com/agent-substrate/substrate.git
cd substrate
git checkout e8d40c33f96f4b7a4984bfea7334a36ab5b45b7b

What do these commands do?

• The clone command copies the canonical repository into a local substrate directory.
• The directory command moves your terminal into that checkout.
• The checkout command selects the exact release commit used throughout this project.

Before you inspect the checkout, predict whether Git will report the full pinned commit hash.

• Reveal the checked-out commit by running this command:

git rev-parse HEAD

What should I see?

The terminal prints e8d40c33f96f4b7a4984bfea7334a36ab5b45b7b. This exact match anchors every script and expected result to Agent Substrate v0.5.0.

Your local source now matches the release used by this walkthrough. That removes version drift from every lifecycle result ahead.

Commit hash does not match?

• Confirm your terminal is inside the local substrate directory.
• Repeat the checkout command if Git remained on another branch or commit.
• Retry the clone after checking your internet connection if the repository download stopped early.

Help me fix my pinned Agent Substrate checkout.

Build and verify the local platform

The pinned repository manages kind v0.33.0 and ko v0.19.1 as Go tools. Its scripts select those versions for the build.

Why kind with gVisor?

The kind workflow keeps the control plane, router, workers, and snapshot store on your machine. This removes cloud accounts from the lifecycle test.

The gVisor counter path works without the KVM-specific environment required by the separate microVM demo. It keeps this project focused on actor suspension and restoration.

The first cluster build and image pulls can take about 15 minutes. A quiet terminal during that window can be normal.

• Create the local cluster and registry by running the repository script:

hack/create-kind-cluster.sh

What does the cluster script create?

• It creates a local cluster named kind.
• It creates the local kind-registry container used by the build workflow.
• It configures kubectl to reach the new cluster.

Before you query the cluster, predict whether kubectl can already reach its control plane.

• Check the cluster connection by running this command:

kubectl cluster-info

What should I see?

The output reports the cluster control plane. This confirms that kubectl is connected to the local kind cluster.

🙋‍♀️ Cluster control plane unavailable?

• Confirm the cluster creation script reached its completion message.
• Confirm the Docker information check still reaches the container runtime.
• Retry the cluster script after restoring the Docker runtime if image pulls were interrupted.

Help me diagnose my local kind cluster creation.

• Deploy the Agent Substrate control plane by running this command:

hack/install-ate-kind.sh --deploy-ate-system --credential-provider='{"name":"k8s.io"}'

What does the system deployment add?

The script deploys the Agent Substrate control-plane workloads to the local cluster. It also deploys PostgreSQL and the in-cluster rustfs snapshot store.

Successful deployment output confirms that the platform services were applied to the kind cluster.

• Deploy the default counter demo by running this command:

hack/install-ate-kind.sh --deploy-demo-counter

What does the counter deployment add?

The demo creates the counter ActorTemplate in ate-demo-counter. Its sandbox uses gvisor.

The template also receives a golden snapshot tag. That snapshot gives new actors a starting state.

🙋‍♀️ Platform deployment stopped early?

• Check the deployment output for an interrupted image download.
• Confirm the cluster connection still reports its control plane.
• Repeat only the deployment command that stopped after the cluster connection is healthy.

Help me diagnose my Agent Substrate deployment.

• Install kubectl-ate from the pinned source by running this command:

go install ./cmd/kubectl-ate

What does this installation use?

Go builds kubectl-ate from the repository commit you verified. The resulting command uses your current kubectl connection to query the local cluster.

• Verify the installed command by running this check:

kubectl ate --version

What should I see?

The command prints the kubectl-ate CLI version and exits. This confirms that the Go tools path is available to your shell.

🙋‍♀️ kubectl-ate unavailable after installation?

• Confirm your shell startup configuration contains $(go env GOPATH)/bin.
• Reload your shell after changing its PATH configuration.
• Repeat the Go installation command from inside the pinned substrate directory.

Help me find my installed kubectl-ate command.

Before you run the final checks, predict whether the source, cluster, and counter template will all confirm the same pinned platform state.

• Verify the complete local platform by running these commands:

git rev-parse HEAD
kubectl cluster-info
kubectl ate get actor-templates -a ate-demo-counter

What proves the platform is ready?

• The first output prints e8d40c33f96f4b7a4984bfea7334a36ab5b45b7b.
• The cluster output reports its control plane.
• The template table includes counter.
• The same template row includes gvisor and a non-empty golden snapshot tag.

🙋‍♀️ Final verification incomplete?

• Return to the source checkout if the commit hash differs.
• Return to the cluster creation check if the control plane is unavailable.
• Repeat the counter deployment if the template table has no counter row.

Help me diagnose my final Agent Substrate verification.

Your local actor platform is ready. Next, Atlas makes its first request and reveals whether a worker remains assigned after the interaction.

Wake Atlas and Send Its First Interaction

Your local Agent Substrate v0.5.0 platform is ready on the kind cluster. Now Atlas can move from a defined persona to a live actor record.

This step tests the platform after one interaction. Does Atlas release its worker as soon as the response finishes?

In this step, get ready to:

• Create atlas-assistant from the deployed counter template.
• Forward local request traffic in a second terminal.
• Capture lifecycle evidence from Atlas's first interaction.

Create Atlas and inspect its initial state

An actor record gives Atlas a stable identity in the ate-demo-counter namespace. It starts without a worker because no request has activated it.

• Create atlas-assistant by typing this command:

kubectl ate create actor atlas-assistant -a ate-demo-counter --template counter

What Does This Command Do?

• The actor name gives the simulated assistant its platform identity.
• The ate-demo-counter value selects the actor namespace.
• The counter template connects Atlas to the deployed counter workload.

• Inspect Atlas's initial state by running:

kubectl ate get actor atlas-assistant -a ate-demo-counter

What Does the Actor Table Show?

• The STATE column reports Atlas's lifecycle state.
• The WORKER IPS column reports its physical worker assignment.

Atlas starts in ACTOR_STATE_SUSPENDED. The WORKER IPS cell is blank because no worker is assigned.

Which States Belong to This Release?

Suspension transitions through ACTOR_STATE_SUSPENDING. A completed suspension settles at ACTOR_STATE_SUSPENDED.

The v0.5.0 state list has no ACTOR_STATE_PAUSING or ACTOR_STATE_PAUSED state.

🙋‍♀️ Actor Missing From the Table?

• Confirm that your current kubectl context still points to the local kind cluster.
• Retype ate-demo-counter as the actor namespace.
• Retype atlas-assistant as the actor name.

Help me find why Atlas is missing from the actor table.

Forward the router in a second terminal

The router receives requests before sending them to the target actor. A port-forward gives that router a local address while keeping the cluster service private.

• Create a second terminal session using your terminal application's new-tab or new-window control.
• Start forwarding local traffic by typing:

kubectl port-forward -n ate-system svc/atenet-router 8000:80

How Does the Port-Forward Work?

• The atenet-router service receives requests for actors.
• The port-forward maps localhost:8000 to service port 80 in ate-system.
• The active terminal session keeps this local route available.

The command continues running while the forwarding route stays active. This terminal now belongs to the router connection.

• Leave this second terminal running.

Port-Forward Stopped?

• Check that the second terminal session is still active.
• Stop the process already using local port 8000 before rerunning the forwarding command.
• Confirm that the Agent Substrate workloads are still running on the local cluster.

Help me diagnose why the router port-forward is not staying active.

Send Atlas its first interaction

An HTTP POST request gives Atlas its first unit of work. The response exposes each state layer through a separate counter.

• Switch back to the repository terminal from earlier.

Before you send the request, do you think Atlas will keep a worker after producing its response?

• Send one interaction by typing this request yourself:

curl -X POST \
  -H "ate-target-actor: ate-demo-counter/atlas-assistant" \
  http://localhost:8000

What Does the First Response Prove?

• The ate-target-actor: ate-demo-counter/atlas-assistant header routes the request to Atlas.
• The response starts with hello from: plus the assigned worker IP.
• The preserved memory count: 1 value represents Atlas remembering its first interaction.
• The preserved file counter: 1 value exposes the durable-file state layer.
• The preserved rootfs counter: 1 value exposes the root-filesystem state layer.

The three values at 1 establish Atlas's baseline before any suspension occurs.

No Response From Atlas?

• Check that the port-forward is still active in the second terminal.
• Check that the request targets ate-demo-counter/atlas-assistant exactly.
• Check that the request uses local port 8000.

Help me diagnose why Atlas did not return its first counter response.

Before you inspect Atlas, do you expect the WORKER IPS cell to be blank or populated?

• Inspect Atlas after the completed request by running:

kubectl ate get actor atlas-assistant -a ate-demo-counter

What Changed After the Request?

• The STATE column now reports ACTOR_STATE_RUNNING because the request activated Atlas.
• The WORKER IPS cell now contains the assigned worker IP.

The interaction is complete. Atlas still holds a worker assignment.

You now have a clean lifecycle baseline. Atlas remembers one interaction while the platform still treats it as running.

Atlas Still Looks Suspended?

• Confirm that the request returned Atlas's three counter values.
• Rerun the actor inspection command after the request finishes.
• Confirm that you are inspecting atlas-assistant in ate-demo-counter.

Help me understand why Atlas still appears suspended after its first routed request.

Your baseline is locked in. Next, you'll suspend Atlas to reclaim the worker. Then you'll test whether its remembered interaction survives.

Suspend Atlas and Restore Its Memory

Atlas is running after its first interaction. The completed request still leaves a worker assigned to it.

Agent Substrate solves this idle-capacity problem through suspension. The platform captures Atlas in a durable snapshot before reclaiming its worker.

A later routed request restores that snapshot on demand. The final counter values reveal whether restoration succeeded.

In this step, get ready to:

• Inspect the counter template's snapshot contract.
• Suspend Atlas to reclaim its assigned worker.
• Send another interaction to prove Atlas resumes with its remembered state.

Confirm Atlas's state contract

The counter template's YAML exposes which state layers suspension captures. Its gVisor sandbox requests memory fidelity for each snapshot.

• Switch back to the substrate repository terminal from earlier.
• Inspect the counter template by running this command:

kubectl ate get actor-template counter -a ate-demo-counter -o yaml

What does this command show?

• The command reads the deployed counter ActorTemplate from ate-demo-counter.
• The preferredFidelity: SNAPSHOT_FIDELITY_MEMORY field selects memory fidelity.
• Memory fidelity includes durable volumes.
• Memory fidelity includes root-filesystem changes.
• Memory fidelity captures process memory for Atlas's remembered interaction count.
• The sandboxClass: gvisor field selects the sandbox class.
• The configName: gvisor-default field selects the matching sandbox configuration.

• Locate preferredFidelity: SNAPSHOT_FIDELITY_MEMORY in the output.
• Locate sandboxClass: gvisor in the output.
• Locate configName: gvisor-default in the output.

Can't find the fidelity fields?

• Check that the command queries the counter template in ate-demo-counter.
• Confirm that -o yaml appears at the end of the command.

Help me find the counter template fields in this output.

Suspend Atlas and inspect the released worker

A running actor occupies a physical worker from the WorkerPool. Suspension makes that capacity available while Atlas's state remains durable.

Before you suspend Atlas, predict whether WORKER IPS will stay populated after the command completes.

• Suspend Atlas by running this command:

kubectl ate suspend actor atlas-assistant -a ate-demo-counter

What does suspension do?

The command asks Agent Substrate to capture Atlas's state in a durable snapshot. The selected fidelity preserves its process memory.

After the snapshot completes, the platform releases Atlas's physical worker. The actor record remains available for a future request.

• Inspect Atlas's state and the available workers by running these commands:

kubectl ate get actor atlas-assistant -a ate-demo-counter
kubectl ate get workers

What do these checks prove?

• The actor query should show ACTOR_STATE_SUSPENDED.
• The WORKER IPS field should now be blank.
• The workers view lets you inspect the pool after Atlas releases its assignment.

That is the reclamation win: Atlas keeps its actor record while its idle worker returns to the pool.

Worker assignment still populated?

• Wait for the suspension command to finish before querying Atlas again.
• Confirm that the actor row reads ACTOR_STATE_SUSPENDED.
• Query atlas-assistant in ate-demo-counter again after the state transition completes.

Help me understand why Atlas still has a worker assignment.

Trigger Atlas's on-demand resume

A request to a suspended actor triggers on-demand activation. The router restores Atlas onto available worker capacity before serving the interaction.

Before you run the final check, predict whether Atlas's process-memory count will continue or reset.

• Verify Atlas's on-demand resume by running these commands:

curl -X POST \
  -H "ate-target-actor: ate-demo-counter/atlas-assistant" \
  http://localhost:8000

kubectl ate get actor atlas-assistant -a ate-demo-counter

What does this verification do?

• The POST request sends Atlas its second routed interaction.
• The ate-target-actor header selects ate-demo-counter/atlas-assistant.
• The actor query reveals Atlas's state after the request completes.

You'll see preserved memory count: 2 in the response. Continuing at 2 proves that Atlas restored its captured process state.

The response also reports preserved file counter: 2. It reports preserved rootfs counter: 2.

The actor row shows ACTOR_STATE_RUNNING. Its WORKER IPS field is populated again.

The worker IP can match the earlier value or change. Counter continuity is the evidence that Atlas's state survived.

Atlas doesn't resume as expected?

• Confirm that the port-forward terminal from earlier is still running.
• Restart the existing port-forward with the same command from the previous step if the router is unreachable.
• Check that the request header targets ate-demo-counter/atlas-assistant if another actor responds.

Help me diagnose Atlas's resume result.

You have closed the lifecycle loop. Atlas now releases idle compute while preserving its visible memory.

Secret mission

Secret Mission: Branch Atlas's Memory into a Clone

Give Atlas's suspended memory a stable snapshot tag. Create a second assistant actor from that checkpoint. Its first interaction reveals which state values the clone actually restores.

Clean Up Your Resources

Clean Up Your Resources

Your Agent Substrate platform runs locally on a kind cluster. Keeping it active creates no cloud charges. It continues to use local memory plus disk space.

Resources you used:

• The local kind cluster.
• The local registry container named kind-registry.
• The Agent Substrate control-plane workloads.
• The in-cluster PostgreSQL instance.
• The in-cluster rustfs snapshot store.
• The counter demo in ate-demo-counter.
• The actor named atlas-assistant.
• The clone actor named atlas-assistant-clone.
• The snapshot tag named atlas-checkpoint.
• The router port-forward from localhost:8000 to service/atenet-router.

Keep everything running

No action needed. Choose this if you want to continue testing Atlas or compare more lifecycle behavior.

• Leave the kind cluster running.
• Keep the router port-forward active in the second terminal.
• Retain atlas-checkpoint while atlas-assistant-clone borrows its snapshot.

The local platform keeps using memory plus disk space. It does not create cloud charges.

Pause - I'll come back to this later

Shut down running processes to free local memory. Keep the cluster state for your return.

• Switch back to the second terminal that is running the router port-forward.
• Press Ctrl+C to stop the port-forward.
• Return to the repository terminal from earlier.
• Suspend the running clone by running this command:

kubectl ate suspend actor atlas-assistant-clone -a ate-demo-counter

What Does This Command Do?

Suspension captures the clone's current state in a durable snapshot. It releases the clone's worker assignment.

The clone keeps its actor record. Its first suspension also gives it ownership of its own snapshot.

• Check the clone's suspended state by running this command:

kubectl ate get actor atlas-assistant-clone -a ate-demo-counter

What Should You See?

You should see ACTOR_STATE_SUSPENDED in the state column. The WORKER IPS field should be blank.

That blank assignment confirms the clone no longer occupies a worker. Both Atlas actors are now suspended.

🙋‍♀️ Clone Still Running?

• Wait for snapshot capture to finish if the clone still shows ACTOR_STATE_SUSPENDING.
• Run the inspection command again after the transition finishes.
• Help me diagnose why atlas-assistant-clone is not reaching ACTOR_STATE_SUSPENDED.

Delete - I don't want to use this again

Remove the local platform resources to start fresh. Your substrate source folder remains on disk.

Deletion Is Permanent

Deleting the cluster is a clean reset. The actors plus their snapshots disappear with the cluster.

Your recorded terminal evidence remains wherever you saved it. The repository source also stays available locally.

• Switch back to the second terminal if the router port-forward is still running.
• Press Ctrl+C to stop the port-forward.
• Return to the repository terminal from earlier.
• Delete the local cluster plus registry by running this command:

hack/delete-kind-cluster.sh

What Does This Command Remove?

The script deletes the default local cluster named kind. This removes every workload plus actor resource stored inside it.

The script also removes kind-registry when Agent Substrate created that registry.

• Wait for the command to return to the terminal prompt.
• Read the final output to confirm the local cluster cleanup completed.

Your local Agent Substrate platform is now stopped. Its cluster resources no longer consume memory.

Cleanup Script Not Running?

• Confirm the repository terminal is still inside the substrate directory.
• Confirm Docker is running if the script cannot access the local container runtime.
• Help me diagnose why the Agent Substrate kind cleanup script is failing.

Nice Work!

Nice Work!

You made it! You built Agent Substrate v0.5.0 on a local kind cluster.

Your Atlas demonstration proved that a simulated assistant can release idle compute while preserving visible memory across suspension.

You've learned how to:

• Build a reproducible local platform from the exact release commit e8d40c33f96f4b7a4984bfea7334a36ab5b45b7b with the default gVisor counter demo.
• Trace Atlas through the actor lifecycle by comparing ACTOR_STATE_RUNNING with ACTOR_STATE_SUSPENDED through the WORKER IPS field.
• Prove worker reclamation through suspension while durable snapshots preserve Atlas's process memory with SNAPSHOT_FIDELITY_MEMORY.
• Secret Mission: Branched Atlas's suspended memory into atlas-assistant-clone through atlas-checkpoint. You recorded the clone's actual memory, file, root-filesystem, and worker-IP values without assuming the result.

Ready to quiz yourself?