Build a Crash-Proof Job Runner

Build a Go job runner that survives Worker crashes with Temporal.

Introduction

30 Second Summary

A batch of jobs can look healthy until the program suddenly closes. When it starts again, finished work may run twice while unfinished work disappears.

In this project, you will build a three-job runner in Go that uses goroutines to expose repeated work after a crash. You will rebuild it with Temporal so completed jobs stay recorded while a replacement Worker finishes the rest.

What You'll Build

Your finished Workflow survives an intentional Worker crash before a replacement Worker delivers all three durable results.

By the end of this project, you'll have:

  • A visible failure demonstration where quick-index completes again after every restart because the original process forgot its progress.
  • A durable concurrent runner that schedules three Activities through one Task Queue. It returns every result after you restart only the Worker.
  • Clear recovery evidence in Temporal Web UI through retry attempts plus Event History. You will also have a local Workflow test that reports PASS without a live Temporal Service.
  • Secret Mission: Add Activity Heartbeats so an interrupted job resumes from recorded progress instead of repeating all its internal work.

Are there any prerequisites?

You need a supported Windows computer with administrator access plus internet access. Basic familiarity with variables and functions is enough.

Before We Start

This is your moment to lock in what you are building before any setup begins. You will compare a process-local goroutine runner with a Temporal runner that preserves completed Workflow and Activity progress after a Worker crash.

Set Up Windows for Temporal Go Development

Your crash-proof runner needs a Linux environment that matches the setup used by Temporal Server contributors on Windows. WSL 2 provides that environment through Ubuntu without a separate virtual machine.

You will install Go plus the Temporal CLI inside Ubuntu. You will connect Visual Studio Code to the same Linux environment so every later command uses one consistent toolchain.

In this step, get ready to:
  • Install WSL 2 with an Ubuntu distribution.
  • Install the pinned Linux development tools.
  • Open an empty WSL workspace through Visual Studio Code.
Install WSL 2 and Ubuntu

WSL 2 runs a Linux environment directly beside Windows. Ubuntu gives your Go tools a Linux home while Visual Studio Code keeps its familiar Windows interface.

Administrator access only enables the Windows features required by WSL. Your project stays local to your computer.

  • Press the Windows key to open Windows search.
  • Type PowerShell.
  • Start PowerShell with administrator permissions.
  • Check the installed Ubuntu distribution by running this command:
wsl --list --verbose

What does this command show?

  • The command lists each Linux distribution installed through WSL.
  • The version column shows whether Ubuntu uses WSL 1 or WSL 2.

✔️ Ubuntu uses WSL 2

Your output lists Ubuntu with version 2. WSL is ready for the Linux tools in this project.

ⓧ Ubuntu uses WSL 1

Ubuntu already exists. Its version needs upgrading before you install the Linux toolchain.

  • Convert the Ubuntu distribution to WSL 2 by running this command:
wsl --set-version Ubuntu 2

What does this command do?

This command converts the existing Ubuntu distribution to WSL 2. Your Linux files remain inside the distribution.

The conversion can take several minutes. A progress percentage confirms Windows is still working.

  • Confirm the conversion by running the distribution check again:
wsl --list --verbose

What should you see?

Ubuntu should now show version 2 in the final column.

Ubuntu still uses WSL 1?

Restart Windows after the conversion finishes. Run the distribution check again after signing back in.

Use help me convert Ubuntu from WSL 1 to WSL 2 if the version remains unchanged.

ⓧ WSL is not installed

This installation downloads Ubuntu and enables the required Windows features. It can take several minutes before PowerShell returns control.

  • Install WSL with its default Ubuntu distribution by running this command:
wsl --install

What does this command do?

The command enables WSL. It installs Ubuntu as the default Linux distribution.

New Linux installations use WSL 2 by default.

  • Restart Windows when the installation asks you to reboot.
  • Press the Windows key after signing back in.
  • Type Ubuntu.
  • Press Enter to launch Ubuntu.
  • Enter a Linux username when Ubuntu prompts you.
  • Enter a Linux password when Ubuntu prompts you.
  • Confirm the password when Ubuntu asks again.

Your password stays invisible while you type. Ubuntu still receives every keystroke.

WSL installation stopped?

Confirm that Windows meets the supported WSL requirements. Windows 10 needs version 2004 with build 19041 or later.

Use help me diagnose my WSL installation for help with the result you see.

That is the Windows layer ready. Ubuntu now gives the rest of this project a Linux-aligned development environment.

Install the Linux toolchain

Go compiles the job runners you build later. The Temporal CLI provides the local service used to preserve Workflow state outside the Worker process.

Architecture names can be fiddly. Intel and AMD computers use the amd64 archives. ARM computers use the arm64 archives.

  • Switch back to the Ubuntu terminal from the WSL setup.
  • Check whether Go is already available by running this command:
go version

What does this command check?

The command prints the installed Go release plus the Linux architecture. The required release for this project is 1.27.3.

✔️ I see the required Go version

Your output contains go1.27.3. Go is ready for the project.

ⓧ I see an older Go version

The older Go tree needs replacing with release 1.27.3.

  • Use go1.27.3.linux-amd64.tar.gz for an Intel or AMD computer.
  • Change the archive value to go1.27.3.linux-arm64.tar.gz for an ARM computer.
  • Set your selected archive filename here: go1.27.3.linux-amd64.tar.gz.
  • Replace the old Go tree with the selected release by running these commands:
curl -LO https://go.dev/dl/[[GO_ARCHIVE="go1.27.3.linux-amd64.tar.gz"]]
sudo rm -rf /usr/local/go
sudo tar -C /usr/local -xzf [[GO_ARCHIVE="go1.27.3.linux-amd64.tar.gz"]]
echo 'export PATH=$PATH:/usr/local/go/bin' >> $HOME/.profile
source $HOME/.profile

What do these commands do?

  • The first command downloads your selected official Go archive.
  • The next two commands replace the existing Go tree under /usr/local/go.
  • The final two commands add Go to your Ubuntu PATH.
  • Verify the replacement by running the version check again:
go version

What should you see?

The output should contain go1.27.3 followed by your Linux architecture.

Go still shows the older version?

Close the current Ubuntu terminal after the installation. Launch Ubuntu again so the updated profile loads in a fresh shell.

Use help me fix my Go PATH in Ubuntu if the older release still appears.

ⓧ Go is not found

Ubuntu needs the official Go archive before it can run Go commands.

  • Use go1.27.3.linux-amd64.tar.gz for an Intel or AMD computer.
  • Change the archive value to go1.27.3.linux-arm64.tar.gz for an ARM computer.
  • Set your selected archive filename here: go1.27.3.linux-amd64.tar.gz.
  • Install the selected Go release by running these commands:
curl -LO https://go.dev/dl/[[GO_ARCHIVE="go1.27.3.linux-amd64.tar.gz"]]
sudo rm -rf /usr/local/go
sudo tar -C /usr/local -xzf [[GO_ARCHIVE="go1.27.3.linux-amd64.tar.gz"]]
echo 'export PATH=$PATH:/usr/local/go/bin' >> $HOME/.profile
source $HOME/.profile

What do these commands do?

  • The first command downloads your selected official Go archive.
  • The next two commands install Go under /usr/local/go.
  • The final two commands make the Go executable available from Ubuntu terminals.
  • Verify the installation by running this command:
go version

What should you see?

The output should contain go1.27.3 followed by your Linux architecture.

Go command still missing?

Close the current Ubuntu terminal after the installation. Launch Ubuntu again so the profile adds Go to the new shell.

Use help me finish my Go installation in Ubuntu if the command remains unavailable.

Go is now available inside Ubuntu. The next version check covers the command that starts Temporal's local development service in a later step.

  • Check whether Temporal CLI is already available by running this command:
temporal --version

What does this command check?

The command prints the installed Temporal CLI release. This project uses release 1.9.1.

✔️ I see the required CLI version

Your output contains 1.9.1. Temporal CLI is ready.

ⓧ I see an older CLI version

The installed binary needs replacing with Temporal CLI release 1.9.1.

  • Use temporal_cli_1.9.1_linux_amd64.tar.gz for an Intel or AMD computer.
  • Change the archive value to temporal_cli_1.9.1_linux_arm64.tar.gz for an ARM computer.
  • Set your selected archive filename here: temporal_cli_1.9.1_linux_amd64.tar.gz.
  • Replace the CLI with the selected release by running these commands:
curl -LO https://github.com/temporalio/cli/releases/download/v1.9.1/[[TEMPORAL_ARCHIVE="temporal_cli_1.9.1_linux_amd64.tar.gz"]]
tar -xzf [[TEMPORAL_ARCHIVE="temporal_cli_1.9.1_linux_amd64.tar.gz"]]
sudo mv temporal /usr/local/bin

What do these commands do?

  • The first command downloads your selected release archive from the official Temporal CLI release.
  • The second command extracts the temporal binary.
  • The final command places that binary on the Ubuntu PATH.
  • Verify the replacement by running the version check again:
temporal --version

What should you see?

The output should contain Temporal CLI release 1.9.1.

Temporal still shows the older version?

A second Temporal binary may appear earlier on your PATH. Confirm that the new binary was moved into /usr/local/bin.

Use help me find the older Temporal CLI binary if the version remains unchanged.

ⓧ Temporal CLI is not found

Ubuntu needs the official Temporal CLI archive before the command becomes available.

  • Use temporal_cli_1.9.1_linux_amd64.tar.gz for an Intel or AMD computer.
  • Change the archive value to temporal_cli_1.9.1_linux_arm64.tar.gz for an ARM computer.
  • Set your selected archive filename here: temporal_cli_1.9.1_linux_amd64.tar.gz.
  • Install the selected Temporal CLI release by running these commands:
curl -LO https://github.com/temporalio/cli/releases/download/v1.9.1/[[TEMPORAL_ARCHIVE="temporal_cli_1.9.1_linux_amd64.tar.gz"]]
tar -xzf [[TEMPORAL_ARCHIVE="temporal_cli_1.9.1_linux_amd64.tar.gz"]]
sudo mv temporal /usr/local/bin

What do these commands do?

  • The first command downloads your selected release archive from the official Temporal CLI release.
  • The second command extracts the temporal binary.
  • The final command places that binary on the Ubuntu PATH.
  • Verify the installation by running this command:
temporal --version

What should you see?

The output should contain Temporal CLI release 1.9.1.

Temporal command still missing?

Confirm that the extracted file was named temporal. Confirm that the move command completed without a permission error.

Use help me finish my Temporal CLI installation if the command remains unavailable.

The Linux toolchain is complete. Go can compile your runner while Temporal CLI can host its durable execution state later.

Connect VS Code to the WSL workspace

The WSL extension keeps the editor interface on Windows while terminals and extensions run inside Ubuntu. This prevents Windows and Linux tool installations from getting mixed together.

  • Press the Windows key to open Windows search.
  • Type Visual Studio Code.
  • Press Enter to start Visual Studio Code.
  • Click Extensions in the left Activity Bar.
  • Enter WSL in the extension search field.
  • Select the WSL extension published by Microsoft.
  • Click Install.

The extension page now shows that WSL is installed. Visual Studio Code can connect its editor to the Ubuntu filesystem.

  • Return to the Ubuntu terminal from earlier.
  • Prepare a Linux Desktop location by running these commands:
mkdir -p ~/Desktop
ls ~

What do these commands do?

The first command ensures your Ubuntu home directory has a Desktop folder. The second command confirms that folder appears in your Linux home directory.

  • Create the empty workspace from your Linux Desktop by running these commands:
cd ~/Desktop
mkdir crashproof-job-runner
cd crashproof-job-runner
pwd

What do these commands do?

These commands create the crashproof-job-runner folder inside Ubuntu. The final command prints its Linux path so you can confirm the terminal is inside the workspace.

Before the final checks, predict what the two version commands should report. Also predict which environment Visual Studio Code should name after it opens.

  • Verify both installed tools from inside the workspace by running these commands:
go version
temporal --version

What should you see?

The Go output should contain go1.27.3. The Temporal output should contain 1.9.1.

  • Open the current WSL workspace in Visual Studio Code by running this command:
code .

What does this command do?

The command opens the current crashproof-job-runner folder in Visual Studio Code. The WSL extension connects the window to Ubuntu.

The first WSL connection can take a moment while Visual Studio Code installs its Linux components. The lower-left connection indicator should name WSL or Ubuntu when the window is ready.

VS Code opened without WSL?

Confirm that you ran the command from the Ubuntu terminal inside crashproof-job-runner. Confirm that the Microsoft WSL extension is installed in Visual Studio Code.

Use help me connect Visual Studio Code to my WSL workspace if the window still opens in Windows mode.

Your Windows machine now has a verified Linux toolchain plus an empty WSL workspace. Next, you will use it to run the goroutine version and watch process-local progress disappear during a crash.

Feel the Failure in a Goroutine Job Runner

Your WSL workspace is open in Visual Studio Code. It gives your Go project a Linux development environment.

Your first runner uses goroutines to start three jobs concurrently. Their progress lives in process-local state.

The call to os.Exit(1) terminates the process immediately. This step lets you watch completed work repeat after a restart.

In this step, get ready to:
  • Create naive/main.go with a concurrent three-job runner.
  • Run the jobs to observe the process terminate during payment.
  • Restart the runner to prove that completed work repeats.
Build the naive runner

A Job value describes one unit of work. Its fields store the job name, duration, plus the intentional crash setting.

The payment job carries the teaching fault. This deliberate process exit belongs only in this local demonstration.

  • Click the New Folder icon beside crashproof-job-runner in the Explorer sidebar.
  • Type naive as the folder name. Press Enter.

You should see the new naive folder beneath crashproof-job-runner in the Explorer sidebar.

  • Click the New File icon beside the naive folder.
  • Type main.go as the file name. Press Enter.

You should see main.go nested inside the naive folder.

  • Add the runner's data model to naive/main.go by copying this code:
package main

import (
	"fmt"
	"os"
	"sync"
	"time"
)

type Job struct {
	Name                string
	DurationSeconds     int
	CrashOnFirstAttempt bool
}

What does this code set up?

  • The fmt package prints visible status lines.
  • The os package provides the intentional process exit.
  • The sync package provides WaitGroup for coordinating the goroutines.
  • The time package controls each job's duration.
  • The Job fields hold the values that control each run.
  • Save naive/main.go.
  • Confirm main.go remains visible beneath naive in the Explorer sidebar.

Is main.go in the wrong place?

  • Expand the naive folder in the Explorer sidebar.
  • Check that the file is named exactly main.go.
  • Drag main.go into naive if it appears beside the folder.

Help me place main.go inside the naive folder.

The main() function creates three jobs. Its loop starts one goroutine for each job.

  • Add main() below the Job type by copying this code:
func main() {
	jobs := []Job{
		{Name: "quick-index", DurationSeconds: 1},
		{Name: "payment", DurationSeconds: 5, CrashOnFirstAttempt: true},
		{Name: "receipt", DurationSeconds: 6},
	}

	var wg sync.WaitGroup
	for _, job := range jobs {
		job := job
		wg.Add(1)
		go func() {
			defer wg.Done()
			fmt.Printf("job=%s status=started\n", job.Name)

			if job.CrashOnFirstAttempt {
				time.Sleep(3 * time.Second)
				fmt.Printf("job=%s status=crashing-process\n", job.Name)
				os.Exit(1)
			}

			time.Sleep(time.Duration(job.DurationSeconds) * time.Second)
			fmt.Printf("job=%s status=completed\n", job.Name)
		}()
	}

	wg.Wait()
	fmt.Println("all jobs completed")
}

What does this code do?

  • The jobs slice defines quick-index, payment, plus receipt.
  • The loop starts one goroutine for each job.
  • The sync.WaitGroup keeps main() active during normal execution.
  • The payment job waits for three seconds before calling os.Exit(1).
  • The fmt.Printf calls expose each job's progress in the terminal.
  • Save naive/main.go.
  • Confirm the unsaved-change dot disappears from the main.go editor tab.

Seeing syntax warnings?

  • Check that func main() appears after the closing brace of Job.
  • Check that every import remains inside the parenthesized import block.
  • Compare every occurrence of CrashOnFirstAttempt with the field name in Job.

Help me fix the Go syntax in naive/main.go.

✔️ Awesome, I've got everything!

Your saved naive/main.go now contains the complete goroutine runner.

ⓧ I'd like to double check the full code

  • Compare naive/main.go with this complete file:
package main

import (
	"fmt"
	"os"
	"sync"
	"time"
)

type Job struct {
	Name                string
	DurationSeconds     int
	CrashOnFirstAttempt bool
}

func main() {
	jobs := []Job{
		{Name: "quick-index", DurationSeconds: 1},
		{Name: "payment", DurationSeconds: 5, CrashOnFirstAttempt: true},
		{Name: "receipt", DurationSeconds: 6},
	}

	var wg sync.WaitGroup
	for _, job := range jobs {
		job := job
		wg.Add(1)
		go func() {
			defer wg.Done()
			fmt.Printf("job=%s status=started\n", job.Name)

			if job.CrashOnFirstAttempt {
				time.Sleep(3 * time.Second)
				fmt.Printf("job=%s status=crashing-process\n", job.Name)
				os.Exit(1)
			}

			time.Sleep(time.Duration(job.DurationSeconds) * time.Second)
			fmt.Printf("job=%s status=completed\n", job.Name)
		}()
	}

	wg.Wait()
	fmt.Println("all jobs completed")
}

How is the file organized?

The file defines the job data first. The main() function starts the concurrent work.

The final wg.Wait() call handles normal completion. The intentional exit prevents the runner from reaching it.

Run the program into a crash

Each goroutine can sleep independently while the other jobs continue. The process still owns all three goroutines.

  • Right-click the naive folder in the Explorer sidebar.
  • Select Open in Integrated Terminal.

The terminal prompt should show that you are inside the naive folder.

Before you run the program, do you think receipt can finish before payment terminates the process?

  • Start the three-job runner by running this command:
go run .

What does this run test?

The command compiles the current main package. It starts all three jobs inside one process.

This run tests whether their progress survives that process ending.

You will see quick-index reach status=completed after about one second. The job start lines may appear in a different order.

Around three seconds into the run, payment reports status=crashing-process. The whole program exits with a status of 1 before receipt completes.

That abrupt stop is intentional. It proves that one goroutine cannot protect the others from a process exit.

Didn't see the intentional crash?

  • Confirm the terminal path ends with the naive folder.
  • Check that payment has CrashOnFirstAttempt: true.
  • Check that os.Exit(1) remains inside the if job.CrashOnFirstAttempt block.

Help me diagnose the naive runner output.

Restart the runner to compare

A restart creates a process with fresh memory. It loads no record that quick-index already completed.

Before you run the program again, do you think quick-index will be skipped or repeated?

  • Restart the same runner from the naive terminal by running:
go run .

What does the second run prove?

The command creates another process with a fresh jobs slice. The new process schedules all three jobs from the beginning.

No durable execution history tells it which job finished during the earlier run.

You will see quick-index start again. You will see it reach status=completed again before the same intentional exit.

  • Scroll upward in the terminal to locate the first run's quick-index completion.
  • Compare it with the second run's quick-index completion.

That is the failure captured: your runner repeats successful work after a crash. The duplicate quick-index completion proves that process memory did not preserve progress.

Can't compare both runs?

  • Increase the terminal panel height to reveal earlier output.
  • Scroll upward to locate both quick-index completion lines.
  • Run the program once more if the first output was cleared.

Help me compare the two naive runner attempts.

You have exposed the reliability gap in a process-only runner. Next, you will use Temporal to preserve completed work outside the Worker process.

Replace Process Memory with Durable Execution

Your goroutine runner proved that completed work disappears when its process exits. A restart repeats quick-index because the only record lived in memory.

Temporal Durable Execution stores Workflow state outside a disposable Worker process. This step moves the same jobs into concurrent Activities whose completed results remain recorded.

In this step, get ready to:
  • Define the Temporal Activity plus the Workflow that schedules three jobs concurrently.
  • Connect a Worker plus a starter to one Task Queue.
  • Test the Workflow before running it against a local Temporal development service.
Define the durable job runner

The go.mod file gives your project a module path. It also pins the Temporal Go SDK to version v1.50.0.

  • Switch back to Visual Studio Code in WSL.
  • Create go.mod inside the crashproof-job-runner folder.
  • Add the module configuration by pasting this code:
module example/crashproofjobrunner

go 1.26.0

require go.temporal.io/sdk v1.50.0

What does this configuration do?

  • module example/crashproofjobrunner gives the local project an import path that the Worker plus starter can share.
  • go 1.26.0 records the language version declared by the pinned SDK module.
  • go.temporal.io/sdk v1.50.0 fixes the project to the SDK release used throughout this project.
  • Save go.mod.

You should see go.mod listed beside the existing naive folder.

Does the module file look different?

Check that the file is named exactly go.mod. Confirm that it sits directly inside crashproof-job-runner.

Compare the blank lines carefully because the Go tool reads this file as module metadata.

help me check my Go module file

✔️ Awesome, I've got everything!

Great. Your project now has a module path plus a pinned Temporal Go SDK dependency.

ⓧ I'd like to double check the full code

Compare your complete go.mod file with this reference:

module example/crashproofjobrunner

go 1.26.0

require go.temporal.io/sdk v1.50.0

A Task Queue connects Workflow tasks plus Activity tasks to a Worker. Your Workflow also uses Futures so all three Activities can begin before their results are collected.

  • Create jobrunner.go inside the crashproof-job-runner folder.
  • Add the package definition plus shared job types by pasting this code:
package jobrunner

import (
	"context"
	"fmt"
	"os"
	"time"

	"go.temporal.io/sdk/activity"
	"go.temporal.io/sdk/workflow"
)

const TaskQueue = "crash-proof-jobs"

type Job struct {
	Name                string
	DurationSeconds     int
	CrashOnFirstAttempt bool
}

What does this code define?

  • TaskQueue gives the starter plus Worker the same routing name.
  • Job carries the name plus duration for one unit of work.
  • CrashOnFirstAttempt controls the intentional Worker-process exit used in the next step.
  • Save jobrunner.go.

You should see jobrunner.go appear beside go.mod in the workspace.

  • Add ProcessJob below the Job type by pasting this code:
func ProcessJob(ctx context.Context, job Job) (string, error) {
	info := activity.GetInfo(ctx)
	fmt.Printf("job=%s attempt=%d status=started\n", job.Name, info.Attempt)

	if job.CrashOnFirstAttempt && info.Attempt == 1 {
		time.Sleep(3 * time.Second)
		fmt.Printf("job=%s attempt=%d status=crashing-worker\n", job.Name, info.Attempt)
		os.Exit(1)
	}

	time.Sleep(time.Duration(job.DurationSeconds) * time.Second)
	result := fmt.Sprintf("%s completed on attempt %d", job.Name, info.Attempt)
	fmt.Printf("job=%s attempt=%d status=completed\n", job.Name, info.Attempt)
	return result, nil
}

How does the Activity work?

  • activity.GetInfo(ctx) reads metadata for the current Activity execution.
  • info.Attempt starts at one. It increases when Temporal retries an unfinished Activity.
  • os.Exit(1) provides the deliberate fault that terminates the Worker process.
  • result records the job name plus the attempt that completed it.

Keep fault injection in the lab

The call to os.Exit(1) is a teaching fault. Production Activity code should return an error plus leave process shutdown to the runtime.

  • Save jobrunner.go.

The file now contains the complete Activity that reports its execution attempt.

  • Add JobRunnerWorkflow below ProcessJob by pasting this code:
func JobRunnerWorkflow(ctx workflow.Context, jobs []Job) ([]string, error) {
	ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
		StartToCloseTimeout: 8 * time.Second,
	})

	futures := make([]workflow.Future, len(jobs))
	for index, job := range jobs {
		futures[index] = workflow.ExecuteActivity(ctx, ProcessJob, job)
	}

	results := make([]string, len(jobs))
	for index, future := range futures {
		if err := future.Get(ctx, &results[index]); err != nil {
			return nil, err
		}
	}

	return results, nil
}

How does the Workflow preserve progress?

  • StartToCloseTimeout limits one Activity attempt to eight seconds.
  • workflow.ExecuteActivity schedules each job without immediately waiting for its result.
  • futures keeps a handle for each scheduled Activity.
  • future.Get waits for each durable result after every Activity has been scheduled.
  • Save jobrunner.go.

Your package now defines the Activity plus the Workflow that coordinates it.

Seeing code errors in jobrunner.go?

Check that ProcessJob appears after the closing brace of Job. Confirm that JobRunnerWorkflow appears after the closing brace of ProcessJob.

The dependency is downloaded later in this step. Import warnings can remain until the module command completes.

help me compare jobrunner.go with the target code

✔️ Awesome, I've got everything!

Good work. Save jobrunner.go before adding the executable programs.

ⓧ I'd like to double check the full code

Compare your complete jobrunner.go file with this reference:

package jobrunner

import (
	"context"
	"fmt"
	"os"
	"time"

	"go.temporal.io/sdk/activity"
	"go.temporal.io/sdk/workflow"
)

const TaskQueue = "crash-proof-jobs"

type Job struct {
	Name                string
	DurationSeconds     int
	CrashOnFirstAttempt bool
}

func ProcessJob(ctx context.Context, job Job) (string, error) {
	info := activity.GetInfo(ctx)
	fmt.Printf("job=%s attempt=%d status=started\n", job.Name, info.Attempt)

	if job.CrashOnFirstAttempt && info.Attempt == 1 {
		time.Sleep(3 * time.Second)
		fmt.Printf("job=%s attempt=%d status=crashing-worker\n", job.Name, info.Attempt)
		os.Exit(1)
	}

	time.Sleep(time.Duration(job.DurationSeconds) * time.Second)
	result := fmt.Sprintf("%s completed on attempt %d", job.Name, info.Attempt)
	fmt.Printf("job=%s attempt=%d status=completed\n", job.Name, info.Attempt)
	return result, nil
}

func JobRunnerWorkflow(ctx workflow.Context, jobs []Job) ([]string, error) {
	ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
		StartToCloseTimeout: 8 * time.Second,
	})

	futures := make([]workflow.Future, len(jobs))
	for index, job := range jobs {
		futures[index] = workflow.ExecuteActivity(ctx, ProcessJob, job)
	}

	results := make([]string, len(jobs))
	for index, future := range futures {
		if err := future.Get(ctx, &results[index]); err != nil {
			return nil, err
		}
	}

	return results, nil
}
Connect the Worker, starter, and test

The Worker polls crash-proof-jobs for tasks. It registers the Workflow plus Activity functions that can execute those tasks.

  • Create worker/main.go inside the crashproof-job-runner folder.
  • Add the Worker program by pasting this code:
package main

import (
	"log"

	"example/crashproofjobrunner"
	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/worker"
)

func main() {
	c, err := client.Dial(client.Options{})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	w := worker.New(c, jobrunner.TaskQueue, worker.Options{})
	w.RegisterWorkflow(jobrunner.JobRunnerWorkflow)
	w.RegisterActivity(jobrunner.ProcessJob)

	log.Println("Worker is polling", jobrunner.TaskQueue)
	if err := w.Run(worker.InterruptCh()); err != nil {
		log.Fatalln("Unable to run worker", err)
	}
}

What does the Worker do?

  • client.Dial connects the program to the local Temporal Service.
  • worker.New creates a Worker for crash-proof-jobs.
  • RegisterWorkflow makes JobRunnerWorkflow available to the Worker.
  • RegisterActivity makes ProcessJob available to the Worker.
  • worker.InterruptCh() lets the Worker stop cleanly when its terminal receives an interrupt.
  • Save worker/main.go.

You should see a worker folder containing main.go in the workspace.

Does the Worker show import errors?

Confirm that the local import is exactly example/crashproofjobrunner. It must match the module path in go.mod.

help me fix the Worker imports

✔️ Awesome, I've got everything!

Your Worker entry point is ready. It can poll the shared Task Queue once the local service starts.

ⓧ I'd like to double check the full code

Compare your complete worker/main.go file with this reference:

package main

import (
	"log"

	"example/crashproofjobrunner"
	"go.temporal.io/sdk/client"
	"go.temporal.io/sdk/worker"
)

func main() {
	c, err := client.Dial(client.Options{})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	w := worker.New(c, jobrunner.TaskQueue, worker.Options{})
	w.RegisterWorkflow(jobrunner.JobRunnerWorkflow)
	w.RegisterActivity(jobrunner.ProcessJob)

	log.Println("Worker is polling", jobrunner.TaskQueue)
	if err := w.Run(worker.InterruptCh()); err != nil {
		log.Fatalln("Unable to run worker", err)
	}
}

The starter creates one Workflow execution. It blocks on run.Get until Temporal returns the durable results.

  • Create start/main.go inside the crashproof-job-runner folder.
  • Add the client setup plus job list by pasting this first section:
package main

import (
	"context"
	"log"

	"example/crashproofjobrunner"
	"go.temporal.io/sdk/client"
)

func main() {
	c, err := client.Dial(client.Options{})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	jobs := []jobrunner.Job{
		{Name: "quick-index", DurationSeconds: 1},
		{Name: "payment", DurationSeconds: 5, CrashOnFirstAttempt: false},
		{Name: "receipt", DurationSeconds: 6},
	}

What does this section prepare?

  • client.Dial creates the connection used to start the Workflow.
  • jobs contains the same three jobs as the naive runner.
  • CrashOnFirstAttempt: false keeps this first Temporal run successful from start to finish.
  • Complete main() by pasting this section below the jobs slice:
	run, err := c.ExecuteWorkflow(
		context.Background(),
		client.StartWorkflowOptions{TaskQueue: jobrunner.TaskQueue},
		jobrunner.JobRunnerWorkflow,
		jobs,
	)
	if err != nil {
		log.Fatalln("Unable to start Workflow", err)
	}

	log.Println("Workflow started. Waiting for a durable result.")
	var results []string
	if err := run.Get(context.Background(), &results); err != nil {
		log.Fatalln("Workflow failed", err)
	}

	log.Println("Workflow completed with durable results:")
	for _, result := range results {
		log.Println(result)
	}
}

How does the starter collect results?

  • c.ExecuteWorkflow starts JobRunnerWorkflow with the three jobs.
  • client.StartWorkflowOptions sends the Workflow to the shared Task Queue.
  • run.Get waits for the Workflow result without executing the jobs in the starter process.
  • results receives the three strings returned by the Workflow.
  • Save start/main.go.

The start folder now contains the program that launches one Workflow execution.

Seeing a syntax error in the starter?

Check that the second section sits directly below the closing brace of the jobs slice. The final brace in the second section closes main().

Confirm that the payment job uses CrashOnFirstAttempt: false for this first successful run.

help me fix start/main.go

✔️ Awesome, I've got everything!

The starter is ready to launch the successful version of your Workflow.

ⓧ I'd like to double check the full code

Compare your complete start/main.go file with this reference:

package main

import (
	"context"
	"log"

	"example/crashproofjobrunner"
	"go.temporal.io/sdk/client"
)

func main() {
	c, err := client.Dial(client.Options{})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	jobs := []jobrunner.Job{
		{Name: "quick-index", DurationSeconds: 1},
		{Name: "payment", DurationSeconds: 5, CrashOnFirstAttempt: false},
		{Name: "receipt", DurationSeconds: 6},
	}

	run, err := c.ExecuteWorkflow(
		context.Background(),
		client.StartWorkflowOptions{TaskQueue: jobrunner.TaskQueue},
		jobrunner.JobRunnerWorkflow,
		jobs,
	)
	if err != nil {
		log.Fatalln("Unable to start Workflow", err)
	}

	log.Println("Workflow started. Waiting for a durable result.")
	var results []string
	if err := run.Get(context.Background(), &results); err != nil {
		log.Fatalln("Workflow failed", err)
	}

	log.Println("Workflow completed with durable results:")
	for _, result := range results {
		log.Println(result)
	}
}

The test environment executes the Workflow without a live Temporal Service. Zero-second jobs keep the test focused on scheduling plus result collection.

  • Create jobrunner_test.go inside the crashproof-job-runner folder.
  • Add the test package plus imports by pasting this code:
package jobrunner

import (
	"testing"

	"go.temporal.io/sdk/testsuite"
)

What does the test import?

The standard testing package supplies testing.T. The SDK test suite supplies an isolated Workflow test environment.

  • Add TestJobRunnerWorkflow below the import block by pasting this code:
func TestJobRunnerWorkflow(t *testing.T) {
	var testSuite testsuite.WorkflowTestSuite
	env := testSuite.NewTestWorkflowEnvironment()
	env.RegisterActivity(ProcessJob)

	jobs := []Job{
		{Name: "alpha", DurationSeconds: 0},
		{Name: "beta", DurationSeconds: 0},
	}
	env.ExecuteWorkflow(JobRunnerWorkflow, jobs)

	if !env.IsWorkflowCompleted() {
		t.Fatal("Workflow did not complete")
	}
	if err := env.GetWorkflowError(); err != nil {
		t.Fatal(err)
	}

	var results []string
	if err := env.GetWorkflowResult(&results); err != nil {
		t.Fatal(err)
	}
	if len(results) != len(jobs) {
		t.Fatalf("got %d results, want %d", len(results), len(jobs))
	}
}

What does the test prove?

  • NewTestWorkflowEnvironment creates the isolated runtime.
  • RegisterActivity makes ProcessJob available during the test.
  • ExecuteWorkflow runs the Workflow with two zero-second jobs.
  • GetWorkflowResult checks that the Workflow returned one result for each job.
  • Save jobrunner_test.go.

You should now see jobrunner_test.go at the same level as jobrunner.go.

Does the test file show errors?

Confirm that the test uses package jobrunner. This gives it direct access to the project functions plus types.

help me fix the Workflow test file

✔️ Awesome, I've got everything!

Your isolated Workflow test is ready to run.

ⓧ I'd like to double check the full code

Compare your complete jobrunner_test.go file with this reference:

package jobrunner

import (
	"testing"

	"go.temporal.io/sdk/testsuite"
)

func TestJobRunnerWorkflow(t *testing.T) {
	var testSuite testsuite.WorkflowTestSuite
	env := testSuite.NewTestWorkflowEnvironment()
	env.RegisterActivity(ProcessJob)

	jobs := []Job{
		{Name: "alpha", DurationSeconds: 0},
		{Name: "beta", DurationSeconds: 0},
	}
	env.ExecuteWorkflow(JobRunnerWorkflow, jobs)

	if !env.IsWorkflowCompleted() {
		t.Fatal("Workflow did not complete")
	}
	if err := env.GetWorkflowError(); err != nil {
		t.Fatal(err)
	}

	var results []string
	if err := env.GetWorkflowResult(&results); err != nil {
		t.Fatal(err)
	}
	if len(results) != len(jobs) {
		t.Fatalf("got %d results, want %d", len(results), len(jobs))
	}
}
  • Return to the Ubuntu terminal in the crashproof-job-runner folder.
  • Download the imported dependencies plus update the module metadata by running:
go mod tidy

What does this command do?

go mod tidy downloads the imported SDK modules. It also records the complete dependency graph for your project.

The first dependency download can take a little time. Terminal activity during the download means the Go tool is still working.

Did the dependency download fail?

Confirm that Ubuntu has internet access. Check that go.mod contains the exact module path plus SDK version from this step.

help me diagnose go mod tidy

Before you run the test, do you expect the Workflow to complete without a live Temporal Service?

  • Run the isolated Workflow test with this command:
go test

Why can this test run offline?

The SDK test environment runs the Workflow plus registered Activity inside the test process. Your local Temporal development service is still stopped at this point.

You should see PASS in the terminal output. That result proves the Workflow completes plus returns one result for each test job.

Test did not pass?

Start with the first file plus line reported by the Go tool. Compare that section with the full-file references above.

Check that jobrunner_test.go registers ProcessJob before it executes JobRunnerWorkflow.

help me fix the failing Workflow test

Run the durable Workflow locally

The test proved that the Workflow logic works in isolation. A persistent local service now records the real execution plus exposes it through Temporal Web UI.

You need three Ubuntu terminals because the service plus Worker keep running while the starter waits for results. Keep each process in its own terminal.

  • Start the persistent local Temporal development service in the current Ubuntu terminal by running:
temporal server start-dev --db-filename temporal.db

What does this service command start?

The command starts the local Temporal Service plus its Web UI. It stores Workflow state in a local SQLite database named temporal.db.

This process keeps control of the terminal. Leave it running while you use the next two terminals.

  • Confirm that temporal.db appears in the Visual Studio Code Explorer.

The new database file proves that the development service has persistent local storage.

Did the local service fail to start?

Confirm that the terminal is inside crashproof-job-runner. Check that another Temporal development service is not already using the default local endpoints.

help me start the Temporal development service

  • Add a second Ubuntu terminal in the Visual Studio Code workspace.
  • Start the Worker from that terminal by running:
go run worker/main.go

What is the Worker waiting for?

The Worker connects to the local service. It polls crash-proof-jobs for Workflow tasks plus Activity tasks.

The process stays active because polling is continuous. Keep this terminal running.

You should see Worker is polling crash-proof-jobs in the Worker terminal.

Worker is not polling?

Confirm that the first terminal still runs the Temporal development service. Check that the Worker terminal is inside crashproof-job-runner.

help me connect the Worker

  • Add a third Ubuntu terminal in the same Visual Studio Code workspace.

The starter waits for the longest job to finish. Before you run it, do you expect all three results to arrive even though the Worker executes the jobs concurrently?

  • Launch one Workflow execution from the third terminal by running:
go run start/main.go

What happens during this run?

The starter submits one Workflow to the local service. The Worker receives the tasks through crash-proof-jobs.

The starter remains connected while run.Get waits for the Workflow result. The three Activities can execute concurrently because each call returns a Future.

You should see Workflow completed with durable results: followed by one result for quick-index, payment, plus receipt.

That is the durable split working. The service owns the Workflow state while the Worker performs disposable task executions.

Is the starter still waiting?

Check that the Worker terminal still shows an active polling process. Confirm that the starter plus Worker both reference crash-proof-jobs.

If the Worker stopped, return to its terminal plus inspect the last reported file or connection problem.

help me diagnose the waiting Workflow

  • Open http://localhost:8233 in your Windows browser.
  • Select the Workflow execution created by the starter.
  • Confirm that its status shows completion.

The completed execution in Temporal Web UI proves that the service recorded the Workflow outside the starter plus Worker processes.

Your first Temporal Workflow now returns all three durable results. Next, you will restore the intentional payment crash plus prove that a replacement Worker can finish the same open Workflow.

Crash the Worker and Prove Recovery

Your first Temporal Workflow completed all three Activities through one Task Queue. The Worker stayed alive for that success path.

Now you will make the Worker exit during the payment job. This reveals whether the same open Workflow can finish after its original process disappears.

Temporal stores Workflow state outside the disposable Worker process. The Start-To-Close Timeout detects the unfinished attempt.

The default Activity Retry Policy then schedules unfinished work for a replacement Worker. Completed work remains recorded throughout the recovery.

In this step, get ready to:
  • Turn on the intentional crash for the first payment attempt.
  • Keep the original starter waiting while a replacement Worker recovers the Workflow.
  • Verify retry evidence in Temporal Web UI.
Enable the payment crash

The CrashOnFirstAttempt field controls the fault injection already built into ProcessJob. The successful run kept this field disabled.

  • In start/main.go, locate the payment Job inside the jobs slice.
  • Find this line:
		{Name: "payment", DurationSeconds: 5, CrashOnFirstAttempt: false},

What does the current value do?

The false value keeps the payment job outside the crash branch. This allowed the first Workflow to finish without interruption.

  • Replace false with true so payment enters the crash branch on its first Activity attempt:
		{Name: "payment", DurationSeconds: 5, CrashOnFirstAttempt: true},

What does the new value do?

The true value activates the existing condition in ProcessJob. Payment attempt 1 now reaches os.Exit(1) after three seconds.

  • Save start/main.go.

✔️ Awesome, I've got everything!

Great. Make sure start/main.go is saved with the payment crash enabled.

ⓧ I'd like to double check the full code

Compare your complete start/main.go file with this version:

package main

import (
	"context"
	"log"

	"example/crashproofjobrunner"
	"go.temporal.io/sdk/client"
)

func main() {
	c, err := client.Dial(client.Options{})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	jobs := []jobrunner.Job{
		{Name: "quick-index", DurationSeconds: 1},
		{Name: "payment", DurationSeconds: 5, CrashOnFirstAttempt: true},
		{Name: "receipt", DurationSeconds: 6},
	}

	run, err := c.ExecuteWorkflow(
		context.Background(),
		client.StartWorkflowOptions{TaskQueue: jobrunner.TaskQueue},
		jobrunner.JobRunnerWorkflow,
		jobs,
	)
	if err != nil {
		log.Fatalln("Unable to start Workflow", err)
	}

	log.Println("Workflow started. Waiting for a durable result.")
	var results []string
	if err := run.Get(context.Background(), &results); err != nil {
		log.Fatalln("Workflow failed", err)
	}

	log.Println("Workflow completed with durable results:")
	for _, result := range results {
		log.Println(result)
	}
}
Crash and replace the Worker

The Temporal Service from the previous step remains available in its own terminal. It owns the durable Workflow state stored in temporal.db.

The starter can keep waiting after its Worker disappears. A replacement Worker can reconnect to the same Task Queue without creating another Workflow.

  • Leave the Temporal development server terminal running.
  • Switch back to the Worker terminal from earlier.
  • Press Ctrl+C to stop the currently polling Worker.
  • Start a fresh Worker from the project folder by running:
go run worker/main.go

What does this command do?

This starts the Worker defined in worker/main.go. It registers JobRunnerWorkflow plus ProcessJob.

The Worker then polls crash-proof-jobs for Workflow Tasks plus Activity Tasks.

  • Switch back to the starter terminal from earlier.

Before you run the starter, do you think the Workflow will fail permanently when its Worker exits?

  • Start one Workflow from the project folder by running:
go run start/main.go

What does this command do?

This creates one Workflow execution with the three jobs from start/main.go. The starter blocks on run.Get while it waits for that execution's durable result.

In the Worker terminal, you will see quick-index reach status=completed. Payment attempt 1 then reaches status=crashing-worker.

The Worker terminal returns to its shell prompt because os.Exit(1) terminated that process. The original starter remains on Workflow started. Waiting for a durable result..

This interruption is the intended failure. The open Workflow still exists inside the Temporal Service.

Worker stayed alive?

Confirm that the payment Job in start/main.go has CrashOnFirstAttempt: true.

Save the file before starting a new Workflow. An earlier Workflow keeps the input values it received when it started.

help me understand why my Worker did not exit

Before you restart the Worker, what do you expect the original starter to do when a replacement begins polling?

A short pause is expected while Temporal detects the missing attempt through the configured timeout. The waiting starter remains connected during that pause.

  • Restart only the Worker from its original terminal by running:
go run worker/main.go

How does the replacement recover?

The replacement Worker polls the same Task Queue. Temporal schedules unfinished Activity Tasks after the timeout plus retry rules allow another attempt.

The completed quick-index result remains in durable Workflow state. The original Futures continue waiting for the missing results.

Back in the original starter terminal, you will eventually see Workflow completed with durable results: followed by one result for each job.

That is the recovery proved: one Workflow survived the Worker crash. A replacement Worker finished its remaining work without another starter command.

Starter still waiting?

Confirm that the Temporal development server terminal is still running. The replacement Worker needs that local service to receive tasks.

Confirm that the replacement Worker reports it is polling crash-proof-jobs.

help me diagnose a waiting Temporal starter

Inspect the recovery evidence

The Event History is the durable record of what happened to the Workflow. Activity attempts above 1 show where Temporal rescheduled interrupted work.

  • Open http://localhost:8233 in your browser.
  • Select the most recent Workflow execution.
  • Confirm that its status is Completed.
  • Open Event History.
  • Inspect the Activity attempts shown for the execution.

You should see the Workflow marked Completed. You should also find interrupted work with an attempt number above 1.

The live execution proves crash recovery against your local service. The Workflow test provides an isolated check of the same application code.

Before you run the test, do you expect the intentional payment crash to affect the zero-duration test jobs?

  • Run the Workflow test from the project folder by running:
go test

What does this check prove?

The command executes TestJobRunnerWorkflow inside the Temporal Go SDK test environment. Its two jobs have zero-second durations without crash flags.

The output ends with PASS when the Workflow completes with the expected number of results.

You have now proved recovery through a live Worker replacement. Your automated Workflow test still passes after the fault injection change.

Test did not pass?

Confirm that the terminal is in the crashproof-job-runner folder containing go.mod plus jobrunner_test.go.

Compare the reported filename with the matching target artifact before changing code.

help me debug my failing Workflow test

Recovery creates an important production concern. An interrupted Activity can execute more than once.

Why idempotency matters

An Activity can finish an external write before its Worker reports completion. A crash during that gap can cause Temporal to retry the Activity.

Idempotency makes a repeated write produce the same safe outcome. Deduplication uses a stable identifier to reject duplicate operations.

The current Activity only sleeps plus logs. Real payments or emails need one of these protections before using automatic retries.

Your runner now demonstrates durable recovery across a real Worker crash. Completed work stays recorded while a replacement Worker retries unfinished Activities.

Secret mission

Resume Work from Heartbeats

Your retry currently repeats an interrupted Activity from the beginning. Add Activity Heartbeats so a replacement Worker restores recorded progress before finishing the open Workflow.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to keep the workspace active, pause its processes, or delete it. Everything runs locally with no ongoing service costs.

Resources you used:

  • A local Temporal development service. This process also provides Temporal Web UI.
  • A local Worker process that polls the Task Queue named crash-proof-jobs.
  • A local crashproof-job-runner folder inside WSL2 Ubuntu. It contains your Go source files. The temporal.db SQLite file holds local Workflow state.

Keep everything running

No action needed. Choose this if you are still testing crash recovery or heartbeat-aware retries.

  • Leave the Temporal development service running to keep the local Web UI available.
  • Leave the Worker running to keep polling crash-proof-jobs.
  • Retain the crashproof-job-runner folder to preserve your source code.
  • Retain temporal.db to preserve the local Workflow history.
  • Leave WSL2 Ubuntu installed for future backend projects.

Pause - I'll come back to this later

Stop the local processes to free their memory. Keep the workspace so you can resume with the same Workflow history.

  • Switch back to the Ubuntu terminal running the Worker.
  • Press Ctrl+C to stop the Worker.
  • Switch back to the Ubuntu terminal running the Temporal development service.
  • Press Ctrl+C to stop the service.
  • Retain the crashproof-job-runner folder with temporal.db inside it.
  • Return to Step 3 when you want to restart the service and Worker.

Your runtime is paused. The local Workflow history remains ready for your next session.

Delete - I don't want to use this again

Deleting this folder is permanent. Your source code and local Workflow history disappear together.

  • Switch back to the Ubuntu terminal running the Worker.
  • Press Ctrl+C to stop the Worker.
  • Switch back to the Ubuntu terminal running the Temporal development service.
  • Press Ctrl+C to stop the service.
  • Return to the stopped Worker terminal inside crashproof-job-runner.
  • Remove the workspace from its parent folder by running these commands:
cd ..
rm -rf crashproof-job-runner

What does this cleanup do?

The first line moves the terminal outside the workspace. The second line removes the crashproof-job-runner folder.

This deletion also removes temporal.db. The stored Workflow history disappears with it.

  • Return to Visual Studio Code.
  • Confirm that the project files no longer appear in the file tree.
  • Confirm that temporal.db no longer appears in the file tree.

Workspace still present?

  • Confirm that both local processes are stopped before retrying the deletion.
  • Confirm that the first command moved your terminal outside the project folder.
  • Help me safely remove my crashproof-job-runner folder from WSL2 Ubuntu.

Cleanup complete. Your project files are gone with their local Workflow history.

WSL2 Ubuntu remains installed as a reusable development environment.

Nice Work!

Nice Work!

You did it! Your crash-proof Go job runner now survives an intentional Worker crash because Temporal preserves its Workflow outside that disposable process.

You've learned how to:

  • Run three jobs concurrently with goroutines. You proved that process-local state forgets completed work when the process exits.
  • Schedule concurrent Activities through one Task Queue. You started a replacement Worker after the intentional exit. The original Workflow returned all three durable results. You also identified why real external writes need idempotency.
  • Inspect Event History in Temporal Web UI. You connected Activity attempts to the retry behavior. You confirmed the Workflow logic with an isolated Temporal Go SDK test environment.
  • Secret Mission: Extend ProcessJob with Activity Heartbeats that restore completedSeconds after a Worker crash. The retried job resumes from recorded progress instead of repeating every completed second.

Ready to quiz yourself?