Build a Bounded Chaos Engine
Build a safe RHEL chaos lab with automated incidents and evidence-led triage.
Introduction
30 Second Summary
Operational failures rarely announce their cause. A server can look healthy while one hidden resource limit turns routine work into an incident.
In this project, you will build a bounded chaos engineering lab on a disposable Red Hat Enterprise Linux 10.2 VM. You will use Bash with systemd to inject four short-lived failure modes before diagnosing a blind incident from command-line evidence.
What You'll Build
You start a surprise drill, trace its live Linux symptom to the hidden fault, and clear the incident through one recovery command.
By the end of this project, you'll have:
- The ability to run four bounded incidents that create visible CPU pressure, disk-space exhaustion, inode exhaustion, or isolated packet loss. Each fault expires automatically.
- A disposable tmpfs mount that keeps destructive storage symptoms away from the VM's root filesystem. An isolated network namespace keeps packet loss away from the management path.
- A workflow for launching blind randomized drills through systemd. You diagnose each drill from command-line evidence before comparing your hypothesis with journaled ground truth in a blameless post-mortem.
- Secret Mission: Combine CPU pressure with inode exhaustion in one stacked incident. Gather enough evidence to isolate both contributing layers.
Are there any prerequisites?
You need a disposable Red Hat Enterprise Linux 10.2 VM with a recent snapshot. You also need console recovery plus administrator access.
Before We Start
Before any fault injection, this checkpoint locks in a disposable RHEL 10.2 VM that has a current snapshot plus console recovery. It also commits every drill to bounded durations within isolated resources on a host reserved for this lab.
Verify the Lab and Safety Boundary
Fault injection deliberately creates stress inside a machine. A stale mount or interface could push that stress beyond the boundary you approved.
This preflight confirms your Red Hat Enterprise Linux 10.2 host can fail closed before it consumes resources. It verifies every required command package against a clean lab baseline.
In this step, get ready to:
- Refresh the rollback boundary for the disposable VM.
- Pass the host preflight.
- Prove no stale lab resources remain.
Refresh the recovery boundary
A fresh snapshot gives this destructive lab a reversible starting point. Console recovery gives you a route back if normal access stops working.
- Create a fresh snapshot of the disposable RHEL 10.2 VM using your provider's snapshot controls.
You should see the new snapshot recorded as available for recovery. That rollback point now reflects the untouched lab host.
- Connect to the same VM using your provider's recovery console.
- Sign in using your lab account.
You should reach a shell prompt on the disposable VM. This confirms that recovery does not depend on the VM's normal network path.
Verify authorization and package versions
Administrative authorization is the first safety gate because every later incident needs controlled root access. The authorization check must succeed before the lab can change mounts or networking.
- Verify your account can authorize elevated commands by running:
sudo -vWhat does this command prove?
The command asks sudo to validate your authorization. A successful return to the shell prompt proves the current account can perform the controlled administrative work used later.
You may be asked for your account password. A successful check returns you to the shell without an authorization failure.
Authorization not working?
- Confirm that you signed in with the administrative lab account assigned to this disposable VM.
- Use the recovery console to correct the account's administrative access before continuing.
- Avoid continuing as an unprivileged account because later cleanup would be unable to remove root-owned resources.
- Help me troubleshoot sudo authorization on my disposable RHEL VM.
Package records prove the VM has the expected builds of Bash plus systemd. Missing command packages would leave cleanup or observation gaps during an incident.
- Confirm the RHEL release and query all seven package records by running:
cat /etc/redhat-release
rpm -q bash systemd iproute iproute-tc iputils util-linux coreutilsWhat do these checks prove?
- The release file identifies the installed RHEL version.
- The rpm query reads the installed package records without changing the VM.
- The seven records cover the shell, service manager, networking commands, filesystem tools, CPU affinity command, timeout command, and core utilities used by the lab.
If a package is missing, DNF can install the verified architecture-neutral build from enabled RHEL repositories. Use the outcome below that matches your package query.
✔️ I see the required versions
That is the dependency gate cleared. Your VM already has every command package the chaos runner needs.
- Confirm the release output names Red Hat Enterprise Linux 10.2.
- Confirm the Bash record begins with bash-5.2.26.
- Confirm the systemd record begins with systemd-257.
- Confirm the iproute record begins with iproute-6.17.0.
- Confirm the traffic-control record begins with iproute-tc-6.17.0.
- Confirm the network-utility record begins with iputils-20240905.
- Confirm the util-linux record begins with util-linux-2.40.2.
- Confirm the coreutils record begins with coreutils-9.5.
ⓧ I see older versions
Older package records can change command behavior during the drill. Bring them to the verified RHEL 10.2 builds before continuing.
- Install the verified builds from enabled repositories by running this command:
- Review the package transaction when DNF pauses for confirmation.
- Approve the transaction only when it lists the seven lab packages from enabled RHEL 10.2 repositories.
sudo dnf install bash-5.2.26-6.el10 systemd-257-23.el10_2.2 iproute-6.17.0-2.el10 iproute-tc-6.17.0-2.el10 iputils-20240905-5.el10 util-linux-2.40.2-18.el10 coreutils-9.5-8.el10_2.1What does this command do?
DNF requests the seven verified builds from the VM's enabled repositories. It also resolves the dependencies required by those packages.
Package transaction not completing?
- Confirm that the VM has access to enabled RHEL 10.2 repositories.
- Restore the fresh snapshot if the transaction proposes removing unrelated system packages.
- Help me diagnose a failed DNF package transaction.
- Query the package records again by running:
rpm -q bash systemd iproute iproute-tc iputils util-linux coreutilsWhat confirms the upgrade?
The repeated query reads the package database after the transaction. All seven records should now begin with the required version prefixes shown in the success tab.
You should now see one installed record for each required package. None of the seven queries should report a missing package.
ⓧ One or more records are missing
A missing record means one or more commands required for injection or cleanup are unavailable. Use the verified install path only when the matching RHEL 10.2 repositories are enabled.
- Install the missing package records by running this command:
- Review the package transaction when DNF pauses for confirmation.
- Approve the transaction only when it lists the seven lab packages from enabled RHEL 10.2 repositories.
sudo dnf install bash-5.2.26-6.el10 systemd-257-23.el10_2.2 iproute-6.17.0-2.el10 iproute-tc-6.17.0-2.el10 iputils-20240905-5.el10 util-linux-2.40.2-18.el10 coreutils-9.5-8.el10_2.1What does this command do?
DNF installs the requested architecture-neutral package builds from enabled repositories. Dependency resolution supplies any supporting packages required by the transaction.
Package still unavailable?
- Confirm that the VM can reach its configured RHEL repositories.
- Confirm that the enabled repositories provide RHEL 10.2 content.
- Stop the preflight if the verified builds remain unavailable.
- Help me investigate a missing RHEL package record.
- Verify the installed package records by running:
rpm -q bash systemd iproute iproute-tc iputils util-linux coreutilsWhat confirms the installation?
The query should now return one installed record for every package. These records prove the required command providers are present before the lab creates resources.
You should now see seven installed package records. A missing-package result means the preflight remains closed.
Prove the lab is clean
A clean package baseline is only half of the safety boundary. Old namespaces, interfaces, mounts, runtime files, or units could collide with the first incident.
Before you run this audit, do you expect any named chaos-lab resource to appear?
Some probes deliberately return a nonzero status when resources are absent. That result is a clean signal in this audit.
- Verify authorization, release, packages, and the empty resource baseline by running:
sudo -v
cat /etc/redhat-release
rpm -q bash systemd iproute iproute-tc iputils util-linux coreutils
ip netns list
ip link show chaos-host
ip link show chaos-peer
findmnt /var/lib/chaos-lab/sandbox
if [ -e "/run/chaos-lab" ]
then
echo "present"
else
echo "absent"
fi
systemctl is-active chaos-runner.service
systemctl is-active chaos-runner.timer
systemctl list-timersWhat does this audit cover?
- The first three commands repeat the required authorization, release, and package checks.
- The network commands search for the lab namespace plus both planned virtual interfaces.
- The filesystem checks search for the planned sandbox mount plus runtime state.
- The systemd checks search for an active service or timer from an earlier lab attempt.
Your release output should name Red Hat Enterprise Linux 10.2. Your RPM query should list all seven records at the required versions.
The namespace list should omit chaos-lab. The interface probes should find neither chaos-host nor chaos-peer.
The mount query should return no lab mount. The runtime check should print absent.
Both unit checks should report an inactive or absent state. The timer list should omit chaos-runner.timer.
That is the safety boundary proven. Your host now has the required package baseline with no leftover incident state.
Found a stale lab resource?
- Stop before creating any new chaos-lab resource.
- Restore the fresh snapshot you created at the start of this step.
- Run the full audit again after the restored VM reaches its shell prompt.
- Help me interpret a stale chaos-lab resource check.
The lab can now accept a bounded incident runner. Next, you will create the runner and observe short-lived CPU pressure.
Build the Runner and Trigger CPU Pressure
Your Red Hat Enterprise Linux lab has passed its safety checks. The next goal is a runner that rejects unsafe inputs before it launches a fault.
The first incident uses CPU affinity to pin one busy worker to CPU 0. A timeout gives that worker a fixed lifetime.
Because you select the CPU incident yourself, its cause is already known. That limitation prepares the lab for a blind drill later.
In this step, get ready to:
- Create a safety-gated Bash incident dispatcher.
- Run one bounded CPU-pressure incident.
- Confirm the incident expires with no active runtime state.
Lay down the safety boundary
The Bash runner owns the complete incident lifecycle. It validates the request before recording state or starting a worker.
- Start a root-owned here-document for /usr/local/sbin/chaos_runner.sh by running this command:
sudo tee /usr/local/sbin/chaos_runner.sh >/dev/null <<'CHAOS_RUNNER'What does this command do?
The command opens a root-owned input stream for the runner file. Your shell keeps collecting the following chunks until it receives the closing delimiter.
- Paste the runner header plus its safety constants into the waiting here-document:
#!/usr/bin/env bash
set -Eeuo pipefail
readonly BASE_DIR="/var/lib/chaos-lab"
readonly RUNTIME_DIR="/run/chaos-lab"
readonly MOUNTPOINT="${BASE_DIR}/sandbox"
readonly REPORTS_DIR="${BASE_DIR}/reports"
readonly DISABLED_FILE="${BASE_DIR}/DISABLED"
readonly LOCK_DIR="${RUNTIME_DIR}/lock"
readonly CURRENT_FILE="${RUNTIME_DIR}/current.env"
readonly TRUTH_FILE="${BASE_DIR}/ground-truth.env"
readonly NETNS="chaos-lab"
readonly HOST_IF="chaos-host"
readonly PEER_IF="chaos-peer"
readonly HOST_CIDR="192.0.2.1/30"
readonly PEER_CIDR="192.0.2.2/30"
readonly TARGET_IP="192.0.2.2"
worker_pid=""
lock_held=0What do these values control?
- The directory constants separate persistent evidence from temporary runtime state.
- The disabled-file path provides a kill switch before any incident begins.
- The network constants reserve isolated names plus documentation addresses for the network module.
- The worker and lock variables let cleanup track resources created by the current process.
- Paste the logging helpers plus the root check into the same here-document:
log() {
printf 'chaos-lab: %s\n' "$*"
}
fail() {
log "ERROR: $*"
exit 1
}
require_root() {
[[ "$(id -u)" -eq 0 ]] || fail "run this command with sudo"
}
prepare_directories() {
mkdir -p "$BASE_DIR" "$RUNTIME_DIR" "$MOUNTPOINT" "$REPORTS_DIR"
chmod 0755 "$BASE_DIR" "$RUNTIME_DIR" "$REPORTS_DIR"
chmod 0700 "$MOUNTPOINT"
}How does the runner fail safely?
- The logging helper gives every message a consistent prefix for later journal filtering.
- The failure helper records the reason before returning a nonzero exit status.
- The root check stops privileged operations when authorization is missing.
- The directory helper applies restrictive permissions to the incident sandbox.
- Paste the dependency preflight below the directory helper:
preflight() {
require_root
local missing=0
local command_name
for command_name in bash systemctl journalctl ip tc ping mount umount mountpoint taskset timeout dd df ps awk date mkdir rm chmod sleep rpm; do
if ! command -v "$command_name" >/dev/null 2>&1; then
log "missing command: $command_name"
missing=1
fi
done
[[ -d /run/systemd/system ]] || { log "systemd is not PID 1"; missing=1; }
grep -qw tmpfs /proc/filesystems || { log "tmpfs is unavailable"; missing=1; }
taskset --cpu-list 0 true >/dev/null 2>&1 || { log "CPU 0 is unavailable to this VM"; missing=1; }
[[ "$missing" -eq 0 ]] || fail "preflight failed"
rpm -q bash systemd iproute iproute-tc iputils util-linux coreutils
log "preflight passed"
}What does the preflight protect?
- The command loop checks every dependency used across the incident modules.
- The system manager check confirms the VM is running systemd as its service manager.
- The filesystem check confirms that bounded temporary mounts are available.
- The affinity check confirms that CPU 0 can host the bounded worker.
- Paste the shared cleanup logic into the same here-document:
namespace_exists() {
[[ -e "/var/run/netns/${NETNS}" ]]
}
cleanup_resources() {
set +e
if [[ -n "${worker_pid:-}" ]]; then
kill "$worker_pid" 2>/dev/null
wait "$worker_pid" 2>/dev/null
fi
tc qdisc del dev "$HOST_IF" root 2>/dev/null
if namespace_exists; then
ip netns del "$NETNS" 2>/dev/null
fi
ip link del "$HOST_IF" 2>/dev/null
if mountpoint -q "$MOUNTPOINT"; then
umount "$MOUNTPOINT"
fi
rm -rf -- "${MOUNTPOINT:?}/"*
rm -f "$CURRENT_FILE"
if [[ "$lock_held" -eq 1 || -d "$LOCK_DIR" ]]; then
rmdir "$LOCK_DIR" 2>/dev/null
fi
set -e
}Why is cleanup centralized?
Every incident can leave a different resource behind. One idempotent cleanup function can safely retry the complete teardown path after success or interruption.
The guarded sandbox path protects the removal command from expanding to an empty location. The current-state file disappears during every cleanup.
- Paste the lock plus duration validation functions below the cleanup logic:
acquire_lock() {
if ! mkdir "$LOCK_DIR" 2>/dev/null; then
fail "another incident is active; stop chaos-runner.service or run cleanup"
fi
lock_held=1
}
validate_duration() {
local value="$1"
[[ "$value" =~ ^[0-9]+$ ]] || fail "duration must be an integer"
(( value >= 10 && value <= 300 )) || fail "duration must be between 10 and 300 seconds"
}How are overlapping incidents blocked?
Creating the lock directory is one atomic operation. A second runner cannot acquire the same directory while the first incident owns it.
Duration validation limits every accepted incident to between 10 seconds and 300 seconds. Invalid values fail before fault injection starts.
- Paste the incident-recording function into the same here-document:
record_incident() {
local module="$1"
local duration="$2"
local incident_id
local started_at
incident_id="$(date -u +%Y%m%dT%H%M%SZ)-$$"
started_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
cat > "$TRUTH_FILE" <<EOF
INCIDENT_ID=${incident_id}
MODULE=${module}
STARTED_AT=${started_at}
DURATION_SECONDS=${duration}
EOF
chmod 0600 "$TRUTH_FILE"
cat > "$CURRENT_FILE" <<EOF
INCIDENT_ID=${incident_id}
STARTED_AT=${started_at}
DURATION_SECONDS=${duration}
STATE=active
EOF
chmod 0644 "$CURRENT_FILE"
log "incident_id=${incident_id} selected_module=${module} duration_seconds=${duration}"
}Why are there two state files?
The ground-truth file includes the selected module. Its restrictive permissions keep that answer sealed during blind diagnosis.
The current-state file omits the module name. It proves that an incident is active without revealing its cause.
Add the bounded incident modules
The dispatcher needs a complete function for each accepted module name. This step exercises only the CPU path.
- Paste the CPU incident function into the waiting here-document:
cpu_incident() {
local duration="$1"
log "injecting bounded CPU pressure on CPU 0"
taskset --cpu-list 0 timeout --signal=TERM --kill-after=2s "${duration}s" bash -c 'while :; do :; done' &
worker_pid=$!
if wait "$worker_pid"; then
return 0
else
local status=$?
[[ "$status" -eq 124 || "$status" -eq 137 ]] || return "$status"
fi
}How is CPU pressure bounded?
- The affinity command confines the busy loop to CPU 0.
- The timeout command sends a termination signal after the requested duration.
- The worker PID gives cleanup a specific process to stop during interruption.
- The accepted timeout statuses keep planned expiry from becoming a runner failure.
- Paste the disk-space incident function below the CPU function:
disk_incident() {
local duration="$1"
log "injecting bounded disk-space pressure in ${MOUNTPOINT}"
mount -t tmpfs -o size=24M,nr_inodes=4096,mode=0700 tmpfs "$MOUNTPOINT"
dd if=/dev/zero of="${MOUNTPOINT}/payload.bin" bs=1M count=30 status=none || true
sleep "$duration"
}How is disk pressure isolated?
The function limits writes to a small tmpfs mounted at the sandbox path. Its payload cannot consume the VM's root filesystem.
- Paste the inode incident function into the same here-document:
inode_incident() {
local duration="$1"
local index=0
log "injecting bounded inode pressure in ${MOUNTPOINT}"
mount -t tmpfs -o size=8M,nr_inodes=128,mode=0700 tmpfs "$MOUNTPOINT"
while (( index < 256 )); do
if ! : > "${MOUNTPOINT}/file-${index}"; then
break
fi
index=$((index + 1))
done
log "created_files=${index}"
sleep "$duration"
}How is inode pressure bounded?
The mount has a fixed inode limit. The loop stops when file creation fails or reaches its own maximum iteration count.
- Paste the isolated network incident function below the inode function:
network_incident() {
local duration="$1"
log "creating isolated network namespace"
ip netns add "$NETNS"
ip link add "$HOST_IF" type veth peer name "$PEER_IF"
ip link set "$PEER_IF" netns "$NETNS"
ip address add "$HOST_CIDR" dev "$HOST_IF"
ip link set "$HOST_IF" up
ip netns exec "$NETNS" ip address add "$PEER_CIDR" dev "$PEER_IF"
ip netns exec "$NETNS" ip link set lo up
ip netns exec "$NETNS" ip link set "$PEER_IF" up
ping -c 1 -W 1 "$TARGET_IP" >/dev/null
tc qdisc add dev "$HOST_IF" root netem loss 100%
log "isolated endpoint is now blackholed"
sleep "$duration"
}How is the network fault contained?
A network namespace owns the peer endpoint. Packet loss applies only to the lab's host-side virtual interface.
The VM's management interface stays outside this topology. Its default route remains unchanged.
- Paste the status function into the same here-document:
show_status() {
if [[ -f "$CURRENT_FILE" ]]; then
cat "$CURRENT_FILE"
else
printf 'STATE=inactive\n'
fi
}What does status reveal?
Active status prints the current incident metadata without its module name. Inactive status proves that no current-state file remains.
- Paste the protected ground-truth reveal function below the status function:
reveal_truth() {
require_root
[[ -f "$TRUTH_FILE" ]] || fail "no ground truth has been recorded"
cat "$TRUTH_FILE"
}Why does reveal require root?
The module name is diagnostic ground truth. Requiring root keeps it unavailable to an unprivileged observer during a blind drill.
- Paste the incident lifecycle function into the waiting here-document:
run_incident() {
local requested_module="$1"
local duration="$2"
local selected_module="$requested_module"
local modules=(cpu disk inode network)
require_root
prepare_directories
preflight
[[ ! -e "$DISABLED_FILE" ]] || fail "kill switch is active at ${DISABLED_FILE}"
validate_duration "$duration"
acquire_lock
trap cleanup_resources EXIT INT TERM HUP
if [[ "$requested_module" == "auto" ]]; then
selected_module="${modules[RANDOM % ${#modules[@]}]}"
fi
case "$selected_module" in
cpu|disk|inode|network) ;;
*) fail "module must be auto, cpu, disk, inode, or network" ;;
esac
record_incident "$selected_module" "$duration"
"${selected_module}_incident" "$duration"
log "incident completed; cleanup follows"
}What controls the incident lifecycle?
- Preflight runs before the kill-switch check or incident selection.
- The lock prevents a second incident from overlapping the active one.
- The signal trap routes normal exit plus interruption through the same cleanup function.
- The dispatcher records the selected module before calling its matching function.
- Paste the command dispatcher below the incident lifecycle function:
main() {
local action="${1:-status}"
local duration="${2:-45}"
case "$action" in
preflight)
prepare_directories
preflight
;;
status)
show_status
;;
reveal)
reveal_truth
;;
cleanup)
require_root
prepare_directories
cleanup_resources
log "cleanup complete"
;;
auto|cpu|disk|inode|network)
run_incident "$action" "$duration"
;;
*)
fail "usage: $0 {preflight|status|reveal|cleanup|auto|cpu|disk|inode|network} [duration_seconds]"
;;
esac
}How does the dispatcher choose an action?
The first argument selects a control action or incident module. The second argument defaults to a 45-second duration.
Unknown actions stop at the usage message. Accepted incident names enter the safety-gated lifecycle.
- Paste the final entry point into the waiting here-document:
main "$@"What starts the runner?
The entry point forwards every command-line argument to the dispatcher. This keeps argument handling in one function.
- Close the here-document plus make the runner executable by entering:
CHAOS_RUNNER
sudo chmod 0755 /usr/local/sbin/chaos_runner.shWhat changes now?
The delimiter commits all collected content to the runner file. The permission change lets root execute that file directly.
Before you run the preflight, which safety checks do you expect the runner to perform?
- Test the completed runner's safety gate by running:
sudo /usr/local/sbin/chaos_runner.sh preflightWhat should the preflight prove?
You should see all seven installed package records. The final runner message should report that the preflight passed.
This proves the required commands are available. It also proves CPU 0 can host the incident worker.
That is the safety gate working. Your runner now refuses to inject faults when its operating assumptions are missing.
Preflight not passing?
- Check the preflight output for the specific dependency or capability it reports as missing.
- Confirm that you ran the script with sudo if the root check stops execution.
- Return to the previous step if one of the seven package records is absent.
- Ask for help with the exact preflight output you received.
✔️ Awesome, I've got everything!
Great. Confirm that the runner is executable before you start the incident.
ⓧ I'd like to double check the full code
Compare your runner with this complete file. Every line should appear in the same order.
#!/usr/bin/env bash
set -Eeuo pipefail
readonly BASE_DIR="/var/lib/chaos-lab"
readonly RUNTIME_DIR="/run/chaos-lab"
readonly MOUNTPOINT="${BASE_DIR}/sandbox"
readonly REPORTS_DIR="${BASE_DIR}/reports"
readonly DISABLED_FILE="${BASE_DIR}/DISABLED"
readonly LOCK_DIR="${RUNTIME_DIR}/lock"
readonly CURRENT_FILE="${RUNTIME_DIR}/current.env"
readonly TRUTH_FILE="${BASE_DIR}/ground-truth.env"
readonly NETNS="chaos-lab"
readonly HOST_IF="chaos-host"
readonly PEER_IF="chaos-peer"
readonly HOST_CIDR="192.0.2.1/30"
readonly PEER_CIDR="192.0.2.2/30"
readonly TARGET_IP="192.0.2.2"
worker_pid=""
lock_held=0
log() {
printf 'chaos-lab: %s\n' "$*"
}
fail() {
log "ERROR: $*"
exit 1
}
require_root() {
[[ "$(id -u)" -eq 0 ]] || fail "run this command with sudo"
}
prepare_directories() {
mkdir -p "$BASE_DIR" "$RUNTIME_DIR" "$MOUNTPOINT" "$REPORTS_DIR"
chmod 0755 "$BASE_DIR" "$RUNTIME_DIR" "$REPORTS_DIR"
chmod 0700 "$MOUNTPOINT"
}
preflight() {
require_root
local missing=0
local command_name
for command_name in bash systemctl journalctl ip tc ping mount umount mountpoint taskset timeout dd df ps awk date mkdir rm chmod sleep rpm; do
if ! command -v "$command_name" >/dev/null 2>&1; then
log "missing command: $command_name"
missing=1
fi
done
[[ -d /run/systemd/system ]] || { log "systemd is not PID 1"; missing=1; }
grep -qw tmpfs /proc/filesystems || { log "tmpfs is unavailable"; missing=1; }
taskset --cpu-list 0 true >/dev/null 2>&1 || { log "CPU 0 is unavailable to this VM"; missing=1; }
[[ "$missing" -eq 0 ]] || fail "preflight failed"
rpm -q bash systemd iproute iproute-tc iputils util-linux coreutils
log "preflight passed"
}
namespace_exists() {
[[ -e "/var/run/netns/${NETNS}" ]]
}
cleanup_resources() {
set +e
if [[ -n "${worker_pid:-}" ]]; then
kill "$worker_pid" 2>/dev/null
wait "$worker_pid" 2>/dev/null
fi
tc qdisc del dev "$HOST_IF" root 2>/dev/null
if namespace_exists; then
ip netns del "$NETNS" 2>/dev/null
fi
ip link del "$HOST_IF" 2>/dev/null
if mountpoint -q "$MOUNTPOINT"; then
umount "$MOUNTPOINT"
fi
rm -rf -- "${MOUNTPOINT:?}/"*
rm -f "$CURRENT_FILE"
if [[ "$lock_held" -eq 1 || -d "$LOCK_DIR" ]]; then
rmdir "$LOCK_DIR" 2>/dev/null
fi
set -e
}
acquire_lock() {
if ! mkdir "$LOCK_DIR" 2>/dev/null; then
fail "another incident is active; stop chaos-runner.service or run cleanup"
fi
lock_held=1
}
validate_duration() {
local value="$1"
[[ "$value" =~ ^[0-9]+$ ]] || fail "duration must be an integer"
(( value >= 10 && value <= 300 )) || fail "duration must be between 10 and 300 seconds"
}
record_incident() {
local module="$1"
local duration="$2"
local incident_id
local started_at
incident_id="$(date -u +%Y%m%dT%H%M%SZ)-$$"
started_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
cat > "$TRUTH_FILE" <<EOF
INCIDENT_ID=${incident_id}
MODULE=${module}
STARTED_AT=${started_at}
DURATION_SECONDS=${duration}
EOF
chmod 0600 "$TRUTH_FILE"
cat > "$CURRENT_FILE" <<EOF
INCIDENT_ID=${incident_id}
STARTED_AT=${started_at}
DURATION_SECONDS=${duration}
STATE=active
EOF
chmod 0644 "$CURRENT_FILE"
log "incident_id=${incident_id} selected_module=${module} duration_seconds=${duration}"
}
cpu_incident() {
local duration="$1"
log "injecting bounded CPU pressure on CPU 0"
taskset --cpu-list 0 timeout --signal=TERM --kill-after=2s "${duration}s" bash -c 'while :; do :; done' &
worker_pid=$!
if wait "$worker_pid"; then
return 0
else
local status=$?
[[ "$status" -eq 124 || "$status" -eq 137 ]] || return "$status"
fi
}
disk_incident() {
local duration="$1"
log "injecting bounded disk-space pressure in ${MOUNTPOINT}"
mount -t tmpfs -o size=24M,nr_inodes=4096,mode=0700 tmpfs "$MOUNTPOINT"
dd if=/dev/zero of="${MOUNTPOINT}/payload.bin" bs=1M count=30 status=none || true
sleep "$duration"
}
inode_incident() {
local duration="$1"
local index=0
log "injecting bounded inode pressure in ${MOUNTPOINT}"
mount -t tmpfs -o size=8M,nr_inodes=128,mode=0700 tmpfs "$MOUNTPOINT"
while (( index < 256 )); do
if ! : > "${MOUNTPOINT}/file-${index}"; then
break
fi
index=$((index + 1))
done
log "created_files=${index}"
sleep "$duration"
}
network_incident() {
local duration="$1"
log "creating isolated network namespace"
ip netns add "$NETNS"
ip link add "$HOST_IF" type veth peer name "$PEER_IF"
ip link set "$PEER_IF" netns "$NETNS"
ip address add "$HOST_CIDR" dev "$HOST_IF"
ip link set "$HOST_IF" up
ip netns exec "$NETNS" ip address add "$PEER_CIDR" dev "$PEER_IF"
ip netns exec "$NETNS" ip link set lo up
ip netns exec "$NETNS" ip link set "$PEER_IF" up
ping -c 1 -W 1 "$TARGET_IP" >/dev/null
tc qdisc add dev "$HOST_IF" root netem loss 100%
log "isolated endpoint is now blackholed"
sleep "$duration"
}
show_status() {
if [[ -f "$CURRENT_FILE" ]]; then
cat "$CURRENT_FILE"
else
printf 'STATE=inactive\n'
fi
}
reveal_truth() {
require_root
[[ -f "$TRUTH_FILE" ]] || fail "no ground truth has been recorded"
cat "$TRUTH_FILE"
}
run_incident() {
local requested_module="$1"
local duration="$2"
local selected_module="$requested_module"
local modules=(cpu disk inode network)
require_root
prepare_directories
preflight
[[ ! -e "$DISABLED_FILE" ]] || fail "kill switch is active at ${DISABLED_FILE}"
validate_duration "$duration"
acquire_lock
trap cleanup_resources EXIT INT TERM HUP
if [[ "$requested_module" == "auto" ]]; then
selected_module="${modules[RANDOM % ${#modules[@]}]}"
fi
case "$selected_module" in
cpu|disk|inode|network) ;;
*) fail "module must be auto, cpu, disk, inode, or network" ;;
esac
record_incident "$selected_module" "$duration"
"${selected_module}_incident" "$duration"
log "incident completed; cleanup follows"
}
main() {
local action="${1:-status}"
local duration="${2:-45}"
case "$action" in
preflight)
prepare_directories
preflight
;;
status)
show_status
;;
reveal)
reveal_truth
;;
cleanup)
require_root
prepare_directories
cleanup_resources
log "cleanup complete"
;;
auto|cpu|disk|inode|network)
run_incident "$action" "$duration"
;;
*)
fail "usage: $0 {preflight|status|reveal|cleanup|auto|cpu|disk|inode|network} [duration_seconds]"
;;
esac
}
main "$@"Trigger CPU pressure
This first run makes the cause visible because you choose the module name. The process view still gives you real evidence of CPU affinity plus load.
- Start a second shell session from your VM console.
- Keep the original shell visible for the runner output.
Before you start the incident, do you expect the pinned worker to rise near the top of the process list?
- Start a 20-second CPU incident in the original shell by running:
sudo /usr/local/sbin/chaos_runner.sh cpu 20What happens during this command?
The runner repeats its preflight before acquiring the lock. It then records the selected module plus active-state metadata.
The command holds this shell for about 20 seconds while the worker runs. Cleanup starts as soon as the timeout expires.
- Inspect CPU usage immediately from the second shell by running:
ps -eo pid,psr,pcpu,comm --sort=-pcpuHow do you read this process view?
- The PID column identifies each running process.
- The PSR column shows the processor currently assigned to the process.
- The PCPU column reports recent CPU usage.
- The COMM column shows the executable name.
You should see a Bash worker near the top of the list. Its processor column should show CPU 0 while the incident is active.
Confirm automatic cleanup
A bounded incident must remove its worker plus runtime markers without manual intervention. This check proves the timeout reached the cleanup trap.
- Return to the original shell after the 20-second incident finishes.
You should see the completion message followed by the return of your shell prompt. The runner has now entered its cleanup path.
Before you inspect the process list again, do you expect the busy worker to remain near the top?
- Inspect the process list again from the second shell by running:
ps -eo pid,psr,pcpu,comm --sort=-pcpuWhat should change?
The timeout-limited busy worker should no longer appear near the top of the list. This proves the CPU-consuming process has exited.
Before you check the runner status, do you expect it to report an active incident?
- Confirm that the active-state file was removed by running:
sudo /usr/local/sbin/chaos_runner.sh statusWhat proves cleanup completed?
You should see STATE=inactive. The status function prints that value only when /run/chaos-lab/current.env is absent.
The same cleanup path also releases the lock. A future incident can now start safely.
Worker or state still active?
- Wait until the original incident command has returned to its shell prompt.
- Check whether the original runner was interrupted before its cleanup trap was installed.
- Use the runner's cleanup action if a previous test left runtime state behind.
- Help me diagnose a CPU incident that did not clean up.
Your runner can now create a visible CPU incident without leaving an active worker behind. Next up, you will use its bounded sandbox to compare disk-space exhaustion with inode exhaustion.
Add Disk and Inode Incidents
Your Bash runner already produces a bounded CPU incident. That first drill proved the shared safety controls can end an incident automatically.
CPU pressure cannot show the difference between exhausted storage blocks and exhausted inodes. This step creates both symptoms inside a small tmpfs sandbox so the VM's root filesystem stays outside the blast radius.
In this step, get ready to:
- Add a bounded disk-space incident to the chaos runner.
- Add a bounded inode incident to the chaos runner.
- Compare byte capacity with inode capacity during explicit drills.
Add the bounded disk-space module
The disk incident mounts a temporary filesystem with a 24 MiB ceiling at /var/lib/chaos-lab/sandbox. Writing a 30 MiB payload forces No space left on device inside that sandbox.
- Return to the root-owned shell workflow you used to create /usr/local/sbin/chaos_runner.sh.
- Locate the existing disk_incident() function.
- Replace that function with this exact definition:
disk_incident() {
local duration="$1"
log "injecting bounded disk-space pressure in ${MOUNTPOINT}"
mount -t tmpfs -o size=24M,nr_inodes=4096,mode=0700 tmpfs "$MOUNTPOINT"
dd if=/dev/zero of="${MOUNTPOINT}/payload.bin" bs=1M count=30 status=none || true
sleep "$duration"
}What Does This Code Do?
- The duration value controls how long the full sandbox remains available for diagnosis.
- The mount command limits the temporary filesystem to 24 MiB and 4096 inodes.
- The dd command attempts to write 30 MiB into the smaller filesystem.
- The || true clause keeps the drill active after the planned write failure.
- Save /usr/local/sbin/chaos_runner.sh.
- Start a second shell session connected to the same disposable VM.
Before you start the drill, consider which filesystem should reach its byte limit while the payload is being written.
- Start the 20-second disk incident from the original shell by running:
sudo /usr/local/sbin/chaos_runner.sh disk 20- Measure the sandbox capacity from the second shell by running:
df -h /var/lib/chaos-lab/sandboxYou should see the sandbox backed by a 24 MiB tmpfs with its available space exhausted. The original shell should report No space left on device while the runner remains active for the requested duration.
Disk Incident Not Reaching Capacity?
- Confirm the original shell is still running the 20-second incident when you execute df.
- Check that the mount options contain size=24M.
- Check that payload.bin is written with a count of 30 MiB blocks.
- Help me diagnose why my bounded tmpfs disk incident does not reach its capacity.
Add the bounded inode module
A filesystem can retain free bytes after it runs out of inodes. This incident demonstrates that condition by allowing only 128 inodes inside an 8 MiB tmpfs.
- Wait for the disk incident to finish its automatic cleanup.
- Locate the existing inode_incident() function in /usr/local/sbin/chaos_runner.sh.
- Replace that function with this exact definition:
inode_incident() {
local duration="$1"
local index=0
log "injecting bounded inode pressure in ${MOUNTPOINT}"
mount -t tmpfs -o size=8M,nr_inodes=128,mode=0700 tmpfs "$MOUNTPOINT"
while (( index < 256 )); do
if ! : > "${MOUNTPOINT}/file-${index}"; then
break
fi
index=$((index + 1))
done
log "created_files=${index}"
sleep "$duration"
}How Does Inode Exhaustion Work?
- The index variable tracks how many empty files the module creates.
- The tmpfs mount limits inode capacity to 128 entries.
- Each empty file-N entry consumes an inode while using almost no data capacity.
- The loop stops when the filesystem rejects another file allocation.
- Save /usr/local/sbin/chaos_runner.sh.
Before you run the inode drill, consider whether byte usage or inode usage should reach its limit first.
- Start the 20-second inode incident from the original shell by running:
sudo /usr/local/sbin/chaos_runner.sh inode 20- Compare byte capacity with inode capacity from the second shell by running:
df -h /var/lib/chaos-lab/sandbox
df -i /var/lib/chaos-lab/sandboxThe first report should retain free bytes. The second report should show that the tmpfs has reached its inode limit.
That distinction is the diagnostic signal. A service can fail to create a tiny file even when df -h still reports available space.
Inodes Not Reaching Their Limit?
- Run both capacity checks while the original shell still holds the incident open.
- Confirm the inode mount option is nr_inodes=128.
- Confirm the loop creates empty files below /var/lib/chaos-lab/sandbox.
- Help me diagnose why my inode incident still reports free inodes.
Confirm automatic storage cleanup
The shared exit trap should unmount the sandbox after each duration expires. It should also remove the generated payload and inode-test files.
- Wait for the inode incident to finish in the original shell.
Before the final check, predict whether the runner should report an active incident after the 20-second duration has expired.
- Check the runner state from the original shell by running:
sudo /usr/local/sbin/chaos_runner.sh status- Check which filesystem now serves the sandbox path from the second shell by running:
df -h /var/lib/chaos-lab/sandboxYou should see STATE=inactive from the runner. The sandbox path should report the VM's underlying filesystem instead of the temporary filesystem used during the drill.
Strong result. Your runner now creates two distinct storage failures inside a disposable mount before cleaning up the active state automatically.
Sandbox Still Mounted?
- Confirm the incident process has returned to your shell prompt.
- Check that cleanup_resources() still unmounts MOUNTPOINT.
- Check that the shared trap still calls cleanup_resources when the incident exits.
- Help me find why the chaos runner left its tmpfs sandbox mounted.
✔️ Awesome, I've got everything!
Great. Double-check that /usr/local/sbin/chaos_runner.sh is saved and remains executable.
ⓧ I'd like to double check the full code
Compare your cumulative runner with this reference file.
#!/usr/bin/env bash
set -Eeuo pipefail
readonly BASE_DIR="/var/lib/chaos-lab"
readonly RUNTIME_DIR="/run/chaos-lab"
readonly MOUNTPOINT="${BASE_DIR}/sandbox"
readonly REPORTS_DIR="${BASE_DIR}/reports"
readonly DISABLED_FILE="${BASE_DIR}/DISABLED"
readonly LOCK_DIR="${RUNTIME_DIR}/lock"
readonly CURRENT_FILE="${RUNTIME_DIR}/current.env"
readonly TRUTH_FILE="${BASE_DIR}/ground-truth.env"
readonly NETNS="chaos-lab"
readonly HOST_IF="chaos-host"
readonly PEER_IF="chaos-peer"
readonly HOST_CIDR="192.0.2.1/30"
readonly PEER_CIDR="192.0.2.2/30"
readonly TARGET_IP="192.0.2.2"
worker_pid=""
lock_held=0
log() {
printf 'chaos-lab: %s\n' "$*"
}
fail() {
log "ERROR: $*"
exit 1
}
require_root() {
[[ "$(id -u)" -eq 0 ]] || fail "run this command with sudo"
}
prepare_directories() {
mkdir -p "$BASE_DIR" "$RUNTIME_DIR" "$MOUNTPOINT" "$REPORTS_DIR"
chmod 0755 "$BASE_DIR" "$RUNTIME_DIR" "$REPORTS_DIR"
chmod 0700 "$MOUNTPOINT"
}
preflight() {
require_root
local missing=0
local command_name
for command_name in bash systemctl journalctl ip tc ping mount umount mountpoint taskset timeout dd df ps awk date mkdir rm chmod sleep rpm; do
if ! command -v "$command_name" >/dev/null 2>&1; then
log "missing command: $command_name"
missing=1
fi
done
[[ -d /run/systemd/system ]] || { log "systemd is not PID 1"; missing=1; }
grep -qw tmpfs /proc/filesystems || { log "tmpfs is unavailable"; missing=1; }
taskset --cpu-list 0 true >/dev/null 2>&1 || { log "CPU 0 is unavailable to this VM"; missing=1; }
[[ "$missing" -eq 0 ]] || fail "preflight failed"
rpm -q bash systemd iproute iproute-tc iputils util-linux coreutils
log "preflight passed"
}
namespace_exists() {
[[ -e "/var/run/netns/${NETNS}" ]]
}
cleanup_resources() {
set +e
if [[ -n "${worker_pid:-}" ]]; then
kill "$worker_pid" 2>/dev/null
wait "$worker_pid" 2>/dev/null
fi
tc qdisc del dev "$HOST_IF" root 2>/dev/null
if namespace_exists; then
ip netns del "$NETNS" 2>/dev/null
fi
ip link del "$HOST_IF" 2>/dev/null
if mountpoint -q "$MOUNTPOINT"; then
umount "$MOUNTPOINT"
fi
rm -rf -- "${MOUNTPOINT:?}/"*
rm -f "$CURRENT_FILE"
if [[ "$lock_held" -eq 1 || -d "$LOCK_DIR" ]]; then
rmdir "$LOCK_DIR" 2>/dev/null
fi
set -e
}
acquire_lock() {
if ! mkdir "$LOCK_DIR" 2>/dev/null; then
fail "another incident is active; stop chaos-runner.service or run cleanup"
fi
lock_held=1
}
validate_duration() {
local value="$1"
[[ "$value" =~ ^[0-9]+$ ]] || fail "duration must be an integer"
(( value >= 10 && value <= 300 )) || fail "duration must be between 10 and 300 seconds"
}
record_incident() {
local module="$1"
local duration="$2"
local incident_id
local started_at
incident_id="$(date -u +%Y%m%dT%H%M%SZ)-$$"
started_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
cat > "$TRUTH_FILE" <<EOF
INCIDENT_ID=${incident_id}
MODULE=${module}
STARTED_AT=${started_at}
DURATION_SECONDS=${duration}
EOF
chmod 0600 "$TRUTH_FILE"
cat > "$CURRENT_FILE" <<EOF
INCIDENT_ID=${incident_id}
STARTED_AT=${started_at}
DURATION_SECONDS=${duration}
STATE=active
EOF
chmod 0644 "$CURRENT_FILE"
log "incident_id=${incident_id} selected_module=${module} duration_seconds=${duration}"
}
cpu_incident() {
local duration="$1"
log "injecting bounded CPU pressure on CPU 0"
taskset --cpu-list 0 timeout --signal=TERM --kill-after=2s "${duration}s" bash -c 'while :; do :; done' &
worker_pid=$!
if wait "$worker_pid"; then
return 0
else
local status=$?
[[ "$status" -eq 124 || "$status" -eq 137 ]] || return "$status"
fi
}
disk_incident() {
local duration="$1"
log "injecting bounded disk-space pressure in ${MOUNTPOINT}"
mount -t tmpfs -o size=24M,nr_inodes=4096,mode=0700 tmpfs "$MOUNTPOINT"
dd if=/dev/zero of="${MOUNTPOINT}/payload.bin" bs=1M count=30 status=none || true
sleep "$duration"
}
inode_incident() {
local duration="$1"
local index=0
log "injecting bounded inode pressure in ${MOUNTPOINT}"
mount -t tmpfs -o size=8M,nr_inodes=128,mode=0700 tmpfs "$MOUNTPOINT"
while (( index < 256 )); do
if ! : > "${MOUNTPOINT}/file-${index}"; then
break
fi
index=$((index + 1))
done
log "created_files=${index}"
sleep "$duration"
}
network_incident() {
local duration="$1"
log "creating isolated network namespace"
ip netns add "$NETNS"
ip link add "$HOST_IF" type veth peer name "$PEER_IF"
ip link set "$PEER_IF" netns "$NETNS"
ip address add "$HOST_CIDR" dev "$HOST_IF"
ip link set "$HOST_IF" up
ip netns exec "$NETNS" ip address add "$PEER_CIDR" dev "$PEER_IF"
ip netns exec "$NETNS" ip link set lo up
ip netns exec "$NETNS" ip link set "$PEER_IF" up
ping -c 1 -W 1 "$TARGET_IP" >/dev/null
tc qdisc add dev "$HOST_IF" root netem loss 100%
log "isolated endpoint is now blackholed"
sleep "$duration"
}
show_status() {
if [[ -f "$CURRENT_FILE" ]]; then
cat "$CURRENT_FILE"
else
printf 'STATE=inactive\n'
fi
}
reveal_truth() {
require_root
[[ -f "$TRUTH_FILE" ]] || fail "no ground truth has been recorded"
cat "$TRUTH_FILE"
}
run_incident() {
local requested_module="$1"
local duration="$2"
local selected_module="$requested_module"
local modules=(cpu disk inode network)
require_root
prepare_directories
preflight
[[ ! -e "$DISABLED_FILE" ]] || fail "kill switch is active at ${DISABLED_FILE}"
validate_duration "$duration"
acquire_lock
trap cleanup_resources EXIT INT TERM HUP
if [[ "$requested_module" == "auto" ]]; then
selected_module="${modules[RANDOM % ${#modules[@]}]}"
fi
case "$selected_module" in
cpu|disk|inode|network) ;;
*) fail "module must be auto, cpu, disk, inode, or network" ;;
esac
record_incident "$selected_module" "$duration"
"${selected_module}_incident" "$duration"
log "incident completed; cleanup follows"
}
main() {
local action="${1:-status}"
local duration="${2:-45}"
case "$action" in
preflight)
prepare_directories
preflight
;;
status)
show_status
;;
reveal)
reveal_truth
;;
cleanup)
require_root
prepare_directories
cleanup_resources
log "cleanup complete"
;;
auto|cpu|disk|inode|network)
run_incident "$action" "$duration"
;;
*)
fail "usage: $0 {preflight|status|reveal|cleanup|auto|cpu|disk|inode|network} [duration_seconds]"
;;
esac
}
main "$@"Your storage drills now reproduce exhausted bytes and exhausted inodes within the same bounded sandbox. Next, you will create a network blackhole without touching the VM's management path.
Add an Isolated Network Blackhole
Your runner now creates bounded CPU pressure. It also distinguishes exhausted data blocks from exhausted inodes.
Network faults carry a sharper recovery risk. A host-level packet-loss rule could sever the management connection you need to repair the VM.
This step confines the fault to a network namespace. A veth pair connects the isolated endpoint to the host. Netem drops traffic only on that lab connection.
In this step, get ready to:
- Add the isolated network module to the chaos runner.
- Trigger complete packet loss to the namespace endpoint.
- Confirm every temporary network resource disappears after the incident.
Add the isolated network module
A network namespace owns its own devices. It also owns its own routes. The VM management interface remains outside that boundary.
Why isolate the network fault?
The lab uses 192.0.2.0/24. This range is reserved for documentation.
The packet-loss rule targets only chaos-host. Your real interface keeps its existing route.
- Return to the open shell session from the previous step.
- Locate the closing brace of inode_incident() in /usr/local/sbin/chaos_runner.sh.
- Insert the following function immediately after that closing brace by using the same shell here-document method from earlier:
network_incident() {
local duration="$1"
log "creating isolated network namespace"
ip netns add "$NETNS"
ip link add "$HOST_IF" type veth peer name "$PEER_IF"
ip link set "$PEER_IF" netns "$NETNS"
ip address add "$HOST_CIDR" dev "$HOST_IF"
ip link set "$HOST_IF" up
ip netns exec "$NETNS" ip address add "$PEER_CIDR" dev "$PEER_IF"
ip netns exec "$NETNS" ip link set lo up
ip netns exec "$NETNS" ip link set "$PEER_IF" up
ping -c 1 -W 1 "$TARGET_IP" >/dev/null
tc qdisc add dev "$HOST_IF" root netem loss 100%
log "isolated endpoint is now blackholed"
sleep "$duration"
}What does this function do?
- The NETNS value names the temporary namespace.
- The host keeps HOST_IF. The runner moves PEER_IF into the namespace.
- The host endpoint receives 192.0.2.1/30.
- The isolated endpoint receives 192.0.2.2/30.
- The baseline ping proves the endpoint responds before injection.
- A root qdisc applies complete loss only to chaos-host.
- The final sleep keeps the fault active for the validated duration.
- Finish the here-document to save /usr/local/sbin/chaos_runner.sh.
Trigger the bounded blackhole
A route baseline gives you evidence that the management path stays unchanged. The open management shell gives you a second signal that the VM remains reachable.
- Start a second shell session to the same disposable VM by using the connection method from earlier.
- Capture the management default route in the second shell by running this command:
ip route show defaultWhat does this baseline show?
The output identifies the route used for traffic outside the VM. Keep this line visible for comparison during the isolated fault.
- Switch back to the original shell session.
The command remains attached for about 20 seconds. A quiet pause after the blackhole log means the fault is active.
- Start the bounded network incident by running this command:
sudo /usr/local/sbin/chaos_runner.sh network 20What happens before the blackhole?
The runner builds the namespace endpoint first. Its silent baseline probe must succeed before Netem applies packet loss.
The blackhole log confirms that setup reached the injection point. The process then holds the fault open for the requested duration.
- Switch to the second shell session.
Before you test the endpoint, do you expect its probes to receive replies while the blackhole is active?
- Test the isolated endpoint during the incident by running this command:
ping -c 2 -W 1 192.0.2.2What should you see?
Both probes time out. The summary reports complete packet loss to 192.0.2.2.
This planned failure comes from the Netem rule on chaos-host.
- Display the management default route during the active incident by running this command:
ip route show defaultWhat does the route comparison prove?
You should see the same default route you captured before the incident. Your second shell also remains connected throughout the test.
Together, those observations prove that packet loss stays inside the lab connection.
Still receiving ping replies?
- Check the original shell for the message that confirms the isolated endpoint is blackholed.
- Repeat the incident command if the bounded duration ended before your probe.
- Confirm that the probe targets 192.0.2.2.
- Ask for help with the isolated fault setup.
Verify automatic network cleanup
The existing cleanup trap deletes the Netem rule. It also removes the namespace plus the host side of the veth pair.
- Wait for the original incident command to return to the shell prompt.
The runner has finished its bounded duration. Its cleanup trap has now run.
Before you check the namespace, do you expect the temporary lab name to remain after the runner exits?
- Check the namespace cleanup in the second shell by running this command:
ip netns listWhat confirms namespace cleanup?
The output no longer contains chaos-lab. Deleting that namespace also removes its chaos-peer endpoint.
The next check asks the kernel for both temporary interfaces. Do you expect either interface to remain after cleanup?
- Query both lab interfaces by running these commands:
ip link show chaos-host
ip link show chaos-peerWhy do failed lookups mean success?
Each lookup returns a nonzero result because its named interface is absent. The host endpoint is gone.
The namespace deletion removed the peer endpoint. No temporary veth resource remains.
- Compare the management route with your original baseline by running this command:
ip route show defaultWhat proves the safety boundary held?
You should see the same default route for a third time. The incident changed only the disposable namespace connection.
Your management session stayed available from injection through cleanup. The network experiment remained inside its boundary.
Namespace or interface still present?
- Confirm that the original incident command has returned to the shell prompt.
- Wait for the bounded duration to finish before repeating the cleanup checks.
- Compare the cleanup function's namespace value with chaos-lab.
- Ask for help removing only the stale lab resources.
That is the safety boundary proven. Your runner can now create a complete endpoint outage without disturbing the VM management route.
✔️ Awesome, I've got everything!
Your isolated network blackhole works. Double-check that you saved the completed runner.
ⓧ I'd like to double check the full code
Compare your saved runner with this complete version.
#!/usr/bin/env bash
set -Eeuo pipefail
readonly BASE_DIR="/var/lib/chaos-lab"
readonly RUNTIME_DIR="/run/chaos-lab"
readonly MOUNTPOINT="${BASE_DIR}/sandbox"
readonly REPORTS_DIR="${BASE_DIR}/reports"
readonly DISABLED_FILE="${BASE_DIR}/DISABLED"
readonly LOCK_DIR="${RUNTIME_DIR}/lock"
readonly CURRENT_FILE="${RUNTIME_DIR}/current.env"
readonly TRUTH_FILE="${BASE_DIR}/ground-truth.env"
readonly NETNS="chaos-lab"
readonly HOST_IF="chaos-host"
readonly PEER_IF="chaos-peer"
readonly HOST_CIDR="192.0.2.1/30"
readonly PEER_CIDR="192.0.2.2/30"
readonly TARGET_IP="192.0.2.2"
worker_pid=""
lock_held=0
log() {
printf 'chaos-lab: %s\n' "$*"
}
fail() {
log "ERROR: $*"
exit 1
}
require_root() {
[[ "$(id -u)" -eq 0 ]] || fail "run this command with sudo"
}
prepare_directories() {
mkdir -p "$BASE_DIR" "$RUNTIME_DIR" "$MOUNTPOINT" "$REPORTS_DIR"
chmod 0755 "$BASE_DIR" "$RUNTIME_DIR" "$REPORTS_DIR"
chmod 0700 "$MOUNTPOINT"
}
preflight() {
require_root
local missing=0
local command_name
for command_name in bash systemctl journalctl ip tc ping mount umount mountpoint taskset timeout dd df ps awk date mkdir rm chmod sleep rpm; do
if ! command -v "$command_name" >/dev/null 2>&1; then
log "missing command: $command_name"
missing=1
fi
done
[[ -d /run/systemd/system ]] || { log "systemd is not PID 1"; missing=1; }
grep -qw tmpfs /proc/filesystems || { log "tmpfs is unavailable"; missing=1; }
taskset --cpu-list 0 true >/dev/null 2>&1 || { log "CPU 0 is unavailable to this VM"; missing=1; }
[[ "$missing" -eq 0 ]] || fail "preflight failed"
rpm -q bash systemd iproute iproute-tc iputils util-linux coreutils
log "preflight passed"
}
namespace_exists() {
[[ -e "/var/run/netns/${NETNS}" ]]
}
cleanup_resources() {
set +e
if [[ -n "${worker_pid:-}" ]]; then
kill "$worker_pid" 2>/dev/null
wait "$worker_pid" 2>/dev/null
fi
tc qdisc del dev "$HOST_IF" root 2>/dev/null
if namespace_exists; then
ip netns del "$NETNS" 2>/dev/null
fi
ip link del "$HOST_IF" 2>/dev/null
if mountpoint -q "$MOUNTPOINT"; then
umount "$MOUNTPOINT"
fi
rm -rf -- "${MOUNTPOINT:?}/"*
rm -f "$CURRENT_FILE"
if [[ "$lock_held" -eq 1 || -d "$LOCK_DIR" ]]; then
rmdir "$LOCK_DIR" 2>/dev/null
fi
set -e
}
acquire_lock() {
if ! mkdir "$LOCK_DIR" 2>/dev/null; then
fail "another incident is active; stop chaos-runner.service or run cleanup"
fi
lock_held=1
}
validate_duration() {
local value="$1"
[[ "$value" =~ ^[0-9]+$ ]] || fail "duration must be an integer"
(( value >= 10 && value <= 300 )) || fail "duration must be between 10 and 300 seconds"
}
record_incident() {
local module="$1"
local duration="$2"
local incident_id
local started_at
incident_id="$(date -u +%Y%m%dT%H%M%SZ)-$$"
started_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
cat > "$TRUTH_FILE" <<EOF
INCIDENT_ID=${incident_id}
MODULE=${module}
STARTED_AT=${started_at}
DURATION_SECONDS=${duration}
EOF
chmod 0600 "$TRUTH_FILE"
cat > "$CURRENT_FILE" <<EOF
INCIDENT_ID=${incident_id}
STARTED_AT=${started_at}
DURATION_SECONDS=${duration}
STATE=active
EOF
chmod 0644 "$CURRENT_FILE"
log "incident_id=${incident_id} selected_module=${module} duration_seconds=${duration}"
}
cpu_incident() {
local duration="$1"
log "injecting bounded CPU pressure on CPU 0"
taskset --cpu-list 0 timeout --signal=TERM --kill-after=2s "${duration}s" bash -c 'while :; do :; done' &
worker_pid=$!
if wait "$worker_pid"; then
return 0
else
local status=$?
[[ "$status" -eq 124 || "$status" -eq 137 ]] || return "$status"
fi
}
disk_incident() {
local duration="$1"
log "injecting bounded disk-space pressure in ${MOUNTPOINT}"
mount -t tmpfs -o size=24M,nr_inodes=4096,mode=0700 tmpfs "$MOUNTPOINT"
dd if=/dev/zero of="${MOUNTPOINT}/payload.bin" bs=1M count=30 status=none || true
sleep "$duration"
}
inode_incident() {
local duration="$1"
local index=0
log "injecting bounded inode pressure in ${MOUNTPOINT}"
mount -t tmpfs -o size=8M,nr_inodes=128,mode=0700 tmpfs "$MOUNTPOINT"
while (( index < 256 )); do
if ! : > "${MOUNTPOINT}/file-${index}"; then
break
fi
index=$((index + 1))
done
log "created_files=${index}"
sleep "$duration"
}
network_incident() {
local duration="$1"
log "creating isolated network namespace"
ip netns add "$NETNS"
ip link add "$HOST_IF" type veth peer name "$PEER_IF"
ip link set "$PEER_IF" netns "$NETNS"
ip address add "$HOST_CIDR" dev "$HOST_IF"
ip link set "$HOST_IF" up
ip netns exec "$NETNS" ip address add "$PEER_CIDR" dev "$PEER_IF"
ip netns exec "$NETNS" ip link set lo up
ip netns exec "$NETNS" ip link set "$PEER_IF" up
ping -c 1 -W 1 "$TARGET_IP" >/dev/null
tc qdisc add dev "$HOST_IF" root netem loss 100%
log "isolated endpoint is now blackholed"
sleep "$duration"
}
show_status() {
if [[ -f "$CURRENT_FILE" ]]; then
cat "$CURRENT_FILE"
else
printf 'STATE=inactive\n'
fi
}
reveal_truth() {
require_root
[[ -f "$TRUTH_FILE" ]] || fail "no ground truth has been recorded"
cat "$TRUTH_FILE"
}
run_incident() {
local requested_module="$1"
local duration="$2"
local selected_module="$requested_module"
local modules=(cpu disk inode network)
require_root
prepare_directories
preflight
[[ ! -e "$DISABLED_FILE" ]] || fail "kill switch is active at ${DISABLED_FILE}"
validate_duration "$duration"
acquire_lock
trap cleanup_resources EXIT INT TERM HUP
if [[ "$requested_module" == "auto" ]]; then
selected_module="${modules[RANDOM % ${#modules[@]}]}"
fi
case "$selected_module" in
cpu|disk|inode|network) ;;
*) fail "module must be auto, cpu, disk, inode, or network" ;;
esac
record_incident "$selected_module" "$duration"
"${selected_module}_incident" "$duration"
log "incident completed; cleanup follows"
}
main() {
local action="${1:-status}"
local duration="${2:-45}"
case "$action" in
preflight)
prepare_directories
preflight
;;
status)
show_status
;;
reveal)
reveal_truth
;;
cleanup)
require_root
prepare_directories
cleanup_resources
log "cleanup complete"
;;
auto|cpu|disk|inode|network)
run_incident "$action" "$duration"
;;
*)
fail "usage: $0 {preflight|status|reveal|cleanup|auto|cpu|disk|inode|network} [duration_seconds]"
;;
esac
}
main "$@"Your runner now creates four bounded incident types without touching the VM management route. Next, you will schedule surprise drills before diagnosing one from evidence.
Schedule and Triage a Blind Incident
Your runner now creates four bounded incident types inside the disposable VM. The explicit commands from earlier drills revealed each cause before diagnosis began.
This step places the runner behind systemd automation with a sealed journald trail. A randomized activation forces you to diagnose symptoms before you reveal the selected module.
In this step, get ready to:
- Install the systemd service and randomized timer.
- Diagnose one blind incident through a five-stage workflow.
- Complete an evidence-led post-mortem for the incident.
Schedule randomized incidents
A service defines how one bounded incident runs. A timer controls when that service starts.
Why systemd instead of cron?
Systemd keeps each incident under a named unit. Its stop path gives you one recovery command.
The service journal preserves the selected module for the controlled reveal. A cron schedule would need separate mechanisms for those controls.
- Create /etc/systemd/system/chaos-runner.service through a shell here-document by running:
sudo tee /etc/systemd/system/chaos-runner.service <<'EOF'
[Unit]
Description=Bounded RHEL chaos incident injector
ConditionPathExists=!/var/lib/chaos-lab/DISABLED
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/chaos_runner.sh auto 45
ExecStopPost=/usr/local/sbin/chaos_runner.sh cleanup
TimeoutStartSec=60
Nice=10
EOFHow does the service stay bounded?
- The ConditionPathExists=!/var/lib/chaos-lab/DISABLED condition blocks activation when the kill switch exists.
- The Type=oneshot setting ties the activation to one runner process.
- The ExecStart=/usr/local/sbin/chaos_runner.sh auto 45 line selects one module for a 45-second incident.
- The ExecStopPost=/usr/local/sbin/chaos_runner.sh cleanup line invokes idempotent cleanup after the service stops.
- The TimeoutStartSec=60 setting limits the activation window.
- The Nice=10 setting lowers the service scheduling priority.
- Check the terminal output for the complete [Unit] section.
- Check the terminal output for the complete [Service] section.
Service file not created?
Confirm the shell session still has sudo authorization. Check that the destination path is /etc/systemd/system/chaos-runner.service.
Select Help me create the chaos runner service file..
- Create /etc/systemd/system/chaos-runner.timer through a second shell here-document by running:
sudo tee /etc/systemd/system/chaos-runner.timer <<'EOF'
[Unit]
Description=Randomized schedule for bounded RHEL chaos incidents
[Timer]
OnBootSec=15min
OnUnitActiveSec=45min
RandomizedDelaySec=15min
AccuracySec=1s
Unit=chaos-runner.service
[Install]
WantedBy=timers.target
EOFHow does the timer choose its schedule?
- The OnBootSec=15min setting places the first activation relative to boot.
- The OnUnitActiveSec=45min setting places later activations relative to the previous service activation.
- The RandomizedDelaySec=15min setting adds up to 15 minutes of randomized delay.
- The AccuracySec=1s setting gives the timer a one-second accuracy window.
- The Unit=chaos-runner.service setting names the service activated by the timer.
- The WantedBy=timers.target setting connects the enabled timer to the system timer target.
- Check the terminal output for the [Timer] section.
- Check the terminal output for the [Install] section.
Timer file incomplete?
Confirm the closing EOF appears at the start of its own line. Run the complete here-document again if the shell continued waiting for input.
Select Help me fix the chaos runner timer file..
✔️ Awesome, I've got both unit files
Both systemd unit files are ready. Continue once their fields match the references.
ⓧ I'd like to double check the full code
[Unit]
Description=Bounded RHEL chaos incident injector
ConditionPathExists=!/var/lib/chaos-lab/DISABLED
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/chaos_runner.sh auto 45
ExecStopPost=/usr/local/sbin/chaos_runner.sh cleanup
TimeoutStartSec=60
Nice=10Service file reference
This is the complete oneshot service file. Its start action runs one blind 45-second incident.
[Unit]
Description=Randomized schedule for bounded RHEL chaos incidents
[Timer]
OnBootSec=15min
OnUnitActiveSec=45min
RandomizedDelaySec=15min
AccuracySec=1s
Unit=chaos-runner.service
[Install]
WantedBy=timers.targetTimer file reference
This is the complete randomized timer file. Enabling it makes the schedule available across later boots.
- Reload the unit definitions by running these commands:
sudo systemctl daemon-reload
sudo systemctl enable --now chaos-runner.timer
sudo systemctl list-timersWhat do these commands do?
- The first command reloads administrator-created unit files from /etc/systemd/system/.
- The second command enables the timer for later boots. It starts the timer now.
- The final command lists timers by their next activation time.
You should see chaos-runner.timer with a future activation. Your lab can now schedule surprise drills without an external scheduler.
Timer missing from the list?
Compare both unit files with the full-code references. Run the reload command again after correcting either file.
Select Help me diagnose the missing timer..
The scheduled timer could overlap the manual blind drill. Pause its activations while you collect controlled evidence.
- Pause timer activations by running this command:
sudo systemctl stop chaos-runner.timerWhy pause the timer?
Stopping the timer leaves it enabled for later boots. It prevents a scheduled activation from competing with the manual drill.
Diagnose and mitigate a blind incident
The service writes the selected module to protected ground truth. Your diagnostic shell begins with symptoms only.
- Start a second shell session to the same disposable VM through your existing connection method.
- Keep the chaos-runner.service journal closed until after mitigation.
The first shell can remain occupied for up to 45 seconds while the selected incident runs. That delay is expected during this drill.
Before you start the service, which symptom do you think will identify the affected layer first?
- Start one blind incident from the first shell by running:
sudo systemctl start chaos-runner.serviceWhat does this start?
The service invokes the runner with auto for 45 seconds. The runner selects from cpu, disk, inode, or network.
The first shell waits for the oneshot service. Use the second shell to gather evidence immediately.
Service did not start?
Confirm the DISABLED kill-switch file is absent from /var/lib/chaos-lab. Confirm another incident does not hold the runtime lock.
Select Help me diagnose the failed blind incident..
- Switch to the second shell session.
- Collect evidence across the runner state, processor activity, filesystem capacity, inode capacity, and isolated network by running:
sudo /usr/local/sbin/chaos_runner.sh status
ps -eo pid,psr,pcpu,comm --sort=-pcpu
df -h /var/lib/chaos-lab/sandbox
df -i /var/lib/chaos-lab/sandbox
ip netns list
ping -c 2 -W 1 192.0.2.2How do you isolate the faulty layer?
- A STATE=active result confirms that the runner owns an active incident.
- A busy Bash process on processor 0 supports a CPU-pressure hypothesis.
- A full tmpfs byte report supports a disk-space hypothesis.
- Free bytes beside exhausted inodes support an inode-pressure hypothesis.
- A network namespace named chaos-lab beside failed probes supports a network-blackhole hypothesis.
- A failed probe without the namespace is weak network evidence because the isolated endpoint does not exist for the other modules.
Incident ended too quickly?
Wait for the first service activation to finish. Start another blind run after confirming the runtime lock has cleared.
Paste the complete diagnostic sweep into the second shell immediately. Select Help me gather evidence within the incident window..
- Mitigate the incident with a UTC timestamp by running these commands in the second shell:
date -u +%Y-%m-%dT%H:%M:%SZ
sudo systemctl stop chaos-runner.serviceWhy does stopping the service recover the lab?
The timestamp anchors the mitigation in your report. Stopping the unit terminates the active runner.
Systemd then invokes the configured cleanup action. The cleanup removes the worker, mount, namespace resources, runtime state, and lock.
- Confirm the active state has cleared by running this verification:
sudo /usr/local/sbin/chaos_runner.sh status
df -h /var/lib/chaos-lab/sandbox
df -i /var/lib/chaos-lab/sandbox
ip netns listWhat proves mitigation worked?
- The runner status should show STATE=inactive.
- The sandbox reports should no longer identify a dedicated tmpfs mount.
- The namespace list should no longer contain chaos-lab.
- The persistent ground-truth file remains available for the controlled reveal.
Transient resources still present?
Run the service stop command again if the runner remains active. Use the runner cleanup action if the mount or namespace remains.
Select Help me verify chaos incident cleanup..
Before you reveal the answer, which MODULE value does your evidence predict?
- Reveal the recorded ground truth by running:
sudo /usr/local/sbin/chaos_runner.sh revealHow do you compare the result?
- The INCIDENT_ID value identifies this drill.
- The MODULE value confirms or corrects your hypothesis.
- The STARTED_AT value anchors the beginning of your timeline.
- The DURATION_SECONDS value records the configured incident window.
That is the blind triage complete. You diagnosed a bounded incident from independent evidence before consulting its recorded cause.
Ground truth unavailable?
Confirm the blind incident reached the runner before the service stopped. Use sudo because the ground-truth file has root-only read permissions.
Select Help me recover the ground-truth record..
Complete the post-mortem
A Markdown post-mortem connects symptoms, evidence, mitigation, and ground truth. Its action items turn lessons into measurable follow-up work.
- Create the report header through the detection section by running:
sudo tee /var/lib/chaos-lab/postmortem-template.md <<'EOF'
# Chaos Incident Post-Mortem
Status: Draft
Incident ID:
Date and time UTC:
Author:
## Summary
Describe what failed and how the incident was resolved in two or three sentences.
## Impact
- Affected endpoint or resource:
- User-visible symptom:
- Start time:
- Mitigation time:
- End time:
- Duration:
## Detection
Describe the first symptom and the command or observation that detected it.
EOFWhat does the opening capture?
The opening fields establish ownership and timing. The impact fields connect the technical incident to an observable consequence.
The detection section preserves the first signal before hindsight changes the story.
- Check the terminal output for the Summary, Impact, and Detection headings.
Opening sections missing?
Run the complete creation command again if any opening heading is absent. The first command replaces an incomplete template.
Select Help me create the opening post-mortem sections..
- Append the first three triage stages by running:
sudo tee -a /var/lib/chaos-lab/postmortem-template.md <<'EOF'
## Five-Stage Triage
### 1. Observe symptoms
Record observations without naming a cause.
### 2. Isolate the layer
State which layer you tested and which alternatives you ruled out.
### 3. Form a hypothesis with CLI evidence
Hypothesis:
Evidence that supports it:
Evidence that could disprove it:
EOFWhy separate observation from hypothesis?
The first stages keep raw symptoms separate from interpretation. That separation exposes assumptions before they become conclusions.
- Check the terminal output for triage stages 1 through 3.
Triage stages replaced the template?
Confirm the command includes the append option. Recreate the opening chunk before appending again if the original content disappeared.
Select Help me restore the post-mortem template..
- Append the mitigation, comparison, and timeline sections by running:
sudo tee -a /var/lib/chaos-lab/postmortem-template.md <<'EOF'
### 4. Mitigate
Record the exact mitigation, its timestamp, and the recovery evidence.
### 5. Compare with ground truth
- Predicted module:
- Revealed module:
- Match or mismatch:
- Evidence you missed or interpreted incorrectly:
## Timeline
| Time UTC | Observation or action | Evidence |
|---|---|---|
| | | |
EOFWhat does the comparison preserve?
The comparison records your prediction beside the revealed module. A mismatch becomes evidence about what the diagnostic process missed.
The timeline links each conclusion to a specific observation or action.
- Check the terminal output for triage stages 4 and 5.
- Check the terminal output for the timeline table.
Timeline table malformed?
Confirm each table row starts with a vertical bar. Run this append chunk again only after removing any malformed duplicate lines.
Select Help me repair the Markdown timeline table..
- Append the learning and action-item sections by running:
sudo tee -a /var/lib/chaos-lab/postmortem-template.md <<'EOF'
## Trigger and Root Cause
Trigger:
Root cause or contributing causes:
## What Went Well
-
## What Went Poorly
-
## Where We Got Lucky
-
## Action Items
| Action | Type | Priority | Owner | Verifiable completion condition |
|---|---|---|---|---|
| | Detect, prevent, mitigate, or process | | | |
EOFWhat makes the follow-up measurable?
The root-cause section distinguishes the trigger from contributing conditions. The learning sections keep the report blameless.
Each action item receives an owner and a verifiable completion condition. That turns a lesson into accountable work.
- Check the terminal output for the three learning sections.
- Check the terminal output for the action-item table.
Final sections missing?
Confirm the append command completed at the closing EOF line. Run the final chunk again if its headings are absent.
Select Help me finish the post-mortem template..
✔️ Awesome, the template is complete
The reusable post-mortem template is installed. Keep it unchanged for future drills.
ⓧ I'd like to double check the full code
# Chaos Incident Post-Mortem
Status: Draft
Incident ID:
Date and time UTC:
Author:
## Summary
Describe what failed and how the incident was resolved in two or three sentences.
## Impact
- Affected endpoint or resource:
- User-visible symptom:
- Start time:
- Mitigation time:
- End time:
- Duration:
## Detection
Describe the first symptom and the command or observation that detected it.
## Five-Stage Triage
### 1. Observe symptoms
Record observations without naming a cause.
### 2. Isolate the layer
State which layer you tested and which alternatives you ruled out.
### 3. Form a hypothesis with CLI evidence
Hypothesis:
Evidence that supports it:
Evidence that could disprove it:
### 4. Mitigate
Record the exact mitigation, its timestamp, and the recovery evidence.
### 5. Compare with ground truth
- Predicted module:
- Revealed module:
- Match or mismatch:
- Evidence you missed or interpreted incorrectly:
## Timeline
| Time UTC | Observation or action | Evidence |
|---|---|---|
| | | |
## Trigger and Root Cause
Trigger:
Root cause or contributing causes:
## What Went Well
-
## What Went Poorly
-
## Where We Got Lucky
-
## Action Items
| Action | Type | Priority | Owner | Verifiable completion condition |
|---|---|---|---|---|
| | Detect, prevent, mitigate, or process | | | |Template reference
This is the complete reusable template. Your personal report fills every section with evidence from the blind incident.
- Copy the reusable template into your home directory by running:
cp /var/lib/chaos-lab/postmortem-template.md ~/chaos-postmortem.mdWhy keep two copies?
The file under /var/lib/chaos-lab remains clean for later drills. The file at ~/chaos-postmortem.md becomes the completed record for this incident.
- Draft the completed report with every heading from the template.
- Include the observed impact in the report.
- Include a timestamped evidence trail in the report.
- Include the mitigation command in the report.
- Include the predicted module beside the revealed module in the report.
- Include at least one measurable action item in the report.
- Write the completed report through a shell here-document by filling the report field below:
cat > ~/chaos-postmortem.md <<'EOF'
[[YOUR_COMPLETED_POSTMORTEM="paste your completed Markdown post-mortem"]]
EOFWhat does this write?
The here-document replaces the copied draft with your completed Markdown report. Quoting the delimiter preserves the report text as entered.
Report still contains template prompts?
Fill every section before running the here-document again. Keep the predicted module even when it differs from the revealed module.
Select Help me complete a blameless post-mortem..
- Display the completed report by running:
cat ~/chaos-postmortem.mdWhat should the report show?
You should see completed impact, detection, triage, timeline, root-cause, learning, and action-item sections. The ground-truth comparison should preserve both your original prediction and the revealed module.
Before the final check, do you expect the timer to be scheduled while the runner remains inactive?
- Resume scheduled drills and verify the final lab state by running:
sudo systemctl start chaos-runner.timer
sudo systemctl list-timers
sudo /usr/local/sbin/chaos_runner.sh status
df -h /var/lib/chaos-lab/sandbox
ip netns listWhat confirms the final state?
- The timer list should contain chaos-runner.timer with a future activation.
- The runner should report STATE=inactive.
- The sandbox should no longer identify a dedicated tmpfs mount.
- The namespace list should not contain chaos-lab.
Strong work. Your timer is ready for future surprise drills while this incident has been cleaned up and documented.
Secret mission
Diagnose a Stacked Incident
A convincing first diagnosis can hide another active fault. Install a bounded stacked profile, gather independent evidence for CPU pressure and inode exhaustion, then expand your post-mortem without discarding your first valid finding.
Clean Up Your Resources
Clean Up Your Resources
Your existing disposable RHEL VM adds no service fee. The systemd timer remains enabled across reboots, so choose whether to resume drills, pause the lab, or remove every project artifact.
Resources you used:
- Two executable Bash scripts at /usr/local/sbin/chaos_runner.sh and /usr/local/sbin/chaos_stacked.sh.
- The chaos-runner.service service unit and enabled chaos-runner.timer timer unit under /etc/systemd/system/.
- Lab state under /var/lib/chaos-lab, including the ground truth, post-mortem template, and empty sandbox directories.
- Transient runtime state under /run/chaos-lab.
- Your completed incident report at ~/chaos-postmortem.md.
Keep everything running
No files need changing. Choose this if you want to continue surprise drills on the disposable VM.
Resuming surprise faults can feel risky. The duration limits, isolated resources, cleanup traps, kill switch, and VM snapshot still contain the blast radius.
- Resume scheduled drills by running this command:
sudo systemctl start chaos-runner.timerWhat Does This Command Do?
The command activates the existing timer without recreating its unit file. The timer can now schedule another bounded incident.
- Confirm the timer has a future activation by running this command:
systemctl list-timersWhat Should I See?
You should see chaos-runner.timer with its next activation time. This confirms that surprise drills are scheduled again.
Your completed report remains at ~/chaos-postmortem.md for future comparisons.
- Keep the VM snapshot available as your recovery point.
Pause - I'll come back to this later
Stop scheduled incidents while preserving every script, unit file, lab record, and report. The disabled-file kill switch also blocks accidental manual incidents when you return to the VM.
- Pause the chaos lab by running these commands:
sudo systemctl disable --now chaos-runner.timer
sudo systemctl stop chaos-runner.service
sudo touch /var/lib/chaos-lab/DISABLEDHow Is the Lab Paused?
- The first command disables the timer across future boots. It also stops the current timer.
- The second command stops an active incident through the service recovery path.
- The final command creates the kill switch checked by chaos_runner.sh.
- Verify the paused state by running these commands:
systemctl list-timers
sudo /usr/local/sbin/chaos_runner.sh statusWhat Should I See?
You should see no active chaos-runner.timer entry. The runner should report STATE=inactive.
Your scripts, incident evidence, template, and completed report remain ready for another session.
Delete - I don't want to use this again
Remove all project resources from the disposable VM. Deletion is permanent, so preserve ~/chaos-postmortem.md first if you want to keep your evidence-backed report.
- Copy ~/chaos-postmortem.md to your computer with the file-transfer feature provided by your VM platform.
- Stop automation before deleting files by running these commands:
sudo systemctl disable --now chaos-runner.timer
sudo systemctl stop chaos-runner.service
sudo /usr/local/sbin/chaos_runner.sh cleanupWhy Run Cleanup First?
The cleanup action removes any remaining mount, network namespace, virtual interface, packet-loss rule, lock, and active-state file. Stopping the service first prevents a new incident from competing with deletion.
- Remove the scripts, unit files, lab directories, and VM copy of the report by running these commands:
sudo rm -f /usr/local/sbin/chaos_runner.sh /usr/local/sbin/chaos_stacked.sh
sudo rm -f /etc/systemd/system/chaos-runner.service /etc/systemd/system/chaos-runner.timer
sudo systemctl daemon-reload
sudo rm -rf /var/lib/chaos-lab /run/chaos-lab
rm -f ~/chaos-postmortem.mdWhat Gets Removed?
These commands delete both incident scripts and both systemd unit files. Reloading systemd removes the deleted unit definitions from the manager.
The final commands remove the ground truth, template, empty sandboxes, transient state, kill switch, and completed report from the VM.
- Verify that every persistent project path is gone by running this check:
# Verify every persistent project path is gone.
if [[ ! -e /usr/local/sbin/chaos_runner.sh &&
! -e /usr/local/sbin/chaos_stacked.sh &&
! -e /etc/systemd/system/chaos-runner.service &&
! -e /etc/systemd/system/chaos-runner.timer &&
! -e /var/lib/chaos-lab &&
! -e /run/chaos-lab &&
! -e "$HOME/chaos-postmortem.md" ]]
then
printf 'All chaos-lab resources removed.\n'
else
printf 'One or more chaos-lab resources remain.\n'
fiWhat Should I See?
You should see All chaos-lab resources removed. This confirms that the scripts, automation, lab state, and report no longer exist on the VM.
Does a Resource Remain?
Run the removal block again if the check reports a remaining resource. Confirm that your shell still has sudo authorization if a protected path could not be deleted.
Use this prompt for help: Help me identify which chaos-lab path remains and remove it safely from my disposable RHEL VM.
Nice Work!
Nice Work!
You did it! You've built a safety-bounded chaos engine on RHEL 10.2 that reproduces real Linux failure symptoms inside a disposable VM. You also proved each diagnosis against sealed ground truth in an evidence-backed engineering report.
You've learned how to:
- Built four bounded incident modules behind one Bash dispatcher. Protected every run with automatic expiry plus idempotent cleanup. Added a systemd recovery path.
- Used bounded tmpfs mounts to reproduce disk-space exhaustion. Reused the same sandbox to expose inode exhaustion without consuming the VM's root filesystem. Confined packet loss to a network namespace so the management path stayed available.
- Scheduled randomized incidents with systemd timers. Recorded sealed ground truth through journald. Completed a five-stage incident triage. Turned the evidence into a blameless post-mortem with measurable action items.
- Completed the optional Secret Mission by diagnosing a stacked incident that combined CPU pressure with inode exhaustion. Preserved both causes in the second-cause correction to ~/chaos-postmortem.md.
Ready to quiz yourself?