Recover a Nested KVM Workload on AWS

Recover a nested KVM guest on AWS with libvirt networking and snapshots.

Introduction

30 Second Summary

Everything can look healthy from the outside while a machine buried inside another machine has quietly lost its connection. Fast recovery starts with finding the exact layer that failed.

In this project, you will build a nested KVM recovery lab on Amazon EC2. You will use libvirt to recover a planned virtual network outage before proving that a snapshot can restore deleted guest state.

What You'll Build

Your demo shows an Ubuntu guest back online after a planned outage with a deleted file restored through snapshot rollback.

By the end of this project, you'll have:

A nested Ubuntu guest you can bring to a login prompt through KVM. A repeatable virsh workflow lets you inspect its state. You can also stop or start the guest.
Network recovery evidence showing the default libvirt network returning to active status. The guest receives an IPv4 lease again.
Snapshot rollback proof showing /var/tmp/recovery-marker returning after you remove it from the guest.
Secret Mission: Run a timed network-outage recovery drill in five minutes or less without following the guided instructions.

What do I need for this project?

You need an AWS account with permission to launch and terminate instances within the project's hard $10 cap.

You also need an Apple silicon Mac with 16 GB or more RAM.

Before We Start

This checkpoint locks in the purpose of your disposable nested KVM recovery lab before the hands-on work begins. The lab prepares you for a virtualization systems role through practice in VM lifecycle operations, virtual networking, troubleshooting, and recovery.

Launch the Nested Virtualization Host

Your recovery lab needs an Intel host that can expose virtualization support to KVM. Nested virtualization on Amazon EC2 gives the future guest a supported three-layer foundation.

This step creates an Ubuntu Server 24.04 LTS host on a c7i-flex.xlarge instance. A restricted security rule keeps SSH access tied to your current public IP address.

In this step, get ready to:

Model the one-hour host cost before launching any resources.
Launch an Ubuntu Server host with nested virtualization enabled.
Protect the private key before connecting from your Mac.

Why KVM for this lab?

AWS supports two L1 hypervisors for nested virtualization: KVM or Hyper-V.

KVM keeps this lab centered on Linux administration. It also gives you direct practice with terminal operations and virtual networking.

Estimate the one-hour cost

The AWS Pricing Calculator models the compute and storage charges before they begin. This billing checkpoint gives the lab a hard $10 ceiling.

Open the AWS Pricing Calculator in your browser.
Set Location type to Region.
Choose US East (N. Virginia).
Keep the tenancy at Shared Instances.
Keep the operating system at Linux.
Confirm the estimate includes one EC2 instance.
Search for c7i-flex.xlarge in the instance specifications.
Select On-Demand usage.

The estimate now models the same compute profile that hosts your nested lab.

Set the usage duration to one hour.
Set the Storage volume to gp3.
Enter 20 in Storage amount.
Save the estimate.

Before you read the total, do you expect the one-hour lab plan to fit below the cap?

Compare the estimated one-hour total with the hard $10 limit.
Stop here if the total exceeds $10.

A total of $10 or less clears the billing gate. Good. Your one-hour plan now has a firm spending boundary before any resource exists.

Launch the configured EC2 host

The AWS Management Console turns your estimate into the L1 host for the lab. Each launch setting controls a specific part of its compute capacity or access boundary.

Sign in to the AWS Management Console with your AWS account.
Open Amazon EC2 using the console search.
Set the console Region to US East (N. Virginia).
Start a new instance launch from the EC2 console.

The machine image determines the operating system that boots on your host. Canonical's owner ID lets you distinguish the official Ubuntu image from similarly named images.

Select the official Canonical Ubuntu Server 24.04 LTS AMD64 image under Application and OS Images (Amazon Machine Image).
Confirm the image publisher is Canonical.
Confirm the owner account is 099720109477.

The Canonical publisher and owner account confirm that you selected the official Ubuntu image.

Select c7i-flex.xlarge under Instance type.
Choose Create new key pair under Key pair (login).
Enter kvm-lab as the key pair name.
Select the .pem private key format.
Finish creating the key pair.

Your browser downloads the private key as kvm-lab.pem. This credential stays on your Mac throughout the lab.

Create a new security group under Network settings.
Set the inbound SSH source to your current public IP address.
Enter 20 GiB for the root volume under Configure storage.
Select gp3 as the root volume type.

The security group now limits administrative access to your current internet connection. The EBS root volume provides 20 GiB for the host software and nested guest artifacts.

Expand Advanced details.
Choose Enable for Nested virtualization.

What does nested virtualization enable?

AWS Nitro operates at L0. Your EC2 host runs at L1. The Ubuntu guest you create later runs at L2.

Enabling nested virtualization lets the L1 host expose virtualization support to that L2 guest.

Launching starts EC2 compute and EBS storage billing. Your saved one-hour estimate keeps this lab within the hard cap.

Review the launch summary for the selected image and instance type.
Launch the EC2 instance.

Expect the two EC2 status checks to take a few minutes. The host is booting during this wait.

Select the new instance in the EC2 instance list.
Wait until the instance state shows running.
Wait until both EC2 status checks pass.
Record the listed public DNS value as instance public DNS name.

Protect the key and connect from macOS

The downloaded .pem file identifies you to the EC2 host. OpenSSH rejects a private key when other users can read it.

Press Cmd+Space on your Mac to open the search bar.
Type Terminal into the search bar.
Press Enter to open macOS Terminal.

Terminal is ready to protect the key before it is used for authentication.

Locate the downloaded kvm-lab.pem file on your Mac.
Record its full path as path to kvm-lab.pem.
Copy the command below into Terminal.
Replace /path/kvm-lab.pem with path to kvm-lab.pem.
Protect the private key by running the edited command:

chmod 0400 /path/kvm-lab.pem

What does chmod 0400 do?

The 0400 permission leaves the file readable only by its owner. OpenSSH can now use the key without exposing it to other local users.

A successful permission change returns you to the Terminal prompt without an error.

Key permissions still rejected?

Confirm that the path points to the downloaded kvm-lab.pem file. Check that the filename has not gained an extra extension.

Ask for help with fixing the permissions on my EC2 private key.

On the first connection, OpenSSH presents the host key fingerprint for confirmation. Confirming it adds the new host to your local known-hosts file.

Copy the SSH command below into Terminal.
Replace /path/kvm-lab.pem with path to kvm-lab.pem.
Replace instance-public-dns-name with instance public DNS name.
Type yes if OpenSSH asks you to accept the new host key.

Before you connect, do you expect the prompt to remain on your Mac or move to the Ubuntu host?

Connect to the Ubuntu host by running the edited command:

ssh -i /path/kvm-lab.pem ubuntu@instance-public-dns-name

What does this SSH command do?

The -i option tells OpenSSH which private key identifies you. The ubuntu value selects the default user for the Ubuntu EC2 image.

The public DNS name directs the encrypted connection to your running host.

Your Terminal prompt changes to the Ubuntu EC2 host. That change proves the instance is running and reachable through the restricted SSH rule.

Unable to reach the Ubuntu host?

Confirm that both EC2 status checks passed. Check that the security group allows SSH from your current public IP address.

Confirm that the command uses the current public DNS name. Check that the username remains ubuntu.

Ask for help with diagnosing my EC2 SSH connection.

✔️ My host is ready

Your Mac now has an authenticated SSH session to the nested-virtualization-capable Ubuntu host.

ⓧ I'd like to double check the configuration

One-hour estimate in US East (N. Virginia) for a Shared Linux On-Demand c7i-flex.xlarge instance and 20 GiB of EBS. You confirmed the estimate is $10 or less before launch.
Running Amazon EC2 c7i-flex.xlarge instance using the official Canonical Ubuntu Server 24.04 LTS AMD64 image owned by Canonical account 099720109477. Nested virtualization is enabled. Both EC2 status checks passed. The host is reachable as user ubuntu over SSH.
Key pair named kvm-lab in PEM format. The private key is stored on your Mac. It is protected with chmod 0400.
A 20 GiB gp3 root EBS volume is attached to the EC2 instance.
The security group allows SSH access only from your current public IP address.

Your protected Ubuntu host is online and accepting administrative connections. Next, you will install the virtualization stack and launch its first nested guest.

Install KVM and Launch the Guest

Your Amazon EC2 host is running with nested virtualization enabled. It still needs a virtualization layer before it can run an Ubuntu guest.

In this step, KVM supplies hardware acceleration. QEMU provides the virtual machine process.

Libvirt manages the domain lifecycle plus its virtual network. The virt-install command combines these pieces into a managed guest.

In this step, get ready to:

Install the pinned virtualization packages.
Download the Ubuntu cloud image.
Launch the nested guest to reach its serial login prompt.

Install the pinned virtualization stack

Pinned package revisions give your lab a reproducible starting point. An installed-package check tells you which setup path your host needs.

Check the installed virtualization package versions by running this command:

apt list --installed qemu-system-x86 libvirt-daemon-system libvirt-clients virtinst

What Does This Check Show?

The --installed filter reads the packages already present on the host. Compare the displayed versions with these required pins.

Use qemu-system-x86=1:8.2.2+ds-0ubuntu1.18 for the QEMU system emulator.
Use libvirt-daemon-system=10.0.0-2ubuntu8.16 for the libvirt system service.
Use libvirt-clients=10.0.0-2ubuntu8.16 for the management commands.
Use virtinst=1:4.1.0-3ubuntu0.1 for the guest installation tools.

Choose the tab that matches the package list on your host.

✔️ I See the Exact Package Versions

Your host already has every required package revision. The virtualization stack is ready for the Ubuntu cloud image.

Continue to the cloud image download below.

ⓧ I See Different Package Versions

Different revisions can produce different behavior during guest creation. Installing the pinned revisions gives this lab the expected package state.

The download can take a few minutes while Apt retrieves the packages. A steady stream of package messages means the installation is progressing.

Refresh the Apt metadata plus install the exact package revisions by running these commands:

sudo apt update
sudo apt install qemu-system-x86=1:8.2.2+ds-0ubuntu1.18 libvirt-daemon-system=10.0.0-2ubuntu8.16 libvirt-clients=10.0.0-2ubuntu8.16 virtinst=1:4.1.0-3ubuntu0.1

What Do These Commands Do?

The first command refreshes the host's package metadata. The second command requests each virtualization package at its pinned revision.

The libvirt system package provides the background service. The client package provides management tools such as virsh.

Confirm the installed revisions by running the package check again:

apt list --installed qemu-system-x86 libvirt-daemon-system libvirt-clients virtinst

What Should the Package List Confirm?

Each requested package should now appear in the output. Its installed revision should match the corresponding pin listed above.

That is the package layer complete. Your host now has the exact virtualization stack used throughout the lab.

Package Revision Unavailable?

Confirm that the host is running Ubuntu Server 24.04 LTS AMD64. A different Ubuntu release can expose a different package set.

Refresh the Apt metadata again if the package index was temporarily unavailable.

Help me diagnose an unavailable pinned Ubuntu virtualization package.

ⓧ One or More Packages Are Missing

A missing package means the EC2 host has not received the complete virtualization stack. The pinned installation supplies the emulator plus the management layer.

Expect the installation to take a few minutes while Apt downloads dependencies. The command may print many package names during this process.

Refresh the Apt metadata plus install the missing virtualization stack by running these commands:

sudo apt update
sudo apt install qemu-system-x86=1:8.2.2+ds-0ubuntu1.18 libvirt-daemon-system=10.0.0-2ubuntu8.16 libvirt-clients=10.0.0-2ubuntu8.16 virtinst=1:4.1.0-3ubuntu0.1

What Do These Commands Install?

Apt first refreshes its package metadata. It then installs QEMU plus the libvirt service.

The same installation adds the libvirt client commands. It also adds virt-install through the pinned virtinst package.

Verify that all four packages are installed by running:

apt list --installed qemu-system-x86 libvirt-daemon-system libvirt-clients virtinst

What Should the Package List Confirm?

The output should now contain one installed entry for each requested package. Each revision should match its required pin.

Good progress. Your EC2 host now has the virtualization tools needed to create a nested guest.

Installation Still Failing?

Check that the SSH session still has internet access to Ubuntu package sources. A stalled metadata refresh can prevent Apt from finding the requested revisions.

Confirm that your package check names all four packages exactly as shown.

Help me troubleshoot the pinned virtualization package installation.

Download the Ubuntu cloud image

A prepared QCOW cloud image gets the nested guest to a bootable state without an interactive ISO installation. Canonical distributes this image with an .img extension for QEMU plus libvirt.

Why Use a Cloud Image?

An ISO installation requires a longer operating-system setup before you can test the virtualization layer. The prepared cloud image lets virt-install import a bootable disk directly.

Download the current Ubuntu 24.04 LTS AMD64 cloud image by running:

wget https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img

What Does This Download Create?

The command retrieves Canonical's current Noble AMD64 cloud image. It stores the disk as noble-server-cloudimg-amd64.img in the current directory on your EC2 host.

The transfer progress reaches completion when the image finishes downloading. The guest disk is now ready for libvirt.

Inspect the libvirt management layer plus its virtual networks by running these commands:

virsh version
virsh net-list --all

What Do These Checks Prove?

The version check confirms that virsh can communicate with the libvirt management layer.
The network check includes inactive definitions in its output.
The default network supplies NAT connectivity for the nested guest.
The same network provides DHCP leases for guest addresses.

You should see libvirt version information. You should also see the default network listed as active.

Default Network Missing or Inactive?

Confirm that the pinned libvirt system package completed its installation. An interrupted package setup can leave the service or its default network unavailable.

Run the installed-package check from the previous substep before repeating these two checks.

Help me diagnose an unavailable libvirt default network.

Launch the guest and reach its console

The virt-install command registers the cloud image as a libvirt domain. It assigns 2048 MiB of memory plus 2 virtual CPUs.

The plain --cloud-init option generates a root password during a ten-second pause. The short pause gives you one chance to store that credential safely.

Prepare a secure password-manager entry for the generated root password.
Launch the nested Ubuntu guest by running this command:

virt-install --name ubuntu-vm-name --noautoconsole --import --memory 2048 --vcpus=2 --osinfo generic --disk bus=virtio,path=noble-server-cloudimg-amd64.img --network default --cloud-init

What Does This Command Build?

The --name option registers a domain named ubuntu-vm-name.
The --import option boots from the existing cloud image.
The disk uses the virtio bus for its virtual device.
The guest interface attaches to the default libvirt network.
The --noautoconsole option leaves you at the host prompt after creation.

Record the generated root password in your prepared password-manager entry during the ten-second pause.

The command defines the domain before starting it. Your cloud image now backs a managed nested virtual machine.

Guest Creation Failed?

Confirm that noble-server-cloudimg-amd64.img finished downloading in the current directory. An incomplete image cannot become the guest disk.

Check that the default network was active in the previous substep.

Help me debug the virt-install guest creation command.

Confirm that the new domain is running by running:

virsh list

What Does the Domain List Confirm?

The command lists active libvirt domains. A row for ubuntu-vm-name confirms that the nested guest is running.

You should see ubuntu-vm-name in the running domain list. The EC2 host is now operating an L2 guest through KVM.

Before you connect, do you expect the nested guest to have finished booting?

Connect to the guest's serial console by running:

virsh console ubuntu-vm-name

What Does the Console Connection Prove?

The serial console gives you direct access to the nested guest through libvirt. Reaching the Ubuntu login prompt proves that the imported image booted successfully.

You should reach the Ubuntu guest login prompt. That is the key result: your nested KVM workload is live inside the EC2 host.

Leave the guest console by pressing Ctrl-].

No Guest Login Prompt?

Wait for the cloud image to finish its first boot before pressing Enter once.
Confirm that ubuntu-vm-name still appears as running in the domain list.
Leave a stuck console with Ctrl-] before reconnecting.

Help me reach the ubuntu-vm-name serial login prompt.

Your nested Ubuntu guest now boots under KVM through libvirt. Next, you will operate its lifecycle before triggering a controlled virtual-network outage.

Operate the Guest and Trigger an Outage

That is a major milestone: your nested Ubuntu guest now reaches a login prompt through libvirt on Amazon EC2. A login prompt alone does not prove that you can inspect its dependencies or manage its lifecycle.

Routine lifecycle commands are only part of a virtualization administrator's job. This step ends with a controlled service-impacting fault that creates the incident for the next diagnosis.

In this step, get ready to:

Capture a healthy baseline for the guest domain.
Exercise the guest shutdown and startup lifecycle.
Trigger a planned outage in the default virtual network.

Inspect the healthy guest baseline

A healthy baseline gives you evidence to compare against the upcoming fault. The domain state proves the guest is alive. The virtual interface plus its DHCP lease prove the network path is healthy.

Capture the domain, interface, and address baseline by running these commands:

virsh dominfo ubuntu-vm-name
virsh domiflist ubuntu-vm-name
virsh domifaddr ubuntu-vm-name --source lease

What do these checks prove?

The domain information confirms that ubuntu-vm-name is running.
The interface list confirms that the guest has a virtual NIC connected to default.
The address check reads the network's lease data. You should see an IPv4 address for the guest.

Guest address missing?

Make sure you are at the EC2 host shell instead of inside the guest console.
Wait briefly for the guest's DHCP lease to appear.
Run the three baseline checks again.

Help me diagnose why the running libvirt guest has no DHCP lease.

Cycle the guest lifecycle

A graceful shutdown asks the guest operating system to close its services cleanly. Starting the defined domain again proves that you can control its lifecycle without rebuilding it.

Request a graceful shutdown by running this command:

virsh shutdown ubuntu-vm-name

What does shutdown change?

The command sends a graceful shutdown request to ubuntu-vm-name.
The request can take a variable amount of time while the guest closes its services.
The domain definition remains available after the guest powers off.

Wait for the guest to complete the graceful shutdown.
Confirm the powered-off state by running:

virsh list --all

How does the status check help?

The --all option includes active and inactive domains. You should see ubuntu-vm-name with a shut off state.

Guest still running?

Wait another moment for the guest operating system to finish shutting down.
Run the domain status check again.

Help me diagnose why my libvirt guest is not completing a graceful shutdown.

Start the defined guest again by running this command:

virsh start ubuntu-vm-name

What does start change?

The start command powers on the existing domain definition. It reuses the guest's current virtual disk, memory allocation, CPU allocation, and network interface configuration.

Confirm that the guest returned to service by running:

virsh list

What should you see?

The running-domain list should include ubuntu-vm-name. Its state should be running again.

Guest not returning to running?

Check that the start command used the exact domain name ubuntu-vm-name.
Review the start command's output for a domain or storage problem.

Help me diagnose why my libvirt guest does not return to the running state.

Trigger the planned network outage

That confirms your lifecycle control is working. The default virtual network is a separate dependency beneath the guest domain.

A controlled incident lets you test whether the network and domain fail together. You are about to interrupt the network layer while leaving the domain configuration untouched.

Before you run the outage command, predict whether stopping the network also stops the guest domain.
Trigger the planned outage by running this command:

virsh net-destroy default

What did the outage command change?

The command immediately stops the default virtual network. It leaves the network definition available for later administration.

The command does not remove the guest's virtual NIC definition. Your verification now reveals whether the guest domain kept running.

The planned outage is now active. Your next check reveals how far the fault spread.

Before you check, predict the network, domain, interface, and lease states you expect to see.
Verify the full impact of the incident by running these commands:

virsh net-list --all
virsh list
virsh domiflist ubuntu-vm-name
virsh domifaddr ubuntu-vm-name --source lease

How do these checks isolate the fault?

The network list shows whether default is active.
The domain list shows whether ubuntu-vm-name survived the network outage.
The interface list confirms whether the virtual NIC definition still points to default.
The lease check reveals whether the guest still has an address from the stopped network.

The network list shows default as inactive. The domain list still shows ubuntu-vm-name as running.

The interface list still includes the virtual NIC on default. The lease check no longer returns the guest's IPv4 address.

Network still active?

Confirm that the outage command targeted the network named default.
Confirm that you ran the command from the EC2 host shell.
Repeat the incident verification commands after the outage command succeeds.

Help me diagnose why the default libvirt network still appears active after the planned outage command.

You now have a clean incident to diagnose: the guest is running while its network is inactive. Next, you'll isolate the failed layer before restoring connectivity.

Diagnose and Recover the Network

Your nested Ubuntu guest stayed running after the default network stopped. The incident left its libvirt network dependency unavailable.

You now need to separate domain health from network health with virsh. That diagnosis lets you restore only the failed layer.

In this step, get ready to:

Isolate the outage to the inactive default network.
Restore the failed network without restarting ubuntu-vm-name.
Confirm the recovered IPv4 DHCP lease.

Diagnose the failed layer

Each check targets a separate layer of the virtualization stack. Their combined output shows where service stopped.

Before you run these checks, predict whether the domain or its network dependency will appear unhealthy.

Compare the three layers from the Ubuntu host's existing SSH session by running these commands:

virsh list
virsh net-list --all
virsh domiflist ubuntu-vm-name

What Do These Checks Prove?

The first command confirms that ubuntu-vm-name remains running.
The second command includes active and inactive networks. It exposes default as the unavailable dependency.
The third command confirms that the guest still has a virtual NIC tied to default.

You should see ubuntu-vm-name running. You should see default inactive.

The guest's virtual NIC still references default. You have isolated the fault to the virtual network layer.

Results Don't Match the Incident?

Run the checks from the Ubuntu host's existing SSH session. Commands issued on your Mac do not inspect the remote virtualization state.
Confirm that the domain name is exactly ubuntu-vm-name.
Confirm that the network name is exactly default.

Help me compare my domain, network, and virtual NIC results.

Restore the network dependency

An inactive NAT network can be started independently of its attached domains. Autostart also brings that network back after a future host boot.

Start default now and configure it for host boot by running these commands:

virsh net-start default
virsh net-autostart default

How Does the Repair Work?

The net-start subcommand activates the defined network immediately.
The net-autostart subcommand configures the network to start when the EC2 host boots.
Neither command restarts the running guest domain.

You should receive a success confirmation for each command. The running guest remains untouched.

That is a focused repair. You restored only the failed dependency.

Network Commands Not Succeeding?

Confirm that you are still using the Ubuntu host's existing SSH session.
Check that the first subcommand is net-start.
Check that the second subcommand is net-autostart.

Help me restore the default libvirt network without restarting my guest.

Verify the recovered lease

The active network confirms that the failed service has returned. The lease confirms that ubuntu-vm-name has an IPv4 address through that service.

Optionally observe the running guest directly by running this command:

virsh console ubuntu-vm-name

What Does the Console Confirm?

The serial console connects directly to the running guest. Reaching its login prompt confirms that the domain remained available during the network repair.

Leave the guest console by pressing Ctrl-].

You should return to the Ubuntu host shell.

Console Not Responding?

Press Enter once if the login prompt is not visible. Leave the console with Ctrl-] if you no longer need direct guest access.

Help me access or leave the serial console for ubuntu-vm-name.

Before the final check, predict whether the restored network has returned the guest's IPv4 lease.

Check the active network and guest lease by running these commands:

virsh net-list
virsh domifaddr ubuntu-vm-name --source lease

What Does the Final Check Prove?

The first command lists active virtual networks. It should now include default.
The second command reads the network's lease data for ubuntu-vm-name.
An IPv4 address confirms that the guest has recovered its network lease.

You should see default in the active network list. You should also see an IPv4 address for ubuntu-vm-name in the lease output.

That recovered lease is your proof that guest connectivity has returned. The domain stayed running throughout the repair.

Lease Still Missing?

Confirm that default appears in the active network list.
Give the running guest a moment to renew its lease.
Run the final two checks again from the Ubuntu host's existing SSH session.

Help me diagnose why ubuntu-vm-name has no DHCP lease after the default network restarted.

The outage is resolved without a guest restart. Next, you will create a recovery point that can reverse a destructive change inside ubuntu-vm-name.

Create and Prove a Recovery Point

Your recovered libvirt network is active again. The Ubuntu guest has recovered its IPv4 DHCP lease.

Network repair restores availability. Disaster recovery also requires proof that a known-good machine state can undo a destructive change.

An internal snapshot captures the guest at a point in time. A marker file gives you visible evidence that rollback restored the captured state.

In this step, get ready to:

Place a recovery marker inside the running guest.
Capture an internal snapshot named clean-state.
Delete the marker before restoring it through snapshot rollback.

Prepare the guest state

The snapshot needs a clear piece of guest state to preserve. An empty marker file provides a simple recovery test with an unambiguous result.

Connect to the guest serial console by running this command:

virsh console ubuntu-vm-name

What does this command do?

The console subcommand attaches your current terminal to the running guest. This gives you direct access through its serial console.

Keep the generated root password private during this guest login.
Enter root at the guest login prompt.
Enter the generated root password at the password prompt.

You will see the guest shell prompt after the login succeeds.

Create the recovery marker inside the guest by running this command:

touch /var/tmp/recovery-marker

What does this command do?

The touch command creates an empty file at /var/tmp/recovery-marker. The file becomes proof of the guest state captured by the snapshot.

Confirm that the marker exists by running this command inside the guest:

ls /var/tmp/recovery-marker

What does this command show?

The ls command checks the exact marker path. A printed path confirms that the file exists before the snapshot.

You have your proof point. The marker path now appears inside the guest.

Leave the guest console by pressing Ctrl-].

Having trouble with the guest console?

Press Enter once if the console looks blank.
Use the generated root password recorded when the guest was created.
Reconnect with the console command above if you detached before creating the marker.

Ask for help if the serial console still blocks the marker setup: help me diagnose why I cannot log in to ubuntu-vm-name through virsh console

Turn the marker into a recovery point

The marker only becomes recoverable after the domain captures it. Naming the snapshot clean-state gives you a clear rollback target.

Create the internal snapshot by running this command on the EC2 host:

virsh snapshot-create-as ubuntu-vm-name clean-state

What does this command do?

The snapshot-create-as subcommand captures the domain state under the name clean-state. The captured disk state includes the recovery marker.

The snapshot also preserves memory state. It preserves device state for the recovery point.

Predict which snapshot name the domain should report.
List the available snapshots by running this command:

virsh snapshot-list ubuntu-vm-name

What does this command show?

The snapshot-list subcommand displays the recovery points available for ubuntu-vm-name. This confirms that libvirt recorded the snapshot metadata.

You will see clean-state in the snapshot list. Your guest now has a named recovery point.

Snapshot missing from the list?

Confirm that the domain name is ubuntu-vm-name.
Confirm that the snapshot name is clean-state.
Run the snapshot creation command above again if the first attempt did not complete.

Ask for help if the snapshot still does not appear: help me diagnose why clean-state is missing from the snapshot list

Prove the rollback

A recovery point matters only when it restores state after damage. The marker deletion creates a controlled destructive change inside the guest.

Return to the guest serial console by running this command:

virsh console ubuntu-vm-name

What does this command do?

This reconnects your terminal to the running guest. The destructive test must happen inside the guest captured by clean-state.

Enter root at the guest login prompt.
Enter the generated root password at the password prompt.

You will see the guest shell prompt after the login succeeds.

Removing the marker is deliberate. This test affects only the empty file created for the recovery exercise.

Remove the recovery marker by running this command inside the guest:

rm /var/tmp/recovery-marker

What does this command do?

The rm command deletes /var/tmp/recovery-marker from the current guest state. The earlier snapshot still contains the version of the disk where the marker existed.

Predict whether the deleted marker path should still resolve.
Test the deleted marker path by running this command:

ls /var/tmp/recovery-marker

This failed check is intentional

The command cannot find the marker because the destructive change removed it. This visible failure proves that the guest has moved beyond the captured state.

Leave the guest console by pressing Ctrl-].

Snapshot reversion discards changes made after clean-state. In this controlled lab the only planned change is the marker deletion.

Restore the captured guest state by running this command on the EC2 host:

virsh snapshot-revert ubuntu-vm-name clean-state --running

What does this command do?

The snapshot-revert subcommand restores ubuntu-vm-name to the state captured by clean-state. The --running flag returns the restored domain to a running state.

Reconnect to the restored guest by running this command:

virsh console ubuntu-vm-name

Why reconnect after rollback?

The revert replaces the guest state that your earlier console session used. A fresh console connection lets you inspect the restored running domain.

Enter root at the guest login prompt.
Enter the generated root password at the password prompt.

You will see the guest shell prompt after the restored system accepts the login.

Before the final check, predict whether rollback restored the deleted path.
Verify the recovered marker by running this command inside the guest:

ls /var/tmp/recovery-marker

What proves the recovery worked?

The command reads the marker path from the restored guest disk. Seeing the path confirms that rollback reversed the deletion made after the snapshot.

You will see /var/tmp/recovery-marker printed in the console. That is your recovery proof: the known-good state survived a destructive test.

Marker still missing after rollback?

Confirm that you reverted to clean-state.
Confirm that the snapshot was created after the first successful marker check.
Reconnect to the guest console after completing the revert.

Ask for help if the restored path is still missing: help me diagnose why snapshot rollback did not restore the recovery marker

Secret mission

Run a Timed KVM Recovery Drill

Repeat the virtual network incident as a five-minute recovery drill. Diagnose the failed layer without restarting the guest. Capture the known-good, failed, and recovered states as incident evidence.

Clean Up Your Resources

Your Amazon EC2 host can keep generating AWS charges while it runs. Decide whether to keep the lab available, pause it for later, or delete it entirely.

Cost warning

The running EC2 host incurs compute charges. Its attached EBS volume incurs storage charges.

Nested virtualization has no additional AWS charge. Your regional EC2 rate remains dynamic.

Resources you used:

One running c7i-flex.xlarge EC2 host with nested virtualization enabled plus the installed KVM and libvirt stack.
One 20 GiB gp3 root volume containing the nested Ubuntu guest, the cloud image, the clean-state snapshot, and the default NAT network.
One EC2 security group allowing SSH from your current public IP address.
One EC2 key pair named kvm-lab plus the protected kvm-lab.pem private key on your Mac.

Keep everything running

No action is needed. Choose this if you plan to continue practising virtualization operations in the lab.

The EC2 host stays available for SSH connections.
EC2 compute charges continue while the host runs.
The root volume preserves ubuntu-vm-name plus its clean-state snapshot.
The security group keeps SSH restricted to your current public IP address.
The kvm-lab key pair remains available for host access.

Pause - I'll come back to this later

Pausing preserves the lab on EBS while EC2 compute is stopped. EBS storage charges continue.

Select your running c7i-flex.xlarge host in the Amazon EC2 console.
Stop the selected instance from its instance controls.

Your SSH session closes during shutdown. The EC2 console eventually shows the host in the stopped state.

Start the stopped EC2 host from the console when you return.
Reconnect from macOS Terminal using your protected kvm-lab.pem key.
Repeat the network and domain checks from your timed recovery drill.

Confirm the default network state before operating ubuntu-vm-name again.

Delete - I don't want to use this again

Termination is permanent. The EC2 host cannot be started again after deletion.

Select the lab host in the Amazon EC2 console.
Choose Instance state.
Choose Terminate (delete) instance.
Confirm with Terminate (delete).

You will see the instance reach the terminated state. The nested guest, cloud image, snapshot, and libvirt configuration on that host are no longer accessible.

Confirm that no intentionally retained 20 GiB gp3 root volume remains in the EBS volume inventory for us-east-1.
Delete the lab security group from the EC2 security group inventory after the instance reaches terminated.
Delete kvm-lab from the EC2 key pair inventory.
Use Finder search on your Mac to locate kvm-lab.pem.
Move kvm-lab.pem to Trash.
Empty Trash to remove the private key from your Mac.

That's the billable lab removed. The EC2 host and its EBS storage can no longer generate charges for this project.

Nice Work!

You made it! You built a three-layer nested virtualization lab on AWS.

You've learned how to:

Build a three-layer lab with an Amazon EC2 host at L1. Run a nested Ubuntu guest under KVM at L2.
Operate ubuntu-vm-name through a libvirt lifecycle workflow. Isolate the outage to the inactive default network while the guest remains running.
Create the clean-state internal snapshot before a destructive change. Revert ubuntu-vm-name to recover /var/tmp/recovery-marker.
Complete the Secret Mission by restoring the inactive default network in five minutes or less. Confirm the recovered DHCP lease without restarting ubuntu-vm-name. Capture incident evidence across the outage timeline.

Ready to quiz yourself?