Run a Service Inside a KVM Guest on AWS

Run an HTTP service in a nested KVM guest on Amazon EC2.

Introduction

30 Second Summary

A web page can load in seconds while the machine serving it stays completely hidden. That makes it hard to prove where the service runs or how its operating system is isolated.

In this project, you will build a nested virtualization lab on AWS that runs a status page inside an Amazon Linux 2023 guest managed by KVM. You will prove the guest works by retrieving its page from the Amazon EC2 host.

What You'll Build

Picture your EC2 host retrieving a page that declares KVM Guest Is Online from a nested guest on its private network.

By the end of this project, you'll have:

Proof that KVM acceleration is available through /dev/kvm on the L1 host.
A managed Amazon Linux guest whose lifecycle you can control with virsh.
A reachable guest service that returns its status page across the private NAT network.
Secret Mission: reduce the guest to one vCPU with 1 GiB of memory, then prove those limits from both sides of the virtualization boundary.

Are there any prerequisites?

You'll need an AWS account with administrator access plus quota for four Standard vCPUs in US East (N. Virginia). The project assumes your Windows computer already has the required local tools.

Before We Start

Before any infrastructure exists, this step gives you a clear reason for building the lab. You are committing to a service inside an L2 KVM guest that demonstrates how a full virtual machine runs its own kernel instead of sharing the L1 host kernel.

Launch the Nested-Virtualization Host

Your L2 guest needs an L1 host that exposes hardware-assisted virtualization. A standard Amazon EC2 launch does not expose KVM automatically.

This step creates the L1 host with its processor extensions enabled. You will secure its browser connection before preparing the virtualization stack.

In this step, get ready to:

Restrict browser-based SSH access to the regional AWS-managed prefix list.
Launch the L1 host with nested virtualization enabled.
Install the virtualization stack and verify KVM acceleration.

Create the restricted SSH security group

Amazon EC2 Instance Connect opens an SSH session in your browser. Its traffic comes from an AWS-managed prefix list for the selected region.

A security group controls which traffic can reach the host. This rule permits SSH from the EC2 Instance Connect service addresses in US East (N. Virginia).

Sign in to the AWS Management Console.
Select US East (N. Virginia) from the region menu.
Enter EC2 in the console search bar.
Select EC2 from the search results.
Choose Security Groups from the EC2 navigation panel.
Click Create security group.
Enter kvm-lab-eic in the Security group name field.
Enter kvm-lab-eic in the Description field.

Why Use the Managed Prefix List?

Browser-based EC2 Instance Connect sends SSH traffic from AWS-managed addresses. The regional prefix list keeps the rule aligned with those addresses.

This gives the host a narrow browser management path. Arbitrary internet addresses remain outside the rule.

Click Add rule in the inbound rules section.
Select SSH for Type.
Select Custom for Source.
Select com.amazonaws.us-east-1.ec2-instance-connect as the source prefix list.
Click Create security group.

Good progress. The new security group now accepts browser SSH sessions from EC2 Instance Connect.

Launch and connect to the L1 host

AWS infrastructure plus the Nitro hypervisor form L0. The EC2 instance becomes L1 when nested virtualization exposes processor extensions for an L2 guest.

Return to Instances in the EC2 navigation panel.
Click Launch instances.
Enter kvm-lab-host as the instance name.
Select Amazon Linux 2023 AMI as the machine image.
Select m8i.xlarge as the instance type.
Select Proceed without a key pair (Not recommended) from the key pair menu.

Why Launch Without a Key Pair?

Skipping a persistent key pair can feel risky. EC2 Instance Connect provides temporary SSH access through the browser.

The security group limits that connection to the AWS-managed service addresses. You do not need a separate host key file for this lab.

The root volume holds the host operating system plus the guest files created later. Deleting this volume with the instance prevents its storage from remaining after teardown.

Set the root volume size to 20 GiB.
Select gp3 as the root volume type.
Set Delete on termination to Yes.
Set Auto-assign public IP to Enable.
Choose Select existing security group.
Select kvm-lab-eic as the security group.

Nested virtualization is the setting that passes virtualization extensions into L1. That boundary allows the host operating system to expose the KVM device.

Expand Advanced details.
Set Nested virtualization to Enable.

Before You Launch

Launching starts an On-Demand EC2 instance with an attached EBS volume. Your AWS credits cover the project usage described here.

The final step terminates the instance. It also verifies that the root volume has been removed.

Click Launch instance.
Confirm the missing key pair choice if the console asks for confirmation.

Expect the instance to take a few minutes to become ready. The status checks show when AWS has finished preparing the host.

Return to Instances.
Wait for kvm-lab-host to show the Running state.
Wait for its status checks to pass.
Select kvm-lab-host from the instance list.
Click Connect.
Select the EC2 Instance Connect tab.
Select Connect using a Public IP.
Click Connect.

The browser terminal confirms that the public address plus the restricted security group provide a working path into L1. Your commands now run on kvm-lab-host.

Browser Terminal Not Connecting?

Confirm that the instance has a public IPv4 address. Confirm that its attached security group is kvm-lab-eic.

Check that the inbound SSH source is com.amazonaws.us-east-1.ec2-instance-connect.

Help me diagnose the browser connection.

Install and verify the virtualization stack

QEMU runs the virtual machine process. libvirt provides the management layer used by virsh.

The host also needs separate sockets for virtual machines plus virtual networking. The default NAT network gives L1 a private path to the guest you create later.

Package installation can take a few minutes while DNF downloads the virtualization stack. A short pause in the browser terminal is expected.

Install the core virtualization packages by running this command:

sudo dnf install -y qemu-kvm qemu-img libvirt libvirt-client edk2-ovmf

What Does This Install?

The QEMU packages provide the hypervisor process plus guest disk tooling.
The libvirt packages provide the management daemons plus the virsh client.
The firmware package provides guest boot firmware.

Review the DNF output for a completed package transaction.

Package Installation Failed?

Review the DNF output for a repository or network failure. Keep the browser terminal open so the failure details remain visible.

Help me diagnose the package installation.

The later NoCloud step needs an ISO creation tool. Amazon Linux provides xorriso through DNF.

Install xorriso by running this command:

sudo dnf install xorriso

Why Install Xorriso?

Xorriso can build ISO9660 images through its mkisofs-compatible interface. The guest provisioning step uses that capability to create first-boot media.

Approve the package transaction if DNF asks for confirmation.

The modular libvirt stack uses one socket for QEMU management. It uses another socket for the virtual network.

Enable the libvirt sockets plus the default NAT network by running these commands:

sudo systemctl enable --now virtqemud.socket
sudo systemctl enable --now virtnetworkd.socket
sudo virsh net-start default
sudo virsh net-autostart default

What Do These Commands Prepare?

The QEMU socket makes virtual machine management available to libvirt.
The network socket makes virtual network management available to libvirt.
The final commands activate the default NAT network plus its automatic startup setting.

If net-start reports network is already active, continue. That result confirms the required network state.

Default Network Did Not Start?

Check the preceding output from virtnetworkd.socket for a failure. The default network depends on that socket.

Help me diagnose the libvirt network.

Before you run the final checks, what evidence do you expect to see when hardware virtualization reaches the L1 operating system?

Verify the processor extension plus the KVM device plus the active hypervisor by running these commands:

grep -o -m1 -E 'vmx|svm' /proc/cpuinfo
ls -l /dev/kvm
sudo virsh version

What Do These Checks Prove?

The processor check confirms that a hardware virtualization extension reaches the instance.
The device check confirms that the Linux KVM interface exists.
The version check confirms that libvirt can communicate with the QEMU hypervisor.

✔️ I see QEMU 11.0.0

You should see vmx plus a character device at /dev/kvm. The version output should include Running hypervisor: QEMU 11.0.0.

That is the host boundary working. Your L1 instance can now run an accelerated L2 virtual machine.

ⓧ I see an older QEMU version

An older hypervisor version indicates that the host does not have the required Amazon Linux virtualization release. The project requires QEMU 11.0.0.

Return to the Amazon EC2 console.
Confirm that kvm-lab-host uses the current Amazon Linux 2023 AMI.
Repeat the package installation command after confirming the current AMI:

sudo dnf install -y qemu-kvm qemu-img libvirt libvirt-client edk2-ovmf

What Does This Command Refresh?

DNF installs the virtualization packages available from the host's configured Amazon Linux repositories. The current repository release supplies the required QEMU version.

Recheck the running hypervisor version by running:

sudo virsh version

What Should You See?

The version output should include Running hypervisor: QEMU 11.0.0.

ⓧ virsh is unavailable

The libvirt client supplies virsh. Reinstall the complete package set before repeating the version check.

Install the complete virtualization package set by running:

sudo dnf install -y qemu-kvm qemu-img libvirt libvirt-client edk2-ovmf

What Restores Virsh?

The libvirt-client package provides the command-line client. The remaining packages provide the hypervisor plus its management services.

Confirm that the client can reach QEMU by running:

sudo virsh version

What Should You See?

The output should identify the running hypervisor as QEMU 11.0.0.

Your nested-virtualization host is connected and ready for a guest. Next, you will boot an Amazon Linux image until it reaches an inaccessible login prompt.

Boot an Unprovisioned Guest

Your nested-virtualization host is ready. It exposes KVM acceleration through /dev/kvm.

You now need to prove that libvirt can boot a complete virtual machine from the official guest image. This step deliberately uses that image before any first-boot configuration is attached.

In this step, get ready to:

Download the pinned Amazon Linux 2023 guest image and validate its checksum.
Define a KVM domain with fixed compute resources and virtual networking.
Reach the guest login prompt through its serial console.

Download and validate the guest image

A QCOW2 image contains the guest operating system disk. Its checksum proves that the downloaded bytes match the pinned Amazon Linux 2023 release.

The image transfer may take a few minutes on a slower connection. A quiet progress meter during the download does not mean the browser terminal has frozen.

Prepare the validated guest disk in the browser terminal from earlier by running these commands:

curl -o al2023-kvm-2023.12.20261026.1-kernel-6.1-x86_64.xfs.gpt.qcow2 https://cdn.amazonlinux.com/al2023/os-images/2023.12.20261026.1/kvm/al2023-kvm-2023.12.20261026.1-kernel-6.1-x86_64.xfs.gpt.qcow2
echo "547b7e403b5b78aad57cdaf4c4adcb6815534fa50ce7ae43e2c14dc36d8b759f  al2023-kvm-2023.12.20261026.1-kernel-6.1-x86_64.xfs.gpt.qcow2" | sha256sum -c
sudo mv al2023-kvm-2023.12.20261026.1-kernel-6.1-x86_64.xfs.gpt.qcow2 /var/lib/libvirt/images/al2023-demo.qcow2

What do these commands do?

The first command downloads the pinned guest image into your home directory.
The second command compares the downloaded image with the official SHA-256 digest.
The third command moves the validated image to the location used by the domain definition.

You should see the transfer complete before the checksum reports the file as valid. That gives your guest a trusted operating system disk at /var/lib/libvirt/images/al2023-demo.qcow2.

Download or checksum failed?

A failed download usually points to a temporary network interruption. Run the three commands again after confirming that the browser terminal remains connected.

A checksum mismatch means the downloaded file is incomplete or different from the pinned image. Remove that download before trying again.

Help me diagnose the image download or checksum failure.

Define and start the KVM domain

A libvirt domain definition uses XML to describe the guest hardware. This definition assigns two virtual CPUs plus 2048 MiB of memory.

The fixed UUID keeps later definitions attached to the same guest identity. The fixed MAC address gives its virtual network interface a stable identity on the default NAT network.

Create al2023-demo.xml in your home directory by running this command:

vi al2023-demo.xml

What does this command do?

This opens al2023-demo.xml in the terminal editor. Saving the new buffer creates the file in your home directory.

Press i to enter insert mode.
Paste the opening domain configuration into al2023-demo.xml from the first line:

<domain type='kvm'>
  <name>al2023-demo</name>
  <uuid>6c984a40-7f9b-4e04-a56c-4eb362d04a33</uuid>
  <memory unit='MiB'>2048</memory>
  <currentMemory unit='MiB'>2048</currentMemory>
  <vcpu>2</vcpu>
  <os>
    <type arch='x86_64' machine='q35'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
  </features>
  <cpu mode='host-passthrough'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/al2023-demo.qcow2'/>
      <target dev='vda' bus='virtio'/>
    </disk>

What does this configuration set?

The resource settings expose two virtual CPUs plus 2048 MiB of memory to the guest.
The host-passthrough CPU mode lets the guest use the processor capabilities exposed to the EC2 host.
The disk block attaches /var/lib/libvirt/images/al2023-demo.qcow2 as the guest's virtio system disk.

Paste the remaining device configuration directly below </disk>.

    <interface type='network'>
      <mac address='52:54:00:4b:56:4d'/>
      <source network='default'/>
      <model type='virtio'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
  </devices>
</domain>

What do these devices provide?

The virtio interface connects the guest to libvirt's default private network.
The fixed MAC address keeps the guest network identity consistent across redefinitions.
The serial device gives virsh a terminal connection to the guest boot process.

Press Esc to return to command mode.
Type :wq.
Press Enter to save al2023-demo.xml.

✔️ Awesome, I've got everything!

Your saved domain definition now matches the unprovisioned guest configuration.

ⓧ I'd like to double check the full code

Compare your saved al2023-demo.xml with this complete reference file:

<domain type='kvm'>
  <name>al2023-demo</name>
  <uuid>6c984a40-7f9b-4e04-a56c-4eb362d04a33</uuid>
  <memory unit='MiB'>2048</memory>
  <currentMemory unit='MiB'>2048</currentMemory>
  <vcpu>2</vcpu>
  <os>
    <type arch='x86_64' machine='q35'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
  </features>
  <cpu mode='host-passthrough'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/al2023-demo.qcow2'/>
      <target dev='vda' bus='virtio'/>
    </disk>
    <interface type='network'>
      <mac address='52:54:00:4b:56:4d'/>
      <source network='default'/>
      <model type='virtio'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
  </devices>
</domain>

What should match?

The complete file contains one system disk plus one virtual network interface. It contains no CD-ROM device.

Build the running guest from the saved definition by running these commands:

sudo virsh define al2023-demo.xml
sudo virsh start al2023-demo
sudo virsh list --all

What do these commands do?

The first command registers the persistent domain definition with libvirt.
The second command boots the domain through KVM.
The third command displays every registered domain plus its current state.

You should see al2023-demo listed with an active state. Your L2 virtual machine is now running inside the L1 EC2 host.

Domain definition or startup failed?

Check that every opening XML element has a matching closing element. Confirm that the disk source points to /var/lib/libvirt/images/al2023-demo.qcow2.

Help me diagnose why the libvirt domain cannot be defined or started.

Inspect the inaccessible serial console

A serial console shows the guest boot process without depending on its virtual network. This is the clearest place to test whether the raw cloud image is ready for direct access.

Before you connect, do you expect the guest to be fully usable after reaching its login prompt?

Attach to the guest serial console by running this command:

sudo virsh console al2023-demo

What does this command do?

This command attaches your browser terminal to the guest's primary serial console. Detaching from that console leaves the virtual machine running.

You'll see Amazon Linux boot messages followed by a login prompt. That prompt proves that KVM booted the guest operating system successfully.

The image has no user password or SSH key by default. The running guest is therefore inaccessible even though its operating system reached the login screen.

Press Ctrl+] to detach from the serial console.

No login prompt in the console?

Allow the guest time to finish its first boot. If the console remains blank, confirm that al2023-demo still has an active state in the domain list.

Help me diagnose why the al2023-demo serial console does not reach a login prompt.

You have proved that the nested KVM stack can boot a complete L2 operating system. Next, you will provision the guest with NoCloud so it has an identity plus a usable SSH credential.

Provision the Guest with NoCloud

Your KVM guest already boots under libvirt. The unreachable login prompt proved that the virtualization layer works.

The failed login comes from missing first-boot configuration in the Amazon Linux 2023 image. A NoCloud seed gives cloud-init the guest identity plus your SSH public key.

In this step, get ready to:

Generate an SSH key pair for guest access.
Build a NoCloud seed containing the guest identity plus the status page.
Attach the seed to the existing domain before verifying SSH access.

Create the NoCloud seed

Why use NoCloud?

NoCloud reads configuration from a local seed volume during first boot. The volume must contain meta-data plus user-data.

This approach turns the unusable cloud image into an identifiable guest with key-based access. The same seed also writes the page used later in the project.

The private key remains on kvm-lab-host. You use its public half to grant access to the guest.

Generate the guest SSH key pair by running this command:

ssh-keygen -t ed25519 -f ~/.ssh/kvm-lab -N ""

What does this command create?

The ~/.ssh/kvm-lab file is the private key used by the host.
The ~/.ssh/kvm-lab.pub file is the public key installed in the guest.
The empty value passed to -N creates the lab key without a passphrase.

Display the public key by running this command:

cat ~/.ssh/kvm-lab.pub

What should you copy?

The command prints one public-key line. Copy the entire line so cloud-init can authorize the matching private key.

Record the printed line here: your SSH public key.
Create meta-data in the host home directory with your terminal text editor by entering this content:

instance-id: kvm-lab-guest-001
local-hostname: kvm-lab-guest

What does this metadata control?

The instance-id value identifies this first-boot configuration to cloud-init.
The local-hostname value gives the guest the hostname kvm-lab-guest.

Save meta-data.
Display the saved metadata by running this command:

cat meta-data

What should you see?

You should see the fixed instance ID on the first line. You should see the guest hostname on the second line.

Create user-data in the host home directory with your terminal text editor by entering this first section:

#cloud-config
users:
  - default
  - name: ec2-user
    ssh_authorized_keys:
      - your-ssh-public-key-here

What does the user section do?

The users section keeps the image defaults. It also authorizes a public key for ec2-user.

Save user-data.
Display the first saved section by running this command:

cat user-data

What should the first section contain?

The output should start with #cloud-config. The public-key placeholder should appear beneath ssh_authorized_keys.

Append the status-page configuration below the existing content in user-data by entering:

write_files:
  - path: /home/ec2-user/site/index.html
    owner: ec2-user:ec2-user
    permissions: '0644'
    content: |
      <!doctype html>
      <html lang="en">
      <head>
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <title>KVM Guest Status</title>
        <style>
          body { font-family: system-ui, sans-serif; max-width: 760px; margin: 4rem auto; padding: 0 1rem; background: #0b1020; color: #e8ecff; }
          main { background: #171f38; border: 1px solid #33416f; border-radius: 16px; padding: 2rem; }
          code { color: #8fe3ff; }
          li { margin: 0.6rem 0; }
        </style>
      </head>
      <body>
        <main>
          <h1>KVM Guest Is Online</h1>
          <p>This page is being served from an Amazon Linux 2023 guest managed by KVM and libvirt.</p>
          <ul>
            <li>L0: AWS infrastructure and the Nitro hypervisor</li>
            <li>L1: The <code>kvm-lab-host</code> EC2 instance</li>
            <li>L2: The <code>kvm-lab-guest</code> virtual machine</li>
          </ul>
        </main>
      </body>
      </html>

What does the file section do?

The write_files section creates /home/ec2-user/site/index.html inside the guest. The page labels the L0 infrastructure plus the L1 host plus the L2 guest.

The ownership setting lets ec2-user manage the page. The permissions make the page readable by the HTTP server used in the next step.

Save user-data.
Display the combined configuration by running this command:

cat user-data

What should the combined file contain?

You should see the user configuration first. The complete write_files section should follow it.

Replace your-ssh-public-key-here in user-data with your SSH public key.
Save user-data.
Confirm the replacement by running this command:

cat user-data

What proves the key is ready?

The line beneath ssh_authorized_keys should now contain your complete public key. The original placeholder should be gone from the working file.

Build seed.iso from both configuration files by running this command:

xorrisofs -output seed.iso -volid cidata -joliet -rock user-data meta-data

What does this command do?

The -output seed.iso option writes the seed image to seed.iso.
The -volid cidata option gives the image the label recognized by NoCloud.
The final two arguments place user-data plus meta-data inside the image.

You should see an image-writing summary before the terminal prompt returns. The host now has a local NoCloud seed.

Seed image not created?

Confirm that meta-data plus user-data are saved in the directory where you ran the command.

Check that the command uses the volume label cidata exactly.

Ask for help with building the NoCloud seed ISO.

Attach the seed to the guest

The running guest started without the seed. A cold restart lets libvirt mount the new CD-ROM before cloud-init performs first-boot provisioning.

Destroying the domain sounds severe. This command powers off the guest process while keeping its disk plus its persistent definition.

Power off the inaccessible guest by running this command:

sudo virsh destroy al2023-demo

Why use destroy here?

The unprovisioned guest cannot complete a clean shutdown through an authenticated session. The forced power-off gives you a stopped domain that can be redefined with the seed attached.

Copy the seed into libvirt's image directory by running this command:

sudo cp seed.iso /var/lib/libvirt/images/seed.iso

Why copy the seed here?

The final domain definition points to /var/lib/libvirt/images/seed.iso as a file-backed CD-ROM. Copying the image to that path makes the definition resolvable when the guest starts.

Confirm the copied seed exists by running this command:

sudo ls -l /var/lib/libvirt/images/seed.iso

What should you see?

You should see one file listing for /var/lib/libvirt/images/seed.iso. This confirms that libvirt can reach the seed at the path used by the domain XML.

Return to al2023-demo.xml in the terminal editor from the previous step.
Find the existing disk block whose source is /var/lib/libvirt/images/al2023-demo.qcow2.
Add this CD-ROM block immediately below that disk block:

    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/seed.iso'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
    </disk>

What does the CD-ROM block do?

The device='cdrom' setting presents the seed as removable media.
The raw driver reads the ISO image without a virtual disk format layer.
The SATA target attaches the read-only seed as sda.

Save al2023-demo.xml.
Redefine the persistent domain before starting it by running these commands:

sudo virsh define al2023-demo.xml
sudo virsh start al2023-demo

What happens during the restart?

The first command updates the existing domain because the name plus UUID remain unchanged. The second command boots that domain with the seed mounted.

Cloud-init reads the seed during startup. It configures the hostname plus the SSH key plus the status page.

The terminal should confirm that the domain definition was accepted. It should also confirm that al2023-demo started.

✔️ Awesome, I've got everything!

Great. Make sure meta-data plus user-data plus al2023-demo.xml are saved.

Your working user-data should contain your real public key in place of the template placeholder.

ⓧ I'd like to double check the full code

Compare your files with these canonical versions. Keep your real public key in the working copy of user-data when the reference shows your-ssh-public-key-here.

instance-id: kvm-lab-guest-001
local-hostname: kvm-lab-guest

Metadata checkpoint

This file fixes the first-boot identity plus the guest hostname.

#cloud-config
users:
  - default
  - name: ec2-user
    ssh_authorized_keys:
      - your-ssh-public-key-here
write_files:
  - path: /home/ec2-user/site/index.html
    owner: ec2-user:ec2-user
    permissions: '0644'
    content: |
      <!doctype html>
      <html lang="en">
      <head>
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <title>KVM Guest Status</title>
        <style>
          body { font-family: system-ui, sans-serif; max-width: 760px; margin: 4rem auto; padding: 0 1rem; background: #0b1020; color: #e8ecff; }
          main { background: #171f38; border: 1px solid #33416f; border-radius: 16px; padding: 2rem; }
          code { color: #8fe3ff; }
          li { margin: 0.6rem 0; }
        </style>
      </head>
      <body>
        <main>
          <h1>KVM Guest Is Online</h1>
          <p>This page is being served from an Amazon Linux 2023 guest managed by KVM and libvirt.</p>
          <ul>
            <li>L0: AWS infrastructure and the Nitro hypervisor</li>
            <li>L1: The <code>kvm-lab-host</code> EC2 instance</li>
            <li>L2: The <code>kvm-lab-guest</code> virtual machine</li>
          </ul>
        </main>
      </body>
      </html>

User data checkpoint

This template authorizes ec2-user plus writes the complete status page. Your saved working file should contain your public key on the authorization line.

<domain type='kvm'>
  <name>al2023-demo</name>
  <uuid>6c984a40-7f9b-4e04-a56c-4eb362d04a33</uuid>
  <memory unit='MiB'>2048</memory>
  <currentMemory unit='MiB'>2048</currentMemory>
  <vcpu>2</vcpu>
  <os>
    <type arch='x86_64' machine='q35'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
  </features>
  <cpu mode='host-passthrough'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/al2023-demo.qcow2'/>
      <target dev='vda' bus='virtio'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/seed.iso'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
    </disk>
    <interface type='network'>
      <mac address='52:54:00:4b:56:4d'/>
      <source network='default'/>
      <model type='virtio'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
  </devices>
</domain>

Domain checkpoint

The final domain retains the original UUID plus resources plus network identity. The only new device is the SATA CD-ROM backed by seed.iso.

Guest does not start?

Check that the CD-ROM block sits inside <devices> after the qcow2 disk block.

Confirm that the seed path in the XML exactly matches /var/lib/libvirt/images/seed.iso.

Ask for help with debugging the redefined libvirt domain.

Find the lease and verify access

The default libvirt network gives the guest a private IPv4 address through DHCP. The lease connects the fixed virtual MAC address to the address you need for SSH.

Before you check the lease, do you expect the guest address to be available immediately after its first provisioned boot?

Check the guest's DHCP lease by running this command:

sudo virsh domifaddr al2023-demo --source lease

What does the lease prove?

A listed IPv4 address proves that the guest's virtio interface joined the default NAT network. It also gives the L1 host a private route to the L2 guest.

Wait briefly if the address column is empty.
Run the lease command again until an IPv4 address appears.
Record the listed address here: <guest-ip>.

The first SSH connection asks you to confirm the guest fingerprint. Accept it only after the destination address matches the lease reported by virsh.

Connect to the guest with the generated private key by running this command:

ssh -i ~/.ssh/kvm-lab ec2-user@[[GUEST_IP="<guest-ip>"]]

What does this connection test?

The command uses the private key stored on the L1 host. A guest shell proves that cloud-init installed the matching public key for ec2-user.

SSH access rejected?

Confirm that the address in your SSH command matches the current lease for al2023-demo.

Check the working user-data file for the complete public key beneath ssh_authorized_keys.

Ask for help with troubleshooting guest SSH access.

Before you run the final check, which hostname plus page heading do you expect the provisioned guest to show?

Verify the guest identity plus the status page from the SSH session by running these commands:

hostname
cat /home/ec2-user/site/index.html

What should you see?

The first command should print kvm-lab-guest. This proves that the NoCloud metadata set the guest identity.

The second command should print the HTML stored at /home/ec2-user/site/index.html. Look for the heading KVM Guest Is Online in the output.

That inaccessible cloud image now accepts your key. It also carries the hostname plus page required for the service test.

Your L2 guest now has an identity plus a private address plus key-based access. Next, you will run the page as a service inside the guest before retrieving it from the L1 host.

Serve and Reach the Guest Application

Your NoCloud-provisioned KVM guest now accepts SSH. Its status page is ready inside L2.

A running guest is only infrastructure. The service must cross the private NAT boundary before the lab proves anything useful.

The test uses Python 3.9's http.server for a lab-only HTTP endpoint. The L1 host reaches that endpoint through libvirt without exposing port 8000 to the internet.

In this step, get ready to:

Serve the status page from inside the guest.
Reach the page from the L1 host through the private NAT network.
Collect virtualization evidence from virsh output plus kernel releases.

Start the service inside the guest

The guest already has the page from its first-boot configuration. A background server makes that file reachable after you leave the guest shell.

Leave kvm-lab-eic unchanged so port 8000 remains private.
Refresh the guest's DHCP lease details by running:

sudo virsh domifaddr al2023-demo --source lease

What does this command show?

The domifaddr command asks libvirt for the guest's network addresses.
The --source lease option reads the address assigned by the default network's DHCP lease.

Record the IPv4 address shown for al2023-demo: <guest-ip>.
Use <guest-ip> wherever a later command shows <guest-ip>.
Connect to the guest with its private key by running:

ssh -i ~/.ssh/kvm-lab ec2-user@<guest-ip>

How does this connection work?

The -i ~/.ssh/kvm-lab option selects the private key created for this guest.
The ec2-user@<guest-ip> destination targets the guest through its private DHCP address.

Your shell prompt now belongs to the L2 guest. The next command starts the server inside that guest.

Start the lab-only server in the background by running:

nohup python3 -m http.server 8000 --bind 0.0.0.0 --directory /home/ec2-user/site > /home/ec2-user/http-server.log 2>&1 &

What does this command do?

The nohup command keeps the process running after the SSH session ends.
The python3 -m http.server 8000 portion starts the test server on port 8000.
The --bind 0.0.0.0 option accepts traffic through the guest's virtual network interface.
The --directory /home/ec2-user/site option serves the directory containing index.html.
The output redirection writes server messages to /home/ec2-user/http-server.log.

Your shell should return control after starting a background job. The server can now continue after the SSH connection closes.

Server did not return control?

Check that you ran the server command from the guest shell.
Check that the directory is spelled /home/ec2-user/site.
Run the server command again if the first attempt contained a typing mistake.

Ask for help with the guest server command:.

Close the SSH connection using the same method you used earlier so you return to the Amazon EC2 Instance Connect terminal.

Reach and inspect the guest service

The default libvirt network gives the L1 host a direct path to the L2 guest. This private route lets you test the service without changing the EC2 security group.

Before you run the request, picture what the L1 host should receive from the L2 guest.

Retrieve the status page from the L1 host by running:

curl http://<guest-ip>:8000/

What does this request prove?

The curl request begins on the L1 host. It reaches the Python server through the guest's private virtual interface.

A successful response proves the L2 operating system can serve traffic across the libvirt NAT network. Port 8000 remains closed to the public internet.

You should see HTML containing KVM Guest Is Online plus the L0, L1, and L2 descriptions. That is the proof you came for. The L1 host can reach a page served inside the L2 guest.

Record the returned page content in your project notes.

Page did not respond?

Confirm your current shell belongs to the L1 host before running curl.
Confirm the address in the request matches the latest DHCP lease.
Reconnect to the guest if you need to start the server command again.

Ask for help tracing the private request:.

Inspect the guest's configured resources by running:

sudo virsh dominfo al2023-demo

What does dominfo capture?

The output identifies the al2023-demo domain.
The output reports the guest's running state.
The output reports its virtual CPU allocation.
The output reports its maximum memory allocation.

Record the virsh dominfo output in your project notes.
Inspect the guest's virtual interface by running:

sudo virsh domiflist al2023-demo

What does domiflist capture?

The output identifies the guest's network interface.
The default source shows that the interface uses libvirt's default network.
The listed MAC address links the running interface to the domain definition.

Record the virsh domiflist output in your project notes.

Compare the host and guest kernels

A full virtual machine has its own kernel boundary. Comparing the two kernel releases makes that boundary visible from both sides.

Before you run the comparison, predict whether the L1 host and L2 guest will report the same kernel release.

Print the host kernel plus the guest kernel by running:

uname -r
ssh -i ~/.ssh/kvm-lab ec2-user@<guest-ip> uname -r

What does this comparison show?

The first uname -r runs directly on kvm-lab-host.
The second uname -r runs inside kvm-lab-guest over SSH.
The separate results show that the guest boots its own kernel.

You should see two distinct kernel release strings. One belongs to the L1 host. The other belongs to the L2 guest.

Record both kernel release strings in your project notes.

How does OpenVZ differ?

KVM gives the guest an independent kernel managed through a virtual-machine boundary. OpenVZ containers share one host kernel across isolated environments.

Your two kernel results are direct evidence of the KVM model. A shared-kernel container model would expose the host's kernel instead.

Before the final check, predict whether the service still responds after your inspection commands.

Verify the guest service one final time by running:

curl http://<guest-ip>:8000/

What does the final response confirm?

The final request confirms that inspecting the domain did not interrupt the guest. The background Python server still serves the status page across the private network.

You should again see KVM Guest Is Online plus the three virtualization layers. Your service now works across the complete L0, L1, and L2 path.

Your nested service is proven from both sides of the virtualization boundary. Next, you can continue to the optional challenge or prepare the lab for teardown.

Secret mission

Right-Size the Guest

Your guest currently has more CPU and memory than its status page needs. Reduce it to one vCPU and 1 GiB of memory, then prove the new limits from both sides of the virtualization boundary without losing the service.

Clean Up Your Resources

This lab uses a billable On-Demand Amazon EC2 host plus Amazon EBS storage. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Cost warning

The On-Demand m8i.xlarge host incurs usage charges while running. Its attached EBS storage continues to incur charges when the instance is stopped.

The project's AWS usage draws from your available credits. Leaving resources running uses more of those credits.

Nested virtualization plus Amazon EC2 Instance Connect add no separate charge.

Resources you used:

On-Demand Amazon EC2 instance kvm-lab-host.
Nested KVM lab stored entirely on kvm-lab-host.
20 GiB gp3 Amazon EBS root volume attached to kvm-lab-host.
Custom Amazon EC2 security group kvm-lab-eic.

Keep everything running

No action is needed while you are actively building with the nested virtualization lab.

The kvm-lab-host instance remains available through the AWS Management Console.
The al2023-demo guest retains its one vCPU limit plus 1024 MiB memory limit.
The Python HTTP service remains available while the guest stays running.
The On-Demand instance continues to use AWS credits while running.
The 20 GiB gp3 root volume remains billable while allocated.
The kvm-lab-eic security group remains ready for future EC2 Instance Connect sessions.

Pause - I'll come back to this later

Pausing shuts down the running guest plus its EC2 host while preserving the root volume for your return.

Request a clean guest shutdown from the browser terminal by running the first command below:
Poll the guest state by running the second command below until shutdown completes:

sudo virsh shutdown al2023-demo
sudo virsh domstate al2023-demo

What do these commands do?

The shutdown command asks the guest operating system to power off cleanly.
The domstate command reports the guest's current lifecycle state.

You should see shut off once the guest finishes its shutdown.

Guest still active?

If the state does not change, allow the guest more time to finish its operating system shutdown.

Ask for help diagnosing the shutdown.

Return to the Instances page in the AWS Management Console.
Select kvm-lab-host.
Choose Instance state.
Choose the option that stops the selected instance.
Wait until the instance state shows stopped.

Stopping the host pauses instance usage charges. The attached EBS root volume continues to incur storage charges.

Delete - I don't want to use this again

Deletion removes the EC2 host plus the nested guest stored on it. The root volume plus custom security group also need to be confirmed as deleted.

Request a clean guest shutdown from the browser terminal by running the first command below:
Poll the guest state by running the second command below until shutdown completes:

sudo virsh shutdown al2023-demo
sudo virsh domstate al2023-demo

What do these commands do?

The shutdown command gives the guest a clean path to power off.
The domstate command confirms when the guest has stopped.

You should see shut off before you delete its EC2 host.

Guest will not shut down?

If the state does not change, allow the guest more time to finish its operating system shutdown.

Ask for help diagnosing the shutdown.

Return to the Instances page in the AWS Management Console.
Select kvm-lab-host.
Choose the Storage tab.
Locate the root volume under Block devices.
Record the root Volume ID as your root volume ID.
Scroll right to the Delete on termination column.
Confirm the root volume displays Yes.

Deleting the host is permanent. Its guest files plus SSH key are stored on the root volume, so termination removes them with it.

Return to Instances.
Select kvm-lab-host.
Choose Instance state.
Choose Terminate (delete) instance.
Confirm the deletion by choosing Terminate (delete).
Wait until the instance state shows terminated.

The host can leave a terminated record on the Instances page briefly. That record does not represent a running resource.

Open Volumes in the Amazon EC2 console.
Filter the page for your root volume ID.

No result means the root volume was deleted automatically. An Available result means the detached volume still needs manual deletion.

Select the volume if its state is Available.
Choose Actions.
Choose Delete volume.
Enter delete in the confirmation field.
Choose Delete.

Open Security Groups in the Amazon EC2 console.
Select kvm-lab-eic.
Choose Actions.
Choose Delete security groups.
Enter Delete if the console requests confirmation text.
Choose Delete.

Security group still in use?

A security group cannot be deleted while it remains associated with an instance or network interface. Wait for instance termination to finish before trying the deletion again.

Ask for help finding the remaining association.

Finish with three console checks in US East (N. Virginia). Each empty result proves that part of the lab is gone.

Filter the Instances page for kvm-lab-host.
Confirm no running or stopped instance matches kvm-lab-host.
Filter the Volumes page for your root volume ID.
Confirm the volume search returns no result.
Filter the Security Groups page for kvm-lab-eic.
Confirm the security group search returns no result.

Your resource state now matches the cleanup choice you made. You know exactly which parts of the lab remain billable.

Nice Work!

You did it! Your service now runs inside a provisioned KVM guest on an Amazon EC2 host. You also proved that the service remains reachable after a cold resize to one vCPU plus 1 GiB of memory.

You've learned how to:

Built a three-layer virtualization lab using QEMU under libvirt management. Verified hardware acceleration through /dev/kvm.
Provisioned an inaccessible cloud image with a NoCloud seed. Gave the guest SSH access through your public key.
Served a status page from the L2 guest. Retrieved it across the private NAT boundary with an HTTP request. Compared the host kernel with the guest kernel to prove that the guest owns an independent kernel.
Completed the optional Secret Mission by cold-resizing al2023-demo to one vCPU plus 1 GiB of memory. Verified the new limits from libvirt. Confirmed the status page still responded.

Ready to quiz yourself?