Data Center Troubleshooting Lab

Simulate server outages and diagnose them using real DCT tools on Linux.

Introduction

30 Second Summary

Every time you stream a show, send a message, or check your bank account, the request travels through a data center. When something breaks inside one of those buildings, a technician has minutes to find and fix the problem. Amazon hires thousands of these technicians every year, and the interview tests one thing above all: can you troubleshoot a real outage under pressure?

In this project, you will set up a Linux server on your Windows machine, run real network diagnostics, and simulate three realistic outages that mirror Amazon DCT interview questions. You will fix each one using a structured troubleshooting flow and document everything in a GitHub portfolio.

What You'll Build

You will open your Linux terminal, diagnose a simulated server outage from symptom to root cause in under five minutes, then point an interviewer to your GitHub repository where every incident is documented in the same format Amazon teams use for shift handoffs.

By the end of this project, you'll have:

  • A fully functional Linux server environment on your Windows machine where you can run diagnostic commands, manage services, and investigate failures exactly like a working technician.
  • Three completed troubleshooting scenarios (a failed web server, a network connectivity issue, and a disk capacity alert) that give you real stories for STAR-format interview answers.
  • A public incident response portfolio on GitHub with structured documentation that proves your diagnostic ability to hiring managers.
  • Secret Mission: Simulate a compound multi-layer outage where a service failure and a network misconfiguration hit at the same time, forcing you to debug both layers from a single symptom.

Are there any prerequisites?

You need a Windows 11 machine with internet access, Git installed, and a GitHub account. No prior Linux experience is required. Step 1 walks you through setting up everything from scratch.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters for your Amazon DCT interview preparation.

Set Up Your Linux Server Environment

Amazon data center technicians diagnose server failures using Linux command-line tools every single shift. To practice those same tools, you need a real Linux environment running on your machine.

In this step, you will install WSL (Windows Subsystem for Linux) with Ubuntu, set up the diagnostic toolkit, and create a Git repository on GitHub to store your incident documentation.

In this step, get ready to:
  • Install WSL 2 with Ubuntu on your Windows machine.
  • Install the networking diagnostic tools used by data center technicians.
  • Initialize a Git repository and push it to GitHub for your incident documentation.
Install WSL and Ubuntu on Windows 11

WSL lets you run a full Linux environment directly inside Windows. One command installs everything you need.

  • Press the Windows key to open your search bar.
  • Type PowerShell in the search box.
  • Right-click Windows PowerShell in the results.
  • Select Run as administrator.
  • Click Yes on the User Account Control prompt that appears.

The title bar should now read Administrator: Windows PowerShell.

  • Install WSL and Ubuntu by running this command:
wsl --install

What does this command do?

The wsl --install command does four things in one shot:

  • Enables the WSL and Virtual Machine Platform components.
  • Downloads and installs the latest Linux kernel.
  • Sets WSL 2 as the default version.
  • Downloads and installs the Ubuntu distribution.

What happens next?

After the command finishes downloading, it will ask you to restart your machine. Close any open work and restart when prompted.

After the restart, an Ubuntu terminal window will open automatically. It will ask you to create a Linux username and password. Type a username (lowercase) and a password you will remember. The password will not show characters as you type.

  • Restart your machine when prompted.
  • After the restart, wait for the Ubuntu window to appear.
  • Enter a lowercase username when prompted.
  • Enter a password when prompted (you will not see characters appear as you type).
  • Re-enter the password to confirm.

Once you see username@PCName followed by a $ sign, your Ubuntu environment is ready.

  • Open a new PowerShell window (does not need to be Administrator this time) and verify the installation by running:
wsl -l -v

✔️ I see Ubuntu with VERSION 2

You should see a table listing Ubuntu with STATE as Running and VERSION as 2. WSL is installed and ready.

ⓧ Ubuntu is not listed or command not recognized

If you see WSL help text instead of a distribution list, the install may not have completed. Try these steps:

  • Make sure you restarted your machine after running wsl --install.
  • Open PowerShell as Administrator again and run wsl --install a second time.
  • If the install hangs, try wsl --install --web-download to download directly from the internet.
  • After a successful install, restart and open the Ubuntu app from the Start menu to create your username and password.

WSL install failing?

Make sure virtualization is enabled in your BIOS settings. Most modern machines have this on by default, but some manufacturers ship with it disabled.

Also confirm you ran PowerShell as Administrator. Without elevated permissions, the command cannot enable the required Windows features.

help me troubleshoot my WSL installation on Windows 11

Update the system and install diagnostic tools

Your Ubuntu installation is fresh and needs updated packages. You will also install the standard networking toolkit that data center technicians rely on.

  • In your Ubuntu terminal, update the system packages by running:
sudo apt update && sudo apt upgrade -y

What does this command do?

The apt update part refreshes the list of available packages from Ubuntu's servers. The apt upgrade -y part installs all available updates, with -y automatically confirming each one so you do not have to press Enter repeatedly.

This may take a minute or two depending on how many packages need updating.

  • Install the networking diagnostic tools and a test web server by running:
sudo apt install -y net-tools iproute2 traceroute dnsutils nginx

What are these tools?

  • net-tools provides classic commands like ifconfig.
  • iproute2 provides the modern ip and ss commands for interface and socket inspection.
  • traceroute maps the network path between your machine and a target host.
  • dnsutils gives you dig and nslookup for DNS troubleshooting.
  • nginx is a real web server you will use as a test service to break and diagnose.
  • Verify nginx is running by checking its status:
systemctl status nginx

You should see active (running) highlighted in green in the output.

  • Open your Windows browser and navigate to http://localhost.

You should see the nginx welcome page with the text "Welcome to nginx" in your browser. This confirms your Linux server is running and serving traffic.

Not seeing the nginx welcome page?

If systemctl status nginx shows inactive, start it manually with sudo systemctl start nginx and check the status again.

If the browser shows an error, make sure you typed http://localhost (not https). WSL 2 automatically forwards ports from Linux to Windows.

help me get nginx running in WSL

Initialize your project repository

Every incident you diagnose in this lab will be documented and committed to a Git repository. This gives you a portfolio of troubleshooting evidence to show interviewers.

  • Create your project directory and move into it by running these commands:
mkdir -p ~/dc-lab && cd ~/dc-lab
  • Initialize a Git repository by running:
git init

You should see a message that says Initialized empty Git repository followed by the path.

  • Create your project files by running these commands:
echo "# Data Center Troubleshooting Lab" > README.md
echo "Incident response documentation and network diagnostics practice for Amazon DCT role." >> README.md
echo "*.img" > .gitignore
  • Confirm the files exist by running:
ls

You should see README.md and .gitignore in the output.

What is the .gitignore for?

Later in the project you will create disk image files (.img) to simulate storage drives. These are large binary files that do not belong in version control. The .gitignore tells Git to skip them.

  • Make your first commit by running:
git add . && git commit -m "Initial commit: project setup"

You should see output confirming 2 files were committed.

  • Open your Windows browser and go to github.com.
  • Click the + icon in the top right corner.
  • Select New repository.
  • Enter dc-troubleshooting-lab as the repository name.
  • Keep the repository set to Public.
  • Do NOT check any of the initialization options (no README, no .gitignore, no license).
  • Click Create repository.

Why keep it empty?

You already created a README and .gitignore locally. If GitHub also creates them, you will have conflicting histories that are annoying to merge. Starting the remote empty avoids this entirely.

  • Back in your Ubuntu terminal, link your local repository to GitHub and push by running these commands:
git remote add origin https://github.com/[[GITHUB_USERNAME="YOUR-USERNAME"]]/dc-troubleshooting-lab.git
git branch -M main
git push -u origin main

What do these commands do?

  • git remote add origin connects your local repo to the GitHub repository you just created.
  • git branch -M main renames your default branch to main.
  • git push -u origin main uploads your commit to GitHub and sets main as the default push target.

Git asking for credentials?

If Git prompts for your username and password, enter your GitHub username and a Personal Access Token (not your GitHub password). You can create a token at github.com under Settings then Developer settings then Personal access tokens.

  • Verify your environment is fully working by running these commands one at a time:
ip addr show

You should see your network interfaces listed, including eth0 with an IP address assigned.

  • Confirm nginx is still serving by running:
systemctl status nginx

You should see active (running) in the output. Open http://localhost in your Windows browser one more time to confirm the welcome page still appears.

Before you check the final result, what do you expect to see when you run git remote -v in your ~/dc-lab directory?

  • Verify your GitHub remote is connected by running:
git remote -v

You should see two lines showing your origin URL pointing to https://github.com/YOUR-USERNAME/dc-troubleshooting-lab.git for both fetch and push.

Git push rejected or remote not found?

Double-check that the repository name on GitHub matches exactly: dc-troubleshooting-lab. A typo in the name or your username will cause the push to fail.

If you initialized the GitHub repository with a README, you will get a merge conflict. Delete the repository on GitHub, recreate it empty, and push again.

help me push my local git repository to GitHub

✔️ Awesome, I've got everything!

Great. Confirm you have saved all files and that your GitHub repository shows the README and .gitignore on the main branch.

ⓧ I'd like to double check the full code

Your ~/dc-lab directory should contain exactly these two files:

README.md:

# Data Center Troubleshooting Lab
Incident response documentation and network diagnostics practice for Amazon DCT role.

.gitignore:

*.img

You can verify the contents by running cat README.md and cat .gitignore in your terminal.

Your Linux server environment is live, your diagnostic tools are installed, and your portfolio repository is on GitHub. Next up, you will use those diagnostic tools to run a full network sweep and capture your server's baseline state.

Run Your First Network Diagnostic Sweep

In Step 1, you set up a fully functional Linux server environment. Now it's time to use it.

Amazon DCT interviews frequently ask "how would you troubleshoot a connectivity issue?" This step teaches you the exact sequence of diagnostic commands that technicians run on production servers. By the end, you'll have real results you can reference in your STAR-format interview answers.

In this step, get ready to:
  • Practice the Layer 1-2-3 network diagnostic sequence used by data center technicians.
  • Verify service connectivity by checking open ports and HTTP responses.
  • Document your baseline readings and push them to GitHub for future reference.
Practice the Layer 1-2-3 diagnostic sequence

Network troubleshooting follows a layered approach. You start at the physical/link layer (are interfaces up?), move to the network layer (do I have an IP and a route?), then test connectivity (can I reach other hosts?). This sequence isolates problems quickly.

  • In your Ubuntu terminal, check that your network interfaces are up by running this command:
ip link show

You should see your eth0 interface listed with state UP in the output.

  • Check your IP address configuration by running:
ip addr show eth0
  • Check your default gateway (the route traffic takes to leave your machine) by running:
ip route show

What do these commands tell you?

These three commands map the first three layers of your network stack. ip link show confirms the interface exists and is active (Layer 1-2). ip addr show eth0 shows your assigned IP address (Layer 3). ip route show reveals where traffic goes when it leaves your machine.

If any of these are missing or wrong, you've found the problem without needing to test anything higher up the stack.

  • Test local connectivity (traffic that never leaves your machine) by running:
ping -c 4 127.0.0.1

You should see 4 packets transmitted and 4 received with 0% packet loss.

  • Test external connectivity by pinging Google's public DNS server:
ping -c 4 8.8.8.8

You should see 4 successful replies. This confirms your WSL instance can reach the internet.

  • Trace the path your traffic takes to reach the external host by running:
traceroute -m 10 8.8.8.8

What does traceroute show?

Each line represents a "hop" between your machine and the destination. In a data center, if a server can't reach a remote host, traceroute shows exactly where the path breaks. The -m 10 flag limits the trace to 10 hops so it finishes quickly.

Some hops may show * * * instead of a time. This means that router chose not to respond to the probe. It does not necessarily indicate a problem.

Ping to 8.8.8.8 failing?

If you see 100% packet loss, your WSL networking may not be configured correctly. Try restarting WSL by running wsl --shutdown in PowerShell, then reopening Ubuntu.

If the issue persists, check that your Windows machine itself has internet access by opening a browser.

Help me troubleshoot why ping is failing in my WSL Ubuntu environment.

Verify service connectivity with port checking

Knowing your network is up is only half the picture. A server can have perfect connectivity but still fail to serve requests if the application isn't listening on the right port. Port checking bridges the gap between "network works" and "service works."

  • Check which ports are currently listening on your machine by running:
ss -tlnp

What does ss -tlnp mean?

Each flag has a purpose: -t shows TCP sockets only. -l filters to listening (waiting for connections) sockets. -n shows port numbers instead of service names. -p shows which process owns each socket.

This command is the modern replacement for netstat. You'll use it constantly when diagnosing "why can't I connect to service X" problems.

  • Confirm that nginx is specifically listening on port 80 by running:
ss -tlnp | grep :80

You should see a line showing nginx listening on *:80 or 0.0.0.0:80. This confirms the web server is ready to accept connections.

Now for the real test. Before you run the next command, predict: will you get a successful HTTP response, or will something go wrong?

  • Test the full HTTP connection to your web server by running:
curl -I http://localhost

You should see HTTP/1.1 200 OK as the first line of the response headers. This confirms the complete path works: network is up, port is listening, and the service is responding correctly.

  • Verify DNS resolution is working by running:
dig google.com

You should see an ANSWER SECTION with one or more IP addresses for google.com. This confirms your machine can translate domain names to IP addresses.

Why does DNS matter for a DCT?

Many "server unreachable" tickets turn out to be DNS failures. The server is fine, the network is fine, but the client can't resolve the hostname. Testing DNS separately helps you rule it out (or identify it) in seconds.

Seeing 'Connection refused' from curl?

This means nginx isn't running. Check its status with systemctl status nginx and start it with sudo systemctl start nginx if it's inactive.

Help me figure out why curl to localhost is returning Connection refused.

Document and commit your baseline

Data center technicians maintain baseline documentation for every server. When something breaks at 3 AM, you compare the current state against the known-good baseline to spot what changed. You're going to capture your server's healthy state right now.

  • Create a baselines directory inside your project by running:
mkdir -p ~/dc-lab/baselines
  • Save your network configuration, listening ports, and nginx status into baseline files by running these commands:
ip addr show > ~/dc-lab/baselines/network.txt
ss -tlnp > ~/dc-lab/baselines/ports.txt
systemctl status nginx > ~/dc-lab/baselines/nginx-status.txt
  • Confirm your baseline files were created by running:
ls ~/dc-lab/baselines/

You should see three files listed: network.txt, nginx-status.txt, and ports.txt.

Why save baselines in Git?

In real data center operations, baselines are part of shift handoff documentation. If a technician on the next shift sees a problem, they compare the current output against the last known-good state. Version-controlling these files means you always have a timestamped "this is what healthy looks like" reference.

  • Commit your baselines and push them to GitHub by running:
cd ~/dc-lab && git add . && git commit -m "Add baseline documentation: network, ports, nginx status"
  • Push your commit to GitHub by running:
git push
  • Verify your commit history shows the new baseline commit by running:
git log --oneline

You should see your new commit at the top: Add baseline documentation: network, ports, nginx status.

Git push asking for credentials?

If Git asks for a username and password, GitHub no longer accepts password authentication. You need a Personal Access Token (PAT). Go to GitHub, click your profile icon, then Settings, then Developer settings, then Personal access tokens. Generate a classic token with repo scope and use it as your password.

Help me set up Git authentication for pushing to GitHub from WSL.

Your server is verified healthy, your diagnostic toolkit is proven, and your baselines are documented on GitHub. Next up, you'll intentionally break this server and diagnose the failure using a structured troubleshooting flow.

Diagnose a Failed Service (Simulated Outage)

"How would you troubleshoot a server that isn't responding?" is the most common Amazon DCT interview question. In this step, you'll experience that scenario firsthand by intentionally breaking your nginx web server, then following a structured diagnostic flow to find and fix the problem.

The baseline you captured in the last step proved everything was healthy. Now you'll see what happens when things go wrong, and practice the exact diagnostic sequence that Amazon technicians use to recover from outages.

In this step, get ready to:
  • Break your web server by introducing a configuration error.
  • Follow a structured troubleshooting flow to diagnose the root cause.
  • Fix the issue, verify the recovery, and document the incident in your portfolio.
Break the service (simulate the outage)

Data center technicians don't get to choose when things break. To practice the diagnostic flow, you'll simulate an outage by stopping the service and corrupting the configuration file.

  • Stop the nginx web server and introduce a configuration error by running these commands:
# Stop the running web server
sudo systemctl stop nginx

# Remove the semicolon from the listen directive (introduces a syntax error)
sudo sed -i 's/listen 80;/listen 80/' /etc/nginx/sites-available/default

# Append an invalid line to the main config file
echo "BROKEN" | sudo tee -a /etc/nginx/nginx.conf

What do these commands do?

The first command stops nginx so it's no longer serving web pages.

The sed command silently removes the semicolon from listen 80; in the config file, creating a syntax error that's hard to spot visually. The echo command appends the word "BROKEN" to the main config, which nginx doesn't recognize as a valid directive.

Together, these simulate what happens when a misconfiguration is introduced during maintenance.

Before you run the next command, think about what you expect to see. The server is stopped and the config is broken. What will happen when you try to load the web page?

  • Try to access the web server by running:
curl -I http://localhost

You'll see a "Connection refused" error. This is exactly what a technician sees when they get an alert that a server isn't responding. The service is down, and right now you don't know why.

Not seeing Connection refused?

If you're still getting a response, the systemctl stop command may not have completed. Run systemctl status nginx to confirm it shows inactive (dead). If it's still active, run the stop command again with sudo.

Help me figure out why my nginx service is still responding after I stopped it.

Follow the structured troubleshooting flow

You know the symptom: the web server isn't responding. Now you'll work through the diagnostic sequence that every Amazon DCT follows: check status, attempt restart, read logs, validate configuration.

  • Start by checking the service status by running:
# Step A - Check service status
systemctl status nginx

You'll see Active: inactive (dead) in the output. This confirms the service isn't running, but doesn't tell you why.

  • Attempt to restart the service by running:
# Step B - Try to start the service
sudo systemctl start nginx

This command will fail silently. Run the status check again to confirm it didn't start:

systemctl status nginx

You'll see it's still inactive or shows a failure state. The service can't start, which points to a configuration problem.

  • Read the service logs to find error details by running:
# Step C - Read the logs
sudo journalctl -u nginx --no-pager -n 20

The logs show nginx encountered an error trying to parse its configuration. You'll see an [emerg] level message pointing to the configuration file and a line number.

  • Run the configuration test to get the exact error location:
# Step D - Validate the configuration
sudo nginx -t

You'll see output like nginx: [emerg] unknown directive "BROKEN" followed by nginx: configuration file /etc/nginx/nginx.conf test failed. Now you know exactly what's wrong and where.

Why this four-step sequence matters

This is the exact diagnostic flow Amazon expects technicians to follow: status check, restart attempt, log analysis, configuration validation. Each step narrows down the problem from "it's broken" to "here's the exact file and line."

In an interview, describing this sequence with a real example demonstrates your structured troubleshooting methodology. Random guessing or jumping straight to config files without checking logs first is a red flag.

Fix the issue, verify, and document

You've identified the root cause: an invalid "BROKEN" directive in nginx.conf. Now you'll fix it, verify the recovery, and write up the incident.

  • Remove the invalid line from the configuration file by running:
# Remove the BROKEN line from nginx.conf
sudo sed -i '/BROKEN/d' /etc/nginx/nginx.conf
  • Validate that the configuration is now correct by running:
sudo nginx -t

You should see syntax is ok and test is successful in the output. The configuration file is valid again.

  • Start the service and verify it's running:
# Restart the service
sudo systemctl start nginx

# Confirm it's active
systemctl status nginx

You'll see Active: active (running) in the status output. The service is back.

Now for the final verification. Before you run the next command, predict what you'll see this time. Will the web server respond?

  • Confirm connectivity is restored by running:
curl -I http://localhost

You'll see HTTP/1.1 200 OK in the response headers. Your web server is fully recovered.

Still seeing Connection refused?

Run systemctl status nginx to check if the service actually started. If it shows a failure, run sudo nginx -t again. The original sed command also removed the semicolon from listen 80; in /etc/nginx/sites-available/default. Open that file with sudo nano /etc/nginx/sites-available/default and add the semicolon back so it reads listen 80; again.

Help me fix my nginx configuration so the service starts successfully.

The technical fix is done. Now you'll document the incident, which is just as important as fixing it. In a data center, every outage gets documented for shift handoff and post-incident review.

  • Create the incidents directory and write your incident report by running:
mkdir -p ~/dc-lab/incidents
cat << 'EOF' > ~/dc-lab/incidents/incident-001.md
# Incident 001: nginx Service Failure

## Symptom
nginx not responding. `curl -I http://localhost` returns "Connection refused."

## Root Cause
Configuration syntax error: missing semicolon on `listen 80` directive in `/etc/nginx/sites-available/default`. Invalid line "BROKEN" appended to `/etc/nginx/nginx.conf`.

## Fix Applied
1. Removed invalid "BROKEN" line from `/etc/nginx/nginx.conf`
2. Validated configuration with `sudo nginx -t` (syntax OK)
3. Started service with `sudo systemctl start nginx`

## Verification
- `systemctl status nginx` shows active (running)
- `curl -I http://localhost` returns HTTP/1.1 200 OK
EOF

Why document every incident?

Amazon data centers require written documentation for every outage and alarm response. This record serves three purposes: shift handoff (the next technician knows what happened), pattern detection (recurring issues get escalated), and interview evidence (you can reference real incidents in STAR-format answers).

  • Verify the file was created by running:
cat ~/dc-lab/incidents/incident-001.md

You should see your full incident report printed to the terminal.

  • Commit and push the incident documentation to GitHub by running:
cd ~/dc-lab && git add . && git commit -m "Incident 001: nginx service failure - config syntax error"
git push
  • Verify the push succeeded by running:
git log --oneline -3

You should see your latest commit Incident 001: nginx service failure - config syntax error at the top of the log.

Git push rejected?

If git push fails with an authentication error, you may need to use a personal access token instead of a password. Go to GitHub, navigate to Settings then Developer settings then Personal access tokens, and generate a token with repo scope. Use that token as your password when prompted.

Help me push my dc-lab commits to GitHub.

You've just completed your first structured incident response. Next up, you'll tackle a different class of problem that Amazon technicians encounter daily: disk capacity alerts.

Respond to a Disk Capacity Alert

In the last step, you diagnosed a service outage from first symptom to full recovery. That was a clear failure: curl returned "Connection refused" and you knew immediately something was wrong.

Disk capacity issues are sneakier. Your server can appear completely healthy while a filling disk silently degrades performance and eventually crashes services. Environmental alarms like "disk at 85%" are the most common alerts data center technicians respond to on every shift.

In this step, get ready to:
  • Simulate a disk capacity alert on a test filesystem.
  • Diagnose the disk issue using filesystem monitoring tools.
  • Remediate, document the incident, and push to GitHub.
Simulate a disk filling up

Data centers use dedicated storage volumes for logs, databases, and temporary files. When one of these volumes fills up, the monitoring system fires an alert long before the disk hits 100%.

You will create a small test filesystem that mimics a separate disk volume, then fill it past the alert threshold.

  • Create a 100MB disk image file and format it as an ext4 filesystem by running these commands:
dd if=/dev/zero of=/tmp/dc-lab-disk.img bs=1M count=100
mkfs.ext4 /tmp/dc-lab-disk.img

What do these commands do?

The dd command creates a 100MB file filled with zeros, simulating a raw disk.

The mkfs.ext4 command formats that file with the ext4 filesystem, the same format used on most Linux servers in data centers.

You should see output from dd showing "100+0 records in" and "104857600 bytes copied", followed by mkfs.ext4 confirming it created the filesystem with blocks and inodes.

  • Mount the new filesystem to simulate attaching a storage volume by running:
sudo mkdir -p /mnt/dc-lab-disk
sudo mount /tmp/dc-lab-disk.img /mnt/dc-lab-disk
  • Confirm the filesystem is mounted and nearly empty by running:
df -h /mnt/dc-lab-disk

You should see the filesystem listed with about 93MB total size and very low usage (around 0-2% used).

  • Now fill the disk to critical capacity by running:
sudo dd if=/dev/zero of=/mnt/dc-lab-disk/fill-file bs=1M count=80
  • Check the disk usage to see the alert state:
df -h /mnt/dc-lab-disk

You should see usage at 80% or higher. In a real data center, this triggers an automated alarm in the monitoring system.

Seeing 'No space left on device'?

If the dd fill command fails with a space error, the filesystem overhead left less than 80MB available. Try filling with 70MB instead: sudo dd if=/dev/zero of=/mnt/dc-lab-disk/fill-file bs=1M count=70.

help me troubleshoot a 'No space left on device' error when filling my test filesystem

Diagnose and identify the problem

Imagine you just received an alert: "Disk at 85% on /mnt/dc-lab-disk." Your job now is to investigate exactly which files are consuming the space.

  • Identify which filesystem is over the threshold by running:
df -h

Look for any row showing usage above 80%. You should see /mnt/dc-lab-disk flagged at critical capacity.

  • Find the largest files on that volume by running:
sudo du -sh /mnt/dc-lab-disk/* | sort -rh | head -5

You should see fill-file listed as the largest file consuming roughly 80MB.

  • Check for log files that might be growing uncontrollably:
sudo find /mnt/dc-lab-disk -name "*.log" -size +10M

No results here means the problem is not runaway logs.

  • Determine file ownership and timestamps to narrow down what created the large file:
ls -la /mnt/dc-lab-disk/

What does this diagnostic sequence mirror?

This is the exact workflow when a DCIM or monitoring system alerts "disk at 85%" during a shift. You identify which volume is full, find the offending files, rule out common causes like runaway logs, and determine file ownership before taking action.

In a real data center, you would also check if the file is actively being written to before deleting it.

Remediate, document, and push

You have identified the root cause: a large temporary file consuming 80% of the volume. Time to fix it and prove the fix worked.

  • Remove the fill file (simulating clearing old logs or temp files) by running:
sudo rm /mnt/dc-lab-disk/fill-file

Before you check the disk usage, what percentage do you expect to see now that the 80MB file is gone?

  • Verify that space was reclaimed by running:
df -h /mnt/dc-lab-disk

You should see usage drop well below 50%. The alert would now clear in a real monitoring system.

Still seeing high usage?

If disk usage has not dropped, confirm the file was actually removed by running ls /mnt/dc-lab-disk/. You should see only lost+found remaining.

If the file still appears, make sure you included sudo in the rm command since the file was created with root permissions.

help me understand why disk usage is still high after I deleted the fill-file

  • Unmount the test filesystem and remove the disk image by running:
sudo umount /mnt/dc-lab-disk
rm /tmp/dc-lab-disk.img
  • Document the incident by creating your report file:
cat > ~/dc-lab/incidents/incident-002.md << EOF
# Incident 002: Disk Capacity Alert

## Timestamp
$(date)

## Alert Trigger
Disk usage at ~85% on /mnt/dc-lab-disk volume.

## Investigation Steps
1. Identified full filesystem with df -h
2. Located largest files with du -sh | sort -rh
3. Checked for growing log files with find
4. Confirmed file ownership with ls -la

## Root Cause
Large temporary file (fill-file, ~80MB) consuming 80% of the 100MB volume.

## Remediation
- Removed fill-file: sudo rm /mnt/dc-lab-disk/fill-file
- Space reclaimed to normal operating levels

## Verification
- df -h /mnt/dc-lab-disk confirms usage below 50%
EOF

Why document every incident?

Data center technicians maintain incident logs for every alert they respond to during a shift. These logs serve as handoff documentation for the next shift team and as evidence of your troubleshooting methodology.

For your Amazon interview, each documented incident becomes a ready-made STAR story: Situation (alert fired), Task (diagnose and fix), Action (your diagnostic steps), Result (space reclaimed, alert cleared).

  • Commit and push your incident report to GitHub by running:
cd ~/dc-lab && git add . && git commit -m "Incident 002: disk capacity alert - temp file consuming 80% of volume"
git push

Before you check your commit history, what do you expect the most recent three commits to show?

  • Verify your GitHub repo shows the new incident by running:
git log --oneline -3

You should see your Incident 002 commit at the top, followed by your Incident 001 commit and your baseline documentation commit.

You have now responded to a disk capacity alert using the same diagnostic flow that data center technicians follow on every shift. Your GitHub portfolio now has two documented incidents showing a structured approach to troubleshooting.

Secret mission

Diagnose a Multi-Layer Outage

Real data center incidents are rarely single-cause. In this challenge, you'll face a compound failure where both a service AND a network misconfiguration break simultaneously. Can you identify both root causes from a single symptom?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally on your machine using WSL, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • WSL Ubuntu installation (free, runs locally on your machine, ~2-3 GB disk space).
  • GitHub repository dc-troubleshooting-lab (public, free, serves as your portfolio piece).

Keep everything running

No action needed. Choose this if you're still actively building or want to keep practicing diagnostic commands.

  • WSL stays installed and ready for future practice. It takes minimal disk space.
  • Your GitHub repository remains as a portfolio piece you can share with interviewers.
  • You can open the Ubuntu terminal any time to run through your troubleshooting scenarios again.

Pause - I'll come back to this later

Close the Ubuntu terminal. WSL stops consuming memory and CPU resources until you open it again.

  • Close any open Ubuntu terminal windows.
  • WSL automatically shuts down after a few seconds of inactivity. Your files and configuration remain intact.
  • To resume later, open Ubuntu from your Start menu. Everything will be exactly as you left it.

Delete - I don't want to use this again

Remove all project resources and reclaim disk space. Your GitHub repository persists independently.

  • Remove the entire WSL Ubuntu installation by opening PowerShell and running:
wsl --unregister Ubuntu

This removes the Ubuntu filesystem and all files inside it, including the ~/dc-lab directory and everything you created during this project.

  • To delete the GitHub repository, go to your dc-troubleshooting-lab repository on GitHub.
  • Click Settings in the top navigation bar.
  • Scroll down to the Danger Zone section at the bottom.
  • Click Delete this repository and follow the confirmation prompts.

Want to keep the portfolio piece?

Consider keeping the GitHub repository even if you delete WSL. It costs nothing and demonstrates your troubleshooting skills to hiring managers reviewing your profile.

Nice Work!

Nice Work!

You just built a fully functional data center troubleshooting lab on your Windows machine and diagnosed three realistic outage scenarios from symptom to root cause. That is the exact skill set Amazon tests in their DCT technical interviews.

Here's what you accomplished:

  • Set up a Linux server environment using WSL on Windows 11 and practiced the exact network diagnostic sequence (ip, ping, traceroute, ss, curl, dig) that Amazon DCT interviews test.
  • Diagnosed and fixed simulated outages using a structured troubleshooting flow: service status check, restart attempt, log analysis, config validation, fix, and verify.
  • Built a public GitHub portfolio with three documented incident reports that demonstrate your troubleshooting skills and documentation discipline to hiring managers.
  • Secret Mission: Diagnosed a compound multi-layer failure (service down combined with a firewall block) and documented it in STAR format for interview practice.

Ready to quiz yourself?