Investigate a Server Incident on EC2

Introduction

30 Second Summary

When a website goes down at 3am, someone needs to log into the server and figure out what went wrong. That person is a DevOps engineer, and their main weapon is a set of Linux commands.

In this project, you will launch a cloud server on AWS, SSH into it from your Windows machine, and investigate a simulated server incident using the same commands professionals use during real outages.

What You'll Build

You will type a single command pipeline into your terminal and watch it instantly rank the top attacking IP addresses from thousands of log entries, proving exactly who is causing your server's errors.

By the end of this project, you'll have:

• A live SSH session to a real EC2 cloud server you launched yourself, where running whoami proves you are connected to real AWS infrastructure.
• A command pipeline that identifies the top attackers from thousands of log lines, answering "who is flooding our server?" in a single line.
• A fixed and verified server where broken file permissions are corrected, the config is updated, and system health checks pass cleanly.
• Secret Mission: Write a bash script that automates the entire health check and schedule it with cron to run every 30 minutes.

Are there any prerequisites?

This project requires WSL 2 with Ubuntu already installed on your Windows machine, and an AWS account. If you need to set up WSL, run wsl --install in a Windows PowerShell terminal first.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

You're about to step into the shoes of a DevOps engineer who just got paged about a misbehaving server. The Linux commands and investigation techniques you'll practice here are exactly what interviewers expect you to know for Cloud and DevOps roles.

Launch an EC2 Instance and SSH In

You just got paged about a misbehaving web server. Before you can investigate anything, you need to get onto the server. Every DevOps task starts with this skill: connecting to a remote machine in the cloud.

In this step, you will launch a virtual server on AWS called an EC2 instance, then connect to it from your local WSL terminal using SSH.

In this step, get ready to:

• Launch a t2.micro EC2 instance running Amazon Linux 2023.
• Connect to it via SSH from your WSL terminal.

Launch the EC2 instance from the AWS Console

You will use the AWS Management Console to spin up a small virtual server that you can connect to over the internet.

• Open the EC2 console in your browser: https://console.aws.amazon.com/ec2/
• In the top-right corner, click the region dropdown.
• Select US East (N. Virginia) us-east-1.
• Click Launch instance (the orange button on the dashboard).
• Under Name and tags, enter linux-incident-lab in the Name field.
• Under Application and OS Images (Amazon Machine Image), click Quick Start and choose Amazon Linux.
• From the Amazon Machine Image (AMI) dropdown, select Amazon Linux 2023 (the first option).
• Under Instance type, confirm t2.micro is selected.

What is a key pair?

A key pair is how AWS authenticates your SSH connection. AWS keeps the public half of the key on the server. You keep the private half (the .pem file) on your local machine.

When you connect, SSH proves you hold the matching private key without ever transmitting it. This is more secure than a password because nobody can guess or brute-force the key.

• Under Key pair (login), click Create new key pair.
• Enter incident-lab-key as the Key pair name.
• For Key pair type, select RSA.
• For Private key file format, select .pem.
• Click Create key pair.

Your browser automatically downloads a file called incident-lab-key.pem. This is your private key. Keep it safe.

• Under Network settings, keep Create security group selected.
• Make sure Allow SSH traffic from is checked and set to My IP.
• Click Launch instance at the bottom of the page.

You will see a confirmation page. Click the instance ID link to go to the Instances page.

• Wait for the Instance state column to change from Pending to Running. This takes about 30 seconds.

What should I see?

Your Instances list now shows one row named linux-incident-lab with the state Running and a green circle icon. You also see a Public IPv4 address in the details below.

Instance stuck in Pending?

Refresh the page. It can take up to a minute for a new instance to transition to Running. If it moves to Terminated instead, your account may not have access to launch instances in this region. Check that you selected t2.micro and try again.

Help me troubleshoot my EC2 instance not launching

Connect via SSH from WSL

Your server is running in the cloud. Now you need a secure tunnel from your local machine into that server. That tunnel is SSH.

• Press the Windows key to open your search bar.
• Type Ubuntu and press Enter to open your WSL terminal.

The .pem file you downloaded is in your Windows Downloads folder. WSL can access your Windows files through the /mnt/c/ path. You need to copy it into your WSL home directory and lock down its permissions.

• Copy your key file into WSL by running this command:

cp /mnt/c/Users/[[WINUSER="your Windows username"]]/Downloads/incident-lab-key.pem ~/incident-lab-key.pem

• Lock down the file permissions by running:

chmod 400 ~/incident-lab-key.pem

Why chmod 400?

SSH refuses to use a key file that other users on the system could read. The chmod 400 command sets the file to read-only for you and no access for anyone else. Without this, your SSH connection will fail with a permissions error.

• Confirm the file exists and has the correct permissions by running:

ls -la ~/incident-lab-key.pem

You should see -r-------- at the start of the line, confirming only you can read it.

Now you need your instance's public IP address to connect. Switch back to the EC2 console.

• On the Instances page, click your linux-incident-lab instance.
• In the details panel below, find the Public IPv4 address field and copy the IP address.
• Back in your WSL terminal, connect to your instance by running:

ssh -i ~/incident-lab-key.pem ec2-user@[[EC2_IP="your EC2 public IPv4 address"]]

What does this command do?

• ssh opens a secure shell connection to a remote machine.
• -i ~/incident-lab-key.pem tells SSH to authenticate using your private key file.
• ec2-user is the default username on Amazon Linux instances.
• The IP address after @ is where to connect.

• When prompted with "Are you sure you want to continue connecting?", type yes and press Enter.

This fingerprint prompt appears the first time you connect to any new server. It confirms you trust this host.

Before you run the next command, think: what username do you expect to see if the connection succeeded?

• Verify your connection by running:

whoami

You should see ec2-user printed. This confirms you are connected to the remote Amazon Linux instance.

Connection refused or timed out?

• Double-check you copied the correct Public IPv4 address (not the Private IP).
• Make sure your security group allows SSH from My IP. If your IP changed (VPN, different network), go back to the security group in the console and update the inbound SSH rule to your current IP.
• If you see "Permission denied (publickey)", confirm you ran chmod 400 on the .pem file and that the filename matches exactly.

Help me troubleshoot my SSH connection to EC2

You are now inside your cloud server. Everything you type in this terminal runs on the remote EC2 instance, not on your local machine.

Your remote access is set up. Next, you will create the incident scenario and start exploring the server filesystem.

Set Up the Incident and Explore the Server

You are connected to a live server, but right now it is a blank slate. In a real incident, you would arrive at a server that has been running for weeks with thousands of log entries and dozens of config files.

In this step, you will set up a simulated incident environment that mimics a production web server under attack. Then you will explore the filesystem and discover why the most obvious approach to reading logs fails completely.

In this step, get ready to:

• Run the incident simulation script to create a realistic server environment.
• Navigate the filesystem using pwd, ls, cd, and find.
• Attempt to read the access log and discover why raw output is overwhelming.

Run the incident simulation script

A production server has web access logs, error logs, config files, and a directory structure built up over time. You need all of that in place before you can investigate anything.

• Download the setup script by running this command:

curl -sL https://raw.githubusercontent.com/nextwork-org/linux-incident-lab/main/setup-incident.sh -o setup-incident.sh

What does this command do?

• -s tells curl to run in silent mode, hiding the progress bar.
• -L tells curl to follow any redirects from the URL.
• -o setup-incident.sh saves the downloaded content into a file called setup-incident.sh.

• Confirm the script downloaded by running:

ls -la setup-incident.sh

You should see the file listed with its size. If the file is 0 bytes or you see an error, the URL may not be available.

File is 0 bytes or missing?

If the download failed, you can create the script manually. Open a new file in nano by running nano setup-incident.sh, then paste the setup script contents from the project's Complete Code section (the setup-incident.sh file).

Save with Ctrl+O then Enter, and exit with Ctrl+X.

Help me create the script manually on my EC2 instance

• Make the script executable by running:

chmod +x setup-incident.sh

• Run the script with elevated privileges by running:

sudo bash setup-incident.sh

You should see output that says "Setting up incident simulation..." followed by a banner reading "INCIDENT SIMULATION READY" with your mission briefing.

What did the script create?

The script built a realistic server environment on your instance. It created web server access logs in /var/log/webapp/, error logs with warnings and failures, config files in /etc/webapp/, and a simple web page in /var/www/html/.

It also planted some problems for you to find: a config file with wrong permissions, an IP address flooding login attempts, and excessive 500 errors.

Seeing "Permission denied" when running the script?

Make sure you included sudo before bash setup-incident.sh. The script creates files in system directories like /var/log/ and /etc/ which require root access.

Help me fix this permission error

Explore the server filesystem

When you SSH into an unfamiliar server during an incident, your first job is to orient yourself. Where are you, what files exist, and where are the logs?

• See your current location by running:

pwd

You should see /home/ec2-user printed. This is your home directory on the instance.

• List everything in the log directory to see what the script created by running:

ls -la /var/log/webapp/

You should see access.log and error.log listed with their sizes and permissions.

• Navigate into the log directory and inspect file sizes by running these commands:

cd /var/log/webapp
ls -lh

The -h flag shows file sizes in human-readable format (KB, MB). Notice the access log is significantly larger than the error log.

• Use find to locate all config files on the server by running:

find /etc/webapp -type f -name "*.conf"

You should see /etc/webapp/app.conf listed. This is the web application's configuration file.

• Find all log files modified today by running:

find /var/log -type f -name "*.log" -mtime -1

What is the find command doing?

• find /etc/webapp searches starting from that directory.
• -type f limits results to files only (not directories).
• -name "*.conf" matches filenames ending in .conf.
• -mtime -1 finds files modified in the last 24 hours.

On unfamiliar servers, find is your best tool for answering "where is that config file?" without knowing the directory structure in advance.

Try reading the access log

You know the access log exists and it is large. The most intuitive way to read a file is cat, which prints the entire file contents to your screen.

Before you run this, what do you think will happen when you try to read the entire access log with cat?

• Try reading the access log by running:

cat access.log

Hundreds of lines just flooded your screen. The output scrolled past faster than you could read any of it.

Why is this a problem?

During a real incident, you need to find specific errors in thousands of log lines. Dumping everything with cat gives you no way to search, filter, or limit the output.

You cannot scroll back far enough to see the beginning. You cannot find the 500 errors hiding among the 200 successes. You need surgical tools, and that is exactly what the next step provides.

You have set up a realistic incident environment and explored the server filesystem. You have also hit the wall that every new engineer hits: cat is useless for investigating large log files. Next up, you will learn grep, tail, and head to slice through logs like a pro.

Hunt for Errors with grep, tail, and head

In the last step, you tried reading the access log with cat and watched hundreds of lines flood your screen. You could not find anything useful that way.

Real DevOps engineers never read raw logs top-to-bottom. They use surgical tools that show only what matters. In this step, you will learn grep to find specific patterns, tail to see recent entries, and head to limit output to a manageable amount.

In this step, get ready to:

• Control log output with tail, head, and wc so you only see what you need.
• Use grep flags to find, count, and show context around specific errors.
• Combine commands with pipes to filter by time and method.

Use tail and head to control output

Instead of dumping an entire file, tail shows lines from the end and head shows lines from the beginning. Combined with wc (word count), you can get a quick picture of any log file without the flood.

• See the last 20 lines of the access log by running this command:

tail -n 20 /var/log/webapp/access.log

You should see exactly 20 lines of log entries. That is a manageable amount to read.

• See the first 10 lines of the log by running:

head -n 10 /var/log/webapp/access.log

• Count the total number of lines in the file by running:

wc -l /var/log/webapp/access.log

You should see a number in the thousands. That is how many lines cat dumped on your screen in the previous step.

What do these commands do?

• tail -n 20 shows only the last 20 lines. Since logs are written chronologically, this shows the most recent entries.
• head -n 10 shows only the first 10 lines. Useful for seeing the format of a file without reading the whole thing.
• wc -l counts lines. Now you know the scale of what you are dealing with.

There is one more trick. You can watch a log file in real time as new entries appear.

• Follow the log live by running:

tail -f /var/log/webapp/access.log

Your terminal will appear to freeze. It is waiting for new log entries to appear. Since this is a simulated log, nothing new will arrive.

• Press Ctrl+C to stop following the log and get your prompt back.

When to use tail -f

On a live production server, tail -f lets you watch requests arrive in real time. It is one of the first things a DevOps engineer runs during an active incident.

Use grep to find specific errors

Controlling how many lines you see is helpful, but you still need to find specific patterns in thousands of lines. grep searches for a text pattern and shows only the lines that match.

• Find all lines containing 500 errors by running:

grep "500" /var/log/webapp/access.log

You should see a focused list of just the lines containing 500 status codes. No more flood.

But how many 500 errors are there total? Before you run the next command, predict: do you think there are more or fewer than 50?

• Count the exact number of 500 errors by running:

grep -c "500" /var/log/webapp/access.log

You should see a single number printed. That is how many lines contain "500" in the access log. Compare that to the wall of text cat gave you. One number tells you more than thousands of scrolling lines ever could.

What does the -c flag do?

The -c flag tells grep to count matching lines instead of printing them. It turns grep from a search tool into a counting tool.

Now let's look at the error log, which uses severity labels like ERROR and CRITICAL.

• Search for all error entries (case-insensitive) by running:

grep -i "error" /var/log/webapp/error.log

The -i flag makes the search case-insensitive. It matches "ERROR", "Error", and "error" all at once.

• Find CRITICAL entries with 2 lines of context above and below by running:

grep -B 2 -A 2 "CRITICAL" /var/log/webapp/error.log

You should see the CRITICAL lines with surrounding context. This tells you what was happening right before and after each critical event.

What do -B and -A mean?

• -B 2 means "show 2 lines Before each match." Think B for Before.
• -A 2 means "show 2 lines After each match." Think A for After.
• Context is critical during incidents. You need to see what triggered the failure, not just the failure itself.

• Search recursively across all log files for "denied" by running:

grep -rn "denied" /var/log/webapp/

You should see results showing the filename, line number, and matching text. The -rn flags search recursively through all files in the directory and print line numbers alongside each match.

Not seeing any results?

Check the file path. The trailing slash in /var/log/webapp/ tells grep to search the directory. A typo in the path returns nothing without an error message.

help me troubleshoot why grep is returning no output

Narrow down the timeframe

You know there are errors. Now you need to narrow the investigation to a specific time window. Pipes let you chain commands together, sending the output of one command as input to the next.

• Find errors from the 03:00 hour (when most problems occurred) by running:

grep "03:0" /var/log/webapp/error.log

You should see a cluster of errors concentrated in the 03:00 hour. That is your incident window.

• Combine two grep commands with a pipe to find only POST requests that returned 500 by running:

grep "500" /var/log/webapp/access.log | grep "POST"

The pipe (|) takes all 500-error lines from the first grep and feeds them into the second grep, which keeps only lines that also contain "POST". Two filters in one command.

• Exclude all successful (200) requests to see only non-200 responses by running:

grep -v "200" /var/log/webapp/access.log | head -20

You should see 20 lines of non-200 responses. The -v flag inverts the match, showing everything that does NOT contain "200".

Why use pipes?

Pipes are the core of Linux command-line power. Each command does one small thing well, and pipes chain them together. You will use this pattern constantly in the next step to build multi-stage analysis pipelines.

Before you move on, confirm your grep skills are working. What do you expect to see if you count 500 errors one more time?

• Verify you can extract a precise count by running:

grep -c "500" /var/log/webapp/access.log

You should see the same count as before. One clean number, extracted from thousands of lines in under a second. That is the difference between flailing and investigating.

What grep flags did you learn?

• -c counts matches instead of printing them.
• -i makes the search case-insensitive.
• -B and -A show lines Before and After each match for context.
• -rn searches recursively and shows line numbers.
• -v inverts the match (shows lines that do NOT match).

You can now slice through thousands of log lines and extract exactly what you need. Next up, you will build powerful analysis pipelines with awk and multi-stage pipes to answer the big question: who is causing these errors?

Deep Analysis with Pipes and awk

You now know there are 500 errors in the access log. But that alone does not close an incident. You need to answer two critical questions: who is causing them, and what is the most common failure pattern.

grep finds matching lines, but it dumps entire lines. You need to extract specific columns (like just the IP address) and rank them by frequency. This is where awk and pipes come in. Chaining commands with pipes is the technique senior DevOps engineers use daily during incidents.

In this step, get ready to:

• Extract specific fields (IPs, status codes, URLs) from log entries using awk.
• Build a pipeline that ranks the top offending IP addresses by request count.
• Correlate findings to confirm which IP is the attacker causing 500 errors.

Extract specific fields with awk

Each line in the access log has multiple fields separated by spaces: the IP address, the timestamp, the request method, the URL, and the status code. awk lets you pull out a single field by its position number.

• Print just the IP addresses (field 1) from the access log by running:

awk '{print $1}' /var/log/webapp/access.log

You see a long list of IPs scrolling by. Every single request is represented, but you cannot tell which IP is most frequent just by eyeballing it.

• Now print both the IP and the status code (fields 1 and 9) and limit to the first 10 lines by running:

awk '{print $1, $9}' /var/log/webapp/access.log | head -10

Now you can see which IP got which status code. You should spot some 200, 500, and 401 codes paired with different IPs.

• Filter to show only the IPs that got 500 errors by running:

awk '$9 == 500 {print $1}' /var/log/webapp/access.log

How does awk field selection work?

awk splits each line by whitespace. $1 is the first field (the IP address), $9 is the ninth field (the HTTP status code in this log format).

The pattern $9 == 500 {print $1} means "if field 9 equals 500, print field 1." This is how you surgically filter and extract in one command.

You see a list of IPs that caused 500 errors, but you still cannot tell which one is the worst offender. You need to count and rank them.

Build the DevOps power pipeline

A pipe (|) takes the output of one command and feeds it as input to the next. By chaining multiple commands together, you build a pipeline that transforms raw data into ranked answers in a single line.

Before you run the next command, think about this: do you expect all IPs to have roughly equal request counts, or will one IP stand out dramatically above the rest?

• Find the top 5 requesting IPs by running this pipeline:

awk '{print $1}' /var/log/webapp/access.log | sort | uniq -c | sort -rn | head -5

You should see 203.0.113.42 sitting at the top with far more requests than any other IP. That is your prime suspect.

What does each part of the pipeline do?

• awk '{print $1}' extracts just the IP addresses.
• sort groups identical IPs next to each other (required for uniq to work).
• uniq -c counts consecutive duplicates.
• sort -rn sorts numerically in reverse (highest count first).
• head -5 shows only the top 5 results.

This exact pipeline pattern works for any "top N" question. Swap the awk extraction and you can rank URLs, error messages, user agents, or anything else.

• Find which URLs are getting the most 500 errors by running:

awk '$9 == 500 {print $7}' /var/log/webapp/access.log | sort | uniq -c | sort -rn | head -5

You should see /api/login at the top. The login endpoint is where most 500 errors are happening.

• Find the top error messages in the error log by running:

grep "ERROR" /var/log/webapp/error.log | awk -F': ' '{print $2}' | sort | uniq -c | sort -rn | head -5

What does -F do in awk?

The -F': ' flag tells awk to split on ": " (colon-space) instead of whitespace. This lets you extract the error message portion after the severity label and timestamp.

You should see "Authentication failed" as the most common error message. The picture is forming: someone at 203.0.113.42 is hammering the login endpoint and failing repeatedly.

Seeing different results?

Make sure you are running commands against /var/log/webapp/access.log (not the error log) for IP and URL queries, and /var/log/webapp/error.log for error message queries. Check for typos in the file path.

help me debug my awk pipeline output

Correlate findings to confirm the attacker

You have a prime suspect (203.0.113.42) and a top failing endpoint (/api/login). Now you need to confirm: is that specific IP actually responsible for the 500 errors, or is it just making a lot of successful requests?

• Check what status codes the suspect IP is getting by running:

grep "203.0.113.42" /var/log/webapp/access.log | awk '{print $9}' | sort | uniq -c | sort -rn

You should see a large number of 500 status codes from this single IP. The correlation is confirmed.

• Check what hour the attack was most intense by running:

grep "203.0.113.42" /var/log/webapp/access.log | awk '{print $4}' | cut -d: -f2 | sort | uniq -c | sort -rn

You should see hour 03 at the top with a spike of requests. The attacker flooded the server around 3 AM.

What does cut -d: -f2 do?

The timestamp in the log looks like [10/Aug/2026:03:15:22. The cut -d: -f2 splits on the colon character and takes the second field, which is the hour (03). Combined with the counting pipeline, this gives you requests per hour.

Your investigation is complete. You have confirmed that IP 203.0.113.42 launched a brute-force attack against /api/login around 3 AM, causing a flood of 500 errors and failed authentication attempts.

Memorize the power pipeline

The pattern awk | sort | uniq -c | sort -rn | head answers any "top N" question during an incident. Swap the awk field to rank IPs, URLs, error messages, user agents, or anything else in your logs.

You have identified the root cause. Next up, you will fix the server by repairing broken file permissions, editing the config to enable rate limiting, and verifying the system is healthy.

Fix the Server and Verify Health

Your investigation uncovered three problems. A suspicious IP is flooding requests, the config file has broken permissions, and rate limiting is disabled.

Now you fix the problems and verify the server is healthy. This completes the incident response cycle: detect, investigate, remediate, confirm.

In this step, get ready to:

• Fix broken file permissions with chmod and chown.
• Edit the server configuration using nano and sed.
• Verify system health with disk, memory, process, and network commands.

Fix file permissions

Remember the error log entry that said Permission denied: /etc/webapp/app.conf? The config file is unreadable because it is owned by root with restrictive permissions. You need to read this file to investigate and fix the server configuration.

Before you run the next command, what do you think will happen when ec2-user tries to read a file owned by root with 600 permissions?

• Try to read the config file by running:

cat /etc/webapp/app.conf

You see cat: /etc/webapp/app.conf: Permission denied. That is the broken permission problem your error log flagged. The file exists, but your user cannot read it.

• Check who owns the file and what permissions are set by running:

ls -la /etc/webapp/app.conf

You see -rw------- with owner root root. Only root can read or write this file. Your ec2-user account has zero access.

What do these permissions mean?

The -rw------- string breaks into three groups of three: owner (rw-), group (---), and others (---). Only the owner (root) has read and write access. Everyone else is locked out.

The numeric equivalent is 600: 6 for owner (read+write), 0 for group, 0 for others.

• Fix the ownership so ec2-user owns the file by running:

sudo chown ec2-user:ec2-user /etc/webapp/app.conf

• Set the permissions to 640 (owner read+write, group read, others nothing) by running:

sudo chmod 640 /etc/webapp/app.conf

• Now try reading the file again:

cat /etc/webapp/app.conf

This time the file contents print to your screen. You can see log_level=debug and enable_rate_limiting=false. Both of these need fixing.

• Confirm the permission fix by running:

ls -la /etc/webapp/app.conf

You should see -rw-r----- with ec2-user ec2-user as the owner and group. The permission problem is fixed.

Still seeing Permission denied?

Make sure you included sudo before both chown and chmod. Without sudo, your ec2-user does not have permission to change ownership of a root-owned file.

Also check for typos in the file path. It must be /etc/webapp/app.conf exactly.

help me fix the Permission denied error on /etc/webapp/app.conf

Edit the config with nano and sed

The config has two dangerous settings. Debug-level logging floods the disk with noise, and rate limiting is turned off, leaving the server wide open to the attacker flooding requests.

• Open the config file in the nano text editor by running:

nano /etc/webapp/app.conf

What is nano?

Nano is a simple terminal text editor. It opens files directly in your SSH session so you can edit them without leaving the terminal. The commands are shown at the bottom of the screen. The ^ symbol means the Ctrl key.

• Find the line that says log_level=debug.
• Delete debug and type warn so the line reads log_level=warn.
• Press Ctrl+O to save, then press Enter to confirm the filename.
• Press Ctrl+X to exit nano.

• Verify your change took effect by running:

grep "log_level" /etc/webapp/app.conf

You should see log_level=warn printed. The noisy debug logging is disabled.

Now try the faster approach. The sed command edits files in place without opening an editor. This is how experienced engineers make single-line config changes over SSH.

• Enable rate limiting with a single sed command:

sed -i 's/enable_rate_limiting=false/enable_rate_limiting=true/' /etc/webapp/app.conf

What does this command do?

The sed -i flag means "edit in place" (modify the file directly). The s/old/new/ pattern tells sed to substitute the first string with the second. No editor needed.

• Verify the rate limiting change by running:

grep "rate_limiting" /etc/webapp/app.conf

You should see enable_rate_limiting=true. Both config fixes are now in place.

Seeing the old value still?

If grep still shows false, check that your sed command used enable_rate_limiting=false exactly as written. Any extra spaces or typos in the search string will cause sed to silently do nothing.

Run cat /etc/webapp/app.conf to see the full file and confirm what the current value is.

help me fix the sed command for /etc/webapp/app.conf

Check system health

The config is fixed. The final step in any incident response is confirming the system itself is not in distress. You need to check disk space, memory, running processes, and network ports.

Before you run these commands, what do you expect to see for a healthy server? Think about what values would concern you versus reassure you.

• Check disk usage by running:

df -h

You see a table of filesystems with their size, used space, and available space in human-readable format. A healthy server has its main partition well below 90% usage.

• Check memory usage by running:

free -h

This shows total, used, and available memory. On a t2.micro instance you have about 1 GB total.

• Check the top CPU-consuming processes by running:

ps aux --sort=-%cpu | head -5

• Check the top memory-consuming processes by running:

ps aux --sort=-%mem | head -5

What does ps aux show?

The ps aux command lists all running processes with their CPU and memory usage. The --sort=-%cpu flag sorts them so the hungriest processes appear first. Piping to head -5 limits the output to the top 5 lines (including the header row).

• Check what is listening on network ports by running:

ss -tulpn

This shows all TCP and UDP sockets currently listening. You can see which processes are bound to which ports. If something unexpected were running on port 8080 or 443, you would spot it here.

What do the ss flags mean?

Each flag targets a specific angle: -t shows TCP sockets, -u shows UDP, -l shows only listening sockets, -p shows the process using each socket, and -n shows port numbers instead of service names.

Your server is healthy. Disk is not full, memory is not exhausted, no runaway processes are consuming all CPU, and no unexpected services are listening on open ports. Incident resolved.

Seeing high CPU or disk usage?

If df -h shows a partition above 90%, the log simulation may have filled it. Run sudo rm -f /var/log/webapp/access.log and re-run the setup script to regenerate clean logs.

If a process is consuming high CPU, identify it from the ps aux output and investigate. On this lab instance, all processes should be lightweight.

help me diagnose high resource usage on my EC2 instance

The incident is closed. You diagnosed the root cause with grep and awk, fixed permissions with chmod and chown, hardened the config with nano and sed, and confirmed system health. Next up, you will automate this entire health check with a bash script so the server monitors itself.

Secret mission

Automate the Health Check with a Bash Script

You have investigated the incident manually. But what if you could automate the entire health check and have it run on a schedule, without you lifting a finger? Write a bash script that does it all for you, then schedule it with cron so the server monitors itself.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses a t2.micro EC2 instance that costs approximately $0.0116 per hour while running.

Cost warning

Your EC2 instance charges by the second while it is in the running state. If you leave it running overnight (8 hours), that is about $0.09. If you forget about it for a month, that is about $8.50.

If you are done with the project, terminate the instance now to stop all charges immediately.

Resources you used:

• 1x EC2 t2.micro instance (linux-incident-lab) in us-east-1.
• 1x Key pair (incident-lab-key).
• 1x Security group (SSH inbound rule on port 22).
• 1x Local .pem file (~/incident-lab-key.pem in WSL).

Keep everything running

No action needed. Choose this if you are still actively experimenting with Linux commands on your EC2 instance and plan to come back within the next few hours.

• Your instance continues to cost approximately $0.0116 per hour.
• Your SSH connection will time out eventually. Reconnect with the same ssh command from your WSL terminal when you return.
• Your cron job will keep running and appending to /home/ec2-user/health-reports.log every 30 minutes.

Pause - I'll come back to this later

Stop the instance to halt compute charges while keeping your files and configuration intact. Storage costs remain at approximately $0.08/month for the default 8GB EBS volume.

• Return to the EC2 console at https://console.aws.amazon.com/ec2/.
• In the left navigation pane, select Instances.
• Select the checkbox next to linux-incident-lab.
• Click the Instance state dropdown menu.
• Select Stop instance and confirm by clicking Stop.
• When you want to come back, select the instance and choose Instance state > Start instance. Note that the public IP address will change, so you will need to update your SSH command.

Delete - I don't want to use this again

Remove all project resources and stop all charges. This is recommended if you have finished the project.

Terminate the EC2 instance:

• Return to the EC2 console at https://console.aws.amazon.com/ec2/.
• In the left navigation pane, select Instances.
• Select the checkbox next to linux-incident-lab.
• Click the Instance state dropdown menu.
• Select Terminate (delete) instance and confirm by clicking Terminate (delete).

Delete the key pair:

• In the left navigation pane, under Network & Security, select Key Pairs.
• Select the checkbox next to incident-lab-key.
• Click Actions, then Delete.
• Type Delete in the confirmation field and click Delete.

Delete the security group:

• In the left navigation pane, under Network & Security, select Security Groups.
• Select the security group you created for this project (not the default one).
• Click Actions, then Delete security groups.
• Confirm by clicking Delete.

Delete the local .pem file from WSL:

• Open your Ubuntu WSL terminal and run:

rm ~/incident-lab-key.pem

• Confirm deletion by running ls ~/incident-lab-key.pem and verifying you see a "No such file or directory" message.

Nice Work!

Nice Work!

You just investigated a server incident on real AWS infrastructure using the same Linux commands DevOps engineers use every day. From SSHing into a remote machine to identifying an attacker IP and automating your monitoring, you handled the full incident response lifecycle.

You've learned how to:

• Launch an EC2 instance and connect via SSH using key-based authentication from your WSL terminal, proving you can access remote cloud servers.
• Hunt through thousands of log lines using grep and build the DevOps power pipeline (awk | sort | uniq -c | sort -rn) to identify the top offending IP addresses in seconds.
• Diagnose and fix broken file permissions with chmod and chown, edit server configs with nano and sed, and verify system health with df, free, ps, and ss.
• Secret Mission: Write an automated bash script using variables, command substitution, and conditionals, then schedule it with cron to run every 30 minutes.

Ready to quiz yourself?