Build a Linux Server Health Monitor

Diagnose real Linux server problems and automate fixes in bash.

Introduction

30 Second Summary

When a website goes down or gets slow, someone has to figure out what went wrong on the server. They look for bad programs eating up resources, search through thousands of Linux log lines for clues, and check that services are still running.

In this project, you will start up a fake broken server using Docker, then find and fix each problem using the same commands that real DevOps engineers use at work.

What You'll Build

You will start a broken server, fix four real problems one by one, and then run one command that prints a color-coded health report you can show to anyone.

By the end of this project, you'll have:

  • A Docker-based Linux lab you can break and rebuild in seconds to practice fixing problems any time you want.
  • The skills to fix a slow server like a DevOps engineer: find the bad process using all the CPU, pull out error patterns from 5000+ log lines using grep and awk, free up wasted disk space, fix unsafe file permissions, and check that network services are working.
  • A bash health-check script you can put on your CV that prints a color-coded health report with one command.
  • Secret Mission: Build the health-check script from scratch with strict mode, functions, threshold logic, and exit codes.

Are there any prerequisites?

You should know basic terminal commands (ls, cd, pwd) and basic programming ideas (variables, loops, if/else) in any language. You also need Docker Desktop installed and running on your Mac.

Before We Start

Before We Start...

Before you dive into building, take a moment to lock in what you're creating and why it matters. Think about what makes a server "healthy" and what a DevOps team would want automatically checked every hour.

Launch Your Linux Server Lab

You need a real Linux environment to practice DevOps troubleshooting on. Your Mac Terminal is great for running commands, but production servers run Linux.

Docker gives you a disposable Ubuntu server you can break and rebuild without any risk to your Mac. In this step, you will launch a container that looks and feels like a production server, install diagnostic tools, and seed it with real problems to solve.

In this step, get ready to:
  • Launch an interactive Ubuntu container that simulates a production server.
  • Install a full DevOps diagnostic toolkit (nginx, process tools, networking utilities).
  • Seed the server with four realistic problems to diagnose in later steps.
Pull and launch an interactive Ubuntu container

Docker containers start from images. You will pull the official Ubuntu 26.04 LTS image and launch an interactive bash session inside it.

  • Launch your Linux server lab by running this command in your Mac Terminal:
docker run -it --hostname prodserver --name devops-lab ubuntu:26.04 bash

What does this command do?

  • docker run creates and starts a new container from the specified image.
  • -it makes the session interactive with a terminal attached, so you can type commands inside.
  • --hostname prodserver sets the container's hostname so your shell prompt looks like a real production server.
  • --name devops-lab gives the container a friendly name so you can easily restart it later.
  • ubuntu:26.04 is the official Ubuntu LTS image from Docker Hub.
  • bash launches a bash shell as the container's main process.

Your terminal prompt changes to something like root@prodserver:/#. You are now inside a Linux server.

  • Verify you are inside Ubuntu by running:
cat /etc/os-release

You should see output containing Ubuntu 26.04 and the codename Resolute.

Seeing 'docker: command not found'?

  • Make sure Docker Desktop is running on your Mac. Check for the Docker whale icon in your menu bar.
  • If Docker Desktop is open but you still see the error, close and reopen your Mac Terminal to refresh the PATH.
  • help me fix 'docker command not found' on my Mac
Install the DevOps diagnostic toolkit

A fresh Ubuntu container is minimal. It has almost no tools installed. You need to install the same diagnostic utilities a real production server would have.

  • Install the required packages by running this command inside the container:
apt-get update && apt-get install -y nginx procps iproute2 curl vim htop

What are these packages?

  • nginx is a web server you will troubleshoot in later steps.
  • procps provides ps, top, and free for process monitoring.
  • iproute2 provides ss for network socket inspection.
  • curl tests HTTP connections.
  • vim is the text editor you will use to write scripts.
  • htop is a color-coded process monitor.

The install takes about 30 seconds. You will see a stream of package downloads.

  • Start the nginx web server by running:
nginx
  • Verify nginx is responding by running:
curl -s http://localhost:80 | head -3

You should see the first few lines of HTML, starting with <!DOCTYPE html>.

Seeing 'Connection refused'?

  • Make sure you ran the nginx command first. It produces no output on success, which is normal.
  • Try running nginx again. If you see address already in use, it means nginx is already running and something else went wrong with the curl command.
  • help me troubleshoot nginx not responding inside my Docker container
Create the problem simulation script

Now you will create a script that seeds your server with four realistic production problems. This transforms your clean Ubuntu container into a broken server that needs diagnosing.

  • Open a new file in vim by running:
vim /root/simulate-problems.sh

New to vim?

Vim is a terminal text editor. You only need three commands for this project.

  • Press i to enter insert mode (so you can type or paste text).
  • Press Esc to exit insert mode.
  • Type :wq and press Enter to save and quit.
  • Press i to enter insert mode.
  • Paste the first section of the script (the header and INFO log generation):
#!/usr/bin/env bash
set -euo pipefail

echo "=== Seeding DevOps Troubleshooting Lab ==="

# Create application log directory
mkdir -p /var/log/myapp

# Generate a large log file with mixed content
echo "Generating application logs..."
> /var/log/myapp/app.log

for i in $(seq 1 5000); do
    printf "2026-07-13 %02d:%02d:%02d INFO  Request processed successfully from 192.168.1.%d\n" \
        $((RANDOM % 24)) $((RANDOM % 60)) $((RANDOM % 60)) $((RANDOM % 255)) \
        >> /var/log/myapp/app.log
done

What does this section do?

  • set -euo pipefail enables strict mode. The script exits immediately if any command fails, uses an unset variable, or has a pipeline failure.
  • The for loop generates 5000 fake INFO log entries with random timestamps. This creates the bulk of the log file you will analyze in a later step.
  • Continue pasting the next section directly below (stay in insert mode). This adds ERROR and WARN entries, shuffles the log, creates disk-filling files, and starts a rogue process:

for i in $(seq 1 200); do
    printf "2026-07-13 %02d:%02d:%02d ERROR Database connection timeout after 30s - host=db-primary.internal port=5432\n" \
        $((RANDOM % 24)) $((RANDOM % 60)) $((RANDOM % 60)) \
        >> /var/log/myapp/app.log
done

for i in $(seq 1 100); do
    printf "2026-07-13 %02d:%02d:%02d WARN  Memory usage above 80%% threshold\n" \
        $((RANDOM % 24)) $((RANDOM % 60)) $((RANDOM % 60)) \
        >> /var/log/myapp/app.log
done

# Shuffle log entries to simulate real interleaved logging
sort -R /var/log/myapp/app.log -o /var/log/myapp/app.log

# Create disk-hogging files
echo "Creating disk pressure..."
dd if=/dev/zero of=/var/log/myapp/debug-old.log bs=1M count=100 2>/dev/null
dd if=/dev/zero of=/tmp/core-dump-001 bs=1M count=50 2>/dev/null
dd if=/dev/zero of=/tmp/core-dump-002 bs=1M count=50 2>/dev/null

# Start a rogue CPU-hogging process
echo "Starting rogue process..."
while true; do : ; done &
echo "Rogue process started with PID: $!"

What does this section do?

  • 200 ERROR lines and 100 WARN lines get mixed into the log. The sort -R shuffles all entries randomly so they look like a real interleaved log file.
  • dd creates three large junk files (100MB + 50MB + 50MB) that waste disk space.
  • while true; do : ; done & launches an infinite loop in the background that consumes CPU. The : is a bash no-op that executes as fast as possible.
  • Paste the final section directly below. This sets up permission problems and prints the lab summary:

# Set up files with incorrect permissions
echo "Setting up permission problems..."
mkdir -p /opt/myapp/scripts
echo "DB_HOST=db-primary.internal" > /opt/myapp/config.env
echo "DB_PASS=supersecret123" >> /opt/myapp/config.env
chmod 777 /opt/myapp/config.env

printf '#!/usr/bin/env bash\necho "Deploying application..."\n' > /opt/myapp/scripts/deploy.sh
chmod 644 /opt/myapp/scripts/deploy.sh

echo ""
echo "=== Lab Ready ==="
echo "Your production server has problems:"
echo "  [1] A process is consuming excessive CPU"
echo "  [2] Application logs contain errors to investigate"
echo "  [3] Disk space is being wasted by large junk files"
echo "  [4] File permissions are misconfigured (security risk)"
echo ""
echo "Diagnose and fix each issue. Good luck!"

What does this section do?

  • chmod 777 on config.env makes a sensitive file (containing a database password) readable and writable by everyone. This is a security risk you will fix later.
  • chmod 644 on deploy.sh removes execute permission from a script that needs to run. You will fix this too.
  • The final echo statements print a summary of all four problems you will diagnose.
  • Press Esc to exit insert mode.
  • Type :wq and press Enter to save the file and quit vim.
  • Make the script executable and run it:
chmod +x /root/simulate-problems.sh
./root/simulate-problems.sh

The script takes about 10 seconds to run. You will see progress messages as it creates each problem.

At the end, you should see the === Lab Ready === banner listing all four problem categories.

✔️ Awesome, I've got everything!

Great. Make sure you see all four problems listed in the output before moving on.

ⓧ I'd like to double check the full code

Scroll up and confirm your script contains these three sections in order:

  • Section 1: The shebang line, strict mode, log directory creation, and the INFO loop (5000 iterations).
  • Section 2: The ERROR loop (200 iterations), WARN loop (100 iterations), log shuffle, disk-filling dd commands, and the rogue CPU process.
  • Section 3: Permission setup for config.env and deploy.sh, plus the final output messages.

If any section is missing or the script failed, delete the file with rm /root/simulate-problems.sh and start the vim step again.

Script errored or vim issues?

  • If you see permission denied, make sure you ran chmod +x before trying to execute the script.
  • If vim is confusing, you can use :q! to quit without saving and try again.
  • If the script shows a syntax error, the most common cause is missing a closing done for one of the for loops. Delete and recreate the file.
  • help me fix my simulate-problems.sh script

Before you run the next verification, predict what you expect to see when you run top. Do you think the CPU will look busy or idle?

  • Run top to see the real-time system view:
top

You should see a process consuming nearly 100% CPU. This is the rogue process your script created. Press q to exit top.

Your production server is now broken in four different ways. In the next step, you will hunt down the CPU-hogging process and terminate it using Linux signals.

Hunt and Kill the Rogue Process

Your Linux lab is running with four seeded problems waiting to be diagnosed. The first crisis to tackle is the most urgent one.

Something is consuming all available CPU on your production server, making it unresponsive. This is the number one fresher-level DevOps interview question: "How would you troubleshoot a slow server?"

In this step, get ready to:
  • Identify the rogue process using real-time monitoring tools and sorted snapshots.
  • Terminate the process safely using Linux signals (SIGTERM and SIGKILL).
  • Verify the server has recovered by checking CPU usage and load averages.
Spot the rogue process with top, htop, and ps

When a server is slow, a DevOps engineer starts with a real-time overview before drilling into specifics. You have three tools for this job, each giving you a different angle on the problem.

  • See the real-time system overview by running:
top

What am I looking at?

The header shows three load averages (1-minute, 5-minute, 15-minute). Below that, each row is a running process.

Look at the %CPU column. You should spot one process consuming nearly 100% CPU. Note its PID (the number in the first column) and its COMMAND (the last column, which shows bash for our rogue infinite loop).

  • Press q to exit top.
  • Try the color-coded alternative by running:
htop

You can see CPU bars at the top and the high-CPU process highlighted. Press q to exit.

  • Get a sorted snapshot that you can copy and reference by running:
ps aux --sort=-%cpu | head -10

This shows the top 10 processes ranked by CPU usage. The rogue process appears at the top, consuming nearly all available CPU.

  • Note the PID number of the rogue process (the number in the second column). You will need this in the next substep.

Interview approach

"I start with top for the real-time overview, then ps aux --sort=-%cpu for the exact PID." This two-step approach shows you understand both live monitoring and point-in-time snapshots.

Terminate the process using signals

Linux uses signals to communicate with processes. The polite approach is to ask the process to shut down gracefully before forcing it.

  • Send SIGTERM (signal 15) to the rogue process by running:
kill [[PID="the PID number you noted above"]]

What is SIGTERM?

SIGTERM (signal 15) is the default signal sent by kill. It asks the process to shut down gracefully, giving it a chance to close connections and flush buffers.

  • Wait 2-3 seconds, then check if the process is gone by running:
ps aux | grep [[PID="the same PID number"]]

If you only see the grep command itself in the output (not the original bash process), the rogue process is dead. If the rogue process still appears, it ignored SIGTERM and you need to escalate.

  • If the process persists, force-kill it with SIGKILL (signal 9) by running:
kill -9 [[PID="the same PID number"]]

Why try SIGTERM before SIGKILL?

SIGTERM lets the process close connections, flush buffers, and release resources cleanly. SIGKILL (signal 9) is a forced termination. The process gets no chance to clean up.

Always try SIGTERM first. SIGKILL is the last resort because it can leave orphaned child processes, corrupt files, or leak resources.

💡 Interview approach

"Always try SIGTERM first. It lets the process close connections and flush buffers. SIGKILL is the last resort because the process gets no chance to clean up."

Verify the server has recovered

Killing a process is only half the job. A DevOps engineer always confirms the system has actually recovered by checking CPU usage and load averages.

  • Before you run the next command, predict: do you think any process will still show abnormally high CPU usage?
  • Check the current CPU state by running:
ps aux --sort=-%cpu | head -5

No process should be above 10% CPU now. The rogue process is gone.

  • Check the system load averages by running:
uptime

You see three load averages at the end of the output: 1-minute, 5-minute, and 15-minute. The 1-minute average drops fastest after killing the rogue process.

  • Check how many CPU cores you have by running:
nproc

What is a healthy load average?

A healthy load average is below the number of CPU cores. If nproc reports 2 cores, a load average below 2.0 means the server is keeping up with demand.

The 1-minute average responds fastest to changes. The 15-minute average takes time to settle because it represents a longer history.

🙋‍♀️ Still seeing high CPU?

If ps aux --sort=-%cpu still shows a bash process at high CPU, your SIGTERM may not have worked. Run kill -9 with that PID to force-kill it.

If you see a different PID than the one you killed, the simulation script may have been run more than once, spawning multiple rogue processes. Kill each one individually.

Help me figure out why the rogue process won't die.

The CPU crisis is resolved. But that rogue process was just a symptom. Next up, you will dig into the application logs to find the root cause of what was going wrong on this server.

Investigate the Logs

You killed the rogue process, but that was just treating a symptom. The application logs hold the actual root cause of what went wrong on this server.

The problem is that /var/log/myapp/app.log has over 5000 lines. You cannot read that by scrolling. You need grep, awk, and pipes to extract signal from noise.

In this step, get ready to:
  • Learn safe techniques for exploring large log files without flooding your terminal.
  • Filter and count errors using grep.
  • Build a full analysis pipeline that reveals the root cause error pattern.
Explore the log file safely

Your first instinct with an unfamiliar log file might be to open it. Let's see what happens when you try.

  • Try displaying the log file with cat by running:
cat /var/log/myapp/app.log

Your terminal floods with thousands of lines scrolling past faster than you can read. Press Ctrl+C to cancel it.

Why never cat a large log file?

The cat command dumps the entire file to your screen at once. For a 5000+ line file, this is useless. You cannot read it, you cannot search it, and it fills your terminal buffer so you lose context from previous commands.

DevOps engineers never use cat on large logs. Instead they use targeted tools: head to learn the format, tail -f for real-time monitoring, and grep to filter.

  • Check how many lines the log file actually has by running:
wc -l /var/log/myapp/app.log

You should see a number around 5300 lines. Now you know the scale of what you are dealing with.

  • Peek at the first 5 lines to understand the log format by running:
head -5 /var/log/myapp/app.log

You can see the format: a date, a time, a log level (INFO, ERROR, or WARN), and then the message. This pattern matters for the awk commands you will run later.

  • View the 10 most recent log entries by running:
tail -10 /var/log/myapp/app.log

This shows the last 10 lines of the file. In a running system, the most recent entries are usually the most relevant.

  • Practice real-time log monitoring with tail -f by running:
tail -f /var/log/myapp/app.log

The -f flag means "follow." In production, new log lines appear live as the application writes them. Since your log file is static, you will just see the last few lines and a waiting cursor. Press Ctrl+C to stop.

The DevOps interview answer

If an interviewer asks "How would you investigate application logs?", your answer is: "Never cat a large log. I use head to learn the format, tail -f for real-time, and grep to filter."

Filter errors with grep

You know the log file has a mix of INFO, ERROR, and WARN entries. To find the root cause, you need to isolate just the errors.

  • Filter the log to show only ERROR lines by running:
grep "ERROR" /var/log/myapp/app.log

Now you see only the error lines. That is already much more manageable than 5000+ lines of everything.

  • Count exactly how many errors exist by running:
grep -c "ERROR" /var/log/myapp/app.log

You should see 200. That is a lot of errors in one log file. The -c flag tells grep to count matching lines instead of displaying them.

  • Look at the first few error lines to see the pattern by running:
grep "ERROR" /var/log/myapp/app.log | head -3

What is the pipe doing here?

The | character (the pipe) takes the output of one command and feeds it as input into the next. Here, grep finds all ERROR lines, then head -3 shows only the first three of those matches.

Pipes are one of the most powerful concepts in Linux. They let you chain simple tools together into complex analysis workflows.

  • Try combining tail -f with grep for live error monitoring by running:
tail -f /var/log/myapp/app.log | grep "ERROR"

In production, this shows only errors as they happen in real time. Since your log file is not actively receiving new entries, you will see existing matches and a waiting cursor. Press Ctrl+C to stop.

Seeing nothing from tail -f | grep?

If you see a blank cursor with no output, that is actually normal for a static log file. tail -f only shows new lines added after you ran the command. Since no application is actively writing to this file, no new errors appear.

Press Ctrl+C and move on. The important takeaway is the pattern itself, which you would use on a live production server.

Why isn't my tail -f | grep showing any output?

Build the full analysis pipeline

You know there are 200 errors. But are they all the same error, or many different problems? You need to rank them by frequency to find the most impactful one.

Before you run the next command, think about this: what do you expect the most common error to be, based on what you saw in the grep | head -3 output earlier?

  • Run the full analysis pipeline to rank errors by frequency:
grep "ERROR" /var/log/myapp/app.log | awk '{$1=$2=$3=""; print $0}' | sort | uniq -c | sort -rn

You should see a ranked count of error types, with "Database connection timeout" as the most frequent at 200 occurrences. That is your root cause.

What does each stage of the pipeline do?

This is the single most important DevOps log analysis pipeline. Each stage does one job:

  • grep "ERROR" filters the 5000+ lines down to only error lines.
  • awk '{$1=$2=$3=""; print $0}' removes the first three columns (date, time, log level) so you are left with just the error message.
  • sort groups identical error messages together (required for uniq to work).
  • uniq -c counts how many times each unique message appears.
  • sort -rn ranks by count in descending order, so the most frequent error is at the top.
  • Get a broader picture by counting all log levels (not just errors) by running:
awk '{print $3}' /var/log/myapp/app.log | sort | uniq -c | sort -rn

You should see the distribution: around 5000 INFO entries, 200 ERROR entries, and 100 WARN entries. This tells you the overall health picture of the application at a glance.

What does awk '{print $3}' do?

The awk command treats each line as columns separated by spaces. $3 means "the third column." In your log format (date, time, level, message), column 3 is the log level: INFO, ERROR, or WARN.

💡 Your interview answer for log analysis

You can now tell an interviewer exactly what was failing: "I ran a grep-awk-sort-uniq pipeline and discovered 200 database connection timeouts to db-primary.internal on port 5432. That was the root cause behind the application errors."

You have diagnosed the root cause using professional log analysis techniques. The server's application was failing due to database connection timeouts. Next up, you will tackle two more stacked problems: the disk is full and file permissions are dangerously misconfigured.

Reclaim Disk Space and Fix Permissions

You killed the rogue process and found the root cause in the logs. But the server still has problems lurking beneath the surface.

In this step, you will discover that the disk is filling up with junk files, and a critical configuration file has dangerous permissions that expose your database password to every user on the system. Two real problems stacking on each other, exactly like production incidents.

In this step, get ready to:
  • Diagnose disk pressure and reclaim ~200MB of wasted storage.
  • Fix insecure file permissions to lock down sensitive credentials.
  • Restart and verify nginx is serving traffic on a healthy server.
Diagnose and reclaim disk space

Production servers crash when they run out of disk space. Log files grow endlessly, core dumps pile up, and critical services stop working because they cannot write to disk.

  • Check the overall disk usage by running:
df -h

Look at the Use% column for the / filesystem. The disk is under pressure from the junk files the simulation created.

  • Drill into the log directory to find the largest files by running:
du -sh /var/log/myapp/* | sort -hr

You should see debug-old.log at the top at 100MB. That is the biggest offender in this directory.

  • Sweep the entire system for any large files hiding anywhere by running:
find / -type f -size +50M 2>/dev/null

What does this command do?

This searches the entire filesystem (/) for regular files (-type f) larger than 50MB (-size +50M). The 2>/dev/null suppresses permission error noise so you only see results.

You will see three files: the 100MB debug log and two 50MB core dumps in /tmp.

  • Remove all three junk files by running:
rm /var/log/myapp/debug-old.log /tmp/core-dump-001 /tmp/core-dump-002
  • Verify the space was reclaimed by running df -h again:
df -h

The Use% column for / has dropped. You just reclaimed ~200MB.

What is the interview answer for disk diagnostics?

When asked how to troubleshoot a full disk: "df -h for the big picture, du -sh to drill into the directory, find -size for sweeping the system." This three-step approach works on any Linux server.

🙋‍♀️ Still seeing high disk usage?

If df -h still shows high usage after removing files, a running process may be holding deleted files open. You can check with lsof +L1 to see deleted-but-open files.

Help me troubleshoot high disk usage after deleting files

Fix file permissions

Disk space is fixed. Now look at the permissions problem. A configuration file with the wrong permissions can expose passwords to any user on the system.

  • Inspect the permissions on the config file by running:
ls -la /opt/myapp/config.env

You should see -rwxrwxrwx (777). This means every user on the system can read, write, and execute this file. That file contains your database password. In production, this is a critical security vulnerability.

  • Lock down the config file so only the owner can read and write it:
chmod 600 /opt/myapp/config.env
  • Verify the change took effect by running:
ls -la /opt/myapp/config.env

You should now see -rw------- (600). Only the owner (root) can read or write this file.

  • Now check the deploy script permissions:
ls -la /opt/myapp/scripts/deploy.sh

You should see -rw-r--r-- (644). This means the file cannot be executed. A deploy script that cannot execute is useless.

  • Make the deploy script executable by running:
chmod 755 /opt/myapp/scripts/deploy.sh

How does the chmod number system work?

Each digit represents a permission group: owner, group, others. The values are: read=4, write=2, execute=1. Add them together for each group.

So 755 means: owner gets 7 (4+2+1 = rwx), group gets 5 (4+0+1 = r-x), others get 5 (4+0+1 = r-x). And 600 means: owner gets 6 (4+2+0 = rw-), group gets 0 (---), others get 0 (---).

Interview answer: "Sensitive configs get 600. Scripts get 755. Never 777 in production."

🙋‍♀️ Getting permission denied when running chmod?

Make sure you are running as root. Check with whoami. If you are not root, these commands will fail because only the file owner or root can change permissions.

Help me fix permission denied errors

Verify the server is fully recovered

You have reclaimed disk space and locked down permissions. Now confirm everything works end-to-end by running the permission audit, testing the deploy script, and restarting nginx.

  • Verify the config file permissions with a numeric check:
stat -c '%a %n' /opt/myapp/config.env

You should see 600 /opt/myapp/config.env.

  • Test that the deploy script actually runs:
/opt/myapp/scripts/deploy.sh

You should see Deploying application... printed to the terminal. The script works now that it has execute permission.

  • Run a permission audit to confirm no world-writable files remain in /opt:
find /opt -type f -perm -o=w 2>/dev/null

This should return no output. An empty result means no file in /opt is writable by random users.

  • Restart nginx to confirm the server can recover fully:
nginx -s stop && nginx

Before you verify, what do you expect to see when you check port 80? Will nginx be listening after the restart?

  • Check that nginx is listening on port 80:
ss -tlnp | grep :80

You should see a line showing nginx listening on port 80. That confirms the process restarted and bound to the port.

  • Confirm nginx actually responds to HTTP requests:
curl -s -o /dev/null -w "%{http_code}" http://localhost:80

You should see 200. All three layers pass: process running, port bound, service responding. The server is fully recovered.

What does this curl pattern do?

The flags break down as: -s silences progress bars, -o /dev/null discards the HTML body, and -w "%{http_code}" prints only the HTTP status code. This is the pattern health-check scripts use to verify a service is responding.

🙋‍♀️ Seeing connection refused or no output?

If ss -tlnp | grep :80 shows nothing, nginx did not restart. Run nginx by itself to see the error message. A common issue is a stale PID file. Remove it with rm -f /run/nginx.pid then run nginx again.

Help me get nginx running again

Your server is back to full health. Disk space reclaimed, permissions locked down, nginx responding. Next up, you will verify network services at all three layers and build the systematic check that DevOps engineers use across fleets of servers.

Verify Network Services

Your server is back on its feet. You killed the rogue process, traced the root cause in logs, reclaimed disk space, and locked down permissions.

But how do you know the service is actually working for users? A DevOps engineer always verifies at three distinct layers: process running, port bound, service responding.

In this step, get ready to:
  • Confirm nginx processes are running and the port is bound.
  • Verify the service responds with a valid HTTP status code.
  • Inspect connection statistics and active connections with the ss networking toolkit.
Confirm the process is running and the port is bound

A running process does not guarantee a working service. The process could be alive but not bound to a port. The port could be bound but returning errors. Checking all three layers catches each of these failure modes.

  • Start with Layer 1. Confirm nginx worker processes exist by running:
ps aux | grep nginx

You should see at least two lines: a master process (started as root) and one or more worker processes. The master manages configuration and the workers handle actual requests.

  • Now check Layer 2. Confirm port 80 is bound by running:
ss -tlnp | grep :80

What do these flags mean?

The ss command shows socket statistics. The flags break down as:

  • -t shows only TCP sockets.
  • -l shows only listening (waiting for connections) sockets.
  • -n shows numeric port numbers instead of service names.
  • -p shows the process name that owns the socket.

You should see nginx listed as the process owning port 80. This confirms the socket is bound.

  • See all listening services on the server (not just port 80) by running:
ss -tlnp

This shows every service waiting for connections on your server. On a production system, you would see many more entries here.

What would you say in an interview?

"I check three layers: process running, port bound, service responding. A process can be alive but not listening. A port can be bound but returning errors. I verify all three before calling a service healthy."

Test the HTTP response layer

Layer 1 and Layer 2 confirm that nginx is running and the port is bound. Layer 3 proves the service actually responds with valid content.

  • Check the actual HTML response from nginx by running:
curl -s http://localhost:80 | head -5

You should see the opening lines of the nginx default HTML page. This proves the server is returning actual content.

  • Get just the HTTP status code (the pattern health-check scripts use) by running:
curl -s -o /dev/null -w "%{http_code}" http://localhost:80

You should see 200 printed on its own. This is the pattern automated health checks use to verify a service is responding.

What does this command do?

  • -s suppresses the progress bar.
  • -o /dev/null discards the response body (you only want the status code).
  • -w "%{http_code}" tells curl to print only the HTTP status code after the request completes.
  • Now test what a failing check looks like. Try a port where nothing is listening by running:
curl -s -o /dev/null -w "%{http_code}" http://localhost:8080

You should see 000 printed. This means connection refused. Nothing is listening on port 8080. In a health-check script, you would flag any response that is not 200 as a problem.

Inspect connection statistics with the network toolkit

Beyond verifying specific services, ss gives you a full picture of network activity on the server. These commands replace the deprecated netstat tool.

  • View a summary of all connection statistics by running:
ss -s

This prints a compact overview showing total TCP connections, their states, and UDP socket counts. Useful for spotting connection leaks or unusual patterns at a glance.

  • View all established TCP connections (not just listening sockets) by running:
ss -tnp

This shows active connections between your server and other hosts. On a busy production server, you would see connections to databases, caches, and upstream services here.

Why ss instead of netstat?

ss is the modern replacement for netstat. It reads directly from the kernel and is faster, especially on servers with thousands of connections.

netstat may not even be installed on newer systems. In interviews, use ss -tlnp for listening ports and ss -tnp for active connections.

Verify your three-layer check end to end

Before you run the final check, predict: will both commands confirm a healthy service?

  • Confirm nginx is bound on port 80 by running:
ss -tlnp | grep :80
  • Confirm the HTTP response is healthy by running:
curl -s -o /dev/null -w "%{http_code}" http://localhost:80

You should see nginx listed on port 80 in the first command, and 200 from the second. Your server is fully recovered and verified at all three layers.

Not seeing nginx on port 80?

  • Nginx may have stopped. Restart it with nginx and re-run the checks.
  • If you see an error about the address already being in use, another process claimed port 80. Run ss -tlnp | grep :80 to identify it.
  • If curl returns 000 for port 80, nginx is not running. Start it with nginx and try again.

Why can't I get a 200 response from nginx?

Linux Interview Questions for Freshers

You just diagnosed and fixed a sick Linux server from top to bottom. That hands-on experience is exactly what interviewers want to hear about.

In this step, you will review 10 common Linux interview questions that freshers face. Each question includes a clear, concise answer you can use to prepare for your next interview.

In this step, get ready to:
  • Review 10 fresher-level Linux interview questions with detailed answers.
  • Connect each answer back to the real commands you used in this project.
Q1: What is the Linux kernel?

Answer

The Linux kernel is the core component of the Linux operating system. It manages hardware resources (CPU, memory, disk, network) and provides an interface between applications and the physical hardware.

Think of it as a translator between your software and your machine. Every command you ran in this project ultimately passed through the kernel to interact with hardware.

Q2: How do you find and kill a process in Linux?

Answer

Use ps aux to list all running processes, or top to see them in real time sorted by resource usage. Once you identify the process ID (PID), use kill <PID> to terminate it gracefully, or kill -9 <PID> to force-kill it.

You did exactly this in Step 2 when you hunted and killed the rogue process consuming CPU.

Q3: What is the difference between su and sudo?

Answer

su (switch user) changes your entire session to another user account. It requires that user's password. sudo (superuser do) runs a single command with elevated privileges using your own password.

sudo is preferred in production because it logs every privileged command and limits the scope of elevated access to one command at a time.

Q4: How do you check disk space usage?

Answer

Use df -h to see disk space usage for all mounted filesystems in human-readable format. Use du -sh <directory> to see how much space a specific directory occupies.

You used both of these commands in Step 4 when you tracked down and reclaimed wasted disk space on the server.

Q5: What are file permissions in Linux?

Answer

Every file and directory in Linux has three permission types: read (r), write (w), and execute (x). These are assigned to three categories: owner, group, and others.

Use chmod to change permissions and chown to change ownership. For example, chmod 755 script.sh gives the owner full access and everyone else read and execute. You fixed broken permissions in Step 4.

Q6: How do you view logs in Linux?

Answer

System logs are stored in /var/log/. Use journalctl to query systemd logs, tail -f <logfile> to follow a log in real time, and grep to filter for specific keywords.

You investigated logs in Step 3 to trace the root cause of the rogue process. This is a daily task for any Linux administrator.

Q7: What is the purpose of the grep command?

Answer

grep searches for patterns in text. It filters the output of other commands or searches inside files. For example, grep "error" /var/log/syslog shows only lines containing the word "error".

You used grep with pipes throughout this project to filter process lists and port information.

Q8: How do you check which ports are open on a Linux server?

Answer

Use ss -tlnp to list all TCP ports currently listening on the server. The flags show TCP sockets, listening state, numeric ports, and the process name.

This is the modern replacement for netstat. You used this exact command in Step 5 to verify nginx was bound to port 80.

Q9: What is a cron job?

Answer

A cron job is a scheduled task that runs automatically at a specified time or interval. You configure them using crontab -e to edit your cron table.

For example, a health-check script that runs every 5 minutes to verify your server is responding (like the curl check you built in Step 5) would be a perfect use case for a cron job.

Q10: What is the Linux file system hierarchy?

Answer

Linux organizes files in a tree structure starting from / (root). Key directories include:

  • /etc stores configuration files.
  • /var/log stores system logs (you investigated these in Step 3).
  • /home stores user home directories.
  • /tmp stores temporary files (you cleaned these up in Step 4).
  • /bin and /usr/bin store executable programs and commands.

Secret mission

Automate It All with a Health-Check Script

You have diagnosed and fixed a broken server manually. A DevOps engineer does this once, then automates it. Build a professional bash health-check script that runs every diagnostic from this project in one command, with color-coded alerts and proper exit codes.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally inside a Docker container, so there are no ongoing costs.

Resources you used:

  • Docker container named devops-lab (free, uses only local disk space).

Keep everything running

No action needed. Choose this if you are still actively building or plan to continue practicing soon.

  • When you type exit inside the container, it stops automatically (bash is the main process).
  • The container stays on disk in a stopped state. It uses no CPU or memory while stopped.
  • Re-enter anytime by running this command in your Mac Terminal:
docker start -ai devops-lab

This reattaches you to the same container with all your scripts and files intact.

Pause - I'll come back to this later

Shut down the container to free up memory. Your files and scripts remain saved inside the container image.

  • Type exit inside the container to return to your Mac Terminal.
  • If the container is still running, stop it explicitly by running:
docker stop devops-lab
  • The container uses zero CPU and memory while stopped.
  • When you are ready to return, re-enter with:
docker start -ai devops-lab

Delete - I don't want to use this again

Remove the container entirely and start fresh. You can always recreate it by running the original docker run command again.

  • Type exit inside the container to return to your Mac Terminal.
  • Remove the container by running:
docker rm devops-lab
  • If you see an error saying the container is still running, stop it first with docker stop devops-lab, then run the docker rm command again.
  • This deletes the container and all files inside it (your scripts, logs, and installed packages).
  • To recreate the lab from scratch, run the original command from Step 1:
docker run -it --hostname prodserver --name devops-lab ubuntu:26.04 bash

Nice Work!

Nice Work!

You just built a complete Linux Server Health Monitor inside a Docker container. You diagnosed real production problems, fixed them using the exact workflow DevOps engineers follow, and automated everything into a professional bash tool.

You've learned how to:

  • Identify a rogue CPU-hogging process and terminate it safely using Linux signals (SIGTERM for graceful shutdown, SIGKILL as last resort).
  • Parse 5000+ log lines using grep, awk, sort, uniq, and pipe chains to extract the root-cause error pattern from noise.
  • Reclaim disk space with df, du, and find, fix dangerous file permissions, and verify network services using the three-layer check (process, port, response).
  • Secret Mission: Built a professional bash health-check script with strict mode, functions, color-coded output, threshold logic, and exit codes that automates all diagnostics into a single command.

Ready to quiz yourself?