Linux Operations Detective

Diagnose logs, fix permissions, and manage processes in a local Linux lab.

Introduction

30 Second Summary

When an app fails, the useful clue is often buried among ordinary messages and files. Finding it quickly depends on knowing where to look.

In this project, you will build a safe local deployment simulation in your Linux terminal. You will follow an operations loop that covers log diagnosis, file permissions, and process control.

What You'll Build

You finish with a deployment simulation that behaves like a small operations incident you can diagnose from one terminal window.

By the end of this project, you'll have:

A repaired deployment workspace whose health check prints service=inventory-api status=healthy.
A log investigation that isolates four error events from a mixture of normal messages, warnings, and failures.
A process-control workflow that starts a background job, captures its process ID, inspects it, and terminates it safely.
Secret Mission: Turn raw error lines into a saved incident summary that groups repeated failures and shows their counts.

Are there any prerequisites?

You need only a Linux computer with a terminal. The walkthrough requires no cloud account, administrator access, or prior command-line experience.

Before We Start

This first checkpoint locks in your goal before any terminal work begins. You are building a broken deployment simulation to practise the inspect, diagnose, change, and verify loop.

Set Up a Safe Operations Workspace

Operations practice is safest when every change stays away from system files. A directory in your home folder gives this deployment simulation that boundary.

This step confirms GNU Bash first. It then verifies every command the lab depends on.

In this step, get ready to:

Confirm GNU Bash is available.
Verify every command used later in the lab.
Create an empty operations workspace in your home directory.

Confirm and start GNU Bash

Your existing Linux terminal can start with a different shell. GNU Bash gives the later job-control commands a predictable environment.

Check the installed GNU Bash version by running this command:

bash --version

What does this command do?

The bash --version command prints Bash version information. It exits after displaying the result.

You should see version information in the terminal. This confirms GNU Bash is available.

Do You See No Version Information?

Confirm that you typed bash --version with two hyphens before version.

If the check still fails, help me diagnose why GNU Bash is unavailable.

Good start. GNU Bash is available for the rest of the lab.

Start a GNU Bash session by running this command:

bash

What does this command do?

The bash command starts a GNU Bash session inside your existing terminal. Your terminal returns to a prompt in the new shell.

You should return to a prompt without an error. The remaining commands now run in GNU Bash.

Verify the required commands

The deployment lab relies on a small command-line toolchain. Checking each command now prevents a missing utility from interrupting a later investigation.

Verify every required command by running these checks:

command -v bash
command -v cp
command -v chmod
command -v grep
command -v sort
command -v uniq
command -v wc
command -v sleep
command -v jobs
command -v kill
command -v man

What Do These Checks Prove?

The command -v command asks the current shell how each command name resolves.
A printed path confirms that an external utility is available.
A printed Bash builtin name confirms that the shell provides the command.

Each check should print a path or a Bash builtin name. Eleven printed results confirm the full toolchain is available.

Does a Check Produce No Result?

Stop before continuing if one check produces no result. The later task that uses that command depends on it being available.

Use this link to help me troubleshoot the missing Linux command.

Create the isolated workspace

Your home directory is a safe location for this local simulation. The lab directory keeps every configuration file and log together.

Before you run these commands, what ending do you expect pwd to print once the workspace is active?

Create the empty workspace inside your home directory by running these commands:

cd ~
mkdir linux-ops-lab
cd linux-ops-lab
pwd

What Do These Commands Do?

The cd ~ command returns you to your home directory.
The mkdir linux-ops-lab command creates the empty lab directory.
The cd linux-ops-lab command enters the new directory.
The pwd command prints your current location.

The final line should show your home path ending in /linux-ops-lab. That ending proves the empty workspace is your current directory.

You have the safety boundary in place. Every later project file stays inside this lab directory.

Does the Path End Differently?

Check that you typed linux-ops-lab the same way in both directory commands. Linux directory names are case-sensitive.

If the path still looks wrong, help me reach the correct lab directory.

Your empty operations workspace is ready. Next, you will stage the deployment files that turn this directory into a realistic incident scene.

Stage the Broken Deployment

Your Linux operations lab now has a safe boundary for every change you make. Your existing GNU Bash session is still inside ~/linux-ops-lab.

A visible deployment layout gives each command an operational purpose. This step also stages a file permission defect that you can investigate in a later step.

In this step, get ready to:

Create four directories for configuration, logs, scripts, and reports.
Populate the deployment configuration and application log.
Stage a health check with owner execute permission removed.

Build the deployment skeleton

A deployment workspace separates files by purpose. This layout keeps configuration, logs, scripts, and generated reports in predictable places.

Create the four deployment directories by running this command:

mkdir config logs scripts reports

What does this command do?

The mkdir command creates each named directory inside ~/linux-ops-lab. These directories form the structure for the staged deployment.

List the new directories by running this command:

ls

What does this listing show?

The ls command displays the contents of your current directory. It gives you an immediate check that the deployment skeleton exists.

You should see config, logs, scripts, and reports in the output.

Missing one of the directories?

Check that your terminal prompt still points to ~/linux-ops-lab.
Repeat the directory creation command above if a directory name is missing.
Compare each directory name with the listing above.

help me find why one of my Linux lab directories was not created.

Add the deployment data

The configuration identifies the service that this deployment represents. It also records the environment, port, and logging level used by the simulation.

Write the service settings to config/app.conf by running this command:

cat > config/app.conf <<'EOF'
SERVICE=inventory-api
ENVIRONMENT=staging
PORT=8080
LOG_LEVEL=info
EOF

How does this create the configuration?

The <<'EOF' syntax supplies literal lines to cat.
The > symbol redirects standard output into config/app.conf.
The SERVICE setting names the simulated application.
The remaining settings describe its staging environment.

Confirm that the configuration file exists by running this command:

ls -l config logs scripts reports

What does this listing prove?

The long listing displays the contents of all four deployment directories. At this point, config should contain app.conf.

You should see app.conf listed under the config heading.

Configuration file missing?

Check that the closing EOF starts at the beginning of its line.
Repeat the configuration command if the terminal was waiting for more input.
Confirm that app.conf appears inside config.

help me fix my Bash here-document for app.conf.

✔️ Awesome, I've got everything!

Your config/app.conf file is complete.

ⓧ I'd like to double check the full code

SERVICE=inventory-api
ENVIRONMENT=staging
PORT=8080
LOG_LEVEL=info

What should this file contain?

The file contains exactly four configuration entries. Compare each line with your saved config/app.conf file.

Application logs record events in the order they occur. This staged log mixes normal activity with warnings and failures so later commands have realistic data to inspect.

Write the eight application events to logs/app.log by running this command:

cat > logs/app.log <<'EOF'
INFO inventory-api started
INFO request accepted
ERROR database_connection_refused
WARN retry scheduled
ERROR database_connection_refused
ERROR upstream_timeout
INFO request accepted
ERROR database_connection_refused
EOF

What is inside this log?

The here-document writes eight events into logs/app.log. Each event occupies one line.

The log includes normal requests, one warning, and four ERROR events. That mixture creates the evidence for your later investigation.

Confirm that the application log exists by running this command:

ls -l config logs scripts reports

What changed in the listing?

The logs section now includes app.log. The configuration file remains under config.

You should now see app.conf under config. You should also see app.log under logs.

Application log missing?

Check that the first line names logs/app.log.
Check that the closing EOF has no spaces before it.
Repeat the log command if the shell did not return to its prompt.

help me fix my Bash here-document for app.log.

✔️ Awesome, I've got everything!

Your logs/app.log file contains all eight events.

ⓧ I'd like to double check the full code

INFO inventory-api started
INFO request accepted
ERROR database_connection_refused
WARN retry scheduled
ERROR database_connection_refused
ERROR upstream_timeout
INFO request accepted
ERROR database_connection_refused

What should this file contain?

The file contains exactly eight application events. Compare their order with your saved logs/app.log file.

Stage the broken health check

An executable file needs an x in its permission string before Linux allows it to run directly. The copied Bash program provides a harmless health check whose owner execute permission can be removed safely.

Copy the Bash executable into scripts with owner execute permission removed by running these commands:

cp "$(command -v bash)" scripts/health-check
chmod u-x scripts/health-check
ls -l scripts/health-check

How is the defect staged?

Command substitution discovers the Bash executable on your Linux system.
The cp command copies that executable to scripts/health-check.
The chmod command removes only the owner's execute permission.
The ls command displays the resulting permission string.

You should see rw- in the owner's permission triplet. The missing x confirms that the health check is staged with the intended defect.

Health check copy missing?

Confirm that the earlier Bash check printed a valid executable path.
Confirm that the scripts directory still exists.
Repeat the staging commands after correcting the missing path or directory.

help me diagnose why the Bash executable was not copied into scripts.

Before you run the final check, which files do you expect to see under each deployment directory?

Inspect the complete staged deployment by running this command:

ls -l config logs scripts reports

What does the final check cover?

This command lists the contents and permission metadata for every deployment directory. It confirms the complete workspace without leaving ~/linux-ops-lab.

Under config, you should see app.conf. Under logs, you should see app.log.

Under scripts, you should see health-check with the owner triplet rw-. The reports directory should contain no files.

Strong start. Your local deployment now contains realistic configuration, application events, and a staged permission defect.

Final listing does not match?

Compare each directory heading with the expected contents above.
Repeat the command that creates any missing artifact.
Check the owner permission triplet on scripts/health-check for rw-.

help me compare my staged deployment workspace with the expected layout.

Your broken deployment is ready for investigation. Next, you will filter its application log to isolate the failures.

Investigate the Application Logs

Your deployment workspace is staged. Its application log now contains the incident trail you will investigate.

Reading an entire log works while the file is small. As logs grow, routine events can hide failures.

This step uses grep to isolate failures. A pipeline sends those matching lines to wc for a reliable count.

In this step, get ready to:

Display the full application log to see its mixed events.
Filter the log to isolate its failure lines.
Count the filtered failures with a pipeline.

Read the full application log

The full log gives you a baseline before you filter anything. This view shows how routine activity can surround the events you need.

Display every event in logs/app.log by running this command:

cat logs/app.log

What does this command show?

cat writes every line from logs/app.log to standard output.
The complete view reveals the different event types stored in the log.

You should see eight events in the terminal. The output contains INFO, WARN, and ERROR lines.

Can't read the log?

Confirm your current terminal is still inside ~/linux-ops-lab.
Check that the path is spelled exactly as logs/app.log.

Help me find out why the application log is not displayed.

Filter the failure lines

The grep command prints lines that match a pattern. Filtering for ERROR removes the routine events from this investigation.

Isolate the failure events by running this command:

grep ERROR logs/app.log

How does the filter work?

grep searches logs/app.log for the matching pattern.
ERROR limits the output to failure events.

ERROR database_connection_refused appears three times. ERROR upstream_timeout appears once.

Don't see the four failures?

Check that ERROR uses uppercase letters.
Confirm the file path remains logs/app.log.

Help me troubleshoot why my log filter does not show the expected failures.

Count the filtered errors

A pipe connects the output from grep to the input of wc. The -l option selects newline counts.

Before you run the count, how many error events do you expect the filtered stream to contain?

Count the filtered error events by running this pipeline:

grep ERROR logs/app.log | wc -l

How does the pipeline work?

grep sends only matching lines through the pipe.
wc -l counts the newlines it receives.
This pattern scales because the counter processes filtered output instead of the entire log.

You should see 4 in the terminal. That result confirms the log contains four error events.

Seeing a different count?

Compare the filtered output with the four failure lines you identified above.
Check that the pipeline contains wc -l.

Help me understand why my error pipeline returns a different count.

Your first log investigation is complete. The pipeline reduced eight mixed events to four confirmed failures.

You have isolated the deployment's failures without changing its log. Next, you will investigate the staged health check.

Reproduce the Permission Failure

Your existing GNU Bash session has already exposed four error events in the application log. The next operational question is whether the deployment's health check can run.

A permission diagnosis becomes meaningful after you reproduce the symptom. In this step, you will test direct execution before using the file listing to explain the result.

In this step, get ready to:

Attempt the staged health check through direct execution.
Use the long listing to identify the cause.

Reproduce the direct-execution result

The copied health check accepts a Bash command string through -c. Direct execution asks the operating system to start the file using its permission metadata.

Before you run it, do you expect the health check itself to start?

Test the staged health check by running this command:

./scripts/health-check -c 'echo "service=inventory-api status=healthy"'

This failure is intentional

The shell should report Permission denied. The healthy status stays absent.

Linux file permissions control whether a file can run as a program. Direct execution stops before Bash can process -c.

🙋‍♀️ Did you get a different result?

If the healthy status prints, the copied file still has execute permission. Your workspace differs from the staged state.

If the file cannot be found, use pwd to confirm your current directory ends with /linux-ops-lab.

Ask for help to restore the intended broken state.

Trace the failure to the owner bits

The shell response gives you the symptom. A long listing exposes the permission string that explains it.

Before you inspect the file, which permission position do you expect to explain the rejection?

Display the health check's long listing by running this command:

ls -l scripts/health-check

How to read the permission evidence

The ls -l output places the owner's permission triplet immediately after the file-type character.
The r grants read access. The w grants write access.
The final - shows that owner execute permission is absent. Direct execution requires x in that position.

You've found the exact gap. The owner's permission triplet is rw-.

Before the final check, do you expect the direct-execution result to change while the permission string remains rw-?

Verify the rejection is reproducible by running this command:

./scripts/health-check -c 'echo "service=inventory-api status=healthy"'

What should you see?

The shell should again report Permission denied. This repeated rejection confirms the defect remains unchanged.

That is the diagnosis complete. You now have the evidence needed for a targeted repair.

Your evidence is clear. Next, you'll restore the owner's execute permission.

Repair the Check and Control a Process

The operating system rejected your health check in the last step. You traced the failure to its missing owner execute permission.

Now you will restore that permission. The inspect, change, and verify loop turns the failed check into a healthy result.

You will then use GNU Bash to track a harmless background process. Its process ID gives you precise control over the running job.

In this step, get ready to:

Restore owner execute permission on the health check.
Track a harmless background process through its saved process ID.
Terminate the background job through its saved process ID.

Turn the rejected check into a healthy result

The existing health check is a copied Bash executable. Its owner's permission triplet currently contains rw-.

The missing x blocks direct execution. Adding u+x restores that capability without changing the file's contents.

Before you run the repair, consider whether the same health-check command will be rejected or succeed.

Apply the repair, inspect the permission string, and test the health check by running this sequence:

chmod u+x scripts/health-check
ls -l scripts/health-check
./scripts/health-check -c 'echo "service=inventory-api status=healthy"'

What does this sequence do?

The chmod command adds owner execute permission through u+x.
The ls command displays the updated permission string for scripts/health-check.
The copied Bash executable reads the health command after -c.
The echo command prints the simulated service status.

You should see the owner's permission triplet include rwx. The final line prints service=inventory-api status=healthy.

You closed the failure loop. The same health-check command that failed now confirms the service is healthy.

Still seeing Permission denied?

Check that your terminal is still inside ~/linux-ops-lab.
Compare the file path with scripts/health-check.
Confirm that the owner's permission triplet includes rwx.
Repeat the command sequence after correcting the location or file path.

Help me diagnose why the health check still cannot run.

Track a background process

A background process keeps running while your terminal accepts more commands. Bash exposes the most recent background process ID through $!.

Saving that number in pid gives later commands a precise target. The job remains harmless because it only waits.

Start the waiting process, save its process ID, and inspect the job by running this sequence:

sleep 1000 &
pid=$!
echo "$pid"
jobs -l

How does Bash track the job?

The sleep command waits for 1000 seconds unless it is interrupted.
The & character starts the command in the background.
The $! value supplies the process ID of the newest background command.
The echo command displays the saved numeric process ID.
The jobs command reports jobs started by the current shell.
The -l option includes process information in the job listing.

You should see a numeric process ID. The detailed job listing shows the sleep job as running.

Your shell is now tracking a live background job through pid. That saved value is the handle you use to stop it.

Missing the process ID or job listing?

Confirm that you ran all four lines in the same GNU Bash session.
Check that the numeric process ID appears before the job listing.
Repeat the sequence if your shell session changed.

Help me troubleshoot why Bash did not save or list my background process.

Stop the background job

A signal is a message sent to a running process. The kill command uses the default TERM signal when you do not specify another one.

Send the default termination signal to the saved process ID by running this command:

kill "$pid"

What does this command do?

The kill command reads the process ID stored in pid. It sends the default TERM signal to that process.

Before you check the job table, predict whether the sleep job still appears as running.

Verify the background job has stopped by running this command:

jobs

What should you see?

GNU Bash may first report that the job terminated. After that status is reported, no running sleep job should remain.

That completes the runtime cleanup. The health check works correctly. The background job is gone.

Still see a running sleep job?

Check that you stayed in the same GNU Bash session because pid exists only in that shell.
Compare the numeric process ID printed earlier with the process information in the earlier job listing.
Repeat the background-process sequence if your shell session changed.

Help me stop the background process that still appears in my Bash job list.

Secret mission

Generate an Error Summary Report

Turn the raw error lines in your application log into a saved incident summary. Your report will group identical failures and show how often each one occurred.

Clean Up Your Resources

Your operations lab lives entirely on your local machine. It has no ongoing costs.

Decide whether to keep the workspace for more practice, pause by closing the terminal, or delete the workspace entirely.

Resources you used:

The local ~/linux-ops-lab workspace containing all practice files.

Keep everything running

No action is needed. Choose this option if you want to keep using the workspace for terminal practice.

Keep the ~/linux-ops-lab directory as a reusable operations practice lab.
Review reports/error-summary.txt when you want to revisit the grouped incident data.

Pause - I'll come back to this later

Shut down the terminal session when you want to pause. Your workspace stays on disk.

The background lab process is already terminated. Closing the terminal leaves no lab job running.

Close the existing terminal window.

Delete - I don't want to use this again

Remove all project resources when you are finished with the lab. This permanently removes only the local lab folder.

Return to your home directory and print its path by running these commands:

cd ~
pwd

Confirm Your Location

The pwd output should show your home directory. The printed path must not end with /linux-ops-lab.

Delete the linux-ops-lab directory by running this command:

rm -r linux-ops-lab

What Does This Command Do?

The rm -r command removes the directory plus everything inside it. Terminal deletion does not place the files in a recycle bin.

Confirm the workspace is gone by running this command:

ls

What Should You See?

The command lists the remaining items in your home directory. You should no longer see linux-ops-lab in the output.

Nice Work!

Case closed! You repaired a local Linux deployment workspace through the operations troubleshooting loop.

You've learned how to:

Build a deployment-style filesystem layout inside a safe local workspace. Navigate its configuration, logs, scripts, and reports from the terminal.
Complete a log investigation with cat. Filter failures with grep. Count four error events through a pipe into wc -l.
Reproduce a real Permission denied failure. Restore owner execute permission with chmod. Manage a background process through its PID.
Secret Mission: Generate a persistent error summary report from the application log. Group repeated failures with sort and uniq -c before saving the result in reports/error-summary.txt.

Ready to quiz yourself?