Document Your RHEL Chaos Lab

Turn a RHEL chaos lab into an evidence-backed observability portfolio.

Introduction

30 Second Summary

A monitoring dashboard can look convincing while leaving another engineer unable to reproduce the incident behind it. Scattered evidence makes strong technical work hard to trust.

In this project, you will turn your working Red Hat Enterprise Linux 10.2 observability lab into a push-ready portfolio repository that another engineer can understand and reproduce. You will connect seven controlled failure experiments to evidence from Spring Boot, systemd, Prometheus, Grafana, and cgroups v2.

What You'll Build

A visitor opens your repository, chooses an experiment, and follows the evidence from user symptom to verified recovery.

By the end of this project, you'll have:

A project story that lets a reader trace traffic from curl through the service to metrics and dashboards. Its safety boundaries make clear where the experiments belong.
A reproducible experiment index covering all seven scenarios. Each experiment gives the reader a hypothesis, trigger, symptom, evidence source, and recovery action.
An incident field guide that separates application symptoms from host, service, and kernel evidence. Descriptive screenshots support each conclusion without replacing the written analysis.
Secret Mission: Build an observability coverage matrix that shows which layer detects each experiment and exposes the remaining blind spots.

Are there any prerequisites?

You need the completed RHEL observability project with observer.service and the monitoring stack already working. A hosted Git remote is optional because this project finishes with a push-ready local repository.

Before We Start

Before any hands-on documentation, this is your moment to define who your portfolio story serves. A clear audience keeps every experiment focused on what the evidence proves.

Set Up the Evidence Workspace

Your experiments already run across your RHEL lab. However, their proof is scattered across several tools.

This step creates one evidence workspace around a verified healthy baseline. You will also prove that screenshots can move safely from Windows into the repository.

In this step, get ready to:

Verify the observer service and its application health response.
Create dedicated folders for experiment guides and evidence images.
Capture a test image and review the resulting Git status.

Confirm the healthy baseline

A trustworthy incident story starts with a known-good baseline. The existing GNU Make target checks the systemd-managed observer.service before you collect evidence.

Spring Boot Actuator provides a separate application-level health response. Together, these checks confirm that the lab begins in its normal state.

Run the repository's existing health target from the RHEL shell:

make chaos-check

What does this check prove?

The chaos-check target is a local interface supplied by your repository. A successful exit confirms that its health check completed without reporting a failure.

Wait for the target to return control to the shell.
Use the documented Actuator request from the earlier project to retrieve /actuator/health.

The response should include an overall status that indicates the application is healthy. This gives you a stable reference point for every later experiment.

Baseline check failing?

Return observer.service to its healthy state using the recovery workflow from the earlier project.
Reuse the documented Actuator request without changing its host or port.

Still stuck? Help me diagnose why my observer health baseline is failing.

Create the documentation workspace

Experiment guides and screenshots serve different purposes. Separate directories keep written analysis away from visual evidence while README.md remains at the repository root.

Create both documentation directories and display their contents by running these commands:

mkdir -p docs/experiments docs/images
ls -R docs

What does this create?

The -p option creates the parent docs directory when it is missing.
The docs/experiments directory will hold your Markdown experiment guides.
The docs/images directory will hold repository-hosted PNG evidence.

The recursive listing should include both experiments and images under docs. Your evidence workspace now has a clear structure.

Directories missing?

Confirm that the shell prompt still points to the root of the observer repository.
Check the command for the exact directory names docs/experiments and docs/images.

Need another pair of eyes? Help me find why my documentation directories are missing.

Capture and verify a test image

A small test capture checks the complete path from Windows Snipping Tool into the repository. Using harmless content now keeps credentials and private information out of your evidence workflow.

Display a small Windows area that contains no credentials or private information.
Press Windows logo key + Shift + S.

You will see the Snipping Tool overlay. The screen is ready for you to select a small region.

Drag across the harmless area to capture it.

The selected region is now ready to save as a test snapshot.

Use the Snipping Tool save control to save the capture as evidence-workspace-test.png.
Use your existing editor or file-transfer workflow to copy the image to docs/images/evidence-workspace-test.png.

Return to the RHEL shell and confirm the copied file appears by running:

ls -R docs

What should you see?

The listing should show docs/experiments and docs/images. Under the image directory, you should see evidence-workspace-test.png.

Your screenshot pipeline now reaches the repository with the intended filename.

Test image missing?

Check that the transfer destination was docs/images inside the observer repository.
Check that the filename ends with the .png extension.
Repeat the existing file-transfer workflow if the image remained on Windows.

Need help tracing the file? Help me locate my transferred evidence image.

Before you run the final check, do you expect the health target to succeed and Git to report only your intentional workspace additions?

Run the combined health and repository status check from the repository root:

make chaos-check && git status --short

What does the final check prove?

The && operator runs the Git status check only after the health target succeeds. Git's short format keeps the working-tree review compact.

This final check connects service health with repository hygiene. You can now distinguish intentional evidence additions from unrelated changes.

The health check should complete successfully. The Git output should contain only the documentation or image additions you intended to make.

Well done. Your healthy observer lab now has a tested home for written guides and visual evidence.

Unexpected Git changes?

Review every unexpected path before continuing.
Return unrelated files to their intended state using your existing Git workflow.
Preserve docs/images/evidence-workspace-test.png as the intentional test image.

Unsure what belongs? Help me interpret my short Git status safely.

✔️ Awesome, I've got everything!

Your service is healthy. Your documentation directories and test image are ready for the experiment evidence.

ⓧ I'd like to double check the workspace

Use this checklist to compare your workspace with the expected end state.

Confirm that observer.service passes the existing health target.
Confirm that the documented request to /actuator/health returns a healthy response.
Confirm that docs/experiments exists for experiment guides.
Confirm that docs/images contains evidence-workspace-test.png.
Confirm that Git reports only intentional documentation or image additions.

Your evidence workspace is ready. Next, you will turn the seven experiments into an accurate map of failure domains and learning objectives.

Build the Chaos Experiment Map

Your evidence workspace now gives each screenshot a stable home. It also gives each experiment guide a stable home.

Calling every scenario chaos hides the failure domain each one targets. This step turns the lab into a map that another engineer can understand before running anything.

In this step, get ready to:

Write a portfolio pitch plus an architecture summary.
Define the lab's safe operating workflow.
Classify all seven experiments in one evidence map.

Write the project story

The opening of README.md tells a reviewer what the lab proves before they read any incident details. Its architecture follows curl into Spring Boot.

Prometheus collects the metrics for Grafana. systemd controls the service lifecycle.

Return to the editor workflow from Step 1.
Use that workflow to open or create README.md at the repository root.
Replace the file's current contents by copying the Markdown below:

# RHEL Observability Chaos Lab

This repository turns a working RHEL observability stack into a reproducible incident lab.

## Project Pitch

This lab compares application, host, service, and cgroup failure domains. Each experiment connects user symptoms to metrics, service state, kernel evidence, and recovery.

## Architecture

Requests begin with `curl` and reach the Spring Boot observer application.
Spring Boot exports Actuator metrics for Prometheus to scrape.
Grafana turns those Prometheus time series into application and host views.
`observer.service` keeps the application under systemd management.

## Safety Boundaries

- Run these experiments only in the isolated lab.
- Keep the failure endpoints and busy loops away from public traffic.
- Use bounded durations and follow each documented recovery action.

## Quick Start

1. Start from the repository root.
2. Run `make chaos-check`.
3. Send the documented `curl` request to `/actuator/health`.
4. Confirm Prometheus remains available.
5. Open the existing Grafana application and host dashboards.

Why These Sections Come First

The pitch tells a reviewer what your working lab proves.
The architecture traces a request from the client to the application.
The metrics path connects Prometheus collection to Grafana evidence.
The safety boundaries keep the failure controls inside the isolated lab.
The quick start gives a reader a short path to a healthy starting state.

Save README.md.
Use your editor's rendered preview to confirm that the project pitch appears above the architecture summary.
Confirm that the preview also shows the safety boundaries plus the quick-start workflow.

Good progress. Your README now explains the lab's purpose before asking a reader to run an experiment.

README Structure Looks Wrong?

Check that README.md sits at the same repository level as docs.
Check that every heading marker has one space before its heading text.
Confirm that you copied the full quick-start list through its final Grafana item.

Still stuck? Help me fix the structure of my README without changing its technical meaning.

Map the seven experiments

A failure domain names the part of the system affected by an experiment. Clear domains stop application-local work from being mistaken for host-wide failure.

Place your cursor after the final quick-start item in README.md.
Append the seven-row experiment map by copying the table below:

## Experiment Map

| Experiment | Hypothesis | Trigger | Expected user symptom | Primary evidence | Recovery action |
| --- | --- | --- | --- | --- | --- |
| Control baseline | A healthy system provides the comparison point for every failure run. | `make chaos-baseline DURATION=30` | Requests complete normally. | Grafana golden-signal panels plus healthy `/actuator/health`. | Let the bounded run finish. Confirm health remains normal. |
| Application-induced latency | `/slow` increases request duration while `/actuator/health` remains `UP`. | `make chaos-latency MS=300 DURATION=30` | Requests respond slowly without a process outage. | Grafana latency panels plus `/actuator/health`. | Let the bounded run finish. Confirm latency returns to baseline. |
| Deterministic HTTP 500 responses | `/error` deliberately returns HTTP 500 while process health can remain `UP`. | `make chaos-errors DURATION=30` | Error requests fail while the normal request path still responds. | Grafana server-error panels plus `/actuator/health`. | Let the bounded run finish. Confirm the error rate returns to baseline. |
| Application-local CPU work | One `/simulate-cpu` request keeps one application request thread busy for about ten seconds. | Send one request to `/simulate-cpu`. | The request remains busy while the application performs local CPU work. | Grafana application CPU panels. | Wait for the bounded request to complete. Confirm application CPU returns to baseline. |
| Single-worker host CPU pressure | One Bash busy loop creates single-worker pressure. It represents full host saturation only when the host has one logical CPU. | `make host-cpu-stress CPU_DURATION=30` | The host has less CPU headroom. User impact can remain limited on a multi-core host. | Grafana host CPU panels. | Let the bounded target finish. Confirm host CPU returns to baseline. |
| Service outage | Stopping `observer.service` removes application availability. | Use the existing stopped-service experiment. | Requests cannot connect to the application. | systemd state, Prometheus target health, and Grafana availability evidence. | Follow the existing service recovery procedure. Run `make chaos-check`. |
| Cgroup memory OOM | Reaching `memory.max` can invoke an OOM kill inside the cgroup. | Use the existing cgroup OOM experiment. | The application terminates or becomes unavailable. | journald, systemd, and cgroup memory evidence. | Follow the existing OOM rollback procedure. Run `make chaos-check`. |

How the Experiment Map Separates Scope

The control baseline gives every later signal a healthy comparison point.
The /slow, /error, and /simulate-cpu endpoints create three different application behaviors.
The host-cpu-stress target is labeled as single-worker pressure because one busy loop does not prove full multi-core saturation.
The service outage row connects loss of availability to service state.
The cgroup OOM row connects a hard memory limit to kernel-level evidence.

Save README.md.
Use your editor's rendered preview to inspect the experiment table.
Confirm that the preview shows one header row plus seven experiment rows.

That is the core classification done. A reader can now see each experiment's scope before running it.

Experiment Table Does Not Render?

Check that the table has one header row plus seven data rows.
Check that every row begins with a vertical bar.
Check that every row ends with a vertical bar.
Confirm that each row contains six cells.

Need another pair of eyes? Help me find the Markdown table formatting problem in my experiment map.

✔️ Awesome, I've got everything!

Your README.md now contains the project story plus the complete seven-row experiment map.

ⓧ I'd like to double check the full code

Compare your complete README.md with this reference file.

# RHEL Observability Chaos Lab

This repository turns a working RHEL observability stack into a reproducible incident lab.

## Project Pitch

This lab compares application, host, service, and cgroup failure domains. Each experiment connects user symptoms to metrics, service state, kernel evidence, and recovery.

## Architecture

Requests begin with `curl` and reach the Spring Boot observer application.
Spring Boot exports Actuator metrics for Prometheus to scrape.
Grafana turns those Prometheus time series into application and host views.
`observer.service` keeps the application under systemd management.

## Safety Boundaries

- Run these experiments only in the isolated lab.
- Keep the failure endpoints and busy loops away from public traffic.
- Use bounded durations and follow each documented recovery action.

## Quick Start

1. Start from the repository root.
2. Run `make chaos-check`.
3. Send the documented `curl` request to `/actuator/health`.
4. Confirm Prometheus remains available.
5. Open the existing Grafana application and host dashboards.

## Experiment Map

| Experiment | Hypothesis | Trigger | Expected user symptom | Primary evidence | Recovery action |
| --- | --- | --- | --- | --- | --- |
| Control baseline | A healthy system provides the comparison point for every failure run. | `make chaos-baseline DURATION=30` | Requests complete normally. | Grafana golden-signal panels plus healthy `/actuator/health`. | Let the bounded run finish. Confirm health remains normal. |
| Application-induced latency | `/slow` increases request duration while `/actuator/health` remains `UP`. | `make chaos-latency MS=300 DURATION=30` | Requests respond slowly without a process outage. | Grafana latency panels plus `/actuator/health`. | Let the bounded run finish. Confirm latency returns to baseline. |
| Deterministic HTTP 500 responses | `/error` deliberately returns HTTP 500 while process health can remain `UP`. | `make chaos-errors DURATION=30` | Error requests fail while the normal request path still responds. | Grafana server-error panels plus `/actuator/health`. | Let the bounded run finish. Confirm the error rate returns to baseline. |
| Application-local CPU work | One `/simulate-cpu` request keeps one application request thread busy for about ten seconds. | Send one request to `/simulate-cpu`. | The request remains busy while the application performs local CPU work. | Grafana application CPU panels. | Wait for the bounded request to complete. Confirm application CPU returns to baseline. |
| Single-worker host CPU pressure | One Bash busy loop creates single-worker pressure. It represents full host saturation only when the host has one logical CPU. | `make host-cpu-stress CPU_DURATION=30` | The host has less CPU headroom. User impact can remain limited on a multi-core host. | Grafana host CPU panels. | Let the bounded target finish. Confirm host CPU returns to baseline. |
| Service outage | Stopping `observer.service` removes application availability. | Use the existing stopped-service experiment. | Requests cannot connect to the application. | systemd state, Prometheus target health, and Grafana availability evidence. | Follow the existing service recovery procedure. Run `make chaos-check`. |
| Cgroup memory OOM | Reaching `memory.max` can invoke an OOM kill inside the cgroup. | Use the existing cgroup OOM experiment. | The application terminates or becomes unavailable. | journald, systemd, and cgroup memory evidence. | Follow the existing OOM rollback procedure. Run `make chaos-check`. |

What Should the Full File Prove?

The opening establishes the lab's purpose plus its safety boundaries. The table then connects every trigger to a symptom, evidence source, and recovery action.

Review the finished diff

A final git diff review catches gaps before the story becomes evidence. Git limits this check to README.md so the documentation change stays easy to inspect.

Before you run the review, do you expect every experiment row to connect one trigger with one recovery action?

Review only README.md by running this command:

git diff -- README.md

What Does This Command Show?

The command compares your working copy of README.md with the committed version. It leaves every other file out of this review.

The diff should show the project pitch plus the architecture summary. It should also show the safety boundaries plus the quick start.

The experiment table should contain all seven scenarios. Every row should contain the hypothesis, trigger, user symptom, primary evidence, and recovery action.

README Changes Missing from the Diff?

Confirm that README.md is saved in the repository root.
Confirm that the experiment table appears in the saved file.
Check that your shell still points at the observer repository root.

Still missing changes? Help me diagnose why Git does not show my README edits.

Your README now gives every chaos experiment an accurate scope plus a recovery path. Next, you will capture the evidence that exposes the green health check's blind spot.

Expose the Health-Check Blind Spot

Your Red Hat Enterprise Linux lab now maps all seven experiments to their real failure domains. That map gives each controlled run a clear purpose.

A process-level health check can stay green while users experience slow or failed requests. In this step, you will compare Spring Boot Actuator status with evidence from Grafana.

In this step, get ready to:

Capture normal application behavior as a control baseline.
Test the health response during application-induced latency.
Test the health response during deterministic server-error traffic.

Capture the control baseline

A control baseline records normal golden signals before an experiment changes them. Prometheus continues scraping the application while Grafana turns those measurements into comparison evidence.

This run lasts 30 seconds. The short wait gives the dashboard a clean comparison window.

Before you run this, consider whether a control run should produce a steady baseline or a visible spike.

Generate the control traffic by running this command in the repository shell:

make chaos-baseline DURATION=30

How the Baseline Target Works

The repository uses GNU Make to run the existing chaos-baseline target.
The DURATION=30 argument keeps the control traffic running for 30 seconds.
This window establishes the normal application signal you will compare with the next two runs.

Switch back to the Grafana application dashboard from earlier.
Record the current latency shown in the application panels.
Record the current server-error level shown in the application panels.
Press Windows logo key + Shift + S to capture the relevant Grafana panels.
Save the capture as baseline-grafana.png in the location used by your existing file-transfer workflow.
Copy baseline-grafana.png into docs/images using that workflow.
Confirm your repository editor lists docs/images/baseline-grafana.png.

You now have a clean control image. It anchors every comparison that follows.

Baseline Missing From Grafana?

Confirm the repository shell still points to the observer repository root.
Wait for the next Grafana refresh before checking the experiment window again.
Confirm Prometheus still shows the observer application as a scraped target.

Still seeing an empty dashboard? Help me trace missing baseline metrics through my observer application, Prometheus, and Grafana.

Test latency against process health

The Actuator health endpoint reports the application's overall status. The Grafana latency panel measures how long requests take.

The active experiment window lasts 30 seconds. A prepared second shell keeps the timing manageable.

Open a second RHEL shell using the terminal workflow from the earlier project.
Prepare the documented Actuator request for /actuator/health in the second shell without submitting it.

Before you start, predict whether the health response changes when requests slow down.

Start the latency experiment in the original repository shell by running:

make chaos-latency MS=300 DURATION=30

How the Latency Target Works

The chaos-latency target generates application-induced latency.
The MS=300 argument configures a 300 millisecond delay for this run.
The DURATION=30 argument keeps the experiment active for 30 seconds.

Switch to the prepared second shell.
Submit the documented Actuator request while the latency experiment is active.

The response still reports the overall status as UP.

Return to the Grafana application dashboard from earlier.
Align the dashboard time window with the latency run.
Inspect the application latency panel.

You will see latency rise during the same window. The process-level health response stays green.

Press Windows logo key + Shift + S to capture the elevated-latency panel.
Save the capture as latency-grafana.png in the location used by your file-transfer workflow.
Copy latency-grafana.png into docs/images using that workflow.
Confirm your repository editor lists docs/images/latency-grafana.png.

That is the first blind-spot proof captured. The application stayed live while its user-facing latency degraded.

Latency Spike Missing?

Repeat the latency command if the 30-second window ended before you submitted the health request.
Check that the Grafana time window includes the minute when the experiment ran.
Wait for the next dashboard refresh before inspecting the latency panel again.

Still seeing a flat latency panel? Help me diagnose why my latency experiment is missing from Grafana.

Test errors and document the blind spot

Deterministic HTTP 500 responses create a clear user-facing failure signal. This run tests whether process status reacts to those failed requests.

You have another 30-second experiment window. Keeping the Actuator request ready leaves enough time to compare both evidence sources.

Return to the second RHEL shell from the latency experiment.
Prepare the documented Actuator request for /actuator/health without submitting it.

Before you start, predict whether deterministic HTTP 500 responses cause the health status to change.

Switch back to the original repository shell.
Start the error experiment by running:

make chaos-errors DURATION=30

How the Error Target Works

The chaos-errors target generates deterministic server-error traffic.
The DURATION=30 argument keeps the experiment active for 30 seconds.
This run gives Grafana an error window to compare with the Actuator health response.

Switch to the prepared second shell.
Submit the documented Actuator request while the error experiment is active.

The response still reports the overall status as UP.

Return to the Grafana application dashboard from earlier.
Align the dashboard time window with the error run.
Inspect the application server-error panel.

You will see server errors increase during the same window. The health response continues to report a healthy process.

Press Windows logo key + Shift + S to capture the server-error panel.
Save the capture as errors-grafana.png in the location used by your file-transfer workflow.
Copy errors-grafana.png into docs/images using that workflow.
Confirm your repository editor lists docs/images/errors-grafana.png.

Error Evidence Missing?

Repeat the error command if the experiment finished before you checked both evidence sources.
Confirm the Grafana time window covers the 30-second run.
Wait for the next dashboard refresh before checking the error panel again.

Still seeing no error increase? Help me trace the error experiment from the observer application into Grafana.

Wait for the error target to finish its 30-second run.

Before you check recovery, predict whether the established health check should succeed after the controlled traffic stops.

Run the same repository health check from the earlier step:

make chaos-check

What the Recovery Check Proves

The existing chaos-check target verifies the observer application's established healthy state. A successful exit confirms the controlled traffic has ended without leaving the lab unhealthy.

The command completes successfully. Your observer application is back in its normal healthy state.

Return to the Grafana application dashboard.
Refresh the dashboard after the experiment window.

The current latency and error signals return toward their baseline levels. The historical experiment windows remain available as evidence.

What Makes This Evidence Useful?

The Baseline section records the normal application panel values. It embeds ../images/baseline-grafana.png.
The Latency section records UP from Actuator. It records elevated latency from Grafana. It embeds ../images/latency-grafana.png.
The Errors section records UP from Actuator. It records increased server errors from Grafana. It embeds ../images/errors-grafana.png.
Each experiment section states the prediction that was tested. Each section records the observed recovery result.
The conclusion explains that process liveness can coexist with degraded user experience.

Create docs/experiments/health-check-blind-spot.md in the existing repository editor.
Write the evidence sections using your observed values plus the relative image paths listed above.
Open the editor's Markdown preview for the guide.

The preview should show the baseline evidence first. It should then connect the latency and error observations to the green health response.

Save docs/experiments/health-check-blind-spot.md.
Add one evidence note below the README experiment table that links to docs/experiments/health-check-blind-spot.md.
Render README.md with the existing Markdown preview workflow.
Follow the new evidence link from the README preview.

The health-check blind-spot guide should open from the README. Its three screenshots support the written observations without replacing them.

Save README.md.

Before the final repository check, predict which documentation files and evidence images Git should list.

Review the intentional evidence additions by running:

git status --short

What Does This Status Show?

The short status from Git lists modified files plus new untracked files in a compact format. This check confirms that your evidence exists inside the repository before the next documentation step.

The output should include these intentional additions:

The updated README.md file.
The new docs/experiments/health-check-blind-spot.md guide.
The baseline-grafana.png evidence image.
The latency-grafana.png evidence image.
The errors-grafana.png evidence image.

You have captured the health-check blind spot with repeatable evidence. Next, you will layer application signals with host, service, kernel, and cgroup evidence to explain each failure precisely.

Explain Failures with Layered Evidence

Your baseline, latency, and error runs exposed the central blind spot. Spring Boot Actuator still reported UP while Grafana showed degraded user experience.

A credible incident explanation correlates application metrics with Prometheus, systemd, journald, and cgroups v2 evidence. Recovery proof closes each guide.

In this step, get ready to:

Document the real scope of the application experiments.
Compare application-local CPU work with single-worker host CPU pressure.
Complete the service-outage and cgroup OOM evidence guides.

Document the application experiments

The endpoint behavior defines the failure scope. Each guide needs to connect that behavior to the symptom a user experiences.

Use One Evidence Pattern

A repeatable evidence pattern makes every experiment easier to compare. Each guide should lead readers from the trigger to verified recovery.

Scope identifies the request thread, process, service, host worker, or cgroup affected by the experiment.
Expected impact states what a user should experience during the experiment.
Layered evidence records the application metric plus the relevant host, service, or kernel signal.
Limitations define what the experiment proves within its controlled conditions.
Recovery proof shows that the affected layer returned to its normal state.

In the editor from earlier, update docs/experiments/baseline.md with the control scope, expected normal signals, observed results, limitations, matching Grafana image, and normal-state verification.
Save docs/experiments/baseline.md.
Check the guide in the rendered Markdown preview.

You should see a control guide that establishes normal availability, latency, errors, and resource behavior before any failure is introduced.

Update docs/experiments/latency.md with the bounded request-thread sleep scope, the requested-versus-actual delay limitation, the UP health result, the matching Grafana image, the recovery action, and the recovery proof.
Save docs/experiments/latency.md.
Check the guide in the rendered Markdown preview.

You should see an explanation that /slow puts one request thread to sleep for a bounded duration. The guide should connect that behavior to elevated request latency while health remains UP.

Update docs/experiments/errors.md with deliberate HTTP 500 behavior from /error, the increased error signal, the UP health result, the matching Grafana image, the limitations, and the recovery proof.
Save docs/experiments/errors.md.
Check the guide in the rendered Markdown preview.

Good progress. The control, latency, and error guides now separate process health from user experience.

Compare application and host CPU scope

The application endpoint keeps one request thread busy for about ten seconds. The existing GNU Make target creates CPU pressure through one Bash busy loop on the host.

The application request takes about ten seconds to finish, so a quiet terminal during that interval is expected.

In the RHEL shell from earlier, send one request to /simulate-cpu with the repository's documented request workflow.
Watch the application CPU panels in Grafana while the request is active.

You should see CPU work associated with the observer application while one request thread remains busy. The signal should settle after the request finishes.

The host-pressure target runs for 30 seconds. Keep Grafana visible so the short test window does not pass unnoticed.

Before you run the host experiment, which CPU signal do you expect to change most clearly?

Start the single-worker host CPU experiment by running this command:

make host-cpu-stress CPU_DURATION=30

What Does This Target Prove?

The host-cpu-stress target starts one Bash busy loop on the host.
The CPU_DURATION=30 value bounds the pressure period to 30 seconds.
One busy worker proves single-worker host pressure. Full multi-core host saturation remains unproven because the experiment does not load every logical CPU.

Watch the host CPU panels in Grafana while the target runs.
Wait for the target to complete its 30-second run.

You should see host CPU pressure rise during the run. The increase may look modest on a multi-core machine because one worker cannot occupy every logical CPU.

After the target exits, you should see the host signal move back toward its normal level.

Host CPU Signal Missing?

Confirm that the Grafana time range includes the full 30-second experiment window.
Check that the host dashboard is showing the same machine where the Make target ran.
Use a narrower dashboard time range if the single-worker increase is difficult to distinguish from the longer baseline.

Still stuck? Help me compare the application CPU and host CPU signals in my Grafana panels.

Update docs/experiments/cpu-pressure.md with both experiment scopes, their user impact, their application and host signals, the single-worker limitation, and normal-state recovery proof.
Save docs/experiments/cpu-pressure.md.
Check the comparison in the rendered Markdown preview.

The guide should now show that /simulate-cpu creates application-local work in one request thread. It should show that host-cpu-stress creates host pressure through one Bash worker.

Document service and cgroup failures

A stopped service and a cgroup OOM are hard failures with evidence below the application layer. Their guides need service state, journal records, cgroup or kernel evidence, and verified recovery.

Why Cgroup Evidence Matters

The memory.max value is the cgroup v2 memory hard limit. If memory usage reaches that limit and cannot be reduced, the OOM killer acts inside the cgroup.

That evidence identifies the failure domain more precisely than a missing application response alone.

The stopped-service run is deliberate and temporary. Your existing recovery procedure returns observer.service to its normal state before you continue.

Before you trigger the outage, which evidence source do you expect to identify the stopped service most directly?

Re-run the stopped-service scenario with its existing documented trigger.
Inspect the corresponding availability result in Prometheus.
Inspect the stopped observer.service state through the existing systemd workflow.
Inspect the matching service records through the existing journald workflow.
Apply the documented service recovery action.
Retrieve the health response from /actuator/health.

You should see the outage reflected in availability evidence plus the service state and journal records. After recovery, the health response should report UP.

Service Did Not Recover?

Confirm that you applied the same restart procedure documented by the original outage experiment.
Check the current service state before repeating the health request.
Review the latest journal records for the first startup problem reported after recovery.

Need another pair of eyes? Help me trace why observer.service did not return to a healthy state.

Capturing both the outage and recovery in one image can be fiddly. Keep the relevant terminal history visible before taking the screenshot.

Keep the stopped-state and recovered-state evidence visible in the terminal history.
Capture that evidence with the screenshot workflow from earlier.
Copy the PNG to docs/images/service-outage-systemd.png.
Confirm that service-outage-systemd.png appears under docs/images in the editor sidebar.

Update docs/experiments/service-outage.md with the outage scope, user impact, Prometheus and Grafana evidence, systemd state, journald evidence, limitations, restart action, and healthy-service proof.
Save docs/experiments/service-outage.md.
Check the service-outage evidence card in the rendered Markdown preview.

The cgroup OOM experiment crosses another safety boundary, so follow the existing memory limit and recovery procedure exactly. The failure remains contained within the lab's established controls.

Before you trigger the OOM scenario, which evidence do you expect to explain the failure most precisely?

Re-run the cgroup OOM scenario with its existing documented trigger.
Inspect the corresponding availability and application results in Prometheus.
Inspect the matching memory and availability panels in Grafana.
Inspect the relevant service records through the existing journald workflow.
Inspect the matching cgroup or kernel OOM evidence.
Apply the documented cgroup OOM recovery action.
Retrieve the health response from /actuator/health.

You should see application or availability symptoms supported by cgroup or kernel-level OOM evidence. After recovery, observer.service should return to its normal healthy state with an UP health response.

OOM Recovery Incomplete?

Confirm that the original memory constraint has been removed or reset through the documented recovery procedure.
Check the current service state before repeating the health request.
Review the latest journal and cgroup evidence for another memory-related termination.

Still seeing an unhealthy service? Help me correlate my cgroup OOM evidence with the service recovery state.

Keep the OOM evidence and recovered service state visible in the terminal history.
Capture that evidence with the screenshot workflow from earlier.
Copy the PNG to docs/images/cgroup-oom-evidence.png.
Confirm that cgroup-oom-evidence.png appears under docs/images in the editor sidebar.

Update docs/experiments/cgroup-oom.md with the failure scope, user impact, Prometheus and Grafana evidence, service state, journald evidence, cgroup or kernel proof, limitations, recovery action, and normal-state proof.
Save docs/experiments/cgroup-oom.md.
Check the cgroup OOM evidence card in the rendered Markdown preview.

Return to the existing experiment table in README.md.
Link each of its seven scenario rows to the matching guide under docs/experiments.
Save README.md.
Follow each guide link in the rendered README preview.

Each relative link should open the guide for the experiment named in that row.

Before the final review, which scenarios should contain service or kernel evidence beyond application metrics?

Return to docs/experiments in the editor from earlier.
Review baseline.md, latency.md, errors.md, cpu-pressure.md, service-outage.md, and cgroup-oom.md against the layered-evidence pattern.
Confirm that both CPU scenarios appear as distinct scopes inside cpu-pressure.md.
Confirm that observer.service is healthy with the existing service check.
Confirm that /actuator/health reports UP.
Confirm that Prometheus is still scraping the observer application.
Confirm that Grafana still shows the normal application and host state.

You should find all seven scenarios documented with scope, expected user impact, application evidence, host or service evidence, relevant kernel or cgroup evidence, limitations, recovery actions, and recovery proof.

That completes the evidence chain. Another engineer can now trace each symptom to the correct failure domain and verify its recovery.

Your experiment guides now explain what every controlled failure proves. Next up, you will turn those guides into a compact incident field guide that readers can follow without guessing.

Publish the Incident Field Guide

Your seven experiment guides now explain every controlled failure across your RHEL lab. Each guide connects the user symptom to evidence from the correct layer.

Raw screenshots and commands are not a portfolio story until a reader can move from symptom to diagnosis. The reader also needs a clear path from evidence to recovery.

This step turns README.md into that path. A final Git review makes the local repository push-ready without assuming that a remote is already configured.

In this step, get ready to:

Complete the reader journey through the root incident field guide.
Embed evidence with accessible repository-relative image references.
Add a five-stage incident runbook before passing the final repository checks.

Complete the reader journey

A useful field guide answers the reader's next question before they need to guess. The root README should lead from context through experimentation to recovery.

Open README.md from the repository root in the editor you used for the experiment guides.
Add a Prerequisites section that names the working observer repository, healthy observer.service, Prometheus, Grafana, GNU Make, curl, and Git.
Add a Key Findings section that explains how /actuator/health can remain UP during latency or error degradation. Include the scope difference between application-local CPU work and single-worker host CPU pressure.
Read the new prerequisites and findings to confirm that a reader can understand the lab assumptions without opening a detailed guide.
Add a Repository Structure section that points readers to README.md, docs/experiments, and docs/images.
Polish the reader path from the existing quick start through the experiment index so every scenario links to its detailed guide.
Read the full README to confirm that the pitch, architecture, prerequisites, quick start, seven-scenario index, safety boundaries, key findings, guide links, and repository structure appear in a logical order.

What makes this a field guide?

The README provides the route through the lab. The detailed guides hold the deeper evidence for each experiment.

This split lets a reviewer understand the overall incident method first. They can then inspect any experiment without losing the wider story.

Embed evidence with accessible context

Repository-relative Markdown image paths keep the evidence portable with the repository. Meaningful alt text also explains what each image contributes when the image cannot be viewed.

Embed docs/images/baseline-grafana.png beside the control baseline summary using alt text that describes the normal Grafana state.
Embed docs/images/latency-grafana.png beside the latency finding using alt text that identifies elevated request latency.
Inspect both image references to confirm that each path starts with docs/images/ and each alt description identifies a different observation.
Embed docs/images/errors-grafana.png beside the HTTP error finding using alt text that identifies increased server errors.
Embed docs/images/service-outage-systemd.png beside the service-outage finding using alt text that describes the service state or recovery evidence.
Inspect the error and outage references to confirm that their surrounding text explains what the screenshots prove.
Embed docs/images/cgroup-oom-evidence.png beside the memory OOM finding using alt text that identifies the cgroup or kernel evidence.
Review every evidence section to confirm that its written conclusion remains understandable without the screenshot.

Let Screenshots Support the Claim

A screenshot proves that an observation occurred at a particular moment. The surrounding text should still explain the scope, limitation, recovery action, and recovery result.

Add the incident runbook and verify

A compact incident runbook converts the evidence collection into a repeatable decision path. Its order keeps diagnosis focused before any recovery action changes the system.

How the Runbook Flows

Detect identifies which golden signals moved away from the baseline.
Scope separates an application symptom from host pressure, service failure, or a cgroup boundary.
Inspect correlates Grafana panels, Prometheus results, systemd state, journald records, and cgroup evidence.
Recover applies the scenario-specific rollback from the detailed guide.
Verify proves that the user-facing signal and supporting system state returned to normal.

Add a Detect stage under a new Incident Runbook section that directs the reader to availability, latency, errors, CPU, and memory signals.
Add a Scope stage that separates application, host, service, and cgroup failure domains.
Read the first two stages to confirm that the reader identifies the symptom before choosing a deeper evidence source.
Add an Inspect stage that directs the reader to the evidence sources relevant to the suspected failure domain.
Add a Recover stage that directs the reader to the rollback documented in the matching experiment guide.
Read the scope and inspection guidance to confirm that neither stage treats one dashboard as a complete diagnosis.

Why This Order Works

Recovery can erase temporary evidence or change the failure state. Inspecting before recovery preserves the chain between the symptom, its scope, and its cause.

Add a Verify stage that checks observer.service, /actuator/health, application metrics, and a normal user request.
Add a closing Lessons Learned section that distinguishes liveness, user experience, resource pressure, and hard failure.

The README now takes a reader from the first symptom through evidence-backed recovery. Its final lesson also explains why a green process-level check cannot represent every dimension of service health.

Save README.md in your editor.

Before you run the first final check, do you expect Git to report a formatting problem or return quietly?

Check the completed diff for whitespace or conflict-marker errors by running:

git diff --check

What Does This Check Do?

The git diff --check command scans the current diff for whitespace errors and conflict markers. It returns control to the shell when the scan finishes.

You should see no output before the shell prompt returns. That quiet result confirms the formatting check passed.

Formatting Check Reports a Problem?

Use the reported file and line number to locate the affected text in your editor.
Remove trailing whitespace or leftover conflict markers before running the check again.

Still seeing a problem? Help me fix the formatting issue reported by my Git diff check.

Before checking the repository status, which README, guide, and PNG paths do you expect Git to list?

Review every changed or untracked path by running:

git status --short

What Does This Check Show?

The git status --short command presents the working-tree status in a compact format. This makes unrelated changes easier to spot before a commit.

You should see README.md plus the guide and PNG paths you intentionally changed. You should not see unrelated application or service files.

Seeing an Unexpected Path?

Compare every listed path with the documentation and evidence files you intentionally changed.
Inspect any unrelated path before you decide whether it belongs in the future commit.

Unsure about a listed file? Help me review an unexpected path in my Git status safely.

That completes the field guide. Your local repository now presents the lab as a reproducible incident investigation with clear evidence, limits, recovery steps, and lessons.

Secret mission

Build an Observability Coverage Matrix

Your incident guides explain each failure separately. Build a coverage matrix that compares all seven experiments, exposes healthy but misleading signals, and identifies the next monitoring gap to close.

Clean Up Your Resources

Everything you created for this project lives in your local observer repository on RHEL. The project costs $0 because it creates no new paid infrastructure.

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

Uncommitted updates to README.md in your Git working tree.
Evidence-backed experiment guides under docs/experiments.
The observability coverage matrix at docs/experiments/observability-coverage-matrix.md.
PNG evidence under docs/images.

Keep everything running

No action needed. Choose this if you want the observability lab to remain ready for demonstrations or a future push.

Keep the completed documentation in your local observer repository.
Leave observer.service in its current healthy state.
Keep Prometheus running in its normal scraping state.
Keep Grafana available with the retained evidence views.

Pause - I'll come back to this later

All controlled experiments have finished. Your observer service and monitoring stack are already in their normal healthy state.

Leave observer.service in its current healthy state.
Leave Prometheus in its normal scraping state.
Leave Grafana in its normal dashboard state.
Close the RHEL shell when you are ready to step away.

Delete - I don't want to use this again

Remove this project's documentation changes if you do not want to keep the portfolio layer. The original monitoring stack remains untouched.

This deletion touches only the portfolio files created or updated during this project.

Review the changed paths in your repository by running this command:

git status --short

What Does This Command Show?

This command lists the changed or untracked paths in Git's short format. Use the output to confirm that the documentation paths match the resources listed above.

Inspect the uncommitted changes in README.md by running:

git diff -- README.md

What Does This Diff Show?

This diff isolates the uncommitted changes in the root incident field guide. It gives you a final chance to review the project story before removing it.

Use the same text editor that you used for the incident guide to reverse each uncommitted change shown for README.md.
Remove the experiment guides and PNG evidence by running this command:

rm -rf docs/experiments docs/images

What Does This Command Remove?

Recursive removal deletes docs/experiments and docs/images from the repository. It does not stop the observer service or monitoring stack.

Folders Still Listed?

Confirm that your shell prompt still points to the observer repository root.
Check that both directory paths match the names shown in the resource list.
Help me remove these folders without touching the original monitoring stack.

Confirm that the documentation changes are gone by running:

git status --short

What Should I See?

Git should no longer list README.md, docs/experiments, or docs/images as changed paths. Any unrelated pre-existing changes remain visible.

Cleanup is complete. Your original observability lab remains intact.

Nice Work!

You did it! Your Red Hat Enterprise Linux observability repository is now a push-ready local incident field guide backed by reproducible evidence.

You've learned how to:

Built a portfolio-ready incident field guide that another engineer can follow from symptom to verified recovery.
Classified failure scope across application requests, single-worker host CPU pressure, service outages, and cgroup OOM events.
Proved that green liveness can hide user-facing degradation during latency and error experiments. Correlated each result using Prometheus metrics, Grafana views, systemd state, and journald records.
Secret Mission: Built an observability coverage matrix for all seven experiments. The matrix exposes healthy-but-misleading signals plus future coverage gaps.

Ready to quiz yourself?