Build an Observable Linux Service

Manage a Spring Boot service with systemd, Prometheus, and Grafana on RHEL.

Introduction

30 Second Summary

A server can answer requests perfectly one minute and disappear the next. Without a clear record of what changed, recovery becomes guesswork.

In this project, you will build a small Spring Boot service and operate it as a systemd-managed Linux workload. You will monitor its host and JVM signals with Prometheus and Grafana before practicing a complete outage response.

What You'll Build

Your demo starts with a healthy dashboard, turns into a firing availability alert, and finishes with every signal healthy again after an evidence-led recovery.

By the end of this project, you'll have:

  • A local service check that proves the application, health endpoint, and metrics endpoint are responding from the VM.
  • A live monitoring view where you can watch service availability, host CPU activity, JVM memory, and alert state change during an outage.
  • A repeatable incident workflow for detecting a failure, diagnosing it with system evidence, restoring service, and preserving the evidence behind your decisions.
  • Secret Mission: Apply a restrictive cgroups v2 memory guardrail, prove why the service fails under the limit, and restore it safely.

Are there any prerequisites?

You need a clean Red Hat Enterprise Linux 10.2 VM with sudo access, outbound HTTPS access, and enabled RHEL repositories. Your Mac needs SSH access to that VM.

Before We Start

This is your moment to lock in the observable Linux service you are building. You will also connect its monitoring baseline to the incident skills you want to prove.

Prepare the RHEL Engineering Workstation

An incident lab is only reliable when its host is known. A hidden version mismatch can make later failures look like service incidents.

Here, you will verify your Red Hat Enterprise Linux 10.2 VM over SSH. You will install Red Hat build of OpenJDK 21 with Apache Maven 3.10.0 before recording a clean Git baseline.

In this step, get ready to:
  • Verify the RHEL host meets the incident lab requirements.
  • Install the pinned build toolchain.
  • Prepare the observer repository for versioned work.
Verify the RHEL host

The first check establishes what machine you are operating. It also proves that administrative package installation is available before the lab depends on it.

  • Press Cmd+Space on your Mac to open Spotlight.
  • Type Terminal into Spotlight.
  • Press Enter to open Terminal.
  • Connect to the RHEL VM by running this command:
ssh [[RHEL_USER="your RHEL username"]]@[[RHEL_HOST="your RHEL VM hostname or IP address"]]

What does this command do?

The ssh client creates an encrypted remote shell on your RHEL VM. The prompt changes to identify the remote user or host after authentication succeeds.

Your authentication details stay between your Mac and the VM during this connection.

  • Compare the displayed host fingerprint with the fingerprint supplied for your VM.
  • Enter yes if the fingerprints match.
  • Complete the authentication prompt with your existing VM credentials.
  • Confirm that your terminal prompt now identifies the RHEL VM.

Cannot reach the VM?

Check that the VM is running. Confirm that the hostname or IP address matches your SSH access details.

If authentication fails after the connection starts, confirm that you selected the expected private key or entered the correct remote username.

Help me diagnose my SSH connection to the RHEL VM.

The host baseline covers release identity, privilege access, software sources, network access, memory, storage, and CPU architecture. Each result rules out a different environmental cause of later failures.

  • Collect the host baseline by running these checks:
cat /etc/redhat-release
sudo whoami
dnf repolist
curl "https://echo-api.3scale.net:443"
free -m
df -h
uname -m

What should these checks prove?

  • The release output identifies RHEL 10.2.
  • The privilege check prints root.
  • The repository check lists enabled software sources.
  • The HTTPS request returns a response from the remote test endpoint.
  • The capacity checks report available memory plus disk space.
  • The architecture check reports x86_64 or aarch64.

You now have evidence that the VM can support the installation path. This baseline separates host setup problems from the service incidents you create later.

Does a baseline check fail?

An empty repository list means the VM cannot supply the RHEL packages used in this project. Restore access to the enabled RHEL repositories before continuing.

A failed HTTPS request points to outbound networking or DNS. A different release or architecture falls outside this lab's verified environment.

Help me interpret my failed RHEL environment check.

Install the pinned toolchain

The observer service needs a Java compiler plus a repeatable build runner. RHEL packages also provide GNU Make plus the shell utilities used by later installation scripts.

The package manager displays a transaction summary before changing the VM. Review the package list before approving the prompt.

  • Install Git, GNU Make, curl, tar, and gzip from the enabled repositories by running:
sudo dnf install git make curl tar gzip

What does this command install?

These packages provide source control, Makefile execution, HTTPS downloads, and archive extraction. DNF resolves their required dependencies from the repositories you just verified.

  • Enter y if the transaction summary contains the expected packages.

Package installation fails?

A repository error means the package source check needs attention. A permission failure means the connected account lacks the required sudo access.

Help me diagnose the failed RHEL package installation.

Java plus Maven may already exist on the VM. Checking first avoids replacing a correct pinned toolchain.

  • Check the installed Java and Maven versions by running:
java -version
mvn -v

How do I choose a tab?

Use the first tab only when both commands report the required versions. Use the older-version tab when either command reports a lower release.

Use the missing-command tab when the shell cannot run Java or Maven.

✔️ I see Java 21 and Maven 3.10.0

Good, the two pinned version checks already pass. Your VM can use the existing Java plus Maven installations.

ⓧ I see an older version

RHEL can keep multiple Java versions installed. You will install OpenJDK 21 before selecting it as the system-wide Java runtime.

The package installer requests approval during the Java installation. The Maven archive download continues after that package transaction finishes.

  • Install OpenJDK 21 plus Apache Maven 3.10.0 by running these commands:
sudo yum install java-21-openjdk-devel
curl --fail --location --show-error --silent --output apache-maven-3.10.0-bin.tar.gz https://downloads.apache.org/maven/maven-3/3.10.0/binaries/apache-maven-3.10.0-bin.tar.gz
tar xzvf apache-maven-3.10.0-bin.tar.gz
sudo mv apache-maven-3.10.0 /opt/apache-maven-3.10.0
echo 'export PATH=/opt/apache-maven-3.10.0/bin:$PATH' >> ~/.bashrc
source ~/.bashrc

What does this installation do?

The first command installs the RHEL development kit. The remaining commands download the pinned Maven binary distribution.

The archive is installed under /opt/apache-maven-3.10.0. The shell configuration places its executable before an older Maven installation.

Selecting the Java version is a little fiddly when several JDKs are present. The alternatives screen lists every installed Java path with a selection number.

  • Display the installed Java alternatives by running:
  • Choose the selection whose path contains the OpenJDK 21 installation.
sudo update-alternatives --config 'java'

What changes here?

The selected alternative becomes the system-wide target for the java command. This keeps older JDKs installed while the observer project uses OpenJDK 21.

ⓧ A command is missing

The missing command confirms that this VM needs the pinned Java plus Maven toolchain. The installation uses the enabled RHEL repositories plus Apache's binary archive.

The package installer requests approval during the Java installation. The Maven archive download continues after that package transaction finishes.

  • Install OpenJDK 21 plus Apache Maven 3.10.0 by running these commands:
sudo yum install java-21-openjdk-devel
curl --fail --location --show-error --silent --output apache-maven-3.10.0-bin.tar.gz https://downloads.apache.org/maven/maven-3/3.10.0/binaries/apache-maven-3.10.0-bin.tar.gz
tar xzvf apache-maven-3.10.0-bin.tar.gz
sudo mv apache-maven-3.10.0 /opt/apache-maven-3.10.0
echo 'export PATH=/opt/apache-maven-3.10.0/bin:$PATH' >> ~/.bashrc
source ~/.bashrc

What does this installation do?

The RHEL package supplies the Java development kit required to compile the service. The Apache archive supplies the exact Maven release used by this project.

The final shell setting makes mvn available in the current session plus future Bash sessions.

Before you run the final toolchain check, which versions plus architecture do you expect the VM to report?

  • Verify every required executable by running these checks:
java -version
mvn -v
git --version
make --version
uname -m

What should I see?

The output reports Java 21 plus Apache Maven 3.10.0. Git plus GNU Make each report an installed version.

The final line reports x86_64 or aarch64.

Still seeing the wrong version?

If Maven still reports an older release, start a new SSH shell so the updated Bash path is loaded. If Java still reports an older major version, repeat the alternatives selection for the OpenJDK 21 path.

Help me fix my Java or Maven version mismatch.

Create the observer repository

A repository gives the lab a known starting point before service code appears. The first commit records the ignore rules that keep build output plus generated evidence out of version control.

  • Create the observer folder on your Desktop by running these commands:
mkdir -p ~/Desktop
cd ~/Desktop
mkdir observer
cd observer
git init

What does this setup create?

The commands place the project at ~/Desktop/observer. Git initializes its hidden repository metadata inside that folder.

  • Create .gitignore with the project ignore rules by running:
cat > .gitignore <<'EOF'
target/
evidence/*.txt
evidence/*.json
*.log
EOF

What do these ignore rules protect?

  • The target/ rule excludes Maven build output.
  • The evidence rules exclude generated text plus JSON captures.
  • The final rule excludes runtime log files.
  • Display the saved ignore rules by running:
cat .gitignore

What should I see?

You should see four entries. They should match the build, evidence, and log patterns shown above.

Ignore rules look incomplete?

Run the file creation command again from ~/Desktop/observer. Confirm that the closing EOF appears on its own line.

Help me fix my .gitignore file.

✔️ Awesome, I've got everything!

Great, your ignore rules are saved. The repository is ready for its first checkpoint.

ⓧ I'd like to double check the full code

target/
evidence/*.txt
evidence/*.json
*.log

Before you create the checkpoint, what commit message do you expect Git to record?

  • Stage the repository files plus create the initial checkpoint by running:
git add .
git commit -m 'Initialize observable Linux service'

What should the checkpoint show?

Git reports a new initial commit named Initialize observable Linux service. The commit includes .gitignore as the repository's first tracked file.

Commit blocked by Git identity?

Git needs an author name plus email before it can create a commit. Configure the identity requested by your terminal before rerunning the two checkpoint commands.

Help me resolve the Git commit failure without changing the project files.

Your RHEL workstation now has a verified host baseline, a pinned build toolchain, and a clean Git checkpoint. Next up, you will turn that baseline into the first runnable observer service.

Build the Observable Service

Your RHEL workstation now has a verified Java 21 toolchain. The repository also has a clean Git baseline.

This step creates a small Spring Boot workload before systemd or dashboards add complexity. Running it in the foreground proves that the Java process serves HTTP responses, health data, and Prometheus metrics.

In this step, get ready to:
  • Create the Maven build for the observer service.
  • Build the Spring Boot application and run it in the foreground.
  • Prove each endpoint works before capturing evidence and committing the workload.
Create the Maven build

A Maven project uses pom.xml to define its identity, Java version, dependencies, and build plugin. You will assemble the file in small sections so each change gets an immediate build check.

  • Create pom.xml in the observer repository with your SSH terminal editor.
  • Add the project metadata with this code:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>4.1.1</version>
    </parent>

    <groupId>com.nextwork</groupId>
    <artifactId>observer</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>observer</name>
    <description>Observable Linux service baseline</description>

    <properties>
        <java.version>21</java.version>
    </properties>
</project>

What does this configuration do?

  • The Spring Boot parent supplies managed dependency and plugin versions for release 4.1.1.
  • The project coordinates produce an artifact named from observer and 0.0.1-SNAPSHOT.
  • The java.version property compiles the project for Java 21.
  • Save pom.xml.
  • Check the project metadata by running this command:
mvn package

What does this build prove?

Maven reads pom.xml and validates the project structure. A successful package phase confirms that the initial build model is valid.

You should see Maven finish successfully. The observer artifact gains its application code in the next chunk.

Maven cannot read the project?

Confirm that pom.xml sits beside .gitignore in the observer repository. Check that the final </project> tag is present.

Compare every XML tag with the reference below. Help me diagnose my pom.xml build error.

The service needs web routing, operational endpoints, and Prometheus-formatted metrics. Maven downloads those capabilities from the dependencies declared in the project model.

  • In pom.xml, find the closing </properties> tag.
  • Insert this dependency group below that tag:
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-webmvc</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-prometheus</artifactId>
        </dependency>
    </dependencies>

What do these dependencies provide?

  • The web starter supplies the controller support used by the root endpoint.
  • Spring Boot Actuator supplies health and operational endpoints.
  • The Micrometer registry converts application measurements into Prometheus format.
  • Save pom.xml.
  • Resolve the new dependencies by running this command:
mvn package

What happens during this build?

Maven resolves the three dependency trees declared in pom.xml. The first dependency download can take longer than later builds because the local Maven cache starts empty.

You should see the package phase complete successfully after the dependencies finish downloading.

Dependency download failed?

Confirm that the VM still has outbound HTTPS connectivity. Check each groupId and artifactId against the code block.

A repository access error can also indicate that the VM lost network access. Help me diagnose my Maven dependency download failure.

The build plugin turns the packaged application into an executable JAR. That artifact becomes the workload managed by systemd later in the project.

  • In pom.xml, find the closing </dependencies> tag.
  • Insert this build configuration below that tag:
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

Why add the build plugin?

The Spring Boot Maven plugin packages the application with the runtime components it needs. The resulting JAR can start through the Java launcher without a separate application server.

  • Save pom.xml.
  • Validate the completed Maven model by running this command:
mvn package

What does this check confirm?

Maven now parses the project metadata, dependencies, and build plugin together. A successful result confirms that the completed pom.xml is ready for application code.

Your Maven build is ready. The next build will compile a real endpoint into target/observer-0.0.1-SNAPSHOT.jar.

Seeing an XML build error?

Confirm that <dependencies> and <build> both sit inside the outer <project> element. Make sure each opening tag has one matching closing tag.

Use the complete reference to check the section order. Help me find the structural error in my completed pom.xml.

Add the service and workflows

The application has one controller with two jobs. It returns a visible service response at the root path and starts the Spring Boot runtime from main().

  • Create the Java package directories by running this command:
mkdir -p src/main/java/com/nextwork/observer src/main/resources

What does this command create?

The command creates the Java package path and the resources path under src/main. Maven reads source code and application configuration from these conventional locations.

  • Confirm the two directories exist by running this command:
ls src/main/java/com/nextwork/observer src/main/resources

What should this list show?

The command prints both directory headings without a missing-path error. Empty output beneath each heading is expected because the files come next.

Directory path missing?

Confirm that your SSH session is inside the observer repository. Run the directory creation command again if either path was typed differently.

The package path must end with com/nextwork/observer. Help me correct my observer source directory structure.

  • Create src/main/java/com/nextwork/observer/ObserverApplication.java with your SSH terminal editor.
  • Add the application class with this code:
package com.nextwork.observer;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@SpringBootApplication
public class ObserverApplication {

    @GetMapping("/")
    public String home() {
        return "observer-service: ok\n";
    }

    public static void main(String[] args) {
        SpringApplication.run(ObserverApplication.class, args);
    }
}

What does this application do?

  • The package declaration matches the directory path so Java can locate the class correctly.
  • The controller maps an HTTP GET request for / to home().
  • The root response provides a simple service-availability signal.
  • The main() method starts the Spring Boot application.
  • Save src/main/java/com/nextwork/observer/ObserverApplication.java.
  • Compile the application by running this command:
mvn package

What does this build prove?

Maven compiles ObserverApplication and repackages it as an executable Spring Boot JAR. The build output is target/observer-0.0.1-SNAPSHOT.jar.

That is the first executable workload complete. Your Java class now compiles into the JAR that the project will operate.

Java compilation failed?

Check that the filename uses the exact capitalization ObserverApplication.java. Confirm that its package path ends with com/nextwork/observer.

Compare the imports and braces with the complete reference. Help me diagnose my ObserverApplication.java compilation error.

The service should listen only on the VM loopback interface. Its operational endpoints expose health and metrics locally without adding a firewall opening.

  • Create src/main/resources/application.properties with your SSH terminal editor.
  • Add the local binding and Actuator configuration with this code:
server.address=127.0.0.1
management.endpoints.web.exposure.include=health,prometheus
management.endpoint.health.show-details=always

What does this configuration control?

  • The server binds to 127.0.0.1 so it accepts requests only from the VM itself.
  • The exposure list publishes the health and Prometheus endpoints through Actuator.
  • The health setting includes component details in the local health response.
  • Save src/main/resources/application.properties.
  • Package the configuration into the JAR by running this command:
mvn package

What changed in the artifact?

Maven copies application.properties into the executable JAR. The next application launch uses the loopback binding and endpoint exposure settings.

You should see the package phase succeed again. The rebuilt JAR now contains both the service code and its runtime configuration.

Configuration build failed?

Confirm that the file is named application.properties and sits directly inside src/main/resources. Check each property for extra spaces inside its key.

Use the complete reference to compare all three lines. Help me check my Spring Boot application.properties file.

A Makefile gives the project short, repeatable commands for building, running, and checking the service. In this step, you will use the build, run, and smoke targets.

  • Create Makefile beside pom.xml with your SSH terminal editor.
  • Add the target declaration and foreground service workflows with this code:
SHELL := /usr/bin/bash

.PHONY: all build run smoke install-app install-monitoring validate-monitoring status incident-down incident-up evidence pause resume uninstall clean

all: build

build:
	mvn package

run:
	mvn spring-boot:run

smoke:
	curl --fail --silent --show-error http://localhost:8080/
	@printf '\n'
	curl --fail --silent --show-error http://localhost:8080/actuator/health
	@printf '\n'
	curl --fail --silent --show-error http://localhost:8080/actuator/prometheus > /dev/null
	@echo "Prometheus endpoint returned successfully"

What do these targets provide?

  • The build target packages the executable JAR.
  • The run target starts the application in the foreground.
  • The smoke target checks the root, health, and Prometheus endpoints.
  • The .PHONY declaration keeps target names from colliding with files of the same name.
  • Save Makefile.
  • Test the first Make workflow by running this command:
make build

What does this target prove?

GNU Make reads the target and executes its Maven recipe. A successful result proves that the Makefile uses valid recipe indentation and can build the observer JAR.

You should see Maven complete successfully through the shorter project command.

Make reports a recipe error?

Makefile recipe lines must begin with a tab. Replace leading spaces before mvn and curl with one tab.

Confirm that the filename is exactly Makefile with a capital M. Help me diagnose my Makefile recipe error.

The remaining targets establish the project interface used by later service operations. Leave them in place without running them during this step because their scripts and managed services belong to later sections.

  • Append these installation and incident-control targets below the smoke target:
install-app: build
	sudo bash scripts/install-app.sh

install-monitoring:
	sudo bash scripts/install-monitoring.sh

validate-monitoring:
	sudo /usr/local/bin/promtool check config /etc/prometheus/prometheus.yml
	sudo /usr/local/bin/promtool check rules /etc/prometheus/alerts.yml

status:
	systemctl is-active observer.service
	systemctl is-active prometheus.service
	systemctl is-active node_exporter.service
	systemctl is-active grafana-server.service

incident-down:
	sudo systemctl stop observer.service

incident-up:
	sudo systemctl start observer.service

Why define these targets now?

These targets keep the repository's operational interface in one Makefile. Their recipes cover application installation, monitoring validation, service status, and controlled outage actions.

  • Save Makefile.
  • Confirm that Make can still parse the expanded file by running this command:
make build

What does this parse check prove?

GNU Make reads the entire file before selecting the build target. A successful build confirms that the newly added target definitions did not break the Makefile syntax.

You should see the observer JAR build successfully again.

Make stopped parsing the file?

Check that each target name ends with a colon. Confirm that every command beneath a target begins with a tab.

Pay close attention to the two commands under validate-monitoring and the four commands under status. Help me locate the syntax error in my expanded Makefile.

  • Append the evidence and lifecycle targets below the incident-up target:
evidence:
	mkdir -p evidence
	systemctl status observer.service > evidence/observer-status.txt || true
	journalctl -u observer.service -b > evidence/observer-journal.txt
	curl --fail --silent --show-error http://localhost:8080/actuator/health > evidence/observer-health.json || printf '{"status":"unreachable"}\n' > evidence/observer-health.json

pause:
	sudo systemctl stop observer.service prometheus.service node_exporter.service grafana-server.service

resume:
	sudo systemctl start node_exporter.service prometheus.service grafana-server.service observer.service

uninstall:
	sudo bash scripts/uninstall.sh

clean:
	mvn clean

What do the final targets cover?

The final group standardizes evidence capture, service pausing, service resumption, removal, and Maven cleanup. Keeping these commands behind named targets makes the administrator workflow reproducible.

  • Save Makefile.
  • Confirm the completed Makefile still builds the service by running this command:
make build

What does the completed build confirm?

The command validates the completed Makefile syntax and rebuilds the configured Spring Boot application. The expected artifact remains target/observer-0.0.1-SNAPSHOT.jar.

The full project interface now parses successfully. You are ready to compare every file before launching the service.

Final Makefile check failed?

Compare the final target group with the reference below. Check the quoting around the fallback JSON in the evidence recipe.

Confirm that the pause, resume, uninstall, and clean targets each have a colon. Help me compare my complete Makefile with the expected version.

✔️ Awesome, I've got everything!

Great. Save every project file before launching the service.

ⓧ I'd like to double check the full code

target/
evidence/*.txt
evidence/*.json
*.log
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>4.1.1</version>
    </parent>

    <groupId>com.nextwork</groupId>
    <artifactId>observer</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>observer</name>
    <description>Observable Linux service baseline</description>

    <properties>
        <java.version>21</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-webmvc</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-prometheus</artifactId>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>
package com.nextwork.observer;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@SpringBootApplication
public class ObserverApplication {

    @GetMapping("/")
    public String home() {
        return "observer-service: ok\n";
    }

    public static void main(String[] args) {
        SpringApplication.run(ObserverApplication.class, args);
    }
}
server.address=127.0.0.1
management.endpoints.web.exposure.include=health,prometheus
management.endpoint.health.show-details=always
SHELL := /usr/bin/bash

.PHONY: all build run smoke install-app install-monitoring validate-monitoring status incident-down incident-up evidence pause resume uninstall clean

all: build

build:
	mvn package

run:
	mvn spring-boot:run

smoke:
	curl --fail --silent --show-error http://localhost:8080/
	@printf '\n'
	curl --fail --silent --show-error http://localhost:8080/actuator/health
	@printf '\n'
	curl --fail --silent --show-error http://localhost:8080/actuator/prometheus > /dev/null
	@echo "Prometheus endpoint returned successfully"

install-app: build
	sudo bash scripts/install-app.sh

install-monitoring:
	sudo bash scripts/install-monitoring.sh

validate-monitoring:
	sudo /usr/local/bin/promtool check config /etc/prometheus/prometheus.yml
	sudo /usr/local/bin/promtool check rules /etc/prometheus/alerts.yml

status:
	systemctl is-active observer.service
	systemctl is-active prometheus.service
	systemctl is-active node_exporter.service
	systemctl is-active grafana-server.service

incident-down:
	sudo systemctl stop observer.service

incident-up:
	sudo systemctl start observer.service

evidence:
	mkdir -p evidence
	systemctl status observer.service > evidence/observer-status.txt || true
	journalctl -u observer.service -b > evidence/observer-journal.txt
	curl --fail --silent --show-error http://localhost:8080/actuator/health > evidence/observer-health.json || printf '{"status":"unreachable"}\n' > evidence/observer-health.json

pause:
	sudo systemctl stop observer.service prometheus.service node_exporter.service grafana-server.service

resume:
	sudo systemctl start node_exporter.service prometheus.service grafana-server.service observer.service

uninstall:
	sudo bash scripts/uninstall.sh

clean:
	mvn clean
Run and prove every endpoint

Foreground execution keeps the Java process and its logs visible in your SSH session. A second SSH terminal can test the loopback-only endpoints while the first terminal stays attached to the service.

  • Start the observer service in the current SSH terminal by running this command:
make run

What happens during foreground execution?

Maven starts the Spring Boot process and keeps control of this terminal. The process listens on the VM loopback interface at port 8080.

The first launch can pause while Maven resolves remaining runtime dependencies. Leave this terminal running when the startup logs settle.

  • Keep the first SSH terminal connected with the service running.
  • Return to the terminal application on your Mac.
  • Create a second terminal window.
  • Reconnect to the same RHEL VM with the SSH command you used earlier.
  • Return to the observer repository in the second SSH session.

Service stopped during startup?

Look in the first terminal for a build or startup failure. Confirm that port 8080 is free and that the latest make build completed successfully.

Keep the application running in the first terminal before testing from the second. Help me diagnose why make run cannot start my observer service.

Before you run the smoke test, which three service capabilities do you expect it to prove?

  • Run the complete endpoint check from the second SSH session:
make smoke

What does the smoke test check?

  • The first request checks the application controller at /.
  • The second request checks application health at /actuator/health.
  • The final request checks that /actuator/prometheus returns metrics successfully.

You should see observer-service: ok, a health response containing UP, and Prometheus endpoint returned successfully.

That is the foreground baseline proven. One Java process now serves the application response, detailed health data, and Prometheus-formatted measurements.

Smoke test cannot reach the service?

Confirm that make run is still active in the first SSH terminal. Run the smoke test inside the VM because the service binds to 127.0.0.1.

A failure on only one path usually points to its controller or Actuator configuration. Help me diagnose the endpoint that fails during make smoke.

  • Display the root service response for the first screenshot by running this command:
curl --fail --silent --show-error http://localhost:8080/

What should this request return?

The request reaches home() through the root GET mapping. You should see the exact response observer-service: ok.

  • Capture the successful root response in a screenshot.
  • Save the screenshot as 01-service-response.png.
  • Display the health response and Prometheus metrics by running these commands:
curl --fail --silent --show-error http://localhost:8080/actuator/health
curl --fail --silent --show-error http://localhost:8080/actuator/prometheus

What should these requests reveal?

The health request should contain UP. The metrics request should print Prometheus-formatted measurement names and values.

  • Capture the terminal showing the health response and the beginning of the metrics output.
  • Save the screenshot as 02-actuator-metrics.png.

Metrics output missing?

Confirm that micrometer-registry-prometheus appears in pom.xml and prometheus appears in the endpoint exposure property. Restart make run after changing either file.

The service must remain active while both curl requests run. Help me restore the missing Actuator metrics output.

The working foreground service is ready for a Git checkpoint. The generated target directory remains excluded by the existing ignore rules.

  • Stop the foreground application in the first SSH terminal with Ctrl+C.
  • Return to the second SSH session.
  • Stage the project files and start the commit by running these commands:
git add .
git commit

What does this checkpoint preserve?

Git stages the Maven model, application source, runtime properties, and Makefile. The commit records the complete foreground-service baseline before managed service installation begins.

  • Enter Build observable service as the commit message when Git opens the configured editor.
  • Save the commit message in that editor.
  • Close the editor to complete the commit.

You should see confirmation that Git created a new commit with the project files.

Git did not create the commit?

If Git reports that your identity is missing, configure the author details requested in the terminal and retry the commit. If the editor remains open, save the message before closing it.

Confirm that git add . completed before the commit. Help me finish the observer service Git checkpoint.

Your observer service now builds, runs, and exposes three verified local endpoints. Next, you will move it from a foreground terminal into a managed systemd service with a complete monitoring stack.

Install the Managed Monitoring Stack

Your observer service already responds with health data and application metrics. A foreground process still depends on an open terminal, so it cannot provide the managed lifecycle an administrator needs.

This step places the workload under systemd. You will also install Prometheus, Prometheus Node Exporter, and Grafana OSS as a local monitoring baseline.

In this step, get ready to:
  • Install the observer application as an enabled systemd service.
  • Provision the Prometheus collection rules and Grafana dashboard.
  • Verify four active services through local SSH tunnels.
Install the observer as a native service

A native service unit gives systemd ownership of the Java process. The unit also fixes the runtime user and restart policy in one version-controlled file.

  • Create a folder named systemd inside the repository using your editor's file explorer.
  • Create observer.service inside the systemd folder.
  • Add the observer service definition to systemd/observer.service by pasting this code:
[Unit]
Description=Observer Spring Boot service
After=network.target

[Service]
Type=simple
User=observer
WorkingDirectory=/opt/observer
ExecStart=/usr/bin/java -jar /opt/observer/observer.jar
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

What does this unit control?

  • The observer account isolates the application from interactive users.
  • The executable JAR runs from /opt/observer/observer.jar.
  • The restart policy lets systemd retry unexpected process failures after five seconds.
  • Save systemd/observer.service.
  • Create a folder named scripts inside the repository using your editor's file explorer.
  • Create install-app.sh inside the scripts folder.
  • Add the application installer to scripts/install-app.sh by pasting this code:
#!/usr/bin/env bash
set -euo pipefail

REPO_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)

if ! id -u observer >/dev/null 2>&1; then
    useradd -r -M -d /opt/observer -s /sbin/nologin observer
fi

install -d -m 0755 -o observer /opt/observer
install -m 0644 -o observer "$REPO_ROOT/target/observer-0.0.1-SNAPSHOT.jar" /opt/observer/observer.jar
install -m 0644 "$REPO_ROOT/systemd/observer.service" /etc/systemd/system/observer.service
restorecon -v /opt/observer/observer.jar /etc/systemd/system/observer.service
systemctl daemon-reload
systemctl enable --now observer.service
systemctl status observer.service

What does the installer do?

  • The script creates a locked service account when the observer user does not exist.
  • It installs the executable JAR under /opt/observer with the observer user as owner.
  • It restores SELinux labels before systemd enables the service.
  • Save scripts/install-app.sh.
  • Install the application service by running this target from the repository:
make install-app

What does this target do?

The target rebuilds the JAR before running the installer with elevated privileges. The final status output should identify observer.service as active.

  • Enter your sudo password when the terminal requests it.
  • Confirm the managed application still responds by running:
make smoke

What should you see?

You should see observer-service: ok followed by a health response containing UP. The last line confirms that the Prometheus endpoint responded successfully.

Observer service did not start?

Confirm that Maven produced target/observer-0.0.1-SNAPSHOT.jar. Check that the unit uses the exact /opt/observer/observer.jar path.

Use the status output to identify a missing file or permission failure. Help me diagnose why observer.service failed after make install-app.

Define the monitoring baseline

The monitoring services need native units plus configuration that agrees on local ports. Every listener stays on the VM loopback interface, which keeps the dashboards private until you create an SSH tunnel.

  • Create prometheus.service inside the systemd folder.
  • Add the Prometheus service definition to systemd/prometheus.service by pasting this code:
[Unit]
Description=Prometheus monitoring server
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --web.listen-address=127.0.0.1:9090
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

How is Prometheus isolated?

Prometheus runs through its locked account. Its web interface listens only on 127.0.0.1:9090.

  • Save systemd/prometheus.service.
  • Create node_exporter.service inside the systemd folder.
  • Add the host exporter definition to systemd/node_exporter.service by pasting this code:
[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
Type=simple
User=node_exporter
ExecStart=/usr/local/bin/node_exporter --web.listen-address=127.0.0.1:9100
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

What does Node Exporter provide?

Node Exporter exposes hardware and kernel metrics from the RHEL host. The loopback binding limits its metrics endpoint to local consumers such as Prometheus.

  • Save systemd/node_exporter.service.
  • Create grafana-override.conf inside the systemd folder.
  • Add the loopback override to systemd/grafana-override.conf by pasting this code:
[Service]
Environment=GF_SERVER_HTTP_ADDR=127.0.0.1

Why override Grafana?

The environment override binds Grafana to the loopback interface. Access from your Mac passes through an encrypted SSH tunnel.

  • Save systemd/grafana-override.conf.
  • Create a folder named monitoring inside the repository using your editor's file explorer.
  • Create prometheus.yml inside the monitoring folder.
  • Add the scrape configuration to monitoring/prometheus.yml by pasting this code:
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/alerts.yml

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: node
    static_configs:
      - targets: ["localhost:9100"]

  - job_name: observer
    metrics_path: /actuator/prometheus
    static_configs:
      - targets: ["localhost:8080"]

What does the scrape configuration do?

  • Prometheus collects its own metrics from port 9090.
  • The node job collects host metrics from port 9100.
  • The observer job collects application metrics from /actuator/prometheus every 15 seconds.
  • Save monitoring/prometheus.yml.
  • Create alerts.yml inside the monitoring folder.
  • Add the availability rule to monitoring/alerts.yml by pasting this code:
groups:
  - name: observer-availability
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "Instance {{ $labels.instance }} down"
          description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes."

How does the alert progress?

The InstanceDown rule matches any target whose up metric becomes 0. The alert remains pending until the failed state lasts five minutes.

After that duration, Prometheus changes the alert state to firing. This delay prevents a brief failed scrape from immediately becoming an incident.

  • Save monitoring/alerts.yml.
  • Create monitoring/grafana/datasources using your editor's file explorer.
  • Create prometheus.yml inside monitoring/grafana/datasources.
  • Add the provisioned data source to monitoring/grafana/datasources/prometheus.yml by pasting this code:
apiVersion: 1
prune: true

datasources:
  - name: Prometheus
    type: prometheus
    uid: prometheus
    access: proxy
    url: http://localhost:9090
    isDefault: true
    editable: false
    jsonData:
      httpMethod: POST
      prometheusType: Prometheus
      prometheusVersion: 3.15.0
      timeInterval: 15s

Why provision the data source?

File provisioning gives Grafana a stable data source with the UID prometheus. The dashboard can reference that UID without manual configuration.

  • Save monitoring/grafana/datasources/prometheus.yml.
  • Create monitoring/grafana/dashboards using your editor's file explorer.
  • Create provider.yml inside monitoring/grafana/dashboards.
  • Add the dashboard provider to monitoring/grafana/dashboards/provider.yml by pasting this code:
apiVersion: 1

providers:
  - name: observable-linux-service
    orgId: 1
    folder: NextWork
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards

Where does Grafana find the dashboard?

The provider watches /var/lib/grafana/dashboards for dashboard JSON. Grafana places the provisioned dashboard in the NextWork folder.

  • Save monitoring/grafana/dashboards/provider.yml.
  • Create observable-linux-service.json inside monitoring/grafana/dashboards.
  • Use the full-code tab to add the version-controlled dashboard to monitoring/grafana/dashboards/observable-linux-service.json.

✔️ Awesome, I've got everything!

Save the dashboard file after confirming that it contains four panels.

ⓧ I'd like to double check the full code

{
  "id": null,
  "uid": "observable-linux-service",
  "title": "Observable Linux Service",
  "tags": ["rhel", "linux", "sre"],
  "timezone": "browser",
  "editable": false,
  "graphTooltip": 1,
  "panels": [
    {
      "id": 1,
      "type": "stat",
      "title": "Observer Target",
      "gridPos": {"x": 0, "y": 0, "w": 6, "h": 5},
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "targets": [
        {
          "expr": "up{job=\"observer\"}",
          "refId": "A",
          "instant": true,
          "range": false
        }
      ],
      "fieldConfig": {
        "defaults": {
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {"color": "red", "value": null},
              {"color": "green", "value": 1}
            ]
          }
        },
        "overrides": []
      },
      "options": {
        "colorMode": "background",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "auto",
        "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false},
        "textMode": "auto"
      }
    },
    {
      "id": 2,
      "type": "stat",
      "title": "InstanceDown Alert",
      "gridPos": {"x": 6, "y": 0, "w": 6, "h": 5},
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "targets": [
        {
          "expr": "ALERTS{alertname=\"InstanceDown\",alertstate=\"firing\"}",
          "refId": "A",
          "instant": true,
          "range": false
        }
      ],
      "fieldConfig": {"defaults": {}, "overrides": []},
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "auto",
        "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false},
        "textMode": "auto"
      }
    },
    {
      "id": 3,
      "type": "timeseries",
      "title": "Host System CPU Time",
      "gridPos": {"x": 0, "y": 5, "w": 12, "h": 8},
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "targets": [
        {
          "expr": "rate(node_cpu_seconds_total{mode=\"system\"}[1m])",
          "legendFormat": "CPU {{cpu}}",
          "refId": "A",
          "range": true
        }
      ],
      "fieldConfig": {"defaults": {}, "overrides": []},
      "options": {
        "legend": {"displayMode": "list", "placement": "bottom", "showLegend": true},
        "tooltip": {"mode": "single", "sort": "none"}
      }
    },
    {
      "id": 4,
      "type": "timeseries",
      "title": "JVM Memory Used",
      "gridPos": {"x": 12, "y": 5, "w": 12, "h": 8},
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "targets": [
        {
          "expr": "jvm_memory_used_bytes",
          "legendFormat": "{{area}} {{id}}",
          "refId": "A",
          "range": true
        }
      ],
      "fieldConfig": {"defaults": {"unit": "bytes"}, "overrides": []},
      "options": {
        "legend": {"displayMode": "list", "placement": "bottom", "showLegend": true},
        "tooltip": {"mode": "single", "sort": "none"}
      }
    }
  ],
  "time": {"from": "now-30m", "to": "now"},
  "timepicker": {"refresh_intervals": ["5s", "10s", "30s", "1m", "5m"]},
  "templating": {"list": []},
  "annotations": {"list": []},
  "refresh": "5s",
  "schemaVersion": 42,
  "version": 0,
  "links": []
}

What will the dashboard show?

The stat panels show observer availability plus the firing state of InstanceDown. The time-series panels graph host CPU time plus JVM memory use.

  • Create install-monitoring.sh inside the scripts folder.
  • Use the full-code tab to add the architecture-aware installer to scripts/install-monitoring.sh.

✔️ Awesome, I've got everything!

Save the installer after confirming that it handles both supported VM architectures.

ⓧ I'd like to double check the full code

#!/usr/bin/env bash
set -euo pipefail

REPO_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
TMP_DIR=$(mktemp -d)
trap 'rm -rf "$TMP_DIR"' EXIT

case "$(uname -m)" in
    x86_64)
        PROM_ARCH=amd64
        GRAFANA_RPM=https://dl.grafana.com/grafana/release/13.2.4/grafana_13.2.4_38252365904_linux_amd64.rpm
        ;;
    aarch64)
        PROM_ARCH=arm64
        GRAFANA_RPM=https://dl.grafana.com/grafana/release/13.2.4/grafana_13.2.4_38252365904_linux_arm64.rpm
        ;;
    *)
        echo "Supported architectures are x86_64 and aarch64."
        exit 1
        ;;
esac

PROMETHEUS_URL="https://github.com/prometheus/prometheus/releases/download/v3.15.0/prometheus-3.15.0.linux-${PROM_ARCH}.tar.gz"
NODE_EXPORTER_URL="https://github.com/prometheus/node_exporter/releases/download/v1.12.1/node_exporter-1.12.1.linux-${PROM_ARCH}.tar.gz"

curl --fail --location --show-error --silent --output "$TMP_DIR/prometheus.tar.gz" "$PROMETHEUS_URL"
curl --fail --location --show-error --silent --output "$TMP_DIR/node_exporter.tar.gz" "$NODE_EXPORTER_URL"

(
    cd "$TMP_DIR"
    tar xzvf prometheus.tar.gz
    tar xzvf node_exporter.tar.gz
)

if ! id -u prometheus >/dev/null 2>&1; then
    useradd -r -M -d /var/lib/prometheus -s /sbin/nologin prometheus
fi
if ! id -u node_exporter >/dev/null 2>&1; then
    useradd -r -M -s /sbin/nologin node_exporter
fi

install -m 0755 "$TMP_DIR/prometheus-3.15.0.linux-${PROM_ARCH}/prometheus" /usr/local/bin/prometheus
install -m 0755 "$TMP_DIR/prometheus-3.15.0.linux-${PROM_ARCH}/promtool" /usr/local/bin/promtool
install -m 0755 "$TMP_DIR/node_exporter-1.12.1.linux-${PROM_ARCH}/node_exporter" /usr/local/bin/node_exporter

install -d -m 0755 /etc/prometheus
install -d -m 0755 -o prometheus /var/lib/prometheus
install -m 0644 "$REPO_ROOT/monitoring/prometheus.yml" /etc/prometheus/prometheus.yml
install -m 0644 "$REPO_ROOT/monitoring/alerts.yml" /etc/prometheus/alerts.yml
install -m 0644 "$REPO_ROOT/systemd/prometheus.service" /etc/systemd/system/prometheus.service
install -m 0644 "$REPO_ROOT/systemd/node_exporter.service" /etc/systemd/system/node_exporter.service

/usr/local/bin/promtool check config /etc/prometheus/prometheus.yml
/usr/local/bin/promtool check rules /etc/prometheus/alerts.yml

dnf install -y "$GRAFANA_RPM"
install -d -m 0755 /etc/grafana/provisioning/datasources
install -d -m 0755 /etc/grafana/provisioning/dashboards
install -d -m 0755 -o grafana -g grafana /var/lib/grafana/dashboards
install -m 0644 "$REPO_ROOT/monitoring/grafana/datasources/prometheus.yml" /etc/grafana/provisioning/datasources/prometheus.yml
install -m 0644 "$REPO_ROOT/monitoring/grafana/dashboards/provider.yml" /etc/grafana/provisioning/dashboards/provider.yml
install -m 0644 -o grafana -g grafana "$REPO_ROOT/monitoring/grafana/dashboards/observable-linux-service.json" /var/lib/grafana/dashboards/observable-linux-service.json
install -d -m 0755 /etc/systemd/system/grafana-server.service.d
install -m 0644 "$REPO_ROOT/systemd/grafana-override.conf" /etc/systemd/system/grafana-server.service.d/override.conf

restorecon -v /usr/local/bin/prometheus /usr/local/bin/promtool /usr/local/bin/node_exporter
restorecon -v /etc/systemd/system/prometheus.service /etc/systemd/system/node_exporter.service
restorecon -R -v /etc/prometheus /etc/grafana/provisioning /var/lib/grafana/dashboards

systemctl daemon-reload
systemctl enable --now node_exporter.service prometheus.service grafana-server.service
systemctl status node_exporter.service
systemctl status prometheus.service
systemctl status grafana-server.service

How does the installer stay portable?

The architecture check maps x86_64 to AMD64 assets. It maps aarch64 to ARM64 assets.

The installer validates both Prometheus files before it starts the services. It also restores SELinux labels across the installed paths.

  • Create uninstall.sh inside the scripts folder.
  • Add the cleanup workflow to scripts/uninstall.sh by pasting this code:
#!/usr/bin/env bash
set -euo pipefail

systemctl disable --now observer.service prometheus.service node_exporter.service grafana-server.service || true
rm -f /etc/systemd/system/observer.service
rm -f /etc/systemd/system/prometheus.service
rm -f /etc/systemd/system/node_exporter.service
rm -rf /etc/systemd/system/observer.service.d
rm -rf /etc/systemd/system/grafana-server.service.d
rm -f /usr/local/bin/prometheus /usr/local/bin/promtool /usr/local/bin/node_exporter
rm -rf /opt/observer /etc/prometheus /var/lib/prometheus
rm -f /etc/grafana/provisioning/datasources/prometheus.yml
rm -f /etc/grafana/provisioning/dashboards/provider.yml
rm -f /var/lib/grafana/dashboards/observable-linux-service.json
dnf remove -y grafana || true
userdel observer 2>/dev/null || true
userdel prometheus 2>/dev/null || true
userdel node_exporter 2>/dev/null || true
systemctl daemon-reload

Why prepare cleanup now?

The uninstall script records every installed path and service account while those details are fresh. You will keep it for the final cleanup decision.

  • Save scripts/uninstall.sh.
  • Create a folder named incident inside the repository using your editor's file explorer.
  • Create README.md inside the incident folder.
  • Add the incident template to incident/README.md by pasting this code:
# Incident: Observer service unavailable

Start time:
Detection source:
Alert state:
User impact:

## Evidence

- systemd state:
- Journal evidence:
- Listening socket evidence:
- cgroups evidence:
- Prometheus target state:
- Grafana screenshot:

## Root cause

The service was deliberately stopped as part of the controlled incident exercise.

## Mitigation

The service was started through systemd and validated through the endpoint, Prometheus target, alert state, and Grafana dashboard.

## Prevention and follow-up

- Keep availability alerts and dashboard provisioning under version control.
- Preserve command output and screenshots with the incident record.
- Add application-level resilience only after the infrastructure baseline is understood.

What will the incident record preserve?

The template separates detection evidence from root cause and mitigation. Later steps fill it with systemd state, journal output, socket evidence, cgroup placement, and monitoring screenshots.

Install and verify the stack

The monitoring installer pins each release and replaces the project-managed configuration. Choose the tab that matches the software state on your VM before you run it.

✔️ I already have the pinned releases

The installer reapplies the project configuration for Prometheus 3.15.0, Node Exporter 1.12.1, and Grafana OSS 13.2.4.

Continue with the common installation target below so the binaries and provisioned files match the repository.

ⓧ I have older versions

The installer places the pinned Prometheus and Node Exporter binaries under /usr/local/bin. It installs the pinned Grafana RPM through DNF.

Continue with the common installation target below to apply the project versions.

ⓧ The stack is not installed

The installer downloads architecture-matched releases. It creates the locked service accounts plus the required configuration directories.

Continue with the common installation target below to build the stack from the repository.

This installation changes system services and installs a package with sudo privileges. The project remains local to your existing RHEL VM, so it creates no cloud resources or metered API usage.

  • Install the monitoring stack by running:
make install-monitoring

What happens during installation?

The script downloads the Prometheus and Node Exporter archives for your architecture. It installs the Grafana RPM plus every provisioned monitoring file.

The download and package installation can take a few minutes. Ongoing extraction or DNF output means the process is still working.

Monitoring installation failed?

Confirm that the VM still has outbound HTTPS access plus enabled RHEL repositories. Check whether the terminal reports x86_64 or aarch64 as the architecture.

A configuration validation failure usually points to spacing or quoting drift in a YAML file. Help me diagnose why make install-monitoring failed on RHEL 10.2.

  • Validate the installed Prometheus configuration and alert rules by running:
make validate-monitoring

What does validation prove?

Promtool checks /etc/prometheus/prometheus.yml plus /etc/prometheus/alerts.yml. Successful checks prove that Prometheus can parse both installed files.

Before you check service state, what do you expect systemd to report for the application plus the three monitoring services?

  • Check all four services by running:
make status

What should you see?

You should see active four times. The lines correspond to observer.service, prometheus.service, node_exporter.service, and grafana-server.service.

One service is not active?

Match the failing line to the target order in the status recipe. Review that service's status before rerunning the installer.

Check that every unit and configuration file matches the supplied path exactly. Help me identify which managed service failed from my make status output.

The services now listen only inside the VM. A local SSH forward lets your Mac browser reach Grafana and Prometheus without opening firewall ports.

  • Record the username from your existing SSH connection here: your RHEL username.
  • Record the VM address from your existing SSH connection here: your VM address.
  • Press Cmd+Space on macOS to open Spotlight.
  • Type Terminal into Spotlight.
  • Press Enter to open a fresh Mac terminal.
  • Create local forwards for Grafana and Prometheus by running:
ssh -L 3000:localhost:3000 -L 9090:localhost:9090 [[SSH_USER="your RHEL username"]]@[[VM_HOST="your VM address"]]

What does the tunnel do?

Local port 3000 forwards to Grafana on the VM. Local port 9090 forwards to Prometheus on the VM.

Keep this terminal session open while you use both browser interfaces. Closing the session removes the forwards.

Tunnel could not start?

A local port may already be in use if another tunnel is running. Close the older SSH session before retrying the command.

Confirm that the username and VM address match the connection you used earlier. Help me troubleshoot my SSH local forwarding command.

Your first Grafana sign-in uses the default local administrator credentials. Change the default password when prompted so the fresh installation no longer uses a shared credential.

  • Open http://localhost:3000 in your Mac browser.
  • Enter admin in the Username field.
  • Enter admin in the Password field.
  • Click Sign in.
  • Click OK when Grafana prompts you to change the password.
  • Set a private administrator password that you do not reuse elsewhere.
  • Open the NextWork folder from the dashboard browser.
  • Select Observable Linux Service.

You should see a green Observer Target panel with a value of 1. You should also see host CPU activity plus JVM memory data.

  • Capture the healthy dashboard as 03-healthy-dashboard.png.
  • Return to the SSH terminal connected to the RHEL VM.
  • Run the four-service status check again:
make status

What does the final check prove?

Four active results prove that the observer and monitoring services remain managed by systemd. The live dashboard proves that Prometheus is scraping application plus host metrics.

  • Capture the four active results as 04-systemd-services.png.
  • Keep the SSH tunnel open while you continue to the controlled incident.

That is the managed baseline in place: four active services now expose live host and JVM evidence through private local interfaces. Next, you will stop only the observer workload and watch the monitoring stack detect the outage.

Trigger a Controlled Outage

Your observer service is now managed by systemd. Prometheus is collecting the healthy baseline.

Healthy dashboards do not prove incident readiness. In this step, you will stop only observer.service so Grafana can expose the failed workload.

In this step, get ready to:
  • Confirm the healthy monitoring baseline.
  • Stop only observer.service while monitoring remains active.
  • Preserve the complete incident evidence.
Establish the failure boundary

A healthy baseline gives you a comparison point for every failed signal that follows. Confirm that baseline before removing the application process.

  • Switch back to the Observable Linux Service dashboard at http://localhost:3000.
  • Confirm the Observer Target panel shows 1.
  • Confirm the Host System CPU Time panel is still receiving data.

Before you create the outage, what do you expect the endpoint to do when systemd removes its only process?

  • Stop only observer.service from your SSH terminal by running:
make incident-down

What does this target stop?

The incident-down target calls sudo systemctl stop observer.service. It leaves the three monitoring services untouched.

Before you test, predict whether the root endpoint can still return its healthy response.

  • Test the service endpoints by running:
make smoke

Why does the smoke check fail?

The request to http://localhost:8080/ fails because no observer process is listening on that loopback port. This is the intended outage.

Make stops the smoke recipe after the first failed request. Prometheus independently confirms that the metrics endpoint is also unreachable.

  • Confirm the three monitoring services stayed active by running these commands:
systemctl is-active prometheus.service
systemctl is-active node_exporter.service
systemctl is-active grafana-server.service

What should you see?

You should see active three times. This proves the monitoring stack survived the application outage.

Outage not behaving as expected?

If the endpoint still responds, confirm that the stop target completed without a sudo or directory error.

If a monitoring service is inactive, return to the previous installation checks before continuing. This incident depends on monitoring remaining available.

Help me diagnose why the observer outage or monitoring service check produced an unexpected result.

Watch the alert progress

Prometheus keeps scraping while the application remains down. Its up metric turns the missing endpoint into a measurable incident.

  • Switch back to the Prometheus page at http://localhost:9090.
  • Enter up{job="observer"} in the query field.
  • Run the query.

You should see a value of 0. This proves the observer scrape is failing.

  • Capture the failed target as 05-target-down.png.
  • Open the Prometheus alert view from the top navigation.
  • Locate InstanceDown.

The alert first shows a pending state. Its for: 5m rule requires the failure to remain active before escalation.

This wait is deliberate, so a quiet few minutes means the alert timer is working. Keep the observer service stopped throughout the wait.

Before you refresh, do you expect the alert to remain pending or advance to its firing state?

  • Leave the outage active for slightly longer than five minutes.
  • Refresh the Prometheus alert view.

You should now see InstanceDown in the FIRING state. The failed scrape has lasted long enough to trigger the rule.

  • Capture the firing alert as 06-alert-firing.png.

Alert still pending?

Count the five-minute duration from the first failed scrape instead of the moment you stopped the service. The first failed scrape can arrive up to one scrape interval later.

Confirm that up{job="observer"} still returns 0 before refreshing the alert view.

Help me work out why InstanceDown has not reached its firing state.

Preserve the failed-state evidence

Incident evidence preserves the system state before recovery changes it. This gives the next step a trustworthy starting point for diagnosis.

  • Capture the failed service state by running:
make evidence

What does this target preserve?

  • The status command writes the inactive service state to evidence/observer-status.txt.
  • The journald command writes the current-boot service log to evidence/observer-journal.txt.
  • The failed health request writes {"status":"unreachable"} to evidence/observer-health.json.
  • Open evidence/observer-status.txt with the file editor you used earlier.

You should see that observer.service is inactive.

  • Open evidence/observer-journal.txt with the same editor.

You should see the observer service lifecycle entries from the current boot.

  • Open evidence/observer-health.json with the same editor.

You should see {"status":"unreachable"}. This records the unavailable health endpoint without hiding the failed request.

  • Keep observer.service stopped for the diagnosis step.

Your controlled outage is now documented. Next, you will diagnose the failed service from Linux evidence before restoring it.

Diagnose and Restore the Service

Your Prometheus alert has exposed the outage. Your Grafana dashboard has preserved the visible symptom.

An alert identifies a symptom. It cannot restore the service.

Linux evidence reveals the cause before mitigation. You will inspect systemd state, journald records, the listening socket, and cgroups v2 placement.

In this step, get ready to:
  • Correlate the inactive service state with logs, socket evidence, and cgroup placement.
  • Restore the observer workload through its managed lifecycle.
  • Complete the incident report and checkpoint the recovery evidence.
Diagnose the stopped service

A stopped unit can resemble several failure modes from a dashboard. Correlating Linux signals identifies the controlled stop before you change the system.

  • Collect the four diagnostic signals before changing the service state by running these commands:
systemctl status observer.service
journalctl -u observer.service -b
ss -tulpn
systemd-cgls

What does this evidence show?

  • The service manager state confirms whether the unit has an active process.
  • The current-boot journal preserves the stop event and earlier application records.
  • The socket list proves whether a process is accepting connections on the observer port.
  • The cgroup tree shows whether systemd currently has a live observer process group.

You should see observer.service reported as inactive. The journal should contain the controlled stop records.

You should see no listener on 127.0.0.1:8080. The inactive service should have no live branch under /sys/fs/cgroup/system.slice/observer.service.

Missing diagnostic evidence?

If journal entries are unavailable, confirm your SSH account can read system logs with sudo access. If port 8080 still has a listener, verify that the process belongs to the observer workload before continuing.

Ask for help correlating the stopped unit with the Linux evidence.

Restore the managed workload

The evidence points to the intended administrative stop. Restoring the unit through its Make target keeps the application inside the same systemd lifecycle.

Before you run the recovery, do you expect the endpoint and all four services to return to a healthy state?

  • Start the observer service and test the recovered stack by running these commands:
make incident-up
make smoke
make status

What does the recovery check prove?

  • The incident-up target asks systemd to start only the observer unit.
  • The smoke target checks the root endpoint, health endpoint, and Prometheus metrics endpoint.
  • The status target checks the observer, Prometheus, Node Exporter, and Grafana services.

You should see observer-service: ok from the root endpoint. The health response should report UP.

The metrics check should succeed. All four services should report active.

That closes the service interruption. The application is serving requests through its managed Linux process again.

Service still unavailable?

Review the service status and journal output from the diagnostic check. A failed startup should leave evidence there.

Ask for help diagnosing the failed recovery.

  • Return to Prometheus through the existing SSH tunnel at http://localhost:9090.
  • Enter up{job="observer"} in the query field.
  • Submit the query.

You should see a value of 1. This proves Prometheus can scrape the observer metrics again.

  • Return to the provisioned Observable Linux Service dashboard in Grafana.
  • Wait for the next dashboard refresh.

The Observer Target panel should show the recovered target. The InstanceDown Alert panel should no longer show a firing alert.

  • Return to systemd/observer.service in the repository.
  • Confirm the service account directive remains User=observer.
Document and checkpoint the incident

An incident report connects monitoring symptoms to Linux evidence. It also records the mitigation that restored service.

  • Edit incident/README.md in your terminal editor.
  • Replace the template with the completed incident record below:
# Incident: Observer service unavailable

Start time: Recorded during the controlled outage exercise.
Detection source: Prometheus target status and InstanceDown alert.
Alert state: FIRING after the observer target remained down for more than five minutes; resolved after recovery.
User impact: The loopback-only observer endpoint and Actuator endpoints were unavailable while observer.service was stopped.

## Evidence

- systemd state: observer.service was inactive after make incident-down and active after make incident-up.
- Journal evidence: journalctl -u observer.service -b captured stop and subsequent startup records.
- Listening socket evidence: no listener was present during the outage; Java listened on 127.0.0.1:8080 after recovery.
- cgroups evidence: observer.service was inspected under /sys/fs/cgroup/system.slice/observer.service.
- Prometheus target state: up{job="observer"} changed from 1 to 0 during the outage and returned to 1 after recovery.
- Grafana screenshot: 05-target-down.png, 06-alert-firing.png, and 07-service-recovered.png.

## Root cause

The service was deliberately stopped as part of the controlled incident exercise.

## Mitigation

The service was started through systemd and validated through the endpoint, Prometheus target, alert state, and Grafana dashboard.

## Prevention and follow-up

- Keep availability alerts and dashboard provisioning under version control.
- Preserve command output and screenshots with the incident record.
- Add application-level resilience only after the infrastructure baseline is understood.

What does this report capture?

  • The opening fields record detection, alert progression, and user impact.
  • The evidence section connects the dashboard symptom to systemd, journal, socket, and cgroup observations.
  • The final sections record the controlled root cause, the recovery action, and the follow-up practices.
  • Save incident/README.md.
  • Compare the saved report with the complete reference below.

✔️ Awesome, I've got everything!

Your incident report now records the complete outage and recovery workflow.

ⓧ I'd like to double check the full code

# Incident: Observer service unavailable

Start time: Recorded during the controlled outage exercise.
Detection source: Prometheus target status and InstanceDown alert.
Alert state: FIRING after the observer target remained down for more than five minutes; resolved after recovery.
User impact: The loopback-only observer endpoint and Actuator endpoints were unavailable while observer.service was stopped.

## Evidence

- systemd state: observer.service was inactive after make incident-down and active after make incident-up.
- Journal evidence: journalctl -u observer.service -b captured stop and subsequent startup records.
- Listening socket evidence: no listener was present during the outage; Java listened on 127.0.0.1:8080 after recovery.
- cgroups evidence: observer.service was inspected under /sys/fs/cgroup/system.slice/observer.service.
- Prometheus target state: up{job="observer"} changed from 1 to 0 during the outage and returned to 1 after recovery.
- Grafana screenshot: 05-target-down.png, 06-alert-firing.png, and 07-service-recovered.png.

## Root cause

The service was deliberately stopped as part of the controlled incident exercise.

## Mitigation

The service was started through systemd and validated through the endpoint, Prometheus target, alert state, and Grafana dashboard.

## Prevention and follow-up

- Keep availability alerts and dashboard provisioning under version control.
- Preserve command output and screenshots with the incident record.
- Add application-level resilience only after the infrastructure baseline is understood.

Report formatting looks different?

Check that each section heading starts on its own line. Confirm the evidence entries remain under the Evidence heading.

Ask for help comparing the incident report.

The recovered dashboard provides the visual proof for your incident record.

  • Return to the Observable Linux Service dashboard.
  • Confirm the observer target remains healthy.
  • Confirm the availability alert remains resolved.
  • Capture the recovered dashboard as 07-service-recovered.png.

Fresh command evidence preserves the post-recovery state beside the earlier outage evidence.

  • Refresh the command-line evidence after recovery by running this command:
make evidence

What does the updated evidence contain?

The evidence target records the current service status, current-boot journal, and health response. The new files now represent the recovered state.

You should now have updated files under evidence/. The health evidence should report UP.

Evidence command incomplete?

A journal permission problem can prevent the log file from updating. Confirm your SSH account still has the required access before rerunning the target.

Ask for help repairing the evidence capture.

A Git checkpoint preserves the report and recovery screenshot as one incident milestone.

  • Stage the incident artifacts and create the checkpoint by running these commands:
git add .
git commit -m 'message'

What does this checkpoint preserve?

The staging command collects the report and screenshot changes. The commit records them in the local repository history.

You should see a Git commit summary in the terminal. It should list the files recorded by the checkpoint.

Commit did not complete?

Git may request an author identity if this VM has never created a commit. Follow the guidance in the terminal before retrying the checkpoint.

Ask for help completing the incident checkpoint.

Before the final check, do you expect the endpoint, managed services, listening socket, and observer cgroup to agree on the recovered state?

  • Run the final Linux recovery check with these commands:
make smoke
make status
ss -tulpn
systemd-cgls

What does the final check prove?

  • The HTTP checks prove the root, health, and metrics endpoints respond.
  • The service checks prove the application and monitoring stack remain active.
  • The socket list proves Java listens on the loopback interface at port 8080.
  • The cgroup tree proves the observer process runs under its systemd service boundary.

You should see observer-service: ok and a health status of UP. Each service should report active.

You should also see a listener on 127.0.0.1:8080. The observer process should appear under /sys/fs/cgroup/system.slice/observer.service.

  • Return to Prometheus through the existing SSH tunnel.
  • Submit up{job="observer"} again.
  • Return to the recovered Grafana dashboard.
  • Confirm the InstanceDown Alert remains resolved.

Prometheus should return 1 for the observer target. Your Linux evidence and monitoring now tell the same recovery story.

Recovery signals disagree?

If the endpoint works while Prometheus still shows 0, wait for the next scrape before checking again. If the socket or cgroup is missing, review the current service status and journal.

Ask for help reconciling the recovery signals.

You have completed the detect, diagnose, restore, and document loop. The observer service is healthy again with a committed incident record behind it.

Secret mission

Prove a cgroups v2 Memory Guardrail

Apply a controlled memory ceiling to the observer service. Use Linux evidence to prove how the guardrail affects the JVM. Remove the limit to restore the monitored service safely.

Clean Up Your Resources

Clean Up Your Resources

Your monitoring stack runs entirely on your existing RHEL VM, so there are no cloud charges or metered API costs. Decide whether to keep the services active, pause them for later, or remove the installed stack.

Resources you used:

  • Managed services: The systemd units observer.service, prometheus.service, node_exporter.service, and grafana-server.service.
  • Observer installation: The executable application under /opt/observer.
  • Prometheus installation: The Prometheus binaries plus configuration under /etc/prometheus.
  • Prometheus data: The time-series data under /var/lib/prometheus.
  • Host telemetry: The Node Exporter binary installed under /usr/local/bin.
  • Dashboard installation: The Grafana package plus its provisioned data source and dashboard files.
  • Locked service accounts: observer, prometheus, and node_exporter.

Your current repository contains the source files, screenshots, incident report, and command evidence. Every cleanup option below leaves that repository untouched.

Keep everything running

No action needed. Choose this if you want the healthy monitoring baseline ready for another incident exercise.

  • The observer application remains active on 127.0.0.1:8080.
  • Prometheus remains active on 127.0.0.1:9090.
  • Node Exporter remains active on 127.0.0.1:9100.
  • Grafana remains active on 127.0.0.1:3000.
  • Your existing SSH forwards continue providing private browser access while the connection stays open.
  • Your Git history and incident evidence remain available in the current repository.

Pause - I'll come back to this later

Shut down the running services to free VM memory. The installed files and monitoring data stay available for your return.

  • Pause the running stack by running this command:
make pause

What Does This Target Pause?

The pause target stops the four services.

  • Unit enablement remains configured.
  • Installed application files remain on disk.
  • Monitoring data remains on disk.
  • Repository evidence remains on disk.
  • Resume the monitoring stack when you return by running this command:
make resume

What Does This Target Restore?

  • The target starts node_exporter.service first.
  • It starts prometheus.service next.
  • It starts grafana-server.service after Prometheus.
  • It starts observer.service last.
  • Confirm the restored service states by running this command:
make status

What Should I See?

You should see active four times. Each line confirms that one service is running again.

Delete - I don't want to use this again

Remove the installed services and their local runtime data if you have finished with this lab. Your current repository remains available with its screenshots and incident evidence.

This command requests sudo access because it removes system services and package-managed data. Reading the script first keeps the deletion predictable.

  • Read scripts/uninstall.sh from top to bottom in the current repository.
  • Confirm the screenshots and incident evidence you want to keep are present in the current repository.
  • Remove the installed runtime stack by running this command:
make uninstall

What Does This Cleanup Remove?

  • The script stops the four services.
  • It disables their boot enablement.
  • It removes the custom unit files and drop-in directories.
  • It removes the observer application and monitoring data.
  • It removes the Prometheus and Node Exporter binaries.
  • It removes the Grafana package and provisioned dashboard files.
  • It removes the three locked service accounts.
  • It reloads systemd after the cleanup.

The source repository stays intact because the cleanup script does not remove it.

Nice Work!

Nice Work!

Mission complete! You built an observable Spring Boot workload on RHEL. You operated it as a recoverable Linux service.

You've learned how to:

  • Built a loopback-only service. Managed its lifecycle through a native systemd unit. Used GNU Make targets for repeatable operations.
  • Provisioned Prometheus as the metrics store. Collected host telemetry through Node Exporter. Displayed service availability in Grafana. Displayed JVM memory in the same dashboard. Tracked the InstanceDown alert from healthy state through recovery.
  • Completed a controlled incident workflow. Diagnosed the outage with journald logs. Confirmed the missing listening socket. Inspected the service's cgroups v2 placement. Restored the managed service. Preserved the incident evidence in Git.
  • Secret Mission: Enforced a restrictive MemoryMax=1500K guardrail against observer.service. Captured the resulting JVM failure. Removed the drop-in safely. Proved endpoint recovery. Confirmed the monitoring stack returned to a healthy state.

Ready to quiz yourself?