Build a Self-Healing OpenShift Service

Build an OpenShift service that self-heals and recovers from Git with Ansible.

Introduction

30 Second Summary

An online service can look dependable until one failure takes it offline. The real test is whether people can keep using it while something goes wrong.

In this project, you will build a public health service on OpenShift. You will test self-healing after a Pod loss before recovering the entire stateless workload from Git with Ansible.

What You'll Build

You will watch a public health endpoint keep returning HTTP 200 while OpenShift replaces a deleted managed Pod.

By the end of this project, you'll have:

  • A timestamped availability watcher that polls a public health endpoint from Windows PowerShell. You can use it to capture HTTP 200 followed by DOWN during the unmanaged Pod failure.
  • A three-replica self-healing service that stays available during one managed Pod deletion. Its Deployment restores three ready Pods after one is deleted.
  • A Git-backed recovery playbook that rebuilds the deleted Deployment, Service, Route, and PodDisruptionBudget from your saved manifests.
  • Secret Mission: Deploy a release with a deliberately broken readiness probe. Prove traffic stays on healthy replicas before performing a manual rollback.

Are there any prerequisites?

Your phone-verified Red Hat account needs access to Developer Sandbox for Red Hat OpenShift with permission to create namespaced resources.

Your Windows computer needs internet access plus Git and kubectl in the same WSL or container environment where Ansible runs.

Before We Start

First, you will define the reliability target that guides every failure test in this lab. This gives you a clear promise to test before anything is created in OpenShift.

Set Up the OpenShift Recovery Workspace

Your reliability target gives every failure drill a clear pass condition. Recovery automation still falls apart when its tools use different files or cluster contexts.

A local Git repository becomes the recovery source for the application files you create later.

The OpenShift CLI connects your shell to the assigned cluster. kubectl uses the same cluster credentials.

Ansible runs the recovery procedure from your existing WSL or container shell.

In this step, get ready to:
  • Verify the command-line tools used by the recovery workflow.
  • Authenticate your cluster shells with Developer Sandbox.
  • Create the local Git repository structure.
Verify your recovery tools

Ansible cannot run natively on Windows as a control node. Keep the recovery workflow inside the WSL or container shell where Ansible already runs.

  • Check Git, kubectl, and Ansible in that shell by running:
git --version
kubectl version --client
ansible-playbook --version

What do these checks prove?

  • The git result confirms that this shell can track the recovery files.
  • The kubectl result confirms that the Kubernetes client is available.
  • The ansible-playbook result identifies the installed ansible-core version.

Git should print its version. Kubectl should print client information.

  • Choose the tab that matches the ansible-core result.

✔️ I see version 2.21.5

Your recovery shell has ansible-core 2.21.5. That playbook runner is ready for the recovery procedure.

ⓧ I see an older version

This project uses ansible-core 2.21.5. This release requires Python 3.12 or newer.

  • Read the Python version shown in the Ansible output.
  • Confirm that the reported Python version is 3.12 or newer.
  • Open the official Ansible installation guide.
  • Use the documented method for your WSL or container environment to install ansible-core 2.21.5.
  • Verify the updated playbook runner by running:
ansible-playbook --version

What should the result confirm?

The output should identify ansible-core 2.21.5. It should also identify the Python environment used by the command.

ⓧ Command not found

Install Ansible inside the WSL or container environment used for this lab. Ansible cannot use native Windows as its control node.

  • Confirm that the selected environment provides Python 3.12 or newer.
  • Open the official Ansible installation guide.
  • Use the documented method for your environment to install ansible-core 2.21.5.
  • Verify the playbook runner by running:
ansible-playbook --version

What should the result confirm?

The output should identify ansible-core 2.21.5. The shell can now run the recovery playbook created later.

A tool check failed?

A command installed only in Windows may be unavailable inside WSL or your container. Install the missing command in the same environment that runs ansible-playbook.

Help me align the recovery tools inside my shell.

Connect your cluster shells

Developer Sandbox for Red Hat OpenShift provides the existing cluster for this lab. Access is free with a Red Hat account.

The Sandbox requires phone verification. It does not require a credit card.

  • Open the Developer Sandbox page in your browser.
  • Sign in with your Red Hat account.
  • Complete the requested phone verification.
  • Start your assigned Sandbox.
  • Continue when the OpenShift web console shows your assigned project.

The oc client should correspond to the current OpenShift server version. A client from the current Sandbox supports the cluster capabilities used in this lab.

  • Switch back to the WSL or container shell used for Ansible.
  • Check the OpenShift client in that shell by running:
oc version

What does this check show?

The command displays information about the installed OpenShift client. A missing-command result means this shell still needs the client supplied by the Sandbox.

  • Switch to Windows PowerShell.
  • Check the OpenShift client in PowerShell by running:
oc version

Why check PowerShell too?

The route watcher runs from Windows PowerShell later. PowerShell needs its own working oc command.

  • Choose the tab that matches the least-ready shell.

✔️ Both shells have the matching client

Both shells can use the client supplied for the current Sandbox. Your command-line environments are ready for authentication.

ⓧ A shell has an older client

Replace the older client with the download supplied by the current Sandbox. This keeps the affected shell aligned with the assigned cluster.

  • Return to the OpenShift web console.
  • Select ? in the console header.
  • Select Command Line Tools.
  • Download the client package that matches the affected shell.
  • Select Download oc for Windows for x86_64 when updating Windows PowerShell.
  • Extract the downloaded package.
  • Place oc on the affected shell's PATH.
  • Verify the replacement client by running:
oc version

What should change?

The affected shell should now report the client downloaded from the current Sandbox. Repeat the same check in the other shell before continuing.

ⓧ Command not found in a shell

Install the current Sandbox client in the shell where the command is missing. The WSL or container shell needs it for Ansible recovery.

  • Return to the OpenShift web console.
  • Select ? in the console header.
  • Select Command Line Tools.
  • Download the client package that matches the affected shell.
  • Select Download oc for Windows for x86_64 when installing it for Windows PowerShell.
  • Extract the downloaded package.
  • Place oc on the affected shell's PATH.
  • Verify the installed client by running:
oc version

What should change?

The affected shell should now print OpenShift client information. Repeat the same check in the other shell before continuing.

Still seeing the wrong client?

An older executable may appear earlier on PATH. Remove that path conflict before repeating the version check.

Help me find which oc executable my shell is using.

Keep the login token private

The console-generated login command contains a live token. Keep it out of screenshots. Keep it out of Git history.

  • Copy the login command supplied by the current OpenShift console.
  • Switch back to the WSL or container shell used for Ansible.
  • Paste the copied command into that shell.
  • Press Enter to authenticate.
  • Confirm that oc can identify your account while kubectl can reach the cluster by running:
oc whoami
kubectl api-versions

What do these results prove?

  • The identity result proves that the OpenShift client can authenticate to your assigned cluster.
  • The API discovery result proves that kubectl can use the current cluster credentials.

You should see your OpenShift username. You should also see a list of API versions from the live cluster.

  • Switch to Windows PowerShell.
  • Paste the same console-generated login command into PowerShell.
  • Press Enter to authenticate.
  • Confirm the PowerShell session by running:
oc whoami

What should PowerShell show?

PowerShell should print the same OpenShift username. The watcher can now discover Routes through this authenticated session.

Cluster authentication failed?

A Sandbox login token can expire. Copy a fresh login command from the current console without sharing its token.

Help me repair my OpenShift login safely.

Create the recovery repository

The repository keeps the failure demonstration separate from the resilient configuration. Dedicated folders also give the recovery playbook stable file locations.

  • Press the Windows key to open search.
  • Type File Explorer into the search field.
  • Press Enter to open File Explorer.
  • Navigate to the Windows folder already mounted into your WSL or container environment.
  • Create a folder named openshift-resilience-lab with File Explorer's folder creation control.

You should now see the empty openshift-resilience-lab folder in File Explorer.

  • Open the openshift-resilience-lab folder.
  • Create a subfolder named lab.
  • Create a subfolder named manifests.

File Explorer should now show lab beside manifests.

  • Create a subfolder named ansible.
  • Create a subfolder named scripts.

You should now see four subfolders inside openshift-resilience-lab.

  • Create a subfolder named secret-mission.

The repository folder should now contain lab, manifests, ansible, scripts, and secret-mission.

  • Switch back to the WSL or container shell used for Ansible.
  • Navigate through the existing filesystem mount to the openshift-resilience-lab folder.
  • Initialize the repository and inspect its status by running:
git init
git status

What do these commands do?

  • The first command turns openshift-resilience-lab into a Git repository.
  • The second command reports the current repository state.
  • Git leaves empty folders untracked until later steps add files inside them.

You should see confirmation that Git initialized the repository. File Explorer should still show all five folders.

Git initialized the wrong folder?

Your shell path should end with openshift-resilience-lab before initialization. Navigate to the folder through the mount used by your existing WSL or container setup.

Help me reach the repository from my recovery shell.

Before you run the final check, which results do you expect to come from local tools? Which result needs the live cluster?

  • Verify the complete recovery workspace by running:
oc whoami
kubectl version --client
oc api-versions
ansible-playbook --version

What does the final check prove?

  • The identity result proves that your OpenShift session is authenticated.
  • The kubectl result proves that the Kubernetes client is available in the recovery shell.
  • The API discovery result proves that the live cluster exposes the resource APIs required by the lab.
  • The Ansible result proves that the recovery playbook runner is available.

You should see your OpenShift username. Both client checks should print version information.

The API list should contain apps/v1. It should contain policy/v1. It should also contain route.openshift.io/v1.

Your authenticated recovery workspace is ready. Next, you will deploy an unmanaged Pod to expose the gap that self-healing must close.

Break an Unmanaged Pod

Your authenticated OpenShift workspace is ready. The next question is what happens when the only application Pod disappears.

An unmanaged Pod has no higher-level controller maintaining its desired state. Deleting it removes the only application backend until you intervene.

In this step, you will expose one bare Pod through a Service and an OpenShift Route. You will delete it while a Windows PowerShell watcher checks the /healthz endpoint.

In this step, get ready to:
  • Create a PowerShell watcher that records the Route health over time.
  • Deploy an unmanaged Pod behind a Service and secure Route.
  • Delete the Pod to reveal the missing self-healing behavior.
Build the Route watcher

A polling watcher turns availability into timestamped evidence. It distinguishes a missing Route from an existing Route whose backend cannot answer.

  • Press the Windows key to open Windows Search.
  • Type Notepad and press Enter to open Windows Notepad.
  • Start the watcher by pasting this runnable polling loop into the blank document:
param(
    [int]$IntervalSeconds = 2
)

Write-Host "Watching the resilience-demo Route. Press Ctrl+C to stop."

while ($true) {
    $timestamp = Get-Date -Format "HH:mm:ss"
    $routeHost = oc get route resilience-demo -o jsonpath='{.spec.host}' 2>$null

    if ($LASTEXITCODE -ne 0 -or -not $routeHost) {
        Write-Host "$timestamp ROUTE_MISSING"
    }

    Start-Sleep -Seconds $IntervalSeconds
}

What does the first watcher version do?

  • The IntervalSeconds parameter controls how long the loop waits between checks.
  • The routeHost variable stores the generated hostname for the resilience-demo Route.
  • The LASTEXITCODE check detects when OpenShift cannot return that Route.
  • The loop prints ROUTE_MISSING with the current timestamp when no Route exists.
  • Find this closing section in the document:
    if ($LASTEXITCODE -ne 0 -or -not $routeHost) {
        Write-Host "$timestamp ROUTE_MISSING"
    }

    Start-Sleep -Seconds $IntervalSeconds
}

Why replace this section?

This section currently handles only a missing Route. The replacement adds an HTTP request for the moments when the Route exists.

  • Replace that closing section with the complete response-checking branch below:
    if ($LASTEXITCODE -ne 0 -or -not $routeHost) {
        Write-Host "$timestamp ROUTE_MISSING"
    }
    else {
        try {
            $response = Invoke-WebRequest `
                -Uri "https://$routeHost/healthz" `
                -Method Get `
                -UseBasicParsing `
                -ErrorAction Stop

            Write-Host "$timestamp HTTP $($response.StatusCode)"
        }
        catch {
            Write-Host "$timestamp DOWN"
        }
    }

    Start-Sleep -Seconds $IntervalSeconds
}

How does the health check work?

  • Invoke-WebRequest sends a request to the Route's /healthz endpoint.
  • The response variable holds the successful HTTP result.
  • The watcher prints the response status code when the endpoint answers.
  • The catch branch prints DOWN when the Route exists without a responding backend.

✔️ Awesome, I've got everything!

Your watcher now distinguishes all three Route states.

ⓧ I'd like to double check the full code

param(
    [int]$IntervalSeconds = 2
)

Write-Host "Watching the resilience-demo Route. Press Ctrl+C to stop."

while ($true) {
    $timestamp = Get-Date -Format "HH:mm:ss"
    $routeHost = oc get route resilience-demo -o jsonpath='{.spec.host}' 2>$null

    if ($LASTEXITCODE -ne 0 -or -not $routeHost) {
        Write-Host "$timestamp ROUTE_MISSING"
    }
    else {
        try {
            $response = Invoke-WebRequest `
                -Uri "https://$routeHost/healthz" `
                -Method Get `
                -UseBasicParsing `
                -ErrorAction Stop

            Write-Host "$timestamp HTTP $($response.StatusCode)"
        }
        catch {
            Write-Host "$timestamp DOWN"
        }
    }

    Start-Sleep -Seconds $IntervalSeconds
}
  • Save the document as scripts/watch-route.ps1 inside the openshift-resilience-lab folder.
  • Switch to a second terminal at the openshift-resilience-lab folder.
  • Start the watcher by running this command:
powershell.exe -ExecutionPolicy Bypass -File .\scripts\watch-route.ps1

What does this command do?

The command runs the watcher with a process-scoped execution policy bypass. The script keeps polling until you stop it.

Good, your watcher can already distinguish a missing Route from a failed backend. You will see timestamped ROUTE_MISSING lines because the application Route does not exist yet.

Watcher not starting?

  • Confirm that the terminal is at the openshift-resilience-lab folder.
  • Confirm that Notepad saved the file as watch-route.ps1 inside the scripts folder.
  • Start the watcher from the shell where the OpenShift client from the previous step is available.

Ask for help with the watcher error.

Create the fragile stack

The first stack deliberately uses a bare Pod. Its labels give the Service a backend without adding a controller that could replace it.

  • Return to Windows Notepad.
  • Create a blank document for lab/fragile-stack.yaml.
  • Add the unmanaged Pod by pasting this definition:
apiVersion: v1
kind: Pod
metadata:
  name: fragile-web
  labels:
    app: resilience-demo
    app.kubernetes.io/name: resilience-demo
    app.kubernetes.io/part-of: resilience-demo
spec:
  containers:
    - name: web
      image: registry.k8s.io/goproxy:0.1
      ports:
        - name: http
          containerPort: 8080
          protocol: TCP
      resources:
        requests:
          cpu: 10m
          memory: 16Mi
        limits:
          cpu: 100m
          memory: 64Mi

What does the Pod definition do?

  • The fragile-web Pod runs the health-check image on port 8080.
  • The app: resilience-demo label gives the Service a stable way to select this Pod.
  • The resource requests reserve a small amount of cluster capacity for the container.
  • The resource limits cap the container's CPU and memory use.
  • Save the document as lab/fragile-stack.yaml inside the openshift-resilience-lab folder.
  • Create the Pod from the saved manifest by running this command in the authenticated shell:
kubectl apply -f lab/fragile-stack.yaml

What does this command do?

The apply operation sends the resource definition from lab/fragile-stack.yaml to the current cluster. Reusing the same command after later edits keeps the cluster aligned with the file.

  • Check the Pod by running this command:
kubectl get pods -l app=resilience-demo

What should you see?

The label selector limits the listing to this lab's workload. You will see fragile-web in the results.

  • Repeat the previous check until the Pod reports that it is ready.

Pod not becoming ready?

  • Check that the image value in lab/fragile-stack.yaml matches the supplied Pod definition.
  • Check that the container port remains 8080.

Help me diagnose why the fragile Pod is not becoming ready.

  • Return to lab/fragile-stack.yaml in Notepad.
  • Append the Service definition below the Pod definition:
---
apiVersion: v1
kind: Service
metadata:
  name: resilience-demo
  labels:
    app.kubernetes.io/part-of: resilience-demo
spec:
  selector:
    app: resilience-demo
  ports:
    - name: http
      port: 8080
      targetPort: http
      protocol: TCP

How does the Service find the Pod?

The Service selects workloads carrying the app: resilience-demo label. It forwards port 8080 to the Pod port named http.

  • Save lab/fragile-stack.yaml.
  • Create the Service from the updated manifest by running this command:
kubectl apply -f lab/fragile-stack.yaml

What changes in the cluster?

The apply operation keeps the existing Pod definition. It adds the Service from the second document in the manifest.

That is the internal connection in place. The Service now selects the Pod through their shared label.

Service apply failed?

  • Confirm that the --- separator sits between the Pod and Service documents.
  • Confirm that the Service selector matches app: resilience-demo.

Help me fix the Service section in my fragile stack manifest.

  • Return to the end of lab/fragile-stack.yaml.
  • Append the secure Route definition below the Service definition:
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: resilience-demo
  labels:
    app.kubernetes.io/part-of: resilience-demo
spec:
  port:
    targetPort: 8080
  to:
    kind: Service
    name: resilience-demo
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect

How does the Route expose the service?

The Route sends public requests to the resilience-demo Service on port 8080. Edge termination handles the secure connection at the OpenShift router.

The redirect policy moves insecure requests to the secure endpoint. The watcher uses that HTTPS address for every health check.

✔️ Awesome, I've got everything!

Your manifest now contains the Pod, Service, and Route.

ⓧ I'd like to double check the full code

apiVersion: v1
kind: Pod
metadata:
  name: fragile-web
  labels:
    app: resilience-demo
    app.kubernetes.io/name: resilience-demo
    app.kubernetes.io/part-of: resilience-demo
spec:
  containers:
    - name: web
      image: registry.k8s.io/goproxy:0.1
      ports:
        - name: http
          containerPort: 8080
          protocol: TCP
      resources:
        requests:
          cpu: 10m
          memory: 16Mi
        limits:
          cpu: 100m
          memory: 64Mi
---
apiVersion: v1
kind: Service
metadata:
  name: resilience-demo
  labels:
    app.kubernetes.io/part-of: resilience-demo
spec:
  selector:
    app: resilience-demo
  ports:
    - name: http
      port: 8080
      targetPort: http
      protocol: TCP
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: resilience-demo
  labels:
    app.kubernetes.io/part-of: resilience-demo
spec:
  port:
    targetPort: 8080
  to:
    kind: Service
    name: resilience-demo
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect
  • Save lab/fragile-stack.yaml.
  • Create the Route from the completed manifest by running this command:
kubectl apply -f lab/fragile-stack.yaml

What completes the request path?

The final apply operation adds the public Route. Requests can now travel through the Route and Service to the Pod.

Within a few polling intervals, the watcher will print timestamped HTTP 200 lines. Those lines prove the full request path is healthy.

Watcher still reporting a failure?

  • Use the earlier Pod listing to confirm that fragile-web is ready.
  • Confirm that the Service selector matches the Pod's app: resilience-demo label.
  • Confirm that the Route points to the resilience-demo Service.

Help me trace why the resilience-demo Route is not returning a successful health response.

Delete the unmanaged Pod

The endpoint is healthy while its only Pod exists. The cluster has no controller responsible for restoring that Pod after deletion.

Before you delete it, do you expect the endpoint to recover on its own?

  • Delete the unmanaged Pod by running this command:
kubectl delete pod fragile-web

What does this command remove?

The command deletes the Pod named fragile-web. The Service and Route remain because they are separate resources.

The watcher changes from HTTP 200 to timestamped DOWN lines. The surviving Route has no healthy backend to answer its request.

Watcher still showing a successful response?

  • Allow the current request to finish before reading the next polling result.
  • Confirm that the deletion command named fragile-web.
  • Confirm that the watcher is checking the resilience-demo Route.

Help me understand why the watcher still reports a successful response after deleting fragile-web.

Before you check the cluster, do you expect to find a replacement Pod with a new name?

  • Check for a replacement Pod by running this command:
kubectl get pods -l app=resilience-demo

What does this result prove?

The listing contains no Pod matching the application label. No controller knows that fragile-web should exist.

The outage is the intended result of this drill. A bare Pod carries no replica count for OpenShift to restore.

Still see a matching Pod?

  • Wait for the deletion to finish before repeating the listing.
  • Confirm that the remaining name is not fragile-web from a deletion still in progress.

Help me identify why a Pod still matches the resilience-demo label after deleting fragile-web.

You have captured the exact failure the resilient design must prevent. Next, you will replace the bare Pod with a managed Deployment that restores its desired replica count.

Build the Self-Healing Service

Your unmanaged Pod stayed gone after deletion. The watcher still reports DOWN because no controller owns a replacement.

A Deployment closes that gap by keeping the cluster aligned with a declared replica count. Its ReplicaSet creates replacements whenever the actual Pod count falls below the desired state.

Readiness probes control which Pods receive traffic. Liveness probes restart containers that stop responding.

In this step, get ready to:
  • Define a three-replica Deployment with health probes plus rolling-update safeguards.
  • Complete the Git-managed Service, OpenShift Route, and PodDisruptionBudget configuration.
  • Apply the resilient stack and confirm that the health endpoint recovers.
Create the three-replica Deployment

The first part of the manifest declares the controller plus its replica target. This valid starting point lets you confirm that the Deployment can create three Pods before you add health checks.

  • Press the Windows key to open the Windows search bar.
  • Type Notepad into the search bar.
  • Press Enter to open Notepad.
  • Use Notepad's save dialog to create resilient-stack.yaml inside the manifests folder in openshift-resilience-lab.
  • Paste this Deployment scaffold into manifests/resilient-stack.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: resilience-demo
  labels:
    app: resilience-demo
    app.kubernetes.io/name: resilience-demo
    app.kubernetes.io/part-of: resilience-demo
spec:
  replicas: 3
  revisionHistoryLimit: 3
  progressDeadlineSeconds: 60
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app: resilience-demo
  template:
    metadata:
      labels:
        app: resilience-demo
        app.kubernetes.io/name: resilience-demo
        app.kubernetes.io/part-of: resilience-demo
    spec:
      containers:
        - name: web
          image: registry.k8s.io/goproxy:0.1

How does the controller scaffold work?

  • The replicas: 3 setting records the desired Pod count.
  • The RollingUpdate strategy preserves current capacity while a replacement starts.
  • The selector links the Deployment to Pods carrying the app: resilience-demo label.
  • The revision limit keeps recent rollout history available for recovery.
  • Save manifests/resilient-stack.yaml.
  • Switch back to the WSL or container shell from earlier.
  • Create the Deployment from the scaffold by running this command:
kubectl apply -f manifests/resilient-stack.yaml

What does this command do?

The command submits the Deployment manifest to the current namespace. The Deployment controller starts working toward three matching Pods.

  • Inspect the controller's first set of Pods by running this command:
kubectl get pods -l app=resilience-demo

What should you see?

You should see three Pod rows created by the Deployment. Each Pod should reach a ready state.

Deployment not creating three Pods?

  • Confirm that Notepad saved the file as resilient-stack.yaml without an added .txt extension.
  • Check the indentation beneath selector plus template.

Ask for help with the manifest:

The Pods now exist under controller ownership. Their health behavior still needs explicit rules before the service can depend on them.

  • Switch back to manifests/resilient-stack.yaml in Notepad.
  • Find the image: registry.k8s.io/goproxy:0.1 line.
  • Insert this configuration directly below the image line:
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /healthz
              port: http
              scheme: HTTP
            initialDelaySeconds: 2
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
          resources:
            requests:
              cpu: 10m
              memory: 16Mi
            limits:
              cpu: 100m
              memory: 64Mi

What do these health settings do?

  • The named http port gives the Service a stable target for traffic.
  • The readiness probe keeps a Pod out of Service endpoints until /healthz responds successfully.
  • The liveness probe restarts the container after repeated health-check failures.
  • The resource requests reserve baseline capacity. The limits cap the container's CPU plus memory use.
  • Save manifests/resilient-stack.yaml.

Before you apply this update, do you expect the watcher to recover when the new Pods become ready?

  • Switch back to the WSL or container shell from earlier.
  • Apply the health configuration and wait for the rollout by running these commands:
kubectl apply -f manifests/resilient-stack.yaml
kubectl rollout status deployment/resilience-demo --timeout=120s

What does this check prove?

The apply command updates the Pod template with the named port plus probes. The rollout command waits for the new ReplicaSet to make the updated Pods available.

  • Switch back to the PowerShell watcher from earlier.

You should see fresh timestamped HTTP 200 results. The controller-backed Pods have restored the Route's healthy backend.

That recovery is your first resilience win. The endpoint is responding through Pods managed by a Deployment.

Watcher still reporting DOWN?

  • Check that ports is aligned with readinessProbe beneath the web container.
  • Confirm that both probes use /healthz plus the named http port.

Diagnose the unhealthy rollout:

Add the network and disruption resources

The healthy Deployment currently uses the Service plus Route left by the fragile stack. Your recovery manifest must declare those resources so the whole application can be reconstructed from one file.

  • Switch back to manifests/resilient-stack.yaml in Notepad.
  • Place the cursor after the final memory: 64Mi line.
  • Append the Service definition below the Deployment:
---
apiVersion: v1
kind: Service
metadata:
  name: resilience-demo
  labels:
    app.kubernetes.io/part-of: resilience-demo
spec:
  selector:
    app: resilience-demo
  ports:
    - name: http
      port: 8080
      targetPort: http
      protocol: TCP

How does the Service find the Pods?

The Service selects Pods with app: resilience-demo. Its targetPort: http setting resolves to the named container port.

  • Place the cursor after the Service's final protocol: TCP line.
  • Append the Route definition below the Service:
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: resilience-demo
  labels:
    app.kubernetes.io/part-of: resilience-demo
spec:
  port:
    targetPort: 8080
  to:
    kind: Service
    name: resilience-demo
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect

How does the Route expose the Service?

The Route sends external requests to the resilience-demo Service on port 8080. Edge termination handles secure traffic at the OpenShift router.

The redirect policy moves insecure requests to the secure endpoint. Your watcher continues polling the Route over HTTPS.

  • Save manifests/resilient-stack.yaml.
  • Switch back to the WSL or container shell from earlier.
  • Reconcile the existing Service plus Route from the new manifest by running this command:
kubectl apply -f manifests/resilient-stack.yaml

What should this apply report?

The command reconciles the Deployment plus the network resources from one manifest. The Service and Route can remain unchanged because their live configuration already matches this file.

  • Switch back to the watcher to confirm that it continues reporting HTTP 200.

Route stopped responding after the apply?

  • Check that the Service selector remains app: resilience-demo.
  • Confirm that the Route points to the resilience-demo Service.

Trace the Route-to-Pod connection:

A disruption budget adds an availability rule for controlled maintenance. This project keeps at least two matching Pods available during voluntary evictions.

  • Switch back to manifests/resilient-stack.yaml in Notepad.
  • Place the cursor after the Route's final insecureEdgeTerminationPolicy: Redirect line.
  • Append the PodDisruptionBudget definition below the Route:
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: resilience-demo
  labels:
    app.kubernetes.io/part-of: resilience-demo
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: resilience-demo

What protection does the budget provide?

The budget asks eviction workflows to keep at least two matching Pods available. Its scope is voluntary eviction.

Deployment reconciliation handles replacement after direct Pod deletion. Node failures remain outside the budget's guarantee.

  • Save manifests/resilient-stack.yaml.
  • Switch back to the WSL or container shell from earlier.
  • Create the disruption budget and inspect it by running these commands:
kubectl apply -f manifests/resilient-stack.yaml
kubectl get poddisruptionbudgets

What should you see?

The apply command creates the resilience-demo PodDisruptionBudget. The listing should show a minimum availability value of 2.

PodDisruptionBudget missing?

  • Confirm that the final document starts with apiVersion: policy/v1.
  • Check that the PDB selector matches app: resilience-demo.

Find the PDB configuration problem:

✔️ Awesome, I've got everything!

Your resilient manifest now contains the Deployment, Service, Route, and PodDisruptionBudget. Save the file before the final verification.

ⓧ I'd like to double check the full code

apiVersion: apps/v1
kind: Deployment
metadata:
  name: resilience-demo
  labels:
    app: resilience-demo
    app.kubernetes.io/name: resilience-demo
    app.kubernetes.io/part-of: resilience-demo
spec:
  replicas: 3
  revisionHistoryLimit: 3
  progressDeadlineSeconds: 60
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app: resilience-demo
  template:
    metadata:
      labels:
        app: resilience-demo
        app.kubernetes.io/name: resilience-demo
        app.kubernetes.io/part-of: resilience-demo
    spec:
      containers:
        - name: web
          image: registry.k8s.io/goproxy:0.1
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /healthz
              port: http
              scheme: HTTP
            initialDelaySeconds: 2
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
          resources:
            requests:
              cpu: 10m
              memory: 16Mi
            limits:
              cpu: 100m
              memory: 64Mi
---
apiVersion: v1
kind: Service
metadata:
  name: resilience-demo
  labels:
    app.kubernetes.io/part-of: resilience-demo
spec:
  selector:
    app: resilience-demo
  ports:
    - name: http
      port: 8080
      targetPort: http
      protocol: TCP
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: resilience-demo
  labels:
    app.kubernetes.io/part-of: resilience-demo
spec:
  port:
    targetPort: 8080
  to:
    kind: Service
    name: resilience-demo
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: resilience-demo
  labels:
    app.kubernetes.io/part-of: resilience-demo
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: resilience-demo
Verify the resilient stack

The final check combines controller progress, Pod readiness, and disruption-budget state. Together, these signals prove that the cluster reached the manifest's declared state.

Before you run the final check, which signal do you expect to confirm that the Deployment reached its replica target?

  • Switch back to the WSL or container shell from earlier.
  • Verify the rollout plus its current availability by running these commands:
kubectl rollout status deployment/resilience-demo --timeout=120s
kubectl get pods -l app=resilience-demo
kubectl get poddisruptionbudgets

What should the final check show?

  • The rollout command should complete successfully.
  • The Pod listing should contain three rows. Each row should show one ready container.
  • The PDB listing should contain resilience-demo with a minimum availability value of 2.
  • Switch back to the PowerShell watcher from earlier.

You should see timestamped HTTP 200 results continuing to appear. The Route now has three healthy backends under controller ownership.

You have closed the outage gap from the unmanaged Pod. The service now returns to its declared replica count through Deployment reconciliation.

Final rollout not healthy?

  • Compare manifests/resilient-stack.yaml with the full-code reference above.
  • Check the Pod listing for any row that has not reached its ready state.
  • Confirm that the watcher uses the same authenticated cluster context as your shell.

Investigate the final health signals:

Your controller-backed service is healthy. Next, you will test that reliability promise during a live Pod failure.

Run a Live Pod-Failure Drill

Your three-replica Deployment is healthy. Its Route currently answers every watcher request.

A replica count on paper is only a promise. This drill tests that promise by removing one managed Pod while requests continue.

In this step, get ready to:
  • Record the current managed Pod names.
  • Delete one managed Pod while the watcher polls the endpoint.
  • Verify that the ReplicaSet restores the desired replica count.
Record the healthy baseline

Generated Pod names let you distinguish the original replicas from the replacement. A baseline gives you the evidence needed for that comparison.

  • Switch back to the cluster-command shell from earlier.
  • List the current managed Pods by running this command:
kubectl get pods -l app=resilience-demo

What does this check prove?

  • The label selector limits the result to Pods in the resilience service.
  • The three generated names form the baseline for spotting the replacement.

You should see three Pod rows. Every row should show that its single container is ready and running.

  • Keep this output visible for comparison.
  • Record one current Pod name here: paste one current Pod name.

Don't see three ready Pods?

  • Wait until the listing contains three ready Pod rows before continuing.
  • Check that your current namespace still contains the healthy Deployment from the previous step.

Help me diagnose why my resilience-demo Deployment does not have three ready Pods.

Delete one managed Pod

Deleting one replica creates a gap between actual state and desired state. The Deployment controller must close that gap while the watcher keeps polling.

  • Return to the PowerShell watcher from earlier.

The latest watcher line should report HTTP 200.

  • Switch back to the cluster-command shell.

Before you run this, what do you expect the next watcher line to report? Make your prediction now.

  • Delete the selected managed Pod by running this command:
kubectl delete pod [[POD_NAME="paste one current Pod name"]]

What does this deletion test?

  • The command removes one named Pod directly.
  • The deletion immediately reduces the actual replica count.
  • The Deployment controller now has to restore the declared count of three.
  • Watch the next several polling lines in the PowerShell watcher.

You should keep seeing timestamped HTTP 200 lines. Your endpoint has just stayed available through a live Pod loss.

Seeing repeated outage results?

  • Confirm that at least two Pods remain ready by using the listing from the first substep.
  • Check that the watcher still resolves the existing resilience-demo Route.
  • Copy another current Pod name if the selected name no longer exists.

Help me diagnose why deleting one resilience-demo Pod interrupted the Route.

Confirm desired-state restoration

The replacement receives a new generated name. That name change proves the controller created a fresh Pod instead of reviving the deleted one.

Before you check, how many ready Pods do you expect after reconciliation? Lock in your prediction now.

  • Check the restored replica count by running this command:
kubectl get pods -l app=resilience-demo

What does the second listing prove?

  • The repeated listing shows the current replica count after deletion.
  • A new generated name identifies the replacement Pod.
  • Three ready rows confirm that reconciliation has completed.
  • Repeat the same Pod listing until every container in three Pod rows is ready.

You should now see three ready Pod rows. One generated name should differ from the baseline.

Still below three ready Pods?

  • Check whether the replacement Pod is still starting.
  • Confirm that the Deployment still declares three replicas.
  • Review the replacement Pod events for a scheduling or image-pull problem.

Help me diagnose why the replacement Pod is not becoming ready.

  • Compare the replacement Pod name with paste one current Pod name.
  • Return to the OpenShift web console for your assigned project.
  • Select Topology from the project navigation.
  • Select the resilience-demo Deployment.
  • Review its recent events for the deletion and replacement activity.

You should see the old Pod leave. You should see a replacement appear beneath the same ReplicaSet.

The replacement then becomes ready. The Deployment settles at three ready replicas.

What protected the endpoint?

The Deployment preserves its declared replica count through the ReplicaSet. The Service keeps routing to ready endpoints. The readiness probe controls when the replacement joins traffic.

The PodDisruptionBudget uses minAvailable: 2 for voluntary evictions. Manual Pod deletion falls outside that protection. Deployment reconciliation replaced the deleted Pod.

You have solid evidence now: one managed Pod disappeared while the public endpoint stayed healthy.

Your Deployment has proved runtime self-healing. Next, you'll test recovery when the whole application stack disappears.

Recover the Deleted Workload from Git

Your managed Pod drill proved that OpenShift restores a missing Pod while its Deployment still exists. A complete application-resource deletion removes the controller that performs that repair.

Git preserves the application’s desired state outside the cluster. Ansible turns that recovery source into an executable procedure.

In this step, get ready to:
  • Create an Ansible playbook that reapplies the Git-managed manifest.
  • Commit the recovery source to Git.
  • Prove full-stack recovery after deleting the application resources.
Build the Git recovery source

The recovery playbook uses a path relative to its own location. The repository can therefore travel between environments without hard-coded local paths.

  • Switch back to Windows Notepad from earlier.
  • Create ansible/recover.yml inside the ansible folder by pasting this first recovery action:
---
- name: Recover the OpenShift resilience demo
  hosts: 127.0.0.1
  connection: local
  gather_facts: false
  vars:
    manifest_path: "{{ playbook_dir }}/../manifests/resilient-stack.yaml"
  tasks:
    - name: Apply the Git-managed application resources
      ansible.builtin.command:
        argv:
          - kubectl
          - apply
          - -f
          - "{{ manifest_path }}"
      register: apply_result
      changed_when: "'created' in apply_result.stdout or 'configured' in apply_result.stdout"

What does this playbook do so far?

  • manifest_path locates manifests/resilient-stack.yaml from the playbook directory.
  • ansible.builtin.command invokes kubectl with separate arguments.
  • apply_result captures the command result for the change check.
  • changed_when reports a change only when the cluster creates or updates a resource.
  • Save ansible/recover.yml.
  • Test the first recovery action from the openshift-resilience-lab folder by running:
ansible-playbook ansible/recover.yml

What does this first run prove?

The playbook can locate the Kubernetes manifest. It can also apply that manifest through your authenticated cluster context.

The existing resources remain configured. The play recap should complete with failed=0.

Did the first playbook run fail?

  • Confirm that your shell is at the openshift-resilience-lab folder if the playbook path cannot be found.
  • Check that the shell still shares your authenticated OpenShift cluster context.
  • Confirm that manifests/resilient-stack.yaml still exists inside the repository.

Help me diagnose why my Ansible recovery playbook cannot apply the existing OpenShift manifest.

Applying the manifest restores resources. A complete recovery procedure must also wait for readiness before reporting the public endpoint.

  • Place these remaining tasks directly below the first task in ansible/recover.yml by pasting:
    - name: Wait for the resilient deployment
      ansible.builtin.command:
        argv:
          - kubectl
          - rollout
          - status
          - deployment/resilience-demo
          - --timeout=120s
      changed_when: false

    - name: Read the restored Route host
      ansible.builtin.command:
        argv:
          - kubectl
          - get
          - route
          - resilience-demo
          - -o
          - jsonpath={.spec.host}
      register: route_host
      changed_when: false

    - name: Show the recovered health endpoint
      ansible.builtin.debug:
        msg: "Recovered route: https://{{ route_host.stdout }}/healthz"

How does the recovery finish?

  • The rollout task waits for deployment/resilience-demo to become available.
  • The host lookup reads the admitted Route into route_host.
  • The final task prints the restored health endpoint from that host.
  • Save ansible/recover.yml.
  • Verify the complete playbook against the healthy stack by running:
ansible-playbook ansible/recover.yml

What does this verification prove?

The playbook now applies the manifest before waiting for the Deployment. It finishes by reading the Route host from the live cluster.

The recap should show failed=0. The final task should print a recovered URL ending in /healthz.

Missing the recovered Route URL?

  • Check that the Route task remains indented under the playbook’s tasks list.
  • Confirm that the host lookup stores its result in route_host.
  • Compare the final message with route_host.stdout if the URL is empty.

Help me fix the Route-reporting tasks in my Ansible recovery playbook.

✔️ Awesome, I've got everything!

Your full recovery playbook now matches the recovery design.

ⓧ I'd like to double check the full code

---
- name: Recover the OpenShift resilience demo
  hosts: 127.0.0.1
  connection: local
  gather_facts: false
  vars:
    manifest_path: "{{ playbook_dir }}/../manifests/resilient-stack.yaml"
  tasks:
    - name: Apply the Git-managed application resources
      ansible.builtin.command:
        argv:
          - kubectl
          - apply
          - -f
          - "{{ manifest_path }}"
      register: apply_result
      changed_when: "'created' in apply_result.stdout or 'configured' in apply_result.stdout"

    - name: Wait for the resilient deployment
      ansible.builtin.command:
        argv:
          - kubectl
          - rollout
          - status
          - deployment/resilience-demo
          - --timeout=120s
      changed_when: false

    - name: Read the restored Route host
      ansible.builtin.command:
        argv:
          - kubectl
          - get
          - route
          - resilience-demo
          - -o
          - jsonpath={.spec.host}
      register: route_host
      changed_when: false

    - name: Show the recovered health endpoint
      ansible.builtin.debug:
        msg: "Recovered route: https://{{ route_host.stdout }}/healthz"

The repository now contains every file required to reproduce the resilient application. Committing that state creates a stable recovery source before the destructive drill.

  • Stage the repository files before recording the recovery source by running these commands:
git add .
git commit -m "Add OpenShift resilience recovery lab"
git status

What do these Git commands record?

  • git add . stages the repository contents.
  • git commit -m "Add OpenShift resilience recovery lab" records the staged recovery source with the required message.
  • git status checks for files that remain outside the commit.

That recovery source is now locked in. Git status should report a clean working tree.

Could Git not create the commit?

  • Follow Git’s guidance to configure your author identity if the commit stops for missing identity details.
  • Run the staging sequence again after correcting the identity configuration.
  • Confirm that ansible/recover.yml appears in the commit summary.

Help me diagnose why Git could not commit my OpenShift recovery files.

Expose the recovery gap

The live Deployment can replace one of its Pods. Removing the Deployment itself removes the controller that holds the desired replica count.

Before you run this drill, predict whether Deployment self-healing can recreate its own deleted Deployment.

  • Delete every application resource defined in manifests/resilient-stack.yaml by running:
kubectl delete -f manifests/resilient-stack.yaml

What did this deletion remove?

The manifest identifies the resilient Deployment. It also identifies the Service, Route, and PodDisruptionBudget.

Deleting the Deployment also removes its managed ReplicaSets and Pods. No workload controller remains to reconstruct the stack.

  • Switch back to the PowerShell watcher from earlier.

You’ll see timestamped ROUTE_MISSING or DOWN results. This is the expected application-resource outage.

  • Confirm that the workload has no remaining matching Pods by running:
kubectl get pods -l app=resilience-demo

Why does no Pod return?

The Pod listing has no controller to satisfy a replica target. The committed manifest now holds the only complete desired state for this application.

The listing contains no matching Pods. The watcher remains unable to reach the health endpoint.

Does the watcher still return HTTP 200?

  • Confirm that the delete command reported removal of the Route.
  • Check that the watcher is polling the resilience-demo Route.
  • Wait for the deletion to finish propagating before checking the next timestamp.

Help me determine why the Route watcher still returns HTTP 200 after deleting the resilient manifest.

Close the outage window

The outage proves that runtime reconciliation needs a surviving controller. Your committed playbook can now reintroduce the desired state from outside the cluster.

A quiet pause of up to 120 seconds is expected while the Deployment waits for healthy replicas.

Before you run the recovery, predict which output will prove that the public endpoint has returned.

  • Execute the committed recovery procedure from the openshift-resilience-lab folder by running:
ansible-playbook ansible/recover.yml

How does the playbook reconstruct the stack?

  • The apply task recreates the resources from manifests/resilient-stack.yaml.
  • The rollout task waits for the new Deployment to become healthy.
  • The Route task reads the newly admitted host.
  • The debug task prints the restored health endpoint.
  • Review the final Ansible recap in the WSL or container shell.

You’ll see failed=0. You’ll also see the recovered HTTPS health URL.

  • Switch back to the PowerShell watcher from earlier.

The timestamped results return to HTTP 200. The outage window is now closed.

Did the recovery playbook stop?

  • Confirm that the recovery shell still shares the authenticated cluster context.
  • Check the failed task name to identify whether apply, rollout, or Route lookup stopped.
  • Compare ansible/recover.yml with the full-code reference if Ansible reports a YAML parsing problem.

Help me diagnose the failed task in my OpenShift recovery playbook.

Before the final check, predict how many ready Pods the restored Deployment should manage.

  • Verify the restored replica count by running:
kubectl get pods -l app=resilience-demo

What confirms full restoration?

A new ReplicaSet creates the replacement Pods from the recovered Deployment. Readiness probes keep each Pod out of Service traffic until its health endpoint responds.

You’ll see three Pods. Each Pod shows ready status.

Are fewer than three Pods ready?

  • Wait for the readiness checks to complete if a Pod is still starting.
  • Compare the Pod status with the watcher to determine whether the Route already has a healthy endpoint.
  • Review the Ansible rollout task output if the replica count remains below three.

Help me diagnose why the recovered Deployment has fewer than three ready Pods.

Strong finish. You deleted the complete application stack before rebuilding its public health endpoint from the Git-backed recovery source.

Secret mission

Reject and Roll Back a Bad Release

A bad release can satisfy the declared configuration while its new Pods remain unready. Apply a deliberately broken readiness probe, prove that the healthy replicas keep serving traffic, then restore the previous revision with a manual rollback.

Clean Up Your Resources

Clean Up Your Resources

The Developer Sandbox for Red Hat OpenShift is free for this lab. Decide whether to keep your resources running, pause them for later, or delete them entirely.

Sandbox Resources Are Temporary

The Sandbox trial is active for 30 days. User Pods are automatically deleted after 12 consecutive hours.

Resources you used:

  • Your active Sandbox application stack with one Deployment, one Service, one OpenShift Route, one PodDisruptionBudget, managed ReplicaSets, and three ready Pods.
  • Your Windows PowerShell watcher process running from scripts/watch-route.ps1.
  • Your local openshift-resilience-lab Git repository containing the lab manifests, recovery playbook, watcher, and Secret Mission manifest.

Keep everything running

No action is needed. Choose this if you want to repeat the failure drills or demonstrate the recovery workflow again.

  • Leave the resilience-demo Deployment at three replicas.
  • Keep the watcher running while you continue testing the health endpoint.
  • Keep the openshift-resilience-lab repository as your recovery source.

Sandbox lifecycle rules still apply. The platform can remove your Pods automatically after 12 consecutive hours.

Pause - I'll come back to this later

Shut down the application Pods to free cluster capacity. Your Deployment configuration and Route remain available for your return.

  • Return to the WSL or container shell from earlier.
  • Scale the Deployment to zero replicas by running this command:
kubectl scale deployment resilience-demo --replicas=0

What Does This Command Do?

Scaling the Deployment to zero removes its application Pods. The Deployment retains the configuration needed to create them again.

Scale Command Failed?

  • Confirm that this shell still uses the authenticated cluster context from earlier.
  • Confirm that the current project contains the resilience-demo Deployment.

Ask for help with the cluster context or permissions: Help me diagnose why my OpenShift Deployment would not scale to zero replicas.

  • Confirm that the application Pods have stopped by running this command:
kubectl get pods -l app=resilience-demo

What Should You See?

The listing should no longer show running Pods for the application. A Pod can briefly show a terminating state while the scale operation finishes.

  • Wait a few seconds if a Pod is still terminating.
  • Repeat the Pod listing until no application Pods remain.
  • Switch back to the Windows PowerShell terminal running the watcher.
  • Press Ctrl+C to stop the watcher.

Your cluster configuration and local Git repository remain ready for another session.

Delete - I don't want to use this again

Remove the active cluster stack if you are finished with the lab. Deleting it is deliberate because your Git repository can reconstruct the application later.

  • Return to the WSL or container shell from earlier.
  • Delete the resources defined in the resilient manifest by running this command:
kubectl delete -f manifests/resilient-stack.yaml

What Does This Command Delete?

The manifest identifies the Deployment, Service, Route, and PodDisruptionBudget. Removing the Deployment also removes the managed ReplicaSets and Pods.

You should see deletion confirmations for the four resources defined in manifests/resilient-stack.yaml.

Deletion Command Failed?

  • Confirm that you are running the command from the openshift-resilience-lab folder.
  • Confirm that this shell still has access to the authenticated Sandbox project.

Get help checking the manifest path or cluster context: Help me diagnose why my OpenShift manifest deletion failed.

  • Verify that the application Pods and PodDisruptionBudget are gone by running these commands:
kubectl get pods -l app=resilience-demo
kubectl get poddisruptionbudgets

What Should You See?

The Pod listing should not contain the application Pods. The PodDisruptionBudget listing should not contain resilience-demo.

Together with the deletion confirmations, these results show that the namespaced application stack is gone.

  • Switch back to the Windows PowerShell terminal running the watcher.
  • Press Ctrl+C to stop the watcher.

Keep the local Git repository as evidence and as a working recovery source. Remove it only if you want a complete local reset.

  • Press the Windows key to open search.
  • Type File Explorer to find the application.
  • Press Enter to open File Explorer.
  • Browse to the location containing the openshift-resilience-lab folder.
  • Select the openshift-resilience-lab folder.
  • Press the Delete key to move the folder to the Recycle Bin.

The openshift-resilience-lab folder should no longer appear in its previous location.

Nice Work!

Nice Work!

You did it! Your OpenShift health service kept responding while a managed Pod was replaced.

You also proved that the entire stateless application stack can be reconstructed from a version-controlled recovery source.

You learned how to:

  • Resilient service: Built a three-replica Deployment with readiness probes. Added liveness probes. Exposed the /healthz endpoint through a secure OpenShift Route. Applied rolling-update safeguards. Added a PodDisruptionBudget for voluntary evictions.
  • Failure testing: Proved that an unmanaged Pod stays deleted. Watched a ReplicaSet replace a deleted managed Pod. Confirmed the service kept returning HTTP 200.
  • Git-backed recovery: Reconstructed the deleted application stack from Git with an Ansible playbook. Confirmed the watcher moved from ROUTE_MISSING back to HTTP 200.
  • Secret Mission: Detected a readiness-blocked rollout while healthy replicas kept serving traffic. Manually rolled the Deployment back to its healthy revision.

Ready to quiz yourself?