Part 8: Parse Logs with Python and YAML

Build a Python tool that parses access logs and updates YAML configuration.

Introduction

30 Second Summary

A long record of website requests can hide one useful signal among thousands of ordinary characters. A 404 inside a page address can resemble a failed request even when the request succeeded.

In this project, you will build a Python utility that turns NGINX-style access logs into a trustworthy HTTP 404 count. The utility saves that count plus a current UTC timestamp in a YAML configuration.

What You'll Build

You'll run log_analyzer.py to see a clean terminal report backed by the same three-response count plus current UTC timestamp in server_config.yaml.

By the end of this project, you'll have:

  • A regular expression demo that extracts every IP-shaped value from a block of text.
  • A field-aware log parser that counts three real 404 statuses without being fooled by /docs/404-guide.
  • A YAML configuration workflow that safely persists the new count plus timestamp from a Python dictionary.
  • Secret Mission: Move the target HTTP status into YAML so the analyzer can inspect 404, 500, or another configured status.

Are there any prerequisites?

This project assumes you completed Python for DevOps Parts 1 through 7. You also need a Linux terminal with Python 3.8 or newer. Step 1 sets up PyYAML 6.0.3.

Before We Start

This checkpoint locks in the DevOps utility you are about to build. Your analyzer will count real HTTP 404 status entries before recording the count plus a current UTC timestamp in YAML.

Set Up and Find Regex Patterns

Log files hide useful signals inside long lines of text. Simple searches reveal those signals quickly.

This step prepares an isolated Python workspace with PyYAML 6.0.3. You will use a regular expression to test how much confidence a bare number match deserves.

In this step, get ready to:
  • Verify that Python is version 3.8 or newer.
  • Create an isolated environment with PyYAML 6.0.3.
  • Compare the first regex match with every matching value.
Prepare the project directory and Python environment

A virtual environment keeps this project's packages inside its own folder. Activating it directs Python package commands to .venv.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create the python-devops-part-8 project directory by running these commands:
mkdir python-devops-part-8
cd python-devops-part-8
  • Confirm that the terminal is inside the new directory by running:
pwd

You should see a path ending in /Desktop/python-devops-part-8.

  • Check the installed Python version by running:
python3 --version

The version line must show 3.8 or higher.

Need a Python Upgrade?

A version below 3.8 cannot install the pinned PyYAML release. Follow the supported Python upgrade process for your Linux distribution before continuing.

Use the official Python downloads page to find the current Linux guidance.

If the upgraded version still does not appear, help me check which Python installation my terminal is using.

  • Create and activate the local environment by running these commands:
python3 -m venv .venv
source .venv/bin/activate

What Do These Commands Do?

The first command creates the isolated .venv directory. The second command activates that environment in your current terminal.

Activation places the environment's Python tools on your shell path. Package installations now stay with this project.

🙋‍♀️ Virtual Environment Failed?

Your Linux installation may be missing its Python virtual-environment package. Install that package through your distribution's supported package manager before rerunning the environment command.

If activation still fails, help me troubleshoot my Python virtual environment.

Install PyYAML and build the first regex

The pinned dependency prepares your environment for the YAML work later in the project. The first script focuses on finding IP-shaped text with Python's regex module.

  • Install the pinned PyYAML dependency by running:
python3 -m pip install 'PyYAML==6.0.3'
  • Verify the installed package version by running:
python3 -m pip freeze

You should see PyYAML==6.0.3 in the package list.

Your isolated environment is ready. PyYAML 6.0.3 now belongs to this project.

PyYAML Installation Failed?

Confirm that the environment is still active before retrying the installation. A network or package-index problem can also interrupt the download.

For help with the output, help me diagnose the PyYAML installation failure.

  • Use your installed Linux text editor to create regex_basics.py inside python-devops-part-8.
  • Paste this code into regex_basics.py:
import re

text = "web-01 10.0.0.5 failed; web-02 10.0.0.8 healthy"
match = re.search(r"\d{1,3}(?:\.\d{1,3}){3}", text)
addresses = re.findall(r"\d{1,3}(?:\.\d{1,3}){3}", text)
print(match.group())
print(addresses)

How Does the Regex Demo Work?

  • The re import provides Python's regular-expression functions.
  • The text variable holds two server names with two IP-shaped values.
  • re.search returns the first match. Calling match.group() retrieves its matched text.
  • re.findall returns every non-overlapping match as a list.
  • The pattern accepts one to three digits in each section. It finds IP-shaped text without validating the numeric range of each section.
  • Save regex_basics.py.
  • Run the regex demo by running:
python3 regex_basics.py

Your terminal should print 10.0.0.5 on the first line. The second line should print ['10.0.0.5', '10.0.0.8'].

Seeing a Match Error?

Confirm that the text line contains both addresses. Check that every backslash and brace in the pattern matches the example.

Save regex_basics.py before running it again.

If the error remains, help me compare my regex demo with the working version.

Probe the limits of literal matching

Small input changes reveal which part of a pattern controls the result. You will test an address change before checking a three-digit search against log-shaped text.

  • In regex_basics.py replace 10.0.0.5 with 172.16.0.10.
  • Save regex_basics.py.
  • Run the changed script by running:
python3 regex_basics.py

The first output line should now be 172.16.0.10.

  • Replace 172.16.0.10 with 10.0.0.5 to restore the original text.
  • Save regex_basics.py.
  • Confirm the original first match has returned by running:
python3 regex_basics.py

The first line should be 10.0.0.5 again.

  • Start Python's interactive prompt by running:
python3

The terminal is now waiting for individual Python expressions. This gives you a quick place to test patterns without changing regex_basics.py.

  • Enter re.findall(r"\b\d{3}\b", "200 404 503") at the interactive prompt.

You should see ['200', '404', '503']. The word boundaries keep each three-digit value separate.

Before the next search, do you think a successful status field protects the request from matching 404?

  • Enter re.findall(r"404", '"GET /docs/404-guide" 200') at the interactive prompt.

You will see ['404']. This is the intended false positive.

Literal Matching Loses Field Context

The search found 404 inside the request path. The pattern had no rule describing where the HTTP status belongs.

The request actually finished with status 200. A bare match cannot distinguish those two fields.

  • Exit the interactive prompt by pressing Ctrl+D.

Avoid Two Regex Traps

  • Write backslash-heavy patterns as raw strings such as r"\d+". Python then passes the backslashes to the regex engine.
  • Check if match: before calling .group() when input may not contain a match.

✔️ Awesome, I've got everything!

Great. Double check that regex_basics.py is saved with the original two addresses.

ⓧ I'd like to double check the full code

Compare your complete regex_basics.py file with this version:

import re

text = "web-01 10.0.0.5 failed; web-02 10.0.0.8 healthy"
match = re.search(r"\d{1,3}(?:\.\d{1,3}){3}", text)
addresses = re.findall(r"\d{1,3}(?:\.\d{1,3}){3}", text)
print(match.group())
print(addresses)

Before the final run, which two lines should return now that the original address is restored?

  • Verify the completed script by running:
python3 regex_basics.py

You should see 10.0.0.5 followed by ['10.0.0.5', '10.0.0.8'].

Your regex workspace is ready. Next, you will use capturing groups to extract a timestamp and status from their exact log fields.

Parse Structured Log Fields

Your first regular expression pulled recognizable IP-shaped values from a loose text string. Access logs pack several fields into each line. A value's position gives it meaning.

The earlier literal search counted 404 inside a successful request path. This step uses capturing groups to read the timestamp and HTTP status from their positions in an NGINX-style log line.

In this step, get ready to:
  • Create a mock access log with successful requests and real error responses.
  • Extract a timestamp and status field with capturing groups.
  • Compare literal text matching with field-aware log parsing.
Create the mock access log

An access log records one request per line. This sample includes three real 404 statuses. A successful request also contains /docs/404-guide in its path.

  • Use your Linux text editor to create server_access.log inside the current project directory.
  • Paste the following five log lines into server_access.log:
10.0.0.5 - - [11/Sep/2026:10:00:00 +0000] "GET / HTTP/1.1" 200 612 "-" "training-client"
10.0.0.8 - - [11/Sep/2026:10:02:00 +0000] "GET /missing HTTP/1.1" 404 128 "-" "training-client"
10.0.0.9 - - [11/Sep/2026:10:03:00 +0000] "GET /old-page HTTP/1.1" 404 128 "-" "training-client"
10.0.0.10 - - [11/Sep/2026:10:04:00 +0000] "GET /docs/404-guide HTTP/1.1" 200 900 "-" "training-client"
10.0.0.11 - - [11/Sep/2026:10:05:00 +0000] "GET /api/item HTTP/1.1" 404 128 "-" "training-client"

What does this log contain?

  • The opening value on each line is the client address.
  • The bracketed value records the request timestamp.
  • The three-digit value after the quoted request is the response status.
  • The /docs/404-guide request succeeds with status 200.
  • Save server_access.log.
  • Check your editor's file list to confirm that server_access.log appears in the current project directory.

Cannot find the log file?

Confirm that the filename is exactly server_access.log. Check that your editor saved it beside regex_basics.py.

Ask for help to find the saved log file.

✔️ Awesome, I've got everything!

Great. Your five-line access log is saved beside regex_basics.py.

ⓧ I'd like to double check the full code

Compare your complete server_access.log file with this reference:

10.0.0.5 - - [11/Sep/2026:10:00:00 +0000] "GET / HTTP/1.1" 200 612 "-" "training-client"
10.0.0.8 - - [11/Sep/2026:10:02:00 +0000] "GET /missing HTTP/1.1" 404 128 "-" "training-client"
10.0.0.9 - - [11/Sep/2026:10:03:00 +0000] "GET /old-page HTTP/1.1" 404 128 "-" "training-client"
10.0.0.10 - - [11/Sep/2026:10:04:00 +0000] "GET /docs/404-guide HTTP/1.1" 200 900 "-" "training-client"
10.0.0.11 - - [11/Sep/2026:10:05:00 +0000] "GET /api/item HTTP/1.1" 404 128 "-" "training-client"

What should match?

Your file should contain these five lines in this order. The fourth line must keep /docs/404-guide with status 200.

Extract timestamp and status groups

Parentheses create extraction slots around the fields you need. The pattern r'\[([^\]]+)\] "[^"]+" (\d{3}) ' captures the bracketed timestamp first. It captures the three-digit status second.

  • Create regex_advanced.py beside server_access.log with your Linux text editor.
  • Paste the following script into regex_advanced.py:
import re

line = '10.0.0.8 - - [11/Sep/2026:10:02:00 +0000] "GET /missing HTTP/1.1" 404 128 "-" "training-client"'
pattern = r'\[([^\]]+)\] "[^"]+" (\d{3}) '
match = re.search(pattern, line)
timestamp, status = match.groups()
print(timestamp)
print(status)

What does this code do?

  • The first group captures the characters inside the timestamp brackets.
  • The second group captures the three-digit status after the quoted request.
  • The match.groups() call returns both captured values as a tuple.
  • The names timestamp and status receive the captured values in pattern order.
  • Save regex_advanced.py.

Before you run the script, which two fields do you expect the groups to return?

  • Run the saved parser in the active virtual environment with this command:
python3 regex_advanced.py

What does this run prove?

Python applies the pattern to the sample line. The two print calls expose each captured field separately.

You should see 11/Sep/2026:10:02:00 +0000 on the first line. You should see 404 on the second line.

Parser stopped before printing?

Compare the quotes and brackets in line with the reference code. A missing character can prevent re.search from returning a match.

Ask for help to debug the capturing groups.

Good work. Your parser now separates two meaningful fields from one dense log line.

Compare literal and field-aware matching

Literal matching sees every occurrence of the characters 404. Field-aware parsing checks where those characters appear before treating them as a response status.

Before you search the log, how many literal 404 matches do you expect to find?

  • Switch to server_access.log in your text editor.
  • Search the file for 404 with your editor's text search.
  • Count the highlighted matches.

You should find four literal occurrences. One belongs to /docs/404-guide on a request whose status is 200.

  • Inspect the three lines where 404 appears after the closing request quote.
  • Inspect the status field on the /docs/404-guide line.

Only three occurrences occupy the status field. Structured parsing removes the misleading fourth match.

Seeing a different count?

Confirm that server_access.log contains all five reference lines. Check that the successful guide request still contains /docs/404-guide.

Ask for help to compare the literal and status counts.

A quick status change shows that the second group follows the field's value. You can test that behavior without changing the pattern.

  • In regex_advanced.py, change the sample line's status field from 404 to 503.
  • Save regex_advanced.py.

Before you rerun the script, what should change in the two-line output?

  • Run the temporary status test with this command:
python3 regex_advanced.py

What changed?

The command runs the saved version of the same parser. Only the sample status value has changed.

The timestamp stays the same. The second output line now shows 503.

  • Change the sample status field from 503 back to 404.
  • Save regex_advanced.py.

Before you run the restored script, which status should appear on the second line?

  • Confirm the restored status by running:
python3 regex_advanced.py

What does this check confirm?

The parser reads the restored sample line. The second captured group returns to 404.

You should see the original timestamp followed by 404.

  • Temporarily replace the final two print lines with one print of match.groups().
  • Save regex_advanced.py.

Before you run this version, what shape do you expect the two captured values to take?

  • Display both groups together by running:
python3 regex_advanced.py

Why use groups together?

The groups() method returns captured subgroups in pattern order. A tuple makes that order visible in one output value.

You should see ('11/Sep/2026:10:02:00 +0000', '404').

  • Restore regex_advanced.py so it matches the full-file reference below:

✔️ Awesome, I've got everything!

Your script again prints the timestamp and status on separate lines. Save the restored file before the final run.

ⓧ I'd like to double check the full code

Compare your complete regex_advanced.py file with this reference:

import re

line = '10.0.0.8 - - [11/Sep/2026:10:02:00 +0000] "GET /missing HTTP/1.1" 404 128 "-" "training-client"'
pattern = r'\[([^\]]+)\] "[^"]+" (\d{3}) '
match = re.search(pattern, line)
timestamp, status = match.groups()
print(timestamp)
print(status)

What should match?

The final file unpacks match.groups() into timestamp and status. It prints each value on its own line.

  • Save the restored regex_advanced.py file.

Before the final run, do you expect the parser to print literal matches or the two captured fields?

  • Complete the final verification by running:
python3 regex_advanced.py

What does the final run prove?

The successful run confirms that the saved pattern matches the expected log structure. It also confirms that both capturing groups are available separately.

You should see 11/Sep/2026:10:02:00 +0000 followed by 404 on the next line.

Final output does not match?

Confirm that the sample line ends with status 404 in the expected field. Compare the final three lines with the full-file reference.

Ask for help to fix the final parser output.

Strong work. Your parser can now extract fields according to the log structure. Next, you'll load readable configuration data into Python.

Load YAML Configuration

Your field-aware regex can separate a real status code from misleading text in a URL. The next challenge is keeping related server settings together without brittle line-by-line text edits.

YAML organizes related values as mappings that become Python dictionaries. In this step, you will use the PyYAML library from earlier to load a configuration file safely.

In this step, get ready to:
  • Create a YAML configuration with nested server settings.
  • Load the YAML mapping into a Python dictionary.
  • Read nested configuration values with dictionary keys.
Structure nested YAML mappings

A YAML mapping stores key-value pairs. Indentation groups related keys beneath a parent such as server or monitoring.

  • Use your Linux text editor to create server_config.yaml beside server_access.log.
  • Add the configuration by copying this YAML into server_config.yaml:
server:
  name: web-01
  environment: staging
monitoring:
  last_404_count: 0
  last_checked_utc: null

What Does This Configuration Store?

  • The server mapping groups the machine name with its deployment environment.
  • The monitoring mapping holds values that the analyzer updates later.
  • The null value represents a timestamp that has not been recorded yet.
  • Save server_config.yaml.
  • Check your text editor's file list for server_config.yaml beside server_access.log.

You should see both files in the same project directory.

YAML File Looks Misaligned?

Use spaces before each nested key. Align name with environment beneath server.

Align both monitoring values beneath monitoring.

help me check my YAML indentation

Load the mapping with PyYAML

The yaml.safe_load function reads basic YAML tags into Python values. The top-level mappings become dictionary keys that your script can access directly.

  • Create yaml_reader.py beside server_config.yaml using your Linux text editor.
  • Add the reader by copying this code into yaml_reader.py:
import yaml

with open("server_config.yaml", "r", encoding="utf-8") as config_file:
    config = yaml.safe_load(config_file)
print(config)
print(config["server"]["name"])

What Does This Code Do?

  • The import yaml line makes PyYAML available to the script.
  • The open call reads server_config.yaml as UTF-8 text.
  • The yaml.safe_load call converts the YAML mapping into the config dictionary.
  • The final lookup reads name from the nested server dictionary.
  • Save yaml_reader.py.

Before you run the reader, predict what its final output line will contain.

  • Load the YAML configuration by running this command:
python3 yaml_reader.py

What Should You See?

The first line shows a Python dictionary containing the server and monitoring mappings. The final line shows web-01.

🙋‍♀️ Reader Did Not Print web-01?

Confirm that yaml_reader.py and server_config.yaml are in the same directory. Check that both files are saved.

If Python reports a YAML parsing problem, replace tabs with spaces. Check the indentation beneath each top-level mapping.

help me debug my YAML reader

Test nested values and restore the configuration

YAML scalar values become matching Python types when PyYAML loads them. These short experiments show how strings and integers behave inside the same nested mapping.

  • Add port: 8080 directly below environment: staging in server_config.yaml.
  • Save server_config.yaml.
  • Reload the expanded mapping by running:
python3 yaml_reader.py

Check the Expanded Mapping

You should see 'port': 8080 inside the first line's server dictionary. The final line still shows web-01.

  • Replace name with port in the final lookup inside yaml_reader.py.
  • Save yaml_reader.py.
  • Read the integer value by running:
python3 yaml_reader.py

Check the Integer Value

You should see 8080 on the final line. PyYAML loads that unquoted YAML value as a Python integer.

  • Replace port with environment in the final lookup.
  • Save yaml_reader.py.
  • Read the environment value by running:
python3 yaml_reader.py

Check the Environment Value

You should see staging on the final line. The YAML text value loads as a Python string.

  • Replace the final print expression with type(config["monitoring"]["last_404_count"]).__name__.
  • Save yaml_reader.py.
  • Inspect the loaded monitoring value by running:
python3 yaml_reader.py

Check the Loaded Type

You should see int on the final line. The YAML value 0 becomes a Python integer.

  • Remove the temporary port: 8080 line from server_config.yaml.
  • Save server_config.yaml.
  • Confirm the temporary key is gone by running:
python3 yaml_reader.py

Check the Restored Mapping

The first line should no longer contain port. The final line still shows int because the reader is inspecting last_404_count.

  • Restore the final lookup to config["server"]["name"] in yaml_reader.py.
  • Save yaml_reader.py.

Before the final run, predict the two things the terminal will show.

  • Verify the restored reader by running:
python3 yaml_reader.py

Final Verification

You should see the complete configuration as a Python dictionary. The next line should show web-01.

That confirms server_config.yaml loads successfully through yaml.safe_load.

🙋‍♀️ Seeing a Missing Key Problem?

Check that the final lookup uses server first. Confirm that the nested key is name.

Compare the spelling with the keys in server_config.yaml because dictionary lookups are case-sensitive.

help me fix my nested YAML lookup

✔️ Awesome, I've got everything!

Your YAML configuration is restored. Double check that you saved server_config.yaml.

ⓧ I'd like to double check the full code

Compare your complete server_config.yaml file with this reference:

server:
  name: web-01
  environment: staging
monitoring:
  last_404_count: 0
  last_checked_utc: null

How to Compare This File

Check every key and value. Confirm that each nested line uses two spaces of indentation.

✔️ Awesome, I've got everything!

Your reader loads the configuration successfully. Double check that you saved yaml_reader.py.

ⓧ I'd like to double check the full code

Compare your complete yaml_reader.py file with this reference:

import yaml

with open("server_config.yaml", "r", encoding="utf-8") as config_file:
    config = yaml.safe_load(config_file)
print(config)
print(config["server"]["name"])

How to Compare This File

Check the filename passed to open. Confirm that the final lookup reads the nested server name.

Your configuration now moves cleanly from readable YAML into a Python dictionary. Next up, you will update that dictionary and save the new state back to the YAML file.

Update and Save YAML

Your YAML reader now turns server_config.yaml into a Python dictionary. The monitoring workflow also needs to preserve new values after the script finishes.

This step uses PyYAML to update one nested value. The yaml.safe_dump function then serializes the dictionary as readable block-style YAML.

In this step, get ready to:
  • Create a writer that loads the existing configuration.
  • Update the saved monitoring count.
  • Persist readable YAML with the original key order.
Create the YAML writer

Writing over a configuration file can feel risky. This script loads the complete mapping before it opens the same file for output.

  • In the text editor from earlier, create yaml_writer.py inside the current Linux project directory.
  • Paste this code into yaml_writer.py:
import yaml

with open("server_config.yaml", "r", encoding="utf-8") as config_file:
    config = yaml.safe_load(config_file)
config["monitoring"]["last_404_count"] = 3
with open("server_config.yaml", "w", encoding="utf-8") as config_file:
    yaml.safe_dump(config, config_file, sort_keys=False)
print("Saved last_404_count: 3")

What Does This Code Do?

  • The first with open block reads the complete configuration into config.
  • The assignment changes last_404_count inside the nested monitoring dictionary.
  • The second with open block uses "w" mode to replace the saved file.
  • The yaml.safe_dump call writes basic YAML values back to the file.
  • The sort_keys=False option preserves the dictionary's insertion order.
  • Save yaml_writer.py.

Before you run the writer, consider whether the saved count will stay at 0 or change to 3.

  • Run the YAML writer by entering this command:
python3 yaml_writer.py

What Happens When It Runs?

The terminal prints Saved last_404_count: 3. The script also rewrites server_config.yaml with the updated dictionary.

  • Switch back to server_config.yaml in the text editor.

Under monitoring, you should see last_404_count: 3. That is the persistence loop working: your Python dictionary has become saved configuration.

Saved Count Still Shows Zero?

  • Confirm your current Linux project directory contains yaml_writer.py.
  • Check that the second open call uses "w" mode.
  • Save yaml_writer.py before rerunning the command.

help me debug why my YAML writer did not save the updated count.

Test alternate values and key order

Temporary edits make serialization behavior visible. You will restore the project version after each behavior has been confirmed.

  • Return to yaml_writer.py from earlier.
  • Replace both 3 values with 7 for the first practice run.
  • Save yaml_writer.py.
  • Run the updated writer by entering this command:
python3 yaml_writer.py

What Changed in the First Practice Run?

The terminal now reports a saved count of 7. The monitoring mapping in server_config.yaml also contains last_404_count: 7.

  • Switch back to server_config.yaml.

You should see last_404_count: 7 under monitoring.

  • Return to yaml_writer.py.
  • Add config["monitoring"]["healthy"] = True below the count assignment.
  • Save yaml_writer.py.
  • Run the writer with the new Boolean value by entering this command:
python3 yaml_writer.py

How Does YAML Store the Boolean?

Python's True value is serialized as healthy: true. YAML keeps the value as a Boolean instead of turning it into text.

  • Switch back to server_config.yaml.

You should see healthy: true inside the monitoring mapping.

  • Return to yaml_writer.py.
  • Change sort_keys=False to sort_keys=True.
  • Save yaml_writer.py.
  • Run the writer with key sorting enabled by entering this command:
python3 yaml_writer.py

What Does Key Sorting Change?

The saved mappings now appear in alphabetical key order. This comparison shows why the project uses sort_keys=False for a more intentional configuration layout.

  • Switch back to server_config.yaml.

You should see the keys arranged alphabetically. The saved values remain unchanged.

The practice runs proved that dictionary values control the saved YAML. Restore the project script before moving on.

  • Return to yaml_writer.py.
  • Restore the project version by replacing the file contents with this code:
import yaml

with open("server_config.yaml", "r", encoding="utf-8") as config_file:
    config = yaml.safe_load(config_file)
config["monitoring"]["last_404_count"] = 3
with open("server_config.yaml", "w", encoding="utf-8") as config_file:
    yaml.safe_dump(config, config_file, sort_keys=False)
print("Saved last_404_count: 3")

What Has Been Restored?

The writer once again saves last_404_count as 3. It also uses sort_keys=False without the temporary healthy assignment.

  • Save yaml_writer.py.

Before the final run, consider which count the restored script should print.

  • Verify the restored writer by entering this command:
python3 yaml_writer.py

What Should You See?

The terminal should print Saved last_404_count: 3. The monitoring mapping should contain last_404_count: 3.

✔️ Awesome, I've got everything!

Great. Double check that you saved yaml_writer.py with the restored count and key-order setting.

ⓧ I'd like to double check the full code

Compare your complete yaml_writer.py file with this project version.

import yaml

with open("server_config.yaml", "r", encoding="utf-8") as config_file:
    config = yaml.safe_load(config_file)
config["monitoring"]["last_404_count"] = 3
with open("server_config.yaml", "w", encoding="utf-8") as config_file:
    yaml.safe_dump(config, config_file, sort_keys=False)
print("Saved last_404_count: 3")
Restore the clean configuration snapshot

A reusable sample configuration starts with no recorded monitoring result. Resetting the sample data keeps the next analyzer run easy to verify.

  • Return to server_config.yaml from earlier.
  • Restore the clean sample configuration by replacing its contents with this YAML:
server:
  name: web-01
  environment: staging
monitoring:
  last_404_count: 0
  last_checked_utc: null

Why Reset These Values?

The writer script remains ready to save a count of 3. The configuration now provides a clean starting point with last_checked_utc: null.

  • Save server_config.yaml.
  • Confirm the monitoring mapping shows last_404_count: 0.

You now have the restored configuration plus a working writer that can persist nested updates.

Avoid Two Save Mistakes

  • Use "w" mode for the output pass because read mode cannot persist dictionary changes.
  • Keep this exercise's configuration comment-free because a PyYAML load-and-save cycle does not preserve original comments or spacing.

help me check my YAML loading and saving workflow.

Your configuration can now carry automation results between runs. Next, you will connect this save workflow to the access-log parser.

Build the Log Analyzer

Your earlier PyYAML workflow can already persist a nested YAML monitoring record. The next challenge is giving that record a trustworthy count from the noisy access log.

The earlier literal search found four occurrences of 404. One occurrence belongs to a successful request path, so a plain text count overstates the problem.

In this step, you will build a Python analyzer that extracts the status field from each line. It will update the configuration with three real errors plus a current UTC timestamp.

In this step, get ready to:
  • Count status-field matches with a reusable function.
  • Persist the count with a current UTC timestamp.
  • Print a summary with controlled input-error messages.
Build the field-aware counter

A field-aware regular expression captures the HTTP status from its documented position. This keeps the successful /docs/404-guide request out of the error count.

  • Create log_analyzer.py in the current project directory using the text editor from earlier.
  • Add the imports plus the status-counting function by pasting this code:
import re
from datetime import datetime, timezone

import yaml

LOG_FILE = "server_access.log"
CONFIG_FILE = "server_config.yaml"
TARGET_STATUS = "404"


def count_status(log_text, target_status):
    count = 0
    pattern = r'\[[^\]]+\] "[^"]+" (\d{3}) '
    for line in log_text.splitlines():
        match = re.search(pattern, line)
        if match and match.group(1) == target_status:
            count += 1
    return count

How Does Field-Aware Counting Work?

  • The constants identify the access log, the configuration file, plus the target status.
  • The pattern captures the three-digit field after the quoted request.
  • The loop checks each log line separately.
  • The comparison increments the count only when the captured status equals 404.
  • The successful documentation request has a captured status of 200.
  • Save log_analyzer.py.
  • Confirm your editor lists log_analyzer.py beside server_access.log.

Cannot Find the New File?

Check that your editor saved the file in the same project directory as server_access.log. Confirm that the filename ends with .py.

Help me locate the analyzer file.

Connect the input files to YAML

The main workflow completes both input reads before opening the configuration for writing. A missing file or invalid YAML therefore stops the update before saved state can be replaced.

  • Append the main workflow below count_status() by pasting this code:
def main():
    try:
        with open(LOG_FILE, "r", encoding="utf-8") as log_file:
            log_text = log_file.read()
        with open(CONFIG_FILE, "r", encoding="utf-8") as config_file:
            config = yaml.safe_load(config_file) or {}
    except FileNotFoundError as error:
        print(f"ERROR: Missing file: {error.filename}")
        return
    except yaml.YAMLError as error:
        print(f"ERROR: Invalid YAML: {error}")
        return

    error_count = count_status(log_text, TARGET_STATUS)
    checked_at = datetime.now(timezone.utc).isoformat(timespec="seconds")
    monitoring = config.setdefault("monitoring", {})
    monitoring["last_404_count"] = error_count
    monitoring["last_checked_utc"] = checked_at

    with open(CONFIG_FILE, "w", encoding="utf-8") as config_file:
        yaml.safe_dump(config, config_file, sort_keys=False)


if __name__ == "__main__":
    main()

How Does the Workflow Protect the Configuration?

  • The try block reads the complete access log before loading the configuration.
  • The exception handlers print controlled messages for missing files or invalid YAML.
  • Each exception handler returns before the write operation.
  • The aware UTC timestamp records an explicit timezone offset.
  • The final dump preserves the dictionary's insertion order with sort_keys=False.
  • Save log_analyzer.py.

Before you run the workflow, predict whether the saved count will be three or four.

  • Run the analyzer by entering this command:
python3 log_analyzer.py

What Did This Run Change?

The call to main() reads both files. It stores the field-aware count with the current UTC time.

The write happens only after both inputs have loaded successfully. Your configuration now contains evidence from this run.

  • Return to server_config.yaml in your text editor.

You should see last_404_count: 3 under monitoring. The last_checked_utc value should end in +00:00.

That is the core pipeline working. Your analyzer has converted noisy log text into a trustworthy configuration update.

Configuration Did Not Update?

Confirm that log_analyzer.py, server_access.log, plus server_config.yaml are in the same project directory.

Check the terminal for a missing-file or invalid-YAML message. That message identifies the input that stopped the update.

Help me debug the configuration update.

The saved configuration makes the result persistent. A terminal report makes the same result visible during routine operations.

  • In log_analyzer.py, find the block containing yaml.safe_dump(config, config_file, sort_keys=False).
  • Insert the report after that block by pasting this code:
    print("Log Analyzer Summary")
    print("--------------------")
    print(f"Status code: {TARGET_STATUS}")
    print(f"Occurrences: {error_count}")
    print(f"Config updated: {CONFIG_FILE}")
    print(f"Timestamp: {checked_at}")

What Does the Report Show?

The report displays the target status plus the matching count. It also names the updated configuration file.

The final line displays the same UTC timestamp stored in YAML. The terminal output now mirrors the persisted monitoring record.

  • Save log_analyzer.py.

Before you run the completed analyzer, predict whether the successful URL containing 404 will change the status-field result.

  • Run the completed analyzer by entering this command:
python3 log_analyzer.py

What Should the Summary Confirm?

You should see Status code: 404 followed by Occurrences: 3.

The remaining lines should name server_config.yaml plus a timestamp ending in +00:00.

  • Return to server_config.yaml in your text editor.

You should see last_404_count: 3 with a refreshed last_checked_utc value. The terminal report now matches the persisted state.

Summary Missing or Incomplete?

Check that all six print lines remain indented inside main(). They should align with the configuration write block.

Confirm that the final if __name__ == "__main__": block remains at the left edge of the file.

Help me fix the analyzer summary.

✔️ Awesome, I've got everything!

Great. Double check that log_analyzer.py is saved before testing its defensive paths.

ⓧ I'd like to double check the full code

Compare your complete log_analyzer.py file with this reference:

import re
from datetime import datetime, timezone

import yaml

LOG_FILE = "server_access.log"
CONFIG_FILE = "server_config.yaml"
TARGET_STATUS = "404"


def count_status(log_text, target_status):
    count = 0
    pattern = r'\[[^\]]+\] "[^"]+" (\d{3}) '
    for line in log_text.splitlines():
        match = re.search(pattern, line)
        if match and match.group(1) == target_status:
            count += 1
    return count


def main():
    try:
        with open(LOG_FILE, "r", encoding="utf-8") as log_file:
            log_text = log_file.read()
        with open(CONFIG_FILE, "r", encoding="utf-8") as config_file:
            config = yaml.safe_load(config_file) or {}
    except FileNotFoundError as error:
        print(f"ERROR: Missing file: {error.filename}")
        return
    except yaml.YAMLError as error:
        print(f"ERROR: Invalid YAML: {error}")
        return

    error_count = count_status(log_text, TARGET_STATUS)
    checked_at = datetime.now(timezone.utc).isoformat(timespec="seconds")
    monitoring = config.setdefault("monitoring", {})
    monitoring["last_404_count"] = error_count
    monitoring["last_checked_utc"] = checked_at

    with open(CONFIG_FILE, "w", encoding="utf-8") as config_file:
        yaml.safe_dump(config, config_file, sort_keys=False)

    print("Log Analyzer Summary")
    print("--------------------")
    print(f"Status code: {TARGET_STATUS}")
    print(f"Occurrences: {error_count}")
    print(f"Config updated: {CONFIG_FILE}")
    print(f"Timestamp: {checked_at}")


if __name__ == "__main__":
    main()
Test the analyzer's defenses

A useful operations tool should respond to changing logs. It should also stop cleanly when an input cannot be trusted.

  • In server_access.log, duplicate the line containing GET /api/item HTTP/1.1.
  • Save server_access.log.

Before you rerun the analyzer, predict how one extra status-field match will change the count.

  • Rerun the analyzer by entering this command:
python3 log_analyzer.py

Did the Counter Respond?

You should now see Occurrences: 4. The counter responds to the captured status field from the added line.

  • Remove the duplicated line from server_access.log.
  • Save server_access.log.

The Log Is Restored

The access log now contains its original five requests. The next check temporarily removes that input from the analyzer's reach.

  • Use your text editor's file list to rename server_access.log to server_access.log.bak.

Before you run the analyzer, predict whether the missing input will produce a traceback or a controlled message.

  • Test the missing-file handler by entering this command:
python3 log_analyzer.py

How Does the Missing-File Check Behave?

The output should begin with ERROR: Missing file: without a traceback. The early return prevents a configuration write.

  • Rename server_access.log.bak back to server_access.log.

The Input File Is Available Again

The analyzer can read the access log again. The next check confirms that malformed configuration data is also rejected safely.

  • In server_config.yaml, replace the spaces before last_404_count with a tab.
  • Save server_config.yaml.

Before you run the analyzer, predict which exception handler will protect the malformed configuration.

  • Test the invalid-YAML handler by entering this command:
python3 log_analyzer.py

How Does the YAML Check Behave?

The output should begin with ERROR: Invalid YAML: without a traceback. Parsing must succeed before the write stage can begin.

  • Restore two spaces before last_404_count in server_config.yaml.
  • Save server_config.yaml.

Before the final run, predict which count should return after restoring both input files.

  • Verify the restored analyzer by entering this command:
python3 log_analyzer.py

What Does the Final Run Prove?

You should see Occurrences: 3 again. The final timestamp should end in +00:00.

Your analyzer has recovered from both controlled failure tests. It is reading the original five-line access log again.

  • Return to server_config.yaml to verify the final persisted state.

You should see last_404_count: 3 with a current last_checked_utc value ending in +00:00.

Final Run Still Failing?

Confirm that the access log has its original filename. Check that both monitoring keys use spaces for indentation.

Compare log_analyzer.py with the full-code tab. Pay close attention to indentation inside try, except, plus main().

Help me debug the final analyzer run.

Secret mission

Make the Target Status Configurable

Turn one hard-coded 404 check into a reusable analyzer controlled by YAML. You will switch between 404 and 500 without changing the regex logic.

Clean Up Your Resources

Clean Up Your Resources

Everything you built stays on your Linux machine with no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • A local Python virtual environment in .venv containing PyYAML version 6.0.3.
  • Five Python scripts: regex_basics.py, regex_advanced.py, yaml_reader.py, yaml_writer.py, and log_analyzer.py.
  • The local log file server_access.log plus the YAML configuration file server_config.yaml.

Keep everything running

No action is needed. Choose this option if you want to reuse the analyzer in Part 9.

  • Keep .venv so the pinned PyYAML dependency remains available.
  • Keep the five scripts plus server_access.log and server_config.yaml for future experiments.

Pause - I'll come back to this later

Exit the active virtual environment to return your shell to its usual Python installation. Your project files stay in the current directory.

  • Deactivate the virtual environment by running this command:
deactivate

What does this command do?

Your pause is complete. The shell has left .venv.

Every project file and installed package remains available on disk.

  • Reactivate the environment later from this project directory by running this command:
source .venv/bin/activate

How do you resume?

This activates .venv again. Your shell can then use the pinned PyYAML installation.

Delete - I don't want to use this again

Deleting these local resources is permanent, so save anything you want to keep first. The commands below target only the named project artifacts in your current directory.

  • Confirm that the current directory contains regex_basics.py and server_config.yaml.
  • Exit the active environment and remove every listed resource by running these commands:
deactivate
rm -rf .venv regex_basics.py regex_advanced.py yaml_reader.py yaml_writer.py log_analyzer.py server_access.log server_config.yaml

What do these commands remove?

The first line exits the active virtual environment. The second line removes the named virtual environment from the current directory.

The same removal command deletes the five scripts, the access log, and the configuration file.

That completes your clean reset. The local project resources are gone from this directory.

Nice Work!

Nice Work!

You made it! You completed Part 8 of Python for DevOps with a Linux-based utility that turns noisy access logs into a reusable YAML monitoring update.

You've learned how to:

  • Use regular expressions with re.search plus re.findall to find log patterns. Apply capturing groups to isolate the HTTP status field. This excludes 404 inside /docs/404-guide from the error count.
  • Use PyYAML to load server_config.yaml as a Python dictionary with yaml.safe_load. Save nested changes as readable YAML with yaml.safe_dump.
  • Build a defensive log analyzer that counts configured status entries. Write the count to server_config.yaml. Record a current UTC timestamp beside it. Show clean messages for missing files. Show clean messages for malformed YAML.
  • Secret Mission: Make the analyzer reusable by reading target_status from YAML. Store each result in last_error_count. Confirm 404 finds three matches while 500 finds zero.

Ready to quiz yourself?