Build a Log Analytics Pipeline

Parse raw web server logs into searchable data with the ELK stack.

Introduction

30 Second Summary

Every web server generates thousands of log lines per hour. When something breaks, the team that can instantly search and visualize those logs finds the root cause while the team scrolling through raw text files is still guessing.

In this project, you will build a local log analytics pipeline using the ELK stack. You'll use Logstash to transform raw web server logs into structured, queryable fields, then visualize traffic patterns in a Kibana dashboard.

What You'll Build

Picture dropping a raw log file into a folder and, seconds later, opening a dashboard that shows exactly which endpoints are failing, which IPs are most active, and how your traffic breaks down by HTTP status code.

By the end of this project, you'll have:

  • A working pipeline where you drop a raw log file into a folder and watch it appear as structured, searchable data in Elasticsearch within seconds.
  • The power to run queries in Kibana Dev Tools that find all 404 errors, top requesting IPs, or busiest endpoints instantly.
  • A live Kibana dashboard with charts showing HTTP traffic by status code, requests over time, and top URLs.
  • Secret Mission: Enrich your logs with a computed response size field and create an alerting rule that fires when server errors spike.

Are there any prerequisites?

This project requires Docker Desktop installed on your machine with at least 4 GB of memory allocated to Docker. You should also be comfortable with basic command line navigation.

Before We Start

Before we dive into Docker containers and pipeline configurations, let's lock in what you're building and why it matters.

Set Up the ELK Stack with Docker Compose

The ELK stack has three moving parts: Elasticsearch for storing and searching data, Logstash for ingesting and transforming logs, and Kibana for visualizing results.

All three need to be running and connected before you can send any log data through the pipeline. Docker Compose lets you define all three services in a single file and start them with one command.

In this step, get ready to:
  • Create the environment variables and Docker Compose configuration files.
  • Launch all three ELK containers with a single command.
  • Verify Elasticsearch and Kibana are accessible in your browser.
Create the environment variables file

Docker Compose can read variables from a .env file and inject them into your service definitions. This keeps sensitive values and version numbers in one place so you can change them without editing the main compose file.

  • In your code editor, create a new file called .env in the root of your elk-pipeline folder and paste the following content:
ELASTIC_PASSWORD=elastic123
STACK_VERSION=9.4.3
CLUSTER_NAME=elk-lab
ES_PORT=9200
KIBANA_PORT=5601
ES_MEM_LIMIT=2147483648
KB_MEM_LIMIT=1073741824
LS_MEM_LIMIT=1073741824

What do these variables control?

  • ELASTIC_PASSWORD sets the password for the elastic superuser account.
  • STACK_VERSION pins all three services to 9.4.3 so they stay in sync.
  • The memory limits allocate 2 GB to Elasticsearch and 1 GB each to Kibana and Logstash (about 4 GB total).
  • Save .env.
  • Confirm the file is saved correctly by running this command in your terminal (from the elk-pipeline directory):
cat .env

You should see all eight variables printed, starting with ELASTIC_PASSWORD=elastic123 and ending with LS_MEM_LIMIT=1073741824.

File is empty or not found?

  • Make sure your terminal's working directory is elk-pipeline. Run ls -a and confirm you see .env in the listing.
  • The filename must start with a dot. Some editors hide dotfiles by default.
  • help me find my .env file in the elk-pipeline directory
Define the Docker Compose services

The docker-compose.yml file defines each ELK component as a service with its own image, ports, and health checks. You will build this file in three chunks, one service at a time.

  • Create a new file called docker-compose.yml in the root of your elk-pipeline folder and paste the Elasticsearch service definition:
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    container_name: elasticsearch
    volumes:
      - esdata:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=elasticsearch
      - cluster.name=${CLUSTER_NAME}
      - discovery.type=single-node
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=false
      - xpack.license.self_generated.type=trial
    mem_limit: ${ES_MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -u elastic:${ELASTIC_PASSWORD} http://localhost:9200 | grep -q 'cluster_name'",
        ]
      interval: 10s
      timeout: 10s
      retries: 30

What does the Elasticsearch service do?

  • discovery.type=single-node tells Elasticsearch to run alone without looking for other cluster members.
  • xpack.security.enabled=true enables authentication so you need a password to access the data.
  • xpack.security.http.ssl.enabled=false disables HTTPS for local development. This keeps the setup simple since everything stays on your machine.
  • The healthcheck curls Elasticsearch every 10 seconds. Other services wait for it to pass before starting.

This file is not complete yet. You will add Kibana and Logstash next.

  • Add the Kibana service definition directly below the Elasticsearch service (still inside the services: block):
  kibana:
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    container_name: kibana
    volumes:
      - kibanadata:/usr/share/kibana/data
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${ELASTIC_PASSWORD}
      - xpack.security.encryptionKey=something_at_least_32_characters_long_for_security
      - xpack.encryptedSavedObjects.encryptionKey=something_at_least_32_characters_long_for_security
      - xpack.reporting.encryptionKey=something_at_least_32_characters_long_for_security
    mem_limit: ${KB_MEM_LIMIT}
    depends_on:
      elasticsearch:
        condition: service_healthy
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 30

What does the Kibana service do?

  • ELASTICSEARCH_HOSTS points Kibana at the Elasticsearch container using Docker's internal DNS name.
  • depends_on with condition: service_healthy means Kibana waits until Elasticsearch's healthcheck passes before starting.
  • The encryption keys are required for Kibana's saved objects and alerting features to work.
  • Save docker-compose.yml and confirm Docker Compose can read your file so far by running:
docker compose config --services

You should see two service names printed, one per line:

elasticsearch
kibana

Seeing a YAML syntax error?

  • YAML is sensitive to indentation. Each service name (elasticsearch, kibana) must be indented exactly 2 spaces under services:.
  • Check that you do not have tabs mixed with spaces. Most editors show this with a setting called "render whitespace".
  • help me fix a YAML syntax error in my docker-compose.yml
  • Add the Logstash service definition below the Kibana service, then add the volumes: section at the very bottom of the file (outside and below the services: block):
  logstash:
    image: docker.elastic.co/logstash/logstash:${STACK_VERSION}
    container_name: logstash
    volumes:
      - logstashdata:/usr/share/logstash/data
      - ./logstash/pipeline/logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro
      - ./logstash/ingest_data:/usr/share/logstash/ingest_data
    ports:
      - 5044:5044
      - 9600:9600
    environment:
      - LS_JAVA_OPTS=-Xms512m -Xmx512m
      - xpack.monitoring.enabled=false
    mem_limit: ${LS_MEM_LIMIT}
    depends_on:
      elasticsearch:
        condition: service_healthy

volumes:
  esdata:
  kibanadata:
  logstashdata:

What does the Logstash service do?

  • The three volume mounts give Logstash access to its pipeline config file and the folder where you will drop log files for ingestion.
  • LS_JAVA_OPTS=-Xms512m -Xmx512m caps Logstash's Java heap at 512 MB to keep it within the 1 GB container limit.
  • The top-level volumes: section declares the three named volumes Docker creates for persistent data storage.
  • Save docker-compose.yml and verify all three services are recognized:
docker compose config --services

You should now see all three service names:

elasticsearch
kibana
logstash

✔️ Awesome, I've got everything!

Great. Double check you have saved docker-compose.yml and that all three services appear in the config output.

ⓧ I'd like to double check the full code

Your docker-compose.yml should contain the Elasticsearch service block at the top, then the Kibana service block, then the Logstash service block, and finally the volumes: section at the very bottom. Scroll up and confirm each chunk is present in order.

Your .env file should have 8 lines starting with ELASTIC_PASSWORD=elastic123 and ending with LS_MEM_LIMIT=1073741824.

Launch the stack and verify

The Logstash service mounts a file called logstash.conf that does not exist yet. Docker would create a directory in its place if the file is missing, which breaks the container.

  • Create an empty placeholder file so Docker can mount it correctly by running:
touch logstash/pipeline/logstash.conf
  • Start all three containers in the background by running:
docker compose up -d

What does this command do?

The -d flag runs the containers in detached mode so they stay running in the background and give you your terminal back. Docker Compose pulls the images on the first run (this may take a few minutes), creates the named volumes, and starts each service in dependency order.

The first run downloads about 2 GB of container images. Wait until your terminal shows all three containers as "Started" or "Healthy".

What about Logstash?

Logstash will start but exit because its pipeline config file is empty. That is expected. Elasticsearch and Kibana do not depend on Logstash, so they continue running normally. You will give Logstash a real pipeline configuration in a later step.

Before you check the result, what do you expect to see when you curl Elasticsearch at port 9200?

  • Test that Elasticsearch is responding by running:
curl -s -u elastic:elastic123 http://localhost:9200

You should see a JSON response containing "cluster_name" : "elk-lab" and "version" showing 9.4.3.

  • Open your browser and navigate to http://localhost:5601.
  • You should see the Kibana login page.
  • Enter elastic as the username and elastic123 as the password, then click Log in.

You should land on the Kibana home page. This confirms Elasticsearch and Kibana are running and connected.

Kibana not loading or showing a connection error?

  • Kibana waits for Elasticsearch to be fully healthy before starting. Run docker compose ps and check that the elasticsearch container shows "healthy". If it shows "starting", wait another 30 seconds and refresh.
  • If elasticsearch shows "unhealthy", check that Docker Desktop has at least 4 GB of memory allocated in its settings.
  • Run docker compose logs kibana to see Kibana's startup output for specific errors.
  • help me troubleshoot Kibana not connecting to Elasticsearch in Docker Compose

Your ELK stack is running. Next up, you will see why raw logs are a problem before introducing Logstash to solve it.

Ingest Raw Logs into Elasticsearch

Your ELK stack is up and running. Before building a Logstash pipeline to transform your data, you need to understand the exact problem it solves.

In this step, you'll dump raw log lines straight into Elasticsearch and try to query them. You'll quickly discover why unstructured text is a dead end for real analytics.

In this step, get ready to:
  • Create a sample access log file with Apache-format entries.
  • Index raw log lines into Elasticsearch through Kibana Dev Tools.
  • Attempt to query by status code and discover the limitation of unstructured data.
Create the sample access log file

Web servers produce access logs that record every incoming request. Each line contains the client IP, timestamp, HTTP method, URL, status code, and more packed into a single string.

  • In your code editor, create a new file at logstash/ingest_data/access.log inside your project folder.
  • Paste the following log entries into the file:
83.149.9.216 - - [20/Jul/2026:10:05:03 +0000] "GET /index.html HTTP/1.1" 200 4523 "http://example.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
83.149.9.216 - - [20/Jul/2026:10:05:04 +0000] "GET /style.css HTTP/1.1" 200 1234 "http://example.com/index.html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
192.168.1.50 - - [20/Jul/2026:10:05:05 +0000] "GET /api/users HTTP/1.1" 200 8923 "-" "curl/7.68.0"
10.0.0.1 - - [20/Jul/2026:10:05:06 +0000] "POST /api/login HTTP/1.1" 200 452 "http://example.com/login" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)"
172.16.0.100 - - [20/Jul/2026:10:05:07 +0000] "GET /images/logo.png HTTP/1.1" 200 15234 "http://example.com/index.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0)"
83.149.9.216 - - [20/Jul/2026:10:05:08 +0000] "GET /favicon.ico HTTP/1.1" 404 209 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
192.168.1.50 - - [20/Jul/2026:10:05:09 +0000] "GET /api/products HTTP/1.1" 200 23456 "-" "curl/7.68.0"
10.0.0.1 - - [20/Jul/2026:10:05:10 +0000] "GET /nonexistent HTTP/1.1" 404 196 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)"
172.16.0.100 - - [20/Jul/2026:10:05:11 +0000] "GET /api/orders HTTP/1.1" 500 89 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0)"
83.149.9.216 - - [20/Jul/2026:10:05:12 +0000] "GET /admin HTTP/1.1" 301 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
192.168.1.50 - - [20/Jul/2026:10:05:13 +0000] "DELETE /api/users/5 HTTP/1.1" 500 102 "-" "curl/7.68.0"
10.0.0.1 - - [20/Jul/2026:10:05:14 +0000] "PUT /api/users/3 HTTP/1.1" 200 345 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)"
172.16.0.100 - - [20/Jul/2026:10:05:15 +0000] "GET /docs/api HTTP/1.1" 200 12890 "http://example.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0)"
83.149.9.216 - - [20/Jul/2026:10:05:16 +0000] "GET /old-page HTTP/1.1" 301 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
192.168.1.50 - - [20/Jul/2026:10:05:17 +0000] "GET /api/health HTTP/1.1" 200 23 "-" "curl/7.68.0"
10.0.0.1 - - [20/Jul/2026:10:05:18 +0000] "POST /api/orders HTTP/1.1" 500 156 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)"
172.16.0.100 - - [20/Jul/2026:10:05:19 +0000] "GET /missing-image.jpg HTTP/1.1" 404 198 "http://example.com/index.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0)"
83.149.9.216 - - [20/Jul/2026:10:05:20 +0000] "GET /api/products/42 HTTP/1.1" 200 1567 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
192.168.1.50 - - [20/Jul/2026:10:05:21 +0000] "GET /api/products/999 HTTP/1.1" 404 87 "-" "curl/7.68.0"
10.0.0.1 - - [20/Jul/2026:10:05:22 +0000] "GET /dashboard HTTP/1.1" 200 9876 "http://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)"
172.16.0.100 - - [20/Jul/2026:10:05:23 +0000] "POST /api/checkout HTTP/1.1" 500 201 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0)"
83.149.9.216 - - [20/Jul/2026:10:05:24 +0000] "GET /about HTTP/1.1" 200 3421 "http://example.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"

What is in this log file?

Each line follows the Apache Combined Log Format. A single line packs the client IP, a timestamp, the HTTP method and URL, the status code, response size in bytes, the referrer, and the user agent into one string.

The file includes a realistic mix of status codes: successful 200 responses, 301 redirects, 404 not-found errors, and 500 server errors.

  • Save access.log.
  • Confirm the file exists by running this in your terminal:
ls logstash/ingest_data/

You should see access.log listed in the output.

Don't see access.log?

Make sure your terminal is in the project root folder (the one containing docker-compose.yml). Check that you created the file inside logstash/ingest_data/ and not in a different directory.

Help me find where my access.log was saved

Index raw log lines into Elasticsearch

Now you'll push a few of these raw log lines directly into Elasticsearch as plain text. This simulates what happens when logs arrive without any parsing.

  • In Kibana, open the navigation menu (hamburger icon, top left).
  • Scroll down and click Dev Tools under the Management section.
  • Clear any existing text in the console, then paste this bulk indexing request:
POST raw-logs/_bulk
{"index": {}}
{"message": "83.149.9.216 - - [20/Jul/2026:10:05:03 +0000] \"GET /index.html HTTP/1.1\" 200 4523 \"http://example.com/\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36\""}
{"index": {}}
{"message": "83.149.9.216 - - [20/Jul/2026:10:05:08 +0000] \"GET /favicon.ico HTTP/1.1\" 404 209 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36\""}
{"index": {}}
{"message": "172.16.0.100 - - [20/Jul/2026:10:05:11 +0000] \"GET /api/orders HTTP/1.1\" 500 89 \"-\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 14_0)\""}
{"index": {}}
{"message": "83.149.9.216 - - [20/Jul/2026:10:05:12 +0000] \"GET /admin HTTP/1.1\" 301 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36\""}
{"index": {}}
{"message": "10.0.0.1 - - [20/Jul/2026:10:05:10 +0000] \"GET /nonexistent HTTP/1.1\" 404 196 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)\""}
  • Click the green play button (or press Ctrl+Enter) to execute the request.

What does this request do?

The POST raw-logs/_bulk command creates a new index called raw-logs and inserts five documents in one request. Each document has a single field called message containing the entire log line as one unbroken string.

No parsing happens here. The IP address, status code, URL, and everything else are mashed together in that one text blob.

You should see a response on the right side of Dev Tools showing "errors": false and five items with "result": "created".

Seeing errors in the response?

Check that each {"index": {}} line is on its own line, not combined with the document line below it. The _bulk API requires strict newline-delimited JSON with alternating action and document lines.

Help me fix my _bulk request errors

Attempt to query by status code

Your five log lines are now stored in Elasticsearch. Each one clearly contains a status code like 200 or 404 if you read the text. The question is: can Elasticsearch search on that status code as a field?

Before you run the next query, predict what will happen. Do you think Elasticsearch can find only the 404 entries when the status code is buried inside a text blob?

  • In Dev Tools, clear the console and paste this query that tries to match on a response field:
GET raw-logs/_search
{
  "query": {
    "match": {
      "response": "404"
    }
  }
}
  • Click the green play button to run it.

You get back zero hits. Elasticsearch has no field called response because you never parsed the logs into separate fields. The status code is trapped inside the message string.

  • Now run this query to see what the documents actually look like:
GET raw-logs/_search
{
  "query": {
    "match_all": {}
  }
}

Look at the results on the right. Every document is just a single message field containing the full log line as one long string. There is no clientip field, no response field, no bytes field.

Why is this a problem?

Without separate fields, you cannot filter logs by status code, aggregate requests by IP address, or calculate average response sizes. Dashboards need structured fields to build charts and alerts.

This is exactly the gap that Logstash fills. It sits between your raw data source and Elasticsearch, parsing each line into named fields before indexing.

✔️ Awesome, I've got everything!

Great. You've now seen firsthand why raw log ingestion without parsing is a dead end for analytics.

ⓧ I'd like to double check the full code

Here is the complete access.log file you created:

83.149.9.216 - - [20/Jul/2026:10:05:03 +0000] "GET /index.html HTTP/1.1" 200 4523 "http://example.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
83.149.9.216 - - [20/Jul/2026:10:05:04 +0000] "GET /style.css HTTP/1.1" 200 1234 "http://example.com/index.html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
192.168.1.50 - - [20/Jul/2026:10:05:05 +0000] "GET /api/users HTTP/1.1" 200 8923 "-" "curl/7.68.0"
10.0.0.1 - - [20/Jul/2026:10:05:06 +0000] "POST /api/login HTTP/1.1" 200 452 "http://example.com/login" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)"
172.16.0.100 - - [20/Jul/2026:10:05:07 +0000] "GET /images/logo.png HTTP/1.1" 200 15234 "http://example.com/index.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0)"
83.149.9.216 - - [20/Jul/2026:10:05:08 +0000] "GET /favicon.ico HTTP/1.1" 404 209 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
192.168.1.50 - - [20/Jul/2026:10:05:09 +0000] "GET /api/products HTTP/1.1" 200 23456 "-" "curl/7.68.0"
10.0.0.1 - - [20/Jul/2026:10:05:10 +0000] "GET /nonexistent HTTP/1.1" 404 196 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)"
172.16.0.100 - - [20/Jul/2026:10:05:11 +0000] "GET /api/orders HTTP/1.1" 500 89 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0)"
83.149.9.216 - - [20/Jul/2026:10:05:12 +0000] "GET /admin HTTP/1.1" 301 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
192.168.1.50 - - [20/Jul/2026:10:05:13 +0000] "DELETE /api/users/5 HTTP/1.1" 500 102 "-" "curl/7.68.0"
10.0.0.1 - - [20/Jul/2026:10:05:14 +0000] "PUT /api/users/3 HTTP/1.1" 200 345 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)"
172.16.0.100 - - [20/Jul/2026:10:05:15 +0000] "GET /docs/api HTTP/1.1" 200 12890 "http://example.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0)"
83.149.9.216 - - [20/Jul/2026:10:05:16 +0000] "GET /old-page HTTP/1.1" 301 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
192.168.1.50 - - [20/Jul/2026:10:05:17 +0000] "GET /api/health HTTP/1.1" 200 23 "-" "curl/7.68.0"
10.0.0.1 - - [20/Jul/2026:10:05:18 +0000] "POST /api/orders HTTP/1.1" 500 156 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)"
172.16.0.100 - - [20/Jul/2026:10:05:19 +0000] "GET /missing-image.jpg HTTP/1.1" 404 198 "http://example.com/index.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0)"
83.149.9.216 - - [20/Jul/2026:10:05:20 +0000] "GET /api/products/42 HTTP/1.1" 200 1567 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
192.168.1.50 - - [20/Jul/2026:10:05:21 +0000] "GET /api/products/999 HTTP/1.1" 404 87 "-" "curl/7.68.0"
10.0.0.1 - - [20/Jul/2026:10:05:22 +0000] "GET /dashboard HTTP/1.1" 200 9876 "http://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)"
172.16.0.100 - - [20/Jul/2026:10:05:23 +0000] "POST /api/checkout HTTP/1.1" 500 201 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0)"
83.149.9.216 - - [20/Jul/2026:10:05:24 +0000] "GET /about HTTP/1.1" 200 3421 "http://example.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"

You've now experienced the core problem: raw logs stored as a single text field are useless for analytics. Next up, you'll build a Logstash pipeline that parses each line into separate, queryable fields.

Build the Logstash Pipeline with Grok Parsing

In the last step, you saw the core problem firsthand: raw log lines dumped into Elasticsearch are unsearchable blobs of text. You cannot filter by status code, IP address, or endpoint because those fields simply do not exist.

This is the problem Logstash solves. Its grok filter uses pattern matching to split each log line into named fields like clientip, verb, response, and bytes. Once parsed, every field becomes queryable and visualizable in Kibana.

In this step, get ready to:
  • Create a Logstash pipeline configuration with grok parsing, date filtering, and type conversion.
  • Restart Logstash to ingest your sample log file through the new pipeline.
  • Verify structured documents appear in Elasticsearch with separate queryable fields.
Create the pipeline configuration file

A Logstash pipeline has three sections: input (where data comes from), filter (how to transform it), and output (where to send the results).

  • In your code editor, create a new file called logstash.conf inside the logstash/pipeline/ folder.
  • Add the input and filter sections by pasting this code:
input {
  file {
    path => "/usr/share/logstash/ingest_data/*.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
  mutate {
    convert => {
      "response" => "integer"
      "bytes" => "integer"
    }
  }
}

What does this code do?

  • The file input reads any .log file from the ingest directory. start_position => "beginning" ensures it reads from the first line. sincedb_path => "/dev/null" means it re-reads the whole file on every restart.
  • The grok filter applies the built-in %{COMBINEDAPACHELOG} pattern. This single pattern extracts clientip, verb, request, response, bytes, referrer, and agent from each log line.
  • The date filter parses the log timestamp and sets it as the event's official @timestamp field for time-based queries.
  • The mutate filter converts response and bytes from strings to integers so you can run range queries and numeric aggregations on them.
  • Below the filter section, add the output section:
output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    user => "elastic"
    password => "elastic123"
    index => "web-logs"
  }
  stdout {
    codec => rubydebug
  }
}

What does the output section do?

  • The elasticsearch output sends each parsed document to a web-logs index in Elasticsearch. It authenticates with the elastic user.
  • The stdout output with rubydebug codec prints each processed event to the container logs so you can see parsing results in real time.
  • Save logstash/pipeline/logstash.conf.

✔️ Awesome, I've got everything!

Great. Double check you have saved the file.

ⓧ I'd like to double check the full code

Your logstash.conf should have three sections in this order:

  • The input block with a file plugin containing path, start_position, and sincedb_path.
  • The filter block with grok, date, and mutate plugins.
  • The output block with elasticsearch and stdout plugins.

Scroll up and confirm each chunk is present and matches the code blocks above.

Restart Logstash to apply the new pipeline

Your docker-compose.yml already mounts ./logstash/pipeline/logstash.conf and ./logstash/ingest_data into the Logstash container. Now that the config file exists, you need to restart the container so Logstash reads it.

  • Restart the Logstash container by running this command:
docker compose restart logstash

Why restart instead of recreate?

The volume mount was already defined in your Compose file. The Logstash container just needs to restart its process to pick up the pipeline config file that now exists on disk.

  • Wait about 30 seconds for Logstash to initialize its pipeline, then check the logs by running:
docker compose logs logstash --tail 50
  • Look for a line that says Pipeline is running with no error messages above it.
  • You should also see rubydebug output showing your parsed log events with fields like clientip, verb, and response printed as structured data.

Not seeing Pipeline is running?

  • Check for typos in your logstash.conf file. A missing closing brace } or mismatched quotes will prevent the pipeline from starting.
  • Confirm that logstash.conf is saved at logstash/pipeline/logstash.conf (not logstash/logstash.conf). The mount path must match exactly.
  • If you see a connection error to Elasticsearch, make sure the Elasticsearch container is healthy by running docker compose ps.

Help me troubleshoot my Logstash pipeline

Verify structured documents in Kibana Dev Tools

The logs show Logstash processed the file. Now confirm the structured documents actually landed in Elasticsearch.

  • Switch back to Kibana in your browser at http://localhost:5601.
  • Open the hamburger menu (three lines in the top left).
  • Scroll down to Management and click Dev Tools.

Before you run this query, what do you expect to see? Do you think each document will have separate fields for IP, method, and status code, or will it still be one big message string?

  • Paste this query into the Dev Tools console and click the play button to run it:
GET web-logs/_search
{
  "size": 1
}

You should see a document with separate fields including clientip, verb, request, response (as an integer), bytes (as an integer), referrer, and agent.

Compare this to the raw-logs index from the previous step. The difference is the entire value of a Logstash pipeline.

What changed between raw-logs and web-logs?

In the raw-logs index, each document had a single message field containing the entire log line as one string.

In the web-logs index, Logstash split that string into individual typed fields. You can now filter on response: 404 or aggregate bytes values because Elasticsearch knows these are separate, searchable fields.

Getting an index_not_found error?

  • Logstash may still be processing. Wait 30 seconds and try the query again.
  • Run docker compose logs logstash --tail 20 to check if any parsing errors occurred. Errors in the grok pattern will prevent documents from being indexed.

Help me fix my web-logs index

Your Logstash pipeline is transforming raw log lines into clean, structured data. Next up, you will query this structured data with precision to answer real questions about your traffic.

Query Structured Data in Elasticsearch

In the last step, you built a Logstash pipeline that transforms raw log lines into structured documents with separate fields for each piece of data.

Now it is time to prove those structured fields are actually useful. You will run Elasticsearch queries directly in Kibana Dev Tools to filter, aggregate, and analyze your log data in ways that were impossible with the raw text blob from Step 2.

In this step, get ready to:
  • Filter for specific HTTP error codes using a match query.
  • Aggregate status codes to see a breakdown of your traffic.
  • Find large responses and identify top endpoints using range queries and terms aggregations.
Filter for 404 errors with a match query

A match query lets you search for documents where a specific field equals a specific value. Since the Logstash pipeline converted the response field to an integer, you can filter precisely by status code.

  • Switch back to Kibana Dev Tools in your browser.
  • Run this match query to find all 404 error documents:
GET /web-logs/_search
{
  "query": {
    "match": {
      "response": 404
    }
  }
}

What does this query do?

The match query searches the response field for documents with the value 404. Because Logstash converted this field to an integer, Elasticsearch can do an exact numeric match instead of searching through a raw text blob.

Compare this to Step 2 where the entire log line was a single message string. There was no way to say "show me only the 404s" without scanning every character of every document.

You should see 4 hits returned, each containing a document with "response": 404. The results include separate fields for clientip, verb, request, bytes, and more.

Seeing 0 hits?

  • Make sure you are querying the web-logs index (not raw-logs). The raw-logs index has unstructured data.
  • Check that Logstash successfully ingested the access log by running GET /web-logs/_count first. You should see a count greater than 0.
  • help me figure out why my Elasticsearch match query returns 0 hits on the web-logs index
Aggregate status codes with a terms aggregation

Individual queries are useful, but aggregations reveal patterns across your entire dataset. A terms aggregation groups documents by unique values in a field and counts how many fall into each bucket.

  • Before you run this next query, predict: how many distinct HTTP status codes do you think exist in your log data?
  • Run this terms aggregation to find out:
GET /web-logs/_search
{
  "size": 0,
  "aggs": {
    "status_codes": {
      "terms": {
        "field": "response"
      }
    }
  }
}

What does this query do?

Setting "size": 0 tells Elasticsearch to skip returning individual documents. You only want the aggregation results.

The terms aggregation on the response field groups all documents by their HTTP status code and returns a count for each unique value. Since response is a numeric field, the terms aggregation works directly on it without needing a .keyword sub-field.

You should see a breakdown of 4 status codes: 200 with 12 documents, 404 with 4 documents, 500 with 4 documents, and 301 with 2 documents.

Find large responses and identify top endpoints

Range queries let you find documents where a numeric field falls within a boundary. Combined with a terms aggregation on text fields, you can answer questions like "which responses are heaviest?" and "which URLs get hit most often?"

  • Run this range query to find responses larger than 10000 bytes:
GET /web-logs/_search
{
  "query": {
    "range": {
      "bytes": {
        "gte": 10000
      }
    }
  }
}

What does this query do?

The range query on bytes with "gte": 10000 finds all documents where the response size is 10,000 bytes or more. This only works because Logstash converted bytes to an integer. You cannot run a range query on a raw text string.

You should see results for the largest responses in your logs, like /images/logo.png (15234 bytes), /api/products (23456 bytes), and /docs/api (12890 bytes).

  • Now run a terms aggregation on the request field to see which endpoints get the most hits:
GET /web-logs/_search
{
  "size": 0,
  "aggs": {
    "top_endpoints": {
      "terms": {
        "field": "request.keyword"
      }
    }
  }
}

Why use request.keyword instead of request?

Elasticsearch maps string fields as text (for full-text search) with a .keyword sub-field (for exact matching and aggregations). The text version splits /api/users into tokens like "api" and "users", which would produce meaningless aggregation buckets.

Using request.keyword preserves the full URL path as a single value, so each bucket represents an actual endpoint.

You should see a list of endpoints ranked by hit count. Endpoints that appear multiple times in the log file (like /api/orders) show higher counts than one-off requests.

Seeing an error about fielddata?

  • If you accidentally used request instead of request.keyword, Elasticsearch returns an error because terms aggregations cannot run on text fields by default. Add .keyword to the field name and try again.
  • help me understand the Elasticsearch fielddata error I am seeing when running a terms aggregation

You have now filtered by exact value, aggregated across all documents, searched within numeric ranges, and identified top endpoints. Every one of these queries was impossible with the unstructured data in Step 2.

Your structured data is ready for the final piece of the pipeline. Next up, you will turn these query results into visual dashboards in Kibana.

Build a Kibana Dashboard

In the previous step, you ran queries and aggregations against your structured log data in Dev Tools. Those results are powerful, but they require knowing the right query syntax.

Dashboards turn query results into visual insights that non-technical stakeholders can understand. In this step, you will complete the full ELK pipeline by building a Kibana dashboard that visualizes HTTP traffic patterns from your structured log data.

In this step, get ready to:
  • Create a Data View in Kibana for the web-logs index.
  • Build two Lens visualizations showing status code distribution and top requested URLs.
  • Assemble a dashboard with your visualizations and a Metric panel.
Create a Data View and a pie chart visualization

Before Kibana can visualize your data, it needs a Data View that points to your Elasticsearch index. A Data View tells Kibana which index to query and which field to use for time-based filtering.

  • In Kibana at http://localhost:5601, open the navigation menu on the left side.
  • Click Dashboards.
  • Click Create dashboard to start with an empty dashboard.
  • Click Create visualization in the dashboard toolbar to open the Lens editor.

What is Lens?

Lens is Kibana's drag-and-drop visualization editor. It lets you build charts by selecting fields and chart types without writing any code.

Kibana will prompt you to create a Data View because none exists yet for your data.

  • Open the data view menu at the top of the Lens editor.
  • Click Create a data view.
  • In the Name field, enter web-logs.
  • In the Index pattern field, type web-logs. You should see your index appear as a match.
  • In the Timestamp field dropdown, select @timestamp.
  • Click Save data view to Kibana.

The Lens editor now loads your web-logs data. You should see a list of available fields in the left sidebar, including response, request, clientip, and bytes.

Expand the time range

If you see a "No results found" message, expand the time range picker in the top right corner. Select Last 1 year or a range that covers the date in your log file (July 20, 2026).

Now build the pie chart showing HTTP status code distribution.

  • Open the Visualization type dropdown at the top of the editor (it may default to Bar).
  • Select Pie.
  • From the Available fields list in the left sidebar, drag response to the Slice by area in the layer pane.
  • Leave the Metric as the default Count aggregation.

You should see a pie chart appear with slices for each status code (200, 301, 404, 500). Each slice represents how many requests returned that status code.

  • Click Save and return to add this visualization to your dashboard.

What does this visualization show?

The pie chart uses the response field (which Logstash converted to an integer) to group documents by HTTP status code. Each slice's size represents the proportion of requests that returned that code.

At a glance, you can see whether most traffic is healthy (200) or if there is an unusual number of errors (500) or not-found responses (404).

Don't see any data in the chart?

Make sure the time range in the top right corner of Kibana covers the dates in your log file. Your logs are dated July 20, 2026, so select a range that includes that date.

If the response field does not appear in the fields list, confirm that documents exist in the web-logs index by returning to Dev Tools and running GET web-logs/_count.

Why isn't my pie chart showing data from my web-logs index?

Create a horizontal bar chart of top requested URLs

Your second visualization answers a different question: which endpoints receive the most traffic? A horizontal bar chart is ideal for ranking categorical data like URL paths.

  • On your dashboard, click Create visualization in the toolbar to open a new Lens editor.
  • Open the Visualization type dropdown and select Bar horizontal.
  • From the Available fields list, drag request to the vertical axis (the Vertical axis area in the layer pane).
  • Click on the request field you just added to open its settings.
  • Set the Number of values to 10 to show the top 10 URLs.
  • Leave the horizontal axis Metric as Count.

You should see a horizontal bar chart with URLs ranked by request count. The longest bars represent your most-hit endpoints.

  • Click Save and return to add this second visualization to the dashboard.

What does this visualization show?

This chart uses the request field to rank URLs by how frequently they appear in your logs. It quickly reveals which endpoints get the heaviest traffic and helps identify unexpected hotspots.

Seeing request.keyword instead of request?

Elasticsearch sometimes creates both a text and a keyword sub-field. If you see request.keyword in the fields list, use that field instead. Keyword fields work better for terms aggregations because they are not tokenized.

Why is my bar chart showing unexpected field names?

Add a Metric panel and save the dashboard

Your dashboard already shows status distribution and top URLs. The final panel adds a quick summary of total requests and error percentage.

  • On your dashboard, click Create visualization in the toolbar to open a new Lens editor.
  • Open the Visualization type dropdown and select Metric.
  • The default metric is Count of all documents. This shows your total request count.
  • Click Save and return to add the metric to the dashboard.

You now have three panels on your dashboard. Before saving, give the dashboard a meaningful name.

  • In the application menu at the top, click Save.
  • In the Title field, enter Web Traffic Dashboard.
  • Click Save.

What is a Kibana dashboard?

A dashboard is a collection of visualization panels arranged on a single page. You can share it with teammates, set time filters, and click on data points to drill into specific subsets of your data.

Each panel queries the same underlying Elasticsearch index through your Data View, so everything stays synchronized when you adjust the time range or add filters.

Before you verify your finished dashboard, take a moment to predict what you expect to see. How many panels should be visible, and what type of chart is each one?

Open your saved dashboard and confirm you see three panels: a pie chart of status codes, a horizontal bar chart of top endpoints, and a metric panel showing total request count.

Missing panels or empty dashboard?

If panels appear empty, expand the time range to cover July 20, 2026. Click the calendar icon in the top right and set an absolute range that includes that date.

If you only see one or two panels, you may not have clicked Save and return after creating each visualization. Click Create visualization to add the missing panel.

Why are my Kibana dashboard panels empty or missing?

Your ELK pipeline is complete. Raw log lines now flow through Logstash into Elasticsearch and surface as interactive visuals in Kibana. Next up, a secret mission that takes your pipeline to the next level.

Secret mission

Enrich Logs and Create an Alert

Real-world ELK pipelines go beyond dashboards. In this bonus challenge, you will add computed fields to your Logstash pipeline using conditional mutate logic, then create a threshold-based alerting rule in Kibana that fires when server errors spike.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally in Docker, so there are no ongoing cloud costs.

Resources you used:

  • Docker containers: elasticsearch, kibana, logstash (running locally, using ~4 GB RAM).
  • Docker volumes: esdata, kibanadata, logstashdata (storing Elasticsearch indexes, Kibana dashboards, and Logstash state).
  • Local project files: docker-compose.yml, .env, logstash/ directory, and access.log.

Keep everything running

No action needed. Choose this if you are still actively experimenting with your ELK stack or want to ingest more log files.

  • Your containers will keep running and using approximately 4 GB of RAM.
  • Your Kibana dashboard, alerting rule, and all indexed data remain available at http://localhost:5601.
  • You can drop additional .log files into logstash/ingest_data/ and Logstash will pick them up automatically.

Pause - I'll come back to this later

Stop all containers to free up RAM without losing any data. Your indexed logs, dashboards, and alerting rules are preserved in the Docker volumes.

  • Open your terminal and navigate to the folder containing your docker-compose.yml file.
  • Stop all containers by running:
docker compose stop
  • When you are ready to resume, start everything back up by running:
docker compose start

All your Elasticsearch data, Kibana dashboards, and pipeline configuration will be exactly as you left them.

Delete - I don't want to use this again

Remove all containers, networks, volumes, and indexed data. This is a full reset. All Elasticsearch indexes, Kibana dashboards, and alerting rules will be permanently deleted.

  • Open your terminal and navigate to the folder containing your docker-compose.yml file.
  • Remove all containers, networks, and volumes by running:
docker compose down -v
  • Confirm the containers are gone by running:
docker ps -a

You should see no containers named elasticsearch, kibana, or logstash in the list.

  • Delete the local project files. On macOS or Linux, run:
cd ~/Desktop
rm -rf elk-lab
  • On Windows, run:
cd ~\Desktop
Remove-Item -Recurse -Force elk-lab

Your machine is now back to its pre-project state.

Nice Work!

Nice Work!

You just built a complete log analytics pipeline from scratch using the ELK stack. Raw, messy access logs go in one end, and actionable dashboards come out the other.

You've learned how to:

  • Deploy a full ELK stack (Elasticsearch, Logstash, Kibana 9.4.3) orchestrated with Docker Compose and experience the problem of unstructured log data firsthand.
  • Build a Logstash pipeline with file input, grok pattern parsing, and date filters that transform raw text into structured, queryable fields in Elasticsearch.
  • Query structured data with match queries, range queries, and aggregations in Kibana Dev Tools, then visualize HTTP traffic patterns in a multi-panel Kibana dashboard.
  • Secret Mission: Enriched logs with computed fields via conditional mutate filters and set up threshold-based alerting in Kibana.

Ready to quiz yourself?