Build a Mini SIEM with Splunk

Ingest security logs, write SPL detections, and build a SOC dashboard.

Introduction

30 Second Summary

Every time you sign in to a website, your action generates a log entry somewhere. Security teams monitor millions of these entries to catch the one attacker hiding among thousands of normal users.

In this project, you will build a mini SIEM using Splunk Enterprise running in Docker. You will ingest realistic security logs, diagnose a timestamp parsing failure, write detection queries that surface attack patterns, and present everything in a monitoring dashboard.

What You'll Build

Picture a security dashboard where you type a suspicious IP address and instantly see every SSH brute-force attempt, firewall block, and web attack it launched across three log sources on one screen.

By the end of this project, you'll have:

  • A running Splunk SIEM where you can search across SSH authentication logs, Linux system logs, and web access logs to spot attacker activity in seconds.
  • Two production-style detection rules that fire on real attack patterns, complete with MITRE ATT&CK mappings, tuning rationale, and triage procedures you can show in a portfolio.
  • A SOC-style monitoring dashboard with attack timelines, top attacker IPs ranked by volume, and categorized web threats that communicates your skills at a glance.
  • Secret Mission: Build a multi-stage correlation detection that links SSH brute-force and web attacks from the same source IP within a time window.

Are there any prerequisites?

You need Docker Desktop installed on your machine and basic comfort with the command line. Familiarity with SIEM and SOC concepts (what logs are, what a detection rule does) will help you move faster.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

This project proves a specific set of skills to a hiring manager. Naming those skills now helps you stay focused on the security engineering outcome, not just clicking through steps.

Launch Splunk and Ingest Security Logs

A SIEM is only useful when it has data flowing into it. Before you can write detection queries or build dashboards, you need a running instance with realistic security logs to search against.

In this step, you will launch Splunk Enterprise in Docker and feed it three log sources that simulate a server under attack.

In this step, get ready to:
  • Launch Splunk Enterprise in a Docker container.
  • Create three realistic security log files on your machine.
  • Upload the logs into Splunk with correct sourcetype assignments.
Run Splunk Enterprise in Docker

Docker lets you run Splunk Enterprise as a self-contained container with no installer or account required. The official splunk/splunk image includes a 60-day Enterprise trial with full alerting and dashboard features.

  • Make sure Docker Desktop is running. Press Cmd+Space (macOS) or the Windows key (Windows), type Docker Desktop, and press Enter to open it.
  • Open your terminal. Press Cmd+Space (macOS) or the Windows key (Windows), type Terminal, and press Enter.
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir splunk-siem
cd splunk-siem
  • Launch the Splunk Enterprise container by running this command:
docker run -d -p 8000:8000 -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_GENERAL_TERMS=--accept-sgt-current-at-splunk-com" -e "SPLUNK_PASSWORD=Splunk123" --name splunk splunk/splunk:latest

What does this command do?

  • -d runs the container in the background so you keep your terminal.
  • -p 8000:8000 maps port 8000 inside the container to port 8000 on your machine, making Splunk Web accessible in your browser.
  • SPLUNK_START_ARGS and SPLUNK_GENERAL_TERMS accept the Splunk license and general terms. Splunk will not start without these.
  • SPLUNK_PASSWORD=Splunk123 sets the admin password you will use to log in.
  • --name splunk names the container so you can reference it easily in future commands.

Splunk takes about 1-2 minutes to finish its internal setup. You can check progress with:

docker logs splunk --tail 5
  • Wait until you see a message containing Ansible playbook complete in the logs. This means Splunk is ready.
  • Open your web browser and navigate to http://localhost:8000.
  • Log in with username admin and password Splunk123.

You should see the Splunk Enterprise home page with the Search & Reporting app listed.

Splunk Web not loading?

  • Run docker logs splunk --tail 20 and look for error messages. If you see a license acceptance error, make sure you copied the full command including both SPLUNK_START_ARGS and SPLUNK_GENERAL_TERMS.
  • If the page does not load, wait another minute and refresh. The first startup requires initialization.
  • If port 8000 is already in use, stop the conflicting process or remove the container with docker rm -f splunk and re-run with a different port mapping (e.g., -p 8080:8000).

Help me troubleshoot my Splunk Docker container not starting.

Create the security log files

Your SIEM needs data to ingest. You will create three log files that simulate a Linux server experiencing an active attack: SSH brute-force attempts, firewall events, and malicious web requests.

  • Create a logs directory inside your project folder by running:
mkdir logs
  • Create the SSH authentication log by running this command:
cat > logs/auth.log << 'EOF'
Aug  9 02:14:01 webserver sshd[2841]: Failed password for invalid user admin from 10.200.50.3 port 44521 ssh2
Aug  9 02:14:03 webserver sshd[2842]: Failed password for invalid user admin from 10.200.50.3 port 44522 ssh2
Aug  9 02:14:05 webserver sshd[2843]: Failed password for invalid user root from 10.200.50.3 port 44523 ssh2
Aug  9 02:14:07 webserver sshd[2844]: Failed password for invalid user test from 10.200.50.3 port 44524 ssh2
Aug  9 02:14:09 webserver sshd[2845]: Failed password for invalid user guest from 10.200.50.3 port 44525 ssh2
Aug  9 02:14:11 webserver sshd[2846]: Failed password for invalid user admin from 10.200.50.3 port 44526 ssh2
Aug  9 02:14:13 webserver sshd[2847]: Failed password for invalid user oracle from 10.200.50.3 port 44527 ssh2
Aug  9 02:14:15 webserver sshd[2848]: Failed password for invalid user postgres from 10.200.50.3 port 44528 ssh2
Aug  9 02:15:00 webserver sshd[2900]: Accepted password for deploy from 192.168.1.10 port 52201 ssh2
Aug  9 02:20:01 webserver sshd[2950]: Failed password for invalid user admin from 10.200.50.3 port 44600 ssh2
Aug  9 02:20:03 webserver sshd[2951]: Failed password for invalid user admin from 10.200.50.3 port 44601 ssh2
Aug  9 02:20:05 webserver sshd[2952]: Failed password for root from 10.200.50.3 port 44602 ssh2
Aug  9 03:00:01 webserver sshd[3100]: Failed password for deploy from 172.16.0.5 port 33010 ssh2
Aug  9 03:00:03 webserver sshd[3101]: Failed password for deploy from 172.16.0.5 port 33011 ssh2
Aug  9 03:00:05 webserver sshd[3102]: Failed password for deploy from 172.16.0.5 port 33012 ssh2
Aug  9 03:00:07 webserver sshd[3103]: Accepted password for deploy from 172.16.0.5 port 33013 ssh2
Aug  9 03:01:00 webserver sshd[3110]: Failed password for deploy from 172.16.0.5 port 33020 ssh2
Aug  9 03:01:02 webserver sshd[3111]: Failed password for deploy from 172.16.0.5 port 33021 ssh2
Aug  9 03:01:04 webserver sshd[3112]: Accepted password for deploy from 172.16.0.5 port 33022 ssh2
Aug  9 04:30:00 webserver sshd[3500]: Accepted password for admin from 192.168.1.10 port 52300 ssh2
Aug  9 06:00:01 webserver sshd[4000]: Failed password for invalid user admin from 10.200.50.3 port 45000 ssh2
Aug  9 06:00:03 webserver sshd[4001]: Failed password for invalid user root from 10.200.50.3 port 45001 ssh2
Aug  9 06:00:05 webserver sshd[4002]: Failed password for invalid user user from 10.200.50.3 port 45002 ssh2
EOF

What is in this log?

This file simulates an SSH authentication log from a Linux server. It contains two distinct attack patterns:

  • A brute-force attack from IP 10.200.50.3 cycling through common usernames (admin, root, test, guest, oracle, postgres).
  • A legitimate monitoring system at 172.16.0.5 that fails a few times before successfully authenticating. This will become important when you tune false positives later.
  • Confirm the file was created by running:
ls logs/

You should see auth.log listed in the output.

  • Create the Linux system log by running:
cat > logs/syslog.log << 'EOF'
Aug  9 02:13:55 webserver kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:16:3e:5e:6c:00 SRC=10.200.50.3 DST=192.168.1.20 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=54321 DF PROTO=TCP SPT=44520 DPT=22 WINDOW=14600 RES=0x00 SYN URGP=0
Aug  9 02:14:00 webserver kernel: [UFW ALLOW] IN=eth0 OUT= MAC=00:16:3e:5e:6c:00 SRC=10.200.50.3 DST=192.168.1.20 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=54322 DF PROTO=TCP SPT=44521 DPT=22 WINDOW=14600 RES=0x00 SYN URGP=0
Aug  9 02:15:00 webserver systemd[1]: Started Session 45 of user deploy.
Aug  9 02:30:00 webserver kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:16:3e:5e:6c:00 SRC=10.200.50.3 DST=192.168.1.20 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=54400 DF PROTO=TCP SPT=50000 DPT=443 WINDOW=14600 RES=0x00 SYN URGP=0
Aug  9 02:30:01 webserver kernel: [UFW ALLOW] IN=eth0 OUT= MAC=00:16:3e:5e:6c:00 SRC=10.200.50.3 DST=192.168.1.20 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=54401 DF PROTO=TCP SPT=50001 DPT=80 WINDOW=14600 RES=0x00 SYN URGP=0
Aug  9 03:00:00 webserver systemd[1]: Started Session 46 of user deploy.
Aug  9 03:01:05 webserver sudo: deploy : TTY=pts/0 ; PWD=/home/deploy ; USER=root ; COMMAND=/bin/systemctl restart nginx
Aug  9 04:30:00 webserver systemd[1]: Started Session 47 of user admin.
Aug  9 06:00:00 webserver kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:16:3e:5e:6c:00 SRC=10.200.50.3 DST=192.168.1.20 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=55000 DF PROTO=TCP SPT=45100 DPT=8080 WINDOW=14600 RES=0x00 SYN URGP=0
EOF

What is in this log?

This file contains UFW firewall events and system session messages. Notice the same attacker IP 10.200.50.3 being blocked on multiple ports (22, 443, 80, 8080). This cross-source overlap with auth.log will be essential for the Secret Mission's correlation detection later.

  • Create the web application access log by running:
cat > logs/web_access.log << 'EOF'
09-Aug-2026 02:30:01.445 | 10.200.50.3 | GET /index.html HTTP/1.1 | 200 | 3042 | Mozilla/5.0 (compatible; Nikto/2.1.6)
09-Aug-2026 02:30:02.112 | 10.200.50.3 | GET /admin HTTP/1.1 | 403 | 287 | Mozilla/5.0 (compatible; Nikto/2.1.6)
09-Aug-2026 02:30:03.887 | 10.200.50.3 | GET /admin/../../../etc/passwd HTTP/1.1 | 400 | 315 | Mozilla/5.0 (compatible; Nikto/2.1.6)
09-Aug-2026 02:30:04.221 | 10.200.50.3 | GET /login?user=admin'%20OR%201=1-- HTTP/1.1 | 403 | 287 | sqlmap/1.4
09-Aug-2026 02:30:05.554 | 10.200.50.3 | GET /login?user=admin&pass=test HTTP/1.1 | 401 | 287 | sqlmap/1.4
09-Aug-2026 02:30:06.012 | 10.200.50.3 | POST /login HTTP/1.1 | 401 | 142 | sqlmap/1.4
09-Aug-2026 02:30:07.334 | 10.200.50.3 | GET /wp-admin/ HTTP/1.1 | 404 | 196 | Mozilla/5.0 (compatible; Nikto/2.1.6)
09-Aug-2026 02:30:08.776 | 10.200.50.3 | GET /phpmyadmin/ HTTP/1.1 | 404 | 196 | Mozilla/5.0 (compatible; Nikto/2.1.6)
09-Aug-2026 02:31:00.100 | 192.168.1.10 | GET /index.html HTTP/1.1 | 200 | 3042 | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
09-Aug-2026 02:31:01.200 | 192.168.1.10 | GET /dashboard HTTP/1.1 | 200 | 8921 | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
09-Aug-2026 02:35:00.500 | 10.200.50.3 | GET /api/users?id=1%20UNION%20SELECT%20*%20FROM%20passwords HTTP/1.1 | 403 | 287 | sqlmap/1.4
09-Aug-2026 02:35:01.800 | 10.200.50.3 | GET /api/users?id=1;%20DROP%20TABLE%20users HTTP/1.1 | 403 | 287 | sqlmap/1.4
09-Aug-2026 03:15:00.100 | 172.16.0.5 | GET /index.html HTTP/1.1 | 200 | 3042 | Mozilla/5.0 (X11; Linux x86_64)
09-Aug-2026 03:15:01.200 | 172.16.0.5 | GET /dashboard HTTP/1.1 | 200 | 8921 | Mozilla/5.0 (X11; Linux x86_64)
09-Aug-2026 06:00:05.100 | 10.200.50.3 | GET /admin/../../../etc/shadow HTTP/1.1 | 400 | 315 | Mozilla/5.0 (compatible; Nikto/2.1.6)
09-Aug-2026 06:00:06.200 | 10.200.50.3 | GET /cgi-bin/test-cgi?*%20cat%20/etc/passwd HTTP/1.1 | 404 | 196 | Mozilla/5.0 (compatible; Nikto/2.1.6)
EOF

What is in this log?

This is a custom-format web access log with pipe-delimited fields. It contains SQL injection attempts (sqlmap user-agent), path traversal attacks (../../../etc/passwd), and vulnerability scanner fingerprints (Nikto).

Notice the timestamp format: 09-Aug-2026 02:30:01.445. This is non-standard. Splunk will not auto-detect it, which sets up the timestamp parsing problem you will diagnose in Step 2.

  • Confirm all three files exist by running:
ls logs/

You should see auth.log, syslog.log, and web_access.log listed.

Missing one or more files?

  • Make sure you ran each cat command from inside the splunk-siem folder. Run pwd to check your current directory.
  • If you see an error about logs/ not existing, run mkdir logs first, then retry the cat command.

Help me fix my log file creation issue.

Upload logs into Splunk

Now that you have log files on your machine, you will upload them into Splunk through the web interface. Each file gets a specific sourcetype that tells Splunk how to parse it.

  • In Splunk Web, click Settings in the top navigation bar.
  • Click Add Data.
  • Click Upload at the bottom of the page.
  • Click Select File and navigate to your Desktop, then splunk-siem, then logs. Select auth.log and click Open.
  • On the Set Source Type step, click the Source type dropdown and type linux_secure. Select it from the list.
  • Click Next to continue to Input Settings.

On the Input Settings page, you will create a dedicated index for all your security data.

  • Under the Index section, click the dropdown and select Create a new Index.
  • In the Index Name field, type security.
  • Leave all other fields at their defaults and click Save.
  • Click Review, then click Submit.

You should see a success message confirming the file has been uploaded.

Why create a separate index?

Indexes in Splunk are like separate databases. Keeping security logs in their own index makes searches faster and access control easier. In production, SOC teams use dedicated security indexes to isolate sensitive data from general application logs.

  • Click Start Searching to verify the data arrived. Run this search in the search bar:
index=security sourcetype=linux_secure

You should see events from your auth.log file with timestamps spread across the timeline. This confirms Splunk correctly parsed the standard syslog timestamp format.

Now repeat the upload process for the remaining two files.

  • Click Settings then Add Data, then Upload.
  • Click Select File and choose syslog.log from your logs folder.
  • Set the source type to syslog.
  • On Input Settings, set the index to security (select it from the dropdown. It already exists from the first upload).
  • Click Review, then Submit.
  • Click Settings then Add Data, then Upload.
  • Click Select File and choose web_access.log from your logs folder.
  • On the Set Source Type step, Splunk will not auto-detect this format. Click the Source type dropdown and type web_custom. Since this sourcetype does not exist yet, click Create or select the option to create a new source type with the name web_custom.
  • On Input Settings, set the index to security.
  • Click Review, then Submit.

All three files are now uploaded. Before you move on, predict what you expect to see when you search all three sourcetypes together. Will the timestamps look correct for all of them?

  • Click Search & Reporting in the left sidebar (or navigate to the Search app from the Splunk home page).
  • Run this search to see event counts by sourcetype:
index=security | stats count by sourcetype

You should see three rows: linux_secure, syslog, and web_custom, each with their event counts.

  • Now run this search to inspect the web access events on the timeline:
index=security sourcetype=web_custom

Look at the timeline at the top of the search results. All the web_custom events are clustered at a single point rather than spread across their actual timestamps. Splunk used the upload time as the timestamp for every event because it could not parse the custom 09-Aug-2026 02:30:01.445 format.

Why are the web timestamps wrong?

Splunk auto-detects many common timestamp formats (like the syslog format in auth.log and syslog.log). However, the pipe-delimited web log uses a non-standard format that Splunk cannot parse automatically.

When Splunk fails to extract a timestamp from the raw event, it falls back to the time the data was indexed. This is the most common ingestion problem SOC engineers troubleshoot in production. You will fix it in the next step.

Your SIEM is alive and ingesting data. In the next step, you will diagnose this timestamp failure and fix it by teaching Splunk how to read the custom format.

Diagnose and Fix the Timestamp Parsing Failure

In the previous step, you noticed something wrong with the web access log timeline. All events are clustered at a single point instead of spread across their actual timestamps.

This is a timestamp parsing failure. It happens because Splunk cannot auto-detect the custom pipe-delimited format used in your web access logs. Instead of reading each event's real timestamp, Splunk falls back to the upload time. This is the most common ingestion problem SOC engineers troubleshoot in production SIEMs.

In this step, get ready to:
  • Investigate the timestamp clustering and confirm Splunk is using ingestion time.
  • Create a custom props.conf to teach Splunk the correct timestamp format.
  • Re-upload the log data and verify correct per-event timestamps.
Investigate the timestamp clustering

Before fixing anything, you need to confirm what is actually wrong. A good SOC engineer diagnoses before prescribing.

  • Switch back to Splunk Web at http://localhost:8000.
  • Click Search & Reporting in the left sidebar.
  • In the search bar, run the following query:
index=security sourcetype=web_custom
  • Set the time range picker to All time and click the green search button.

Look at the timeline bar chart above the results. All the web events are stacked in a single narrow bar rather than spread across multiple hours.

  • Click on any event in the results list to expand it.
  • Look at the _time field in the event details on the left.
  • Now look at the raw event text. You can see a timestamp like 09-Aug-2026 02:30:01.445 at the start of the line.

The _time field shows the moment you uploaded the file. The timestamp in the raw text shows when the event actually occurred. They do not match.

Why did this happen?

Splunk uses pattern matching to auto-detect timestamps in log data. It recognizes common formats like Aug 9 02:14:01 (used in auth.log and syslog) but does not recognize the custom format 09-Aug-2026 02:30:01.445 followed by pipe delimiters.

When Splunk cannot find a recognizable timestamp, it assigns the current time (ingestion time) to every event. That is why they all cluster at one point.

Create a custom props.conf to fix timestamp parsing

The fix is a configuration file called props.conf that tells Splunk exactly how to read timestamps in the web_custom sourcetype.

  • Open a terminal on your machine (the same one you used for Docker commands in Step 1).
  • Shell into the Splunk container as the splunk user by running this command:
docker exec -u splunk -it splunk bash

Your terminal prompt changes to show you are now inside the container.

  • Create the props.conf file at /opt/splunk/etc/system/local/props.conf by running this command:
cat > /opt/splunk/etc/system/local/props.conf << 'EOF'
[web_custom]
TIME_FORMAT = %d-%b-%Y %H:%M:%S.%3N
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 28
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
EOF

What does this configuration do?

  • TIME_FORMAT tells Splunk the exact pattern to look for: day-month-year hours:minutes:seconds.milliseconds.
  • TIME_PREFIX = ^ means the timestamp starts at the beginning of each line.
  • MAX_TIMESTAMP_LOOKAHEAD = 28 limits how many characters Splunk scans for the timestamp (the format is 24 characters, with a small buffer).
  • SHOULD_LINEMERGE = false prevents Splunk from combining multiple lines into one event.
  • LINE_BREAKER tells Splunk each new line is a separate event.
  • Confirm the file was created correctly by running:
cat /opt/splunk/etc/system/local/props.conf

You should see the six lines of configuration you just wrote, starting with [web_custom].

File is empty or missing?

Make sure you ran the cat > command exactly as shown, including the EOF markers. If the file is wrong, re-run the full command to overwrite it.

Also confirm you are inside the container (your prompt should show a container hostname, not your Mac's name).

Help me create the props.conf file inside my Splunk Docker container

Clear old data, restart Splunk, and re-upload

For the new timestamp configuration to take effect on the web logs, you need to restart Splunk and re-ingest the data. You will clear the entire security index and re-upload all three log files.

  • Still inside the container, stop Splunk by running:
/opt/splunk/bin/splunk stop
  • Clean all event data from the security index by running:
/opt/splunk/bin/splunk clean eventdata -index security -f

Why delete all three sourcetypes?

The clean eventdata command removes everything in the index. This is the simplest approach because you will re-upload all three files in a moment. The auth and syslog events will parse correctly again using their built-in sourcetypes.

  • Start Splunk back up by running:
/opt/splunk/bin/splunk start

Wait for the startup messages to finish. You will see a message indicating Splunk is ready.

  • Exit the container shell by running:
exit
  • Confirm Splunk Web is accessible again by refreshing http://localhost:8000 in your browser.

Splunk Web not loading?

Give it 30-60 seconds after the start command completes. Splunk needs time to initialize all services.

If it still does not load, check the container is running with docker ps in your terminal. If it is not listed, run docker start splunk.

Help me get Splunk Web back up after restarting inside Docker

Now re-upload all three log files using the same process from Step 1.

  • In Splunk Web, click Settings in the top navigation bar.
  • Click Add Data.
  • Click Upload.
  • Upload auth.log with sourcetype set to linux_secure and index set to security.
  • Upload syslog.log with sourcetype set to syslog and index set to security.
  • Upload web_access.log with sourcetype set to web_custom and index set to security.

Same upload process as before

Follow the same Settings > Add Data > Upload workflow you used in Step 1 for each file. Select the sourcetype manually and choose the security index on the Input Settings page.

Before you run the verification query, what do you expect the web_custom timeline to look like now?

  • In Search & Reporting, run this verification query:
index=security sourcetype=web_custom | head 5 | table _time, _raw

The _time values now match the timestamps visible at the start of each raw log line. Events are spread across the timeline instead of clustered at a single point.

  • Also run the original stats query to confirm all three sourcetypes are back:
index=security | stats count by sourcetype

You should see three sourcetypes with event counts. The timeline now shows events distributed across their actual timestamps for all three sources.

web_custom timestamps still showing upload time?

Double-check your props.conf by shelling back into the container and running cat /opt/splunk/etc/system/local/props.conf. The stanza header must be exactly [web_custom] with no extra spaces. The TIME_FORMAT must match %d-%b-%Y %H:%M:%S.%3N exactly.

If you edited the file, you need to stop Splunk, clean the index again, start Splunk, and re-upload web_access.log. The config only applies to newly ingested data.

Help me fix my web_custom timestamp parsing in Splunk

You have diagnosed and resolved the most common SIEM ingestion problem. All three log sources now have accurate timestamps, which means your searches and detections will reflect when events actually happened. Next up, you will write SPL queries that surface attack patterns hidden in these logs.

Write SPL Detection Queries

Your SIEM now has three clean log sources with accurate timestamps. Raw events are just noise until you ask the right questions.

In this step, you will write SPL queries that surface attack patterns buried in those logs. This is detection engineering in action: turning log data into security intelligence.

In this step, get ready to:
  • Write an SSH brute-force detection query and confirm it identifies the attacking IP.
  • Write a web attack detection query that categorizes hits by attack type.
  • Document both detections with MITRE ATT&CK mappings and triage procedures.
Write the SSH brute-force detection query

SSH brute-force attacks have a distinct pattern: many failed login attempts from the same IP address in a short time window. Your detection counts failures per source IP within 5-minute bins and flags any IP that exceeds a threshold.

  • Switch back to Splunk Web at http://localhost:8000.
  • Click Search & Reporting in the left sidebar to open the Search app.
  • Set the time range picker to All time.
  • Paste the following SPL query into the search bar:
index=security sourcetype=linux_secure "Failed password"
| rex "from (?<src_ip>\d+\.\d+\.\d+\.\d+)"
| bin _time span=5m
| stats count as attempts by src_ip, _time
| where attempts > 5

What does this query do?

  • index=security sourcetype=linux_secure "Failed password" filters to only SSH authentication failures.
  • rex extracts the source IP address from each log line into a new field called src_ip.
  • bin _time span=5m groups events into 5-minute time windows.
  • stats count as attempts by src_ip, _time tallies failures per IP per window.
  • where attempts > 5 keeps only IPs that exceeded the threshold.

Before you run this query, think: how many source IPs in the auth log do you expect to exceed 5 failures in a single 5-minute window?

  • Click the green search button (or press Enter).

You should see one result: 10.200.50.3 with 8 attempts in the 02:10-02:15 time bin. The other IPs in the logs never exceed the threshold of 5 in a single window.

Not seeing any results?

  • Make sure the time range picker is set to All time, not a recent window.
  • Verify the index and sourcetype names match exactly: index=security sourcetype=linux_secure.
  • Check that the rex pattern matches your log format by first running just index=security sourcetype=linux_secure "Failed password" to confirm events exist.

Help me debug my brute-force detection query.

Write the web attack detection query

Web attacks leave signatures in request URLs and user-agent strings. SQL injection injects keywords like UNION SELECT. Path traversal uses ../ sequences to escape directory boundaries. Automated tools like sqlmap and Nikto announce themselves in the user-agent header.

  • Clear the search bar and paste this query:
index=security sourcetype=web_custom
| rex field=_raw "\| (?<src_ip>\d+\.\d+\.\d+\.\d+) \| (?<request>[^|]+) \| (?<status>\d+) \| (?<bytes>\d+) \| (?<user_agent>[^|]+)"
| regex _raw="(\.\./|OR%201=1|UNION%20SELECT|DROP%20TABLE|sqlmap|Nikto)"
| eval attack_type=case(
    match(request, "(?i)(OR%201=1|UNION%20SELECT|DROP%20TABLE)"), "SQL Injection",
    match(request, "\.\./"), "Path Traversal",
    match(user_agent, "(?i)(sqlmap|Nikto)"), "Scanner",
    1=1, "Other")
| table _time, src_ip, request, status, user_agent, attack_type

What does this query do?

  • rex extracts structured fields from the pipe-delimited log format into named fields like src_ip, request, and user_agent.
  • regex filters to only events containing attack signatures (path traversal sequences, SQL keywords, or known tool names).
  • eval attack_type=case(...) categorizes each matching event. The case function evaluates in order, so SQL injection is identified first, then path traversal, then generic scanner activity.
  • The patterns use %20 because the raw log stores URL-encoded spaces. Searching for literal OR 1=1 with actual spaces would miss these events.
  • Click the search button to run the query.

You should see approximately 12 results, all from 10.200.50.3. The attack_type column categorizes them into SQL Injection, Path Traversal, and Scanner.

Seeing fewer results than expected?

  • Confirm the sourcetype is web_custom (the name you assigned during upload).
  • If regex patterns are not matching, check that you pasted the %20 URL-encoded versions, not actual spaces.
  • Run index=security sourcetype=web_custom alone first to confirm events are present.

Help me debug my web attack detection query.

Document both detections

A detection query without documentation is unusable in a team. Professional detection engineering requires recording the logic, context, and response procedure alongside the query itself.

  • Open a text editor or note-taking tool of your choice.
  • Document your SSH brute-force detection using this format:
# Detection: SSH Brute-Force Login Attempts

## Data Source
linux_secure (SSH authentication logs)

## SPL Query
index=security sourcetype=linux_secure "Failed password"
| rex "from (?<src_ip>\d+\.\d+\.\d+\.\d+)"
| bin _time span=5m
| stats count as attempts by src_ip, _time
| where attempts > 5

## MITRE ATT&CK
T1110.001 - Brute Force: Password Guessing

## Trigger Logic
Fires when a single source IP generates more than 5 failed
SSH login attempts within any 5-minute window.
## Expected False Positives
- Monitoring systems that authenticate with occasional failures.
- Service accounts with expired or rotating credentials.
- Legitimate users who mistype passwords multiple times.

## Triage Steps
1. Verify the source IP is not a known internal system.
2. Check if any successful authentication followed the failures.
3. Review target accounts (root, admin are higher priority).
4. If external and unauthorized, block IP and escalate.

Why MITRE ATT&CK mapping?

MITRE ATT&CK is a shared framework that gives every attack technique a unique ID. Mapping your detections to ATT&CK IDs lets other analysts instantly understand what behavior the rule catches, regardless of the tool or query language used.

  • Now document your web attack detection using the same format:
# Detection: Web Application Attack Patterns

## Data Source
web_custom (Web application access logs)

## SPL Query
index=security sourcetype=web_custom
| rex field=_raw "\| (?<src_ip>...) \| (?<request>...) ..."
| regex _raw="(\.\./|OR%201=1|UNION%20SELECT|DROP%20TABLE|sqlmap|Nikto)"
| eval attack_type=case(...)
| table _time, src_ip, request, status, user_agent, attack_type

## MITRE ATT&CK
T1190 - Exploit Public-Facing Application

## Trigger Logic
Fires when any request matches SQL injection syntax, path traversal
sequences, or arrives from a known attack tool user-agent.
## Expected False Positives
- Authorized penetration testing or vulnerability scanning.
- URLs that legitimately contain special characters.
- Web application firewalls injecting test payloads.

## Triage Steps
1. Check if the source IP belongs to an authorized scanner.
2. Verify whether any attack received a 200 response (success).
3. Correlate with other log sources for the same source IP.
4. If unauthorized, block IP and review for compromise.

What does T1190 cover?

T1190 (Exploit Public-Facing Application) covers any attempt to exploit a vulnerability in an internet-facing system. SQL injection, path traversal, and automated scanning all fall under this technique because they target the web application layer.

Save your detection documentation. You now have two production-style detections, each with a clear query, a MITRE mapping, and a defined response procedure.

✔️ Awesome, I've got everything!

Great. Confirm you have saved both detection documents with all fields: detection name, SPL query, data source, MITRE ATT&CK ID, trigger logic, expected false positives, and triage steps.

ⓧ I'd like to double check the full queries

Here is the brute-force detection query:

index=security sourcetype=linux_secure "Failed password"
| rex "from (?<src_ip>\d+\.\d+\.\d+\.\d+)"
| bin _time span=5m
| stats count as attempts by src_ip, _time
| where attempts > 5

Here is the web attack detection query:

index=security sourcetype=web_custom
| rex field=_raw "\| (?<src_ip>\d+\.\d+\.\d+\.\d+) \| (?<request>[^|]+) \| (?<status>\d+) \| (?<bytes>\d+) \| (?<user_agent>[^|]+)"
| regex _raw="(\.\./|OR%201=1|UNION%20SELECT|DROP%20TABLE|sqlmap|Nikto)"
| eval attack_type=case(
    match(request, "(?i)(OR%201=1|UNION%20SELECT|DROP%20TABLE)"), "SQL Injection",
    match(request, "\.\./"), "Path Traversal",
    match(user_agent, "(?i)(sqlmap|Nikto)"), "Scanner",
    1=1, "Other")
| table _time, src_ip, request, status, user_agent, attack_type

You have two tested detections that surface real attack patterns from your log data. Next up, you will save these as scheduled alerts, tune out false positives, and investigate a triggered alert like a working SOC analyst.

Create Alerts, Tune False Positives, and Investigate

Your detection queries surface attack patterns, but only when you manually run them. In a real SOC, detections run automatically on a schedule and alert analysts when something fires.

The problem is that automated detections often catch legitimate activity alongside real threats. A detection that fires constantly trains analysts to ignore it. In this step, you will save both queries as scheduled alerts, tune the noisy one to suppress a known false positive, and investigate a triggered alert like a working analyst.

In this step, get ready to:
  • Save both detection queries as scheduled Splunk alerts.
  • Tune the brute-force detection by excluding a legitimate monitoring IP.
  • Investigate a triggered web attack alert across all three log sources.
Save both detection queries as scheduled alerts

A saved alert turns your manual search into an automated monitor. Splunk will run the query on a schedule and log the results on the Triggered Alerts page whenever the trigger condition is met.

  • Return to the Splunk Search bar in Splunk Web.
  • Paste your SSH brute-force detection query (the one you wrote in the previous step) and run it to confirm it still returns results.
  • Click Save As in the top menu.
  • Select Alert.

The Save As Alert dialog opens. Configure it with these settings:

  • Set Title to SSH Brute-Force Detection.
  • Set Alert type to Scheduled.
  • Under scheduling, select Run on Cron Schedule.
  • Enter the cron expression */5 * * * * to run every 5 minutes.
  • Set Trigger condition to Number of Results and set it to trigger when the number of results is greater than 0.
  • Under Trigger Actions, click Add Actions.
  • Select Add to Triggered Alerts.
  • Click Save.

What does this configuration do?

The cron expression */5 * * * * tells Splunk to execute the search every 5 minutes. The trigger condition fires only when results exist (more than 0).

The Add to Triggered Alerts action logs each firing so you can review it later on a dedicated page.

  • Repeat the same process for your web attack detection query.
  • Set Title to Web Attack Detection.
  • Use the same cron schedule (*/5 * * * *), trigger condition (Number of Results > 0), and Add to Triggered Alerts action.
  • Click Save.

Now verify the alerts fire. It may take up to 5 minutes for the scheduled search to run.

  • In the Splunk navigation bar at the top, click Activity.
  • Select Triggered Alerts.

You should see your two alerts listed on the Triggered Alerts page with their fire times. Both will show results because the attack data in your logs matches both detection queries.

Alerts not appearing yet?

Scheduled alerts run on their cron cycle. Wait up to 5 minutes and refresh the page. If they still do not appear, confirm Add to Triggered Alerts was selected as the action when you saved the alert.

Help me troubleshoot my scheduled alerts

Tune the brute-force detection to exclude a false positive

Look at the brute-force alert results. One of the IPs that triggered the detection is 172.16.0.5. This is actually a legitimate monitoring system that authenticates with a few failures before succeeding.

  • On the Triggered Alerts page, click the SSH Brute-Force Detection alert entry to view its search results.
  • Confirm that 172.16.0.5 appears in the results alongside the real attacker 10.200.50.3.

In the auth logs, 172.16.0.5 fails a few times then succeeds. This is a deployment system retrying authentication. A detection that fires on this IP every cycle creates noise that buries real alerts.

  • Return to the Search bar.
  • Add a where clause to your brute-force query to exclude the monitoring IP. Append this line before any final output commands:
| where src_ip!="172.16.0.5"

Why exclude instead of raising the threshold?

You could raise the failure threshold to filter out 172.16.0.5, but that also raises the bar for detecting real attacks. An explicit IP exclusion is more surgical.

It documents exactly what you excluded and why, which matters when another analyst reviews the rule.

  • Run the updated query and confirm 172.16.0.5 no longer appears in the results.
  • Confirm 10.200.50.3 still appears as the detected attacker.

The tuned detection now catches the real attack while ignoring the noisy legitimate system.

  • Update the saved alert with the tuned query. Click Save As and select Alert again with the same title SSH Brute-Force Detection to overwrite it with the new query.
  • Update your detection documentation to include the tuning rationale. Add a note: "Excluded 172.16.0.5 (internal monitoring system). Pattern: few failures followed by successful auth from internal RFC 1918 address. Not indicative of brute-force."

Document every exclusion

In production SOCs, undocumented exclusions are a security risk. If an attacker compromises the excluded IP, the detection is blind to it. Writing down the rationale ensures the next analyst can re-evaluate the exclusion.

Investigate a triggered web attack alert

Detection is only half the job. When an alert fires, a SOC analyst must investigate it to determine whether the activity is truly malicious and what further action is needed.

  • Navigate back to Activity in the top bar.
  • Select Triggered Alerts.
  • Click the Web Attack Detection alert entry to view its results.
  • Note the source IP that triggered the alert: 10.200.50.3.

Now pivot to see all activity from this IP across every log source. This is how analysts build a timeline of an attacker's behavior.

  • Run the following pivot query in the Search bar to pull all events from this IP across all three sourcetypes:
index=security "10.200.50.3"
| table _time, sourcetype, _raw
| sort _time

What does this pivot query do?

Searching for the string "10.200.50.3" across the entire security index pulls every event mentioning that IP, regardless of sourcetype.

Sorting by _time gives you a chronological timeline of the attacker's activity across SSH, firewall, and web logs.

Review the results. You should see a timeline that tells a story:

  • Starting at 02:13, firewall logs show connection attempts to port 22 from 10.200.50.3.
  • From 02:14 to 02:20, SSH brute-force attempts are logged in linux_secure.
  • At 02:30, the same IP begins web scanning and SQL injection attempts in web_custom.
  • The activity resumes at 06:00 with more SSH and web attacks.

This cross-source timeline is the evidence you reference in your analyst finding.

  • Write a brief analyst finding (2-3 sentences) documenting what the evidence shows. Reference specific timestamps and what further investigation is warranted.

Here is an example format for your finding:

Analyst Finding: Source IP 10.200.50.3 conducted SSH brute-force
attempts against the webserver beginning at 02:14, followed by web
application scanning (Nikto) and SQL injection attempts (sqlmap)
starting at 02:30. No successful authentication was observed from
this IP. Recommend blocking at the network perimeter and checking
host-based logs for signs of compromise.

What makes a good analyst finding?

A strong finding separates evidence from conclusions. State what you observed (timestamps, tools, IPs) and what action you recommend.

Do not state that an attacker succeeded unless the logs show successful compromise. In this case, all SSH attempts failed and all web requests returned 400/403/404 status codes.

Not seeing events from all three sourcetypes?

Make sure your search time range in the Search bar covers the full range of your log data. Set it to All time. If you set a narrow time range, some events may fall outside it.

Help me troubleshoot my pivot query results

Before you run the final verification, predict what you expect to see. Does the tuned brute-force alert still catch the real attacker? Does it still fire on the excluded IP?

  • Run your tuned brute-force query one more time in the Search bar.

You should see 10.200.50.3 in the results but NOT 172.16.0.5. The Triggered Alerts page shows both of your alerts. Your investigation note references specific SPL output and timestamps from the cross-source pivot.

Your alerts are live, your detection is tuned, and you have investigated a real alert. Next up, you will present all of this in a SOC-style dashboard.

Build a SOC Security Monitoring Dashboard

You have detection rules firing and an investigation note documenting suspicious activity. The missing piece is a single view where you can watch all of this in real time.

Dashboards are how SOC teams monitor their environment at a glance. A well-designed security dashboard is also the highest-impact portfolio artifact because it visually communicates your detection engineering skills to hiring managers.

In this step, get ready to:
  • Create a SOC monitoring dashboard in Dashboard Studio with a failed-SSH-attempts timechart.
  • Add panels showing top attacker IPs and web attack categories.
  • Add a findings summary panel and write a portfolio description.
Create the dashboard and add the SSH attack timeline

Before adding a panel, verify your timechart query returns data in the Search app. This confirms the SPL works before you wire it into a dashboard.

  • Return to the Search & Reporting app in Splunk Web.
  • Run the following SPL query to see failed SSH authentication attempts over time:
index=security sourcetype=linux_secure "Failed password" | timechart span=1h count

What does this query do?

This searches the security index for all failed password events in the SSH authentication logs.

The timechart span=1h count command groups results into one-hour buckets and counts events per bucket. This produces a time-series visualization showing attack spikes.

  • Confirm you see a line chart with spikes around the hours when brute-force attempts occurred (around 02:00 and 06:00).

Now create the dashboard that will hold all four panels.

  • Click Dashboards in the top navigation bar.
  • Click Create New Dashboard.
  • Enter SOC Security Monitoring as the title.
  • Select Dashboard Studio as the dashboard type.
  • Select Grid layout for easy panel snapping.
  • Click Create.

Why Grid layout?

Grid layout automatically aligns panels into rows without manual positioning. This is faster for building a multi-panel dashboard and keeps things tidy for screenshots.

You are now in the Dashboard Studio visual editor. Add your first panel.

  • Click the Add Chart button in the editing toolbar and select Line.
  • In the Select data source panel, click + Create search.
  • Name the data source SSH Failed Auth Timeline.
  • Paste the following into the SPL query field:
index=security sourcetype=linux_secure "Failed password" | timechart span=1h count
  • Click Apply and close.

The line chart panel appears on your dashboard canvas showing the SSH attack timeline. You should see spikes in the chart where brute-force attempts clustered.

Panel showing no data?

Make sure the Global Time Range on the dashboard covers the time range of your log data. Click the time range picker at the top and select All time to ensure all events appear.

Also confirm the SPL query has no typos by comparing it to the query you ran successfully in the Search app.

Why is my dashboard panel empty?

Add the top attacker IPs table and web attack bar chart

The timeline shows when attacks happen. The next two panels answer who is attacking and what techniques they are using.

  • Click the Add Chart button and select Table.
  • Click + Create search in the data source panel.
  • Name the data source Top Attacker IPs.
  • Paste the following SPL into the query field:
index=security sourcetype=linux_secure "Failed password"
| rex "from (?<src_ip>\d+\.\d+\.\d+\.\d+)"
| stats count as attempts by src_ip
| sort - attempts
  • Click Apply and close.

What does this query do?

The rex command extracts the source IP address from the "from X.X.X.X" pattern in each failed password log line.

The stats count as attempts by src_ip groups by IP and counts how many failures each one generated. The sort - attempts puts the most aggressive attacker at the top.

You should see a table appear with IP addresses ranked by attempt count. 10.200.50.3 should be at the top.

Now add the web attack categorization panel.

  • Click the Add Chart button and select Bar.
  • Click + Create search in the data source panel.
  • Name the data source Web Attacks by Type.
  • Paste the following SPL into the query field:
index=security sourcetype=web_custom
| regex _raw="(\.\./|OR\s+1=1|UNION\s+SELECT|DROP\s+TABLE|/etc/passwd|/etc/shadow|sqlmap|Nikto)"
| eval attack_type=case(
    match(_raw, "\.\./|/etc/passwd|/etc/shadow"), "Path Traversal",
    match(_raw, "OR\s+1=1|UNION\s+SELECT|DROP\s+TABLE"), "SQL Injection",
    match(_raw, "sqlmap|Nikto"), "Scanner")
| stats count by attack_type
  • Click Apply and close.

What does this query do?

The regex command filters to only events matching known attack signatures.

The eval attack_type=case(...) classifies each hit into a category: Path Traversal, SQL Injection, or Scanner. This gives the bar chart distinct categories to display.

Your bar chart should now show three bars representing the different attack types. SQL Injection and Path Traversal are the dominant categories in your web access logs.

Bar chart only showing one category?

The case() function evaluates conditions in order and stops at the first match. If a log line matches both a scanner user-agent and an SQL injection pattern, it is classified by whichever condition appears first in the case statement.

Check that all three conditions are present and correctly separated by commas in your query.

Help me fix my web attack categorization query

Add the findings summary and save the dashboard

The final panel ties everything together with a summary of the most concerning activity. This is what a SOC manager would look at first to decide where to focus.

  • Click the Add Chart button and select Table.
  • Click + Create search in the data source panel.
  • Name the data source Security Findings Summary.
  • Paste the following SPL into the query field:
index=security sourcetype=linux_secure "Failed password"
| rex "from (?<src_ip>\d+\.\d+\.\d+\.\d+)"
| where src_ip!="172.16.0.5"
| stats count as attempts by src_ip
| where attempts > 5
| eval finding="Brute-force: ".attempts." failed SSH attempts"
| table src_ip finding attempts

What does this query do?

This surfaces the high-severity findings: source IPs that exceeded the brute-force threshold after excluding your known monitoring system (172.16.0.5).

The eval finding column creates a human-readable description of each finding. This is the panel a SOC manager would scan first to see what needs attention.

  • Click Apply and close.
  • Click Save in the top-right corner to save your dashboard.

Before you check your dashboard, what do you expect to see across the four panels? Think about which IPs should appear, how many attack categories you have, and what shape the timeline takes.

Click the back arrow or navigate to Dashboards and open SOC Security Monitoring to view it in presentation mode.

Your dashboard should display four panels: a line chart with SSH attack spikes, a table with attacker IPs ranked by volume (10.200.50.3 at the top), a bar chart distinguishing SQL Injection, Path Traversal, and Scanner categories, and a findings table showing the brute-force alert for the attacker IP.

Some panels showing no results?

Ensure the time range picker at the top of the dashboard is set to All time. If it is set to a narrow window like "Last 15 minutes," your historical log data will not appear.

If one panel works but another does not, open the dashboard in edit mode and check the failing panel's SPL for typos by comparing against the queries above.

Help me troubleshoot my SOC dashboard panels

Finally, write a brief portfolio description for your dashboard. This is the text you would include alongside a screenshot when sharing your work.

  • Write 3-4 sentences in a text file or note-taking app that cover: what the dashboard monitors (SSH brute-force, web application attacks, and security findings), what data sources it uses (SSH auth logs, web access logs, Linux system logs), and what security engineering skills it demonstrates (SPL query writing, detection engineering, false-positive tuning, and alert investigation).

Make it portfolio-ready

A strong portfolio description is specific. Instead of "monitors security events," say something like "surfaces SSH brute-force campaigns and web exploitation attempts across three log sources with tuned detection logic and documented triage procedures."

Secret mission

Multi-Stage Correlation Detection

Real attackers don't stop at one technique. They probe SSH, scan for web vulnerabilities, and exploit whatever they find. Your single-source detections catch each attack in isolation, but they miss the bigger picture: a coordinated campaign from a single IP. Build a cross-source correlation detection that links SSH brute-force and web attacks from the same source IP within a 30-minute window.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely in Docker, so there are no ongoing cloud costs.

Resources you used:

  • Docker container splunk running Splunk Enterprise (trial license, no cost).
  • Docker image splunk/splunk:latest downloaded to your machine.
  • Indexed log data in the security index (stored inside the container).
  • Local project files (log files, props.conf, and detection documentation).

Keep everything running

No action needed. Choose this if you are still actively building on your SIEM or plan to continue exploring SPL queries and detections.

  • Your splunk container continues running and consuming system resources (CPU and RAM).
  • The 60-day Enterprise trial license remains active. After 60 days it converts to a perpetual Free license with alerting disabled.
  • To stop the container temporarily without losing data, run docker stop splunk and restart later with docker start splunk.

Pause - I'll come back to this later

Shut down the running container to free up system memory and CPU. Your data, configurations, and dashboards persist inside the stopped container.

  • Stop the Splunk container by running:
  • When you are ready to resume, start it again with docker start splunk. Splunk Web will be available again at localhost:8000 after about 30 seconds.
docker stop splunk

Your indexed data, alerts, dashboards, and props.conf all remain intact inside the container.

Delete - I don't want to use this again

Remove all project resources and start fresh. This permanently deletes the container, all indexed data, your alerts, dashboards, and props.conf configuration.

  • Remove the Splunk container and all its data by running:
docker rm -f splunk
  • Remove the Splunk Docker image to free disk space by running:
docker rmi splunk/splunk:latest
  • Delete your local project files (log files and detection documentation) by running:
rm -rf ~/Desktop/logs
rm -f ~/Desktop/detection_multi_stage.md

After running these commands, all traces of the project are removed from your machine.

Nice Work!

Nice Work!

Nice work! You just built a fully operational mini SIEM from scratch, proving you can ingest security logs, engineer detections, and investigate alerts like a working SOC analyst.

You've learned how to:

  • Deploy Splunk Enterprise in Docker, ingest three realistic security log sources, and diagnose a timestamp parsing failure by writing a custom props.conf configuration.
  • Write SPL detection queries mapped to MITRE ATT&CK techniques, tune a noisy rule with a documented exclusion, and investigate triggered alerts by pivoting across log sources.
  • Build a SOC-style security monitoring dashboard with attack timelines, top attacker IP rankings, and categorized web attack panels ready for a portfolio screenshot.
  • Secret Mission: Built a cross-source correlation detection linking SSH brute-force and web attacks from the same IP within a 30-minute window, with proper evidence and conclusion separation.

Ready to quiz yourself?