Monitor Spring Boot Logs with Splunk

Ingest Spring Boot logs into Splunk Cloud, build dashboards, and set alerts.

Introduction

30 Second Summary

When an e-commerce site crashes during a flash sale, the first thing an operations team does is check the application logs. Knowing how to search, visualize, and alert on log data is a core skill for anyone working in IT operations.

In this project, you will upload realistic Spring Boot application logs to Splunk Cloud Platform, write SPL queries to investigate application health, build a monitoring dashboard, and create a scheduled alert that fires when errors spike.

What You'll Build

A complete application monitoring setup in Splunk Cloud Platform for a fictional e-commerce Spring Boot service, from log ingestion through dashboards and alerting.

By the end of this project, you'll have:

  • A working Splunk Cloud instance ingesting ~5,000 realistic Spring Boot log events covering normal traffic, slow queries, and application errors.
  • SPL queries that extract key metrics like error rates, response time percentiles, and top failing endpoints.
  • A monitoring dashboard with 4 panels showing error timelines, top errors by class, slow endpoints, and request volume.
  • A scheduled alert configured to fire when error counts exceed a threshold.
  • Secret Mission: a transaction search that correlates a user's full request lifecycle across multiple log lines.

Are there any prerequisites?

A basic understanding of application logs and log levels (INFO, WARN, ERROR) is helpful. Familiarity with web application concepts like HTTP requests and database queries will make the log data feel intuitive.

No prior Splunk experience is required. This project walks you through everything from account creation to dashboard building.

Set Up Splunk Cloud Platform Trial

Before you can search through application logs, you need a platform to send them to. Splunk Cloud Platform is an enterprise-grade tool that ingests machine data, lets you search it in real time, and helps you build dashboards and alerts.

In this step, you will create a free Splunk Cloud Platform trial account. The trial gives you a fully functional cloud instance for 15 days with no credit card required.

In this step, get ready to:
  • Create a Splunk Cloud Platform trial account.
  • Log in to Splunk Web and verify access to the Search & Reporting app.
Create a splunk.com account and start the Splunk Cloud Platform trial

Splunk Cloud Platform runs entirely in the browser. You will register on splunk.com and launch a trial instance that Splunk hosts for you.

  • Fill in the registration form with your details:
  • Enter your email address in the Business Email field.
  • Create a password in the Password field.
  • Enter your First Name and Last Name.
  • Enter any value for Job Title, Phone Number, Company, and Zip / Postal Code.
  • Check the box to accept the terms and conditions.
  • Click Create your account.

Use a real email address

Splunk sends your trial URL and login credentials to the email you provide. Use an address you can check right away.

  • Confirm you are not a robot when prompted.
  • Click Start Trial.
  • Accept the Terms of Service.
  • A confirmation page appears with the message: "Your Splunk Cloud Platform Trial is Ready!"
  • Click View My Instance.

Splunk Web launches automatically in your browser. You will also receive a Welcome email containing your trial URL and login credentials.

What are the trial credentials?

The Welcome email contains the URL to your personal Splunk Cloud instance and your login details. The default username is sc_admin. The password is included in the same email.

Keep this email accessible. You will need these credentials any time you log in during the 15-day trial window.

Verify access to Splunk Web

Now that your trial is provisioned, confirm you can access Splunk Web and see the main interface.

  • Log in using the credentials from the Welcome email. Enter sc_admin as the username and the password from the email.
  • If a welcome window appears inviting you to "Drop your data file here", close it by clicking the X in the corner. You will upload data in the next step.
  • Confirm you can see the Splunk Web home screen with the Search & Reporting app listed in the left sidebar or on the home page.

What is Splunk Web?

Splunk Web is the browser-based interface for Splunk Cloud Platform. Everything you do in this project (searching logs, building dashboards, creating alerts) happens through this interface.

The Search & Reporting app is where you will write SPL queries and investigate your application logs.

Can't find the Welcome email?

Check your spam or junk folder. The email comes from Splunk and may take a few minutes to arrive. If you still do not see it after 10 minutes, try clearing your browser cache and logging in again at splunk.com.

Help me troubleshoot my Splunk Cloud trial login

Your Splunk Cloud environment is ready. Next up, you will generate realistic Spring Boot log data and upload it to Splunk for analysis.

Generate and Upload Spring Boot Log Data

Your Splunk Cloud Platform instance is live and ready for data. Now you need something to search.

In this step, you will create a realistic log file that simulates 3 hours of production traffic from a fictional e-commerce Spring Boot application. Then you will upload it to Splunk Cloud so you can start investigating application health in the next step.

In this step, get ready to:
  • Create a Spring Boot log file with realistic INFO, WARN, and ERROR events.
  • Upload the log file to Splunk Cloud via the Add Data wizard.
  • Verify that approximately 5,000 events are indexed and searchable.
Create a realistic Spring Boot log file

Before Splunk can analyze anything, it needs data to ingest. You will create a sample log file that mirrors what a real Spring Boot e-commerce application produces in production.

  • Open your text editor (Notepad on Windows, or TextEdit in plain-text mode on macOS).
  • Create a new file and save it as ecommerce-app.log on your Desktop.
  • Paste the following sample log content into the file:
2024-11-15T09:00:01.234Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : GET /api/products completed in 45ms
2024-11-15T09:00:01.567Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-2] c.e.controller.PaymentController : POST /api/payments completed in 234ms
2024-11-15T09:00:02.891Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-3] c.e.service.OrderService : Processing order #48291 for customer cust-7823
2024-11-15T09:00:03.456Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : GET /api/products/category/electronics completed in 89ms
2024-11-15T09:00:04.012Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-4] c.e.service.OrderService : Order #48291 confirmed successfully
2024-11-15T09:00:05.789Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-2] o.apache.catalina.connector.CoyoteAdapter : Request processed in 12ms
2024-11-15T09:00:12.345Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-5] c.e.controller.PaymentController : POST /api/checkout completed in 567ms
2024-11-15T09:00:15.678Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-3] c.e.repository.ProductRepository : findAll query returned 142 results in 34ms
2024-11-15T09:00:22.123Z  WARN 8432 --- [ecommerce-app] [http-nio-8080-exec-6] c.e.repository.ProductRepository : Slow query detected - query took 5200ms for findByCategory
2024-11-15T09:00:28.456Z  WARN 8432 --- [ecommerce-app] [http-nio-8080-exec-1] o.apache.catalina.connector.CoyoteAdapter : Connection pool nearing capacity - 48/50 active connections
2024-11-15T09:00:35.789Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-7] c.e.service.OrderService : Processing order #48292 for customer cust-1156
2024-11-15T09:00:42.012Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-2] o.s.web.servlet.DispatcherServlet : GET /api/orders/48292 completed in 23ms
2024-11-15T09:00:55.345Z  ERROR 8432 --- [ecommerce-app] [http-nio-8080-exec-8] c.e.service.OrderService : Failed to process order #48293
java.lang.NullPointerException: Cannot invoke method on null reference
	at com.ecommerce.service.OrderService.processOrder(OrderService.java:142)
	at com.ecommerce.controller.OrderController.createOrder(OrderController.java:67)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:897)
2024-11-15T09:01:02.678Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : GET /api/products completed in 38ms
2024-11-15T09:01:15.901Z  WARN 8432 --- [ecommerce-app] [http-nio-8080-exec-9] c.e.repository.ProductRepository : Slow query detected - query took 4800ms for findByPriceRange
2024-11-15T09:01:23.234Z  ERROR 8432 --- [ecommerce-app] [http-nio-8080-exec-3] c.e.controller.PaymentController : Payment gateway timeout for order #48294 - connection timed out after 30000ms
2024-11-15T09:01:45.567Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-4] c.e.service.OrderService : Processing order #48295 for customer cust-9012
2024-11-15T09:01:52.890Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-5] o.s.web.servlet.DispatcherServlet : POST /api/cart/add completed in 156ms
2024-11-15T09:02:10.123Z  ERROR 8432 --- [ecommerce-app] [http-nio-8080-exec-6] c.e.repository.ProductRepository : Database connection failed
org.springframework.dao.DataAccessResourceFailureException: Unable to acquire JDBC Connection
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:82)
	at com.ecommerce.repository.ProductRepository.findById(ProductRepository.java:56)
	at com.ecommerce.service.OrderService.validateProducts(OrderService.java:98)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms
	at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:181)
2024-11-15T09:02:35.456Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-7] o.s.web.servlet.DispatcherServlet : GET /api/products/search?q=laptop completed in 67ms
2024-11-15T09:02:48.789Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-8] c.e.controller.PaymentController : POST /api/payments completed in 345ms
2024-11-15T09:03:01.012Z  WARN 8432 --- [ecommerce-app] [http-nio-8080-exec-2] c.e.service.OrderService : Order #48296 retry attempt 2 of 3
2024-11-15T09:03:15.345Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : GET /api/orders/history?customerId=cust-7823 completed in 189ms
2024-11-15T09:03:30.678Z  ERROR 8432 --- [ecommerce-app] [http-nio-8080-exec-9] c.e.service.OrderService : HTTP 500 Internal Server Error - order processing failed for #48297 completed in 5023ms
2024-11-15T09:03:45.901Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-3] c.e.repository.ProductRepository : findAll query returned 89 results in 28ms
2024-11-15T09:04:02.234Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-4] o.apache.catalina.connector.CoyoteAdapter : Request processed in 15ms
2024-11-15T09:04:15.567Z  ERROR 8432 --- [ecommerce-app] [http-nio-8080-exec-5] c.e.controller.PaymentController : Payment declined for order #48298 - insufficient funds
2024-11-15T09:04:30.890Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-6] c.e.service.OrderService : Processing order #48299 for customer cust-3344
2024-11-15T09:04:45.123Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-7] o.s.web.servlet.DispatcherServlet : GET /api/products/featured completed in 52ms
2024-11-15T09:05:00.456Z  WARN 8432 --- [ecommerce-app] [http-nio-8080-exec-8] o.apache.catalina.connector.CoyoteAdapter : Connection pool nearing capacity - 49/50 active connections
2024-11-15T09:05:15.789Z  ERROR 8432 --- [ecommerce-app] [http-nio-8080-exec-10] c.e.service.OrderService : OutOfMemoryError during batch processing
java.lang.OutOfMemoryError: Java heap space
	at com.ecommerce.service.OrderService.processBatchOrders(OrderService.java:210)
	at com.ecommerce.scheduler.BatchScheduler.runHourlyBatch(BatchScheduler.java:45)
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2024-11-15T09:05:30.012Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : GET /api/products completed in 41ms
2024-11-15T09:05:45.345Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-2] c.e.controller.PaymentController : POST /api/payments completed in 278ms
2024-11-15T09:06:00.678Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-3] c.e.service.OrderService : Processing order #48300 for customer cust-5567
2024-11-15T09:06:15.901Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-4] o.s.web.servlet.DispatcherServlet : POST /api/cart/add completed in 134ms
2024-11-15T09:06:30.234Z  WARN 8432 --- [ecommerce-app] [http-nio-8080-exec-5] c.e.repository.ProductRepository : Slow query detected - query took 6100ms for findByCategory
2024-11-15T09:06:45.567Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-6] o.apache.catalina.connector.CoyoteAdapter : Request processed in 9ms
2024-11-15T09:07:00.890Z  ERROR 8432 --- [ecommerce-app] [http-nio-8080-exec-7] c.e.controller.PaymentController : Payment gateway timeout for order #48301 - connection timed out after 30000ms
2024-11-15T09:07:15.123Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-8] c.e.service.OrderService : Order #48300 confirmed successfully
2024-11-15T09:07:30.456Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-9] o.s.web.servlet.DispatcherServlet : GET /api/products/category/clothing completed in 72ms
2024-11-15T09:07:45.789Z  INFO 8432 --- [ecommerce-app] [http-nio-8080-exec-1] c.e.repository.ProductRepository : findAll query returned 215 results in 45ms

What does this log file contain?

This sample follows the standard Spring Boot Logback format. Each line starts with an ISO-8601 timestamp, then a log level (INFO, WARN, or ERROR), the process ID, the application name, the thread name, the logger class, and finally the message.

The file simulates different scenarios an e-commerce app would produce: normal HTTP requests with response times, slow database queries, connection timeouts, and critical errors with multi-line stack traces (like NullPointerException and OutOfMemoryError).

The sample above contains about 40 events covering a few minutes of traffic. To reach approximately 5,000 events (which gives better visualizations later), you need to expand this file.

  • Copy the entire block of log lines you just pasted.
  • Paste it again below the existing content, changing the hour in each timestamp from 09 to 10.
  • Repeat this process, incrementing the hour each time (10, 11), until you have content spanning 3 hours (09:00 to 12:00).
  • Within each pasted block, vary the minute and second values slightly to avoid exact duplicates.
  • Aim for roughly 100-130 copies of the block to reach approximately 5,000 total lines.

Faster with find-and-replace

Use your text editor's find-and-replace feature to change timestamps in bulk. For example, replace 2024-11-15T09:0 with 2024-11-15T10:0 across an entire pasted block to shift it forward one hour.

  • Save the file as ecommerce-app.log on your Desktop.

Why do I need 5,000 events?

Splunk's timechart command groups events into time buckets. With only a few dozen events, your charts will look sparse and trends will be hard to spot.

5,000 events across 3 hours gives you enough density to see realistic spikes and patterns in later steps. If you have fewer events, the project still works. The charts will just be less dramatic.

Not sure if you have enough events?

Open your file and check the line count in your text editor. In Notepad, the status bar at the bottom shows the total lines. Aim for at least 3,000 lines to get meaningful results from Splunk queries.

help me figure out how to bulk-generate enough log events for my file

Upload the log file to Splunk Cloud

With your log file ready, you will now feed it into Splunk Cloud using the Add Data upload wizard. This is the same workflow teams use to ingest one-off log files for investigation.

  • In Splunk Web, click **Settings** in the top navigation bar.
  • Click Add Data in the menu that appears.
  • Click Upload.
  • Click Select File, browse to your Desktop, select ecommerce-app.log, and click Open.
  • Click Next to proceed to the Set Source Type page.

What is a source type?

A source type tells Splunk how to parse your data into individual events. It controls how timestamps are extracted and where one event ends and the next begins.

Splunk auto-detects a source type based on your file's format. If the preview looks correct (each log line appears as a separate event with the right timestamp), the auto-detected type is working.

  • On the Set Source Type page, review the preview pane on the right. Each log line should appear as a separate event with a green timestamp.
  • If Splunk correctly identifies each event, click Next to proceed.
  • If events are incorrectly merged or timestamps are wrong, click the Source type: drop-down. Hover over the Application category and select log4j as the closest match.

Save as a custom source type

If you want a more descriptive label, click Save As on the Set Source Type page. Name it spring_boot_log, select Application as the category, and click Save. This is optional but makes your data easier to find later.

  • On the Input Settings page, confirm the Index is set to Default (which maps to the main index).
  • Leave the Host value as the default.
  • Click Review to see a summary of your settings.
  • Verify the file name, source type, and index are correct, then click Submit.
  • On the Success screen, click Start Searching to jump directly into the Search & Reporting app with your data loaded.

Upload stuck or failing?

Make sure your file is under 500 MB. The Splunk Cloud trial allows 5 GB/day of indexing, but the browser upload has a practical size limit.

If the upload hangs, try a smaller file first (even 100 events will work for testing). You can always upload additional data later through the same wizard.

help me troubleshoot my Splunk Cloud file upload

Verify data ingestion

The upload is complete, but you should always confirm that Splunk parsed your data correctly before running analysis queries.

  • In the Search & Reporting app, type the following search into the search bar and press Enter:
index=main | head 20

What does this search do?

index=main tells Splunk to look in the default index where your data was stored. The pipe | sends results to the next command. head 20 limits output to the first 20 events so results load quickly.

  • Confirm you see log events in the results with proper timestamps and the full log message visible.
  • Now run this search to count your total events:
index=main | stats count

What does this search do?

stats count is an SPL aggregation command. It counts every event in the results and returns a single number. This tells you exactly how many events Splunk ingested from your file.

  • Verify that the count returns approximately 5,000 events (the exact number depends on how many times you replicated the log block).

Seeing 0 results or far fewer than expected?

Check the time range picker in the top-right corner of the search page. Make sure it is set to All time rather than a short window like "Last 15 minutes." Your log data has timestamps from November 2024, so a recent-time filter would exclude everything.

If results still show 0, go back to Settings > Add Data > Upload and re-upload your file. Make sure you clicked Submit on the Review page (not just Next).

help me figure out why my Splunk search returns 0 results after uploading data

Your Splunk instance now has thousands of realistic application events to work with. Next up, you will write SPL queries to investigate error patterns and application performance.

Search and Investigate with SPL

Your log data is indexed and ready. Now comes the real work of an IT operations engineer: searching through thousands of events to find the signals that matter.

In this step, you will write SPL (Search Processing Language) queries to filter events, extract custom fields with regular expressions, and uncover patterns in your application's behavior.

In this step, get ready to:
  • Filter log events by level and explore field extraction with rex.
  • Investigate error patterns to find the most common exceptions and failing classes.
  • Analyze performance metrics with timechart and stats commands.
Filter and explore log levels

The first thing an operations engineer does when investigating an application is check for errors. A simple keyword search lets you instantly narrow thousands of events down to only the ones that matter.

  • In Splunk Web, click Search & Reporting in the Apps bar to open the Search view.
  • Click the Time Range Picker (to the right of the Search bar) and select All time under Presets to ensure you see all uploaded events.
  • Search for all ERROR events by typing the following into the Search bar and pressing Enter:
index=main ERROR

What does this search do?

This is the simplest form of an SPL search. It retrieves all events from the main index that contain the keyword ERROR anywhere in the raw event text.

You should see a filtered list of events containing only ERROR-level log lines. The event count appears at the top left of the results.

  • Look at the Fields sidebar on the left side of the events list.
  • Under Interesting Fields, notice the fields Splunk has auto-extracted from your data (like host, source, and sourcetype).

What are Interesting Fields?

Splunk automatically identifies fields that appear in at least 20% of your events and lists them under Interesting Fields. You can click any field name to see its values and distribution.

However, Splunk did not extract the log level as a separate field because it is embedded in unstructured text. You will use the rex command to extract it manually.

  • Extract the log level from each event and count by level by running this search:
index=main | rex field=_raw "^\S+\s+(?<log_level>\w+)\s+" | stats count by log_level

What does this search do?

The rex command uses a regular expression to extract a new field called log_level from the raw event text. The pattern skips the timestamp (\S+), then captures the next word (INFO, WARN, or ERROR).

The stats count by log_level command then groups all events by their extracted level and counts how many of each exist.

You should see a table with three rows showing the count of INFO, WARN, and ERROR events. This gives you the overall health distribution of your application.

Seeing zero results or unexpected field values?

Make sure the Time Range Picker is set to All time. The default is Last 24 hours, which may not cover your uploaded log data's timestamps.

If log_level shows unexpected values, click any event to expand it and verify the raw format matches the pattern TIMESTAMP LEVEL PID ---.

help me troubleshoot why my rex extraction is not returning the expected log levels

Investigate error patterns

Knowing that errors exist is only the beginning. An operations engineer needs to find which classes produce the most errors and what types of exceptions are occurring.

  • Find the top error-producing classes by running this search:
index=main ERROR | rex field=_raw "\]\s+(?<logger>\S+)\s+:" | stats count by logger | sort -count

What does this search do?

This search first filters to ERROR events only. Then rex extracts the logger field by matching the class name that appears after the closing bracket ] and before the colon : in each log line.

The sort -count command orders results in descending order so you see the noisiest error-producing classes at the top.

You should see a table listing classes like c.e.service.OrderService and c.e.controller.PaymentController ranked by error count.

  • Search for exception types across all events by running:
index=main "Exception" | rex field=_raw "(?<exception_type>\w+Exception)" | stats count by exception_type | sort -count

What does this search do?

This search finds all events containing the word "Exception" (including stack traces). The rex pattern captures any word ending in "Exception" as the exception_type field.

You will see the distribution of exception types like NullPointerException, OutOfMemoryError, and connection-related exceptions ranked by frequency.

  • Find ERROR events with slow response times (over 3000ms) by running:
index=main ERROR | rex field=_raw "completed in (?<response_time_ms>\d+)ms" | where response_time_ms > 3000 | table _time, response_time_ms, _raw

What does this search do?

This search combines three techniques. First, rex extracts the numeric response time from messages like "completed in 5200ms". Then where filters to only events exceeding 3000ms.

Finally, table displays a clean output with just the timestamp, response time, and raw event text. This reveals which errors are also performance problems.

You should see a table of slow error events. These are the highest-priority incidents because they combine both a failure and a degraded user experience.

Getting empty results for the response time search?

Not all ERROR events contain response time information. Only events with the text "completed in" followed by a number will match the rex pattern. If you see no results, try lowering the threshold from 3000 to 1000 to verify the extraction works.

help me debug why my response time extraction returns no results

Analyze performance metrics

Individual events tell a story, but patterns over time reveal the bigger picture. The timechart command transforms raw events into a time series visualization.

  • Chart the error rate over time by running this search:
index=main ERROR | timechart span=10m count as error_count

What does this search do?

The timechart command groups events into 10-minute buckets (span=10m) and counts the errors in each bucket. The result is a time series showing when errors spiked during the 3-hour window.

The X-axis is always time. The Y-axis shows the count of ERROR events per 10-minute interval.

  • Click the Visualization tab above the results to see a line chart of error spikes over time.

You should see a line chart with peaks and valleys. The peaks indicate time periods where something went wrong in the application.

  • Identify the slowest endpoints by running this search:
index=main "completed in" | rex field=_raw "completed in (?<response_time_ms>\d+)ms" | rex field=_raw "(?<endpoint>/\S+)" | stats avg(response_time_ms) as avg_ms, max(response_time_ms) as max_ms, count by endpoint | sort -avg_ms

What does this search do?

This search uses two rex extractions: one for the response time in milliseconds and one for the URL endpoint (any text starting with /). It then calculates the average and maximum response time for each endpoint.

The result is a table of endpoints ranked by average response time. Endpoints at the top are the slowest and likely candidates for performance optimization.

You should see a table with columns for endpoint, avg_ms, max_ms, and count. The highest avg_ms values appear at the top.

Timechart showing a flat line or single point?

If your timechart shows only one data point, your time range may be too narrow. Confirm the Time Range Picker is set to All time so it covers the full 3-hour window of log data.

If the chart is completely flat, check that your uploaded log file contains events spread across multiple 10-minute intervals rather than all sharing the same timestamp.

help me understand why my timechart only shows one data point

You now have a full set of investigative SPL queries at your disposal. Next up, you will turn these searches into a persistent monitoring dashboard so the data is always at your fingertips.

Build a Monitoring Dashboard

In the previous step, you wrote SPL queries to investigate error patterns, performance bottlenecks, and log level distributions. Each query answered a specific question about your application's health.

Running individual searches one by one is fine for investigation. But in production, operations teams need a single view that shows all key metrics at a glance. That single view is a dashboard. In this step, you will build a 4-panel monitoring dashboard that visualizes error trends, top failing classes, slow endpoints, and overall request volume.

In this step, get ready to:
  • Create the "Spring Boot Health" dashboard with an Error Timeline panel.
  • Add panels for top errors, slow endpoints, and request volume.
  • Add a shared time range picker that controls all four panels.
Create the dashboard and first panel

The fastest way to create a new dashboard in Splunk is to run a search, then save the results directly as a dashboard panel. This creates the dashboard and your first panel in one step.

  • Switch back to the Search & Reporting app in Splunk Web.
  • Run the error timeline search by pasting this SPL into the search bar:
index=main ERROR | timechart span=10m count as errors

What does this search do?

This query filters for ERROR-level events and groups them into 10-minute time buckets. The timechart command counts how many errors occurred in each bucket, producing data that Splunk can render as a timeline visualization.

  • Set the time range to All time and click the green search button (or press Enter).
  • Click the Visualization tab above the results.
  • Select Line Chart as the visualization type.
  • Click Save As in the toolbar above the search bar.
  • Select Dashboard Panel.

A dialog appears asking you to configure the new dashboard and panel.

  • For Dashboard, click New.
  • For Dashboard Title, type Spring Boot Health.
  • For Dashboard Description, type Application monitoring for ecommerce-app.
  • For Dashboard Permissions, keep the default setting Private.
  • For Panel Title, type Error Timeline.
  • Click Save.

A confirmation dialog appears. You now have a dashboard with one panel.

  • Click View Dashboard to see your new dashboard.

You should see the "Spring Boot Health" dashboard with a line chart showing error counts over 10-minute intervals.

Don't see any data in the chart?

Make sure the time range picker at the top of the dashboard is set to All time. If it is set to a short window like "Last 15 minutes", your historical log data falls outside that range.

Help me troubleshoot why my dashboard panel shows no data.

Add remaining panels

Your dashboard needs three more panels to give a complete picture of application health. You will add each one from the dashboard's edit mode.

  • Click Edit at the top of the dashboard to open the dashboard editor.
  • Click Add Panel.
  • Expand the New category and select Column Chart.
  • For the panel title, type Top Errors by Class.
  • Paste this search string into the search field:
index=main ERROR | rex field=_raw "\]\s+(?<logger>\S+)\s+:" | stats count by logger | sort -count | head 10
  • Click Add to Dashboard.

What does this panel show?

This panel extracts the Java class name (logger) from each ERROR line using a rex pattern, then counts errors per class. The column chart makes it easy to spot which parts of the codebase produce the most errors.

Now add the third panel for slow endpoints.

  • Click Add Panel again.
  • Expand New and select Table.
  • For the panel title, type Slow Endpoints.
  • Paste this search string:
index=main "completed in" | rex field=_raw "completed in (?<response_time_ms>\d+)ms" | rex field=_raw "(?<endpoint>/\S+)" | stats avg(response_time_ms) as avg_response_ms, count as requests by endpoint | sort -avg_response_ms | head 10
  • Click Add to Dashboard.

Finally, add the fourth panel for overall request volume.

  • Click Add Panel one more time.
  • Expand New and select Area Chart.
  • For the panel title, type Request Volume.
  • Paste this search string:
index=main | timechart span=10m count as total_requests
  • Click Add to Dashboard.

You should now see all four panels in the dashboard editor. Each panel renders its own visualization.

Why four separate panels?

Each panel answers a different question. Error Timeline shows when problems happen. Top Errors by Class shows where in the code they originate. Slow Endpoints reveals performance bottlenecks. Request Volume provides baseline traffic context so you can tell whether error spikes correlate with traffic spikes.

Panel showing "No results found"?

Double-check that you pasted the full search string without any line breaks. The regex patterns must be on a single line. Also confirm the panel's time range is not restricting results to a window with no data.

Help me fix a Splunk dashboard panel that shows no results.

Add a time range picker and save

Right now, each panel uses whatever time range was set when you created it. A shared time range picker lets you control all panels from one dropdown at the top of the dashboard.

  • In the dashboard editor (you should still be in edit mode), click Add Input.
  • Select Time.

A time range picker input appears at the top of the dashboard. Now configure it.

  • Click the pencil icon (Edit Input) on the time range picker.
  • Set Label to Time Range.
  • Set Default to All time.

Next, connect each panel to this shared time picker so they all update together.

  • On the Error Timeline panel, click the Edit Search icon (magnifying glass).
  • In the Edit Search dialog, set Time range to Shared Time Picker.
  • Click Apply.
  • Repeat for the Top Errors by Class panel: click the Edit Search icon, set Time range to Shared Time Picker, click Apply.
  • Repeat for the Slow Endpoints panel.
  • Repeat for the Request Volume panel.
  • Click Save at the top of the dashboard editor to finalize.

Your dashboard exits edit mode. Try changing the time range picker to a different value and confirm that all four panels update their visualizations.

Why link panels to a shared time picker?

Without a shared time picker, each panel shows data for whatever time range was hardcoded when you created it. Linking them means you can zoom in on a suspicious 30-minute window and see all four metrics for that same window simultaneously.

Time picker not updating all panels?

Make sure you set each panel's time range to Shared Time Picker individually. If a panel still shows the old time range, click Edit, then click that panel's magnifying glass icon and verify the selection.

Help me connect my Splunk dashboard panels to the shared time picker.

✔️ Awesome, I've got everything!

Great. Double check that your dashboard has all four panels and the time range picker updates them all.

ⓧ I'd like to double check the full searches

Here are all four panel searches for reference:

Panel 1: Error Timeline (Line Chart)

index=main ERROR | timechart span=10m count as errors

Panel 2: Top Errors by Class (Column Chart)

index=main ERROR | rex field=_raw "\]\s+(?<logger>\S+)\s+:" | stats count by logger | sort -count | head 10

Panel 3: Slow Endpoints (Table)

index=main "completed in" | rex field=_raw "completed in (?<response_time_ms>\d+)ms" | rex field=_raw "(?<endpoint>/\S+)" | stats avg(response_time_ms) as avg_response_ms, count as requests by endpoint | sort -avg_response_ms | head 10

Panel 4: Request Volume (Area Chart)

index=main | timechart span=10m count as total_requests

Your monitoring dashboard is complete. Next up, you will create a scheduled alert that automatically fires when error counts spike above a threshold.

Create a Scheduled Alert for Error Spikes

Your dashboard gives you a powerful view of application health. But dashboards only help when someone is actively looking at them.

In this step, you will create a scheduled alert that automatically checks for error spikes every hour. If the error count exceeds a threshold, Splunk Cloud logs a triggered alert so you know something needs attention.

In this step, get ready to:
  • Write an alert search that counts ERROR events.
  • Configure a scheduled alert with an hourly schedule and trigger condition.
  • Verify the alert appears in Splunk's alert management.
Create the alert search

Before saving an alert, you need a search that produces the metric you want to monitor. For error spike detection, a simple count of ERROR events is the most effective starting point.

  • In Splunk Web, click Search & Reporting in the left sidebar to open the Search page.
  • Enter the following SPL query in the search bar and click the green search button (or press Enter):
index=main ERROR | stats count as error_count

What does this search do?

This search finds all events in the main index that contain the word ERROR, then counts them into a single field called error_count.

The result is a single row with one number. Splunk's alert system uses this result to decide whether the trigger condition is met.

  • Confirm the search returns a single row showing the error count. The number should be in the hundreds (since your log data contains several hundred ERROR events across the 3-hour window).

Search returns no results?

Make sure the time range picker above the search bar is set to All time. If it is set to a narrow window like "Last 15 minutes", your historical log data will not appear.

Also confirm the search text is exactly index=main ERROR | stats count as error_count with no extra spaces or typos.

Why is my Splunk search returning zero results even though I uploaded data?

Save as a scheduled alert

Now that the search works, you can save it as a scheduled alert. Splunk will run this search automatically on a schedule and check whether the results meet your trigger condition.

  • With the search results still showing, click Save As in the toolbar above the search bar.
  • Select Alert from the dropdown menu.

A dialog appears with several fields to configure. Fill them in as follows:

  • In the Title field, type High Error Rate - Ecommerce App.
  • For Alert type, select Scheduled.
  • For Time Range, set it to run every hour.
  • Under Trigger condition, select Number of Results.
  • Set the condition to is greater than and enter 50 as the threshold.
  • Under Alert Actions, select Add to Triggered Alerts.
  • Click Save.

Why 'Add to Triggered Alerts' instead of email?

Email-based alert actions require SMTP server configuration, which is not available in the Splunk Cloud Platform trial. The "Add to Triggered Alerts" action logs the alert in a dedicated page you can check anytime.

In a production environment, teams typically configure email, Slack webhooks, or PagerDuty integrations as alert actions.

Why set the threshold at 50?

The trigger condition "Number of Results is greater than 50" means the alert fires only when more than 50 errors occur in the search window. This avoids noisy alerts from occasional errors while catching genuine spikes.

In production, you would tune this threshold based on your application's normal error baseline.

Don't see the Alert option under Save As?

Make sure you have run the search first. The Save As menu options are only active after a search has completed.

Help me troubleshoot why Save As > Alert is not available in Splunk Web.

Verify the alert configuration

After saving, verify that Splunk registered the alert correctly and that you know where triggered alerts will appear.

  • Click Settings in the top navigation bar.
  • Click Triggered Alerts to see where alerts appear when they fire. This page will be empty for now since the alert has not triggered yet.
  • Click Settings again, then click Searches, Reports, and Alerts.
  • Confirm that High Error Rate - Ecommerce App is listed with type Alert.
  • Click Edit next to the alert to review the configuration. Confirm the schedule, trigger condition, and action match what you set.

What should I see?

In the Searches, Reports, and Alerts list, your alert should show with the title "High Error Rate - Ecommerce App", type "Alert", and a scheduled run interval of every hour.

The Edit view should confirm: trigger condition set to "Number of Results is greater than 50" and the alert action set to "Add to Triggered Alerts".

Alert not showing in the list?

Check the App dropdown filter at the top of the Searches, Reports, and Alerts page. Make sure it is set to All or to Search & Reporting.

Also check the Owner filter. If set to a different user, your alert will be hidden.

Why can't I find my saved alert in Splunk's Searches, Reports, and Alerts page?

Your monitoring pipeline is complete. You can now ingest logs, investigate issues with SPL, visualize health on a dashboard, and get alerted when things go wrong. Next up, the Secret Mission challenges you to trace an entire request lifecycle across multiple log lines.

Secret mission

Correlate a User Request Lifecycle with Transaction Search

A single log line only tells part of the story. In production, a slow response might touch the dispatcher, the service layer, the repository, and the error handler before returning to the user. Can you trace a full request lifecycle across multiple log lines using Splunk's transaction command?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses the free Splunk Cloud Platform trial, so there are no ongoing costs. The trial auto-expires after 15 days and all data is deleted.

Resources you used:

  • Splunk Cloud Platform trial instance (uploaded log data, SPL queries, dashboard, and scheduled alert).
  • Local file: ecommerce-app.log.

Keep everything running

No action needed. Choose this if you are still actively building or want to explore more SPL queries and dashboard features.

  • Your Splunk Cloud Platform trial remains active for 15 days from signup. You can continue searching, building dashboards, and creating alerts during this period.
  • Your local ecommerce-app.log file stays on your computer for reference.
  • Your splunk.com account remains active even after the trial expires.

Pause - I'll come back to this later

Since Splunk Cloud runs entirely in the browser, there are no local processes to stop. Your trial instance continues running in the cloud until it expires.

  • Close your Splunk Web browser tab.
  • Bookmark your trial URL (starts with https://prd-p-) so you can return to it later.
  • Keep your ecommerce-app.log file in case you want to re-upload or modify it.
  • Remember that your trial expires 15 days after signup. All data and configurations are deleted at expiry.

Delete - I don't want to use this again

Remove all project resources and start fresh. The Splunk Cloud trial instance auto-deletes when it expires, so no manual cloud cleanup is required.

  • Delete your local log file. On macOS/Linux, open your terminal and run:
rm ~/Desktop/ecommerce-app.log
  • On Windows, open PowerShell and run:
Remove-Item ~\Desktop\ecommerce-app.log
  • Your Splunk Cloud Platform trial instance (including all uploaded data, saved searches, dashboards, and alerts) is automatically deleted when the 15-day trial expires. No manual deletion is needed.

What about my splunk.com account?

Your splunk.com account persists after the trial ends, but it holds no data or running resources. It does not incur any charges.

That's a Wrap!

That's a Wrap!

Nice work! You just built a complete application monitoring pipeline in Splunk Cloud Platform, from raw log ingestion all the way to proactive alerting.

You've learned how to:

  • Generated realistic Spring Boot log data and ingested ~5,000 events into Splunk Cloud using the upload workflow with a custom source type.
  • Written SPL queries using rex, stats, timechart, and where to investigate error patterns, exception types, and slow endpoints.
  • Built a 4-panel monitoring dashboard with a shared time range picker and created a scheduled alert that fires when error counts exceed a threshold.
  • Secret Mission: Used the transaction command to correlate multi-line request lifecycles and identify performance bottlenecks across related log events.

Ready to quiz yourself?