Event-Driven Order Processing with Kafka
Build a secure Spring Boot order system with Kafka, PostgreSQL, and AWS SQS locally.
Introduction
30 Second Summary
When you place an order online, dozens of systems coordinate behind the scenes to make it happen. If one of those systems fails mid-process, the order can't just disappear.
In this project, you will build an event-driven order processing system that runs entirely on your local machine. You'll create a secured REST API that publishes order events to Kafka, processes them with automatic retries, routes failures to an AWS SQS dead-letter queue, and deploys the whole thing to a local Kubernetes cluster through an automated CI/CD pipeline.
What You'll Build
A Spring Boot REST API secured with JWT authentication that publishes order events to Kafka, persists state to PostgreSQL, forwards failed events to an SQS dead-letter queue via LocalStack, and deploys to a local Kubernetes cluster using Helm charts orchestrated by Jenkins and GitHub Actions.
By the end of this project, you'll have:
- A Spring Boot application with secured REST endpoints for order creation, backed by Spring Security JWT authentication.
- A Kafka producer/consumer pipeline with retry logic and dead-letter topic routing for failed events.
- An AWS SQS dead-letter queue with IAM least-privilege policies, orchestrated locally with Docker Compose.
- A local CI/CD pipeline using nektos/act, GitHub Actions, Jenkins, and Helm to deploy to a Kubernetes cluster powered by kind.
- Secret Mission: Implement idempotent message consumption with a deduplication table to prevent duplicate processing.
Are there any prerequisites?
This project assumes basic Java and Spring Boot experience, along with basic knowledge of Kafka produce/consume patterns. You should also have Docker Desktop installed on your machine.
Set Up Docker Compose and Project Scaffold
Event-driven systems need multiple infrastructure services running together. This project uses Kafka for messaging, PostgreSQL for persistence, and LocalStack to emulate AWS services locally.
In this step, you will spin up all three services with Docker Compose, scaffold a Spring Boot project from Spring Initializr, and wire everything together so the application starts cleanly.
In this step, get ready to:
- Spin up Kafka, PostgreSQL, and LocalStack containers with Docker Compose.
- Generate a Spring Boot project with all required dependencies from Spring Initializr.
- Configure the application to connect to each service and verify it starts without errors.
Create the Docker Compose file and start containers
Docker Compose lets you define multi-container environments in a single YAML file. You will create one that orchestrates Kafka, PostgreSQL, and LocalStack on your machine.
- Open a terminal window. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
- Type Terminal (macOS) or PowerShell (Windows) and press Enter to open it.
- Move to your Desktop by running this command:
cd ~/Desktop- Create your project folder and move into it by running these commands:
mkdir order-processing
cd order-processing- Confirm the folder exists by running:
lsYou should see the order-processing folder listed in the output.
- Create a new file called docker-compose.yml by running:
touch docker-compose.yml- Open docker-compose.yml in your preferred text editor and paste the following content:
version: '3.8'
services:
kafka:
image: apache/kafka:3.9.2
ports:
- '9092:9092'
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
postgres:
image: postgres:16
ports:
- '5432:5432'
environment:
POSTGRES_DB: orderdb
POSTGRES_USER: orderuser
POSTGRES_PASSWORD: orderpass
volumes:
- postgres_data:/var/lib/postgresql/data
localstack:
image: localstack/localstack:latest
ports:
- '4566:4566'
environment:
LOCALSTACK_AUTH_TOKEN: ${LOCALSTACK_AUTH_TOKEN}
ENFORCE_IAM: '1'
SERVICES: sqs,iam
volumes:
- localstack_data:/var/lib/localstack
volumes:
postgres_data:
localstack_data:What does this file do?
- kafka runs Apache Kafka 3.9.2 in KRaft combined mode. This means it acts as both broker and controller with no ZooKeeper needed.
- postgres runs PostgreSQL 16 with a database called orderdb pre-created on startup.
- localstack emulates AWS services (SQS and IAM) locally on port 4566. The ENFORCE_IAM=1 setting enables IAM policy evaluation so access control actually works.
- The ${LOCALSTACK_AUTH_TOKEN} reference pulls your token from the shell environment so it never gets committed to version control.
Docker Compose needs your LocalStack auth token available as an environment variable. You will create a .env file in the project folder that Docker Compose reads automatically.
- Create a .env file in the order-processing folder by running:
echo "LOCALSTACK_AUTH_TOKEN=your-localstack-auth-token" > .env- Open .env in your text editor and replace your-localstack-auth-token with the actual token from your LocalStack account.
Keep .env out of version control
Never commit .env files containing tokens to Git. If you initialize a repository later, add .env to your .gitignore immediately.
- Make sure Docker Desktop is running. Then start all three containers by running:
docker compose up -d- Verify all containers are healthy by running:
docker compose psYou should see three containers listed with a status of Up or Running.
Containers not starting?
- If you see a message about LOCALSTACK_AUTH_TOKEN being unset, check that your .env file is in the same folder as docker-compose.yml and contains the correct token.
- If port 5432 is already in use, you may have a local PostgreSQL instance running. Stop it or change the host port in the compose file.
- If Docker Desktop is not running, start it first. Docker Compose requires the Docker daemon.
Help me troubleshoot my Docker Compose containers not starting
✔️ Awesome, I've got everything!
Great. Make sure you have saved docker-compose.yml and confirmed all three containers show Up status.
ⓧ I'd like to double check the full code
version: '3.8'
services:
kafka:
image: apache/kafka:3.9.2
ports:
- '9092:9092'
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
postgres:
image: postgres:16
ports:
- '5432:5432'
environment:
POSTGRES_DB: orderdb
POSTGRES_USER: orderuser
POSTGRES_PASSWORD: orderpass
volumes:
- postgres_data:/var/lib/postgresql/data
localstack:
image: localstack/localstack:latest
ports:
- '4566:4566'
environment:
LOCALSTACK_AUTH_TOKEN: ${LOCALSTACK_AUTH_TOKEN}
ENFORCE_IAM: '1'
SERVICES: sqs,iam
volumes:
- localstack_data:/var/lib/localstack
volumes:
postgres_data:
localstack_data:Generate the Spring Boot project from Spring Initializr
Spring Initializr generates a ready-to-run Maven project with all the boilerplate handled for you. You will configure it with the exact dependencies this project needs.
- Open your browser and navigate to https://start.spring.io.
- Under Project, select Maven.
- Under Language, select Java.
- Under Spring Boot, select 3.5.15.
- Under Project Metadata, set Group to com.nextwork.
- Set Artifact to order-processing.
- Set Java version to 21.
- Click the Add Dependencies button and add each of the following one at a time:
- Type Spring Web and select it.
- Type Spring Security and select it.
- Type Spring for Apache Kafka and select it.
- Type Spring Data JPA and select it.
- Type PostgreSQL Driver and select it.
- Type Validation and select it.
- Click Generate to download the project as a zip file.
What does Spring Initializr give you?
The downloaded zip contains a complete Maven project with a pom.xml already configured with Spring Boot 3.5.15 as the parent. All six dependencies are declared so you do not need to add them manually.
The Spring Boot parent POM manages versions for these core starters automatically. You only specify versions for third-party libraries that are not part of the Spring Boot BOM.
Now you need to extract the generated project into your existing order-processing folder (which already contains docker-compose.yml).
macOS
- Extract the zip into your project folder by running:
unzip -o ~/Downloads/order-processing.zip -d ~/Desktop/This merges the generated project files (pom.xml, src/, mvnw) into your existing order-processing folder alongside docker-compose.yml.
Windows
- Extract the zip into your project folder by running in PowerShell:
Expand-Archive -Path $HOME\Downloads\order-processing.zip -DestinationPath $HOME\Desktop\ -ForceThis merges the generated project files (pom.xml, src\, mvnw) into your existing order-processing folder alongside docker-compose.yml.
- Verify the merge worked by running:
ls ~/Desktop/order-processingYou should see docker-compose.yml, pom.xml, src, and mvnw all in the same folder.
- Open IntelliJ IDEA. Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
- Type IntelliJ and press Enter to open it.
- Click Open on the welcome screen.
- Navigate to Desktop and select the order-processing folder.
- Click Open.
IntelliJ detects the pom.xml and imports it as a Maven project automatically. Wait for the indexing and dependency download to complete (you will see a progress bar in the bottom status bar).
IntelliJ not recognizing the Maven project?
- If IntelliJ does not detect Maven automatically, right-click on pom.xml in the Project pane and select Add as Maven Project.
- If dependencies fail to download, check your internet connection and click the Reload All Maven Projects button (circular arrows icon) in the Maven tool window.
Help me fix IntelliJ not recognizing my Spring Boot Maven project
Configure application properties and add remaining dependencies
Spring Initializr gave you the core Spring starters, but this project also needs the AWS SDK for SQS and a JWT library for authentication. You will add these to pom.xml and create the application.yml configuration file.
- In IntelliJ, open pom.xml from the Project pane on the left.
- Find the <dependencies> section and add the following four dependencies inside it, after the existing Spring starters:
<!-- AWS SDK for SQS -->
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>sqs</artifactId>
<version>2.25.0</version>
</dependency>
<!-- JWT support for authentication -->
<dependency>
<groupId>io.jsonwebtoken</groupId>
<artifactId>jjwt-api</artifactId>
<version>0.11.5</version>
</dependency>
<dependency>
<groupId>io.jsonwebtoken</groupId>
<artifactId>jjwt-impl</artifactId>
<version>0.11.5</version>
</dependency>
<dependency>
<groupId>io.jsonwebtoken</groupId>
<artifactId>jjwt-jackson</artifactId>
<version>0.11.5</version>
</dependency>Why do these need explicit versions?
Spring Boot's parent POM manages versions for its own starters (Spring Web, Spring Kafka, etc.), so you omit versions for those. Third-party libraries like the AWS SDK and JJWT are not managed by the Spring Boot BOM, so you must pin their versions explicitly.
- Save pom.xml.
- Click the Reload All Maven Projects button (circular arrows icon) in the Maven tool window on the right side to download the new dependencies.
Now you will create the application configuration file that tells Spring Boot how to connect to Kafka, PostgreSQL, and LocalStack.
- In the Project pane, expand src > main > resources.
- If an application.properties file exists, right-click it and select Refactor > Rename to rename it to application.yml.
- Replace the contents of application.yml with the following configuration:
spring:
datasource:
url: jdbc:postgresql://localhost:5432/orderdb
username: orderuser
password: orderpass
jpa:
hibernate:
ddl-auto: update
show-sql: true
kafka:
bootstrap-servers: localhost:9092
consumer:
group-id: order-group
auto-offset-reset: earliest
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
properties:
spring.json.trusted.packages: '*'
producer:
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
app:
jwt:
expiration-ms: 3600000
aws:
endpoint: http://localhost:4566
region: us-east-1
access-key: test
secret-key: testWhat does this configuration do?
- spring.datasource connects to the PostgreSQL container using the credentials defined in docker-compose.yml.
- spring.jpa.hibernate.ddl-auto: update tells Hibernate to create or modify database tables automatically based on your entity classes.
- spring.kafka configures the Kafka producer with JSON serialization and the consumer with JSON deserialization. The trusted.packages setting allows deserialization of any class.
- app.jwt sets a one-hour token expiration. The RSA key pair for asymmetric signing is generated at application startup (you will configure this in the next step).
- aws points at LocalStack on port 4566 with dummy credentials (LocalStack accepts any credentials when ENFORCE_IAM is set to control access via policies, not credential validity).
- Save application.yml.
Time to verify everything works together.
- Make sure your Docker containers are still running. Return to your terminal and run:
docker compose ps- In IntelliJ, open the OrderProcessingApplication.java file (located in src/main/java/com/nextwork/orderprocessing/).
- Click the green Run arrow next to the main method, or press Ctrl+Shift+F10 (Windows) / Ctrl+Shift+R (macOS).
The application should start and you should see log output confirming Kafka and PostgreSQL connections. Look for lines mentioning the Kafka bootstrap server and Hibernate creating tables.
What should I see?
In the console output, look for:
- A log line containing Started OrderProcessingApplication confirming the app booted successfully.
- Hibernate SQL statements showing table creation (because of show-sql: true).
- No connection-refused errors for port 9092 or 5432.
💡 Spring Security blocks all requests by default
You may notice a generated security password in the console. Spring Security locks down all endpoints by default until you configure it. You will override this behavior in the next step.
🙋♀️ Application fails to start?
- If you see a connection refused error on port 5432, your PostgreSQL container may not be running. Run docker compose ps and restart any stopped containers with docker compose up -d.
- If you see a Kafka broker unavailable error, give the Kafka container 10-15 seconds after starting before running the app. KRaft mode needs time to elect a controller.
- If you see an error about missing jjwt classes, make sure you reloaded Maven after adding the dependencies to pom.xml.
Help me debug my Spring Boot application failing to connect to Kafka or PostgreSQL
✔️ Awesome, I've got everything!
Great. Double check you have saved both pom.xml and application.yml and that your application is running without errors.
ⓧ I'd like to double check the full code
Here is the pom.xml dependencies section (your file will have additional XML boilerplate from Spring Initializr):
<project>
<groupId>com.nextwork</groupId>
<artifactId>order-processing</artifactId>
<version>0.0.1-SNAPSHOT</version>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>3.5.15</version>
</parent>
<dependencies>
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-web</artifactId></dependency>
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-security</artifactId></dependency>
<dependency><groupId>org.springframework.kafka</groupId><artifactId>spring-kafka</artifactId></dependency>
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-data-jpa</artifactId></dependency>
<dependency><groupId>org.postgresql</groupId><artifactId>postgresql</artifactId></dependency>
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-validation</artifactId></dependency>
<dependency><groupId>software.amazon.awssdk</groupId><artifactId>sqs</artifactId><version>2.25.0</version></dependency>
<dependency><groupId>io.jsonwebtoken</groupId><artifactId>jjwt-api</artifactId><version>0.11.5</version></dependency>
<dependency><groupId>io.jsonwebtoken</groupId><artifactId>jjwt-impl</artifactId><version>0.11.5</version></dependency>
<dependency><groupId>io.jsonwebtoken</groupId><artifactId>jjwt-jackson</artifactId><version>0.11.5</version></dependency>
</dependencies>
</project>Here is the full application.yml:
spring:
datasource:
url: jdbc:postgresql://localhost:5432/orderdb
username: orderuser
password: orderpass
jpa:
hibernate:
ddl-auto: update
show-sql: true
kafka:
bootstrap-servers: localhost:9092
consumer:
group-id: order-group
auto-offset-reset: earliest
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
properties:
spring.json.trusted.packages: '*'
producer:
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
app:
jwt:
expiration-ms: 3600000
aws:
endpoint: http://localhost:4566
region: us-east-1
access-key: test
secret-key: testYour local infrastructure is running and your Spring Boot application connects to all three services. Next up, you will build a secured REST API with JWT authentication to accept order requests.
Build the REST API with Spring Security JWT
Your infrastructure is running and your Spring Boot project connects to both Kafka and PostgreSQL. Now it's time to build the core of your system: a secured REST API.
Any production API needs authentication. Without it, anyone can create orders in your system. You'll use Spring Security with asymmetric JWT tokens (RSA-signed) to ensure only authenticated users can access your order endpoints. The private key signs tokens and the public key verifies them.
In this step, get ready to:
- Create a JWT filter and security configuration that locks down your API.
- Build a token endpoint, order entity, and order creation endpoint.
- Test the secured API to confirm authentication works end-to-end.
Create the JWT filter
Before configuring security rules, you need a filter that intercepts every request, extracts the JWT from the Authorization header, and verifies its RSA signature against the public key. If valid, it sets the authenticated user in Spring's security context.
- In IntelliJ IDEA, create a new package com.nextwork.orderprocessing.security inside src/main/java.
- Create a new Java class called JwtFilter in the security package with the following code:
package com.nextwork.orderprocessing.security;
import io.jsonwebtoken.Claims;
import io.jsonwebtoken.Jwts;
import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.springframework.security.authentication.UsernamePasswordAuthenticationToken;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;
import java.io.IOException;
import java.security.KeyPair;
import java.util.List;
@Component
public class JwtFilter extends OncePerRequestFilter {
// RSA key pair - the public key verifies JWT signatures
private final KeyPair keyPair;
public JwtFilter(KeyPair keyPair) {
this.keyPair = keyPair;
}
@Override
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain chain)
throws ServletException, IOException {
// Extract the Authorization header
String header = request.getHeader("Authorization");
if (header != null && header.startsWith("Bearer ")) {
String token = header.substring(7);
try {
// Verify the JWT signature using the RSA public key
Claims claims = Jwts.parserBuilder()
.setSigningKey(keyPair.getPublic())
.build()
.parseClaimsJws(token)
.getBody();
// Set the authenticated user in Spring's security context
UsernamePasswordAuthenticationToken auth =
new UsernamePasswordAuthenticationToken(claims.getSubject(), null, List.of());
SecurityContextHolder.getContext().setAuthentication(auth);
} catch (Exception ignored) {
// Invalid token - let the request continue unauthenticated
}
}
// Continue the filter chain regardless
chain.doFilter(request, response);
}
}What does this code do?
This filter runs once per request. It checks the Authorization header for a Bearer token.
If a valid token exists, the filter verifies its signature using the RSA public key from the injected KeyPair. Only the matching private key can produce a valid signature. If verification passes, the filter extracts the username (subject claim) and sets it as the authenticated principal.
- Save JwtFilter.java.
Create the security configuration
Now you need to tell Spring Security which endpoints are public and which require a valid JWT. The SecurityConfig class wires everything together.
- Create a new Java class called SecurityConfig in the security package with the following code:
package com.nextwork.orderprocessing.security;
import io.jsonwebtoken.SignatureAlgorithm;
import io.jsonwebtoken.security.Keys;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
import org.springframework.security.config.http.SessionCreationPolicy;
import org.springframework.security.web.SecurityFilterChain;
import org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter;
import java.security.KeyPair;
@Configuration
@EnableWebSecurity
public class SecurityConfig {
// Inject the JWT filter created in the previous step
private final JwtFilter jwtFilter;
public SecurityConfig(JwtFilter jwtFilter) {
this.jwtFilter = jwtFilter;
}
@Bean
public static KeyPair jwtKeyPair() {
// Generate an RSA-2048 key pair for asymmetric JWT signing and verification
return Keys.keyPairFor(SignatureAlgorithm.RS256);
}
@Bean
public SecurityFilterChain filterChain(HttpSecurity http) throws Exception {
http
// Disable CSRF since this is a stateless REST API using tokens
.csrf(csrf -> csrf.disable())
// No server-side sessions - each request carries its own JWT
.sessionManagement(sm -> sm.sessionCreationPolicy(SessionCreationPolicy.STATELESS))
// Define authorization rules
.authorizeHttpRequests(auth -> auth
.requestMatchers("/api/auth/token").permitAll()
.requestMatchers("/api/orders/**").authenticated()
.anyRequest().authenticated()
)
// Insert the JWT filter before Spring's default username/password filter
.addFilterBefore(jwtFilter, UsernamePasswordAuthenticationFilter.class);
return http.build();
}
}What does this configuration do?
The jwtKeyPair() bean generates an RSA-2048 key pair at startup using JJWT's Keys utility. The private key signs new tokens and the public key verifies them. The method is static so Spring can create the bean without instantiating SecurityConfig first, avoiding a circular dependency with JwtFilter.
The SecurityFilterChain bean defines three key policies. CSRF protection is disabled because this is a stateless API that uses Bearer tokens instead of cookies. Session management is set to STATELESS so Spring never creates an HTTP session.
The authorization rules permit unauthenticated access to /api/auth/token (so users can get a token) and require authentication for everything else. The addFilterBefore call ensures your JwtFilter runs before Spring's default authentication filter.
- Save SecurityConfig.java.
✔️ Awesome, I've got everything!
Great. Double check you've saved both files in the security package.
ⓧ I'd like to double check the full code
Here is the full JwtFilter.java:
package com.nextwork.orderprocessing.security;
import io.jsonwebtoken.Claims;
import io.jsonwebtoken.Jwts;
import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.springframework.security.authentication.UsernamePasswordAuthenticationToken;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;
import java.io.IOException;
import java.security.KeyPair;
import java.util.List;
@Component
public class JwtFilter extends OncePerRequestFilter {
private final KeyPair keyPair;
public JwtFilter(KeyPair keyPair) {
this.keyPair = keyPair;
}
@Override
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain chain)
throws ServletException, IOException {
String header = request.getHeader("Authorization");
if (header != null && header.startsWith("Bearer ")) {
String token = header.substring(7);
try {
Claims claims = Jwts.parserBuilder()
.setSigningKey(keyPair.getPublic())
.build()
.parseClaimsJws(token)
.getBody();
UsernamePasswordAuthenticationToken auth =
new UsernamePasswordAuthenticationToken(claims.getSubject(), null, List.of());
SecurityContextHolder.getContext().setAuthentication(auth);
} catch (Exception ignored) {}
}
chain.doFilter(request, response);
}
}Here is the full SecurityConfig.java:
package com.nextwork.orderprocessing.security;
import io.jsonwebtoken.SignatureAlgorithm;
import io.jsonwebtoken.security.Keys;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
import org.springframework.security.config.http.SessionCreationPolicy;
import org.springframework.security.web.SecurityFilterChain;
import org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter;
import java.security.KeyPair;
@Configuration
@EnableWebSecurity
public class SecurityConfig {
private final JwtFilter jwtFilter;
public SecurityConfig(JwtFilter jwtFilter) { this.jwtFilter = jwtFilter; }
@Bean
public static KeyPair jwtKeyPair() {
return Keys.keyPairFor(SignatureAlgorithm.RS256);
}
@Bean
public SecurityFilterChain filterChain(HttpSecurity http) throws Exception {
http
.csrf(csrf -> csrf.disable())
.sessionManagement(sm -> sm.sessionCreationPolicy(SessionCreationPolicy.STATELESS))
.authorizeHttpRequests(auth -> auth
.requestMatchers("/api/auth/token").permitAll()
.requestMatchers("/api/orders/**").authenticated()
.anyRequest().authenticated()
)
.addFilterBefore(jwtFilter, UsernamePasswordAuthenticationFilter.class);
return http.build();
}
}Application fails to start?
Make sure the @Component annotation is on JwtFilter. Without it, Spring cannot inject the filter into SecurityConfig.
Check that both files are in the com.nextwork.orderprocessing.security package. Spring Boot only scans subpackages of your main application class.
help me debug my Spring Security configuration
Create the token endpoint and order API
With security in place, you need two things: an endpoint that issues JWT tokens, and the order creation endpoint that security protects. You also need the Order entity and repository to persist orders in PostgreSQL.
- Create a new package com.nextwork.orderprocessing.controller inside src/main/java.
- Create a new Java class called TokenController in the controller package with the following code:
package com.nextwork.orderprocessing.controller;
import io.jsonwebtoken.Jwts;
import io.jsonwebtoken.SignatureAlgorithm;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.web.bind.annotation.*;
import java.security.KeyPair;
import java.util.Date;
import java.util.Map;
@RestController
@RequestMapping("/api/auth")
public class TokenController {
// RSA key pair - the private key signs new tokens
private final KeyPair keyPair;
// Token expiration time in milliseconds (default: 1 hour)
@Value("${app.jwt.expiration-ms}")
private long expirationMs;
public TokenController(KeyPair keyPair) {
this.keyPair = keyPair;
}
@PostMapping("/token")
public Map<String, String> getToken(@RequestBody Map<String, String> body) {
// Extract the username from the request body
String username = body.get("username");
// Build a JWT signed with the RSA private key
String token = Jwts.builder()
.setSubject(username)
.setIssuedAt(new Date())
.setExpiration(new Date(System.currentTimeMillis() + expirationMs))
.signWith(keyPair.getPrivate(), SignatureAlgorithm.RS256)
.compact();
return Map.of("token", token);
}
}What does this code do?
The TokenController is a simple token issuer. It accepts a POST request with a username field and returns a signed JWT. The token contains the username as the subject, an issued-at timestamp, and an expiration time.
The .signWith(keyPair.getPrivate(), SignatureAlgorithm.RS256) call signs the token with the RSA private key using the RS256 algorithm. Only the corresponding public key (used by JwtFilter) can verify this signature. This asymmetric approach means the signing key never needs to be shared with services that only verify tokens.
- Save TokenController.java.
Next, create the Order entity that maps to a database table.
- Create a new package com.nextwork.orderprocessing.model inside src/main/java.
- Create a new Java class called Order in the model package with the following code:
package com.nextwork.orderprocessing.model;
import jakarta.persistence.*;
import java.math.BigDecimal;
import java.time.LocalDateTime;
@Entity
@Table(name = "orders")
public class Order {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
// Customer placing the order
private String customerName;
// Product being ordered
private String product;
// Number of items
private int quantity;
// Price per unit
private BigDecimal price;
// Order lifecycle status (CREATED, COMPLETED, FAILED)
private String status;
// Timestamp of order creation
private LocalDateTime createdAt;
// Getters and setters
public Long getId() { return id; }
public void setId(Long id) { this.id = id; }
public String getCustomerName() { return customerName; }
public void setCustomerName(String customerName) { this.customerName = customerName; }
public String getProduct() { return product; }
public void setProduct(String product) { this.product = product; }
public int getQuantity() { return quantity; }
public void setQuantity(int quantity) { this.quantity = quantity; }
public BigDecimal getPrice() { return price; }
public void setPrice(BigDecimal price) { this.price = price; }
public String getStatus() { return status; }
public void setStatus(String status) { this.status = status; }
public LocalDateTime getCreatedAt() { return createdAt; }
public void setCreatedAt(LocalDateTime createdAt) { this.createdAt = createdAt; }
}What does this entity represent?
The @Entity annotation marks this class as a JPA entity mapped to the orders table. Spring Data JPA uses the ddl-auto: update setting in application.yml to automatically create this table when the application starts.
The status field tracks the order through its lifecycle: CREATED when first submitted, COMPLETED after successful processing, and FAILED if something goes wrong.
- Save Order.java.
Now create the repository and controller that use this entity.
- Create a new package com.nextwork.orderprocessing.repository inside src/main/java.
- Create a new Java interface called OrderRepository in the repository package with the following code:
package com.nextwork.orderprocessing.repository;
import com.nextwork.orderprocessing.model.Order;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.stereotype.Repository;
// Spring Data JPA generates all CRUD methods automatically
@Repository
public interface OrderRepository extends JpaRepository<Order, Long> {
}- Save OrderRepository.java.
- Create a new Java class called OrderController in the controller package with the following code:
package com.nextwork.orderprocessing.controller;
import com.nextwork.orderprocessing.model.Order;
import com.nextwork.orderprocessing.repository.OrderRepository;
import jakarta.validation.Valid;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;
import java.time.LocalDateTime;
@RestController
@RequestMapping("/api/orders")
public class OrderController {
private final OrderRepository orderRepository;
public OrderController(OrderRepository orderRepository) {
this.orderRepository = orderRepository;
}
@PostMapping
public ResponseEntity<Order> createOrder(@Valid @RequestBody Order order) {
// Set initial status and timestamp before persisting
order.setStatus("CREATED");
order.setCreatedAt(LocalDateTime.now());
// Save the order to PostgreSQL
Order saved = orderRepository.save(order);
return ResponseEntity.status(HttpStatus.CREATED).body(saved);
}
}What does this code do?
The OrderController exposes a single POST endpoint at /api/orders. When a valid request arrives with a JWT, the controller sets the order status to CREATED, timestamps it, and saves it to PostgreSQL via OrderRepository.
The @Valid annotation enables bean validation on the request body. The ResponseEntity.status(HttpStatus.CREATED) returns a 201 status code, which is the correct HTTP response for resource creation.
- Save OrderController.java.
- Restart your Spring Boot application in IntelliJ IDEA.
- You should see the application start without errors and the console should show Hibernate creating the orders table.
✔️ Awesome, I've got everything!
Great. Double check you've saved all four files across the controller, model, and repository packages.
ⓧ I'd like to double check the full code
Here is TokenController.java:
package com.nextwork.orderprocessing.controller;
import io.jsonwebtoken.Jwts;
import io.jsonwebtoken.SignatureAlgorithm;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.web.bind.annotation.*;
import java.security.KeyPair;
import java.util.Date;
import java.util.Map;
@RestController
@RequestMapping("/api/auth")
public class TokenController {
private final KeyPair keyPair;
@Value("${app.jwt.expiration-ms}")
private long expirationMs;
public TokenController(KeyPair keyPair) {
this.keyPair = keyPair;
}
@PostMapping("/token")
public Map<String, String> getToken(@RequestBody Map<String, String> body) {
String username = body.get("username");
String token = Jwts.builder()
.setSubject(username)
.setIssuedAt(new Date())
.setExpiration(new Date(System.currentTimeMillis() + expirationMs))
.signWith(keyPair.getPrivate(), SignatureAlgorithm.RS256)
.compact();
return Map.of("token", token);
}
}Here is Order.java:
package com.nextwork.orderprocessing.model;
import jakarta.persistence.*;
import java.math.BigDecimal;
import java.time.LocalDateTime;
@Entity
@Table(name = "orders")
public class Order {
@Id @GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
private String customerName;
private String product;
private int quantity;
private BigDecimal price;
private String status;
private LocalDateTime createdAt;
public Long getId() { return id; }
public void setId(Long id) { this.id = id; }
public String getCustomerName() { return customerName; }
public void setCustomerName(String customerName) { this.customerName = customerName; }
public String getProduct() { return product; }
public void setProduct(String product) { this.product = product; }
public int getQuantity() { return quantity; }
public void setQuantity(int quantity) { this.quantity = quantity; }
public BigDecimal getPrice() { return price; }
public void setPrice(BigDecimal price) { this.price = price; }
public String getStatus() { return status; }
public void setStatus(String status) { this.status = status; }
public LocalDateTime getCreatedAt() { return createdAt; }
public void setCreatedAt(LocalDateTime createdAt) { this.createdAt = createdAt; }
}Here is OrderRepository.java:
package com.nextwork.orderprocessing.repository;
import com.nextwork.orderprocessing.model.Order;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.stereotype.Repository;
@Repository
public interface OrderRepository extends JpaRepository<Order, Long> {
}Here is OrderController.java:
package com.nextwork.orderprocessing.controller;
import com.nextwork.orderprocessing.model.Order;
import com.nextwork.orderprocessing.repository.OrderRepository;
import jakarta.validation.Valid;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;
import java.time.LocalDateTime;
@RestController
@RequestMapping("/api/orders")
public class OrderController {
private final OrderRepository orderRepository;
public OrderController(OrderRepository orderRepository) { this.orderRepository = orderRepository; }
@PostMapping
public ResponseEntity<Order> createOrder(@Valid @RequestBody Order order) {
order.setStatus("CREATED");
order.setCreatedAt(LocalDateTime.now());
Order saved = orderRepository.save(order);
return ResponseEntity.status(HttpStatus.CREATED).body(saved);
}
}Seeing a table creation error?
If Hibernate fails to create the orders table, make sure your PostgreSQL container is running. Run docker compose ps and confirm the postgres container shows a healthy status.
Also verify the datasource URL, username, and password in application.yml match the environment variables in docker-compose.yml (orderdb, orderuser, orderpass).
help me fix my PostgreSQL connection
Test the secured API
Time to verify the security layer works. You'll use curl to request a token, then use that token to create an order. You'll also confirm that unauthenticated requests are rejected.
- Make sure your Spring Boot application is running.
- Open a terminal and test that unauthenticated requests are rejected by running:
curl -X POST http://localhost:8080/api/orders \
-H "Content-Type: application/json" \
-d '{"customerName":"Alice","product":"Widget","quantity":5,"price":9.99}'- You should see a 401 Unauthorized response. This confirms Spring Security is blocking unauthenticated access.
- Request a JWT token by running:
curl -X POST http://localhost:8080/api/auth/token \
-H "Content-Type: application/json" \
-d '{"username":"alice"}'- You should see a JSON response containing a token field with a long encoded string.
- Copy the token value and use it to create an order by running the following (replace YOUR_TOKEN with the actual token):
curl -X POST http://localhost:8080/api/orders \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_TOKEN" \
-d '{"customerName":"Alice","product":"Widget","quantity":5,"price":9.99}'- You should see a 201 response with the order object including an id, status: "CREATED", and a createdAt timestamp.
What should I see?
The successful response looks like this (your id and timestamp will differ):
{
"id": 1,
"customerName": "Alice",
"product": "Widget",
"quantity": 5,
"price": 9.99,
"status": "CREATED",
"createdAt": "2026-06-16T14:30:00"
}🙋♀️ Getting 403 instead of 401?
Spring Security returns 403 Forbidden when CSRF is not disabled. Confirm your SecurityConfig includes .csrf(csrf -> csrf.disable()) and restart the application.
If the token endpoint returns 401, check that .requestMatchers("/api/auth/token").permitAll() is in your security configuration. The order of matchers matters.
help me debug my JWT authentication flow
Your API is locked down and working. Next up, you'll connect Kafka to publish order events and build a consumer that processes them.
Add Kafka Producer and Consumer with Dead-Letter Topic
Your REST API persists orders to PostgreSQL, but right now nothing else happens after that. In a real system, other services need to react to new orders.
Event-driven architecture solves this by publishing events to a message broker whenever something important happens. In this step, you will wire Apache Kafka into your application as the event backbone. Your controller will publish order events, and a consumer will process them with automatic retry and dead-letter routing for failures.
In this step, get ready to:
- Configure a Kafka producer that publishes order events after each order is created.
- Build a consumer that processes order events and updates order status.
- Add retry logic and dead-letter topic routing for failed events.
Create the event model and Kafka producer
Before publishing events, you need a data class that represents the event payload. This OrderEvent class carries the essential order information through Kafka.
- Create a new file OrderEvent.java inside the src/main/java/com/nextwork/orderprocessing/model/ package:
package com.nextwork.orderprocessing.model;
import java.math.BigDecimal;
import java.time.LocalDateTime;
// Carries order data through Kafka as a serializable event
public class OrderEvent {
private Long orderId;
private String product;
private int quantity;
private BigDecimal price;
private String status;
private LocalDateTime timestamp;
// Getters and setters
public Long getOrderId() { return orderId; }
public void setOrderId(Long orderId) { this.orderId = orderId; }
public String getProduct() { return product; }
public void setProduct(String product) { this.product = product; }
public int getQuantity() { return quantity; }
public void setQuantity(int quantity) { this.quantity = quantity; }
public BigDecimal getPrice() { return price; }
public void setPrice(BigDecimal price) { this.price = price; }
public String getStatus() { return status; }
public void setStatus(String status) { this.status = status; }
public LocalDateTime getTimestamp() { return timestamp; }
public void setTimestamp(LocalDateTime timestamp) { this.timestamp = timestamp; }
}What does this class do?
OrderEvent is a plain Java object (POJO) that Spring Kafka serializes to JSON before sending it to the broker. It mirrors the key fields from the Order entity but is decoupled from the database model.
This separation means you can evolve your database schema and event schema independently.
- Save OrderEvent.java.
Next, you need a Kafka configuration class that creates a KafkaTemplate bean with a ProducerListener for logging.
- Create a new package kafka under src/main/java/com/nextwork/orderprocessing/ and add a file KafkaConfig.java:
package com.nextwork.orderprocessing.kafka;
import com.nextwork.orderprocessing.model.OrderEvent;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.core.ProducerFactory;
import org.springframework.kafka.support.ProducerListener;
@Configuration
public class KafkaConfig {
private static final Logger log = LoggerFactory.getLogger(KafkaConfig.class);
// Creates a KafkaTemplate with a listener that logs publish outcomes
@Bean
public KafkaTemplate<String, OrderEvent> kafkaTemplate(ProducerFactory<String, OrderEvent> producerFactory) {
KafkaTemplate<String, OrderEvent> template = new KafkaTemplate<>(producerFactory);
template.setProducerListener(new ProducerListener<>() {
@Override
public void onSuccess(ProducerRecord<String, OrderEvent> record, RecordMetadata metadata) {
log.info("Published event for order {}", record.value().getOrderId());
}
@Override
public void onError(ProducerRecord<String, OrderEvent> record, RecordMetadata metadata, Exception ex) {
log.error("Failed to publish event for order {}: {}", record.value().getOrderId(), ex.getMessage());
}
});
return template;
}
}What does this code do?
The KafkaTemplate is Spring's high-level abstraction for sending messages to Kafka topics. Spring Boot auto-configures the ProducerFactory using the serializer settings in your application.yml.
The ProducerListener callbacks fire after each send attempt completes. This gives you immediate visibility into whether events are reaching the broker.
- Save KafkaConfig.java.
Now update OrderController to publish an event every time an order is created.
- In OrderController.java, replace the entire class with the following updated version that injects KafkaTemplate and publishes after persisting:
package com.nextwork.orderprocessing.controller;
import com.nextwork.orderprocessing.model.Order;
import com.nextwork.orderprocessing.model.OrderEvent;
import com.nextwork.orderprocessing.repository.OrderRepository;
import jakarta.validation.Valid;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.web.bind.annotation.*;
import java.time.LocalDateTime;
@RestController
@RequestMapping("/api/orders")
public class OrderController {
private final OrderRepository orderRepository;
private final KafkaTemplate<String, OrderEvent> kafkaTemplate;
public OrderController(OrderRepository orderRepository, KafkaTemplate<String, OrderEvent> kafkaTemplate) {
this.orderRepository = orderRepository;
this.kafkaTemplate = kafkaTemplate;
}
@PostMapping
public ResponseEntity<Order> createOrder(@Valid @RequestBody Order order) {
order.setStatus("CREATED");
order.setCreatedAt(LocalDateTime.now());
Order saved = orderRepository.save(order);
// Build an event from the persisted order and publish to Kafka
OrderEvent event = new OrderEvent();
event.setOrderId(saved.getId());
event.setProduct(saved.getProduct());
event.setQuantity(saved.getQuantity());
event.setPrice(saved.getPrice());
event.setStatus(saved.getStatus());
event.setTimestamp(saved.getCreatedAt());
kafkaTemplate.send("order-events", String.valueOf(saved.getId()), event);
return ResponseEntity.status(HttpStatus.CREATED).body(saved);
}
}What does this code do?
After saving the order to PostgreSQL, the controller builds an OrderEvent and publishes it to the order-events topic. The order ID is used as the Kafka message key, which ensures all events for the same order land on the same partition.
The kafkaTemplate.send() call is asynchronous. The ProducerListener you configured earlier logs the outcome once the broker acknowledges receipt.
- Save OrderController.java.
- Restart your Spring Boot application. In the console output, you should see a log line indicating the Kafka producer connected to localhost:9092.
Application fails to start?
Check that your Docker containers are still running with docker compose ps. The Kafka container must be healthy before the application can connect.
If you see a serialization error, confirm the producer.value-serializer in application.yml is set to org.springframework.kafka.support.serializer.JsonSerializer.
help me debug my Spring Boot Kafka producer connection
Build the consumer with dead-letter routing
Publishing events is only half the picture. You need a consumer that listens on order-events, processes valid orders, and gracefully handles failures.
- Create a new file OrderEventConsumer.java in the src/main/java/com/nextwork/orderprocessing/kafka/ package:
package com.nextwork.orderprocessing.kafka;
import com.nextwork.orderprocessing.model.OrderEvent;
import com.nextwork.orderprocessing.repository.OrderRepository;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
@Component
public class OrderEventConsumer {
private final OrderRepository orderRepository;
public OrderEventConsumer(OrderRepository orderRepository) {
this.orderRepository = orderRepository;
}
// Listens on order-events and processes each incoming event
@KafkaListener(topics = "order-events", groupId = "order-group")
public void consume(OrderEvent event) {
// Reject orders with quantity above 100 to simulate a business rule failure
if (event.getQuantity() > 100) {
throw new IllegalArgumentException("Quantity " + event.getQuantity() + " exceeds maximum of 100");
}
// On success, update order status to COMPLETED
orderRepository.findById(event.getOrderId()).ifPresent(order -> {
order.setStatus("COMPLETED");
orderRepository.save(order);
});
}
}What does this code do?
The @KafkaListener annotation tells Spring Kafka to poll the order-events topic and invoke consume() for each message.
If the quantity exceeds 100, the consumer throws an exception. This simulates a business validation failure that triggers retry logic. For valid events, the consumer looks up the order by ID and sets its status to COMPLETED.
- Save OrderEventConsumer.java.
The consumer works, but if an event causes an exception the default behaviour is to log and skip. You want controlled retries and dead-letter routing instead.
- Return to KafkaConfig.java and add the following kafkaListenerContainerFactory bean below the existing kafkaTemplate bean:
// Adds retry logic and dead-letter routing to the consumer
@Bean
public ConcurrentKafkaListenerContainerFactory<String, OrderEvent> kafkaListenerContainerFactory(
ConsumerFactory<String, OrderEvent> consumerFactory,
KafkaTemplate<String, OrderEvent> kafkaTemplate) {
ConcurrentKafkaListenerContainerFactory<String, OrderEvent> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory);
DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(kafkaTemplate);
DefaultErrorHandler errorHandler = new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3));
factory.setCommonErrorHandler(errorHandler);
return factory;
}What does this code do?
The DefaultErrorHandler intercepts exceptions thrown by your consumer. FixedBackOff(1000L, 3) means it retries up to 3 times with a 1-second delay between each attempt.
When all retries are exhausted, the DeadLetterPublishingRecoverer publishes the failed record to a dead-letter topic. By default, Spring Kafka appends -dlt to the original topic name, creating order-events-dlt.
💡 Why use a dead-letter topic?
In production, you cannot afford to drop failed messages silently. A dead-letter topic acts as a parking lot where poisoned messages wait for investigation or reprocessing.
This pattern keeps your main consumer healthy while giving you full visibility into failures.
- Add the required imports at the top of KafkaConfig.java:
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.core.ConsumerFactory;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.kafka.listener.DefaultErrorHandler;
import org.springframework.util.backoff.FixedBackOff;- Save KafkaConfig.java.
- Restart the application. You should see log lines confirming the consumer joined the order-group consumer group.
Consumer not starting?
Verify that spring.kafka.consumer.value-deserializer is set to org.springframework.kafka.support.serializer.JsonDeserializer in your application.yml.
Also confirm that spring.json.trusted.packages is set to '*' so the deserializer accepts your OrderEvent class.
help me debug my Kafka consumer configuration in Spring Boot
✔️ Awesome, I've got everything!
Great. Double check you have saved both KafkaConfig.java and OrderEventConsumer.java.
ⓧ I'd like to double check the full code
Here is the complete KafkaConfig.java with both beans:
package com.nextwork.orderprocessing.kafka;
import com.nextwork.orderprocessing.model.OrderEvent;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.core.ConsumerFactory;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.core.ProducerFactory;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.kafka.listener.DefaultErrorHandler;
import org.springframework.kafka.support.ProducerListener;
import org.springframework.util.backoff.FixedBackOff;
@Configuration
public class KafkaConfig {
private static final Logger log = LoggerFactory.getLogger(KafkaConfig.class);
@Bean
public KafkaTemplate<String, OrderEvent> kafkaTemplate(ProducerFactory<String, OrderEvent> producerFactory) {
KafkaTemplate<String, OrderEvent> template = new KafkaTemplate<>(producerFactory);
template.setProducerListener(new ProducerListener<>() {
@Override
public void onSuccess(ProducerRecord<String, OrderEvent> record, RecordMetadata metadata) {
log.info("Published event for order {}", record.value().getOrderId());
}
@Override
public void onError(ProducerRecord<String, OrderEvent> record, RecordMetadata metadata, Exception ex) {
log.error("Failed to publish event for order {}: {}", record.value().getOrderId(), ex.getMessage());
}
});
return template;
}
@Bean
public ConcurrentKafkaListenerContainerFactory<String, OrderEvent> kafkaListenerContainerFactory(
ConsumerFactory<String, OrderEvent> consumerFactory,
KafkaTemplate<String, OrderEvent> kafkaTemplate) {
ConcurrentKafkaListenerContainerFactory<String, OrderEvent> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory);
DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(kafkaTemplate);
DefaultErrorHandler errorHandler = new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3));
factory.setCommonErrorHandler(errorHandler);
return factory;
}
}And the complete OrderEventConsumer.java:
package com.nextwork.orderprocessing.kafka;
import com.nextwork.orderprocessing.model.OrderEvent;
import com.nextwork.orderprocessing.repository.OrderRepository;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
@Component
public class OrderEventConsumer {
private final OrderRepository orderRepository;
public OrderEventConsumer(OrderRepository orderRepository) {
this.orderRepository = orderRepository;
}
@KafkaListener(topics = "order-events", groupId = "order-group")
public void consume(OrderEvent event) {
if (event.getQuantity() > 100) {
throw new IllegalArgumentException("Quantity " + event.getQuantity() + " exceeds maximum of 100");
}
orderRepository.findById(event.getOrderId()).ifPresent(order -> {
order.setStatus("COMPLETED");
orderRepository.save(order);
});
}
}Verify the happy path and failure path
You now have a producer, a consumer, and error handling all wired up. Time to test both paths: a successful order and a failing one.
- First, request a fresh JWT token by running:
curl -X POST http://localhost:8080/api/auth/token \
-H "Content-Type: application/json" \
-d '{"username": "testuser"}'- Copy the token value from the response.
- Create a valid order (quantity 5) by running:
curl -X POST http://localhost:8080/api/orders \
-H "Content-Type: application/json" \
-H "Authorization: Bearer your-token-here" \
-d '{"customerName": "Alice", "product": "Widget", "quantity": 5, "price": 29.99}'- Check the application logs. You should see a log line saying the event was published, followed by the consumer processing it.
- Verify the order status changed to COMPLETED by querying PostgreSQL directly. Open a psql session by running:
docker exec -it order-processing-postgres-1 psql -U orderuser -d orderdb -c "SELECT id, status FROM orders;"What should I see?
You should see your order with status COMPLETED. This confirms the full round trip: API received the order, persisted it, published to Kafka, and the consumer updated its status.
- Now create a failing order with quantity above 100 by running:
curl -X POST http://localhost:8080/api/orders \
-H "Content-Type: application/json" \
-H "Authorization: Bearer your-token-here" \
-d '{"customerName": "Bob", "product": "MegaWidget", "quantity": 150, "price": 99.99}'- Watch the application logs. You should see the consumer attempt to process the event, fail, retry 3 times (with 1-second pauses), then forward the message to the dead-letter topic.
- Verify the order status remains CREATED in PostgreSQL (the consumer never processed it successfully):
docker exec -it order-processing-postgres-1 psql -U orderuser -d orderdb -c "SELECT id, status FROM orders;"What should I see?
The second order (Bob's MegaWidget) should still show status CREATED. This is correct. The consumer rejected it, and after 3 retries the event was routed to the dead-letter topic order-events-dlt.
In your application logs, look for lines showing the retry attempts and the final DLT publish.
🙋♀️ Order status went to COMPLETED when it should not have?
Make sure your quantity validation in OrderEventConsumer checks event.getQuantity() > 100. If you used >= 100 instead, adjust the threshold.
Also confirm you are sending quantity 150 in your curl request, not 15.
help me debug my Kafka dead-letter topic routing
Your event backbone is alive. Next up, you will forward those dead-letter events from Kafka into an AWS SQS queue using LocalStack, and lock down access with IAM policies.
Integrate AWS SQS Dead-Letter Queue with IAM via LocalStack
Your Kafka dead-letter topic catches failed events, but those failures are only visible to Kafka consumers. In production, operations teams need access to failed events through standard AWS tooling.
In this step, you will bridge the Kafka DLT to an SQS dead-letter queue running on LocalStack. You will also create an IAM user with least-privilege access, proving that only authorized credentials can read from the queue.
In this step, get ready to:
- Add the AWS IAM SDK dependency and configure SQS and IAM client beans.
- Create a DLT-to-SQS forwarder and an IAM setup service with least-privilege policies.
- Run the full end-to-end flow and verify IAM enforcement on the SQS queue.
Add the IAM SDK dependency and create AWS client configuration
Your pom.xml already includes the SQS SDK. You also need the IAM SDK to create users and attach policies programmatically.
- In IntelliJ IDEA, open pom.xml at the root of your project.
- Find the existing SQS dependency block and add the IAM dependency directly below it:
<dependency><groupId>software.amazon.awssdk</groupId><artifactId>iam</artifactId><version>2.25.0</version></dependency>What does this dependency do?
The software.amazon.awssdk:iam artifact gives you IamClient for creating IAM users and attaching policies. Combined with the SQS SDK already in your project, you can now manage both queue operations and access control from Java.
- Click the Maven reload icon (or press Cmd+Shift+I on macOS / Ctrl+Shift+O on Windows) to download the new dependency.
Next, create a configuration class that wires up both AWS clients pointing to LocalStack.
- Create a new package com.nextwork.orderprocessing.aws and add a file named AwsConfig.java.
- Add the following configuration class:
@Configuration
public class AwsConfig {
@Value("${aws.endpoint}")
private String endpoint;
@Value("${aws.region}")
private String region;
@Value("${aws.access-key}")
private String accessKey;
@Value("${aws.secret-key}")
private String secretKey;
@Bean
public SqsClient sqsClient() {
// Build an SQS client pointing to LocalStack
return SqsClient.builder()
.endpointOverride(URI.create(endpoint))
.region(Region.of(region))
.credentialsProvider(StaticCredentialsProvider.create(
AwsBasicCredentials.create(accessKey, secretKey)))
.build();
}
@Bean
public IamClient iamClient() {
// Build an IAM client pointing to LocalStack
return IamClient.builder()
.endpointOverride(URI.create(endpoint))
.region(Region.of(region))
.credentialsProvider(StaticCredentialsProvider.create(
AwsBasicCredentials.create(accessKey, secretKey)))
.build();
}
}What does this code do?
Both beans override the default AWS endpoint to http://localhost:4566 (LocalStack). They use StaticCredentialsProvider with the test credentials from your application.yml.
The test credentials act as root-level access in LocalStack. With ENFORCE_IAM=1 enabled in your Docker Compose, LocalStack evaluates IAM policies for any non-root credentials.
- Save AwsConfig.java.
Import errors in AwsConfig?
Make sure Maven finished downloading the IAM dependency. If IamClient is unresolved, click the Maven reload icon again and wait for the download to complete.
help me fix unresolved AWS SDK imports in my AwsConfig class
Create the DLT-to-SQS forwarder and IAM setup service
You need two new components. A Kafka listener that consumes from the DLT and pushes messages into SQS. An initialization service that creates the SQS queue and configures least-privilege IAM access on startup.
- In the kafka package, create a new file named DltToSqsForwarder.java.
- Add the following class:
@Component
public class DltToSqsForwarder {
private final SqsClient sqsClient;
private final ObjectMapper objectMapper;
@Value("${aws.endpoint}")
private String endpoint;
public DltToSqsForwarder(SqsClient sqsClient, ObjectMapper objectMapper) {
this.sqsClient = sqsClient;
this.objectMapper = objectMapper;
}
@KafkaListener(topics = "order-events.DLT", groupId = "dlt-group")
public void forwardToSqs(OrderEvent event) throws JsonProcessingException {
// Look up the SQS queue URL by name
String queueUrl = sqsClient.getQueueUrl(
GetQueueUrlRequest.builder().queueName("order-failures-dlq").build()).queueUrl();
// Send the failed event as JSON to SQS
sqsClient.sendMessage(SendMessageRequest.builder()
.queueUrl(queueUrl)
.messageBody(objectMapper.writeValueAsString(event))
.build());
log.info("Forwarded failed event for order {} to SQS DLQ", event.getOrderId());
}
}What does this code do?
The @KafkaListener annotation subscribes this method to the order-events.DLT topic with a separate consumer group (dlt-group). When a failed event arrives, it serializes the event to JSON and sends it to the order-failures-dlq SQS queue.
This uses the root SqsClient bean (with test credentials), so it has full permissions to send messages. The restricted dlq-reader user you create next will only be able to receive and delete.
- Save DltToSqsForwarder.java.
- Now create IamSetupService.java in the aws package:
@Service
public class IamSetupService {
private final IamClient iamClient;
private final SqsClient sqsClient;
public IamSetupService(IamClient iamClient, SqsClient sqsClient) {
this.iamClient = iamClient;
this.sqsClient = sqsClient;
}
@PostConstruct
public void setup() {
// Create SQS queue
CreateQueueResponse queueResponse = sqsClient.createQueue(
CreateQueueRequest.builder().queueName("order-failures-dlq").build());
String queueUrl = queueResponse.queueUrl();
String queueArn = sqsClient.getQueueAttributes(
GetQueueAttributesRequest.builder()
.queueUrl(queueUrl)
.attributeNames(QueueAttributeName.QUEUE_ARN)
.build()).attributes().get(QueueAttributeName.QUEUE_ARN);
// Create IAM user dlq-reader
iamClient.createUser(CreateUserRequest.builder().userName("dlq-reader").build());
// Attach least-privilege policy
String policyDoc = "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":[\"sqs:ReceiveMessage\",\"sqs:DeleteMessage\"],\"Resource\":\"" + queueArn + "\"}]}";
iamClient.putUserPolicy(PutUserPolicyRequest.builder()
.userName("dlq-reader")
.policyName("dlq-read-only")
.policyDocument(policyDoc)
.build());
}
}What does this code do?
The @PostConstruct method runs at application startup, before any Kafka messages are consumed. It creates the order-failures-dlq queue on LocalStack and retrieves its ARN.
It then creates an IAM user called dlq-reader and attaches an inline policy named dlq-read-only. This policy grants only sqs:ReceiveMessage and sqs:DeleteMessage on the specific queue ARN. Any attempt to sqs:SendMessage with dlq-reader credentials will be denied by LocalStack's IAM enforcement.
- Save IamSetupService.java.
Why least-privilege for the DLQ reader?
In production, the service that reads from a dead-letter queue is typically a separate operations tool or alerting pipeline. It should only be able to read and acknowledge messages.
Granting sqs:SendMessage to a reader would let a compromised consumer inject fake failures into the queue. Least-privilege prevents that lateral movement.
✔️ Awesome, I've got everything!
Great. Double check you have saved all three files before continuing.
ⓧ I'd like to double check the full code
Here is the complete AwsConfig.java:
@Configuration
public class AwsConfig {
@Value("${aws.endpoint}")
private String endpoint;
@Value("${aws.region}")
private String region;
@Value("${aws.access-key}")
private String accessKey;
@Value("${aws.secret-key}")
private String secretKey;
@Bean
public SqsClient sqsClient() {
return SqsClient.builder()
.endpointOverride(URI.create(endpoint))
.region(Region.of(region))
.credentialsProvider(StaticCredentialsProvider.create(
AwsBasicCredentials.create(accessKey, secretKey)))
.build();
}
@Bean
public IamClient iamClient() {
return IamClient.builder()
.endpointOverride(URI.create(endpoint))
.region(Region.of(region))
.credentialsProvider(StaticCredentialsProvider.create(
AwsBasicCredentials.create(accessKey, secretKey)))
.build();
}
}Here is the complete DltToSqsForwarder.java:
@Component
public class DltToSqsForwarder {
private final SqsClient sqsClient;
private final ObjectMapper objectMapper;
@Value("${aws.endpoint}")
private String endpoint;
public DltToSqsForwarder(SqsClient sqsClient, ObjectMapper objectMapper) {
this.sqsClient = sqsClient;
this.objectMapper = objectMapper;
}
@KafkaListener(topics = "order-events.DLT", groupId = "dlt-group")
public void forwardToSqs(OrderEvent event) throws JsonProcessingException {
String queueUrl = sqsClient.getQueueUrl(
GetQueueUrlRequest.builder().queueName("order-failures-dlq").build()).queueUrl();
sqsClient.sendMessage(SendMessageRequest.builder()
.queueUrl(queueUrl)
.messageBody(objectMapper.writeValueAsString(event))
.build());
log.info("Forwarded failed event for order {} to SQS DLQ", event.getOrderId());
}
}Here is the complete IamSetupService.java:
@Service
public class IamSetupService {
private final IamClient iamClient;
private final SqsClient sqsClient;
public IamSetupService(IamClient iamClient, SqsClient sqsClient) {
this.iamClient = iamClient;
this.sqsClient = sqsClient;
}
@PostConstruct
public void setup() {
// Create SQS queue
CreateQueueResponse queueResponse = sqsClient.createQueue(
CreateQueueRequest.builder().queueName("order-failures-dlq").build());
String queueUrl = queueResponse.queueUrl();
String queueArn = sqsClient.getQueueAttributes(
GetQueueAttributesRequest.builder()
.queueUrl(queueUrl)
.attributeNames(QueueAttributeName.QUEUE_ARN)
.build()).attributes().get(QueueAttributeName.QUEUE_ARN);
// Create IAM user dlq-reader
iamClient.createUser(CreateUserRequest.builder().userName("dlq-reader").build());
// Attach least-privilege policy
String policyDoc = "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":[\"sqs:ReceiveMessage\",\"sqs:DeleteMessage\"],\"Resource\":\"" + queueArn + "\"}]}";
iamClient.putUserPolicy(PutUserPolicyRequest.builder()
.userName("dlq-reader")
.policyName("dlq-read-only")
.policyDocument(policyDoc)
.build());
}
}Run the end-to-end flow and verify IAM enforcement
Time to prove the full pipeline works. You will create a failing order, watch it flow through Kafka retry to the DLT, and confirm it arrives in SQS. Then you will test that dlq-reader can receive but cannot send.
- Stop your Spring Boot application if it is running.
- Restart your Docker containers to ensure a clean LocalStack state by running:
docker compose down -v && docker compose up -d- Wait about 10 seconds for all containers to become healthy.
- Start the Spring Boot application from IntelliJ (click the green play button or press Shift+F10 on Windows / Ctrl+R on macOS).
- Watch the console for the startup log confirming the queue was created. You should see a line indicating IamSetupService ran successfully.
Now trigger a failing order through the secured API.
- Request a JWT token by running this in your terminal:
curl -s -X POST http://localhost:8080/api/auth/token \
-H "Content-Type: application/json" \
-d '{"username":"testuser"}'- Copy the token value from the response.
- Create a failing order (quantity exceeds 100) using your token:
curl -s -X POST http://localhost:8080/api/orders \
-H "Content-Type: application/json" \
-H "Authorization: Bearer your-token-here" \
-d '{"customerName":"Test User","product":"Widget","quantity":150,"price":9.99}'What should I see?
You should receive a 201 response with status CREATED. In the application logs, after a few seconds you will see the consumer reject the event 3 times (quantity 150 exceeds 100), then the dead-letter recoverer routes it to order-events.DLT.
Shortly after, the DltToSqsForwarder picks up the event and logs that it forwarded to SQS.
Now verify IAM enforcement from the command line using awslocal.
- Create access keys for the dlq-reader user by running:
awslocal iam create-access-key --user-name dlq-reader- Copy the AccessKeyId and SecretAccessKey from the JSON output.
- Export the credentials in your terminal so subsequent awslocal commands run as dlq-reader:
export AWS_ACCESS_KEY_ID=your-access-key-id
export AWS_SECRET_ACCESS_KEY=your-secret-access-key- Attempt to receive a message from the DLQ (this should succeed):
awslocal sqs receive-message --queue-url http://localhost:4566/000000000000/order-failures-dlqWhat should I see?
You should see a JSON response containing the failed order event in the Body field. The message includes the order ID, product, quantity, and price.
- Now attempt to send a message with the dlq-reader credentials (this should be denied):
awslocal sqs send-message \
--queue-url http://localhost:4566/000000000000/order-failures-dlq \
--message-body '{"test":"unauthorized"}'What should I see?
You should see an AccessDeniedException error. This proves that ENFORCE_IAM=1 is active and the dlq-reader user only has sqs:ReceiveMessage and sqs:DeleteMessage permissions.
- Unset the credentials to return to root access for further testing:
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEYReceive returns empty or send is not denied?
If receive-message returns nothing, the forwarder may not have processed the DLT event yet. Wait 5-10 seconds and try again.
If send-message succeeds instead of being denied, confirm that ENFORCE_IAM: '1' is set in your docker-compose.yml under the localstack service environment. Also verify you exported the dlq-reader credentials correctly.
help me debug why LocalStack IAM enforcement is not denying my SQS send-message request
Your event-driven pipeline is now complete. Failed orders flow securely from Kafka through to SQS, and IAM policies ensure only authorized consumers can access the dead-letter queue. Next up, you will automate deployment with a local CI/CD pipeline using GitHub Actions, Jenkins, and Kubernetes.
Deploy to Kubernetes via GitHub Actions and Jenkins Pipeline
Your event-driven pipeline handles orders, retries, and dead-letter routing. But right now, deploying the application means manually running docker compose up. In a real team, every code change should flow through an automated pipeline that builds, packages, and deploys without human intervention.
In this step, you will build a fully local CI/CD pipeline. You will use nektos/act to trigger a GitHub Actions workflow on your machine. That workflow calls a Jenkins pipeline running in Docker, which builds a container image of your app, packages it with a Helm chart, and deploys it to a local Kubernetes cluster powered by kind.
In this step, get ready to:
- Create a local Kubernetes cluster with kind and install Helm.
- Write a Helm chart and Jenkinsfile for the Spring Boot application.
- Trigger a GitHub Actions workflow locally with nektos/act to deploy via Jenkins.
Set up a kind cluster and install Helm
Before you can deploy to Kubernetes, you need a cluster. kind (Kubernetes in Docker) creates a full cluster inside Docker containers, which fits perfectly with your existing Docker-based setup.
- Install kind by running this command:
# Download and install kind
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.23.0/kind-$(uname -s | tr '[:upper:]' '[:lower:]')-$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind- Create a kind cluster by running:
# Create a single-node Kubernetes cluster
kind create cluster --name order-processingWhat does this command do?
This creates a single-node Kubernetes cluster running inside a Docker container. The cluster is named order-processing so you can identify it later. kind automatically configures your kubectl context to point at this cluster.
- Verify the cluster is running by running:
kubectl cluster-info --context kind-order-processing- You should see output showing the Kubernetes control plane address (typically https://127.0.0.1:<port>).
- Install Helm by running:
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash- Verify Helm is installed by running:
helm version- You should see the Helm version output (v3.x).
Kind cluster not starting?
Make sure Docker Desktop is running and has at least 4 GB of memory allocated. Kind needs Docker to be healthy before it can create the cluster container.
If you see a port conflict, another kind cluster or service may be using the same ports. Run kind delete cluster --name order-processing and try again.
help me troubleshoot why my kind cluster won't start
Create the Helm chart and Jenkinsfile
Your deployment needs two things: a Helm chart that describes how the application runs on Kubernetes, and a Jenkinsfile that orchestrates the build-and-deploy pipeline.
- Create the Helm chart directory structure by running:
mkdir -p helm/order-processing/templates- Create helm/order-processing/Chart.yaml with the following content:
# Helm chart metadata
apiVersion: v2
name: order-processing
description: Event-driven order processing Spring Boot application
type: application
version: 0.1.0
appVersion: "1.0.0"- Create helm/order-processing/values.yaml with the following content:
# Default configuration values
replicaCount: 1
image:
repository: order-processing
tag: latest
pullPolicy: IfNotPresent
service:
type: ClusterIP
port: 8080
resources:
limits:
memory: 512Mi
requests:
memory: 256Mi- Create helm/order-processing/templates/deployment.yaml with the following content:
# Kubernetes Deployment resource
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Release.Name }}
labels:
app: {{ .Release.Name }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app: {{ .Release.Name }}
template:
metadata:
labels:
app: {{ .Release.Name }}
spec:
containers:
- name: order-processing
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- containerPort: 8080
resources:
{{- toYaml .Values.resources | nindent 12 }}- Create helm/order-processing/templates/service.yaml with the following content:
# Kubernetes Service resource
apiVersion: v1
kind: Service
metadata:
name: {{ .Release.Name }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: 8080
protocol: TCP
selector:
app: {{ .Release.Name }}What does this Helm chart do?
A Helm chart is a package of Kubernetes resource templates plus default values. The values.yaml file defines configurable parameters (image name, replica count, resource limits).
The templates use Go templating ({{ .Values.image.repository }}) to inject values at deploy time. This means you can deploy the same chart to different environments by overriding values, without editing the templates.
- Create a Dockerfile at the project root (if you don't already have one):
# Multi-stage build for Spring Boot
FROM maven:3.9-eclipse-temurin-17 AS build
WORKDIR /app
COPY pom.xml .
COPY src ./src
RUN mvn clean package -DskipTests
FROM eclipse-temurin:17-jre
WORKDIR /app
COPY --from=build /app/target/*.jar app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "app.jar"]- Create the Jenkinsfile at the project root:
// Jenkins declarative pipeline for building and deploying to Kubernetes
pipeline {
agent any
environment {
IMAGE_NAME = 'order-processing'
IMAGE_TAG = 'latest'
}
stages {
stage('Build Docker Image') {
steps {
// Build the Spring Boot application image
sh "docker build -t ${IMAGE_NAME}:${IMAGE_TAG} ."
}
}
stage('Load Image into Kind') {
steps {
// Load the image directly into the kind cluster
sh "kind load docker-image ${IMAGE_NAME}:${IMAGE_TAG} --name order-processing"
}
}
stage('Deploy with Helm') {
steps {
// Install or upgrade the Helm release on the kind cluster
sh "helm upgrade --install order-processing ./helm/order-processing --set image.repository=${IMAGE_NAME} --set image.tag=${IMAGE_TAG}"
}
}
}
post {
success {
echo 'Deployment to Kubernetes completed successfully.'
}
failure {
echo 'Pipeline failed. Check the logs above for details.'
}
}
}What does this Jenkinsfile do?
The pipeline has three stages. First, it builds a Docker image from your Dockerfile. Then it loads that image directly into the kind cluster (since kind runs inside Docker, it needs images loaded explicitly with kind load). Finally, it runs helm upgrade --install to deploy (or update) the application on Kubernetes.
The --install flag makes the command idempotent. If the release doesn't exist, Helm creates it. If it already exists, Helm upgrades it in place.
- Save all files (Chart.yaml, values.yaml, deployment.yaml, service.yaml, Dockerfile, Jenkinsfile).
- Verify the Helm chart structure by running:
helm lint ./helm/order-processing- You should see 1 chart(s) linted, 0 chart(s) failed in the output.
Helm lint showing errors?
Check that your Chart.yaml has apiVersion: v2 (not v1). Also verify that template files use correct Go template syntax with matching {{ }} braces.
help me fix Helm chart linting errors
Write the GitHub Actions workflow and deploy with act
Now you will create a GitHub Actions workflow that triggers the Jenkins pipeline, then use nektos/act to run it locally without pushing to GitHub.
- Start a Jenkins container by running:
docker run -d --name jenkins \
--network host \
-v /var/run/docker.sock:/var/run/docker.sock \
-v $(pwd):/workspace \
-e JAVA_OPTS="-Djenkins.install.runSetupWizard=false" \
jenkins/jenkins:ltsWhat does this command do?
This starts Jenkins in a Docker container with host networking (so it can reach Docker and the kind cluster). The Docker socket mount lets Jenkins build images. The workspace mount gives Jenkins access to your project files. The JAVA_OPTS flag skips the first-time setup wizard for automation.
- Wait about 30 seconds for Jenkins to start, then verify it is running:
curl -s http://localhost:8080/login | head -5- You should see HTML output from the Jenkins login page.
- Create the GitHub Actions workflow directory and file:
mkdir -p .github/workflows- Create .github/workflows/deploy.yml with the following content:
# GitHub Actions workflow that triggers a Jenkins pipeline
name: Deploy to Kubernetes
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Trigger Jenkins pipeline
run: |
# Wait for Jenkins to be ready
until curl -s http://localhost:8080/login > /dev/null 2>&1; do
echo "Waiting for Jenkins..."
sleep 5
done
# Create and trigger the pipeline job
curl -X POST http://localhost:8080/createItem?name=order-processing \
-H "Content-Type: application/xml" \
-d '<flow-definition plugin="workflow-job"><definition class="org.jenkinsci.plugins.workflow.cps.CpsFlowDefinition" plugin="workflow-cps"><script>'"$(cat Jenkinsfile)"'</script><sandbox>true</sandbox></definition></flow-definition>'
# Trigger the build
curl -X POST http://localhost:8080/job/order-processing/build
# Wait for build to complete
echo "Jenkins pipeline triggered. Waiting for completion..."
sleep 30
- name: Verify deployment
run: |
# Check that pods are running on the kind cluster
kubectl get pods --context kind-order-processing
kubectl get svc --context kind-order-processingWhat does this workflow do?
The workflow checks out your code, then triggers a Jenkins pipeline job via the Jenkins REST API. It posts the Jenkinsfile content as a pipeline script definition, triggers a build, and finally verifies the deployment by checking Kubernetes pods and services.
When you run this with nektos/act, the entire flow executes locally. Act creates a Docker container that mimics the GitHub Actions runner, and that container communicates with Jenkins (on the host network) to kick off the pipeline.
- Install nektos/act by running:
curl -s https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash- Verify act is installed by running:
act --version- You should see act version 0.2.88 or later.
- Trigger the GitHub Actions workflow locally by running:
act push --container-architecture linux/amd64What does this command do?
The act push command simulates a git push event, which triggers your deploy.yml workflow. The --container-architecture flag ensures consistent behaviour on Apple Silicon Macs by forcing the runner container to use the amd64 architecture.
Act will pull the runner image on first run (this may take a minute), then execute your workflow steps inside a Docker container.
- Once the pipeline completes, verify the deployment on your kind cluster by running:
kubectl get pods --context kind-order-processing- You should see a pod named order-processing-<hash> with status Running.
- Check the Helm release status by running:
helm list --kube-context kind-order-processing- You should see the order-processing release with status deployed.
Pod stuck in ImagePullBackOff?
This means the kind cluster cannot find your Docker image. Verify that the kind load docker-image step ran successfully in the Jenkins console output. You can manually load the image by running kind load docker-image order-processing:latest --name order-processing.
Also confirm that imagePullPolicy in your values.yaml is set to IfNotPresent (not Always). With Always, Kubernetes tries to pull from a remote registry which doesn't exist for local images.
help me fix ImagePullBackOff on my kind cluster
Your application now deploys to Kubernetes through a fully automated local pipeline. A single act push command triggers GitHub Actions, which calls Jenkins to build, package, and deploy with Helm. Next up, tackle the Secret Mission and make your consumer idempotent.
Secret mission
Idempotent Consumer with Deduplication
In production Kafka systems, consumers can receive the same message more than once due to rebalances, retries, or network issues. Your current consumer processes every event it receives, which means a duplicated message could complete the same order twice. Can you add a deduplication layer that guarantees exactly-once processing?
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely locally via Docker containers, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.
Resources you used:
- Docker containers (Kafka, PostgreSQL, LocalStack, Jenkins) running via Docker Compose and standalone.
- LocalStack resources (SQS queue, IAM user and policy) that are ephemeral and destroyed when the container stops.
- A kind Kubernetes cluster (order-processing) running as a Docker container.
- Helm release (order-processing) deployed on the kind cluster.
- Local project files in your order-processing folder.
- PostgreSQL data stored in a Docker volume.
Keep everything running
No action needed. Choose this if you are still actively building or experimenting with the event-driven system.
- Your Docker containers will continue running in the background.
- Your Spring Boot application can be started and stopped from IntelliJ IDEA as needed.
- LocalStack resources (the SQS queue and IAM user) are recreated each time your Spring Boot app starts, so they persist as long as the LocalStack container is running.
- The kind cluster and Jenkins container will remain running until you explicitly stop them.
Pause - I'll come back to this later
Shut down running containers to free up memory while preserving your data volumes and project files.
- Stop your Spring Boot application in IntelliJ IDEA if it is running.
- Stop the Docker Compose containers by running this command from your project directory:
docker compose stop- Stop the Jenkins container:
docker stop jenkins- Your PostgreSQL data, Docker volumes, kind cluster, and project files are preserved.
- To resume later, run docker compose start and docker start jenkins from the same directory.
Delete - I don't want to use this again
Remove all Docker containers, volumes, the kind cluster, and project files to start completely fresh.
- Stop your Spring Boot application in IntelliJ IDEA if it is running.
- Delete the kind Kubernetes cluster by running:
kind delete cluster --name order-processing- Stop and remove the Jenkins container:
docker rm -f jenkins- Remove all Docker Compose containers and volumes by running this command from your project directory:
docker compose down -vWhat does this command do?
The -v flag removes all named volumes declared in your docker-compose.yml (including postgres_data and localstack_data). This permanently deletes your PostgreSQL database and any LocalStack state.
- Verify the containers are gone by running:
docker ps -a- You should see no containers related to Kafka, PostgreSQL, LocalStack, Jenkins, or kind in the output.
- Delete your project folder. On macOS or Linux:
rm -rf ~/Desktop/order-processing- On Windows:
Remove-Item -Recurse -Force ~\Desktop\order-processing- If you set LOCALSTACK_AUTH_TOKEN as an environment variable in your shell profile, remove that line from your .bashrc, .zshrc, or system environment variables.
That's a Wrap!
That's a Wrap!
Nice work! You've just built a fully local event-driven order processing system with an automated CI/CD pipeline. Your application uses Spring Boot, Kafka, and AWS services emulated through LocalStack, and deploys to Kubernetes via Jenkins and GitHub Actions.
You've learned how to:
- Built a Spring Boot REST API secured with JWT authentication that publishes order events to Kafka and routes failures through a dead-letter pipeline to SQS.
- Implemented IAM least-privilege policies with LocalStack to restrict which credentials can access the SQS dead-letter queue.
- Deployed the application to a local Kubernetes cluster using a Helm chart, orchestrated by a Jenkins pipeline triggered through GitHub Actions running locally with nektos/act.
- Secret Mission: Implemented idempotent message consumption with a deduplication table to prevent duplicate event processing.
Ready to quiz yourself?