Real-Time Order Processing with Kafka

Build a pub/sub order pipeline with Apache Kafka and three C# microservices.

Introduction

30 Second Summary

When you order something online, multiple systems react within milliseconds. Your payment gets charged, fraud checks run, and a shipping label gets created.

In this project, you will build a real-time order processing system using Apache Kafka and C#. You'll create three independent services that communicate through event streaming, all running on your local machine.

What You'll Build

A local event-driven architecture with three C# console applications connected through an Apache Kafka broker running in Docker.

By the end of this project, you'll have:

  • A running Kafka broker processing real-time events on your local machine.
  • Three independent C# microservices communicating through a shared Kafka topic.
  • A working demonstration of the pub/sub pattern where one event triggers multiple independent reactions.
  • Secret Mission: Add a live analytics consumer that tracks sales totals by product in real time.

Are there any prerequisites?

You should have basic C# knowledge (variables, classes, and loops) and Docker Desktop already installed on your machine. No prior Kafka experience is needed.

Set Up Your Development Environment

Before you can build an event-driven system, you need the right tools on your machine. This project uses Apache Kafka for message streaming and C# with .NET for building the microservices that produce and consume events.

In this step, you'll install the .NET 8 SDK, pull the Kafka Docker image, and scaffold three separate console applications that will become your producer and consumers.

In this step, get ready to:
  • Install and verify the .NET 8 SDK.
  • Pull the Apache Kafka 4.3.0 Docker image.
  • Create three C# console app projects with the Kafka client library.
Install the .NET 8 SDK

The .NET 8 SDK gives you the dotnet command-line tool for creating projects, adding packages, and running your C# applications. .NET 8 is the current Long Term Support (LTS) release, which means it receives updates and security patches for three years.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal (macOS) or Command Prompt (Windows) and press Enter to open it.
  • Check whether you already have the .NET SDK installed by running this command:
dotnet --version

Look at the output in your terminal and pick the tab below that matches what you see.

✔️ I see version 8.x.x or higher

Your .NET 8 SDK is ready to go. You should see a version number starting with 8. (for example, 8.0.419). Move on to the next substep.

ⓧ I see an older version

You have a .NET SDK installed, but it's an older version (like 6.x.x or 7.x.x). .NET supports side-by-side installations, so you can install .NET 8 alongside your existing version without conflicts.

  • Visit the .NET 8 download page: https://dotnet.microsoft.com/download/dotnet/8.0
  • Download the SDK installer for your operating system (not the Runtime).
  • Run the installer and accept the default settings.
  • Close your current terminal and open a new one so the updated PATH takes effect.
  • Run dotnet --version again and confirm the output starts with 8. (for example, 8.0.419).

Still seeing the old version?

If dotnet --version still shows the old version after reopening your terminal, check that the .NET 8 installer completed successfully. On macOS, the /usr/local/share/dotnet/dotnet path must be on your PATH. On Windows, restarting your machine can force the PATH to refresh.

Help me troubleshoot why dotnet --version is showing an old version after installing .NET 8

ⓧ Command not found

The .NET SDK is not installed on your machine yet. Let's install it now.

  • Visit the .NET 8 download page: https://dotnet.microsoft.com/download/dotnet/8.0
  • Download the SDK installer for your operating system (not the Runtime).
  • Run the installer and accept the default settings.
  • Close your current terminal and open a new one so the PATH takes effect.
  • Run dotnet --version and confirm the output starts with 8. (for example, 8.0.419).

Command still not found?

Make sure you downloaded the SDK and not just the Runtime. The Runtime alone does not include the dotnet CLI tools needed to create and build projects.

Help me install the .NET 8 SDK on my machine

Pull the Kafka Docker image

Apache Kafka 4.3.0 runs in KRaft mode, which means it operates as a single container with no additional dependencies like ZooKeeper. You'll pull the official image now so it's ready when you start the broker in the next step.

  • Pull the Kafka Docker image by running this command:
docker pull apache/kafka:4.3.0

What does this command do?

docker pull downloads a container image from Docker Hub to your local machine. The apache/kafka:4.3.0 part specifies the official Apache Kafka image pinned to version 4.3.0.

  • Confirm the image was downloaded by running:
docker images

You should see apache/kafka with the tag 4.3.0 in the output list.

Seeing a connection error?

Make sure Docker Desktop is running before you try to pull. You can check by looking for the Docker whale icon in your system tray (Windows) or menu bar (macOS).

If you see a timeout or network error, check your internet connection and try the docker pull command again.

Help me troubleshoot why docker pull is failing

Create the project directory structure

Your system will have three independent C# console applications, each representing a separate microservice. They all live inside a single root folder called kafka-order-system.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir kafka-order-system
cd kafka-order-system
  • Confirm the folder exists by running:
ls

You should see the kafka-order-system folder listed if you run ls from the Desktop. Since you've already moved into it, the folder should appear empty for now.

  • Create three console app projects by running these commands:
dotnet new console -n OrderProducer
dotnet new console -n PaymentsConsumer
dotnet new console -n FraudDetectionConsumer

What do these commands do?

dotnet new console -n ProjectName scaffolds a new C# console application in a subfolder named after the project. Each project gets a Program.cs file (your main code) and a .csproj file (project configuration).

You're creating three separate projects because each microservice runs as its own process, just like it would in a real distributed system.

  • Verify the three project folders were created by running:
ls

You should see OrderProducer, PaymentsConsumer, and FraudDetectionConsumer in the output.

Missing a project folder?

If a folder is missing, check that you ran the dotnet new console command from inside the kafka-order-system directory. Run pwd (macOS/Linux) or cd (Windows) to confirm your current location, then rerun the missing command.

Help me figure out why dotnet new console didn't create my project folder

  • Add the Confluent.Kafka NuGet package to each project by running these commands:
dotnet add OrderProducer package Confluent.Kafka --version 2.14.0
dotnet add PaymentsConsumer package Confluent.Kafka --version 2.14.0
dotnet add FraudDetectionConsumer package Confluent.Kafka --version 2.14.0

What is Confluent.Kafka?

Confluent.Kafka is the official .NET client library for Apache Kafka, maintained by Confluent (the company behind Kafka). It wraps a high-performance C library called librdkafka and handles all the low-level communication between your C# code and the Kafka broker.

The --version 2.14.0 flag pins the exact package version so every learner gets the same behaviour.

  • Verify the package was added to one of the projects by checking its .csproj file:
cat OrderProducer/OrderProducer.csproj

You should see a PackageReference entry for Confluent.Kafka with version 2.14.0 listed inside an ItemGroup element.

Package restore failed?

If you see an error about failing to restore packages, make sure you have an internet connection. The dotnet add package command downloads the package from NuGet.org.

Also confirm you're running the commands from the kafka-order-system root folder (not from inside a project subfolder). The dotnet add OrderProducer package syntax expects to find the OrderProducer folder relative to your current directory.

Help me fix the dotnet add package error

Your development environment is locked and loaded. Next up, you'll start a Kafka broker and create the topic that all three services will communicate through.

Start Kafka and Create Your Topic

Your development environment is ready with .NET 8, Docker, and three project folders waiting for code.

Now you need a running Apache Kafka broker to receive and distribute events. In this step, you will start a Kafka broker in a Docker container, create a topic for your order events, and verify that messages flow from a producer to a consumer.

In this step, get ready to:
  • Start a Kafka container and verify it is running.
  • Create the orders topic with 3 partitions.
  • Send a test message through Kafka to prove the pipeline works.
Start the Kafka container

Kafka needs to be running before any of your C# services can connect to it. You will run Kafka inside a Docker container, which keeps your machine clean and makes the setup repeatable.

  • Start the Kafka broker by running this command:
docker run -d --name kafka-broker -p 9092:9092 apache/kafka:4.3.0

What does this command do?

  • -d runs the container in the background so you can keep using the terminal.
  • --name kafka-broker gives the container a name you can reference in later commands.
  • -p 9092:9092 maps the container's Kafka port to your machine's localhost:9092.
  • apache/kafka:4.3.0 is the official Kafka image running in KRaft mode, which means no ZooKeeper is required.
  • Verify the container is running:
docker ps

You should see a row with kafka-broker under the NAMES column and a STATUS showing Up.

Container not showing as Up?

  • If you see no output, the container may have failed to start. Run docker ps -a to see stopped containers and check the STATUS column for an error.
  • If you see a name conflict error, a container named kafka-broker already exists from a previous attempt. Remove it with docker rm -f kafka-broker and then re-run the docker run command.
  • Make sure Docker Desktop is running. The Docker daemon must be active for any container commands to work.

Help me troubleshoot my Kafka container not starting.

Create the orders topic

A topic is a named channel where messages are stored in Kafka. Your three services will all communicate through a single topic called orders.

  • Create the orders topic by running this command:
docker exec --workdir /opt/kafka/bin/ kafka-broker sh -c "./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders --partitions 3 --replication-factor 1"

What does this command do?

This command runs kafka-topics.sh inside the running container to create a new topic.

  • --topic orders names the topic your services will publish to and subscribe from.
  • --partitions 3 splits the topic into 3 segments. Kafka distributes messages across partitions, which allows multiple consumers to read in parallel.
  • --replication-factor 1 keeps a single copy of each message. For local development, one replica is fine.
  • Verify the topic was created by running:
docker exec --workdir /opt/kafka/bin/ kafka-broker sh -c "./kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders"

You should see output showing the topic name orders with PartitionCount: 3 and ReplicationFactor: 1.

Topic creation failed?

  • If you see a connection error, the Kafka broker may not be fully ready yet. Wait 10 seconds and try the create command again.
  • If you see a "topic already exists" message, that means the topic was already created successfully. Run the describe command to confirm.

Help me troubleshoot my Kafka topic creation.

Quick smoke test with console producer and consumer

Before writing any C# code, you can verify that Kafka is working correctly using built-in command-line tools. You will open two terminals and send a message through Kafka manually.

  • In your current terminal, start a console consumer that listens for messages on the orders topic by running:
docker exec --workdir /opt/kafka/bin/ -it kafka-broker sh -c "./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders --from-beginning"

What does this command do?

This starts an interactive consumer that waits for messages and prints them to your terminal as they arrive. The --from-beginning flag tells the consumer to read all messages stored in the topic, not just new ones.

The consumer is now waiting. It will look like nothing is happening, which is expected.

  • Open a second terminal window.
  • In the second terminal, start a console producer by running:
docker exec --workdir /opt/kafka/bin/ -it kafka-broker sh -c "./kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders"
  • Type any test message (for example hello kafka) and press Enter.
  • Switch to your first terminal. You should see your message appear there.
  • Press Ctrl+C in both terminals to stop the producer and consumer.

What just happened?

You just demonstrated the pub/sub pattern at its most basic level. A producer published a message to the orders topic, and a consumer subscribed to that topic received it in real time.

Your C# services will do exactly the same thing, but with structured JSON data instead of plain text.

Message not appearing in the consumer?

  • Make sure you pressed Enter after typing your message in the producer terminal. The message is not sent until you press Enter.
  • Confirm both commands use the same topic name orders. A typo means the producer and consumer are on different channels.
  • If the consumer exited immediately, the Kafka broker may have gone down. Run docker ps to check that kafka-broker is still Up.

Help me troubleshoot my Kafka smoke test.

Your Kafka broker is running and the orders topic is ready to receive events. Next up, you will build the Order Producer in C# to generate realistic order events and publish them to this topic.

Build the Order Producer

Your Kafka broker is running and the orders topic is ready to receive messages. Now you need a data source that produces events in real time.

In this step, you'll build the Order Producer. This is a C# console app that generates random e-commerce orders and publishes them as JSON events to your Kafka topic every second.

In this step, get ready to:
  • Define a data model for order events.
  • Write producer logic that generates random orders and publishes them to Kafka.
  • Test the producer to confirm events flow into the topic.
Define the Order model

Every event flowing through Kafka carries a payload of data. You need a class that defines the shape of an order event so your producer can generate structured data and your consumers can deserialize it later.

  • Create a new file called Order.cs inside the OrderProducer folder.
  • Add the following code to define the Order class:
namespace OrderProducer;

public class Order
{
    public int OrderId { get; set; }
    public string Product { get; set; } = string.Empty;
    public decimal Amount { get; set; }
    public string City { get; set; } = string.Empty;
    public DateTime Timestamp { get; set; }
}

What does this code do?

This class defines the structure of every order event that flows through your system.

  • OrderId uniquely identifies each order and will serve as the Kafka message key.
  • Product and Amount describe what was purchased and for how much.
  • City records where the order originated. Your fraud detection consumer will use this later.
  • Timestamp captures when the order was placed.
  • Save Order.cs.

Seeing red squiggles in your editor?

Make sure the namespace matches the project folder name exactly: OrderProducer. A mismatch will cause the project to fail to compile.

help me fix namespace errors in my Order.cs file

Write the producer logic

The producer is the engine of your system. It connects to Kafka, generates random orders from a pool of products and cities, serializes each order to JSON, and publishes it to the orders topic.

  • Open Program.cs in the OrderProducer folder.
  • Delete the existing Console.WriteLine("Hello, World!"); line and replace the entire file contents with this code:
using System.Text.Json;
using Confluent.Kafka;
using OrderProducer;

// Configure the producer to connect to Kafka on localhost
var config = new ProducerConfig
{
    BootstrapServers = "localhost:9092"
};

// Define pools of products and cities for random order generation
var products = new[] { "Laptop", "Mouse", "Keyboard", "Monitor", "Headset" };
var cities = new[] { "Chicago", "New York", "Dallas", "London", "Lagos", "Minsk", "Seattle" };
var random = new Random();
var orderIdCounter = 1000;

// Build the producer with string key and string value types
using var producer = new ProducerBuilder<string, string>(config).Build();

Console.WriteLine("Order Producer started. Press Ctrl+C to stop.");

// Set up graceful shutdown on Ctrl+C
var cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) =>
{
    e.Cancel = true;
    cts.Cancel();
};

try
{
    while (!cts.Token.IsCancellationRequested)
    {
        // Generate a random order
        var order = new Order
        {
            OrderId = orderIdCounter++,
            Product = products[random.Next(products.Length)],
            Amount = Math.Round((decimal)(random.NextDouble() * 2000 + 50), 2),
            City = cities[random.Next(cities.Length)],
            Timestamp = DateTime.UtcNow
        };

        // Serialize to JSON and publish to the orders topic
        var json = JsonSerializer.Serialize(order);
        var result = await producer.ProduceAsync("orders", new Message<string, string>
        {
            Key = order.OrderId.ToString(),
            Value = json
        });

        Console.WriteLine($"Produced order {order.OrderId}: {order.Product} - ${order.Amount} to {order.City} [{result.Partition}]");

        // Wait 1 second before generating the next order
        await Task.Delay(1000, cts.Token);
    }
}
catch (OperationCanceledException)
{
    Console.WriteLine("Producer stopped.");
}

// Flush any remaining messages before exiting
producer.Flush(TimeSpan.FromSeconds(5));

What does this code do?

This program connects to your local Kafka broker and publishes a new order event every second.

  • ProducerConfig tells the Kafka client where to find the broker (localhost:9092).
  • ProducerBuilder<string, string> creates a producer that uses string keys and string values. The key is the OrderId, and the value is the JSON-serialized order.
  • Using the OrderId as the message key means orders with the same ID always land in the same partition. This is how Kafka guarantees ordering per key.
  • The CancellationTokenSource and Console.CancelKeyPress handler let the program shut down cleanly when you press Ctrl+C.
  • producer.Flush() ensures any buffered messages are delivered before the process exits.
  • Save Program.cs.

Seeing build errors about missing types?

If your editor shows errors for ProducerConfig or ProducerBuilder, confirm that Confluent.Kafka is referenced in your project. Check that OrderProducer.csproj contains a PackageReference for Confluent.Kafka.

help me fix Confluent.Kafka build errors in my OrderProducer project

✔️ Awesome, I've got everything!

Great. Double check you've saved both Order.cs and Program.cs in the OrderProducer folder.

ⓧ I'd like to double check the full code

Here is the complete Order.cs:

namespace OrderProducer;

public class Order
{
    public int OrderId { get; set; }
    public string Product { get; set; } = string.Empty;
    public decimal Amount { get; set; }
    public string City { get; set; } = string.Empty;
    public DateTime Timestamp { get; set; }
}

Here is the complete Program.cs:

using System.Text.Json;
using Confluent.Kafka;
using OrderProducer;

var config = new ProducerConfig
{
    BootstrapServers = "localhost:9092"
};

var products = new[] { "Laptop", "Mouse", "Keyboard", "Monitor", "Headset" };
var cities = new[] { "Chicago", "New York", "Dallas", "London", "Lagos", "Minsk", "Seattle" };
var random = new Random();
var orderIdCounter = 1000;

using var producer = new ProducerBuilder<string, string>(config).Build();

Console.WriteLine("Order Producer started. Press Ctrl+C to stop.");

var cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) =>
{
    e.Cancel = true;
    cts.Cancel();
};

try
{
    while (!cts.Token.IsCancellationRequested)
    {
        var order = new Order
        {
            OrderId = orderIdCounter++,
            Product = products[random.Next(products.Length)],
            Amount = Math.Round((decimal)(random.NextDouble() * 2000 + 50), 2),
            City = cities[random.Next(cities.Length)],
            Timestamp = DateTime.UtcNow
        };

        var json = JsonSerializer.Serialize(order);
        var result = await producer.ProduceAsync("orders", new Message<string, string>
        {
            Key = order.OrderId.ToString(),
            Value = json
        });

        Console.WriteLine($"Produced order {order.OrderId}: {order.Product} - ${order.Amount} to {order.City} [{result.Partition}]");

        await Task.Delay(1000, cts.Token);
    }
}
catch (OperationCanceledException)
{
    Console.WriteLine("Producer stopped.");
}

producer.Flush(TimeSpan.FromSeconds(5));
Test the producer

Your producer code is ready. Time to run it and confirm that order events are flowing into Kafka.

  • Make sure you are in the kafka-order-system folder in your terminal.
  • Start the Order Producer by running:
dotnet run --project OrderProducer

You should see output like this, with a new order printed every second:

Order Producer started. Press Ctrl+C to stop.
Produced order 1000: Laptop - $899.99 to Chicago [1]
Produced order 1001: Mouse - $72.45 to New York [0]
Produced order 1002: Monitor - $1523.80 to Lagos [2]
  • Let it run for a few messages to confirm it works, then press Ctrl+C to stop it.

Each line shows the OrderId, product name, amount, city, and which Kafka partition the message landed in (the number in brackets).

Don't see any output or seeing an error?

  • If you see a connection refused error, confirm the kafka-broker container is still running with docker ps. Restart it with docker start kafka-broker if needed.
  • If you see a build error, make sure you're running the command from the kafka-order-system folder (not from inside the OrderProducer subfolder).
  • If you see an error about Order not being found, verify Order.cs is saved inside the OrderProducer folder and its namespace is OrderProducer.

help me debug my OrderProducer that won't start

Your Order Producer is live and publishing events into Kafka. Next up, you'll build a consumer that listens to those events and processes payments for every order that comes through.

Build the Payments Consumer

Your Order Producer is publishing events into Kafka, but nobody is listening yet.

In this step, you will build a Payments Consumer that subscribes to the orders topic and processes every order as it arrives. This is your first taste of the pub/sub pattern in action.

In this step, get ready to:
  • Define the Order model for the PaymentsConsumer project.
  • Write consumer logic that subscribes to the orders topic and processes payments.
  • Run both services together and confirm real-time message flow.
Define the Order model

The consumer receives JSON strings from Kafka and needs to deserialize them back into C# objects. You need the same Order class in this project so the deserializer knows what shape to expect.

  • Create a new file called Order.cs in the PaymentsConsumer folder.
  • Add the following Order class definition:
namespace PaymentsConsumer;

public class Order
{
    public int OrderId { get; set; }
    public string Product { get; set; } = string.Empty;
    public decimal Amount { get; set; }
    public string City { get; set; } = string.Empty;
    public DateTime Timestamp { get; set; }
}

What does this code do?

This class mirrors the Order model in the producer. Each property maps to a JSON field in the messages flowing through Kafka.

The namespace is PaymentsConsumer instead of OrderProducer. In a real microservices system, each service owns its own copy of shared models.

  • Save Order.cs.

✔️ Awesome, I've got everything!

Great. Make sure you have saved the file before moving on.

ⓧ I'd like to double check the full code

namespace PaymentsConsumer;

public class Order
{
    public int OrderId { get; set; }
    public string Product { get; set; } = string.Empty;
    public decimal Amount { get; set; }
    public string City { get; set; } = string.Empty;
    public DateTime Timestamp { get; set; }
}
Write the consumer logic

A Kafka consumer works by joining a consumer group and subscribing to one or more topics. Kafka then delivers messages to the consumer in a continuous loop.

  • Open Program.cs in the PaymentsConsumer folder.
  • Replace all of its contents with the following code:
using System.Text.Json;
using Confluent.Kafka;
using PaymentsConsumer;

// Configure connection to Kafka and consumer group membership
var config = new ConsumerConfig
{
    BootstrapServers = "localhost:9092",
    GroupId = "payments-service",
    AutoOffsetReset = AutoOffsetReset.Earliest
};

// Build the consumer and subscribe to the orders topic
using var consumer = new ConsumerBuilder<string, string>(config).Build();

consumer.Subscribe("orders");

Console.WriteLine("Payments Consumer started. Press Ctrl+C to stop.");

// Set up graceful cancellation on Ctrl+C
var cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) =>
{
    e.Cancel = true;
    cts.Cancel();
};

try
{
    while (!cts.Token.IsCancellationRequested)
    {
        // Block until a message arrives from Kafka
        var consumeResult = consumer.Consume(cts.Token);
        var order = JsonSerializer.Deserialize<Order>(consumeResult.Message.Value);

        if (order is not null)
        {
            Console.WriteLine($"[PAYMENTS] Charging ${order.Amount} for {order.Product} from {order.City} (Order #{order.OrderId})");
        }
    }
}
catch (OperationCanceledException)
{
    Console.WriteLine("Payments Consumer stopped.");
}

// Commit final offsets and leave the consumer group cleanly
consumer.Close();

What does this code do?

  • ConsumerConfig tells the consumer where Kafka is and which consumer group to join. The GroupId (payments-service) identifies this service so Kafka can track what messages it has already read.
  • AutoOffsetReset.Earliest means the consumer starts reading from the beginning of the topic if it has no prior position saved.
  • consumer.Consume(cts.Token) blocks the loop until a new message arrives. When it does, the JSON value is deserialized back into an Order object.
  • consumer.Close() commits the final offsets and leaves the consumer group cleanly so Kafka knows where this consumer stopped.

Why does GroupId matter?

Each consumer group gets its own independent copy of every message on the topic. In the next step, you will create a second consumer with a different GroupId. Both consumers will receive all orders independently. This is how one event triggers multiple reactions in an event-driven system.

  • Save Program.cs.

Seeing a build error?

  • Make sure you created Order.cs in the PaymentsConsumer folder, not in the OrderProducer folder. The namespace must match the project.
  • Check that the using PaymentsConsumer; line is at the top of Program.cs. Without it, the compiler cannot find the Order class.

Help me fix a build error in my PaymentsConsumer project.

✔️ Awesome, I've got everything!

Great. Double check you have saved the file before testing.

ⓧ I'd like to double check the full code

using System.Text.Json;
using Confluent.Kafka;
using PaymentsConsumer;

var config = new ConsumerConfig
{
    BootstrapServers = "localhost:9092",
    GroupId = "payments-service",
    AutoOffsetReset = AutoOffsetReset.Earliest
};

using var consumer = new ConsumerBuilder<string, string>(config).Build();

consumer.Subscribe("orders");

Console.WriteLine("Payments Consumer started. Press Ctrl+C to stop.");

var cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) =>
{
    e.Cancel = true;
    cts.Cancel();
};

try
{
    while (!cts.Token.IsCancellationRequested)
    {
        var consumeResult = consumer.Consume(cts.Token);
        var order = JsonSerializer.Deserialize<Order>(consumeResult.Message.Value);

        if (order is not null)
        {
            Console.WriteLine($"[PAYMENTS] Charging ${order.Amount} for {order.Product} from {order.City} (Order #{order.OrderId})");
        }
    }
}
catch (OperationCanceledException)
{
    Console.WriteLine("Payments Consumer stopped.");
}

consumer.Close();
Test producer and consumer together

Now for the satisfying part. You will run both services simultaneously and watch orders flow from the producer through Kafka into the consumer in real time.

  • In your terminal, make sure you are inside the kafka-order-system folder.
  • Start the Order Producer by running:
dotnet run --project OrderProducer
  • Open a second terminal window.
  • Navigate to the same kafka-order-system folder in the second terminal.
  • Start the Payments Consumer by running:
dotnet run --project PaymentsConsumer
  • Watch the second terminal. You should see output like:
[PAYMENTS] Charging $899.99 for Laptop from Chicago (Order #1000)
[PAYMENTS] Charging $127.50 for Mouse from New York (Order #1001)
[PAYMENTS] Charging $1450.00 for Monitor from Dallas (Order #1002)

What should I see?

The consumer first catches up on any orders the producer already sent (because of AutoOffsetReset.Earliest). After catching up, new orders appear in the consumer within less than a second of being produced.

Every order the producer prints should have a matching payment charge in the consumer. The product names, amounts, and cities will match between the two terminals.

  • Press Ctrl+C in both terminals to stop the services.

Consumer starts but no messages appear?

  • Confirm the kafka-broker container is still running. Run docker ps and check that its status says Up.
  • Make sure the producer is running in the other terminal. The consumer only prints output when there are messages to consume.
  • Check that both projects reference the same topic name. The producer publishes to "orders" and the consumer subscribes to "orders". A typo in either location means messages flow to nowhere.

Help me debug why my Kafka consumer is not receiving messages.

Your payments service is live and processing every order in real time. Next up, you will build a Fraud Detection Consumer that reads from the same topic but applies business rules to flag suspicious orders.

Build the Fraud Detection Consumer and Run the Full System

Your payments consumer is processing orders in real time. Now it's time to prove the true power of Kafka's pub/sub pattern.

You'll build a second consumer that reads from the same topic with completely different business logic. Both consumers will receive every message independently, proving that adding new services to a Kafka-based system requires zero changes to the producer.

In this step, get ready to:
  • Define the Order model and fraud detection rules.
  • Write the fraud detection consumer logic.
  • Run all three services simultaneously to see the full event-driven system.
Define the Order model and fraud rules

The fraud detection consumer needs the same Order class to deserialize incoming messages. It also needs business rules that define what counts as suspicious.

Your fraud rules will flag any order where the amount exceeds $1500 or the city is in a high-risk list.

  • Create a new file called Order.cs inside the FraudDetectionConsumer folder.
  • Add the Order class by pasting the following code:
namespace FraudDetectionConsumer;

public class Order
{
    public int OrderId { get; set; }
    public string Product { get; set; } = string.Empty;
    public decimal Amount { get; set; }
    public string City { get; set; } = string.Empty;
    public DateTime Timestamp { get; set; }
}

What does this code do?

This is the same Order model you used in the producer and payments consumer, but in the FraudDetectionConsumer namespace. Each microservice has its own copy of the model because they are independent projects.

  • Save Order.cs.
  • Confirm the file appears inside your FraudDetectionConsumer folder alongside the existing FraudDetectionConsumer.csproj.
Write the fraud detection consumer

The fraud consumer follows the same pattern as the payments consumer. The key difference is the GroupId and the processing logic.

Why does GroupId matter?

The payments consumer uses GroupId payments-service and the fraud consumer uses fraud-service. Because they have different group IDs, Kafka delivers every message to both consumers independently.

If they shared the same GroupId, Kafka would split messages between them instead. Different group IDs are what enable the pub/sub pattern where one event triggers multiple reactions.

  • Open Program.cs in the FraudDetectionConsumer folder.
  • Replace the entire contents with the following code:
using System.Text.Json;
using Confluent.Kafka;
using FraudDetectionConsumer;

// Configure the consumer with a unique group ID for the fraud service
var config = new ConsumerConfig
{
    BootstrapServers = "localhost:9092",
    GroupId = "fraud-service",
    AutoOffsetReset = AutoOffsetReset.Earliest
};

// Define fraud detection rules
var highRiskCities = new HashSet<string> { "Lagos", "Minsk" };
const decimal amountThreshold = 1500m;

using var consumer = new ConsumerBuilder<string, string>(config).Build();

consumer.Subscribe("orders");

Console.WriteLine("Fraud Detection Consumer started. Press Ctrl+C to stop.");

// Handle graceful shutdown on Ctrl+C
var cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) =>
{
    e.Cancel = true;
    cts.Cancel();
};

try
{
    while (!cts.Token.IsCancellationRequested)
    {
        var consumeResult = consumer.Consume(cts.Token);
        var order = JsonSerializer.Deserialize<Order>(consumeResult.Message.Value);

        if (order is not null)
        {
            // Apply fraud rules: flag if amount exceeds threshold OR city is high-risk
            bool isFraudulent = order.Amount > amountThreshold || highRiskCities.Contains(order.City);

            if (isFraudulent)
            {
                Console.WriteLine($"[FRAUD ALERT] Suspicious order #{order.OrderId}: ${order.Amount} from {order.City}");
            }
            else
            {
                Console.WriteLine($"[FRAUD] OK - Order #{order.OrderId}: ${order.Amount} from {order.City}");
            }
        }
    }
}
catch (OperationCanceledException)
{
    Console.WriteLine("Fraud Detection Consumer stopped.");
}

consumer.Close();

What does this code do?

  • highRiskCities is a HashSet containing cities flagged for extra scrutiny. HashSet gives fast lookups when checking each order.
  • amountThreshold is the dollar ceiling. Any order above $1500 triggers a fraud alert.
  • isFraudulent combines both rules with an OR operator. An order only needs to match one rule to be flagged.
  • consumer.Close() commits final offsets and cleanly leaves the consumer group when the service shuts down.
  • Save Program.cs.

Seeing a build error?

  • Make sure Order.cs uses the namespace FraudDetectionConsumer (not OrderProducer or PaymentsConsumer).
  • Check that both Order.cs and Program.cs are inside the FraudDetectionConsumer folder, not the project root.

Help me fix a build error in my FraudDetectionConsumer project.

✔️ Awesome, I've got everything!

Great. Make sure you've saved both files before running the full system.

ⓧ I'd like to double check the full code

Here is the complete Order.cs:

namespace FraudDetectionConsumer;

public class Order
{
    public int OrderId { get; set; }
    public string Product { get; set; } = string.Empty;
    public decimal Amount { get; set; }
    public string City { get; set; } = string.Empty;
    public DateTime Timestamp { get; set; }
}

Here is the complete Program.cs:

using System.Text.Json;
using Confluent.Kafka;
using FraudDetectionConsumer;

var config = new ConsumerConfig
{
    BootstrapServers = "localhost:9092",
    GroupId = "fraud-service",
    AutoOffsetReset = AutoOffsetReset.Earliest
};

var highRiskCities = new HashSet<string> { "Lagos", "Minsk" };
const decimal amountThreshold = 1500m;

using var consumer = new ConsumerBuilder<string, string>(config).Build();

consumer.Subscribe("orders");

Console.WriteLine("Fraud Detection Consumer started. Press Ctrl+C to stop.");

var cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) =>
{
    e.Cancel = true;
    cts.Cancel();
};

try
{
    while (!cts.Token.IsCancellationRequested)
    {
        var consumeResult = consumer.Consume(cts.Token);
        var order = JsonSerializer.Deserialize<Order>(consumeResult.Message.Value);

        if (order is not null)
        {
            bool isFraudulent = order.Amount > amountThreshold || highRiskCities.Contains(order.City);

            if (isFraudulent)
            {
                Console.WriteLine($"[FRAUD ALERT] Suspicious order #{order.OrderId}: ${order.Amount} from {order.City}");
            }
            else
            {
                Console.WriteLine($"[FRAUD] OK - Order #{order.OrderId}: ${order.Amount} from {order.City}");
            }
        }
    }
}
catch (OperationCanceledException)
{
    Console.WriteLine("Fraud Detection Consumer stopped.");
}

consumer.Close();
Run the complete system

This is the moment everything comes together. You'll run all three services simultaneously and watch orders flow from the producer to both consumers in real time.

  • Open three terminal windows side by side. Make sure each terminal is in the kafka-order-system folder.
  • In Terminal 1, start the producer by running:
dotnet run --project OrderProducer
  • In Terminal 2, start the payments consumer by running:
dotnet run --project PaymentsConsumer
  • In Terminal 3, start the fraud detection consumer by running:
dotnet run --project FraudDetectionConsumer

Watch the output across all three terminals. Every order the producer generates appears in both the payments and fraud consumers.

What should I see?

Terminal 1 prints produced orders every second. Terminal 2 shows a [PAYMENTS] line for every order. Terminal 3 shows either [FRAUD ALERT] for suspicious orders or [FRAUD] OK for safe ones.

Orders from Lagos or Minsk, or orders above $1500, trigger fraud alerts. All other orders pass through cleanly.

  • Let the system run for 10-15 seconds so you can observe a mix of fraud alerts and safe orders.
  • Press Ctrl+C in each terminal to stop all three services.

Why is this powerful?

The producer has no idea how many consumers exist. You added the fraud service without changing a single line in the producer or the payments consumer. This is the pub/sub pattern in action.

In a real system, you could add a fifth, tenth, or hundredth consumer and the producer would never know or care. Each consumer group processes the full stream independently.

Consumer not receiving messages?

  • Make sure the kafka-broker container is still running. Check with docker ps.
  • Confirm each terminal is in the kafka-order-system directory, not inside a project subfolder.
  • If you see connection errors, restart the broker with docker start kafka-broker and try again.

Help me troubleshoot why my Kafka consumer isn't receiving messages.

Secret mission

Add a Live Analytics Consumer

Your three-service system reacts to events, but none of them remember anything. Can you build a fourth consumer that maintains running state, tracking sales counts and revenue totals per product as orders stream through Kafka in real time?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally using Docker, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Docker container kafka-broker running the Apache Kafka broker on port 9092.
  • The kafka-order-system folder on your Desktop containing all project files.

Keep everything running

No action needed. Choose this if you're still actively building or want to experiment further with your Kafka setup.

  • The kafka-broker container uses minimal CPU and memory when idle.
  • Your topic data and consumer offsets persist inside the container.
  • You can restart your producers and consumers at any time by running dotnet run --project <ProjectName> from the kafka-order-system folder.

Pause - I'll come back to this later

Stop the running container to free up memory. Your container and topic data are preserved for next time.

  • Stop the Kafka container by running:
docker stop kafka-broker
  • When you're ready to resume, restart it by running:
docker start kafka-broker
  • After restarting, your orders topic and all consumer group offsets will still be there.

Delete - I don't want to use this again

Remove all project resources completely. This deletes the container, all topic data, and your project code.

  • Remove the Kafka container and all its data by running:
docker rm -f kafka-broker
  • Remove the Kafka Docker image to free up disk space by running:
docker rmi apache/kafka:4.3.0
  • Delete the project folder. On macOS or Linux, run:
rm -rf ~/Desktop/kafka-order-system
  • On Windows, run:
rmdir /s /q %USERPROFILE%\Desktop\kafka-order-system
  • Verify the container is gone by running:
docker ps -a
  • You should no longer see kafka-broker in the list.

That's a Wrap!

That's a Wrap!

Nice work! You just built a complete event-driven microservices system using Apache Kafka and C#.

You now have hands-on experience with the same architecture that powers real-time systems at companies like Netflix, Uber, and LinkedIn.

You've learned how to:

  • Set up Apache Kafka 4.3.0 locally using Docker with KRaft mode (no ZooKeeper required).
  • Build a Producer that generates random order events and publishes them as JSON to a Kafka topic.
  • Build independent Consumers with different consumer groups, demonstrating the pub/sub pattern where one event triggers multiple services reacting independently.
  • Secret Mission: Built a stateful Analytics Consumer that tracks live sales metrics across the event stream in real time.

Ready to quiz yourself?