Messaging with Kafka

Modern distributed systems require high-throughput, scalable, and fault-tolerant messaging to handle large volumes of events in real time. In Spring Boot microservices, Apache Kafka acts as a powerful event streaming platform that serves as the backbone for event-driven architectures.

Kafka: Distributed Event Streaming Platform

Apache Kafka is a distributed event streaming platform designed for durability, horizontal scalability, and event replay. Unlike traditional message queues, Kafka treats messages as immutable events stored in a distributed log.

Kafka uses a publish–subscribe event streaming model:

• Producers publish events to topics
• Topics are divided into partitions
• Partitions store events in an ordered, append-only log
• Consumers read events independently using consumer groups

Kafka is widely used for event-driven microservices, real-time analytics, data pipelines, and stream processing systems.

Spring Boot integrates Kafka using Spring for Apache Kafka, providing seamless configuration and developer-friendly APIs.

Kafka Core Concepts

Kafka is built around a few fundamental concepts that enable scalability and fault tolerance.

1. Topics

A topic is a logical category or stream where events are published.

• Topics represent event types such as order-events, payment-events
• Events remain in topics for a configurable retention period
• Multiple consumers can read the same topic independently

2. Partitions

Each topic is split into partitions to enable parallelism.

Key characteristics:

• Events in a partition are strictly ordered
• Partitions allow Kafka to scale horizontally
• Each partition is consumed by only one consumer per group

3. Replicas

Partitions are replicated across brokers to ensure fault tolerance.

• One replica acts as the leader
• Others act as followers
• If a broker fails, Kafka automatically promotes a replica

4. Producers

Producers publish events to topics.

• Can choose partitioning strategy
• Support batching and compression
• Ensure high throughput and durability

5. Consumers & Consumer Groups

Consumers read events from topics.

• Consumer Groups enable horizontal scaling
• Each partition is consumed by only one consumer per group
• Multiple groups can read the same data independently

Kafka Architecture Flow

Producer
   ↓
Topic
   ↓
Partition (ordered log)
   ↓
Consumer Group
   ↓
Consumer

Kafka Producers & Consumers

Spring Boot makes it extremely simple to build Kafka producers and consumers using Spring for Apache Kafka. With minimal configuration, you can publish and consume events in a clean, scalable way.

Add Dependency

Add the Spring Kafka dependency in your pom.xml:

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

Kafka Producer Example

A producer publishes messages (events) to a Kafka topic using KafkaTemplate.

@Service
public class OrderProducer {

    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    public void sendMessage(String message) {
        kafkaTemplate.send("order-topic", message);
    }
}

What Happens Here?

• KafkaTemplate is the main class used to send messages.
• "order-topic" is the Kafka topic.
• Messages are published asynchronously.
• Kafka handles partitioning and durability internally.

Kafka Consumer Example

A consumer listens to a topic using the @KafkaListener annotation.

@Service
public class OrderConsumer {

    @KafkaListener(topics = "order-topic", groupId = "order-group")
    public void consume(String message) {
        System.out.println("Received: " + message);
    }
}

What Happens Here?

• @KafkaListener automatically subscribes to the topic.
• groupId enables consumer group functionality.
• Kafka manages offsets for each consumer group.
• Multiple instances can scale horizontally.

Offsets & Consumer Groups Explained

Kafka tracks how messages are consumed using offsets and consumer groups. These two concepts are the foundation of Kafka’s scalability, fault tolerance, and reliability.

What Is an Offset?

An offset is a unique sequential ID assigned to each message within a Kafka partition.

• Represents the position of a record in a partition
• Offsets are immutable and always increase
• Maintained per consumer group, not per consumer
• Kafka uses offsets to know which messages are already processed

Consumer Groups

A consumer group is a set of consumers that work together to consume data from a topic.

• Enable horizontal scaling
• Allow parallel message processing
• Improve fault tolerance

Important Rules:

• Each partition is consumed by only one consumer within a group
• Multiple consumer groups can consume the same topic independently
• If a consumer fails, Kafka rebalances partitions automatically

Offset Management Strategies

Kafka supports two offset commit strategies:

Auto Commit (Default)

• Kafka commits offsets automatically at intervals
• Simple, but risk of message loss if processing fails

Manual Commit (Recommended)

• Application commits offset after successful processing
• Provides better reliability and control

Disable auto-commit in configuration:

spring.kafka.consumer.enable-auto-commit=false

Manual Acknowledgment

Spring Kafka supports manual acknowledgment for precise offset control.

@KafkaListener(topics = "order-topic", groupId = "order-group")
public void listen(ConsumerRecord<String, String> record,
                   Acknowledgment ack) {

    // process message
    System.out.println("Received: " + record.value());

    // commit offset manually
    ack.acknowledge();
}

Dead Letter Queues (DLQ)

A Dead Letter Queue (DLQ) is used to handle failed messages safely without crashing the system.

When a message cannot be processed successfully after retries, it is redirected to a separate topic for further analysis instead of blocking the consumer.

DLQ in Kafka

Spring Kafka provides DeadLetterPublishingRecoverer.

@Bean
public DefaultErrorHandler errorHandler(
        KafkaTemplate<Object, Object> template) {

    DeadLetterPublishingRecoverer recoverer =
            new DeadLetterPublishingRecoverer(template);

    return new DefaultErrorHandler(recoverer);
}

Failed Messages Location

By default, failed messages are sent to:

original-topic.DLT

Why Use a DLQ?

• Prevents consumer crashes
• Isolates problematic messages
• Enables debugging and reprocessing
• Improves system reliability

Spring Cloud Stream with Kafka

Spring Cloud Stream abstracts Kafka-specific implementation details and provides a declarative, event-driven programming model.

Instead of writing Kafka-specific code, you work with logical bindings and message channels.

Why Use Spring Cloud Stream?

• Reduces Kafka/RabbitMQ-specific boilerplate code
• Makes it easier to switch messaging brokers
• Promotes clean event-driven architecture
• Simplifies producer and consumer development

Add Dependency

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-stream-kafka</artifactId>
</dependency>

Functional Programming Model

Spring Cloud Stream supports a functional programming model, where message producers and consumers are defined as simple Java functions.

This approach removes boilerplate code and makes event handling clean and expressive.

Consumer Example

@Bean
public Consumer<String> processOrder() {
    return message -> {
        System.out.println("Processing: " + message);
    };
}

Kafka Binder Configuration

To explicitly configure Kafka as the messaging broker, set the default binder:

spring.cloud.stream.defaultBinder=kafka

Spring Cloud Stream will now route all message bindings through Kafka.

Kafka Strengths

Kafka is designed as a high-performance event streaming platform, making it ideal for large-scale distributed systems.

Key Advantages

• Extremely high throughput
• Built-in event replay capability
• Horizontal scalability with partitions
• Strong ordering guarantees per partition
• Ideal for event backbones and streaming pipelines

Difference Between Kafka vs RabbitMQ

Conclusion

Apache Kafka is a powerful event streaming platform built for high throughput, scalability, and reliability. With Spring Boot and Spring for Apache Kafka, developers can easily build scalable producers and consumers, manage offsets, handle failures using Dead Letter Topics, and process events in real time.

Kafka is ideal for event-driven microservices, streaming pipelines, and systems that require event replay, making it the perfect choice for building modern, data-driven architectures.