Your Task API publishes a task.created event. The notification service consumes it and sends an email. But what happens if the consumer crashes after sending the email but before committing the offset? When it restarts, it re-reads the same event---and sends a second email. Your user receives duplicate notifications. Or worse: what if the consumer commits the offset before processing, crashes mid-processing, and the email never sends at all?
These aren't edge cases. In production systems handling millions of events daily, they happen regularly. Network partitions, broker restarts, consumer crashes, and deployment rollouts all create situations where the relationship between message consumption and processing outcome becomes uncertain.
This chapter examines the three delivery semantic guarantees Kafka can provide, the trade-offs each involves, and the practical patterns that make at-least-once---the most common choice---work reliably. By the end, you'll have a decision framework for choosing the right semantic for your use case and the implementation knowledge to build consumers that handle duplicates gracefully.
Every distributed messaging system must answer: how many times will a consumer see each message? The answer depends on where failures can occur and how the system handles them.
With at-most-once delivery, a message is delivered zero or one times. If anything fails, the message is lost rather than redelivered.
How it works:
Output:
When at-most-once is acceptable:
When at-most-once is dangerous:
With at-least-once delivery, every message is delivered one or more times. If anything fails, the message is redelivered rather than lost.
How it works:
Output (normal operation):
Output (crash and recovery):
Why at-least-once is the most common choice:
With exactly-once delivery, every message is delivered exactly one time---no losses, no duplicates. This requires coordinating the consumer's processing with Kafka's offset commits atomically.
How it works:
Output:
The hidden costs of exactly-once:
Which delivery semantic should you choose? The answer depends on your specific requirements:
Ask these questions in order:
1. Can you lose messages?
2. Can your consumer handle duplicates?
3. Can you make your consumer idempotent?
Why idempotent consumers are usually better than exactly-once:
Exactly-once in Kafka only works when:
Most real-world consumers write to databases, call APIs, send emails, or update caches. These operations don't participate in Kafka transactions. Making these consumers idempotent is simpler and more reliable than attempting exactly-once semantics.
The key insight: if your consumer can safely process the same message multiple times with the same result, at-least-once becomes as good as exactly-once from a business logic perspective.
Store processed event IDs and check before processing:
Output:
Use database constraints to prevent duplicates:
Output:
For state changes, check current state before applying:
Output:
Deduplication key (Pattern 1):
Database upsert (Pattern 2):
Version/state check (Pattern 3):
When designing your event processing strategy, you might start with a simple question and discover the nuances through exploration.
Your initial question:
"My notification service sends emails when tasks are created. How do I prevent duplicate emails?"
Exploring the problem:
This is actually two separate concerns:
For consumer reliability, use at-least-once (commit after processing). For idempotency, you need to track which emails you've already sent.
Discovering the implementation:
A simple approach uses Redis to track sent notifications:
Refining the approach:
But what if sending the email succeeds and Redis fails? You'd send the email again on the next attempt. For truly idempotent email sending, you might need:
What emerged from this exploration:
You built a kafka-events skill in Chapter 1. Test and improve it based on what you learned.
Ask yourself:
If you found gaps:
Apply these delivery semantics to your own event processing scenarios.
Setup: Open Claude Code or your preferred AI assistant in your Kafka project directory.
Prompt 1: Analyze Your Processing Requirements
What you're learning: Mapping real business requirements to delivery semantics and implementing appropriate idempotency patterns for each event type.
Prompt 2: Design Idempotency for External API Calls
What you're learning: Handling idempotency for external systems that don't natively support it---a common real-world challenge.
Prompt 3: Evaluate Exactly-Once vs Idempotent Consumer
What you're learning: Critical evaluation of exactly-once vs at-least-once trade-offs for a specific architecture, not just theoretical understanding.
Safety Note: Test your idempotency logic by deliberately causing failures (kill consumer mid-processing, simulate network partitions). Idempotency bugs often only surface under failure conditions that are hard to reproduce in development.