Kafka
A distributed, replayable log that decouples producers from consumers at massive scale.
Introduction
Apache Kafka is a distributed event streaming platform built around an append-only, partitioned, replicated log. Producers append events to topics; consumers read them at their own pace. Unlike a traditional queue, Kafka retains events and lets multiple independent consumers replay the same stream, which makes it the backbone of event-driven architectures.
Why it exists
Systems need to move high volumes of events between many producers and many consumers without coupling them together, losing data, or forcing everyone to process at the same speed. Point-to-point integrations become an unmaintainable mesh. Kafka exists to be a durable, high-throughput central nervous system: producers publish once, any number of consumers subscribe, and events are retained so consumers can fail, catch up, or replay history.
Analogy
Kafka is like a newspaper archive rather than a mailbox. A mailbox (traditional queue) delivers a letter once and it's gone. The newspaper is printed to an ordered archive; many readers read at their own pace, new subscribers can start from any past date, and re-reading yesterday's paper is trivial. The paper isn't removed when you read it.
How it works
A topic is split into partitions, each an ordered, append-only log. Every record in a partition has a monotonically increasing offset. Ordering is guaranteed only within a partition — records are routed to a partition by a key (same key → same partition → ordered).
Partitions are replicated across brokers for durability; one replica is the leader, others followers. Producers choose a durability level via acks (0/1/all). Consumers track their position by committing offsets; because Kafka retains records, a consumer can replay from an earlier offset. Consumers form consumer groups: each partition is assigned to exactly one consumer in a group, so parallelism is capped by the partition count, and a rebalance reassigns partitions when consumers join or leave.
Delivery is at-least-once by default (commit offsets after processing), so consumers must be idempotent. Kafka scales horizontally by adding partitions and brokers, and retains data by time or size regardless of consumption.
- 1
A producer serializes an event and picks a partition via the record key (hash) or round-robin.
- 2
The partition leader appends the record and replicates it to followers per the acks setting.
- 3
The record gets the next offset and is retained for the configured time/size.
- 4
A consumer in a group polls its assigned partitions and processes records in offset order.
- 5
After processing, the consumer commits its offset; on restart it resumes from the last commit (replayable).
Interactive
Architecture
Topic → partitions → consumer group
Topic 'orders' (ordered per partition)
Consumer group (1 partition → 1 consumer)
Records with the same key land on the same partition and stay ordered; each partition feeds exactly one consumer in the group.
In code
import { Kafka } from "kafkajs"
const kafka = new Kafka({ brokers: ["broker:9092"] })
const producer = kafka.producer()
export async function emitOrderEvent(orderId: string, event: OrderEvent) {
await producer.send({
topic: "orders",
// key = orderId → all events for one order go to the same partition → ordered.
messages: [{ key: orderId, value: JSON.stringify(event) }],
acks: -1, // 'all': wait for in-sync replicas → strongest durability
})
}await consumer.subscribe({ topic: "orders", fromBeginning: false })
await consumer.run({
eachMessage: async ({ message }) => {
const evt = JSON.parse(message.value!.toString())
// Redelivery is normal after a crash → dedupe on a stable id.
const isNew = await markProcessed(evt.eventId) // INSERT ... ON CONFLICT DO NOTHING
if (!isNew) return // duplicate → skip
await applyOrderEvent(evt)
// Offset is committed after processing → at-least-once semantics.
},
})In the real world
Where you've seen this
A commerce platform publishes every domain event (OrderPlaced, PaymentSucceeded, InventoryReserved) to Kafka. Independent consumers do different jobs off the same streams: one updates the read model, one sends emails, one feeds analytics, one drives fraud detection. Adding a new consumer (say, a recommendation trainer) requires zero changes to producers — it just subscribes and can replay history.
When to use it
Reach for it when
- High-throughput event streaming to many independent consumers.
- Event-driven architectures, event sourcing, and CDC pipelines.
- When you need replay / reprocessing of historical events.
- Decoupling services so producers and consumers evolve independently.
Avoid it when
- Simple task queues with a single consumer and low volume (a broker like RabbitMQ/SQS is simpler).
- Request/response or low-latency RPC (Kafka is a log, not an RPC system).
- Small apps that don't need retention, replay, or high fan-out — Kafka's operational weight isn't justified.
- Per-message priority or complex routing (RabbitMQ fits better).
Trade-offs
Advantages
- Very high throughput and horizontal scalability via partitions.
- Durable, replayable retention decoupled from consumption.
- Multiple independent consumer groups read the same stream.
- Strong ordering within a partition and tunable durability.
Disadvantages
- Operationally heavy to run and tune (partitions, retention, rebalancing).
- Only partition-level ordering, not global.
- At-least-once by default — consumers must be idempotent.
- Rebalances can pause consumption; hot partitions skew load.
Kafka trades operational complexity and latency for durability, throughput, and replay. Partition count trades parallelism/ordering: more partitions mean more consumer parallelism but weaker cross-entity ordering and more overhead. acks trades latency for durability (0 fast/lossy, all slow/durable). Retention trades storage cost for replay ability. Compared to RabbitMQ/SQS, Kafka wins on throughput, retention, and fan-out but loses on simplicity, per-message routing, and low-latency single-consumer tasks.
| Kafka vs RabbitMQ | Kafka | RabbitMQ |
|---|---|---|
| Model | Distributed log (pull, retained) | Message broker (push, ack-and-delete) |
| Replay | Yes — re-read by offset | No — consumed messages are gone |
| Ordering | Per partition | Per queue |
| Throughput | Very high | High (lower than Kafka) |
| Best for | Streaming, event sourcing, fan-out | Task queues, complex routing, RPC-ish |
Common mistakes
Watch out for
- Expecting global ordering across a topic (it's per-partition only).
- Too few partitions, capping consumer parallelism; or a bad key causing hot partitions.
- Assuming exactly-once for free instead of building idempotent consumers.
- Committing offsets before processing (risking data loss) — commit after.
- Using Kafka as a database or as low-latency RPC.
Failure thinking
What breaks it
A topic has 3 partitions and you run 5 consumers in one group. What happens?
Only 3 consumers get a partition each; the other 2 sit idle as hot standbys, because a partition is assigned to at most one consumer per group. Parallelism is capped by partition count — to use 5 consumers you need at least 5 partitions. The idle consumers still provide fast failover on a rebalance.
Kafka (or a broker) goes down mid-stream. Do you lose events?
With replication (replication.factor ≥ 3) and acks=all, a committed record survives a broker loss because it exists on in-sync replicas; a follower is promoted to leader. Producers may retry (dedupe via idempotent producer / consumer). Data loss mainly occurs if you used acks=0/1 with too few in-sync replicas — durability is a configuration choice you make explicitly.
Think like a senior
Senior Engineer Insight
The two facts that reveal understanding: ordering is per-partition (so the partition key is a design decision, not an afterthought), and delivery is at-least-once (so idempotent consumers are mandatory, not optional).
Senior Engineer Insight
Kafka is a log, not a queue. That single reframing explains replay, multiple consumer groups, retention, and why you don't 'delete' a message after reading it.
Remember
Ordering is guaranteed within a partition, never across a topic.
Remember
Kafka is at-least-once — consumers must be idempotent.
Interview questions
How does Kafka guarantee ordering, and how do you scale consumers?
Active recall
Check yourself
In Kafka, what guarantees message ordering?
Practical challenges
Ordered, idempotent order pipeline
Model an order event pipeline on Kafka with correct ordering and duplicate safety.
Requirements
- Choose topic(s) and a partition key that preserves per-order ordering.
- Implement at least two consumer groups reading the same stream.
- Make each consumer idempotent against redelivery.
Constraints
- Commit offsets after processing.
- Assume at-least-once delivery.
Acceptance criteria
- Events for one order are always processed in order.
- A forced redelivery does not create a duplicate order or duplicate email.
- Adding a new consumer group requires no producer changes.
Edge cases
- A rebalance mid-batch.
- A poison message that always fails.
Bonus
- Add a dead-letter topic for poison messages.
- Show replay by resetting a group's offsets.
Reflection
- Why does the partition key double as your ordering and load-balancing decision?
Architecture challenge
Design the event backbone for an order pipeline: OrderPlaced → PaymentRequested → PaymentSucceeded → InventoryReserved → OrderConfirmed, with separate email and analytics consumers. Choose topics, partition keys, and partition counts; guarantee per-order ordering; and ensure a consumer crash or redelivery can't create duplicate orders or emails.
Flashcards
Summary
Kafka is a distributed, partitioned, replicated log that decouples producers from many independent consumers and retains events for replay. Ordering holds within a partition (chosen by key), parallelism is bounded by partition count within a consumer group, and default at-least-once delivery makes idempotent consumers mandatory. It excels at high-throughput streaming and event-driven architectures but is operationally heavy and wrong for simple queues or RPC.
Key takeaways
- Topic → partitions → ordered logs; ordering is per partition, set by the key.
- One partition per consumer in a group ⇒ parallelism capped by partition count.
- Retention enables replay and many independent consumer groups.
- At-least-once by default ⇒ consumers must be idempotent.
Your notes
Saved to this device