Kafka

Apache Kafka is a distributed event streaming platform organized around a single data structure, the append-only log. Built at LinkedIn to move activity data between systems and open sourced in 2011, it now claims more than eighty percent of the Fortune 100 as users. A cluster of brokers stores named streams called topics, producers append records to them, and consumers read at their own pace by advancing an offset.

The mental shift that makes Kafka click is that it is not a queue. A queue deletes a message once a worker acknowledges it. Kafka retains every record for a configured window regardless of who has read it, and each reader is nothing more than a cursor position. That one choice buys cheap fan-out to many independent consumers, replay of history, and durable buffering between systems that run at different speeds.

Almost everything else follows from three numbers you choose per topic, the partition count, the replication factor, and the retention window, plus one field you choose per record, the key.

A replicated log, not a queue

A Kafka cluster is a set of brokers that store topics, and a topic is split into partitions. Each partition is an ordered, immutable sequence of records on disk that is only ever appended to. Every record gets a sequential offset, and reading a record does not remove it. A traditional message broker deletes a message once a consumer acknowledges it, so the queue itself is the source of truth about what work remains. Kafka instead keeps everything for a retention window while each consumer moves a cursor forward, so the same bytes can be read by any number of independent readers, at different speeds, and again from the beginning when something needs to be rebuilt.

This is the idea Jay Kreps laid out in The Log. The write-ahead log inside every database, the replication stream between a primary and its replicas, and the state machine replication literature are all the same structure, a totally ordered sequence of records. Two deterministic consumers that apply the same log end up in the same state. Kafka takes the log out of the database and makes it a first-class, shared piece of infrastructure that any system can write to or follow.

The append-only restriction is also why a system that writes everything to disk is fast. The official design docs cite roughly 600 MB/s for linear writes on a six-disk 7200 rpm array against about 100 KB/s for random writes, a gap of more than six thousand times. Kafka never builds a large in-heap cache. It writes into the OS page cache and lets the kernel manage memory, and on the read path it uses the sendfile system call to move bytes from page cache to socket without copying them through user space. Producers batch and compress records before sending, so I/O stays large and sequential on both ends.

A record carries a value plus an optional key, timestamp, and headers. The key controls partition placement, and ordering is guaranteed only inside one partition, never across a topic. A topic is the logical name, partitions are the physical unit of parallelism, ordering, and replication, and each partition lives on several brokers as one leader plus followers.

How to use it

The native client is Java, and most other languages wrap the C library librdkafka. In Python that wrapper is confluent-kafka. The example below is a small order pipeline, a producer inside the checkout service and a consumer inside the billing service.

enable.idempotence=true silently upgrades the producer to acks=all with safe retries and a bounded number of in-flight batches, and it has been the default since Kafka 3.0. linger.ms trades a few milliseconds of latency for fuller batches, and compression multiplies effective throughput because Kafka pays mostly per byte, not per record.

On the consumer side the order of operations is the whole contract. Doing the work first and committing second gives at-least-once processing, since a crash between the two steps replays the record. Committing first would give at-most-once. Handlers therefore need to be idempotent, for example by writing the consumed offset into the same database transaction as the side effect, or by enforcing a unique constraint on the order id.

import json

from confluent_kafka import Producer, Consumer

producer = Producer({
    "bootstrap.servers": "kafka1:9092,kafka2:9092,kafka3:9092",
    "enable.idempotence": True,   # implies acks=all + safe retries
    "compression.type": "lz4",
    "linger.ms": 5,               # wait up to 5 ms to fill a batch
})

# same key -> same partition -> strict per-order ordering
producer.produce(
    topic="orders",
    key=order["order_id"],
    value=json.dumps(order),
)
producer.flush()

consumer = Consumer({
    "bootstrap.servers": "kafka1:9092",
    "group.id": "billing",            # one cursor per group
    "auto.offset.reset": "earliest",  # first run starts at offset 0
    "enable.auto.commit": False,      # commit only after real work
})
consumer.subscribe(["orders"])

while True:
    msg = consumer.poll(timeout=1.0)
    if msg is None or msg.error():
        continue
    charge_customer(msg.key(), msg.value())  # do the work first
    consumer.commit(msg)                     # then advance the cursor

Partitions, offsets, and consumer groups

When a producer sends a record, the partitioner decides where it lands. With a key, the default is murmur2(key) % num_partitions, so all records for one key form a strictly ordered subsequence. Without a key, modern clients use a sticky partitioner that fills a batch for one partition and then rotates, which spreads load evenly at the cost of any ordering relationship between records.

That modulo has a sharp edge. Changing the partition count changes the mapping for every key, so old events for a user sit in one partition while new ones land in another, and per-key ordering silently breaks. The standard practices are to over-partition on day one, and when a topic truly must grow, to create a new topic with more partitions, switch producers to it, let consumers drain the old topic to zero lag, then move them over.

Consumers join a consumer group, and the group coordinator assigns each partition to exactly one consumer in that group. Two consumers over three partitions means one of them owns two. A seventh consumer over six partitions sits idle. When membership changes the group rebalances, and the classic eager protocol paused the whole group during reassignment. The KIP-848 protocol, generally available since Kafka 4.0 and enabled on brokers by default, moves assignment into the broker and rebalances incrementally, so one joining consumer no longer stops everyone. Consumers opt in by setting group.protocol=consumer (release announcement).

Progress is just a number. Each group periodically commits, per partition, the offset it will read next into an internal compacted topic named __consumer_offsets, fifty partitions by default. The gap between a partition's head offset and the committed offset is consumer lag, the most watched Kafka health metric. Because groups are independent entries in that topic, a new team can attach a brand new group to an old topic and start from offset zero without touching anyone else, which is exactly what the diagram shows.

Keyed routing concentrates load, so a viral key becomes a hot partition. The mitigations are all key engineering. Salt the key with a small random suffix and re-aggregate downstream, widen it into a compound key such as ad id plus region, or drop the key entirely when ordering does not matter. None of this is automatic, which is why key choice is the main design decision you make with Kafka.

Producers hash order ids onto three partitions. The billing group splits them between two consumers near the head of the log, while the analytics group reads the same data from offsets around thirty thousand records behind. Neither group affects the other, and both persist their cursors in __consumer_offsets.

ISR replication, acks, and exactly-once

Each partition has one leader replica that takes all writes and, by default, all reads, plus followers on other brokers that fetch from it continuously. Replicas of the same partition never share a broker, and with the usual replication factor of three a partition survives two broker losses. The controller spreads leaders across the cluster so no single broker carries all the traffic.

The leader tracks the in-sync replica set, the followers that are caught up to within replica.lag.time.max.ms, thirty seconds by default. A record counts as committed once every ISR member has it, and only ISR members are eligible for leader election, so a committed record survives as long as one in-sync replica lives. Electing a stale replica anyway, called unclean leader election, trades data loss for availability and is off by default.

The producer chooses its point on the durability curve with acks. acks=0 is fire and forget. acks=1 returns once the leader has the record, which loses data if the leader dies before followers fetch it. acks=all returns only after the full ISR has the record. Paired with min.insync.replicas=2 on a replication factor of three, the cluster keeps accepting writes with one broker down and starts rejecting them, rather than quietly weakening the guarantee, when two are down. Producer acks and consumer offset commits are unrelated mechanisms, one confirms delivery in, the other records progress out.

Out of the box the end-to-end guarantee is at-least-once. A producer retry after a lost acknowledgment can write a duplicate, and a consumer crash between processing and committing replays a record.

The idempotent producer fixes the duplicate write. Each producer holds a producer id and stamps every batch with a per-partition sequence number, the broker persists the highest sequence in the replicated log itself, and a retried batch with an already-seen sequence is discarded. Because the sequence lives in the log, a newly elected leader keeps deduplicating correctly. Confluent measured the overhead as negligible, and the feature is on by default since Kafka 3.0 (Confluent).

Transactions, added with the rest of exactly-once in Kafka 0.11 (2017), extend this across partitions. A producer with a transactional.id registers with a transaction coordinator, which journals transaction state to an internal log topic and writes commit or abort markers into every touched partition. Consumers running with isolation.level=read_committed skip aborted data. The decisive trick is that offset commits are themselves writes to a topic, so consuming, processing, and producing can commit atomically, which is exactly-once stream processing. Confluent measured about a three percent throughput cost versus plain acks=all for 100 ms transactions, and Kafka Streams enables the whole stack with processing.guarantee=exactly_once_v2. The guarantee stops at Kafka's edge, a call to an external API inside the loop still needs its own idempotency.

# durability
acks=all                      # wait for the full in-sync replica set
replication.factor=3
min.insync.replicas=2         # with RF 3: tolerate one broker down
unclean.leader.election.enable=false

# exactly-once producer
enable.idempotence=true       # producer id + sequence numbers
transactional.id=billing-svc-7

# exactly-once consumer
isolation.level=read_committed

Retention, log compaction, and replay

By default a topic uses cleanup.policy=delete with retention.ms of seven days and no size cap (retention.bytes=-1). The broker deletes whole closed segments once they age out, and consumption has no effect on retention at all. Retention is purely a storage budget decision, a topic can keep an hour of data or a year of it.

cleanup.policy=compact changes the contract from a window of events to the latest value per key. A background cleaner rewrites old segments once the uncleaned fraction passes a dirty ratio threshold, keeping at least the newest record for every key. A record with a null value acts as a tombstone that marks its key deleted, and the tombstone itself is removed after delete.retention.ms, one day by default. Compacted topics are how Kafka stores changelogs, consumer offsets, database change capture feeds, and Kafka Streams state store backups, since a new reader can bootstrap the full current state and then follow live updates from one topic.

Retention plus offsets makes replay a first-class operation. Reset a fixed consumer's offsets to reprocess a bad week, attach a fresh group at earliest to rebuild a cache or backfill a search index, or treat the topic as the system of record in an event-sourced design. Tiered storage (KIP-405) pushes closed segments to object storage, so long retention is priced like S3 rather than like broker disks.

KRaft: the cluster manages its own metadata

Kafka originally outsourced coordination to ZooKeeper, which stored topic configuration, tracked broker liveness, and elected the controller broker that assigns partition leaders. That meant operating a second distributed system with its own tooling and failure modes, and it set a practical ceiling on cluster size, because a newly elected controller had to reload the entire metadata state from ZooKeeper before acting, and with hundreds of thousands of partitions that pause got long.

KIP-500 replaced all of it with KRaft. A small quorum of controller nodes runs the Raft consensus protocol and stores cluster metadata as an event-sourced internal log that brokers replicate and replay. Standby controllers already hold current state, so failover is nearly immediate, and the design lifts the partition ceiling into the millions. There is a pleasing symmetry in it, Kafka's own coordination layer is now a replicated log, the same abstraction it sells to everyone else.

KRaft shipped as early access in Kafka 2.8 in 2021 and was declared production ready in 3.3 in 2022. Kafka 4.0, released on March 18, 2025, removed ZooKeeper support entirely, so every supported cluster now runs KRaft (release announcement).

When a queue beats the log

Kafka has no per-message acknowledgment. A consumer can only say it has finished everything up to offset n, so one poisonous record blocks its whole partition until you move it aside, retries and dead letter queues are patterns you build with extra topics rather than features you enable, and there is no per-message priority or delayed delivery. Consumer parallelism inside a group is hard-capped by the partition count.

For a plain work queue those are exactly the features you want, which is why SQS or RabbitMQ often wins for job dispatch. They give per-message acks, visibility timeouts, redelivery counts, built-in dead letter queues, and as many competing workers as load requires with no partition arithmetic. A web crawler or an email sender rarely cares about ordering and cares a great deal about retrying one failed item in isolation.

The log wins when the data outlives a single handling. Several independent consumer groups over one stream, replay and backfill, per-key ordering, sustained throughput in the hundreds of megabytes per second, and stateful stream processing with Flink or Kafka Streams are all natural on Kafka and awkward on a queue.

A few misuses recur. Kafka is not blob storage, so keep records around a megabyte or smaller and pass an object-store pointer for anything large. It is not a request-response bus, since the pull model and batching add latency a synchronous call should not pay. Global ordering across a whole topic requires a single partition, which caps throughput at one broker. And at small scale a managed queue is simply less to operate. KIP-932 share groups, in early access as of Kafka 4.0, add real per-record acknowledgment on top of the log, so the gap between the two models is narrowing (Confluent).

In production

Kafka's home installation is still the best scale reference. LinkedIn reported passing seven trillion messages per day in 2019 across more than one hundred clusters, with over four thousand brokers serving one hundred thousand topics and seven million partitions, carrying everything from activity tracking to database replication (LinkedIn Engineering). They run a house fork with patches such as a maintenance mode for draining brokers and an enforced minimum replication factor on topic creation.

Per-machine throughput has been the headline since the beginning. Kreps' 2014 benchmark pushed about two million 100-byte writes per second through three commodity machines with three-way replication (Benchmarking Apache Kafka). Confluent's 2020 run of the OpenMessaging benchmark on identical hardware measured Kafka peaking at 605 MB/s against 305 MB/s for Pulsar and 38 MB/s for RabbitMQ, while holding a p99 latency of about 5 ms at 200 MB/s (Confluent benchmark).

For capacity sketches in a design discussion, a well-provisioned broker is commonly estimated at about a terabyte of hot data and up to around a million small messages per second. A modest cluster covers most systems, so the interesting engineering lives in key choice and partition counts rather than raw capacity, and hot keys fail long before hardware does.

In practice Kafka arrives with an ecosystem. Kafka Connect moves data in and out of databases and object stores, Debezium turns database write-ahead logs into topics for change data capture, Flink and Kafka Streams consume topics for stateful processing, and managed services such as Confluent Cloud and AWS MSK run the brokers. The project's claim of more than eighty percent of the Fortune 100 matches how often it appears as the default backbone between microservices.

Follow-up questions

• How does Kafka guarantee ordering, and where does that guarantee end? Ordering holds only within a single partition. All records with the same key hash to the same partition via murmur2(key) % num_partitions and are appended in arrival order, so per-key ordering is guaranteed as long as the partition count never changes. There is no ordering across partitions or across a topic, and resizing a topic remaps keys and breaks the guarantee for in-flight history.
• What does acks=all actually promise, and what can still go wrong? The producer gets an acknowledgment only after every replica in the in-sync replica set has the record, so the write survives any failure that leaves one ISR member alive. The trap is ISR shrinkage. If followers fall behind and drop out, acks=all can degrade to just the leader, which is why you pair it with min.insync.replicas=2 so the broker rejects writes instead of silently weakening the contract.
• How does Kafka achieve exactly-once semantics? Two mechanisms layered together. The idempotent producer attaches a producer id and per-partition sequence numbers persisted in the log, so retried batches are deduplicated even across leader failover. Transactions add a coordinator, commit and abort markers, and read_committed consumers, and because consumer offsets are themselves stored in a topic, a consume-process-produce cycle commits atomically. The guarantee covers Kafka-to-Kafka flows only, external side effects still need their own idempotency.
• A consumer processed a message but crashed before committing the offset. What happens? After rebalancing, the partition's new owner resumes from the last committed offset and processes the record again, which is Kafka's default at-least-once behavior. You handle it with idempotent processing, for example storing the offset in the same database transaction as the side effect, or by moving the whole pipeline onto transactions with read_committed.
• Your keyed topic has one partition melting under a viral key. What are the options? Salt the key with a bounded random suffix and re-aggregate downstream, widen it to a compound key like ad id plus region, or drop the key entirely if ordering does not matter. Adding partitions alone does not help the hot key and breaks existing key-to-partition mappings, so the durable fix is key design plus generous over-partitioning from day one.
• When would you pick SQS or RabbitMQ over Kafka? For task queues where each message is independent work. They offer per-message acks, visibility timeouts, native retries and dead letter queues, priorities, and unbounded competing consumers. Kafka wins when multiple consumer groups need the same stream, when replay and long retention matter, when per-key ordering matters, or when throughput reaches hundreds of megabytes per second. KIP-932 share groups are adding queue semantics to Kafka, but they only reached early access in 4.0.

References

1. Apache Kafka documentation, Design, Primary source for the page cache and zero-copy rationale, the pull model, ISR semantics, compaction mechanics, and delivery guarantees.
2. Narkhede, Exactly-Once Semantics Are Possible (Confluent, 2017), How the idempotent producer and transactions work, including the roughly three percent transaction overhead measurement.
3. Apache Kafka 4.0.0 release announcement (2025), ZooKeeper removal, the KIP-848 consumer rebalance protocol going GA, and KIP-932 share groups in early access.
4. LinkedIn Engineering, Kafka at 7 trillion messages per day (2019), Cluster, broker, topic, and partition counts behind the largest publicly documented deployment.
5. Confluent, Benchmarking RabbitMQ vs Kafka vs Pulsar (2020), 605 MB/s peak throughput and roughly 5 ms p99 latency at 200 MB/s on identical hardware across the three systems.
6. Kreps, The Log (LinkedIn, 2013), The mental-model essay presenting the log as the unifying abstraction behind databases, replication, and stream processing.