Inside Kafka: partitions, consumer groups and exactly-once

Apache Kafka is usually described as a message queue, and used as one. Underneath, it is something simpler and more useful: a partitioned, replicated, append-only log. Almost every property people rely on, and almost every surprise they run into, follows from that one design decision.

This article builds Kafka up from the log: how partitions give it scale and ordering, why it is fast on ordinary disks, how replication decides what is committed, how consumer groups share work and what a rebalance costs, and what “exactly once” does and doesn’t mean.

A log, not a queue

In a traditional queue, a message is delivered to one consumer and then deleted. In Kafka, a topic is a log: records are appended to the end, each gets a sequential offset, and they stay there until a retention policy removes them, whether or not anyone has read them. Consumers don’t remove anything. Each consumer group just remembers how far it has read.

That inversion has big consequences:

• Many independent consumers can read the same data at their own pace, without the producer knowing.
• A consumer can rewind and reprocess history, to recover from a bug or to build a new view of old data.
• A slow consumer doesn’t slow anyone else; it just has a growing lag.

Jay Kreps’s essay “The Log” (2013) argues that this abstraction, the same one inside every database’s write-ahead log and every consensus protocol, is the natural way to move data between systems.

Partitions and keys

A single log on a single machine can’t scale, so each topic is split into partitions, each an independent log that can live on a different broker. Order is guaranteed within a partition, and nowhere else.

Producers choose the partition. By default, a record with a key goes to murmur2(key) mod partitions, so every record with the same key lands in the same partition, in order. Use the entity id as the key (the order id, the user id) and you get exactly the ordering most applications need: all events for one order, in sequence. Records without a key are spread across partitions for balance.

Why it is fast

Kafka routinely moves hundreds of megabytes per second per broker, mostly by avoiding work rather than doing it cleverly:

• Sequential I/O only. Appends go to the end of a segment file, and reads are sequential scans. Disks, even spinning ones, are fast at this. The same insight drives LSM-trees.
• The OS page cache, not an application cache. Kafka writes to the file system and lets the kernel cache it. Consumers that keep up read recent data straight from memory.
• Zero-copy reads. For plaintext connections, data goes from the page cache to the socket with sendfile, without being copied through the JVM.
• Batching end to end. Producers batch records per partition and compress the batch. The broker stores the batch as is, and consumers decompress it. Nothing is re-encoded in between.

Each partition is stored as a sequence of segment files, each with a sparse index from offset to file position. Finding offset 1,000,000 is a binary search in a small index, then a short scan.

Retention and compaction

Records leave the log in one of two ways, set per topic:

• Delete retention drops whole segments older than a time or beyond a size limit. This suits event streams: clicks, logs, metrics.
• Log compaction keeps at least the latest record for every key and discards older ones in the background. A record with a null value, a tombstone, deletes its key after a grace period. A compacted topic is a durable, replayable table: read it from the beginning and you rebuild the current state. Kafka uses this itself, to store consumer offsets.

Replication

Each partition has a replication factor, typically 3. One replica is the leader and handles all writes. The others are followers that fetch from it, like consumers. Replicas that are caught up, within replica.lag.time.max.ms, form the in-sync replica set (ISR).

The high watermark is the highest offset that every ISR member has. Records below it are committed, and only they are visible to consumers. If the leader fails, a new one is chosen from the ISR, and because every ISR member has everything below the high watermark, nothing a consumer has read is lost.

The producer chooses its durability with acks. With acks=0 it doesn’t wait at all; with acks=1 it waits for the leader only, and loses data if the leader dies before followers copy it; with acks=all it waits until every current ISR member has the record. Pair acks=all with min.insync.replicas=2: if the ISR shrinks to one, the leader refuses writes rather than accepting them on a single copy.

Which broker leads which partition is metadata, and metadata needs consensus. Kafka originally kept it in ZooKeeper. Since KRaft (KIP-500), a quorum of controller nodes stores it in an internal Raft log, and Kafka 4.0 (2025) removed ZooKeeper support altogether. See the consensus article for how Raft works.

Consumer groups

Consumers that share a group.id divide a topic’s partitions among themselves. Each partition is assigned to exactly one consumer in the group, which is what preserves per-partition order. Different groups are fully independent and each sees every record.

Consumers report progress by committing offsets to an internal compacted topic. The committed offset is the next record to read, and it is where the partition resumes when its owner changes. When a consumer joins, leaves, or stops sending heartbeats, the group coordinator rebalances: it reassigns partitions among the live members.

Rebalances are the main operational pain in Kafka consumers, and several improvements target them:

• Eager rebalancing, the original protocol, revokes every partition from every member, then reassigns them all. The whole group stops, even for partitions that don’t move.
• Cooperative incremental rebalancing moves only the partitions that need to move, over a couple of rounds, while the rest keep processing.
• Static membership (group.instance.id) lets a restarting consumer reclaim its partitions without triggering a rebalance, as long as it returns within the session timeout. It is useful for rolling deploys.
• KIP-848, generally available in Kafka 4.0, moves assignment from the clients to the broker and makes rebalancing incremental by default.

A common cause of unexpected rebalances is slow processing: if the gap between two poll() calls exceeds max.poll.interval.ms, the consumer is considered dead, even though it is busy working.

At-most, at-least, exactly once

The delivery guarantee you get depends on when you commit, relative to when you process:

• Commit, then process: at-most-once. A crash after the commit skips the records that weren’t processed.
• Process, then commit: at-least-once. A crash before the commit reprocesses them. This is the usual choice, and it is what the figure shows.
• Exactly-once means the effect of each record happens once. Kafka provides it for pipelines that read from and write to Kafka. For side effects anywhere else, such as a database write, an email or an API call, you still need an idempotent consumer, as in the sagas article.

How exactly-once works

Kafka’s exactly-once semantics (KIP-98, 2017) are built from two mechanisms:

1. The idempotent producer. Each producer gets an id, and numbers its batches per partition. The broker remembers the last sequence number from each producer and discards duplicates, so a retry after a lost acknowledgement doesn’t write the record twice. This is on by default in modern clients.
2. Transactions. A producer with a transactional.id can write to several partitions and commit consumer offsets as one atomic unit. A transaction coordinator writes commit or abort markers into every partition involved, and consumers with isolation.level=read_committed only see records from committed transactions. The transactional id also fences zombies: when a restarted instance registers with the same id, the old one’s writes are rejected.

Together they make a consume–transform–produce step atomic: the output records and the input offsets commit together. If the step crashes, both are rolled back, and the input is reprocessed with no visible duplicates downstream. Kafka Streams uses exactly this when processing.guarantee=exactly_once_v2 is set.

Operating it

• Watch consumer lag, the gap between a partition’s end and a group’s committed offset. It is the single most useful health metric: rising lag means consumers can’t keep up.
• Beware hot partitions. One very active key, such as a large customer, puts all its traffic on one partition and one consumer. Spread such keys deliberately, at the cost of their ordering.
• Keep processing per record bounded, or move slow work off the poll loop, so consumers don’t exceed max.poll.interval.ms and trigger rebalances.
• Make retention match recovery. If a consumer can be down for a weekend, retention must be longer than a weekend, or it will come back to find its offset deleted.

Consume, transform, produce

A transactional pipeline stage with kafkajs: the idempotent, transactional producer; a read-committed consumer with auto-commit off; and the offsets sent inside the transaction, so outputs and progress commit together.

// Consume–transform–produce with Kafka transactions (kafkajs). The output records and
// the consumer's offsets commit atomically: either both happen or neither does.
import { Kafka } from "kafkajs";

const kafka = new Kafka({ clientId: "enricher", brokers: ["kafka-1:9092"] });
const producer = kafka.producer({
  transactionalId: "enricher-0", // stable per instance: fences off zombies after a restart
  idempotent: true,              // broker dedupes retried batches by (producer id, sequence)
  maxInFlightRequests: 1,
});
// read_committed: never see records from transactions that aborted or are still open.
const consumer = kafka.consumer({ groupId: "enricher", readUncommitted: false });

await Promise.all([producer.connect(), consumer.connect()]);
await consumer.subscribe({ topic: "orders" });

await consumer.run({
  autoCommit: false, // offsets are committed inside the transaction instead
  eachBatch: async ({ batch }) => {
    const tx = await producer.transaction();
    try {
      await tx.send({
        topic: "orders-enriched",
        messages: batch.messages.map((m) => ({ key: m.key, value: enrich(m.value) })),
      });
      const next = (BigInt(batch.lastOffset()) + 1n).toString(); // commit the NEXT offset to read
      await tx.sendOffsets({
        consumerGroupId: "enricher",
        topics: [{ topic: batch.topic, partitions: [{ partition: batch.partition, offset: next }] }],
      });
      await tx.commit();
    } catch (err) {
      await tx.abort(); // outputs are discarded and offsets not advanced: the batch is reprocessed
      throw err;
    }
  },
});

References

1. Kreps, J., Narkhede, N. & Rao, J. (2011). Kafka: a Distributed Messaging System for Log Processing. NetDB. microsoft.com/research
2. Kreps, J. (2013). The Log: What every software engineer should know about real-time data’s unifying abstraction. LinkedIn Engineering. linkedin.com/blog/engineering
3. Wang, G. et al. (2015). Building a Replicated Logging System with Apache Kafka. VLDB. vldb.org
4. Apache Kafka. KIP-98: Exactly Once Delivery and Transactional Messaging. cwiki.apache.org
5. Apache Kafka. KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum. cwiki.apache.org
6. Apache Kafka. KIP-848: The Next Generation of the Consumer Rebalance Protocol. cwiki.apache.org
7. Apache Kafka. Documentation: Design. kafka.apache.org
8. Kleppmann, M. (2017). Designing Data-Intensive Applications, ch. 11: Stream Processing. O’Reilly. dataintensive.net