Apache Kafka Fundamentals

Apache Kafka moves event records between systems and retains them for later reading. It combines messaging with a distributed, replayable log.

That combination changes how you design data flows. A consumer does not remove a record when it reads it. Kafka keeps the record according to the topic's retention policy. Consumers track their own position and can reread retained records.

The problem Kafka solves

Applications often need the same event for different purposes. An order event might feed fulfillment, fraud detection, analytics, and customer notifications.

Direct point-to-point integrations couple every producer to every consumer. Each new destination adds another connection, failure mode, and deployment dependency.

Kafka puts a durable event stream between those systems. Producers write records without knowing every consumer. Consumers read records at their own pace and keep independent positions.

This model supports four common purposes:

Publish and subscribe to event streams.
Store event streams for a configured period.
Process event streams as they arrive or after replay.
Connect databases and services through reusable source and sink connectors.

The core path

Keep this path in mind:

Producer → topic → partition → broker → consumer group

A producer sends a record to a topic. The record usually contains a key, value, timestamp, and optional headers.

A topic names a stream of related records. Kafka splits each topic into one or more partitions.

A partition is an ordered, append-only log. Each record receives an offset, which identifies its position within that partition.

A Kafka server is a broker. A cluster spreads topic partitions across brokers. Each partition has one leader replica and may have follower replicas.

A consumer fetches records. Consumers that share a consumer group divide the group's assigned partitions among themselves.

Partitions define scale and order

Partitions are Kafka's unit of storage, replication, and parallel consumption. They are also the boundary of Kafka's ordering guarantee.

Records within one partition have a defined order. Kafka does not create one total order across every partition in a topic.

The producer's partitioning strategy decides where a record goes. A stable key can route related records to the same partition. That preserves their relative order within that partition.

The tradeoff is direct. More partitions allow more parallel work, but they split the stream into more independent orderings.

Within one consumer group, one partition is assigned to one consumer at a time. Extra consumers beyond the partition count do not increase active consumption parallelism.

Different consumer groups are independent. An analytics group and a billing group can each read the same topic without competing for the same records.

Replication and durability

Kafka can replicate each partition across brokers. The leader handles reads and writes for the partition. Followers copy the leader's log.

If the leader broker fails, Kafka can elect an eligible follower. Replication therefore protects availability and data, but only when configuration and operating conditions support it.

Three controls matter together:

The replication factor sets how many replicas a partition has.
The producer's acks setting controls which broker acknowledgment is required before a send completes.
min.insync.replicas sets the minimum in-sync replica count required for writes when the producer requires all acknowledgments.

Replication factor alone is not a durability promise. Weak acknowledgments can confirm a write before enough replicas have it. A strict minimum can reject writes when too few replicas remain in sync.

Offsets, replay, and delivery behavior

An offset is a position in one partition. A consumer group commits offsets so it can resume after a restart or partition reassignment.

The committed offset represents the next record the group expects to process. Committing before processing risks losing work from the application's perspective. Committing after processing can cause a record to run again after a failure.

This leads to familiar delivery behaviors:

At most once: commit before processing; failures can skip work.
At least once: process before committing; failures can repeat work.
Exactly once: coordinate input offsets and output writes atomically within a defined processing boundary.

Kafka supports idempotent production and transactions. Kafka Streams can use Kafka transactions for exactly-once processing within Kafka. External side effects still require coordination or idempotent application logic.

Treat duplicates as a design concern even when your Kafka path uses transactions. A payment API or email service does not join a Kafka transaction automatically.

Retention and compaction

Kafka stores records independently of whether consumers have read them. A topic's cleanup policy controls what remains.

The delete policy removes old log segments after configured time or size limits. It is the default cleanup policy.

The compact policy retains the latest value for each key over time. Compaction does not turn a topic into an immediately cleaned key-value table. Old versions can remain until compaction processes eligible segments.

A topic can use both delete and compact policies. Choose based on the recovery and history your application needs.

Retention makes replay possible. A new consumer can process retained history, and an existing group can reset its position. Replay also repeats application side effects unless the consumer handles them safely.

Kafka's surrounding tools

Kafka includes more than the broker and basic clients.

Kafka Connect moves data between Kafka and external systems through connectors. Source connectors import records. Sink connectors export records.

Kafka Streams is a Java library for building applications that transform and join Kafka records. It manages partitioned processing, local state, recovery, and Kafka-backed results.

The Admin API manages topics, configurations, and other cluster resources programmatically.

Modern Kafka clusters use KRaft controllers for metadata management. Controllers form a metadata quorum, while brokers store and serve topic data. You should understand the distinction before operating a cluster.

Where Kafka fits

Kafka fits when several consumers need the same event stream, replay matters, or sustained data flow must survive component outages.

Representative uses include event-driven applications, log aggregation, database change distribution, data integration, stream processing, and activity tracking.

Kafka is a poor default when you need direct request-and-response communication, tiny operational scope, or arbitrary row updates and indexed queries.

It also adds real costs. You must plan partitions, replication, retention, security, upgrades, capacity, lag monitoring, and failure recovery. Managed Kafka moves some work to a provider, but it does not remove application-level choices.

A practical learning path

Run the official quickstart and trace one record from producer to consumer.
Create a multi-partition topic and observe key-based partitioning.
Run two consumers in one group, then compare a second independent group.
Stop a consumer and inspect lag and committed offsets.
Test producer acknowledgments, idempotence, retries, and failure behavior.
Learn retention, compaction, replication, and partition reassignment.
Build one Kafka Connect flow and one Kafka Streams application.
Study security, observability, capacity planning, and KRaft operations before production ownership.