Course Preview

Apache Kafka Fundamentals

Apache Kafka is a distributed event-streaming platform that stores and delivers high-throughput streams of records. Producers write events to topics partitioned across a cluster, and consumers read them independently, enabling decoupled real-time data pipelines and event-driven architectures.

itDistributed systems, messaging, and integration | OpenSkills.info

Course pathWalk it in order

Look it upDip in anytime

Go furtherLeaves this page

Don't Panic — Apache Kafka Fundamentals

Apache Kafka is a shared, durable record of events: a producer writes an event once, and several consumer groups can each read it without pulling it away from the others. That sounds suspiciously like a message broker with ambitions. The important difference is that Kafka keeps the event according to retention policy, so a consumer can return later and read history again.

The useful picture is a topic divided into partitions. A partition is an append-only log with its own order and offsets. Kafka has not misplaced the global order; it never promised one across partitions. A stable key sends related records to the same partition, which is how an order's created and paid events avoid becoming an interpretive dance.

A consumer group remembers its own position in each partition. Members of one group share partitions for parallel work; a different group receives an independent view of the same records. That gives billing and analytics separate subscriptions without building a direct connection from every producer to every destination. It also means offsets matter: commit before work and a failure can skip it; commit after work and a failure can repeat it.

Replication makes a partition survive broker failures when the replication, acknowledgments, and in-sync-replica settings agree. No single setting gets to wear the whole durability hat. Retention and compaction answer a different question: how much past remains available for replay or rebuilding keyed state. Reading is not deletion, which is wonderfully useful until a replay repeats a payment or an email.

Kafka Connect moves records between Kafka and outside systems when the flow is reusable. Kafka Streams runs transformations, joins, and aggregations in a Java application. KRaft controllers manage metadata while brokers handle partition data and client traffic. Those pieces are related, but they are not interchangeable boxes with unusually confident names.

Read the Intro for the complete architecture and failure behavior. Use the Slides when the producer-to-partition-to-group path needs a visual reset. Keep the Cheatsheet nearby for settings and decision rules. Then run the Practice Reference and Exercise to watch partitions, groups, offsets, and replay behave in a local broker before any external side effect is invited to the party.

Where this skill leads

Relevant careers

See how this topic contributes to broader role-level skill maps.

Sources