Don't Panic: Event Streaming

Event streaming is what happens when a system stops sending a fact to one waiting recipient and writes it down where several readers can find it. An order is placed, a payment is authorized, or a sensor reports a reading. The fact becomes an event, and the producer can then get out of the way. This is courteous. It also prevents the producer from becoming the world's least popular switchboard.

The useful trick is retention. A work queue often removes a message after somebody handles it. A retained stream keeps records long enough for a new consumer to read old events, for an existing consumer to rebuild a view, or for a bug fix to have something to work with besides regret. That makes a stream closer to a shared history than a row of errands.

The history has compartments. A topic groups related events, and a partition is one ordered subdivision of that topic. Use a stable key when one customer's or order's events need to stay together. More partitions allow more parallel processing, which is excellent news for throughput and less excellent news for anyone expecting one grand universal order. That order does not appear merely because the topic name looks authoritative.

Consumers keep a reading position called an offset or cursor. A consumer group shares work among its members, while separate groups each get their own view of the same events. This is how one order event can lead to fulfillment, a notification, analytics, and fraud checks without the order service knowing the cast list.

The awkward part is failure, because it always arrives carrying paperwork. At-least-once delivery can repeat an event. A broker transaction has a boundary, but it does not automatically make a separate database or payment API repeat-safe. Give events stable identifiers and make the destination operation idempotent when possible. The Practice tab puts that rule under a lamp with a local broker and duplicate records.

Read the Intro for the full model and its limits. Use Slides when the producer-to-consumer shape needs to stay in your head. Keep the Cheatsheet nearby for delivery, partition, and recovery decisions. The Quiz checks the vocabulary, while Field Notes covers the costs that wait behind the cheerful words replay and parallelism. The point is not to worship the stream. It is to know when several independent reactions and readable history are worth the operational work.