DT4: Streams, big data and file formats
Events and Kafka
🎯You'll be able to
Explain events, topics, partitions, offsets and consumer groups, and why event keys decide ordering.
🏠The everyday picture
A railway announcement board logs every train event in order: arrived, departed, delayed. Different people read it at their own pace, and each remembers where they stopped reading.
📘What it really is
- An event is a record that something happened:
order_placed,payment_captured,rider_assigned, with a time and details. -
Apache Kafka stores events in topics (one per kind of event). Managed versions include Confluent Cloud, Amazon MSK and Redpanda (compatible with Kafka).
-
A topic is split into partitions, so many computers can write and read in parallel. Within one partition events keep their order; across partitions they don't.
-
The key decides the partition: all events with key
order SS-100562go to the same partition, so that order's events stay in order. -
Each event has an offset (its position). A consumer group shares the partitions between its members and remembers the offset each has reached, so a restarted consumer carries on where it stopped.
-
Events are kept for a set time (for example 7 days), so a new consumer can read history, and a broken one can re-read after a fix.
- Delivery: most set-ups give at-least-once delivery: an event can arrive twice after a retry. Consumers must handle repeats (Lesson 2 of DT3: idempotent writes keyed on an event id).
📝A worked example
Input: SabziSathi publishes order_status_changed events with the order id as the key, to a topic with 6 partitions. A consumer group of 3
workers updates a live order table.
Expected output (illustrative, not run in the kit): each worker reads 2 partitions. All of order SS-100562's status changes (placed → packed → delivered) arrive at the same worker in order. If a worker stops, its 2 partitions move to the other workers, which carry on from the saved offsets. Some events may be processed twice, so the table is updated by order id, not appended.
🌍Where it's used
- Banks and fintech: transaction events shared by fraud, ledger and notification services.
- E-commerce: order, stock and delivery events.
- Data platforms: change data capture (CDC) streams each database change as an event, for example with Debezium.
📝Case scenario
A delivery app used the rider's id as the key for order events. When one order changed riders, its events went to two partitions and arrived out of order: "delivered" was processed before "picked up". Keying by order id kept each order's events in order.
🖱️Try it
For each event, choose a key that keeps the right events in order: payment_captured, stock_level_changed, customer_address_updated.
Success: payment id or order id, product and store together, and customer id, each with a reason.
⚠️Common mistakes
- Expecting order across a whole topic. Sign: events arrive "out of order" across partitions. Fix: order only matters per key; pick the key carefully.
- Assuming each event arrives once. Sign: double counts after a restart. Fix: idempotent writes keyed on an event id.
- Too few partitions. Sign: consumers can't keep up even with more workers. Fix: plan partitions for growth; more workers than partitions sit idle.
🛠️For your project
Not in the kit's tests. The capstone's design notes ask which events SabziSathi would stream first, and with which key.
✅Quick check
Where does Kafka keep events in order?
Show the answer
Within a partition, so events with the same key stay in order.
What does a consumer group's offset do?
Show the answer
It records how far each member has read, so a restart carries on from there.