IO8: Store-and-forward and edge computing
The server side: duplicates, gaps and late data
🎯You'll be able to
Store device data so resends don't double-count, gaps are visible, late data lands in the right place, and device health is clear.
🏠The everyday picture
A teacher collecting homework ticks off each roll number once, even if a pupil hands in two copies, and notices who hasn't handed anything in.
📘What it really is
-
Duplicates are normal: QoS 1 and lost PUBACKs mean the same reading can arrive twice (IO7). Store readings with a unique key of device and
seq, so a second copy is ignored (Module 57). -
Gaps: if
seq311 and 313 arrive but not 312, the server knows something is missing. Report gaps, and compare them with the dropped count the device reports. -
Late data: a backlog sent after an outage arrives hours late. Use the reading's own time (
ts) for charts, not the time it arrived, and let alerts know the data is old. -
Clocks: a device's clock can drift or reset. Sync it when connected (for example with NTP) and sanity-check times on the server.
-
Device health: last seen, battery, readings waiting, readings dropped and firmware version per device, so the fleet view shows which devices need attention.
-
Ordering: readings can arrive out of order. Sort by
seqorts, not by arrival.
📝A worked example
Input: T07 sends seq 310 to 312. The PUBACK for 311 is lost, so 311 arrives twice. Later, 314 arrives but 313 doesn't.
Steps:
- Store with a unique key (device, seq): the second 311 is ignored.
- The server sees 313 missing and checks T07's status: dropped count 0, readings waiting 1. So 313 is still on the device.
- On the next send 313 arrives, and the gap closes.
Expected output: one copy of each reading, and a gap that's explained instead of a mystery.
🌍Where it's used
- Meter data platforms: one bill per reading, however many times it arrived.
- Logistics: late scans from a lorry put in the right place on the timeline.
- Fleet dashboards: last-seen and battery for every device.
📝Case scenario
A water utility's dashboard showed usage spikes every few days. The spikes were resends after lost acknowledgements, counted twice. A unique key on device and sequence number removed them.
🖱️Try it
A device sends readings 1 to 10. Readings 4 and 7 arrive twice, and 9 is missing. Write what the server stores and what it reports. Success: one copy each of 1 to 8 and 10, and a gap at 9.
⚠️Common mistakes
- Counting every message. Sign: totals too high after network trouble. Fix: unique key on device and
seq. - Charting by arrival time. Sign: a burst of hours of data at one moment. Fix: chart by the reading's time.
- No health view. Sign: dead devices found weeks later. Fix: last seen and battery per device.
🛠️For your project
The kit's sim/collector.py stores one copy per (device, seq) and counts copies, so Project 2's tests can check resends.
✅Quick check
How does the server ignore a resent reading?
Show the answer
It stores readings with a unique key of device and sequence number, so a second copy isn't added.
Why chart readings by their own time rather than arrival time?
Show the answer
Because a backlog can arrive hours late, and arrival time would put it in the wrong place.