My progress
Menu

IoT and Smart Systems Engineer · Focus area 3: Device-to-cloud communication

IO8: Store-and-forward and edge computing

Goal: keep readings safe on the device until the server confirms them, survive power cuts part-way through a write, reconnect without draining the battery or flooding the server, decide on the device which readings are worth sending, and handle duplicates, gaps and late data on the server.

You'll need: IO3, IO4, IO7, Module 24 (Asynchronous Programming), Module 57 (Transactions and Indexes), Module 94 (Distributed Systems), and the JalSetu practice kit (firmware/buffer.py, firmware/uplink.py, firmware/edge.py, sim/flash.py, sim/collector.py, tests/test_buffer.py).

Where this is used: water and energy meters, cold-chain trackers in lorries, farm sensors, vehicle data loggers and factory machines all record while the network is away and send later.

How it differs: a web app that loses its connection shows an error and the person tries again. A device in a field has nobody to press retry; it has to keep its data, survive the battery being pulled, and catch up on its own when the network returns.

Time: reading about 3 hours · practice (the Try it tasks) about 3 hours · assignment about 5 hours.

Lessons

  1. 1Store first, send laterReading 15–25 minutes, plus practice✓
  2. 2Surviving power cuts and torn writesReading 15–25 minutes, plus practice✓
  3. 3Reconnecting without draining the batteryReading 15–25 minutes, plus practice✓
  4. 4Edge computing: deciding on the deviceReading 15–25 minutes, plus practice✓
  5. 5The server side: duplicates, gaps and late dataReading 15–25 minutes, plus practice✓

🛠️Assignment: JalSetu's buffer and edge logic

  1. Build the flash buffer: append-only records with length and CRC32, seq, and buf.ack written via a temporary file and rename.
  2. Send oldest first with QoS 1 in batches of up to 20, removing readings only after PUBACK.
  3. Add Wi-Fi timeouts and exponential backoff with a cap, and report readings waiting and dropped in the status.
  4. Add a deadband, an hourly heartbeat and immediate alarms.
  5. Write docs/buffer-design.md with the sections the test asks for.
  6. Run pytest tests/test_buffer.py until all 10 tests pass.

Self-check before you finish:

✅Module check

Explain in a few sentences, in your own words:

How should a device keep and send data when its network comes and goes?

Show a good answer

A good answer mentions: writing each reading to flash first with a sequence number and removing it only after the broker confirms it; append-only records with checksums and write-then-rename so power cuts leave a usable buffer; a capacity with counted, reported drops; timeouts, exponential backoff with a cap and jitter, and batches oldest first; edge filtering with a deadband, a heartbeat and alarms that bypass filters; and a server that stores one copy per device and sequence number, reports gaps, and charts by the reading's own time.

📚Official references

Official documentation and guidance for this module's topics, chosen on 2 November 2026. Tools, laws and standards change, so check the current version before you rely on a detail. If a link has moved, search the same site.

📝Module quiz

10 questions · 10 marks each · 100 marks in all · pass mark 70% · retry as often as you like ·

1. A sensor keeps its unsent readings in a RAM list and sends them hourly. Brownouts reset it often. What's the most important fix?

Why each option is right or wrong

  • “Send every reading straight away, so nothing ever waits in the device for long” (Incorrect): Readings still vanish whenever the network is down at reset.
  • “Add a bigger battery so the brownouts in the area happen a little less often” (Incorrect): Brownouts come from the supply, and resets still happen.
  • “Double the size of the RAM list, so it can hold two hours of readings at once” (Incorrect): A bigger RAM list is still wiped by a reset.
  • “Write each reading to flash first, and remove it only after the server confirms it” is correct.

    Flash survives resets, and confirmation means nothing leaves too early.

    • Kit: firmware/buffer.py.

📖 Revise: Lesson 1: Store first, send later

Question ID sp-iot-08-q1 · Report a mistake in this question
2. A device sends readings 501 to 520 in one batch. The connection drops after PUBACKs for 501 to 508. What should be in the buffer?

Why each option is right or wrong

  • “Nothing at all, because every one of the 20 readings in the batch was sent before the drop” (Incorrect): Sent isn't the same as confirmed.
  • “Readings 501 to 520, so the whole batch is sent again in the same order” (Incorrect): 501 to 508 were confirmed and can go.
  • “Only reading 520, since it was the last one sent before the connection dropped” (Incorrect): 509 to 519 weren't confirmed either.
  • “Readings 509 to 520, kept in flash and waiting to be sent again on the next connection” is correct.

    Only confirmed readings leave the buffer.

    • Then: the next send starts at 509.

📖 Revise: Lesson 1: Store first, send later

Question ID sp-iot-08-q2 · Report a mistake in this question
3. After a power cut, a device's buf.log ends with half a record. What should start-up code do?

Why each option is right or wrong

  • “Treat the whole file as empty, since part of it couldn't be read correctly” (Incorrect): That throws away good data.
  • “Stop the device and wait for a technician to visit and repair the flash file” (Incorrect): A field device has to recover by itself.
  • “Leave the half record in place and append the next readings after it as usual” (Incorrect): The damaged record would break reading the file later.
  • “Keep all the whole records before it, cut off only the damaged tail, and then carry on” is correct.

    Every confirmed and complete reading stays.

    • How: length and CRC checks find the tail.

📖 Revise: Lesson 2: Surviving power cuts and torn writes

Question ID sp-iot-08-q3 · Report a mistake in this question
4. Why does JalSetu write buf.ack to a temporary name and then rename it?

Why each option is right or wrong

  • “Renaming is faster than writing, so the device can get back to sleep sooner” (Incorrect): Speed isn't the reason.
  • “Renamed files take less space, so the flash chip can hold more readings” (Incorrect): Size doesn't change.
  • “A temporary name hides the file from attackers who read the device's flash” (Incorrect): It's about power cuts, not secrecy.
  • “A power cut part-way then leaves either the old file or the new one, not a mix of both” is correct.

    The rename swaps them in one step on many file systems.

    • Kit: Project 2's test 2.

📖 Revise: Lesson 2: Surviving power cuts and torn writes

Question ID sp-iot-08-q4 · Report a mistake in this question
5. Wi-Fi is down for a day. A device tries to join on every sending wake and waits 30 seconds each time. What's the best change?

Why each option is right or wrong

  • “Keep trying on every wake but leave the radio on longer, in case the network comes back” (Incorrect): That spends even more battery.
  • “Stop trying entirely after the first failure until someone resets the device by hand” (Incorrect): Nobody may visit for weeks.
  • “Turn off the buffer during an outage, so the device has less data to send later” (Incorrect): Readings would be lost.
  • “Give up on each join sooner, and back off exponentially between tries, up to a cap” is correct.

    Fewer, shorter attempts during an outage.

    • Kit: 8 seconds, then 2, 4, 8 and 16 wakes.

📖 Revise: Lesson 3: Reconnecting without draining the battery

Question ID sp-iot-08-q5 · Report a mistake in this question
6. After a 20-minute broker outage, 50,000 devices reconnect in the same few seconds, and the broker fails again. What helps most?

Why each option is right or wrong

  • “Make every device reconnect at exactly the same time, but a minute later than now” (Incorrect): The wave arrives a minute later.
  • “Remove the retry logic so devices only reconnect when they're switched off and on” (Incorrect): Devices would stay offline.
  • “Send larger batches on reconnect, so each device finishes its upload faster” (Incorrect): That makes the wave heavier.
  • “Add a small random delay (jitter) to each device's reconnect, along with backoff” is correct.

    Reconnects spread out over minutes.

    • Name: avoiding a "thundering herd".

📖 Revise: Lesson 3: Reconnecting without draining the battery

Question ID sp-iot-08-q6 · Report a mistake in this question
7. A tank sensor keeps a reading only when the level changes by 10 mm. A tank stays full for two days. What might the server wrongly conclude?

Why each option is right or wrong

  • “That the device is sending far too much data and needs a bigger deadband” (Incorrect): It's sending almost nothing.
  • “That the tank's level is changing quickly and the water is being used” (Incorrect): Nothing is changing.
  • “That the device is running the wrong firmware version and needs an update” (Incorrect): Nothing points to a version problem.
  • “That the device is dead, because a steady tank and a dead sensor look the same” is correct.

    A heartbeat tells them apart.

    • Kit: an hourly heartbeat.

📖 Revise: Lesson 4: Edge computing: deciding on the device

Question ID sp-iot-08-q7 · Report a mistake in this question
8. A low-tank alarm happens on a wake when the device isn't due to send. What should it do?

Why each option is right or wrong

  • “Wait for the next sending wake, which is at most an hour away at the usual rate” (Incorrect): An alarm can't wait.
  • “Skip the alarm, because the deadband filter says the change isn't big enough yet” (Incorrect): Alarms bypass filters.
  • “Store it in RAM and send it after the next reset, to save the battery for later” (Incorrect): RAM is wiped, and the alarm is late.
  • “Send it straight away, outside the usual sending schedule, then go back to sleep” is correct.

    Urgent events skip the wait.

    • Kit: Project 2's test 9.

📖 Revise: Lesson 4: Edge computing: deciding on the device

Question ID sp-iot-08-q8 · Report a mistake in this question
9. Daily water use on a dashboard shows spikes after every network problem. Readings carry device and seq. What's the likely cause and fix?

Why each option is right or wrong

  • “The devices' clocks are wrong; set them by hand on each visit to the site” (Incorrect): Clocks don't double values.
  • “The deadband is too small; raise it so that the devices send far fewer readings each day” (Incorrect): Filtering doesn't cause double counts.
  • “Batteries are low after outages; replace them after each network problem” (Incorrect): Batteries don't make spikes.
  • “Resends after lost PUBACKs are counted twice; store one copy per device and seq” is correct.

    Lost PUBACKs cause resends; a unique key ignores copies.

    • Kit: sim/collector.py counts copies.

📖 Revise: Lesson 5: The server side: duplicates, gaps and late data

Question ID sp-iot-08-q9 · Report a mistake in this question
10. A backlog of 6 hours of readings arrives at 9 am after an outage. How should the chart show them?

Why each option is right or wrong

  • “All at 9 am, because that's when the server received them from the device” (Incorrect): That makes a burst at the wrong time.
  • “Left off the chart entirely, since late readings can't be trusted at all” (Incorrect): They're valid readings, just late.
  • “Averaged into one point at 9 am, so the chart doesn't look too crowded” (Incorrect): That hides what happened.
  • “At each reading's own time (ts), spread across the six hours” is correct.

    The chart shows when things happened.

    • Also: sort by seq or ts, not arrival.

📖 Revise: Lesson 5: The server side: duplicates, gaps and late data

Question ID sp-iot-08-q10 · Report a mistake in this question

ProLearners Assist