This blog is a preview of my session at Sirus Connects on October 9, 2026, about keeping a 24/7 sensing platform running: the failures we catalogued, the principles that came out of them, and the two operations we run to put missing data back.
Speaker:
The hard part of streaming isn't that things fail. It's that you can't redo the work.
It’s a Wednesday morning. You get in, and there’s a mail from a client: data is missing from their report. So you go and look. A sensor that reports every fifteen minutes, months of clean data, and then a flat stretch of nothing between six in the morning and noon on a Tuesday. Nobody was paged. No alarm fired. By the time the client noticed, the source had already moved on. The data existed once. It doesn’t any more.
I’m Hannes, data engineer at Sirus, and I spend my days keeping Urban Sense alive: a 24/7 sensing platform across seven Belgian cities, with data arriving continuously from hundreds of devices, for consumers who expect it to just keep coming. I have personally broken most of what I’ll describe on stage. Why that gap happened matters, but it isn’t what this talk is about. The talk is about putting the data back while everything around it keeps running.
Why streaming isn't batch
In batch, failure is comparatively gentle. A job fails at two in the morning, somebody fixes the bug and runs it again on Thursday. That works because of one property nobody writes down: the input was still there. Streaming takes that away. A source that pushes may keep no record of what it sent. A source you pull from may only ever offer the last fifteen minutes. A queue has a retention setting, and when it expires, so does your second chance. The hard part of streaming isn’t that things fail. It’s that you can’t redo the work.
A catalogue of failures
So we sat down and wrote out every category of failure our platform can have. Not incidents, categories. Eleven of them, grouped by where in the pipeline they happen: getting data in, doing something with it, and writing it somewhere. Some announce themselves. An unavailable source throws an error and a format change makes a transformation fail loudly. The dangerous ones are quiet: a gap, or a poll that never fired because the platform itself was down. Those sit there for days without a line in any log, and they reach a client before they reach us.
Four principles
Four things kept coming up in the answers next to those eleven entries. Treat the raw incoming data as the only source of truth and everything else as something you can rebuild. Make redoing work safe, so a replay corrects wrong output instead of duplicating it. Fail fast and loudly, validate what arrives and alert on what doesn’t, and give the rejected data somewhere to wait. And accept that some downtime is unavoidable, then narrow the part of the system that genuinely has to stay up to the path from the source to the log.
Two ways to put the data back
Then there is what we actually do once something has gone wrong. There are only two operations: repair one window from the archive, or replay the whole log. Most of the difficulty is knowing which one you need, and the answer is decided by a number most teams set once and never revisit. The retention window on your log is the length of your undo history.
What I'll show at Sirus Connects
At Sirus Connects I’ll walk through our failure catalogue in a way you can repeat for your own platform, the four principles with the concrete patterns behind them, the two repair operations step by step, and the handful of things I would get right on day one if I were starting again. None of it is clever. All of it came from something actually breaking.
If you’ve ever stared at a gap in your time series wondering how you resolve it, come find me at Sirus Connects on October 9.
Want to follow this live?
Join this session at Sirus Connects on October 9, 2026, from 14h15 to 15h. Register and secure your spot.
Can’t make it? Reach out to us, we are happy to share our experience and discuss how to make your streaming platform resilient to change.
