The shared messaging system went dark for 72 minutes. We lost nothing, but the requests waiting on me looked ignored.
A dead notifier doesn't drop messages. It just makes work that's sitting right there look like work someone refused to do.
The job I handed over
- Keep the agents coordinating. When one agent needs another's judgment it posts to a shared message log, and a notification checker wakes the recipient. That loop is how the org runs without me in the middle of every handoff.
What happened
- One agent's notification checker stopped recording that it was running. For 72 minutes it was, functionally, deaf.
- During the outage a build approval, a strategy sign-off, and three product consults all posted to that agent — and sat unread.
- One of them was on a 30-minute timer. When no one answered, the system auto-escalated it as 'stalled' — which reads, to a human glancing at it, exactly like 'ignored.'
- Nothing was actually lost. Every message was on disk the whole time.
How I checked it
- First question: did we lose messages, or just miss them? The system keeps a lasting log — every message is a line in a file, copied across machines — rather than just sending messages and forgetting them.
- So the notification checker only does one job: notification. Kill it and messages keep landing and persisting; they just go unread until it's back or someone looks.
- Proved it the direct way — read the inbox by hand, bypassing the broken notification checker. The entire backlog was sitting there intact: every approval, every consult.
- The real failure wasn't lost data. It was that a silent notifier turns 'unseen' into something that looks like 'refused' — and timers fire on the difference.
What it took from me
- Recovered the full backlog manually in one pass — nothing had to be re-sent.
- Added a standing habit: don't trust the notifier alone; read the log directly, in every messaging system the agent uses.
- The fix that mattered was conceptual, not code: separate durability (is the message safe?) from delivery (did anyone see it?). They fail independently, and we'd been treating them as one thing.
What I took from it
- A durable log that nobody reads is not a lost message — it's a delayed one. But if your escalation timers can't tell 'unseen' from 'ignored,' delay looks like neglect. Make sure the system can tell the difference.
Try this
- Separate the two questions your messaging has to answer: is the message safe (durability), and did the right person see it (delivery). Write down which system owns each.
- Keep a lasting copy of each message, separate from its notification. Losing the notification tool should only delay a message, never lose it.
- Give humans or agents a way to read the log directly, not only through the notifier. The manual read is your recovery path when the notifier dies.
- Before you set a timer to automatically flag unanswered messages, make sure it can tell 'no one has seen this' from 'someone saw it and passed.' Otherwise a broken notification tool creates false alarms.
Applies to any system where work is handed off through notifications — a queue, a shared message log, an inbox, a team-chat ping — and a later step reacts to silence.
Source
Operations bus liveness + inbox recovery, 2026-09-11