Most of our bots run on a schedule with nobody watching. When one of them breaks loudly, we find out. The failures that cost us were the quiet ones, where a broken bot and a slow week produce exactly the same output: nothing.
We found four of these in the first weeks of running the fleet. They look different, but they are one problem.
A signed-out inbox looks like an empty one
Our business mail was first read through a browser session, because the plan we were on had no API. A bot checked the inbox and reported what was new. When the session expired, the bot saw no new mail and reported no new mail. Nothing in that report was false, and all of it was useless.
The first fix was a reachability line in the weekly rollup: did the bot actually reach the inbox, yes or no. The real fix was building a mail path a bot could read without a browser session, which we wrote up separately.
An empty balance looks like a quiet feed
Reading X through our connector costs credits per result. A routine that sweeps mentions every weekday will spend the balance down, and once it is empty the sweep returns nothing. That reads as a quiet day on X. We added a rule: every daily summary reports the balance, no unattended run spends more than a dollar, and the bot stops and asks below five.
Silent on success hides silent on failure
One routine pushed our ops repository every evening and was told to stay silent when it worked. That is a reasonable request until it stops working, because then it is also silent. Our fleet audit flagged it for other reasons, but this is the same gap: a routine whose only signal is its absence cannot tell you anything.
A strict filter looks like a dry market
Our lead-scoring bot's first run surfaced three leads. We tightened the fit gate, had it re-score all eight from that run, and it surfaced none. That was the right result for those leads. But a gate that is too strict and a week with no good leads produce the same empty list, and only one of them is working as intended.
So the bot keeps an append-only record of every lead it has seen, including the ones it dropped. An empty report with a full record behind it means the gate held. An empty report with an empty record means the bot did not look.
What to take from it
- For every unattended job, ask what its output looks like when it is broken. If the answer is "the same as a quiet day," it needs another signal.
- Report that the bot reached the source, not just what it found there.
- Report the resources a job spends, every run, so empty does not get mistaken for quiet.
- "Silent on success" needs a matching "loud on failure," or it is just silent.
- Keep a record of what was seen and rejected. A filter that shows its work can be told apart from a filter that did no work.