Some problems to detect: - When a bot goes down. This is somewhat tricky, because we currently don't list bots anywhere on the server side, they just connect at will. So all we know that a bot didn't check in for 24 hours, but we don't know if that's expected. - When a bot goes into a state where it fails every time. For example, one bot had a stale git lockfile today, so it couldn't process any patches. There is some status stored that should make it possible to come up with heuristics (e.g. when one bot didn't report a pass in 24 hours, but others worked fine). - When tests are in a state such that bots have to retry most of the time. Seems somewhat tricky to formalize.