For weeks, some bots have not run all the tests without anyone noticing. https://bugs.webkit.org/show_bug.cgi?id=83075 https://bugs.webkit.org/show_bug.cgi?id=83076 https://bugs.webkit.org/show_bug.cgi?id=83077 Increasingly, gardeners don't look at the waterfall since they just use garden-o-matic. As such they don't see when NRWT is bailing early due to too many unexpected crash/timeout. Also the waterfall doesn't give a big warning either (http://code.google.com/p/chromium/issues/detail?id=121229).
Is there a bit in the results JSON we can look for? We already have UI for losing test coverage. We just need to trigger that UI when we bail early for too many failures.
(In reply to comment #1) > Is there a bit in the results JSON we can look for? We already have UI for losing test coverage. We just need to trigger that UI when we bail early for too many failures. No, not directly (I don't think), but it would certainly be easy to add one. I am tempted to want to make the bot (or at least the step) go purple when this happens, but normally purple is reserved for the infrastructure code failing, which isn't what's happening here.
Looks to me like there's an "interrupted" bit in full_results.json that covers this.
Looks like it works: http://test-results.appspot.com/testfile?builder=Webkit%20Mac10.6&name=full_results.json has "interrupted":true.
(In reply to comment #4) > Looks like it works: http://test-results.appspot.com/testfile?builder=Webkit%20Mac10.6&name=full_results.json has "interrupted":true. Yeah, I guess I had forgotten about that, but that's what that's for.
I believe "interupted" is also used by results.html to show a warning. :)
Still an issue?
AFAIK