Bug 83078 – garden-o-matic should show a *BIG* warning when a builder didn't run to completion

• Status: NEW
• Component: Tools / Tests

[Ojan Vafai]

For weeks, some bots have not run all the tests without anyone noticing.

https://bugs.webkit.org/show_bug.cgi?id=83075
https://bugs.webkit.org/show_bug.cgi?id=83076
https://bugs.webkit.org/show_bug.cgi?id=83077

Increasingly, gardeners don't look at the waterfall since they just use garden-o-matic. As such they don't see when NRWT is bailing early due to too many unexpected crash/timeout. Also the waterfall doesn't give a big warning either (http://code.google.com/p/chromium/issues/detail?id=121229).

[Adam Barth]

Is there a bit in the results JSON we can look for?  We already have UI for losing test coverage.  We just need to trigger that UI when we bail early for too many failures.

[Dirk Pranke]

(In reply to comment #1)
> Is there a bit in the results JSON we can look for?  We already have UI for losing test coverage.  We just need to trigger that UI when we bail early for too many failures.

No, not directly (I don't think), but it would certainly be easy to add one.

I am tempted to want to make the bot (or at least the step) go purple when this happens, but normally purple is reserved for the infrastructure code failing, which isn't what's happening here.

[Ojan Vafai]

Looks to me like there's an "interrupted" bit in full_results.json that covers this.

[Ojan Vafai]

Looks like it works: http://test-results.appspot.com/testfile?builder=Webkit%20Mac10.6&name=full_results.json has "interrupted":true.

[Dirk Pranke]

(In reply to comment #4)
> Looks like it works: http://test-results.appspot.com/testfile?builder=Webkit%20Mac10.6&name=full_results.json has "interrupted":true.

Yeah, I guess I had forgotten about that, but that's what that's for.

[Eric Seidel (no email)]

I believe "interupted" is also used by results.html to show a warning. :)

[Eric Seidel (no email)]

Still an issue?

[Ojan Vafai]

AFAIK