Bug 83078
| Summary: | garden-o-matic should show a *BIG* warning when a builder didn't run to completion | ||
|---|---|---|---|
| Product: | WebKit | Reporter: | Ojan Vafai <ojan> |
| Component: | Tools / Tests | Assignee: | Nobody <webkit-unassigned> |
| Status: | NEW | ||
| Severity: | Normal | CC: | abarth, dglazkov, dpranke, eric |
| Priority: | P2 | ||
| Version: | 528+ (Nightly build) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
Ojan Vafai
For weeks, some bots have not run all the tests without anyone noticing.
https://bugs.webkit.org/show_bug.cgi?id=83075
https://bugs.webkit.org/show_bug.cgi?id=83076
https://bugs.webkit.org/show_bug.cgi?id=83077
Increasingly, gardeners don't look at the waterfall since they just use garden-o-matic. As such they don't see when NRWT is bailing early due to too many unexpected crash/timeout. Also the waterfall doesn't give a big warning either (http://code.google.com/p/chromium/issues/detail?id=121229).
| Attachments | ||
|---|---|---|
| Add attachment proposed patch, testcase, etc. |
Adam Barth
Is there a bit in the results JSON we can look for? We already have UI for losing test coverage. We just need to trigger that UI when we bail early for too many failures.
Dirk Pranke
(In reply to comment #1)
> Is there a bit in the results JSON we can look for? We already have UI for losing test coverage. We just need to trigger that UI when we bail early for too many failures.
No, not directly (I don't think), but it would certainly be easy to add one.
I am tempted to want to make the bot (or at least the step) go purple when this happens, but normally purple is reserved for the infrastructure code failing, which isn't what's happening here.
Ojan Vafai
Looks to me like there's an "interrupted" bit in full_results.json that covers this.
Ojan Vafai
Looks like it works: http://test-results.appspot.com/testfile?builder=Webkit%20Mac10.6&name=full_results.json has "interrupted":true.
Dirk Pranke
(In reply to comment #4)
> Looks like it works: http://test-results.appspot.com/testfile?builder=Webkit%20Mac10.6&name=full_results.json has "interrupted":true.
Yeah, I guess I had forgotten about that, but that's what that's for.
Eric Seidel (no email)
I believe "interupted" is also used by results.html to show a warning. :)
Eric Seidel (no email)
Still an issue?
Ojan Vafai
AFAIK