Bug 63729
| Summary: | sheriffbot should use the list of flakey tests from the TestFailures page | ||
|---|---|---|---|
| Product: | WebKit | Reporter: | Jessie Berlin <jberlin> |
| Component: | Tools / Tests | Assignee: | Nobody <webkit-unassigned> |
| Status: | NEW | ||
| Severity: | Normal | CC: | abarth, aroben, eric |
| Priority: | P2 | ||
| Version: | 528+ (Nightly build) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| URL: | http://build.webkit.org/TestFailures/ | ||
Jessie Berlin
When sheriffbot complains on the IRC channel about a commit breaking the tests on a bot, and it turns out only to be a flakey test, people start ignoring sheriffbot.
sheriffbot should use the list of flakey tests, possibly the one that the TestFailures page creates, to make sure it only yells about real breakages as much as is possible.
| Attachments | ||
|---|---|---|
| Add attachment proposed patch, testcase, etc. |
Eric Seidel (no email)
I support this. Python already has logic to avoid flaky tests. It would be nice if the two approaches could be combined. :)
Eric Seidel (no email)
Hmm.. at least I think he does.
http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/net/failuremap.py
http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/tool/commands/sheriffbot.py#L79
I thought he waited to see something fail twice before reporting.
Eric Seidel (no email)
Maybe his flaky test detection got deleted when we stopped having him kick builders? Adam?
Adam Barth
Jessie is saying he should use information about http://build.webkit.org/TestFailures/