RESOLVED FIXED 59279
The bots should learn from expected failures without having to retry 
https://bugs.webkit.org/show_bug.cgi?id=59279
Summary The bots should learn from expected failures without having to retry
Adam Barth
Reported 2011-04-23 09:31:17 PDT
The bots should learn from expected failures without having to retry
Attachments
Patch (1.54 KB, patch)
2011-04-23 09:33 PDT, Adam Barth 		no flags
DetailsFormatted DiffDiff
now with test case (4.68 KB, patch)
2011-05-01 23:28 PDT, Eric Seidel (no email) 		no flags
DetailsFormatted DiffDiff
Updated with more tests (12.90 KB, patch)
2011-05-02 00:32 PDT, Eric Seidel (no email) 		no flags
DetailsFormatted DiffDiff
Patch for landing (13.10 KB, patch)
2011-05-02 00:47 PDT, Eric Seidel (no email) 		eric: commit-queue+
DetailsFormatted DiffDiff
Show Obsolete (3) View All Add attachment proposed patch, testcase, etc.
Adam Barth
Comment 1 2011-04-23 09:33:06 PDT
Created attachment 90853 [details] Patch
Eric Seidel (no email)
Comment 2 2011-04-23 09:35:44 PDT
Comment on attachment 90853 [details] Patch Cute. But we need a test.
Adam Barth
Comment 3 2011-04-23 11:00:07 PDT
> Cute. But we need a test. Do you think we should report failure at the end?
Eric Seidel (no email)
Comment 4 2011-04-23 11:35:28 PDT
Comment on attachment 90853 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=90853&action=review I'm not sure what you mean. > Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py:217 > + if self._expected_failures.failures_were_expected(first_results): This also could be complicated by flaky tests. We won't get to here in the flaky test case (we would have stopped to report them already).
Adam Barth
Comment 5 2011-04-23 11:44:46 PDT
(In reply to comment #4) > (From update of attachment 90853 [details]) > View in context: https://bugs.webkit.org/attachment.cgi?id=90853&action=review > > I'm not sure what you mean. In the line where I added a FIXME, we could report failure instead of returning False (which causes a retry). > > Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py:217 > > + if self._expected_failures.failures_were_expected(first_results): > > This also could be complicated by flaky tests. We won't get to here in the flaky test case (we would have stopped to report them already). The only way flaky tests cause problem here is if we think one test failed turing the clean run when it should have passed. The result of that is only slightly less test coverage.
Ojan Vafai
Comment 6 2011-04-26 17:02:42 PDT
Comment on attachment 90853 [details] Patch R- per lack of test
Eric Seidel (no email)
Comment 7 2011-05-01 22:55:18 PDT
Comment on attachment 90853 [details] Patch This should be easy to test. :)
Eric Seidel (no email)
Comment 8 2011-05-01 23:28:24 PDT
Created attachment 91886 [details] now with test case
Eric Seidel (no email)
Comment 9 2011-05-02 00:32:01 PDT
Created attachment 91890 [details] Updated with more tests
Adam Barth
Comment 10 2011-05-02 00:39:48 PDT
Comment on attachment 91890 [details] Updated with more tests View in context: https://bugs.webkit.org/attachment.cgi?id=91890&action=review This is a patch of amazingness now. Thanks you. :) > Tools/Scripts/webkitpy/tool/bot/commitqueuetask_unittest.py:88 > + self._test_run_counter = -1 Why does this start at -1?
Eric Seidel (no email)
Comment 11 2011-05-02 00:47:36 PDT
Created attachment 91892 [details] Patch for landing
Eric Seidel (no email)
Comment 12 2011-05-02 00:47:44 PDT
(In reply to comment #10) > (From update of attachment 91890 [details]) > View in context: https://bugs.webkit.org/attachment.cgi?id=91890&action=review > > This is a patch of amazingness now. Thanks you. :) > > > Tools/Scripts/webkitpy/tool/bot/commitqueuetask_unittest.py:88 > > + self._test_run_counter = -1 > > Why does this start at -1? I've updated the comments to explain.
Eric Seidel (no email)
Comment 13 2011-05-02 00:53:17 PDT
Committed r85463: <http://trac.webkit.org/changeset/85463>
Note You need to log in before you can comment on or make changes to this bug.