The bots should learn from expected failures without having to retry
Created attachment 90853 [details] Patch
Comment on attachment 90853 [details] Patch Cute. But we need a test.
> Cute. But we need a test. Do you think we should report failure at the end?
Comment on attachment 90853 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=90853&action=review I'm not sure what you mean. > Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py:217 > + if self._expected_failures.failures_were_expected(first_results): This also could be complicated by flaky tests. We won't get to here in the flaky test case (we would have stopped to report them already).
(In reply to comment #4) > (From update of attachment 90853 [details]) > View in context: https://bugs.webkit.org/attachment.cgi?id=90853&action=review > > I'm not sure what you mean. In the line where I added a FIXME, we could report failure instead of returning False (which causes a retry). > > Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py:217 > > + if self._expected_failures.failures_were_expected(first_results): > > This also could be complicated by flaky tests. We won't get to here in the flaky test case (we would have stopped to report them already). The only way flaky tests cause problem here is if we think one test failed turing the clean run when it should have passed. The result of that is only slightly less test coverage.
Comment on attachment 90853 [details] Patch R- per lack of test
Comment on attachment 90853 [details] Patch This should be easy to test. :)
Created attachment 91886 [details] now with test case
Created attachment 91890 [details] Updated with more tests
Comment on attachment 91890 [details] Updated with more tests View in context: https://bugs.webkit.org/attachment.cgi?id=91890&action=review This is a patch of amazingness now. Thanks you. :) > Tools/Scripts/webkitpy/tool/bot/commitqueuetask_unittest.py:88 > + self._test_run_counter = -1 Why does this start at -1?
Created attachment 91892 [details] Patch for landing
(In reply to comment #10) > (From update of attachment 91890 [details]) > View in context: https://bugs.webkit.org/attachment.cgi?id=91890&action=review > > This is a patch of amazingness now. Thanks you. :) > > > Tools/Scripts/webkitpy/tool/bot/commitqueuetask_unittest.py:88 > > + self._test_run_counter = -1 > > Why does this start at -1? I've updated the comments to explain.
Committed r85463: <http://trac.webkit.org/changeset/85463>