Bug 59279 – The bots should learn from expected failures without having to retry

• Component: New Bugs
• Status: RESOLVED FIXED

[Adam Barth]

The bots should learn from expected failures without having to retry

[Adam Barth]

Created attachment 90853 [details]
Patch

[Eric Seidel (no email)]

Comment on attachment 90853 [details]
Patch

Cute.  But we need a test.

[Adam Barth]

> Cute.  But we need a test.

Do you think we should report failure at the end?

[Eric Seidel (no email)]

Comment on attachment 90853 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=90853&action=review

I'm not sure what you mean.

> Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py:217
> +        if self._expected_failures.failures_were_expected(first_results):

This also could be complicated by flaky tests.  We won't get to here in the flaky test case (we would have stopped to report them already).

[Adam Barth]

(In reply to comment #4)
> (From update of attachment 90853 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=90853&action=review
> 
> I'm not sure what you mean.

In the line where I added a FIXME, we could report failure instead of returning False (which causes a retry).

> > Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py:217
> > +        if self._expected_failures.failures_were_expected(first_results):
> 
> This also could be complicated by flaky tests.  We won't get to here in the flaky test case (we would have stopped to report them already).

The only way flaky tests cause problem here is if we think one test failed turing the clean run when it should have passed.  The result of that is only slightly less test coverage.

[Ojan Vafai]

Comment on attachment 90853 [details]
Patch

R- per lack of test

[Eric Seidel (no email)]

Comment on attachment 90853 [details]
Patch

This should be easy to test. :)

[Eric Seidel (no email)]

Created attachment 91886 [details]
now with test case

[Eric Seidel (no email)]

Created attachment 91890 [details]
Updated with more tests

[Adam Barth]

Comment on attachment 91890 [details]
Updated with more tests

View in context: https://bugs.webkit.org/attachment.cgi?id=91890&action=review

This is a patch of amazingness now.  Thanks you.  :)

> Tools/Scripts/webkitpy/tool/bot/commitqueuetask_unittest.py:88
> +        self._test_run_counter = -1

Why does this start at -1?

[Eric Seidel (no email)]

Created attachment 91892 [details]
Patch for landing

[Eric Seidel (no email)]

(In reply to comment #10)
> (From update of attachment 91890 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=91890&action=review
> 
> This is a patch of amazingness now.  Thanks you.  :)
> 
> > Tools/Scripts/webkitpy/tool/bot/commitqueuetask_unittest.py:88
> > +        self._test_run_counter = -1
> 
> Why does this start at -1?

I've updated the comments to explain.

[Eric Seidel (no email)]

Committed r85463: <http://trac.webkit.org/changeset/85463>