Bug 123450 - WebKit2 bots don't exit after 50 failures or timeouts
Summary: WebKit2 bots don't exit after 50 failures or timeouts
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Lucas Forschler
Depends on:
Reported: 2013-10-29 10:59 PDT by Alexey Proskuryakov
Modified: 2013-10-29 12:24 PDT (History)
5 users (show)

See Also:

set shouldAbortEarly to True (1.24 KB, patch)
2013-10-29 11:14 PDT, Lucas Forschler
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Alexey Proskuryakov 2013-10-29 10:59:29 PDT
So when there are lots of timeouts, they get really late.

Looks like we just don't pass exit-after-n-crashes-or-timeouts option, should be a config file fix I hope.
Comment 1 Lucas Forschler 2013-10-29 11:12:01 PDT
These are currently set to NOT abort early.  Simply changing this to True should set the timeout same as the Webkit1 tests.

class RunWebKit2Tests(RunWebKitTests):
    def start(self):
        self.setProperty("shouldAbortEarly", False)
        self.setCommand(self.command + ["--webkit-test-runner"])

        return RunWebKitTests.start(self)
Comment 2 Lucas Forschler 2013-10-29 11:14:45 PDT
Created attachment 215400 [details]
set shouldAbortEarly to True
Comment 3 Alexey Proskuryakov 2013-10-29 11:36:43 PDT
Comment on attachment 215400 [details]
set shouldAbortEarly to True

This was changed in bug 94506, apparently because WK2 used to have lots and lots of flaky tests, so letting these run to completion still gave you some information about non-flakily failing tests.

We are not there any more with respect to how horrible tests are, and allowing an unlimited number of crashes or timeouts grinds bots to a halt after a bad commit that crashes everywhere.

Alternatively to landing this change, we could roll out the one from bug 94506. What do you think?
Comment 4 Lucas Forschler 2013-10-29 11:39:36 PDT
I don't think we should roll out the previous one.  It is nice that it adds this property.  Let's keep the shouldAbortEarly property, and just set it to False with this change.
Comment 5 Lucas Forschler 2013-10-29 11:47:44 PDT
That should read, set it to True :)
Comment 6 WebKit Commit Bot 2013-10-29 12:18:36 PDT
Comment on attachment 215400 [details]
set shouldAbortEarly to True

Clearing flags on attachment: 215400

Committed r158207: <http://trac.webkit.org/changeset/158207>
Comment 7 WebKit Commit Bot 2013-10-29 12:18:39 PDT
All reviewed patches have been landed.  Closing bug.
Comment 8 Lucas Forschler 2013-10-29 12:24:29 PDT
restarted master.  This should be live.