WebKit2 bots don't exit after 50 failures or timeouts

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Alexey Proskuryakov]

So when there are lots of timeouts, they get really late.

Looks like we just don't pass exit-after-n-crashes-or-timeouts option, should be a config file fix I hope.

[Lucas Forschler]

These are currently set to NOT abort early.  Simply changing this to True should set the timeout same as the Webkit1 tests.

class RunWebKit2Tests(RunWebKitTests):
    def start(self):
        self.setProperty("shouldAbortEarly", False)
        self.setCommand(self.command + ["--webkit-test-runner"])

        return RunWebKitTests.start(self)

[Lucas Forschler]

Created attachment 215400 [details]
set shouldAbortEarly to True

[Alexey Proskuryakov]

Comment on attachment 215400 [details]
set shouldAbortEarly to True

This was changed in bug 94506, apparently because WK2 used to have lots and lots of flaky tests, so letting these run to completion still gave you some information about non-flakily failing tests.

We are not there any more with respect to how horrible tests are, and allowing an unlimited number of crashes or timeouts grinds bots to a halt after a bad commit that crashes everywhere.

Alternatively to landing this change, we could roll out the one from bug 94506. What do you think?

[Lucas Forschler]

I don't think we should roll out the previous one.  It is nice that it adds this property.  Let's keep the shouldAbortEarly property, and just set it to False with this change.

[Lucas Forschler]

That should read, set it to True :)

[WebKit Commit Bot]

Comment on attachment 215400 [details]
set shouldAbortEarly to True

Clearing flags on attachment: 215400

Committed r158207: <http://trac.webkit.org/changeset/158207>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Lucas Forschler]

restarted master.  This should be live.