Bug 212755 – EWS layout test bots should not automatically retry with more than N failures or an early exit

• Status: RESOLVED
• Resolution: INVALID
• Component: Tools / Tests

[David Kilzer (:ddkilzer)]

EWS layout test bots should not automatically retry with more than N failures or an early exit.

Looking at a recent failure picked at random, I see EWS layout test bots have a limit of 30 failing tests before ending early:
<https://ews-build.webkit.org/#/builders/24/builds/19073>

I think N=30 or an early exit seems reasonable to stop retrying to free up the bot resources.

[Alexey Proskuryakov]

There were 30 failures on "run-layout-tests-without-patch", so the tree was in a bad state. It is correct for EWS to keep trying until the tree is fixed.

Stopping would be hostile to the patch author, and freeing up resources is not meaningful because all patches will fail until the tree is fixed.

[David Kilzer (:ddkilzer)]

(In reply to Alexey Proskuryakov from comment #1)
> There were 30 failures on "run-layout-tests-without-patch", so the tree was
> in a bad state. It is correct for EWS to keep trying until the tree is fixed.
> 
> Stopping would be hostile to the patch author, and freeing up resources is
> not meaningful because all patches will fail until the tree is fixed.

This was more about preventing a denial-of-service on EWS after the bad commit was rolled out.  I guess this can be fixed in other ways.

This definitely seems related:
Bug 203698: [EWS] Layout testers can go in an infinite RETRY loop when there are 30+ failures on trunk