Bug 212755 - EWS layout test bots should not automatically retry with more than N failures or an early exit
Summary: EWS layout test bots should not automatically retry with more than N failures...
Status: RESOLVED INVALID
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: Other
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-06-04 09:00 PDT by David Kilzer (:ddkilzer)
Modified: 2020-06-04 10:39 PDT (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description David Kilzer (:ddkilzer) 2020-06-04 09:00:30 PDT
EWS layout test bots should not automatically retry with more than N failures or an early exit.

Looking at a recent failure picked at random, I see EWS layout test bots have a limit of 30 failing tests before ending early:
<https://ews-build.webkit.org/#/builders/24/builds/19073>

I think N=30 or an early exit seems reasonable to stop retrying to free up the bot resources.
Comment 1 Alexey Proskuryakov 2020-06-04 09:35:42 PDT
There were 30 failures on "run-layout-tests-without-patch", so the tree was in a bad state. It is correct for EWS to keep trying until the tree is fixed.

Stopping would be hostile to the patch author, and freeing up resources is not meaningful because all patches will fail until the tree is fixed.
Comment 2 David Kilzer (:ddkilzer) 2020-06-04 10:39:02 PDT
(In reply to Alexey Proskuryakov from comment #1)
> There were 30 failures on "run-layout-tests-without-patch", so the tree was
> in a bad state. It is correct for EWS to keep trying until the tree is fixed.
> 
> Stopping would be hostile to the patch author, and freeing up resources is
> not meaningful because all patches will fail until the tree is fixed.

This was more about preventing a denial-of-service on EWS after the bad commit was rolled out.  I guess this can be fixed in other ways.

This definitely seems related:
Bug 203698: [EWS] Layout testers can go in an infinite RETRY loop when there are 30+ failures on trunk