Bug 212755

Summary:	EWS layout test bots should not automatically retry with more than N failures or an early exit
Product:	WebKit	Reporter:	David Kilzer (:ddkilzer) <ddkilzer>
Component:	Tools / Tests	Assignee:	Nobody <webkit-unassigned>
Status:	RESOLVED INVALID
Severity:	Normal	CC:	aakash_jain, ap
Priority:	P2
Version:	Other
Hardware:	Unspecified
OS:	Unspecified
See Also:	https://bugs.webkit.org/show_bug.cgi?id=212756 https://bugs.webkit.org/show_bug.cgi?id=212746 https://bugs.webkit.org/show_bug.cgi?id=203698

Description David Kilzer (:ddkilzer) 2020-06-04 09:00:30 PDT

EWS layout test bots should not automatically retry with more than N failures or an early exit.

Looking at a recent failure picked at random, I see EWS layout test bots have a limit of 30 failing tests before ending early:
<https://ews-build.webkit.org/#/builders/24/builds/19073>

I think N=30 or an early exit seems reasonable to stop retrying to free up the bot resources.

Comment 1 Alexey Proskuryakov 2020-06-04 09:35:42 PDT

There were 30 failures on "run-layout-tests-without-patch", so the tree was in a bad state. It is correct for EWS to keep trying until the tree is fixed.

Stopping would be hostile to the patch author, and freeing up resources is not meaningful because all patches will fail until the tree is fixed.

Comment 2 David Kilzer (:ddkilzer) 2020-06-04 10:39:02 PDT

(In reply to Alexey Proskuryakov from comment #1)
> There were 30 failures on "run-layout-tests-without-patch", so the tree was
> in a bad state. It is correct for EWS to keep trying until the tree is fixed.
> 
> Stopping would be hostile to the patch author, and freeing up resources is
> not meaningful because all patches will fail until the tree is fixed.

This was more about preventing a denial-of-service on EWS after the bad commit was rolled out.  I guess this can be fixed in other ways.

This definitely seems related:
Bug 203698: [EWS] Layout testers can go in an infinite RETRY loop when there are 30+ failures on trunk