Bug 218618

Summary: [ews] Patch which break layout-test shouldn't cause infinite RETRY loop
Product: WebKit Reporter: Aakash Jain <aakash_jain>
Component: Tools / TestsAssignee: Aakash Jain <aakash_jain>
Severity: Normal CC: aakash_jain, angelos, ap, dean_johnson, dewei_zhu, jbedard, ryanhaddad, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: Other   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=215944
Description Flags
Patch none

Description Aakash Jain 2020-11-05 08:34:33 PST
Currently we retry layout-tests when there is an exception in running the tests (exit code 254). We do it to handle any unexpected infrastructure issues (like simulators not booting up properly). We keep retrying until the issue is fixed.

However, because of this infinite retry, patches which actually break layout-test are also retried, and go into an infinite retry loop (until the patch is obsolete, r-, bug is closed, or someone manually stops the build). e.g.: https://bugs.webkit.org/show_bug.cgi?id=218352#c2 caused continuos retries (https://ews-build.webkit.org/#/builders/30/builds/21779, https://ews-build.webkit.org/#/builders/30/builds/21778, https://ews-build.webkit.org/#/builders/30/builds/21777 etc.)

We shouldn't automatically retry immediately when layout-tests unexpectedly fail. We should re-run the tests on clean tree as well. If the clean tree passes, it indicates that the issue is with the patch, and the build should be marked as FAILURE (and red ews bubble). Otherwise (if clean-tree results are also failure) we should retry the build.
Comment 1 Aakash Jain 2020-11-05 08:46:43 PST
This is also a problem (infinite retry) when a patch breaks layout-tests in some other manner (without exit code 254). e.g.: patch in  https://bugs.webkit.org/show_bug.cgi?id=218306#c4 caused continuous retries (https://ews-build.webkit.org/#/builders/51/builds/672, https://ews-build.webkit.org/#/builders/51/builds/680)
Comment 2 Aakash Jain 2020-11-05 08:46:52 PST
Created attachment 413305 [details]
Comment 3 EWS 2020-11-05 12:40:37 PST
Committed r269467: <https://trac.webkit.org/changeset/269467>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 413305 [details].
Comment 4 Radar WebKit Bug Importer 2020-11-05 12:41:19 PST