Bug 218618 – [ews] Patch which break layout-test shouldn't cause infinite RETRY loop

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Aakash Jain]

Currently we retry layout-tests when there is an exception in running the tests (exit code 254). We do it to handle any unexpected infrastructure issues (like simulators not booting up properly). We keep retrying until the issue is fixed.

However, because of this infinite retry, patches which actually break layout-test are also retried, and go into an infinite retry loop (until the patch is obsolete, r-, bug is closed, or someone manually stops the build). e.g.: https://bugs.webkit.org/show_bug.cgi?id=218352#c2 caused continuos retries (https://ews-build.webkit.org/#/builders/30/builds/21779, https://ews-build.webkit.org/#/builders/30/builds/21778, https://ews-build.webkit.org/#/builders/30/builds/21777 etc.)

We shouldn't automatically retry immediately when layout-tests unexpectedly fail. We should re-run the tests on clean tree as well. If the clean tree passes, it indicates that the issue is with the patch, and the build should be marked as FAILURE (and red ews bubble). Otherwise (if clean-tree results are also failure) we should retry the build.

[Aakash Jain]

This is also a problem (infinite retry) when a patch breaks layout-tests in some other manner (without exit code 254). e.g.: patch in  https://bugs.webkit.org/show_bug.cgi?id=218306#c4 caused continuous retries (https://ews-build.webkit.org/#/builders/51/builds/672, https://ews-build.webkit.org/#/builders/51/builds/680)

[Aakash Jain]

Created attachment 413305 [details]
Patch

[EWS]

Committed r269467: <https://trac.webkit.org/changeset/269467>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 413305 [details].

[Radar WebKit Bug Importer]

<rdar://problem/71089620>