Bug 218618 - [ews] Patch which break layout-test shouldn't cause infinite RETRY loop
Summary: [ews] Patch which break layout-test shouldn't cause infinite RETRY loop
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: Other
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Aakash Jain
Keywords: InRadar
Depends on:
Reported: 2020-11-05 08:34 PST by Aakash Jain
Modified: 2024-02-01 05:48 PST (History)
8 users (show)

See Also:

Patch (5.62 KB, patch)
2020-11-05 08:46 PST, Aakash Jain
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Aakash Jain 2020-11-05 08:34:33 PST
Currently we retry layout-tests when there is an exception in running the tests (exit code 254). We do it to handle any unexpected infrastructure issues (like simulators not booting up properly). We keep retrying until the issue is fixed.

However, because of this infinite retry, patches which actually break layout-test are also retried, and go into an infinite retry loop (until the patch is obsolete, r-, bug is closed, or someone manually stops the build). e.g.: https://bugs.webkit.org/show_bug.cgi?id=218352#c2 caused continuos retries (https://ews-build.webkit.org/#/builders/30/builds/21779, https://ews-build.webkit.org/#/builders/30/builds/21778, https://ews-build.webkit.org/#/builders/30/builds/21777 etc.)

We shouldn't automatically retry immediately when layout-tests unexpectedly fail. We should re-run the tests on clean tree as well. If the clean tree passes, it indicates that the issue is with the patch, and the build should be marked as FAILURE (and red ews bubble). Otherwise (if clean-tree results are also failure) we should retry the build.
Comment 1 Aakash Jain 2020-11-05 08:46:43 PST
This is also a problem (infinite retry) when a patch breaks layout-tests in some other manner (without exit code 254). e.g.: patch in  https://bugs.webkit.org/show_bug.cgi?id=218306#c4 caused continuous retries (https://ews-build.webkit.org/#/builders/51/builds/672, https://ews-build.webkit.org/#/builders/51/builds/680)
Comment 2 Aakash Jain 2020-11-05 08:46:52 PST
Created attachment 413305 [details]
Comment 3 EWS 2020-11-05 12:40:37 PST
Committed r269467: <https://trac.webkit.org/changeset/269467>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 413305 [details].
Comment 4 Radar WebKit Bug Importer 2020-11-05 12:41:19 PST