Bug 201607 - [EWS] API testers can go in an infinite RETRY loop when there is build failure in trunk
Summary: [EWS] API testers can go in an infinite RETRY loop when there is build failur...
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: Other
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Aakash Jain
Keywords: InRadar
Depends on:
Reported: 2019-09-09 10:51 PDT by Aakash Jain
Modified: 2019-09-10 13:35 PDT (History)
5 users (show)

See Also:

Patch (1.84 KB, patch)
2019-09-09 13:12 PDT, Aakash Jain
jbedard: review+
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Aakash Jain 2019-09-09 10:51:03 PDT
If there is a build failure on top of tree, and the patch being tested have API test failures, then the API tester would fail to build and RETRY the build, and keep RETRYing, until the patch is obsoleted/closed.

This happens in very specific scenario, when following conditions are met:
1) builder was able to build irrespective of ToT having build failure (because it does incremental build, not clean build by default).
2) API-tester found a test failure with the patch.
3) API-tester failed to build top-of-tree (since it does clean-build).

In that case, tester will retry the build and since it's testing the exact same revision, it will fail again and keep RETRYing in this manner. We shouldn't RETRY in such cases.

For e.g.: 
For Patch 378021 (in https://bugs.webkit.org/show_bug.cgi?id=201471), Builder was able to build in  https://ews-build.webkit.org/#/builders/13/builds/8775, In that build the top-of-tree was at 1d2cc7a93bcdec77250d1c4ab9686e5cc32c170a (equivalent to r249500), which had a build failure at that time (Build was broken in r249498 and fixed in r249525 in https://bugs.webkit.org/show_bug.cgi?id=201468). Builder was able to build since it does incremental build, and the patch probably didn't modify any files in the part which was broken.

The corresponding API tester noticed API test failures, and tried to build ToT and failed and keep retrying, 
Comment 1 Aakash Jain 2019-09-09 10:53:08 PDT
Note that the corresponding layout-testers worked in this scenario, e.g.: https://ews-build.webkit.org/#/builders/18/builds/3039
Comment 2 Aakash Jain 2019-09-09 13:12:18 PDT
Created attachment 378400 [details]
Comment 3 Aakash Jain 2019-09-09 13:36:15 PDT
Committed r249663: <https://trac.webkit.org/changeset/249663>
Comment 4 Radar WebKit Bug Importer 2019-09-09 13:37:27 PDT
Comment 5 Aakash Jain 2019-09-10 13:35:03 PDT
Deployed in production.

Working fine, e.g: https://ews-build.webkit.org/#/builders/9/builds/8174