[EWS] API testers can go in an infinite RETRY loop when there is build failure in trunk

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Aakash Jain]

If there is a build failure on top of tree, and the patch being tested have API test failures, then the API tester would fail to build and RETRY the build, and keep RETRYing, until the patch is obsoleted/closed.

This happens in very specific scenario, when following conditions are met:
1) builder was able to build irrespective of ToT having build failure (because it does incremental build, not clean build by default).
2) API-tester found a test failure with the patch.
3) API-tester failed to build top-of-tree (since it does clean-build).

In that case, tester will retry the build and since it's testing the exact same revision, it will fail again and keep RETRYing in this manner. We shouldn't RETRY in such cases.

For e.g.: 
For Patch 378021 (in https://bugs.webkit.org/show_bug.cgi?id=201471), Builder was able to build in  https://ews-build.webkit.org/#/builders/13/builds/8775, In that build the top-of-tree was at 1d2cc7a93bcdec77250d1c4ab9686e5cc32c170a (equivalent to r249500), which had a build failure at that time (Build was broken in r249498 and fixed in r249525 in https://bugs.webkit.org/show_bug.cgi?id=201468). Builder was able to build since it does incremental build, and the patch probably didn't modify any files in the part which was broken.

The corresponding API tester noticed API test failures, and tried to build ToT and failed and keep retrying, 
e.g.: 
https://ews-build.webkit.org/#/builders/9/builds/7585
https://ews-build.webkit.org/#/builders/9/builds/7589
https://ews-build.webkit.org/#/builders/9/builds/7594
https://ews-build.webkit.org/#/builders/9/builds/7599
https://ews-build.webkit.org/#/builders/9/builds/7603
https://ews-build.webkit.org/#/builders/9/builds/8094

[Aakash Jain]

Note that the corresponding layout-testers worked in this scenario, e.g.: https://ews-build.webkit.org/#/builders/18/builds/3039

[Aakash Jain]

Created attachment 378400 [details]
Patch

[Aakash Jain]

Committed r249663: <https://trac.webkit.org/changeset/249663>

[Radar WebKit Bug Importer]

<rdar://problem/55195942>

[Aakash Jain]

Deployed in production.

Working fine, e.g: https://ews-build.webkit.org/#/builders/9/builds/8174