201607 2019-09-09 10:51:03 -0700 [EWS] API testers can go in an infinite RETRY loop when there is build failure in trunk 2019-09-10 13:35:03 -0700 1 1 1 Unclassified WebKit Tools / Tests Other Unspecified Unspecified RESOLVED FIXED InRadar P2 Normal --- 1 aakash_jain aakash_jain ap jbedard ryanhaddad webkit-bug-importer wenson_hsieh oldest_to_newest 1569189 0 aakash_jain 2019-09-09 10:51:03 -0700 If there is a build failure on top of tree, and the patch being tested have API test failures, then the API tester would fail to build and RETRY the build, and keep RETRYing, until the patch is obsoleted/closed. This happens in very specific scenario, when following conditions are met: 1) builder was able to build irrespective of ToT having build failure (because it does incremental build, not clean build by default). 2) API-tester found a test failure with the patch. 3) API-tester failed to build top-of-tree (since it does clean-build). In that case, tester will retry the build and since it's testing the exact same revision, it will fail again and keep RETRYing in this manner. We shouldn't RETRY in such cases. For e.g.: For Patch 378021 (in https://bugs.webkit.org/show_bug.cgi?id=201471), Builder was able to build in https://ews-build.webkit.org/#/builders/13/builds/8775, In that build the top-of-tree was at 1d2cc7a93bcdec77250d1c4ab9686e5cc32c170a (equivalent to r249500), which had a build failure at that time (Build was broken in r249498 and fixed in r249525 in https://bugs.webkit.org/show_bug.cgi?id=201468). Builder was able to build since it does incremental build, and the patch probably didn't modify any files in the part which was broken. The corresponding API tester noticed API test failures, and tried to build ToT and failed and keep retrying, e.g.: https://ews-build.webkit.org/#/builders/9/builds/7585 https://ews-build.webkit.org/#/builders/9/builds/7589 https://ews-build.webkit.org/#/builders/9/builds/7594 https://ews-build.webkit.org/#/builders/9/builds/7599 https://ews-build.webkit.org/#/builders/9/builds/7603 https://ews-build.webkit.org/#/builders/9/builds/8094 1569192 1 aakash_jain 2019-09-09 10:53:08 -0700 Note that the corresponding layout-testers worked in this scenario, e.g.: https://ews-build.webkit.org/#/builders/18/builds/3039 1569263 2 378400 aakash_jain 2019-09-09 13:12:18 -0700 Created attachment 378400 Patch 1569275 3 aakash_jain 2019-09-09 13:36:15 -0700 Committed r249663: <https://trac.webkit.org/changeset/249663> 1569276 4 webkit-bug-importer 2019-09-09 13:37:27 -0700 <rdar://problem/55195942> 1569640 5 aakash_jain 2019-09-10 13:35:03 -0700 Deployed in production. Working fine, e.g: https://ews-build.webkit.org/#/builders/9/builds/8174 378400 2019-09-09 13:12:18 -0700 2019-09-09 13:19:39 -0700 Patch bug-201607-20190909161217.patch text/plain 1881 aakash_jain SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDI0OTY1NykKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDE0IEBACisyMDE5LTA5LTA5ICBBYWthc2ggSmFpbiAgPGFha2FzaF9qYWluQGFwcGxlLmNv bT4KKworICAgICAgICBbRVdTXSBBUEkgdGVzdGVycyBjYW4gZ28gaW4gYW4gaW5maW5pdGUgUkVU UlkgbG9vcCB3aGVuIHRoZXJlIGlzIGJ1aWxkIGZhaWx1cmUgaW4gdHJ1bmsKKyAgICAgICAgaHR0 cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTIwMTYwNworCisgICAgICAgIFJl dmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisgICAgICAgICogQnVpbGRTbGF2ZVN1cHBvcnQv ZXdzLWJ1aWxkL3N0ZXBzLnB5OgorICAgICAgICAoQW5hbHl6ZUFQSVRlc3RzUmVzdWx0cy5hbmFs eXplUmVzdWx0cyk6IERvIG5vdCBSRVRSWSB3aGVuIGNsZWFuX3RyZWVfcmVzdWx0cyBhcmUgbWlz c2luZy4KKyAgICAgICAgKEFuYWx5emVBUElUZXN0c1Jlc3VsdHMuYW5hbHl6ZVJlc3VsdHMuZ2V0 QVBJVGVzdEZhaWx1cmVzKTogR3JhY2VmdWxseSBoYW5kbGUgTm9uZSByZXN1bHQuCisKIDIwMTkt MDktMDkgIERhbmllbCBCYXRlcyAgPGRhYmF0ZXNAYXBwbGUuY29tPgogCiAgICAgICAgIFJlbW92 ZSBhbGwgc2VsZWN0aW9uIHZpZXcgYW5pbWF0aW9ucyBiZWZvcmUgZHVtcGluZyByZXN1bHRzCklu ZGV4OiBUb29scy9CdWlsZFNsYXZlU3VwcG9ydC9ld3MtYnVpbGQvc3RlcHMucHkKPT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PQotLS0gVG9vbHMvQnVpbGRTbGF2ZVN1cHBvcnQvZXdzLWJ1aWxkL3N0ZXBzLnB5CShyZXZpc2lv biAyNDk2NTcpCisrKyBUb29scy9CdWlsZFNsYXZlU3VwcG9ydC9ld3MtYnVpbGQvc3RlcHMucHkJ KHdvcmtpbmcgY29weSkKQEAgLTEzNTcsMTIgKzEzNTcsMTQgQEAgY2xhc3MgQW5hbHl6ZUFQSVRl c3RzUmVzdWx0cyhidWlsZHN0ZXAuQgogICAgICAgICBzZWNvbmRfcnVuX3Jlc3VsdHMgPSBzZWxm LnJlc3VsdHMuZ2V0KFJlUnVuQVBJVGVzdHMubmFtZSkKICAgICAgICAgY2xlYW5fdHJlZV9yZXN1 bHRzID0gc2VsZi5yZXN1bHRzLmdldChSdW5BUElUZXN0c1dpdGhvdXRQYXRjaC5uYW1lKQogCi0g ICAgICAgIGlmIG5vdCAoZmlyc3RfcnVuX3Jlc3VsdHMgYW5kIHNlY29uZF9ydW5fcmVzdWx0cyBh bmQgY2xlYW5fdHJlZV9yZXN1bHRzKToKKyAgICAgICAgaWYgbm90IChmaXJzdF9ydW5fcmVzdWx0 cyBhbmQgc2Vjb25kX3J1bl9yZXN1bHRzKToKICAgICAgICAgICAgIHNlbGYuZmluaXNoZWQoUkVU UlkpCiAgICAgICAgICAgICBzZWxmLmJ1aWxkLmJ1aWxkRmluaXNoZWQoWydVbmFibGUgdG8gcGFy c2UgQVBJIHRlc3QgcmVzdWx0cyddLCBSRVRSWSkKICAgICAgICAgICAgIHJldHVybiAtMQogCiAg ICAgICAgIGRlZiBnZXRBUElUZXN0RmFpbHVyZXMocmVzdWx0KToKKyAgICAgICAgICAgIGlmIG5v dCByZXN1bHQ6CisgICAgICAgICAgICAgICAgcmV0dXJuIHNldChbXSkKICAgICAgICAgICAgICMg VE9ETzogQW5hbHl6ZSBUaW1lLW91dCwgQ3Jhc2ggYW5kIEZhaWx1cmUgaW5kZXBlbmRlbnRseQog ICAgICAgICAgICAgcmV0dXJuIHNldChbZmFpbHVyZS5nZXQoJ25hbWUnKSBmb3IgZmFpbHVyZSBp biByZXN1bHQuZ2V0KCdUaW1lZG91dCcsIFtdKV0gKwogICAgICAgICAgICAgICAgIFtmYWlsdXJl LmdldCgnbmFtZScpIGZvciBmYWlsdXJlIGluIHJlc3VsdC5nZXQoJ0NyYXNoZWQnLCBbXSldICsK