Bug 203756 – [ews] Perform validation of patch before retrying API and layout tests

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Aakash Jain]

Some tests (like API and layout-tests) takes a while to run (15-50 mins depending on test-suite). In case of failure, we retry the tests, and if failures are noticed in retry as well, then we un-apply the patch, build the ToT and run the tests again. All these steps might take a while to complete. 

Meanwhile the patch might get obsolete or r-. EWS should not continue to process such patches, as it would be wasteful. We should check the patch status before running any time-consuming step (like running tests, or building without patch). 

This is similar to old EWS behavior in: https://bugs.webkit.org/show_bug.cgi?id=160739


Note that if the bug is closed, we should still continue to process the patch, since re-trying tests indicates that the patch might have broken something, and having the EWS results would be valuable even after patch is landed.

Also note that this doesn't not terminate the on-going steps, which Bug 201924 aims to achieve.

[Aakash Jain]

Created attachment 382627 [details]
Patch

[Aakash Jain]

Sample runs:
API-tests: https://ews-build.webkit-uat.org/#/builders/4/builds/78

Layout-tests: https://ews-build.webkit-uat.org/#/builders/4/builds/78 

(Note that in above builds, for testing, I skipped the obsolete check in first few validate-patch steps).

[WebKit Commit Bot]

Comment on attachment 382627 [details]
Patch

Clearing flags on attachment: 382627

Committed r252037: <https://trac.webkit.org/changeset/252037>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/56891516>

[Aakash Jain]

This change seems to be quite helpful in improving EWS efficiency, by skipping the patches when they become obsolete.

Few examples:
API tests:
https://ews-build.webkit.org/#/builders/9/builds/11447
https://ews-build.webkit.org/#/builders/9/builds/11430
https://ews-build.webkit.org/#/builders/9/builds/11410
https://ews-build.webkit.org/#/builders/9/builds/11397
https://ews-build.webkit.org/#/builders/9/builds/11396
https://ews-build.webkit.org/#/builders/9/builds/11392
https://ews-build.webkit.org/#/builders/9/builds/11390

Layout tests:
https://ews-build.webkit.org/#/builders/17/builds/5480
https://ews-build.webkit.org/#/builders/17/builds/5461
https://ews-build.webkit.org/#/builders/17/builds/5423
https://ews-build.webkit.org/#/builders/17/builds/5422
https://ews-build.webkit.org/#/builders/17/builds/5421