Bug 207064 – [ews] Do not run tests without patch when there are no consistent failures in two layout-test runs

• Component: Tools / Tests
• Status: RESOLVED INVALID

[Aakash Jain]

Currently in EWS when layout-tests run notices any test failure, we re-run the tests. If the re-run also notices any test failure, we run the tests on clean tree (after unapplying the patch and building webkit again).

Due to flaky tests, it is possible that the first and second runs have different failures, and no common test failure. In that case, our analysis logic will mark the build as passed, irrespective of clean tree results, especially after the change in https://trac.webkit.org/changeset/253049/webkit

It's inefficient and wasteful to run tests on clean tree in such case. We shouldn't skip clean-tree run in such cases. Also we should update the build summary with the list of flaky tests, which would make it easier for bot-watchers to notice flaky tests.

Few examples where first and second layout-test runs had different failures:

https://ews-build.webkit.org/#/builders/24/builds/9565
https://ews-build.webkit.org/#/builders/24/builds/9446
https://ews-build.webkit.org/#/builders/24/builds/9394
https://ews-build.webkit.org/#/builders/24/builds/9393
https://ews-build.webkit.org/#/builders/24/builds/9352

https://ews-build.webkit.org/#/builders/32/builds/1744
https://ews-build.webkit.org/#/builders/32/builds/1708
https://ews-build.webkit.org/#/builders/32/builds/1703


https://ews-build.webkit.org/#/builders/30/builds/1783
https://ews-build.webkit.org/#/builders/30/builds/1696
https://ews-build.webkit.org/#/builders/30/builds/1648

https://ews-build.webkit.org/#/builders/31/builds/1905
https://ews-build.webkit.org/#/builders/31/builds/1737
https://ews-build.webkit.org/#/builders/31/builds/1519

[Aakash Jain]

Created attachment 389393 [details]
Patch

[Aakash Jain]

Created attachment 389409 [details]
Patch

[Aakash Jain]

Sample run: https://ews-build.webkit-uat.org/#/builders/16/builds/45

[Alexey Proskuryakov]

> In that case, our analysis logic will mark the build as passed, irrespective of clean tree results

This sounds terrible.

I think that we do want to re-run in this situation though. E.g. if try and retry both have 5 different failures, and retry without patch has 0 failures, that sounds like a failure for the patch.

[Aakash Jain]

(In reply to Alexey Proskuryakov from comment #4)
> I think that we do want to re-run in this situation though. E.g. if try and retry both have 5 different failures, and retry without patch has 0 failures, that sounds like a failure for the patch.
Agree, will do that instead.