Bug 248553 – Skip run-layout-tests-without-change step when first and second layout test run have only flaky failures

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Aakash Jain]

In https://bugs.webkit.org/show_bug.cgi?id=204769, we changed ews behavior to ignore the flaky test failures. However, entire logic is applied at the very end of the build (after we run layout tests twice, and then compile-webkit and then run layout-tests without change).

However each of these steps might take significant time (especially important now that layout-test runtime has increased significantly in past few years). In order to make EWS more efficient, we can apply part of same logic just after re-run-layout-tests. The logic being, if both steps (layout-tests and re-run-layout-tests) completed and produced list of failures (without exiting early), and failures are completely different in both the runs (no single test consistently failed in both the runs), then we should simply ignore the failure and not blame the change being tested. 

Note that this is not at all a behavior/logic change, even currently we do the same thing in analyze-layout-tests-results step. Just that we apply the logic early and prevent extra steps of compiling-webkit and running layout-test without change.


For example in:
https://ews-build.webkit.org/#/builders/60/builds/44715

First run failures: fast/events/message-port-gc-after-removing-event-listener.html, fast/images/animated-heics-draw.html, http/wpt/service-workers/fetch-service-worker-preload.https.html
Second run failures: fast/forms/file/entries-api/image-no-transcode-open-panel.html, fast/images/animated-heics-verify.html

We still run compile-webkit-without-change and run-layout-tests-without-change

Since there were no consistent failures in first and second run, `compile-webkit-without-change` marked the build as successful. But we can reach that same conclusion without running layout-test on clean tree.

[Aakash Jain]

Looking at the current code at https://github.com/WebKit/WebKit/blob/main/Tools/CISupport/ews-build/steps.py#L3735, if self. _results_failed_different_tests is true and tests_that_consistently_failed is empty, then the logic is always going to return self.report_pre_existing_failures() and mark build as successful. We can just do the same thing in such scenario without running layout-tests on clean tree.

[Radar WebKit Bug Importer]

<rdar://problem/102868074>

[Aakash Jain]

Pull request: https://github.com/WebKit/WebKit/pull/7034

[EWS]

Committed 257307@main (e167851130c2): <https://commits.webkit.org/257307@main>

Reviewed commits have been landed. Closing PR #7034 and removing active labels.

[Aakash Jain]

This change did help in speeding up layout-test queues. Few examples where it helped (by avoiding 'compile-webkit-without-change' and 'run-layout-tests-without-change' steps):

https://ews-build.webkit.org/#/builders/60/builds/45458
https://ews-build.webkit.org/#/builders/60/builds/45470
https://ews-build.webkit.org/#/builders/60/builds/45493
https://ews-build.webkit.org/#/builders/60/builds/45511
https://ews-build.webkit.org/#/builders/60/builds/45537
https://ews-build.webkit.org/#/builders/60/builds/45569
https://ews-build.webkit.org/#/builders/60/builds/45570
https://ews-build.webkit.org/#/builders/60/builds/45571
https://ews-build.webkit.org/#/builders/60/builds/45577
https://ews-build.webkit.org/#/builders/60/builds/45586
https://ews-build.webkit.org/#/builders/60/builds/45595
https://ews-build.webkit.org/#/builders/60/builds/45598
https://ews-build.webkit.org/#/builders/60/builds/45620
https://ews-build.webkit.org/#/builders/60/builds/45649
https://ews-build.webkit.org/#/builders/70/builds/22906
https://ews-build.webkit.org/#/builders/70/builds/22986
https://ews-build.webkit.org/#/builders/70/builds/23147
https://ews-build.webkit.org/#/builders/70/builds/23232
https://ews-build.webkit.org/#/builders/70/builds/23262
https://ews-build.webkit.org/#/builders/70/builds/23282
https://ews-build.webkit.org/#/builders/70/builds/23294
https://ews-build.webkit.org/#/builders/70/builds/23295
https://ews-build.webkit.org/#/builders/70/builds/23300
https://ews-build.webkit.org/#/builders/70/builds/23302
https://ews-build.webkit.org/#/builders/73/builds/21389
https://ews-build.webkit.org/#/builders/73/builds/21438
https://ews-build.webkit.org/#/builders/73/builds/21502
https://ews-build.webkit.org/#/builders/73/builds/21516
https://ews-build.webkit.org/#/builders/73/builds/21578
https://ews-build.webkit.org/#/builders/74/builds/5115
https://ews-build.webkit.org/#/builders/74/builds/5207
https://ews-build.webkit.org/#/builders/78/builds/6618
https://ews-build.webkit.org/#/builders/78/builds/6793
https://ews-build.webkit.org/#/builders/78/builds/6813
https://ews-build.webkit.org/#/builders/78/builds/6821
https://ews-build.webkit.org/#/builders/78/builds/6861

[Aakash Jain]

For future reference, url to find occurrences of all builds with specific string in build summary (e.g.: "in re-run"): https://ews-build.webkit.org/api/v2/builds?state_string__contains=in%20re-run