RESOLVED FIXED 248553
Skip run-layout-tests-without-change step when first and second layout test run have only flaky failures 
https://bugs.webkit.org/show_bug.cgi?id=248553
Summary Skip run-layout-tests-without-change step when first and second layout test r...
Aakash Jain
Reported 2022-11-30 13:44:28 PST
In https://bugs.webkit.org/show_bug.cgi?id=204769, we changed ews behavior to ignore the flaky test failures. However, entire logic is applied at the very end of the build (after we run layout tests twice, and then compile-webkit and then run layout-tests without change). However each of these steps might take significant time (especially important now that layout-test runtime has increased significantly in past few years). In order to make EWS more efficient, we can apply part of same logic just after re-run-layout-tests. The logic being, if both steps (layout-tests and re-run-layout-tests) completed and produced list of failures (without exiting early), and failures are completely different in both the runs (no single test consistently failed in both the runs), then we should simply ignore the failure and not blame the change being tested. Note that this is not at all a behavior/logic change, even currently we do the same thing in analyze-layout-tests-results step. Just that we apply the logic early and prevent extra steps of compiling-webkit and running layout-test without change. For example in: https://ews-build.webkit.org/#/builders/60/builds/44715 First run failures: fast/events/message-port-gc-after-removing-event-listener.html, fast/images/animated-heics-draw.html, http/wpt/service-workers/fetch-service-worker-preload.https.html Second run failures: fast/forms/file/entries-api/image-no-transcode-open-panel.html, fast/images/animated-heics-verify.html We still run compile-webkit-without-change and run-layout-tests-without-change Since there were no consistent failures in first and second run, `compile-webkit-without-change` marked the build as successful. But we can reach that same conclusion without running layout-test on clean tree.
Attachments
Add attachment proposed patch, testcase, etc.
Aakash Jain
Comment 1 2022-11-30 13:45:12 PST
Looking at the current code at https://github.com/WebKit/WebKit/blob/main/Tools/CISupport/ews-build/steps.py#L3735, if self. _results_failed_different_tests is true and tests_that_consistently_failed is empty, then the logic is always going to return self.report_pre_existing_failures() and mark build as successful. We can just do the same thing in such scenario without running layout-tests on clean tree.
Radar WebKit Bug Importer
Comment 2 2022-12-01 10:50:05 PST
<rdar://problem/102868074>
Aakash Jain
Comment 3 2022-12-01 10:55:24 PST
Pull request: https://github.com/WebKit/WebKit/pull/7034
EWS
Comment 4 2022-12-02 13:43:30 PST
Committed 257307@main (e167851130c2): <https://commits.webkit.org/257307@main> Reviewed commits have been landed. Closing PR #7034 and removing active labels.
Aakash Jain
Comment 5 2022-12-12 19:17:19 PST
This change did help in speeding up layout-test queues. Few examples where it helped (by avoiding 'compile-webkit-without-change' and 'run-layout-tests-without-change' steps): https://ews-build.webkit.org/#/builders/60/builds/45458 https://ews-build.webkit.org/#/builders/60/builds/45470 https://ews-build.webkit.org/#/builders/60/builds/45493 https://ews-build.webkit.org/#/builders/60/builds/45511 https://ews-build.webkit.org/#/builders/60/builds/45537 https://ews-build.webkit.org/#/builders/60/builds/45569 https://ews-build.webkit.org/#/builders/60/builds/45570 https://ews-build.webkit.org/#/builders/60/builds/45571 https://ews-build.webkit.org/#/builders/60/builds/45577 https://ews-build.webkit.org/#/builders/60/builds/45586 https://ews-build.webkit.org/#/builders/60/builds/45595 https://ews-build.webkit.org/#/builders/60/builds/45598 https://ews-build.webkit.org/#/builders/60/builds/45620 https://ews-build.webkit.org/#/builders/60/builds/45649 https://ews-build.webkit.org/#/builders/70/builds/22906 https://ews-build.webkit.org/#/builders/70/builds/22986 https://ews-build.webkit.org/#/builders/70/builds/23147 https://ews-build.webkit.org/#/builders/70/builds/23232 https://ews-build.webkit.org/#/builders/70/builds/23262 https://ews-build.webkit.org/#/builders/70/builds/23282 https://ews-build.webkit.org/#/builders/70/builds/23294 https://ews-build.webkit.org/#/builders/70/builds/23295 https://ews-build.webkit.org/#/builders/70/builds/23300 https://ews-build.webkit.org/#/builders/70/builds/23302 https://ews-build.webkit.org/#/builders/73/builds/21389 https://ews-build.webkit.org/#/builders/73/builds/21438 https://ews-build.webkit.org/#/builders/73/builds/21502 https://ews-build.webkit.org/#/builders/73/builds/21516 https://ews-build.webkit.org/#/builders/73/builds/21578 https://ews-build.webkit.org/#/builders/74/builds/5115 https://ews-build.webkit.org/#/builders/74/builds/5207 https://ews-build.webkit.org/#/builders/78/builds/6618 https://ews-build.webkit.org/#/builders/78/builds/6793 https://ews-build.webkit.org/#/builders/78/builds/6813 https://ews-build.webkit.org/#/builders/78/builds/6821 https://ews-build.webkit.org/#/builders/78/builds/6861
Aakash Jain
Comment 6 2022-12-12 19:24:17 PST
For future reference, url to find occurrences of all builds with specific string in build summary (e.g.: "in re-run"): https://ews-build.webkit.org/api/v2/builds?state_string__contains=in%20re-run
Note You need to log in before you can comment on or make changes to this bug.