306336 2026-01-27 08:10:24 -0800 [webkitpy][run-webkit-tests] Wrong exit code and report when a test is repeated (via --repeat-each=X) and there is a mix of unexpected and expected results 2026-01-28 14:33:03 -0800 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified REOPENED https://bugs.webkit.org/show_bug.cgi?id=306451 https://bugs.webkit.org/show_bug.cgi?id=306477 InRadar P2 Normal --- 306460 1 clopez clopez bugs-noreply commit-queue csaavedra webkit-bug-importer oldest_to_newest 2175430 0 clopez 2026-01-27 08:10:24 -0800 This has been observed here https://ews-build.webkit.org/#/builders/34/builds/107879 : - On the step `layout-tests-repeat-failures` the bot runs: Tools/Scripts/run-webkit-tests --no-build --no-show-results --no-new-test-results --clobber-old-results --release --wpe --results-directory layout-test-results --debug-rwt-logging --skip-failing-tests --fully-parallel --repeat-each=10 compositing/repaint/composited-document-element.html http/tests/blink/sendbeacon/beacon-cookie.html http/tests/security/contentSecurityPolicy/connect-src-eventsource-blocked.html http/tests/xmlhttprequest/logout.html imported/w3c/web-platform-tests/webrtc/RTCRtpSender-setParameters-keyFrame.html - The result is: 05:40:06.162 521977 Testing completed, Exit status: 1 => Results: 36/50 tests passed (72.0%) => Tests to be fixed (2): 1 crashes (50.0%) => Tests that will only be fixed if they crash (WONTFIX) (0): Unexpected flakiness: text-only failures (2) http/tests/blink/sendbeacon/beacon-cookie.html [ Pass Failure ] http/tests/xmlhttprequest/logout.html [ Pass Failure ] Unexpected flakiness: crashes (1) imported/w3c/web-platform-tests/webrtc/RTCRtpSender-setParameters-keyFrame.html [ Crash Pass ] The exit code (1) is wrong. It should be a zero exit code because all the tests were marked as flaky and not as regressions on the run. This causes an infrastructure error on the EWS logic because run-webkit-tests should not return error (non-zero) unless it also produced a list of failed tests and the EWS explicitly checks for this to guard against a patch that breaks the runner itself. 2175444 1 clopez 2026-01-27 08:47:37 -0800 Pull request: https://github.com/WebKit/WebKit/pull/57336 2175916 2 ews-feeder 2026-01-28 12:58:11 -0800 Committed 306367@main (96d2789262f7): <https://commits.webkit.org/306367@main> Reviewed commits have been landed. Closing PR #57336 and removing active labels. 2175917 3 webkit-bug-importer 2026-01-28 12:59:15 -0800 <rdar://problem/169119844> 2175930 4 clopez 2026-01-28 13:23:33 -0800 I have discovered that this patch will break the step "run-layout-tests-in-stress-mode" that the EWS uses to find new flakies added. In that step it is expected that it exits with error when there is a flaky test. See https://ews-build.webkit.org/#/builders/169/builds/2488
 So I will revert this, land the anti-gardening at bug 306451 and go back to the drawing board.. 2175939 5 commit-queue 2026-01-28 13:34:02 -0800 Re-opened since this is blocked by bug 306460 2175976 6 clopez 2026-01-28 14:26:41 -0800 In the previous patch i assumed "a repeated test should only be considered a regression if _all_ of the results it generated where unexpected. Otherwise, if there is only one PASS or only one expected failure it should be considered flaky instead." but maybe that is wrong and it should be considered a regression if any (instead of all) of the results it generated where unexpected. Anyway, this is a complex topic, I think I'm going to fix first the EWS logic instead to deal with the case run-webkit-tests exists with error and there is only a list of flakies (but not non-flaky errors) 2175984 7 clopez 2026-01-28 14:33:03 -0800 (In reply to Carlos Alberto Lopez Perez from comment #6) > Anyway, this is a complex topic, I think I'm going to fix first the EWS > logic instead to deal with the case run-webkit-tests exists with error and > there is only a list of flakies (but not non-flaky errors) See bug 306477