imported/w3c/web-platform-tests/webrtc/simulcast/setParameters-active.https.html Is a flaky failure on iOS (Release & Debug) and macOS Debug. History: https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fwebrtc%2Fsimulcast%2FsetParameters-active.https.html Diff: --- /Volumes/Data/worker/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/webrtc/simulcast/setParameters-active.https-expected.txt +++ /Volumes/Data/worker/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/webrtc/simulcast/setParameters-active.https-actual.txt @@ -1,3 +1,3 @@ -PASS Simulcast setParameters active=false stops sending frames +FAIL Simulcast setParameters active=false stops sending frames assert_equals: expected 7 but got 8
Result page: https://build.webkit.org/results/Apple-iOS-14-Simulator-Release-WK2-Tests/r282882%20(3320)/results.html#
<rdar://problem/83422662>
Marked test expectations: https://trac.webkit.org/changeset/282893/webkit
Diff: --- /Volumes/Data/worker/bigsur-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/webrtc/simulcast/setParameters-active.https-expected.txt +++ /Volumes/Data/worker/bigsur-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/webrtc/simulcast/setParameters-active.https-actual.txt @@ -1,3 +1,5 @@ -PASS Simulcast setParameters active=false stops sending frames +Harness Error (TIMEOUT), message = null +TIMEOUT Simulcast setParameters active=false stops sending frames Test timed out +
I was able to reproduce the failure (Same diff as one comment up) on BigSur using run-webkit-tests --force --debug --iterations 100 --exit-after-n-failures 1 imported/w3c/web-platform-tests/webrtc/simulcast/setParameters-active.https.html -f
I was able to narrow the regression range down to r281500 - r281800. I was able to reproduce the failure with the diff (+FAIL Simulcast setParameters active=false..) for all reproductions except TOT. With r281500 and older revisions, the test timed out instead of failing.
r281517 - failed with same diff. r281510 - failed with a different diff: -PASS Simulcast setParameters active=false stops sending frames +Harness Error (TIMEOUT), message = null +TIMEOUT Simulcast setParameters active=false stops sending frames Test timed out +
r281511 failed with the diff (+FAIL Simulcast setParameters active=false..). From the reproduction, it looks like the test started failing with the current diff since this change: https://trac.webkit.org/changeset/281511/webkit, but not sure if this is the regression point because the test also failed with r281510 (but with a different diff)
This test was imported with https://commits.webkit.org/241880@main, and scrolling back that far in test history suggests it has been flaky from the start.
This should be fixed in https://bugs.webkit.org/show_bug.cgi?id=270790. Once it lands, we should update the test expectations.
Resolving as configuration changed. Tests are now only flaky on Monterey x86_64, which is tracked in https://bugs.webkit.org/show_bug.cgi?id=271059