Bug 230653 – [ iOS & macOS wk2 Debug ] imported/w3c/web-platform-tests/webrtc/simulcast/setParameters-active.https.html is a flaky failure

• Component: WebRTC
• Status: RESOLVED CONFIGURATION CHANGED

imported/w3c/web-platform-tests/webrtc/simulcast/setParameters-active.https.html

Is a flaky failure on iOS (Release & Debug) and macOS Debug.

History: https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fwebrtc%2Fsimulcast%2FsetParameters-active.https.html

Diff: 

--- /Volumes/Data/worker/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/webrtc/simulcast/setParameters-active.https-expected.txt
+++ /Volumes/Data/worker/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/webrtc/simulcast/setParameters-active.https-actual.txt
@@ -1,3 +1,3 @@
 
-PASS Simulcast setParameters active=false stops sending frames
+FAIL Simulcast setParameters active=false stops sending frames assert_equals: expected 7 but got 8

Result page: https://build.webkit.org/results/Apple-iOS-14-Simulator-Release-WK2-Tests/r282882%20(3320)/results.html#

[Radar WebKit Bug Importer]

<rdar://problem/83422662>

Marked test expectations: https://trac.webkit.org/changeset/282893/webkit

Diff:

--- /Volumes/Data/worker/bigsur-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/webrtc/simulcast/setParameters-active.https-expected.txt
+++ /Volumes/Data/worker/bigsur-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/webrtc/simulcast/setParameters-active.https-actual.txt
@@ -1,3 +1,5 @@
 
-PASS Simulcast setParameters active=false stops sending frames
+Harness Error (TIMEOUT), message = null
 
+TIMEOUT Simulcast setParameters active=false stops sending frames Test timed out
+

I was able to reproduce the failure (Same diff as one comment up) on BigSur using run-webkit-tests --force --debug --iterations 100 --exit-after-n-failures 1 imported/w3c/web-platform-tests/webrtc/simulcast/setParameters-active.https.html -f

I was able to narrow the regression range down to r281500 - r281800. 

I was able to reproduce the failure with the diff (+FAIL Simulcast setParameters active=false..) for all reproductions except TOT. With r281500 and older revisions, the test timed out instead of failing.

r281517 - failed with same diff.
r281510 - failed with a different diff: 

-PASS Simulcast setParameters active=false stops sending frames
+Harness Error (TIMEOUT), message = null
 
+TIMEOUT Simulcast setParameters active=false stops sending frames Test timed out
+

r281511 failed with the diff (+FAIL Simulcast setParameters active=false..). 

From the reproduction, it looks like the test started failing with the current diff since this change: https://trac.webkit.org/changeset/281511/webkit, but not sure if this is the regression point because the test also failed with r281510 (but with a different diff)

[Ryan Haddad]

This test was imported with https://commits.webkit.org/241880@main, and scrolling back that far in test history suggests it has been flaky from the start.

[youenn fablet]

This should be fixed in https://bugs.webkit.org/show_bug.cgi?id=270790.
Once it lands, we should update the test expectations.

[youenn fablet]

Resolving as configuration changed.
Tests are now only flaky on Monterey x86_64, which is tracked in https://bugs.webkit.org/show_bug.cgi?id=271059