Bug 230653 - Regression (r281511?): [ iOS & macOS wk2 Debug ] imported/w3c/web-platform-tests/webrtc/simulcast/setParameters-active.https.html is a flaky failure
Summary: Regression (r281511?): [ iOS & macOS wk2 Debug ] imported/w3c/web-platform-te...
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebRTC (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2021-09-22 16:31 PDT by ayumi_kojima
Modified: 2021-10-13 15:07 PDT (History)
5 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description ayumi_kojima 2021-09-22 16:31:25 PDT
imported/w3c/web-platform-tests/webrtc/simulcast/setParameters-active.https.html

Is a flaky failure on iOS (Release & Debug) and macOS Debug.

History: https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fwebrtc%2Fsimulcast%2FsetParameters-active.https.html

Diff: 

--- /Volumes/Data/worker/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/webrtc/simulcast/setParameters-active.https-expected.txt
+++ /Volumes/Data/worker/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/webrtc/simulcast/setParameters-active.https-actual.txt
@@ -1,3 +1,3 @@
 
-PASS Simulcast setParameters active=false stops sending frames
+FAIL Simulcast setParameters active=false stops sending frames assert_equals: expected 7 but got 8
Comment 2 Radar WebKit Bug Importer 2021-09-22 16:32:19 PDT
<rdar://problem/83422662>
Comment 3 ayumi_kojima 2021-09-22 16:46:14 PDT
Marked test expectations: https://trac.webkit.org/changeset/282893/webkit
Comment 4 ayumi_kojima 2021-09-29 14:20:14 PDT
Diff:

--- /Volumes/Data/worker/bigsur-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/webrtc/simulcast/setParameters-active.https-expected.txt
+++ /Volumes/Data/worker/bigsur-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/webrtc/simulcast/setParameters-active.https-actual.txt
@@ -1,3 +1,5 @@
 
-PASS Simulcast setParameters active=false stops sending frames
+Harness Error (TIMEOUT), message = null
 
+TIMEOUT Simulcast setParameters active=false stops sending frames Test timed out
+
Comment 5 ayumi_kojima 2021-09-29 15:39:59 PDT
I was able to reproduce the failure (Same diff as one comment up) on BigSur using run-webkit-tests --force --debug --iterations 100 --exit-after-n-failures 1 imported/w3c/web-platform-tests/webrtc/simulcast/setParameters-active.https.html -f
Comment 6 ayumi_kojima 2021-09-29 17:01:36 PDT
I was able to narrow the regression range down to r281500 - r281800. 

I was able to reproduce the failure with the diff (+FAIL Simulcast setParameters active=false..) for all reproductions except TOT. With r281500 and older revisions, the test timed out instead of failing.
Comment 7 ayumi_kojima 2021-10-07 17:26:47 PDT
r281517 - failed with same diff.
r281510 - failed with a different diff: 

-PASS Simulcast setParameters active=false stops sending frames
+Harness Error (TIMEOUT), message = null
 
+TIMEOUT Simulcast setParameters active=false stops sending frames Test timed out
+
Comment 8 ayumi_kojima 2021-10-13 15:07:10 PDT
r281511 failed with the diff (+FAIL Simulcast setParameters active=false..). 

From the reproduction, it looks like the test started failing with the current diff since this change: https://trac.webkit.org/changeset/281511/webkit, but not sure if this is the regression point because the test also failed with r281510 (but with a different diff)