Bug 230485

Summary: Regression (r282341?): [ iOS macOS wk2] imported/w3c/web-platform-tests/webrtc/RTCSctpTransport-events.html is a flaky failure
Product: WebKit Reporter: ayumi_kojima
Component: WebRTCAssignee: Nobody <webkit-unassigned>
Status: NEW ---    
Severity: Normal CC: koivisto, simon.fraser, webkit-bot-watchers-bugzilla, webkit-bug-importer, youennf
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   

Description ayumi_kojima 2021-09-20 10:25:13 PDT
imported/w3c/web-platform-tests/webrtc/RTCSctpTransport-events.html

Is flaky failing on iOS and macOS wk2.

History: https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fwebrtc%2FRTCSctpTransport-events.html

According to the history, it looks like the test first failed at r282059.

Result page: https://build.webkit.org/results/Apple-BigSur-Release-AppleSilicon-WK2-Tests/r282745%20(4008)/results.html

Diff:

--- /Volumes/Data/worker/bigsur-release-applesilicon-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/webrtc/RTCSctpTransport-events-expected.txt
+++ /Volumes/Data/worker/bigsur-release-applesilicon-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/webrtc/RTCSctpTransport-events-actual.txt
@@ -1,4 +1,6 @@
+
+Harness Error (TIMEOUT), message = null
 
 PASS SctpTransport objects are created at appropriate times
-PASS SctpTransport reaches connected and closed state
+TIMEOUT SctpTransport reaches connected and closed state Test timed out
Comment 1 Radar WebKit Bug Importer 2021-09-20 10:25:45 PDT
<rdar://problem/83312654>
Comment 2 ayumi_kojima 2021-09-20 10:30:49 PDT
Updated test expectations: https://trac.webkit.org/changeset/282757/webkit
Comment 3 ayumi_kojima 2021-09-20 15:25:00 PDT
I was able to reproduce the failure at TOT using run-webkit-tests --force --iterations 50 --exit-after-n-failures 1 imported/w3c/web-platform-tests/webrtc/RTCSctpTransport-events.html

With older revisions (r282000-r282103) though, the test failed but with a different diff:

--- /Volumes/Data/Builds/GoldenGate/test-282103/layout-test-results/imported/w3c/web-platform-tests/webrtc/RTCSctpTransport-events-expected.txt
+++ /Volumes/Data/Builds/GoldenGate/test-282103/layout-test-results/imported/w3c/web-platform-tests/webrtc/RTCSctpTransport-events-actual.txt
@@ -1,4 +1,6 @@
 
-PASS SctpTransport objects are created at appropriate times
-PASS SctpTransport reaches connected and closed state
+Harness Error (FAIL), message = Unhandled rejection: undefined is not an object (evaluating 'transport.state')
 
+FAIL SctpTransport objects are created at appropriate times assert_equals: expected (undefined) undefined but got (object) null
+FAIL SctpTransport reaches connected and closed state promise_test: Unhandled rejection with value: object "TypeError: undefined is not an object (evaluating 'transport.state')"
+

The test failed at the oldest revision available (r281695) with the same diff above.
Comment 4 ayumi_kojima 2021-09-20 16:38:40 PDT
It appears that it started at https://trac.webkit.org/changeset/282341/webkit

I was able to reproduce the failure at r282341 and newer revision and the test passed at r282340: 

Diff:

--- /Volumes/Data/worker/bigsur-release-applesilicon-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/webrtc/RTCSctpTransport-events-expected.txt
+++ /Volumes/Data/worker/bigsur-release-applesilicon-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/webrtc/RTCSctpTransport-events-actual.txt
@@ -1,4 +1,6 @@
+
+Harness Error (TIMEOUT), message = null
 
 PASS SctpTransport objects are created at appropriate times
-PASS SctpTransport reaches connected and closed state
+TIMEOUT SctpTransport reaches connected and closed state Test timed out

Weird thing is that r282059 failed with the same diff above on the testing machine, but I could not reproduce the same failure locally.
Comment 5 Simon Fraser (smfr) 2021-09-20 16:43:55 PDT
Seems unlikely that the cssom-view tests would cause an unrelated test to fail.