Bug 171605 – Multiple LayoutTest/imported/w3c/web-platform-tests/webrtc/ are flaky failures

• Component: New Bugs
• Status: NEW

[Matt Lewis]

The Test imported/w3c/web-platform-tests/webrtc/no-media-call.html is a flaky failure on Mac-wk2

The test consistently fails at the same time as:
imported/w3c/web-platform-tests/webrtc/promises-call.html
imported/w3c/web-platform-tests/webrtc/datachannel-emptystring.html

History:
https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2Fwebrtc%2Fno-media-call.html

Build log:
https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/r216117%20(1135)/results.html

[Ryan Haddad]

*** Bug 170820 has been marked as a duplicate of this bug. ***

[Matt Lewis]

Created attachment 308922 [details]
Mark test as flaky.

[Ryan Haddad]

Comment on attachment 308922 [details]
Mark test as flaky.

Clearing flags on attachment: 308922

Committed r216125: <http://trac.webkit.org/changeset/216125>

[Radar WebKit Bug Importer]

<rdar://problem/31977318>

[Matt Lewis]

As of 5/22 and revision https://trac.webkit.org/changeset/217225/webkit

the tests above are now failing on iOS Simulator release and Sierra release consistently.

Diff for imported/w3c/web-platform-tests/webrtc/promises-call.html:

--- /Volumes/Data/slave/ios-simulator-10-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/webrtc/promises-call-expected.txt
+++ /Volumes/Data/slave/ios-simulator-10-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/webrtc/promises-call-actual.txt
@@ -1,6 +1,8 @@
 
-PASS Can set up a basic WebRTC call with only data using promises. 
+Harness Error (TIMEOUT), message = null
+
+TIMEOUT Can set up a basic WebRTC call with only data using promises. Test timed out
 iceConnectionState info
 
-First: completed
-Second: connected
+First: checking
+Second: new

Diff for imported/w3c/web-platform-tests/webrtc/datachannel-emptystring.html:

--- /Volumes/Data/slave/ios-simulator-10-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/webrtc/datachannel-emptystring-expected.txt
+++ /Volumes/Data/slave/ios-simulator-10-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/webrtc/datachannel-emptystring-actual.txt
@@ -1,8 +1,7 @@
 
-FAIL Can send empty strings across a WebRTC data channel. assert_equals: Empty message not received expected "" but got "done"
+Harness Error (TIMEOUT), message = null
+
+TIMEOUT Can send empty strings across a WebRTC data channel. Test timed out
 Messages exchanged
 
-Sent: [empty string]
-Sent: "done"
-Received: "done"

[Ryan Haddad]

(In reply to Matt Lewis from comment #5)
> As of 5/22 and revision https://trac.webkit.org/changeset/217225/webkit
> 
> the tests above are now failing on iOS Simulator release and Sierra release
> consistently.
> 
> Diff for imported/w3c/web-platform-tests/webrtc/promises-call.html:
> 
> ---
> /Volumes/Data/slave/ios-simulator-10-release-tests-wk2/build/layout-test-
> results/imported/w3c/web-platform-tests/webrtc/promises-call-expected.txt
> +++
> /Volumes/Data/slave/ios-simulator-10-release-tests-wk2/build/layout-test-
> results/imported/w3c/web-platform-tests/webrtc/promises-call-actual.txt
> @@ -1,6 +1,8 @@
>  
> -PASS Can set up a basic WebRTC call with only data using promises. 
> +Harness Error (TIMEOUT), message = null
> +
> +TIMEOUT Can set up a basic WebRTC call with only data using promises. Test
> timed out
>  iceConnectionState info
>  
> -First: completed
> -Second: connected
> +First: checking
> +Second: new
> 
> Diff for imported/w3c/web-platform-tests/webrtc/datachannel-emptystring.html:
> 
> ---
> /Volumes/Data/slave/ios-simulator-10-release-tests-wk2/build/layout-test-
> results/imported/w3c/web-platform-tests/webrtc/datachannel-emptystring-
> expected.txt
> +++
> /Volumes/Data/slave/ios-simulator-10-release-tests-wk2/build/layout-test-
> results/imported/w3c/web-platform-tests/webrtc/datachannel-emptystring-
> actual.txt
> @@ -1,8 +1,7 @@
>  
> -FAIL Can send empty strings across a WebRTC data channel. assert_equals:
> Empty message not received expected "" but got "done"
> +Harness Error (TIMEOUT), message = null
> +
> +TIMEOUT Can send empty strings across a WebRTC data channel. Test timed out
>  Messages exchanged
>  
> -Sent: [empty string]
> -Sent: "done"
> -Received: "done"

Added iOS TestExpectations in https://trac.webkit.org/changeset/217257/webkit.