Bug 173041 – Layout Test http/tests/websocket/tests/hybi/handshake-ok-with-legacy-sec-websocket-response-headers.html is a flaky failure

• Component: New Bugs
• Status: NEW

[Matt Lewis]

http/tests/websocket/tests/hybi/handshake-ok-with-legacy-sec-websocket-response-headers.html is a flaky failure.
According to the flakiness dashboard the test has failed intermittently on iOS Simulator WK2 Release and El Capitan WK2 Release:
https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=http%2Ftests%2Fwebsocket%2Ftests%2Fhybi%2Fhandshake-ok-with-legacy-sec-websocket-response-headers.html

https://build.webkit.org/builders/Apple%20iOS%2010%20Simulator%20Release%20WK2%20(Tests)/builds/2018
https://build.webkit.org/results/Apple%20iOS%2010%20Simulator%20Release%20WK2%20(Tests)/r217848%20(2018)/results.html


diff:
--- /Volumes/Data/slave/ios-simulator-10-release-tests-wk2/build/layout-test-results/http/tests/websocket/tests/hybi/handshake-ok-with-legacy-sec-websocket-response-headers-expected.txt
+++ /Volumes/Data/slave/ios-simulator-10-release-tests-wk2/build/layout-test-results/http/tests/websocket/tests/hybi/handshake-ok-with-legacy-sec-websocket-response-headers-actual.txt
@@ -1,3 +1,4 @@
+CONSOLE MESSAGE: WebSocket network error: The operation couldn’t be completed. Connection reset by peer
 Test whether WebSocket handshake is OK with legacy response headers 'Sec-WebSocket-Location' and 'Sec-WebSocket-Origin'.
 
 On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".

[Ryan Haddad]

This is still happening:

--- /Volumes/Data/slave/highsierra-release-tests-wk2/build/layout-test-results/http/tests/websocket/tests/hybi/handshake-ok-with-legacy-sec-websocket-response-headers-expected.txt
+++ /Volumes/Data/slave/highsierra-release-tests-wk2/build/layout-test-results/http/tests/websocket/tests/hybi/handshake-ok-with-legacy-sec-websocket-response-headers-actual.txt
@@ -1,3 +1,5 @@
+CONSOLE MESSAGE: WebSocket network error: The operation couldn’t be completed. Connection reset by peer
+CONSOLE MESSAGE: WebSocket network error: The operation couldn’t be completed. Connection reset by peer
 Test whether WebSocket handshake is OK with legacy response headers 'Sec-WebSocket-Location' and 'Sec-WebSocket-Origin'.
 
 On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".

https://build.webkit.org/results/Apple%20High%20Sierra%20Release%20WK2%20(Tests)/r237073%20(7241)/results.html

[Ryan Haddad]

Marked test as flaky on iOS in https://trac.webkit.org/changeset/237443/webkit

[Shawn Roberts]

Test is now showing on flakiness dashboard with Mojave and High Sierra WK2 as a flaky text diff failure. 

Test is highly flaky. Reproducible with only 1 child process. Recent failures are more consistent approximately 1/500 failures until about r242500 and then its 1/20000

Reproduced with :

run-webkit-tests http/tests/websocket/tests/hybi/handshake-ok-with-legacy-sec-websocket-response-headers.html --iterations 500 --child-processes 1

Due to high flakiness, changing test expectations to include Mac WK2.

Diff:

@@ -1,3 +1,5 @@
+CONSOLE MESSAGE: WebSocket network error: The operation couldn’t be completed. Connection reset by peer
+CONSOLE MESSAGE: WebSocket network error: The operation couldn’t be completed. Connection reset by peer
 Test whether WebSocket handshake is OK with legacy response headers 'Sec-WebSocket-Location' and 'Sec-WebSocket-Origin'.
 
 On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".

[Shawn Roberts]

Marked test as flaky on Mac OS in https://trac.webkit.org/changeset/242887/webkit