Bug 199013

Summary: Layout Tests in imported/w3c/web-platform-tests/websockets/ are flakey failures after r246406
Product: WebKit Reporter: Russell Epstein <repstein>
Component: Tools / TestsAssignee: youenn fablet <youennf>
Status: RESOLVED FIXED    
Severity: Normal CC: achristensen, commit-queue, mrobinson, repstein, ryanhaddad, sroberts, tsavell, webkit-bug-importer, youennf
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Update Test Expectations
none
Update TestExpectations
none
Patch
none
Patch none

Description Russell Epstein 2019-06-19 10:19:06 PDT 
imported/w3c/web-platform-tests/websockets/Create-Secure-verify-url-set-non-default-port.any.html flakily failing since import in 246406

Confirmed issue is not exclusive to WK2 locally with MacOS WK1 repro steps listed below.

Steps to Reproduce:

MacOS WK1 with Build 246585:

run-webkit-tests --iter 1000 -f --exit-after-n-failures 20 imported/w3c/web-platform-tests/websockets/Create-Secure-verify-url-set-non-default-port.any.html -1


iOS-Simulator WK2 with Build 246585:

run-webkit-tests --iter 1000 --child-processes 1 --exit-after-n-failures 20 --ios-simulator imported/w3c/web-platform-tests/websockets/Create-Secure-verify-url-set-non-default-port.any.html 

Flakiness Dashboard:

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2Fwebsockets%2FCreate-Secure-verify-url-set-non-default-port.any.html
Comment 1 Russell Epstein 2019-06-19 10:28:15 PDT 
Created attachment 372475 [details]
Update Test Expectations
Comment 2 Truitt Savell 2019-06-19 10:29:49 PDT 
Comment on attachment 372475 [details]
Update Test Expectations

Clearing flags on attachment: 372475

Committed r246594: <https://trac.webkit.org/changeset/246594>
Comment 3 Radar WebKit Bug Importer 2019-06-19 10:31:52 PDT 
<rdar://problem/51902173>
Comment 4 Russell Epstein 2019-06-19 16:09:28 PDT 
Identified additional tests introduced in https://trac.webkit.org/changeset/246406/webkit that have been flakey.

Flakey Tests:

imported/w3c/web-platform-tests/websockets/close-invalid.any.html
imported/w3c/web-platform-tests/websockets/Create-Secure-verify-url-set-non-default-port.any.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/001.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/close/close-replace.html
imported/w3c/web-platform-tests/websockets/constructor/017.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/send/003.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/send/004.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/002.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/003.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/004.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/readyState/001.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/send/001.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/001.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/002.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/006.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/007.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/004.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/bufferedAmount/bufferedAmount-initial.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/004.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/003.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/bufferedAmount/bufferedAmount-readonly.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/bufferedAmount/bufferedAmount-initial.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/003.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/008.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/003.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/007.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/008.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/009.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/004.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/002.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/003.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/001.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/004.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/006.html
Comment 5 youenn fablet 2019-06-19 16:14:59 PDT 
Best might be to skip for now the newly imported tests that are flaky, at least the WSS (secure web socket ones) which seem to be difficult for us to pass since quite a bit of time.
Comment 6 Russell Epstein 2019-06-19 16:27:49 PDT 
Created attachment 372499 [details]
Update TestExpectations
Comment 7 Truitt Savell 2019-06-19 16:30:07 PDT 
Comment on attachment 372499 [details]
Update TestExpectations

Clearing flags on attachment: 372499

Committed r246614: <https://trac.webkit.org/changeset/246614>
Comment 8 Russell Epstein 2019-06-19 16:30:52 PDT 
Cleaned up list of flakey tests by removing duplicate entries.

Updated List:

imported/w3c/web-platform-tests/websockets/close-invalid.any.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/001.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/close/close-replace.html
imported/w3c/web-platform-tests/websockets/constructor/017.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/send/003.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/send/004.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/002.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/003.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/004.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/readyState/001.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/send/001.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/001.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/002.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/006.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/003.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/bufferedAmount/bufferedAmount-readonly.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/bufferedAmount/bufferedAmount-initial.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/003.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/007.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/008.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/009.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/004.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/002.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/001.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/004.html
Comment 9 Russell Epstein 2019-06-19 16:32:09 PDT 
Marked test list as flakey in https://trac.webkit.org/changeset/246614/webkit.
Comment 10 youenn fablet 2019-11-05 01:22:09 PST 
Created attachment 382811 [details]
Patch
Comment 11 youenn fablet 2019-11-05 08:26:09 PST 
Created attachment 382826 [details]
Patch
Comment 12 youenn fablet 2019-11-07 16:07:38 PST 
Ping review?
Comment 13 youenn fablet 2019-11-12 18:14:02 PST 
Ping review
Comment 14 Alex Christensen 2019-11-13 11:17:58 PST 
Comment on attachment 382826 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=382826&action=review

> LayoutTests/TestExpectations:3607
> +imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/close/close-replace.html [ DumpJSConsoleLogInStdErr ]

Why not just the whole directory?
Comment 15 youenn fablet 2019-11-13 12:49:11 PST 
(In reply to Alex Christensen from comment #14)
> Comment on attachment 382826 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=382826&action=review
> 
> > LayoutTests/TestExpectations:3607
> > +imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/close/close-replace.html [ DumpJSConsoleLogInStdErr ]
> 
> Why not just the whole directory?

Yep, we should probably do that for fetch, xhr, service worker and web sockets.
Or maybe the whole WPT folder.
Comment 16 WebKit Commit Bot 2019-11-13 13:47:16 PST 
Comment on attachment 382826 [details]
Patch

Clearing flags on attachment: 382826

Committed r252429: <https://trac.webkit.org/changeset/252429>
Comment 17 WebKit Commit Bot 2019-11-13 13:47:18 PST 
All reviewed patches have been landed.  Closing bug.
Comment 18 Martin Robinson 2021-08-11 04:51:23 PDT 
imported/w3c/web-platform-tests/websockets/Create-Secure-verify-url-set-non-default-port.any.htm no longer exists, so I'll remove it from test expectations.