Bug 199013 - Layout Tests in imported/w3c/web-platform-tests/websockets/ are flakey failures after r246406
Summary: Layout Tests in imported/w3c/web-platform-tests/websockets/ are flakey failur...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: youenn fablet
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2019-06-19 10:19 PDT by Russell Epstein
Modified: 2021-08-11 04:51 PDT (History)
9 users (show)

See Also:


Attachments
Update Test Expectations (2.56 KB, patch)
2019-06-19 10:28 PDT, Russell Epstein
no flags Details | Formatted Diff | Diff
Update TestExpectations (4.26 KB, patch)
2019-06-19 16:27 PDT, Russell Epstein
no flags Details | Formatted Diff | Diff
Patch (12.36 KB, patch)
2019-11-05 01:22 PST, youenn fablet
no flags Details | Formatted Diff | Diff
Patch (11.43 KB, patch)
2019-11-05 08:26 PST, youenn fablet
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Russell Epstein 2019-06-19 10:19:06 PDT
imported/w3c/web-platform-tests/websockets/Create-Secure-verify-url-set-non-default-port.any.html flakily failing since import in 246406

Confirmed issue is not exclusive to WK2 locally with MacOS WK1 repro steps listed below.

Steps to Reproduce:

MacOS WK1 with Build 246585:

run-webkit-tests --iter 1000 -f --exit-after-n-failures 20 imported/w3c/web-platform-tests/websockets/Create-Secure-verify-url-set-non-default-port.any.html -1


iOS-Simulator WK2 with Build 246585:

run-webkit-tests --iter 1000 --child-processes 1 --exit-after-n-failures 20 --ios-simulator imported/w3c/web-platform-tests/websockets/Create-Secure-verify-url-set-non-default-port.any.html 

Flakiness Dashboard:

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2Fwebsockets%2FCreate-Secure-verify-url-set-non-default-port.any.html
Comment 1 Russell Epstein 2019-06-19 10:28:15 PDT
Created attachment 372475 [details]
Update Test Expectations
Comment 2 Truitt Savell 2019-06-19 10:29:49 PDT
Comment on attachment 372475 [details]
Update Test Expectations

Clearing flags on attachment: 372475

Committed r246594: <https://trac.webkit.org/changeset/246594>
Comment 3 Radar WebKit Bug Importer 2019-06-19 10:31:52 PDT
<rdar://problem/51902173>
Comment 4 Russell Epstein 2019-06-19 16:09:28 PDT
Identified additional tests introduced in https://trac.webkit.org/changeset/246406/webkit that have been flakey.

Flakey Tests:

imported/w3c/web-platform-tests/websockets/close-invalid.any.html
imported/w3c/web-platform-tests/websockets/Create-Secure-verify-url-set-non-default-port.any.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/001.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/close/close-replace.html
imported/w3c/web-platform-tests/websockets/constructor/017.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/send/003.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/send/004.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/002.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/003.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/004.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/readyState/001.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/send/001.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/001.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/002.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/006.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/007.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/004.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/bufferedAmount/bufferedAmount-initial.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/004.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/003.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/bufferedAmount/bufferedAmount-readonly.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/bufferedAmount/bufferedAmount-initial.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/003.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/008.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/003.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/007.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/008.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/009.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/004.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/002.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/003.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/001.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/004.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/006.html
Comment 5 youenn fablet 2019-06-19 16:14:59 PDT
Best might be to skip for now the newly imported tests that are flaky, at least the WSS (secure web socket ones) which seem to be difficult for us to pass since quite a bit of time.
Comment 6 Russell Epstein 2019-06-19 16:27:49 PDT
Created attachment 372499 [details]
Update TestExpectations
Comment 7 Truitt Savell 2019-06-19 16:30:07 PDT
Comment on attachment 372499 [details]
Update TestExpectations

Clearing flags on attachment: 372499

Committed r246614: <https://trac.webkit.org/changeset/246614>
Comment 8 Russell Epstein 2019-06-19 16:30:52 PDT
Cleaned up list of flakey tests by removing duplicate entries.

Updated List:

imported/w3c/web-platform-tests/websockets/close-invalid.any.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/001.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/close/close-replace.html
imported/w3c/web-platform-tests/websockets/constructor/017.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/send/003.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/send/004.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/002.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/003.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/004.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/readyState/001.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/send/001.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/001.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/002.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/006.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/003.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/bufferedAmount/bufferedAmount-readonly.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/bufferedAmount/bufferedAmount-initial.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/003.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/007.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/008.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/009.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/004.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/002.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/001.html
imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/004.html
Comment 9 Russell Epstein 2019-06-19 16:32:09 PDT
Marked test list as flakey in https://trac.webkit.org/changeset/246614/webkit.
Comment 10 youenn fablet 2019-11-05 01:22:09 PST
Created attachment 382811 [details]
Patch
Comment 11 youenn fablet 2019-11-05 08:26:09 PST
Created attachment 382826 [details]
Patch
Comment 12 youenn fablet 2019-11-07 16:07:38 PST
Ping review?
Comment 13 youenn fablet 2019-11-12 18:14:02 PST
Ping review
Comment 14 Alex Christensen 2019-11-13 11:17:58 PST
Comment on attachment 382826 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=382826&action=review

> LayoutTests/TestExpectations:3607
> +imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/close/close-replace.html [ DumpJSConsoleLogInStdErr ]

Why not just the whole directory?
Comment 15 youenn fablet 2019-11-13 12:49:11 PST
(In reply to Alex Christensen from comment #14)
> Comment on attachment 382826 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=382826&action=review
> 
> > LayoutTests/TestExpectations:3607
> > +imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/close/close-replace.html [ DumpJSConsoleLogInStdErr ]
> 
> Why not just the whole directory?

Yep, we should probably do that for fetch, xhr, service worker and web sockets.
Or maybe the whole WPT folder.
Comment 16 WebKit Commit Bot 2019-11-13 13:47:16 PST
Comment on attachment 382826 [details]
Patch

Clearing flags on attachment: 382826

Committed r252429: <https://trac.webkit.org/changeset/252429>
Comment 17 WebKit Commit Bot 2019-11-13 13:47:18 PST
All reviewed patches have been landed.  Closing bug.
Comment 18 Martin Robinson 2021-08-11 04:51:23 PDT
imported/w3c/web-platform-tests/websockets/Create-Secure-verify-url-set-non-default-port.any.htm no longer exists, so I'll remove it from test expectations.