Bug 199013

Summary:

Layout Tests in imported/w3c/web-platform-tests/websockets/ are flakey failures after r246406

Product:

WebKit

Reporter:

Russell Epstein <repstein>

Component:

Tools / Tests

Assignee:

youenn fablet <youennf>

Status:

RESOLVED FIXED

Severity:

Normal

CC:

achristensen, commit-queue, mrobinson, repstein, ryanhaddad, sroberts, tsavell, webkit-bug-importer, youennf

Priority:

Keywords:

InRadar

Version:

WebKit Nightly Build

Hardware:

Unspecified

OS:

Unspecified

Attachments:

Description	Flags
Update Test Expectations	none
Update TestExpectations	none
Patch	none
Patch	none

Russell Epstein

Reported 2019-06-19 10:19:06 PDT

imported/w3c/web-platform-tests/websockets/Create-Secure-verify-url-set-non-default-port.any.html flakily failing since import in 246406 Confirmed issue is not exclusive to WK2 locally with MacOS WK1 repro steps listed below. Steps to Reproduce: MacOS WK1 with Build 246585: run-webkit-tests --iter 1000 -f --exit-after-n-failures 20 imported/w3c/web-platform-tests/websockets/Create-Secure-verify-url-set-non-default-port.any.html -1 iOS-Simulator WK2 with Build 246585: run-webkit-tests --iter 1000 --child-processes 1 --exit-after-n-failures 20 --ios-simulator imported/w3c/web-platform-tests/websockets/Create-Secure-verify-url-set-non-default-port.any.html Flakiness Dashboard: https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2Fwebsockets%2FCreate-Secure-verify-url-set-non-default-port.any.html

Attachments
Update Test Expectations (2.56 KB, patch) 2019-06-19 10:28 PDT, Russell Epstein	no flags	Details Formatted Diff Diff
Update TestExpectations (4.26 KB, patch) 2019-06-19 16:27 PDT, Russell Epstein	no flags	Details Formatted Diff Diff
Patch (12.36 KB, patch) 2019-11-05 01:22 PST, youenn fablet	no flags	Details Formatted Diff Diff
Patch (11.43 KB, patch) 2019-11-05 08:26 PST, youenn fablet	no flags	Details Formatted Diff Diff
Show Obsolete (3) View All Add attachment proposed patch, testcase, etc.

Russell Epstein

Comment 1 2019-06-19 10:28:15 PDT

Created attachment 372475 [details] Update Test Expectations

Truitt Savell

Comment 2 2019-06-19 10:29:49 PDT

Comment on attachment 372475 [details] Update Test Expectations Clearing flags on attachment: 372475 Committed r246594: <https://trac.webkit.org/changeset/246594>

Radar WebKit Bug Importer

Comment 3 2019-06-19 10:31:52 PDT

<rdar://problem/51902173>

Russell Epstein

Comment 4 2019-06-19 16:09:28 PDT

Identified additional tests introduced in https://trac.webkit.org/changeset/246406/webkit that have been flakey. Flakey Tests: imported/w3c/web-platform-tests/websockets/close-invalid.any.html imported/w3c/web-platform-tests/websockets/Create-Secure-verify-url-set-non-default-port.any.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/001.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/close/close-replace.html imported/w3c/web-platform-tests/websockets/constructor/017.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/send/003.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/send/004.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/002.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/003.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/004.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/readyState/001.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/send/001.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/001.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/002.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/006.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/007.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/004.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/bufferedAmount/bufferedAmount-initial.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/004.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/003.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/bufferedAmount/bufferedAmount-readonly.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/bufferedAmount/bufferedAmount-initial.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/003.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/008.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/003.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/007.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/008.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/009.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/004.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/002.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/003.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/001.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/004.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/006.html

youenn fablet

Comment 5 2019-06-19 16:14:59 PDT

Best might be to skip for now the newly imported tests that are flaky, at least the WSS (secure web socket ones) which seem to be difficult for us to pass since quite a bit of time.

Russell Epstein

Comment 6 2019-06-19 16:27:49 PDT

Created attachment 372499 [details] Update TestExpectations

Truitt Savell

Comment 7 2019-06-19 16:30:07 PDT

Comment on attachment 372499 [details] Update TestExpectations Clearing flags on attachment: 372499 Committed r246614: <https://trac.webkit.org/changeset/246614>

Russell Epstein

Comment 8 2019-06-19 16:30:52 PDT

Cleaned up list of flakey tests by removing duplicate entries. Updated List: imported/w3c/web-platform-tests/websockets/close-invalid.any.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/001.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/close/close-replace.html imported/w3c/web-platform-tests/websockets/constructor/017.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/send/003.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/send/004.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/002.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/003.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/url/004.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/readyState/001.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/send/001.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/001.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/002.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/006.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/003.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/bufferedAmount/bufferedAmount-readonly.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/bufferedAmount/bufferedAmount-initial.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/003.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/007.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/008.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/009.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/004.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/002.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/001.html imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/constants/004.html

Russell Epstein

Comment 9 2019-06-19 16:32:09 PDT

Marked test list as flakey in https://trac.webkit.org/changeset/246614/webkit.

youenn fablet

Comment 10 2019-11-05 01:22:09 PST

Created attachment 382811 [details] Patch

youenn fablet

Comment 11 2019-11-05 08:26:09 PST

Created attachment 382826 [details] Patch

youenn fablet

Comment 12 2019-11-07 16:07:38 PST

Ping review?

youenn fablet

Comment 13 2019-11-12 18:14:02 PST

Ping review

Alex Christensen

Comment 14 2019-11-13 11:17:58 PST

Comment on attachment 382826 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=382826&action=review > LayoutTests/TestExpectations:3607 > +imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/close/close-replace.html [ DumpJSConsoleLogInStdErr ] Why not just the whole directory?

youenn fablet

Comment 15 2019-11-13 12:49:11 PST

(In reply to Alex Christensen from comment #14) > Comment on attachment 382826 [details] > Patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=382826&action=review > > > LayoutTests/TestExpectations:3607 > > +imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/close/close-replace.html [ DumpJSConsoleLogInStdErr ] > > Why not just the whole directory? Yep, we should probably do that for fetch, xhr, service worker and web sockets. Or maybe the whole WPT folder.

WebKit Commit Bot

Comment 16 2019-11-13 13:47:16 PST

Comment on attachment 382826 [details] Patch Clearing flags on attachment: 382826 Committed r252429: <https://trac.webkit.org/changeset/252429>

WebKit Commit Bot

Comment 17 2019-11-13 13:47:18 PST

All reviewed patches have been landed. Closing bug.

Martin Robinson

Comment 18 2021-08-11 04:51:23 PDT

imported/w3c/web-platform-tests/websockets/Create-Secure-verify-url-set-non-default-port.any.htm no longer exists, so I'll remove it from test expectations.

Note You need to log in before you can comment on or make changes to this bug.