Bug 161142

Summary: REGRESSION (r204844): imported/w3c/web-platform-tests/html/dom/interfaces.html is a flaky failure
Product: WebKit Reporter: Alexey Proskuryakov <ap>
Component: Tools / TestsAssignee: youenn fablet <youennf>
Status: NEW ---    
Severity: Normal CC: cdumez, lforschler, ryanhaddad, youennf
Priority: P2    
Version: Other   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=161067
https://bugs.webkit.org/show_bug.cgi?id=161145
https://bugs.webkit.org/show_bug.cgi?id=161231
Bug Depends on: 161290    
Bug Blocks:    

Description Alexey Proskuryakov 2016-08-24 09:44:27 PDT
https://build.webkit.org/results/Apple%20El%20Capitan%20Release%20WK1%20(Tests)/r204895%20(9065)/imported/w3c/web-platform-tests/html/dom/interfaces-diff.txt has stared to flakily fail yesterday:

@@ -1,3 +1,4 @@
+Blocked access to external URL http://www1.localhost:8800/
 CONSOLE MESSAGE: line 236: callback not yet supported
 CONSOLE MESSAGE: line 236: callback not yet supported
 CONSOLE MESSAGE: line 236: callback not yet supported

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&showExpectations=true&revision=204895&tests=imported%2Fw3c%2Fweb-platform-tests%2Fhtml%2Fdom%2Finterfaces.html
Comment 1 Alexey Proskuryakov 2016-08-24 09:47:09 PDT
There is a number of tests using www1.localhost, which are almost certainly not doing anything useful in our environment.
Comment 2 Chris Dumez 2016-08-24 10:09:04 PDT
Temporarily marked as flaky in <http://trac.webkit.org/changeset/204902>. I'll investigate later today but this is likely caused by the testRunner.setCanOpenWindows() call I added.
Comment 3 youenn fablet 2016-08-24 10:11:05 PDT
> +Blocked access to external URL http://www1.localhost:8800/

If the issue is about trying to get http://www1.localhost:8800/, the test might need to be changed here and in WPT repo.

For fetch tests, we fixed this issue through /LayoutTests/imported/w3c/web-platform-tests/fetch/api/resources/get-host-info.sub.js.
Comment 4 youenn fablet 2016-08-24 10:33:07 PDT
(In reply to comment #3)
> > +Blocked access to external URL http://www1.localhost:8800/
> 
> If the issue is about trying to get http://www1.localhost:8800/, the test
> might need to be changed here and in WPT repo.
> 
> For fetch tests, we fixed this issue through
> /LayoutTests/imported/w3c/web-platform-tests/fetch/api/resources/get-host-
> info.sub.js.

Doing a quick grep at imported tests in WebKit, there are about 30 files to change.
I'll try to check how we could apply the same solution.
Comment 5 youenn fablet 2016-08-24 10:33:26 PDT
... tomorrow
Comment 6 Chris Dumez 2016-08-24 11:11:16 PDT
Thanks for looking Youenn, this is likely the source of the flakiness of the newly imported html/browsers tests.
Comment 7 Chris Dumez 2016-08-24 12:08:01 PDT
Skipped tests trying to load www1.localhost via <http://trac.webkit.org/changeset/204915> instead of skipping imported/w3c/web-platform-tests/html/dom/interfaces.html.
Comment 8 youenn fablet 2016-08-25 02:35:50 PDT
I uploaded some PR to fix some of these issues.
See https://github.com/w3c/web-platform-tests/pull/3543 and https://github.com/w3c/web-platform-tests/pull/3544.

Some WPT tests use 3 domains while we only have two in WebKit (localhost and 127.0.0.1). Can we find a way to have 3 domains in WebKit bots?

See for instance:
- html/infrastructure/urls/dynamic-changes-to-base-urls/dynamic-urls.sub.xhtml
- html/browsers/origin/cross-origin-objects/cross-origin-objects.sub.html
- dom/events/EventListener-incumbent-global-1.sub.html
- dom/events/EventListener-incumbent-global-2.sub.html

Maybe some of these tests could be modified to only use 2 domains.
In the meantime, maybe we should remove these tests and stop importing them.
Comment 9 Alexey Proskuryakov 2016-08-25 09:54:52 PDT
> we only have two in WebKit (localhost and 127.0.0.1)

There are also have http vs https, and several ports (8000 and 8080 for httpd).
Comment 10 youenn fablet 2016-08-25 09:58:55 PDT
(In reply to comment #9)
> > we only have two in WebKit (localhost and 127.0.0.1)
> 
> There are also have http vs https, and several ports (8000 and 8080 for
> httpd).

It is 2 domains, not only 2 origins unfortunately.
This has an impact on cookies for instance.
Also, is subdomain relationship (localhost <-> www.localhost) used in some cases?
Comment 11 Alexey Proskuryakov 2016-08-25 10:06:45 PDT
Do any of the tests that you listed as impacted test cookies? That's not clear from the names.
Comment 12 Alexey Proskuryakov 2016-08-25 10:07:50 PDT
We can test some sub-domain functionality by allowing changing the domain from 127.0.0.1 to 0.0.1 (only in tests, as opt-in IIRC). That's not a very good solution, of course.
Comment 13 youenn fablet 2016-08-25 10:16:51 PDT
(In reply to comment #11)
> Do any of the tests that you listed as impacted test cookies? That's not
> clear from the names.

I haven't precisely checked for these tests.
They seem to be more related to frame/iframe.

There is one fetch api test that is about that, see fetch/api/cors-cookies.html
Comment 14 Alexey Proskuryakov 2016-08-25 15:51:54 PDT
This test now passes on regular bots, but it's super flaky on EWS:

@@ -5,6 +5,7 @@
 CONSOLE MESSAGE: line 236: callback not yet supported
 CONSOLE MESSAGE: line 236: callback not yet supported
 CONSOLE MESSAGE: line 236: callback not yet supported
+CONSOLE MESSAGE: WebSocket network error: The operation couldn’t be completed. (kCFErrorDomainCFNetwork error 2.)
 HTML IDL tests
Comment 15 Chris Dumez 2016-08-25 15:56:13 PDT
(In reply to comment #14)
> This test now passes on regular bots, but it's super flaky on EWS:
> 
> @@ -5,6 +5,7 @@
>  CONSOLE MESSAGE: line 236: callback not yet supported
>  CONSOLE MESSAGE: line 236: callback not yet supported
>  CONSOLE MESSAGE: line 236: callback not yet supported
> +CONSOLE MESSAGE: WebSocket network error: The operation couldn’t be
> completed. (kCFErrorDomainCFNetwork error 2.)
>  HTML IDL tests

This is a different error and is probably related to the fact that this tests has checks related to WebSocket. I'll investigate but in the mean time, we can probably comment out the WebSocket checks.
Comment 16 youenn fablet 2016-08-26 02:01:30 PDT
(In reply to comment #15)
> (In reply to comment #14)
> > This test now passes on regular bots, but it's super flaky on EWS:
> > 
> > @@ -5,6 +5,7 @@
> >  CONSOLE MESSAGE: line 236: callback not yet supported
> >  CONSOLE MESSAGE: line 236: callback not yet supported
> >  CONSOLE MESSAGE: line 236: callback not yet supported
> > +CONSOLE MESSAGE: WebSocket network error: The operation couldn’t be
> > completed. (kCFErrorDomainCFNetwork error 2.)
> >  HTML IDL tests
> 
> This is a different error and is probably related to the fact that this
> tests has checks related to WebSocket. I'll investigate but in the mean
> time, we can probably comment out the WebSocket checks.

I filed bug 161231 to update wpt tests. I'll handle that issue as part of this one.
That should also fix some issues with ww1.localhost URLs.
Comment 17 Ryan Haddad 2016-08-26 14:56:57 PDT
(In reply to comment #14)
> This test now passes on regular bots, but it's super flaky on EWS:
> 
> @@ -5,6 +5,7 @@
>  CONSOLE MESSAGE: line 236: callback not yet supported
>  CONSOLE MESSAGE: line 236: callback not yet supported
>  CONSOLE MESSAGE: line 236: callback not yet supported
> +CONSOLE MESSAGE: WebSocket network error: The operation couldn’t be
> completed. (kCFErrorDomainCFNetwork error 2.)
>  HTML IDL tests

Temporarily skipped test in https://trac.webkit.org/changeset/205049