Bug 161142 – REGRESSION (r204844): imported/w3c/web-platform-tests/html/dom/interfaces.html is a flaky failure

• Status: NEW
• Component: Tools / Tests

[Alexey Proskuryakov]

https://build.webkit.org/results/Apple%20El%20Capitan%20Release%20WK1%20(Tests)/r204895%20(9065)/imported/w3c/web-platform-tests/html/dom/interfaces-diff.txt has stared to flakily fail yesterday:

@@ -1,3 +1,4 @@
+Blocked access to external URL http://www1.localhost:8800/
 CONSOLE MESSAGE: line 236: callback not yet supported
 CONSOLE MESSAGE: line 236: callback not yet supported
 CONSOLE MESSAGE: line 236: callback not yet supported

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&showExpectations=true&revision=204895&tests=imported%2Fw3c%2Fweb-platform-tests%2Fhtml%2Fdom%2Finterfaces.html

[Alexey Proskuryakov]

There is a number of tests using www1.localhost, which are almost certainly not doing anything useful in our environment.

[Chris Dumez]

Temporarily marked as flaky in <http://trac.webkit.org/changeset/204902>. I'll investigate later today but this is likely caused by the testRunner.setCanOpenWindows() call I added.

[youenn fablet]

> +Blocked access to external URL http://www1.localhost:8800/

If the issue is about trying to get http://www1.localhost:8800/, the test might need to be changed here and in WPT repo.

For fetch tests, we fixed this issue through /LayoutTests/imported/w3c/web-platform-tests/fetch/api/resources/get-host-info.sub.js.

[youenn fablet]

(In reply to comment #3)
> > +Blocked access to external URL http://www1.localhost:8800/
> 
> If the issue is about trying to get http://www1.localhost:8800/, the test
> might need to be changed here and in WPT repo.
> 
> For fetch tests, we fixed this issue through
> /LayoutTests/imported/w3c/web-platform-tests/fetch/api/resources/get-host-
> info.sub.js.

Doing a quick grep at imported tests in WebKit, there are about 30 files to change.
I'll try to check how we could apply the same solution.

[youenn fablet]

... tomorrow

[Chris Dumez]

Thanks for looking Youenn, this is likely the source of the flakiness of the newly imported html/browsers tests.

[Chris Dumez]

Skipped tests trying to load www1.localhost via <http://trac.webkit.org/changeset/204915> instead of skipping imported/w3c/web-platform-tests/html/dom/interfaces.html.

[youenn fablet]

I uploaded some PR to fix some of these issues.
See https://github.com/w3c/web-platform-tests/pull/3543 and https://github.com/w3c/web-platform-tests/pull/3544.

Some WPT tests use 3 domains while we only have two in WebKit (localhost and 127.0.0.1). Can we find a way to have 3 domains in WebKit bots?

See for instance:
- html/infrastructure/urls/dynamic-changes-to-base-urls/dynamic-urls.sub.xhtml
- html/browsers/origin/cross-origin-objects/cross-origin-objects.sub.html
- dom/events/EventListener-incumbent-global-1.sub.html
- dom/events/EventListener-incumbent-global-2.sub.html

Maybe some of these tests could be modified to only use 2 domains.
In the meantime, maybe we should remove these tests and stop importing them.

[Alexey Proskuryakov]

> we only have two in WebKit (localhost and 127.0.0.1)

There are also have http vs https, and several ports (8000 and 8080 for httpd).

[youenn fablet]

(In reply to comment #9)
> > we only have two in WebKit (localhost and 127.0.0.1)
> 
> There are also have http vs https, and several ports (8000 and 8080 for
> httpd).

It is 2 domains, not only 2 origins unfortunately.
This has an impact on cookies for instance.
Also, is subdomain relationship (localhost <-> www.localhost) used in some cases?

[Alexey Proskuryakov]

Do any of the tests that you listed as impacted test cookies? That's not clear from the names.

[Alexey Proskuryakov]

We can test some sub-domain functionality by allowing changing the domain from 127.0.0.1 to 0.0.1 (only in tests, as opt-in IIRC). That's not a very good solution, of course.

[youenn fablet]

(In reply to comment #11)
> Do any of the tests that you listed as impacted test cookies? That's not
> clear from the names.

I haven't precisely checked for these tests.
They seem to be more related to frame/iframe.

There is one fetch api test that is about that, see fetch/api/cors-cookies.html

[Alexey Proskuryakov]

This test now passes on regular bots, but it's super flaky on EWS:

@@ -5,6 +5,7 @@
 CONSOLE MESSAGE: line 236: callback not yet supported
 CONSOLE MESSAGE: line 236: callback not yet supported
 CONSOLE MESSAGE: line 236: callback not yet supported
+CONSOLE MESSAGE: WebSocket network error: The operation couldn’t be completed. (kCFErrorDomainCFNetwork error 2.)
 HTML IDL tests

[Chris Dumez]

(In reply to comment #14)
> This test now passes on regular bots, but it's super flaky on EWS:
> 
> @@ -5,6 +5,7 @@
>  CONSOLE MESSAGE: line 236: callback not yet supported
>  CONSOLE MESSAGE: line 236: callback not yet supported
>  CONSOLE MESSAGE: line 236: callback not yet supported
> +CONSOLE MESSAGE: WebSocket network error: The operation couldn’t be
> completed. (kCFErrorDomainCFNetwork error 2.)
>  HTML IDL tests

This is a different error and is probably related to the fact that this tests has checks related to WebSocket. I'll investigate but in the mean time, we can probably comment out the WebSocket checks.

[youenn fablet]

(In reply to comment #15)
> (In reply to comment #14)
> > This test now passes on regular bots, but it's super flaky on EWS:
> > 
> > @@ -5,6 +5,7 @@
> >  CONSOLE MESSAGE: line 236: callback not yet supported
> >  CONSOLE MESSAGE: line 236: callback not yet supported
> >  CONSOLE MESSAGE: line 236: callback not yet supported
> > +CONSOLE MESSAGE: WebSocket network error: The operation couldn’t be
> > completed. (kCFErrorDomainCFNetwork error 2.)
> >  HTML IDL tests
> 
> This is a different error and is probably related to the fact that this
> tests has checks related to WebSocket. I'll investigate but in the mean
> time, we can probably comment out the WebSocket checks.

I filed bug 161231 to update wpt tests. I'll handle that issue as part of this one.
That should also fix some issues with ww1.localhost URLs.

[Ryan Haddad]

(In reply to comment #14)
> This test now passes on regular bots, but it's super flaky on EWS:
> 
> @@ -5,6 +5,7 @@
>  CONSOLE MESSAGE: line 236: callback not yet supported
>  CONSOLE MESSAGE: line 236: callback not yet supported
>  CONSOLE MESSAGE: line 236: callback not yet supported
> +CONSOLE MESSAGE: WebSocket network error: The operation couldn’t be
> completed. (kCFErrorDomainCFNetwork error 2.)
>  HTML IDL tests

Temporarily skipped test in https://trac.webkit.org/changeset/205049