161142 2016-08-24 09:44:27 -0700 REGRESSION (r204844): imported/w3c/web-platform-tests/html/dom/interfaces.html is a flaky failure 2016-08-27 07:59:31 -0700 1 1 1 Unclassified WebKit Tools / Tests Other Unspecified Unspecified NEW https://bugs.webkit.org/show_bug.cgi?id=161067 https://bugs.webkit.org/show_bug.cgi?id=161145 https://bugs.webkit.org/show_bug.cgi?id=161231 P2 Normal --- 161290 1 ap youennf cdumez lforschler ryanhaddad youennf oldest_to_newest 1222675 0 ap 2016-08-24 09:44:27 -0700 https://build.webkit.org/results/Apple%20El%20Capitan%20Release%20WK1%20(Tests)/r204895%20(9065)/imported/w3c/web-platform-tests/html/dom/interfaces-diff.txt has stared to flakily fail yesterday: @@ -1,3 +1,4 @@ +Blocked access to external URL http://www1.localhost:8800/ CONSOLE MESSAGE: line 236: callback not yet supported CONSOLE MESSAGE: line 236: callback not yet supported CONSOLE MESSAGE: line 236: callback not yet supported https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&showExpectations=true&revision=204895&tests=imported%2Fw3c%2Fweb-platform-tests%2Fhtml%2Fdom%2Finterfaces.html 1222677 1 ap 2016-08-24 09:47:09 -0700 There is a number of tests using www1.localhost, which are almost certainly not doing anything useful in our environment. 1222692 2 cdumez 2016-08-24 10:09:04 -0700 Temporarily marked as flaky in <http://trac.webkit.org/changeset/204902>. I'll investigate later today but this is likely caused by the testRunner.setCanOpenWindows() call I added. 1222695 3 youennf 2016-08-24 10:11:05 -0700 > +Blocked access to external URL http://www1.localhost:8800/ If the issue is about trying to get http://www1.localhost:8800/, the test might need to be changed here and in WPT repo. For fetch tests, we fixed this issue through /LayoutTests/imported/w3c/web-platform-tests/fetch/api/resources/get-host-info.sub.js. 1222709 4 youennf 2016-08-24 10:33:07 -0700 (In reply to comment #3) > > +Blocked access to external URL http://www1.localhost:8800/ > > If the issue is about trying to get http://www1.localhost:8800/, the test > might need to be changed here and in WPT repo. > > For fetch tests, we fixed this issue through > /LayoutTests/imported/w3c/web-platform-tests/fetch/api/resources/get-host- > info.sub.js. Doing a quick grep at imported tests in WebKit, there are about 30 files to change. I'll try to check how we could apply the same solution. 1222711 5 youennf 2016-08-24 10:33:26 -0700 ... tomorrow 1222742 6 cdumez 2016-08-24 11:11:16 -0700 Thanks for looking Youenn, this is likely the source of the flakiness of the newly imported html/browsers tests. 1222770 7 cdumez 2016-08-24 12:08:01 -0700 Skipped tests trying to load www1.localhost via <http://trac.webkit.org/changeset/204915> instead of skipping imported/w3c/web-platform-tests/html/dom/interfaces.html. 1223015 8 youennf 2016-08-25 02:35:50 -0700 I uploaded some PR to fix some of these issues. See https://github.com/w3c/web-platform-tests/pull/3543 and https://github.com/w3c/web-platform-tests/pull/3544. Some WPT tests use 3 domains while we only have two in WebKit (localhost and 127.0.0.1). Can we find a way to have 3 domains in WebKit bots? See for instance: - html/infrastructure/urls/dynamic-changes-to-base-urls/dynamic-urls.sub.xhtml - html/browsers/origin/cross-origin-objects/cross-origin-objects.sub.html - dom/events/EventListener-incumbent-global-1.sub.html - dom/events/EventListener-incumbent-global-2.sub.html Maybe some of these tests could be modified to only use 2 domains. In the meantime, maybe we should remove these tests and stop importing them. 1223094 9 ap 2016-08-25 09:54:52 -0700 > we only have two in WebKit (localhost and 127.0.0.1) There are also have http vs https, and several ports (8000 and 8080 for httpd). 1223096 10 youennf 2016-08-25 09:58:55 -0700 (In reply to comment #9) > > we only have two in WebKit (localhost and 127.0.0.1) > > There are also have http vs https, and several ports (8000 and 8080 for > httpd). It is 2 domains, not only 2 origins unfortunately. This has an impact on cookies for instance. Also, is subdomain relationship (localhost <-> www.localhost) used in some cases? 1223102 11 ap 2016-08-25 10:06:45 -0700 Do any of the tests that you listed as impacted test cookies? That's not clear from the names. 1223103 12 ap 2016-08-25 10:07:50 -0700 We can test some sub-domain functionality by allowing changing the domain from 127.0.0.1 to 0.0.1 (only in tests, as opt-in IIRC). That's not a very good solution, of course. 1223113 13 youennf 2016-08-25 10:16:51 -0700 (In reply to comment #11) > Do any of the tests that you listed as impacted test cookies? That's not > clear from the names. I haven't precisely checked for these tests. They seem to be more related to frame/iframe. There is one fetch api test that is about that, see fetch/api/cors-cookies.html 1223293 14 ap 2016-08-25 15:51:54 -0700 This test now passes on regular bots, but it's super flaky on EWS: @@ -5,6 +5,7 @@ CONSOLE MESSAGE: line 236: callback not yet supported CONSOLE MESSAGE: line 236: callback not yet supported CONSOLE MESSAGE: line 236: callback not yet supported +CONSOLE MESSAGE: WebSocket network error: The operation couldn’t be completed. (kCFErrorDomainCFNetwork error 2.) HTML IDL tests 1223297 15 cdumez 2016-08-25 15:56:13 -0700 (In reply to comment #14) > This test now passes on regular bots, but it's super flaky on EWS: > > @@ -5,6 +5,7 @@ > CONSOLE MESSAGE: line 236: callback not yet supported > CONSOLE MESSAGE: line 236: callback not yet supported > CONSOLE MESSAGE: line 236: callback not yet supported > +CONSOLE MESSAGE: WebSocket network error: The operation couldn’t be > completed. (kCFErrorDomainCFNetwork error 2.) > HTML IDL tests This is a different error and is probably related to the fact that this tests has checks related to WebSocket. I'll investigate but in the mean time, we can probably comment out the WebSocket checks. 1223443 16 youennf 2016-08-26 02:01:30 -0700 (In reply to comment #15) > (In reply to comment #14) > > This test now passes on regular bots, but it's super flaky on EWS: > > > > @@ -5,6 +5,7 @@ > > CONSOLE MESSAGE: line 236: callback not yet supported > > CONSOLE MESSAGE: line 236: callback not yet supported > > CONSOLE MESSAGE: line 236: callback not yet supported > > +CONSOLE MESSAGE: WebSocket network error: The operation couldn’t be > > completed. (kCFErrorDomainCFNetwork error 2.) > > HTML IDL tests > > This is a different error and is probably related to the fact that this > tests has checks related to WebSocket. I'll investigate but in the mean > time, we can probably comment out the WebSocket checks. I filed bug 161231 to update wpt tests. I'll handle that issue as part of this one. That should also fix some issues with ww1.localhost URLs. 1223732 17 ryanhaddad 2016-08-26 14:56:57 -0700 (In reply to comment #14) > This test now passes on regular bots, but it's super flaky on EWS: > > @@ -5,6 +5,7 @@ > CONSOLE MESSAGE: line 236: callback not yet supported > CONSOLE MESSAGE: line 236: callback not yet supported > CONSOLE MESSAGE: line 236: callback not yet supported > +CONSOLE MESSAGE: WebSocket network error: The operation couldn’t be > completed. (kCFErrorDomainCFNetwork error 2.) > HTML IDL tests Temporarily skipped test in https://trac.webkit.org/changeset/205049