Some W3C tests make request on dynamically computed URLs such as "http://www2."+location.host. See for instance https://github.com/w3c/web-platform-tests/blob/master/XMLHttpRequest/send-redirect-to-cors.htm This breaks the underlying test framework. First, "www2." + location.host cannot be referenced without changing /etc/hosts. Mapping should be set to 127.0.0.1 Second, "www2." + location.host will not be authorized by DumpRenderTrees as it is neither "localhost" nor 127.0.0.1
> First, "www2." + location.host cannot be referenced without changing /etc/hosts. Mapping should be set to 127.0.0.1 The simplest approach would be to require proper setting of /etc/hosts, for instance by adding entries for "www.web-platform.test", "www2.web-platform.test"... > Second, "www2." + location.host will not be authorized by DumpRenderTree as it is neither "localhost" nor 127.0.0.1 One solution is to authorize URLS with specific prefixes (.test or .local e.g.) Another solution is to check that the host resolves to 127.0.0.1.
Created attachment 233362 [details] WIP: Enable all *.localhost URLS
I don't think that we should be doing this. If I were a new contributor to submit my first patch, I'd certainly refuse to work on a project that insists on modifying my /etc/hosts, or on making any other persistent changes on my machine.
(In reply to comment #3) > I don't think that we should be doing this. > > If I were a new contributor to submit my first patch, I'd certainly refuse > to work on a project that insists on modifying my /etc/hosts, or on making > any other persistent changes on my machine. This is how it is suggested to be done in https://github.com/w3c/web-platform-tests. Bots can be made to work for these tests quite easily, but that does not help contributors much. Do you see any other practical alternative? I am not sure a proxy would be the right solution here. I do not know exactly how much tests require these URLs. The XHR tests are covered by other WebKit tests so this is probably ok for these tests. It would be nice though to have a bot dedicated to WPT conformance, trying to pass all tests and doing reports. At least we would know whether these tests (and other WPT) are passing or not.
In WebKit tests, we've been using the distinction between 127.0.0.1 and localhost, as well as http/https and multiple ports. Perhaps W3C tests could be changed to not require local setup? That seems like it would be a win for everyone. > It would be nice though to have a bot dedicated to WPT conformance That's a good idea. Do you know if there is anyone who would track the progress, and/or actively work on increasing the pass rate?
(In reply to comment #5) > In WebKit tests, we've been using the distinction between 127.0.0.1 and > localhost, as well as http/https and multiple ports. Perhaps W3C tests could > be changed to not require local setup? That seems like it would be a win for > everyone. That is probably not straightforward as WPT tests are made to work both locally as well as from w3c-tests.org. > > It would be nice though to have a bot dedicated to WPT conformance > > That's a good idea. Do you know if there is anyone who would track the > progress, and/or actively work on increasing the pass rate? I am interested in that. cdumez did some work to improve the pass rate, for instance on the binding generator. I do not know the context of this activity though.
> I do not know exactly how much tests require these URLs. 27 -expected.txt files exhibit a "Blocked access message". There may be other tests not running properly due to these limitations though.
(In reply to comment #3) > I don't think that we should be doing this. > > If I were a new contributor to submit my first patch, I'd certainly refuse > to work on a project that insists on modifying my /etc/hosts, or on making > any other persistent changes on my machine. Well, you don't have to as a developer but then you would have some tests failing locally. The fact is that - at the moment - anyone working on web-platform-tests has to modify their hosts file. Sure, it'd be nice if we could work with upstream to try and get them to adopt a configuration that is more convenient for us. We can certainly try this route but I have my doubts it will be successful. I have also been told that out configuration is insufficient to cover all cases the tests are actually covering. If we cannot get upstream tests to change then I think we need to deal with this downstream somehow. Right now, the solution is to skip to tests or have their baseline claiming the test fails even though it would pass if the machine was properly configured. I think we have to do better than this. I personally think having a script called "prepare-machine-for-development" or so (which would add the right entries to /etc/hosts for you) would be a better solution than what we have right now.
> Sure, it'd be nice if we could work with upstream to try and get them to > adopt a configuration that is more convenient for us. We can certainly try > this route but I have my doubts it will be successful. I have also been told > that out configuration is insufficient to cover all cases the tests are > actually covering. Right, some tests require different hostnames, not just different ports. Some other tests require subdomain relationships (www.localhost/localhost for instance).
I think should add a feature at a low level of WebKit itself that can let us test in this fashion with development builds. It can be something that the test runners turn on and something that we can turn on with some kind of debugging menu. There must be a relatively small number of bottlenecks for URL and host name processing that we could hook into. I agree that we don’t want to make test running require changing the configuration of the computer unless it is absolutely necessary and I don’t think it is.
*** Bug 168220 has been marked as a duplicate of this bug. ***
Simon found another related quirk - in Chrome, subdomains of localhost all resolve to 127.0.0.1. This seems to have started with some weird requirements in <https://tools.ietf.org/html/rfc6761#section-6.3>. Chrome chose to follow this RFC, but no one else appears convinced. Some history in <https://bugs.chromium.org/p/chromium/issues/detail?id=489973> and <https://tools.ietf.org/html/draft-west-let-localhost-be-localhost>.
See also: https://github.com/web-platform-tests/wpt/issues/3243 https://github.com/web-platform-tests/wpt/pull/3302 https://github.com/web-platform-tests/wpt/issues/15692
<rdar://problem/71449599>
I'm making this block bug 171934 ; localhost and loopback IP addresses are considered non-mixed content so being able to access other addresses for testing purpose would be useful (otherwise we need to disable the preferences when running tests). This is also causing bug 218795.
(and incidentally supporting .localhost as done in attachment bug 233362 and the ipv6 loopback ip address as suggested in the FIXME would help for testing bug 218627 and bug 218623)
(In reply to Darin Adler from comment #10) > I think should add a feature at a low level of WebKit itself that can let us > test in this fashion with development builds. It can be something that the > test runners turn on and something that we can turn on with some kind of > debugging menu. There must be a relatively small number of bottlenecks for > URL and host name processing that we could hook into. > > I agree that we don’t want to make test running require changing the > configuration of the computer unless it is absolutely necessary and I don’t > think it is. Note that Chrome and Firefox both have such functionality in their shipping browsers, which matters if we want to be able to run WPT tests against Safari without changing the system configuration (which is, of course, impossible on iOS on device). This does also mean that Safari is the only browser that actually requires /etc/hosts to be modified to run the tests through wptrunner. At least currently, WebKit doesn't have any way to intercept the DNS request within the network stack, and I don't know how easy that is for the different network stacks currently supported. Moving away from localhost/loopback addresses is almost certainly needed to match the expected behaviour in some tests, given localhost is special-cased in some places in the web platform. Also, there's also nonexistent.[wpt domain] that's meant to fail to resolve, which we probably need to deal with too. (I know some Gecko developers had issues with some ISPs where the DNS provider intercepted any failure to redirect to a search page, hence it _wasn't_ failing to resolve.)
(In reply to Sam Sneddon [:gsnedders] from comment #17) > At least currently, WebKit doesn't have any way to intercept the DNS request > within the network stack, and I don't know how easy that is for the > different network stacks currently supported. FYI, for the libsoup backend it is quite easy. We just write a new GResolver, have it wrap the default system GResolver, and handle our special hostnames differently. It would need a WebKit internal API to enable use of the test GResolver.
(In reply to Michael Catanzaro from comment #18) > (In reply to Sam Sneddon [:gsnedders] from comment #17) > > At least currently, WebKit doesn't have any way to intercept the DNS request > > within the network stack, and I don't know how easy that is for the > > different network stacks currently supported. > > FYI, for the libsoup backend it is quite easy. We just write a new > GResolver, have it wrap the default system GResolver, and handle our special > hostnames differently. It would need a WebKit internal API to enable use of > the test GResolver. It would be an interesting experiment to add the necessary logic in WebKit to make use of that GResolver. We might need something like: - Add an internal feature to toggle this on/off - Update test runner source code to allow non localhost/127.0.0.1 hosts. Is there any GTK folk interested in that? I'll be happy to help.
Not sure that can help, but FWIW I had experimented non-loopback address for cocoa API tests on https://bugs.webkit.org/show_bug.cgi?id=235083 ; the patch seemed to work locally on my mac but not on EWS.
Initial work on this done here: https://github.com/WebKit/WebKit/pull/2489