Bug 127676 - WebKit tests should run with access to multiple hostnames that all resolve to the test web server
Summary: WebKit tests should run with access to multiple hostnames that all resolve to...
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: youenn fablet
URL:
Keywords:
: 168220 (view as bug list)
Depends on:
Blocks: 147782
  Show dependency treegraph
 
Reported: 2014-01-27 01:40 PST by youenn fablet
Modified: 2020-07-24 09:58 PDT (History)
5 users (show)

See Also:


Attachments
WIP: Enable all *.localhost URLS (3.07 KB, patch)
2014-06-19 08:11 PDT, youenn fablet
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description youenn fablet 2014-01-27 01:40:13 PST
Some W3C tests make request on dynamically computed URLs such as "http://www2."+location.host.
See for instance https://github.com/w3c/web-platform-tests/blob/master/XMLHttpRequest/send-redirect-to-cors.htm

This breaks the underlying test framework.
First, "www2." + location.host cannot be referenced without changing /etc/hosts. Mapping should be set to 127.0.0.1
Second, "www2." + location.host will not be authorized by DumpRenderTrees as it is neither "localhost" nor 127.0.0.1
Comment 1 youenn fablet 2014-01-27 01:47:22 PST
> First, "www2." + location.host cannot be referenced without changing /etc/hosts. Mapping should be set to 127.0.0.1

The simplest approach would be to require proper setting of /etc/hosts, for instance by adding entries for "www.web-platform.test", "www2.web-platform.test"...

> Second, "www2." + location.host will not be authorized by DumpRenderTree as it is neither "localhost" nor 127.0.0.1

One solution is to authorize URLS with specific prefixes (.test or .local e.g.) Another solution is to check that the host resolves to 127.0.0.1.
Comment 2 youenn fablet 2014-06-19 08:11:59 PDT
Created attachment 233362 [details]
WIP: Enable all *.localhost URLS
Comment 3 Alexey Proskuryakov 2016-01-12 10:35:45 PST
I don't think that we should be doing this.

If I were a new contributor to submit my first patch, I'd certainly refuse to work on a project that insists on modifying my /etc/hosts, or on making any other persistent changes on my machine.
Comment 4 youenn fablet 2016-01-12 12:31:41 PST
(In reply to comment #3)
> I don't think that we should be doing this.
> 
> If I were a new contributor to submit my first patch, I'd certainly refuse
> to work on a project that insists on modifying my /etc/hosts, or on making
> any other persistent changes on my machine.

This is how it is suggested to be done in https://github.com/w3c/web-platform-tests.
Bots can be made to work for these tests quite easily, but that does not help contributors much.

Do you see any other practical alternative?
I am not sure a proxy would be the right solution here.

I do not know exactly how much tests require these URLs.
The XHR tests are covered by other WebKit tests so this is probably ok for these tests.
It would be nice though to have a bot dedicated to WPT conformance, trying to pass all tests and doing reports. At least we would know whether these tests (and other WPT) are passing or not.
Comment 5 Alexey Proskuryakov 2016-01-12 13:15:46 PST
In WebKit tests, we've been using the distinction between 127.0.0.1 and localhost, as well as http/https and multiple ports. Perhaps W3C tests could be changed to not require local setup? That seems like it would be a win for everyone.

> It would be nice though to have a bot dedicated to WPT conformance

That's a good idea. Do you know if there is anyone who would track the progress, and/or actively work on increasing the pass rate?
Comment 6 youenn fablet 2016-01-12 14:22:36 PST
(In reply to comment #5)
> In WebKit tests, we've been using the distinction between 127.0.0.1 and
> localhost, as well as http/https and multiple ports. Perhaps W3C tests could
> be changed to not require local setup? That seems like it would be a win for
> everyone.

That is probably not straightforward as WPT tests are made to work both locally as well as from w3c-tests.org.

> > It would be nice though to have a bot dedicated to WPT conformance
> 
> That's a good idea. Do you know if there is anyone who would track the
> progress, and/or actively work on increasing the pass rate?

I am interested in that.
cdumez did some work to improve the pass rate, for instance on the binding generator. I do not know the context of this activity though.
Comment 7 youenn fablet 2016-01-15 08:55:01 PST
> I do not know exactly how much tests require these URLs.

27 -expected.txt files exhibit a "Blocked access message".
There may be other tests not running properly due to these limitations though.
Comment 8 Chris Dumez 2017-01-20 10:04:41 PST
(In reply to comment #3)
> I don't think that we should be doing this.
> 
> If I were a new contributor to submit my first patch, I'd certainly refuse
> to work on a project that insists on modifying my /etc/hosts, or on making
> any other persistent changes on my machine.

Well, you don't have to as a developer but then you would have some tests failing locally. 

The fact is that - at the moment - anyone working on web-platform-tests has to modify their hosts file.

Sure, it'd be nice if we could work with upstream to try and get them to adopt a configuration that is more convenient for us. We can certainly try this route but I have my doubts it will be successful. I have also been told that out configuration is insufficient to cover all cases the tests are actually covering.

If we cannot get upstream tests to change then I think we need to deal with this downstream somehow. Right now, the solution is to skip to tests or have their baseline claiming the test fails even though it would pass if the machine was properly configured. I think we have to do better than this. I personally think having a script called "prepare-machine-for-development" or so (which would add the right entries to /etc/hosts for you) would be a better solution than what we have right now.
Comment 9 youenn fablet 2017-01-20 10:10:13 PST
> Sure, it'd be nice if we could work with upstream to try and get them to
> adopt a configuration that is more convenient for us. We can certainly try
> this route but I have my doubts it will be successful. I have also been told
> that out configuration is insufficient to cover all cases the tests are
> actually covering.

Right, some tests require different hostnames, not just different ports.
Some other tests require subdomain relationships (www.localhost/localhost for instance).
Comment 10 Darin Adler 2017-01-20 15:21:38 PST
I think should add a feature at a low level of WebKit itself that can let us test in this fashion with development builds. It can be something that the test runners turn on and something that we can turn on with some kind of debugging menu. There must be a relatively small number of bottlenecks for URL and host name processing that we could hook into.

I agree that we don’t want to make test running require changing the configuration of the computer unless it is absolutely necessary and I don’t think it is.
Comment 11 Alexey Proskuryakov 2017-02-12 23:32:15 PST
*** Bug 168220 has been marked as a duplicate of this bug. ***
Comment 12 Alexey Proskuryakov 2017-02-12 23:45:45 PST
Simon found another related quirk - in Chrome, subdomains of localhost all resolve to 127.0.0.1. This seems to have started with some weird requirements in <https://tools.ietf.org/html/rfc6761#section-6.3>. Chrome chose to follow this RFC, but no one else appears convinced.

Some history in <https://bugs.chromium.org/p/chromium/issues/detail?id=489973> and <https://tools.ietf.org/html/draft-west-let-localhost-be-localhost>.