Bug 178277 – WPT server is 10x slower than Apache

• Status: NEW
• Component: Tools / Tests

[Alexey Proskuryakov]

Fetching WPT resources is so slow that it makes WPT tests very slow overall. Furthermore, resources aren't cacheable, so the harness keeps being fetched over and over.

On my MacBook Pro:

$ time curl http://localhost:8800/FileAPI/url/url_xmlhttprequest.html -o /dev/null
...
real	0m0.231s

$ time curl http://localhost:8000/loading/simple-subframe.html -o /dev/null
...
real	0m0.025s

$ curl -i  http://localhost:8800/FileAPI/url/url_xmlhttprequest.html
HTTP/1.1 200 OK
Content-Type: text/html
Server: BaseHTTP/0.3 Python/2.7.10
Date: Fri, 13 Oct 2017 19:01:37 GMT
# where is Last-Modified?
...

[youenn fablet]

We should measure how much WPT tests are slower compared to file system for instance through run-webkit-tests.
I did such measurements a year or so ago and it was more like 10/20% slower.

If the bottleneck is related to some specific URLS like testharness.js/testharnessreport.js, is there a way for DRT/WTR to intercept/optimize these fetches?

[youenn fablet]

We should bring that to W3C and ask for optimizing WPT server.

[Chris Dumez]

The results on my machine (using nsurl instead of curl to be more representative) confirm it is slower but not 10x:


==========================================

$ time nscurl http://localhost:8800/FileAPI/url/url_xmlhttprequest.html -o /dev/null

real	0m0.170s
user	0m0.024s
sys	0m0.021s

$ time nscurl http://localhost:8800/FileAPI/url/url_xmlhttprequest.html -o /dev/null

real	0m0.160s
user	0m0.022s
sys	0m0.018s

$ time nscurl http://localhost:8800/FileAPI/url/url_xmlhttprequest.html -o /dev/null

real	0m0.162s
user	0m0.022s
sys	0m0.019s

real AVG: 0m0.164

==========================================

$ time nscurl http://localhost:8000/loading/simple-subframe.html -o /dev/null

real	0m0.068s
user	0m0.021s
sys	0m0.017s

$ time nscurl http://localhost:8000/loading/simple-subframe.html -o /dev/null

real	0m0.058s
user	0m0.021s
sys	0m0.018s

$ time nscurl http://localhost:8000/loading/simple-subframe.html -o /dev/null

real	0m0.055s
user	0m0.020s
sys	0m0.017s

real AVG: 0m0.060s (2.7x faster)

[Chris Dumez]

Headers are pretty different and I agree this is interesting:
HTTP/1.1 200 OK
Content-Type: text/html
Server: BaseHTTP/0.3 Python/2.7.10
Date: Fri, 13 Oct 2017 20:44:48 GMT

vs

HTTP/1.1 200 OK
Date: Fri, 13 Oct 2017 20:45:07 GMT
Server: Apache/2.4.27 (Unix) PHP/7.1.7 LibreSSL/2.2.7
Last-Modified: Fri, 10 Feb 2017 20:11:59 GMT
ETag: "96-54832b5c981c0"
Accept-Ranges: bytes
Content-Length: 150
Content-Type: text/html

[youenn fablet]

Created attachment 327431 [details]
IDB sub test list

Tests used to measure Apache/WPT performance differences

[youenn fablet]

I rerun the test list (in attachment) through WPT and Apache.
I observe a 20% slowdown with WPT when using run-webkit-tests.
When using run-webkit-tests with -f option, there is no observable difference in time needed to run the test suite.

This might mean the WPT lack of speed might not have a huge impact on running WPT tests. This should be completed with other tests, typically smaller tests where the difference might still be noticeable.

[youenn fablet]

Created attachment 327432 [details]
Patch to apply to WPT server to have Content-Length set more often

[youenn fablet]

I just tried the patch to compute Content-Length for WPT served files.
I no longer observe any speed difference between Apache and WPT.

[Alexey Proskuryakov]

> The results on my machine (using nsurl instead of curl to be more representative) confirm it is slower but not 10x:

The results seem to depend on the state of the system. Right now, there is virtually no difference between Apache and WPT on my machine, both take ~35 ms per resource. I'm not sure offhand what could be causing such dramatic changes.

[youenn fablet]

(In reply to Alexey Proskuryakov from comment #9)
> > The results on my machine (using nsurl instead of curl to be more representative) confirm it is slower but not 10x:
> 
> The results seem to depend on the state of the system. Right now, there is
> virtually no difference between Apache and WPT on my machine, both take ~35
> ms per resource. I'm not sure offhand what could be causing such dramatic
> changes.

There was this issue about TIME_WAIT (https://bugs.webkit.org/show_bug.cgi?id=164822).
Could it be that it would slow down things?
Maybe run some wpt tests on a running WPT server and shortly after do the nsurl check, or repetitively do the nsurl check.

[youenn fablet]

(In reply to youenn fablet from comment #8)
> I just tried the patch to compute Content-Length for WPT served files.
> I no longer observe any speed difference between Apache and WPT.

Without the patch, the 25% difference is still showing in my MacBook.
I do not see any difference at all on my iMac (different OS versions though).

[Sam Sneddon [:gsnedders]]

On a Mac Studio (2022), running macOS 14.4:

```
gsnedders@gsnedders-marsha WebKit % cp LayoutTests/http/tests/loading/simple-subframe.html LayoutTests/http/wpt/loading/simple-subframe.html
gsnedders@gsnedders-marsha WebKit % hyperfine -w 3 -m 250 'nscurl http://localhost:8800/loading/simple-subframe.html -o /dev/null' 'nscurl http://localhost:8000/loading/simple-subframe.html -o /dev/null'
Benchmark 1: nscurl http://localhost:8800/loading/simple-subframe.html -o /dev/null
  Time (mean ± σ):     111.5 ms ±  28.9 ms    [User: 10.9 ms, System: 11.2 ms]
  Range (min … max):    46.2 ms … 165.4 ms    250 runs
 
Benchmark 2: nscurl http://localhost:8000/loading/simple-subframe.html -o /dev/null
  Time (mean ± σ):     117.8 ms ±  26.1 ms    [User: 11.3 ms, System: 11.7 ms]
  Range (min … max):    48.7 ms … 181.3 ms    250 runs
 
Summary
  nscurl http://localhost:8800/loading/simple-subframe.html -o /dev/null ran
    1.06 ± 0.36 times faster than nscurl http://localhost:8000/loading/simple-subframe.html -o /dev/null
```

…which tends to suggest it's a wash at this point, and largely dominated by noise, at least with a single request being made at a time?

Even what one might expect to be the case that Apache is better tuned for, with concurrent connections, comes back with almost no difference, again dominated by noise:

```
gsnedders@gsnedders-marsha OpenSource % hyperfine -w1 -m 30 'parallel -N0 nscurl http://localhost:8800/loading/simple-subframe.html -o /dev/null ::: {1..30}' 'parallel -N0 nscurl http://localhost:8000/loading/simple-subframe.html -o /dev/null ::: {1..30}'
Benchmark 1: parallel -N0 nscurl http://localhost:8800/loading/simple-subframe.html -o /dev/null ::: {1..30}
  Time (mean ± σ):      2.166 s ±  0.027 s    [User: 2.525 s, System: 2.099 s]
  Range (min … max):    2.131 s …  2.255 s    30 runs
 
Benchmark 2: parallel -N0 nscurl http://localhost:8000/loading/simple-subframe.html -o /dev/null ::: {1..30}
  Time (mean ± σ):      2.159 s ±  0.026 s    [User: 2.525 s, System: 2.089 s]
  Range (min … max):    2.116 s …  2.233 s    30 runs
 
Summary
  parallel -N0 nscurl http://localhost:8000/loading/simple-subframe.html -o /dev/null ::: {1..30} ran
    1.00 ± 0.02 times faster than parallel -N0 nscurl http://localhost:8800/loading/simple-subframe.html -o /dev/null ::: {1..30}
```

I suggest we close this and the upstream issue at this point?