Fetching WPT resources is so slow that it makes WPT tests very slow overall. Furthermore, resources aren't cacheable, so the harness keeps being fetched over and over. On my MacBook Pro: $ time curl http://localhost:8800/FileAPI/url/url_xmlhttprequest.html -o /dev/null ... real 0m0.231s $ time curl http://localhost:8000/loading/simple-subframe.html -o /dev/null ... real 0m0.025s $ curl -i http://localhost:8800/FileAPI/url/url_xmlhttprequest.html HTTP/1.1 200 OK Content-Type: text/html Server: BaseHTTP/0.3 Python/2.7.10 Date: Fri, 13 Oct 2017 19:01:37 GMT # where is Last-Modified? ...
We should measure how much WPT tests are slower compared to file system for instance through run-webkit-tests. I did such measurements a year or so ago and it was more like 10/20% slower. If the bottleneck is related to some specific URLS like testharness.js/testharnessreport.js, is there a way for DRT/WTR to intercept/optimize these fetches?
We should bring that to W3C and ask for optimizing WPT server.
The results on my machine (using nsurl instead of curl to be more representative) confirm it is slower but not 10x: ========================================== $ time nscurl http://localhost:8800/FileAPI/url/url_xmlhttprequest.html -o /dev/null real 0m0.170s user 0m0.024s sys 0m0.021s $ time nscurl http://localhost:8800/FileAPI/url/url_xmlhttprequest.html -o /dev/null real 0m0.160s user 0m0.022s sys 0m0.018s $ time nscurl http://localhost:8800/FileAPI/url/url_xmlhttprequest.html -o /dev/null real 0m0.162s user 0m0.022s sys 0m0.019s real AVG: 0m0.164 ========================================== $ time nscurl http://localhost:8000/loading/simple-subframe.html -o /dev/null real 0m0.068s user 0m0.021s sys 0m0.017s $ time nscurl http://localhost:8000/loading/simple-subframe.html -o /dev/null real 0m0.058s user 0m0.021s sys 0m0.018s $ time nscurl http://localhost:8000/loading/simple-subframe.html -o /dev/null real 0m0.055s user 0m0.020s sys 0m0.017s real AVG: 0m0.060s (2.7x faster)
Headers are pretty different and I agree this is interesting: HTTP/1.1 200 OK Content-Type: text/html Server: BaseHTTP/0.3 Python/2.7.10 Date: Fri, 13 Oct 2017 20:44:48 GMT vs HTTP/1.1 200 OK Date: Fri, 13 Oct 2017 20:45:07 GMT Server: Apache/2.4.27 (Unix) PHP/7.1.7 LibreSSL/2.2.7 Last-Modified: Fri, 10 Feb 2017 20:11:59 GMT ETag: "96-54832b5c981c0" Accept-Ranges: bytes Content-Length: 150 Content-Type: text/html
Created attachment 327431 [details] IDB sub test list Tests used to measure Apache/WPT performance differences
I rerun the test list (in attachment) through WPT and Apache. I observe a 20% slowdown with WPT when using run-webkit-tests. When using run-webkit-tests with -f option, there is no observable difference in time needed to run the test suite. This might mean the WPT lack of speed might not have a huge impact on running WPT tests. This should be completed with other tests, typically smaller tests where the difference might still be noticeable.
Created attachment 327432 [details] Patch to apply to WPT server to have Content-Length set more often
I just tried the patch to compute Content-Length for WPT served files. I no longer observe any speed difference between Apache and WPT.
> The results on my machine (using nsurl instead of curl to be more representative) confirm it is slower but not 10x: The results seem to depend on the state of the system. Right now, there is virtually no difference between Apache and WPT on my machine, both take ~35 ms per resource. I'm not sure offhand what could be causing such dramatic changes.
(In reply to Alexey Proskuryakov from comment #9) > > The results on my machine (using nsurl instead of curl to be more representative) confirm it is slower but not 10x: > > The results seem to depend on the state of the system. Right now, there is > virtually no difference between Apache and WPT on my machine, both take ~35 > ms per resource. I'm not sure offhand what could be causing such dramatic > changes. There was this issue about TIME_WAIT (https://bugs.webkit.org/show_bug.cgi?id=164822). Could it be that it would slow down things? Maybe run some wpt tests on a running WPT server and shortly after do the nsurl check, or repetitively do the nsurl check.
(In reply to youenn fablet from comment #8) > I just tried the patch to compute Content-Length for WPT served files. > I no longer observe any speed difference between Apache and WPT. Without the patch, the 25% difference is still showing in my MacBook. I do not see any difference at all on my iMac (different OS versions though).
On a Mac Studio (2022), running macOS 14.4: ``` gsnedders@gsnedders-marsha WebKit % cp LayoutTests/http/tests/loading/simple-subframe.html LayoutTests/http/wpt/loading/simple-subframe.html gsnedders@gsnedders-marsha WebKit % hyperfine -w 3 -m 250 'nscurl http://localhost:8800/loading/simple-subframe.html -o /dev/null' 'nscurl http://localhost:8000/loading/simple-subframe.html -o /dev/null' Benchmark 1: nscurl http://localhost:8800/loading/simple-subframe.html -o /dev/null Time (mean ± σ): 111.5 ms ± 28.9 ms [User: 10.9 ms, System: 11.2 ms] Range (min … max): 46.2 ms … 165.4 ms 250 runs Benchmark 2: nscurl http://localhost:8000/loading/simple-subframe.html -o /dev/null Time (mean ± σ): 117.8 ms ± 26.1 ms [User: 11.3 ms, System: 11.7 ms] Range (min … max): 48.7 ms … 181.3 ms 250 runs Summary nscurl http://localhost:8800/loading/simple-subframe.html -o /dev/null ran 1.06 ± 0.36 times faster than nscurl http://localhost:8000/loading/simple-subframe.html -o /dev/null ``` …which tends to suggest it's a wash at this point, and largely dominated by noise, at least with a single request being made at a time? Even what one might expect to be the case that Apache is better tuned for, with concurrent connections, comes back with almost no difference, again dominated by noise: ``` gsnedders@gsnedders-marsha OpenSource % hyperfine -w1 -m 30 'parallel -N0 nscurl http://localhost:8800/loading/simple-subframe.html -o /dev/null ::: {1..30}' 'parallel -N0 nscurl http://localhost:8000/loading/simple-subframe.html -o /dev/null ::: {1..30}' Benchmark 1: parallel -N0 nscurl http://localhost:8800/loading/simple-subframe.html -o /dev/null ::: {1..30} Time (mean ± σ): 2.166 s ± 0.027 s [User: 2.525 s, System: 2.099 s] Range (min … max): 2.131 s … 2.255 s 30 runs Benchmark 2: parallel -N0 nscurl http://localhost:8000/loading/simple-subframe.html -o /dev/null ::: {1..30} Time (mean ± σ): 2.159 s ± 0.026 s [User: 2.525 s, System: 2.089 s] Range (min … max): 2.116 s … 2.233 s 30 runs Summary parallel -N0 nscurl http://localhost:8000/loading/simple-subframe.html -o /dev/null ::: {1..30} ran 1.00 ± 0.02 times faster than parallel -N0 nscurl http://localhost:8800/loading/simple-subframe.html -o /dev/null ::: {1..30} ``` I suggest we close this and the upstream issue at this point?