Bug 20616 – Incorporate V8 benchmarks in testing

• Status: RESOLVED
• Resolution: FIXED
• Component: JavaScriptCore

[Gavin Sherlock]

V8 has a benchmark suite that would be good to include in speed regression testing for JavaScript.  Looks like V8 beats squirrelfish by some margin, at least on Windows at the moment.  (It also beats squirrelfish on some of the Sunspider benchmarks too : http://kourge.net/node/122).

[Cameron Zwarich (cpst)]

I was thinking of making a SunSpider test suite from the V8 benchmarks so we can use the SunSpider driver, which gives more meaningful information.

[Cameron Zwarich (cpst)]

Created attachment 23159 [details]
Proposed patch

[Cameron Zwarich (cpst)]

Here are results on my machine with r36082:

============================================
RESULTS (means and 95% confidence intervals)
--------------------------------------------
Total:             578.3ms +/- 0.6%
--------------------------------------------

  v8:              578.3ms +/- 0.6%
    crypto:        172.3ms +/- 0.5%
    deltablue:      22.6ms +/- 1.6%
    earley-boyer:   74.7ms +/- 0.5%
    raytrace:      300.0ms +/- 0.8%
    richards:        8.7ms +/- 4.0%

This is too wide of a spread, so Maciej and I agreed we should increase the number of iterations in some uniform way. His suggestion was to make them all take some fixed amount of time in V8, since the benchmarks were chosen to highlight V8's strengths. I'll do that later tonight or tomorrow, but I'll clear the review request for now.

[Cameron Zwarich (cpst)]

Created attachment 23165 [details]
Revised proposed patch

I calibrated the benchmarks so that they all run in a roughly equal time with V8. Also, I did not make a separate subdirectory for the v8 benchmarks like the ubench benchmarks. Since there are no categories in the V8 benchmarks, the results look slightly stupid if they are in their own directory.

[Cameron Zwarich (cpst)]

Landed in r36095.