Bug 20616

Summary: Incorporate V8 benchmarks in testing
Product: WebKit Reporter: Gavin Sherlock <gsherloc>
Component: JavaScriptCoreAssignee: Cameron Zwarich (cpst) <zwarich>
Severity: Normal CC: zwarich
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: All   
OS: OS X 10.5   
URL: http://code.google.com/apis/v8/run.html
Description Flags
Proposed patch
Revised proposed patch oliver: review+

Description Gavin Sherlock 2008-09-03 00:55:23 PDT
V8 has a benchmark suite that would be good to include in speed regression testing for JavaScript.  Looks like V8 beats squirrelfish by some margin, at least on Windows at the moment.  (It also beats squirrelfish on some of the Sunspider benchmarks too : http://kourge.net/node/122).
Comment 1 Cameron Zwarich (cpst) 2008-09-04 01:03:35 PDT
I was thinking of making a SunSpider test suite from the V8 benchmarks so we can use the SunSpider driver, which gives more meaningful information.
Comment 2 Cameron Zwarich (cpst) 2008-09-04 01:40:51 PDT
Created attachment 23159 [details]
Proposed patch
Comment 3 Cameron Zwarich (cpst) 2008-09-04 01:57:39 PDT
Here are results on my machine with r36082:

RESULTS (means and 95% confidence intervals)
Total:             578.3ms +/- 0.6%

  v8:              578.3ms +/- 0.6%
    crypto:        172.3ms +/- 0.5%
    deltablue:      22.6ms +/- 1.6%
    earley-boyer:   74.7ms +/- 0.5%
    raytrace:      300.0ms +/- 0.8%
    richards:        8.7ms +/- 4.0%

This is too wide of a spread, so Maciej and I agreed we should increase the number of iterations in some uniform way. His suggestion was to make them all take some fixed amount of time in V8, since the benchmarks were chosen to highlight V8's strengths. I'll do that later tonight or tomorrow, but I'll clear the review request for now.
Comment 4 Cameron Zwarich (cpst) 2008-09-04 04:24:29 PDT
Created attachment 23165 [details]
Revised proposed patch

I calibrated the benchmarks so that they all run in a roughly equal time with V8. Also, I did not make a separate subdirectory for the v8 benchmarks like the ubench benchmarks. Since there are no categories in the V8 benchmarks, the results look slightly stupid if they are in their own directory.
Comment 5 Cameron Zwarich (cpst) 2008-09-04 11:09:44 PDT
Landed in r36095.