Bug 20616 - Incorporate V8 benchmarks in testing
Summary: Incorporate V8 benchmarks in testing
Alias: None
Product: WebKit
Classification: Unclassified
Component: JavaScriptCore (show other bugs)
Version: 528+ (Nightly build)
Hardware: All OS X 10.5
: P2 Normal
Assignee: Cameron Zwarich (cpst)
URL: http://code.google.com/apis/v8/run.html
Depends on:
Reported: 2008-09-03 00:55 PDT by Gavin Sherlock
Modified: 2008-09-04 11:09 PDT (History)
1 user (show)

See Also:

Proposed patch (390.48 KB, patch)
2008-09-04 01:40 PDT, Cameron Zwarich (cpst)
no flags Details | Formatted Diff | Diff
Revised proposed patch (391.08 KB, patch)
2008-09-04 04:24 PDT, Cameron Zwarich (cpst)
oliver: review+
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Gavin Sherlock 2008-09-03 00:55:23 PDT
V8 has a benchmark suite that would be good to include in speed regression testing for JavaScript.  Looks like V8 beats squirrelfish by some margin, at least on Windows at the moment.  (It also beats squirrelfish on some of the Sunspider benchmarks too : http://kourge.net/node/122).
Comment 1 Cameron Zwarich (cpst) 2008-09-04 01:03:35 PDT
I was thinking of making a SunSpider test suite from the V8 benchmarks so we can use the SunSpider driver, which gives more meaningful information.
Comment 2 Cameron Zwarich (cpst) 2008-09-04 01:40:51 PDT
Created attachment 23159 [details]
Proposed patch
Comment 3 Cameron Zwarich (cpst) 2008-09-04 01:57:39 PDT
Here are results on my machine with r36082:

RESULTS (means and 95% confidence intervals)
Total:             578.3ms +/- 0.6%

  v8:              578.3ms +/- 0.6%
    crypto:        172.3ms +/- 0.5%
    deltablue:      22.6ms +/- 1.6%
    earley-boyer:   74.7ms +/- 0.5%
    raytrace:      300.0ms +/- 0.8%
    richards:        8.7ms +/- 4.0%

This is too wide of a spread, so Maciej and I agreed we should increase the number of iterations in some uniform way. His suggestion was to make them all take some fixed amount of time in V8, since the benchmarks were chosen to highlight V8's strengths. I'll do that later tonight or tomorrow, but I'll clear the review request for now.
Comment 4 Cameron Zwarich (cpst) 2008-09-04 04:24:29 PDT
Created attachment 23165 [details]
Revised proposed patch

I calibrated the benchmarks so that they all run in a roughly equal time with V8. Also, I did not make a separate subdirectory for the v8 benchmarks like the ubench benchmarks. Since there are no categories in the V8 benchmarks, the results look slightly stupid if they are in their own directory.
Comment 5 Cameron Zwarich (cpst) 2008-09-04 11:09:44 PDT
Landed in r36095.