Bug 71799

Summary: Extend sunspider driver to be able to run kraken
Product: WebKit Reporter: Andy Wingo <wingo>
Component: Tools / TestsAssignee: Nobody <webkit-unassigned>
Status: RESOLVED FIXED    
Severity: Normal CC: ap, fpizlo, ggaren, mjs, sam, webkit.review.bot
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Patch none

Description Andy Wingo 2011-11-08 04:55:42 PST
Kraken (http://krakenbenchmark.mozilla.org/, http://code.google.com/p/kraken-mirror/) is a JS benchmark that can be browser-hosted, or it can be run standalone.  It uses the sunspider test harness, modified to allow a "data" file to be loaded beforehand.  Loading the data file is not included in the timings.

The patch to be attached allows JSC's sunspider harness to run kraken tests.  You can run it as ./sunspider --shell==..../jsc --suite=/absolute/path/to/kraken-1.1/.
Comment 1 Andy Wingo 2011-11-08 04:58:06 PST
Created attachment 114042 [details]
Patch
Comment 2 Andy Wingo 2011-11-08 04:59:22 PST
Does not seem to affect sunspider-1.0 runs:


TEST                   COMPARISON            FROM                 TO             DETAILS

=============================================================================

** TOTAL **:           ??                162.2ms +/- 1.0%   164.5ms +/- 1.5%     not conclusive: might be *1.014x as slow*

=============================================================================

  3d:                  ??                 25.3ms +/- 5.2%    25.9ms +/- 7.7%     not conclusive: might be *1.024x as slow*
    cube:              ??                 10.1ms +/- 12.7%    10.7ms +/- 18.9%     not conclusive: might be *1.059x as slow*
    morph:             -                   7.2ms +/- 4.2%     7.1ms +/- 3.2% 
    raytrace:          ??                  8.0ms +/- 0.0%     8.1ms +/- 2.8%     not conclusive: might be *1.012x as slow*

  access:              -                  21.2ms +/- 2.1%    20.9ms +/- 1.9% 
    binary-trees:      -                   2.0ms +/- 0.0%     2.0ms +/- 0.0% 
    fannkuch:          -                  10.9ms +/- 2.1%    10.8ms +/- 2.8% 
    nbody:             -                   5.3ms +/- 6.5%     5.1ms +/- 4.4% 
    nsieve:            -                   3.0ms +/- 0.0%     3.0ms +/- 0.0% 

  bitops:              ??                 14.3ms +/- 2.4%    14.8ms +/- 3.1%     not conclusive: might be *1.035x as slow*
    3bit-bits-in-byte: -                   2.0ms +/- 0.0%     2.0ms +/- 0.0% 
    bits-in-byte:      -                   5.2ms +/- 5.8%     5.2ms +/- 5.8% 
    bitwise-and:       -                   3.0ms +/- 0.0%     3.0ms +/- 0.0% 
    nsieve-bits:       *1.122x as slow*    4.1ms +/- 5.5%     4.6ms +/- 8.0%     significant

  controlflow:         ??                  1.0ms +/- 0.0%     1.2ms +/- 25.1%     not conclusive: might be *1.20x as slow*
    recursive:         ??                  1.0ms +/- 0.0%     1.2ms +/- 25.1%     not conclusive: might be *1.20x as slow*

  crypto:              -                  10.1ms +/- 2.2%    10.1ms +/- 2.2% 
    aes:               -                   6.0ms +/- 0.0%     6.0ms +/- 0.0% 
    md5:               -                   2.1ms +/- 10.8%     2.1ms +/- 10.8% 
    sha1:              -                   2.0ms +/- 0.0%     2.0ms +/- 0.0% 

  date:                *1.033x as slow*   18.1ms +/- 1.2%    18.7ms +/- 2.6%     significant
    format-tofte:      ??                 10.1ms +/- 2.2%    10.4ms +/- 3.5%     not conclusive: might be *1.030x as slow*
    format-xparb:      ??                  8.0ms +/- 0.0%     8.3ms +/- 4.2%     not conclusive: might be *1.038x as slow*

  math:                *1.037x as slow*   16.1ms +/- 1.4%    16.7ms +/- 2.1%     significant
    cordic:            *1.118x as slow*    5.1ms +/- 4.4%     5.7ms +/- 6.1%     significant
    partial-sums:      -                   8.0ms +/- 0.0%     8.0ms +/- 0.0% 
    spectral-norm:     -                   3.0ms +/- 0.0%     3.0ms +/- 0.0% 

  regexp:              ??                 12.3ms +/- 2.8%    12.5ms +/- 3.0%     not conclusive: might be *1.016x as slow*
    dna:               ??                 12.3ms +/- 2.8%    12.5ms +/- 3.0%     not conclusive: might be *1.016x as slow*

  string:              -                  43.8ms +/- 1.0%    43.7ms +/- 0.8% 
    base64:            -                   4.0ms +/- 0.0%     4.0ms +/- 0.0% 
    fasta:             -                   6.2ms +/- 4.9%     6.2ms +/- 4.9% 
    tagcloud:          -                  11.0ms +/- 0.0%    10.9ms +/- 2.1% 
    unpack-code:       -                  17.6ms +/- 2.1%    17.6ms +/- 2.1% 
    validate-input:    -                   5.0ms +/- 0.0%     5.0ms +/- 0.0%
Comment 3 Alexey Proskuryakov 2011-11-08 10:05:37 PST
Could you please describe what the rationale for making this change is? What makes running Kraken tests inside Sunspider useful?
Comment 4 Andy Wingo 2011-11-09 02:07:01 PST
(In reply to comment #3)
> Could you please describe what the rationale for making this change is? What makes running Kraken tests inside Sunspider useful?

Hi Alexey,

The goal is to make it easier to benchmark changes to JSC.  Kraken is a useful and widely-used benchmark, as seen in bug 71686.  Also, SunSpider isn't terrible useful for testing the DFG JIT.  Kraken's sunspider-compare-results is broken, but WebKit's sunspider can't run Kraken, so currently you have to use one tool from one and one from the other.

Eventually, and for the same reasons that the V8 benchmarks can be run with the sunspider harness, I would like to import Kraken's benchmarks into the WebKit repo.  This change to the driver would be necessary.  However it is useful even if we do not import Kraken, and otherwise harmless.

Andy.
Comment 5 Sam Weinig 2011-11-09 09:39:54 PST
I believe the 'bencher' script is the intended replacement for the sunspider harness, and it can run SunSpider, V8, and Kraken.

Phil, is that correct?
Comment 6 Andy Wingo 2011-11-09 10:52:25 PST
FWIW, I'm fine with using whatever works.  However, (1) this patch might still be a good idea, and (2) whatever recommended benchmarking procedure is decided on should probably be documented somewhere so other people don't lose so much time.
Comment 7 Filip Pizlo 2011-11-09 12:35:02 PST
(In reply to comment #6)
> FWIW, I'm fine with using whatever works.  However, (1) this patch might still be a good idea, and (2) whatever recommended benchmarking procedure is decided on should probably be documented somewhere so other people don't lose so much time.

I think that this is still a good idea.
Comment 8 Filip Pizlo 2011-11-09 12:36:08 PST
(In reply to comment #4)
> (In reply to comment #3)
> > Could you please describe what the rationale for making this change is? What makes running Kraken tests inside Sunspider useful?
> 
> Hi Alexey,
> 
> The goal is to make it easier to benchmark changes to JSC.  Kraken is a useful and widely-used benchmark, as seen in bug 71686.  Also, SunSpider isn't terrible useful for testing the DFG JIT.  Kraken's sunspider-compare-results is broken, but WebKit's sunspider can't run Kraken, so currently you have to use one tool from one and one from the other.
> 
> Eventually, and for the same reasons that the V8 benchmarks can be run with the sunspider harness, I would like to import Kraken's benchmarks into the WebKit repo.  This change to the driver would be necessary.  However it is useful even if we do not import Kraken, and otherwise harmless.
> 
> Andy.

As well, the -data.js feature of Kraken, which this patch adds to our harness, is generally useful.  We might like to use it for any future benchmarks.
Comment 9 Andy Wingo 2011-11-15 09:39:28 PST
Comment on attachment 114042 [details]
Patch

Setting cq?.
Comment 10 WebKit Review Bot 2011-11-15 13:04:08 PST
Comment on attachment 114042 [details]
Patch

Clearing flags on attachment: 114042

Committed r100313: <http://trac.webkit.org/changeset/100313>
Comment 11 WebKit Review Bot 2011-11-15 13:04:12 PST
All reviewed patches have been landed.  Closing bug.