Bug 71799 – Extend sunspider driver to be able to run kraken

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Andy Wingo]

Kraken (http://krakenbenchmark.mozilla.org/, http://code.google.com/p/kraken-mirror/) is a JS benchmark that can be browser-hosted, or it can be run standalone.  It uses the sunspider test harness, modified to allow a "data" file to be loaded beforehand.  Loading the data file is not included in the timings.

The patch to be attached allows JSC's sunspider harness to run kraken tests.  You can run it as ./sunspider --shell==..../jsc --suite=/absolute/path/to/kraken-1.1/.

[Andy Wingo]

Created attachment 114042 [details]
Patch

[Andy Wingo]

Does not seem to affect sunspider-1.0 runs:


TEST                   COMPARISON            FROM                 TO             DETAILS

=============================================================================

** TOTAL **:           ??                162.2ms +/- 1.0%   164.5ms +/- 1.5%     not conclusive: might be *1.014x as slow*

=============================================================================

  3d:                  ??                 25.3ms +/- 5.2%    25.9ms +/- 7.7%     not conclusive: might be *1.024x as slow*
    cube:              ??                 10.1ms +/- 12.7%    10.7ms +/- 18.9%     not conclusive: might be *1.059x as slow*
    morph:             -                   7.2ms +/- 4.2%     7.1ms +/- 3.2% 
    raytrace:          ??                  8.0ms +/- 0.0%     8.1ms +/- 2.8%     not conclusive: might be *1.012x as slow*

  access:              -                  21.2ms +/- 2.1%    20.9ms +/- 1.9% 
    binary-trees:      -                   2.0ms +/- 0.0%     2.0ms +/- 0.0% 
    fannkuch:          -                  10.9ms +/- 2.1%    10.8ms +/- 2.8% 
    nbody:             -                   5.3ms +/- 6.5%     5.1ms +/- 4.4% 
    nsieve:            -                   3.0ms +/- 0.0%     3.0ms +/- 0.0% 

  bitops:              ??                 14.3ms +/- 2.4%    14.8ms +/- 3.1%     not conclusive: might be *1.035x as slow*
    3bit-bits-in-byte: -                   2.0ms +/- 0.0%     2.0ms +/- 0.0% 
    bits-in-byte:      -                   5.2ms +/- 5.8%     5.2ms +/- 5.8% 
    bitwise-and:       -                   3.0ms +/- 0.0%     3.0ms +/- 0.0% 
    nsieve-bits:       *1.122x as slow*    4.1ms +/- 5.5%     4.6ms +/- 8.0%     significant

  controlflow:         ??                  1.0ms +/- 0.0%     1.2ms +/- 25.1%     not conclusive: might be *1.20x as slow*
    recursive:         ??                  1.0ms +/- 0.0%     1.2ms +/- 25.1%     not conclusive: might be *1.20x as slow*

  crypto:              -                  10.1ms +/- 2.2%    10.1ms +/- 2.2% 
    aes:               -                   6.0ms +/- 0.0%     6.0ms +/- 0.0% 
    md5:               -                   2.1ms +/- 10.8%     2.1ms +/- 10.8% 
    sha1:              -                   2.0ms +/- 0.0%     2.0ms +/- 0.0% 

  date:                *1.033x as slow*   18.1ms +/- 1.2%    18.7ms +/- 2.6%     significant
    format-tofte:      ??                 10.1ms +/- 2.2%    10.4ms +/- 3.5%     not conclusive: might be *1.030x as slow*
    format-xparb:      ??                  8.0ms +/- 0.0%     8.3ms +/- 4.2%     not conclusive: might be *1.038x as slow*

  math:                *1.037x as slow*   16.1ms +/- 1.4%    16.7ms +/- 2.1%     significant
    cordic:            *1.118x as slow*    5.1ms +/- 4.4%     5.7ms +/- 6.1%     significant
    partial-sums:      -                   8.0ms +/- 0.0%     8.0ms +/- 0.0% 
    spectral-norm:     -                   3.0ms +/- 0.0%     3.0ms +/- 0.0% 

  regexp:              ??                 12.3ms +/- 2.8%    12.5ms +/- 3.0%     not conclusive: might be *1.016x as slow*
    dna:               ??                 12.3ms +/- 2.8%    12.5ms +/- 3.0%     not conclusive: might be *1.016x as slow*

  string:              -                  43.8ms +/- 1.0%    43.7ms +/- 0.8% 
    base64:            -                   4.0ms +/- 0.0%     4.0ms +/- 0.0% 
    fasta:             -                   6.2ms +/- 4.9%     6.2ms +/- 4.9% 
    tagcloud:          -                  11.0ms +/- 0.0%    10.9ms +/- 2.1% 
    unpack-code:       -                  17.6ms +/- 2.1%    17.6ms +/- 2.1% 
    validate-input:    -                   5.0ms +/- 0.0%     5.0ms +/- 0.0%

[Alexey Proskuryakov]

Could you please describe what the rationale for making this change is? What makes running Kraken tests inside Sunspider useful?

[Andy Wingo]

(In reply to comment #3)
> Could you please describe what the rationale for making this change is? What makes running Kraken tests inside Sunspider useful?

Hi Alexey,

The goal is to make it easier to benchmark changes to JSC.  Kraken is a useful and widely-used benchmark, as seen in bug 71686.  Also, SunSpider isn't terrible useful for testing the DFG JIT.  Kraken's sunspider-compare-results is broken, but WebKit's sunspider can't run Kraken, so currently you have to use one tool from one and one from the other.

Eventually, and for the same reasons that the V8 benchmarks can be run with the sunspider harness, I would like to import Kraken's benchmarks into the WebKit repo.  This change to the driver would be necessary.  However it is useful even if we do not import Kraken, and otherwise harmless.

Andy.

[Sam Weinig]

I believe the 'bencher' script is the intended replacement for the sunspider harness, and it can run SunSpider, V8, and Kraken.

Phil, is that correct?

[Andy Wingo]

FWIW, I'm fine with using whatever works.  However, (1) this patch might still be a good idea, and (2) whatever recommended benchmarking procedure is decided on should probably be documented somewhere so other people don't lose so much time.

[Filip Pizlo]

(In reply to comment #6)
> FWIW, I'm fine with using whatever works.  However, (1) this patch might still be a good idea, and (2) whatever recommended benchmarking procedure is decided on should probably be documented somewhere so other people don't lose so much time.

I think that this is still a good idea.

[Filip Pizlo]

(In reply to comment #4)
> (In reply to comment #3)
> > Could you please describe what the rationale for making this change is? What makes running Kraken tests inside Sunspider useful?
> 
> Hi Alexey,
> 
> The goal is to make it easier to benchmark changes to JSC.  Kraken is a useful and widely-used benchmark, as seen in bug 71686.  Also, SunSpider isn't terrible useful for testing the DFG JIT.  Kraken's sunspider-compare-results is broken, but WebKit's sunspider can't run Kraken, so currently you have to use one tool from one and one from the other.
> 
> Eventually, and for the same reasons that the V8 benchmarks can be run with the sunspider harness, I would like to import Kraken's benchmarks into the WebKit repo.  This change to the driver would be necessary.  However it is useful even if we do not import Kraken, and otherwise harmless.
> 
> Andy.

As well, the -data.js feature of Kraken, which this patch adds to our harness, is generally useful.  We might like to use it for any future benchmarks.

[Andy Wingo]

Comment on attachment 114042 [details]
Patch

Setting cq?.

[WebKit Review Bot]

Comment on attachment 114042 [details]
Patch

Clearing flags on attachment: 114042

Committed r100313: <http://trac.webkit.org/changeset/100313>

[WebKit Review Bot]

All reviewed patches have been landed.  Closing bug.