Kraken (http://krakenbenchmark.mozilla.org/, http://code.google.com/p/kraken-mirror/) is a JS benchmark that can be browser-hosted, or it can be run standalone. It uses the sunspider test harness, modified to allow a "data" file to be loaded beforehand. Loading the data file is not included in the timings. The patch to be attached allows JSC's sunspider harness to run kraken tests. You can run it as ./sunspider --shell==..../jsc --suite=/absolute/path/to/kraken-1.1/.
Created attachment 114042 [details] Patch
Does not seem to affect sunspider-1.0 runs: TEST COMPARISON FROM TO DETAILS ============================================================================= ** TOTAL **: ?? 162.2ms +/- 1.0% 164.5ms +/- 1.5% not conclusive: might be *1.014x as slow* ============================================================================= 3d: ?? 25.3ms +/- 5.2% 25.9ms +/- 7.7% not conclusive: might be *1.024x as slow* cube: ?? 10.1ms +/- 12.7% 10.7ms +/- 18.9% not conclusive: might be *1.059x as slow* morph: - 7.2ms +/- 4.2% 7.1ms +/- 3.2% raytrace: ?? 8.0ms +/- 0.0% 8.1ms +/- 2.8% not conclusive: might be *1.012x as slow* access: - 21.2ms +/- 2.1% 20.9ms +/- 1.9% binary-trees: - 2.0ms +/- 0.0% 2.0ms +/- 0.0% fannkuch: - 10.9ms +/- 2.1% 10.8ms +/- 2.8% nbody: - 5.3ms +/- 6.5% 5.1ms +/- 4.4% nsieve: - 3.0ms +/- 0.0% 3.0ms +/- 0.0% bitops: ?? 14.3ms +/- 2.4% 14.8ms +/- 3.1% not conclusive: might be *1.035x as slow* 3bit-bits-in-byte: - 2.0ms +/- 0.0% 2.0ms +/- 0.0% bits-in-byte: - 5.2ms +/- 5.8% 5.2ms +/- 5.8% bitwise-and: - 3.0ms +/- 0.0% 3.0ms +/- 0.0% nsieve-bits: *1.122x as slow* 4.1ms +/- 5.5% 4.6ms +/- 8.0% significant controlflow: ?? 1.0ms +/- 0.0% 1.2ms +/- 25.1% not conclusive: might be *1.20x as slow* recursive: ?? 1.0ms +/- 0.0% 1.2ms +/- 25.1% not conclusive: might be *1.20x as slow* crypto: - 10.1ms +/- 2.2% 10.1ms +/- 2.2% aes: - 6.0ms +/- 0.0% 6.0ms +/- 0.0% md5: - 2.1ms +/- 10.8% 2.1ms +/- 10.8% sha1: - 2.0ms +/- 0.0% 2.0ms +/- 0.0% date: *1.033x as slow* 18.1ms +/- 1.2% 18.7ms +/- 2.6% significant format-tofte: ?? 10.1ms +/- 2.2% 10.4ms +/- 3.5% not conclusive: might be *1.030x as slow* format-xparb: ?? 8.0ms +/- 0.0% 8.3ms +/- 4.2% not conclusive: might be *1.038x as slow* math: *1.037x as slow* 16.1ms +/- 1.4% 16.7ms +/- 2.1% significant cordic: *1.118x as slow* 5.1ms +/- 4.4% 5.7ms +/- 6.1% significant partial-sums: - 8.0ms +/- 0.0% 8.0ms +/- 0.0% spectral-norm: - 3.0ms +/- 0.0% 3.0ms +/- 0.0% regexp: ?? 12.3ms +/- 2.8% 12.5ms +/- 3.0% not conclusive: might be *1.016x as slow* dna: ?? 12.3ms +/- 2.8% 12.5ms +/- 3.0% not conclusive: might be *1.016x as slow* string: - 43.8ms +/- 1.0% 43.7ms +/- 0.8% base64: - 4.0ms +/- 0.0% 4.0ms +/- 0.0% fasta: - 6.2ms +/- 4.9% 6.2ms +/- 4.9% tagcloud: - 11.0ms +/- 0.0% 10.9ms +/- 2.1% unpack-code: - 17.6ms +/- 2.1% 17.6ms +/- 2.1% validate-input: - 5.0ms +/- 0.0% 5.0ms +/- 0.0%
Could you please describe what the rationale for making this change is? What makes running Kraken tests inside Sunspider useful?
(In reply to comment #3) > Could you please describe what the rationale for making this change is? What makes running Kraken tests inside Sunspider useful? Hi Alexey, The goal is to make it easier to benchmark changes to JSC. Kraken is a useful and widely-used benchmark, as seen in bug 71686. Also, SunSpider isn't terrible useful for testing the DFG JIT. Kraken's sunspider-compare-results is broken, but WebKit's sunspider can't run Kraken, so currently you have to use one tool from one and one from the other. Eventually, and for the same reasons that the V8 benchmarks can be run with the sunspider harness, I would like to import Kraken's benchmarks into the WebKit repo. This change to the driver would be necessary. However it is useful even if we do not import Kraken, and otherwise harmless. Andy.
I believe the 'bencher' script is the intended replacement for the sunspider harness, and it can run SunSpider, V8, and Kraken. Phil, is that correct?
FWIW, I'm fine with using whatever works. However, (1) this patch might still be a good idea, and (2) whatever recommended benchmarking procedure is decided on should probably be documented somewhere so other people don't lose so much time.
(In reply to comment #6) > FWIW, I'm fine with using whatever works. However, (1) this patch might still be a good idea, and (2) whatever recommended benchmarking procedure is decided on should probably be documented somewhere so other people don't lose so much time. I think that this is still a good idea.
(In reply to comment #4) > (In reply to comment #3) > > Could you please describe what the rationale for making this change is? What makes running Kraken tests inside Sunspider useful? > > Hi Alexey, > > The goal is to make it easier to benchmark changes to JSC. Kraken is a useful and widely-used benchmark, as seen in bug 71686. Also, SunSpider isn't terrible useful for testing the DFG JIT. Kraken's sunspider-compare-results is broken, but WebKit's sunspider can't run Kraken, so currently you have to use one tool from one and one from the other. > > Eventually, and for the same reasons that the V8 benchmarks can be run with the sunspider harness, I would like to import Kraken's benchmarks into the WebKit repo. This change to the driver would be necessary. However it is useful even if we do not import Kraken, and otherwise harmless. > > Andy. As well, the -data.js feature of Kraken, which this patch adds to our harness, is generally useful. We might like to use it for any future benchmarks.
Comment on attachment 114042 [details] Patch Setting cq?.
Comment on attachment 114042 [details] Patch Clearing flags on attachment: 114042 Committed r100313: <http://trac.webkit.org/changeset/100313>
All reviewed patches have been landed. Closing bug.