71799 2011-11-08 04:55:42 -0800 Extend sunspider driver to be able to run kraken 2011-11-15 13:04:12 -0800 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified RESOLVED FIXED P2 Normal --- 0 wingo webkit-unassigned ap fpizlo ggaren mjs sam webkit.review.bot oldest_to_newest 497953 0 wingo 2011-11-08 04:55:42 -0800 Kraken (http://krakenbenchmark.mozilla.org/, http://code.google.com/p/kraken-mirror/) is a JS benchmark that can be browser-hosted, or it can be run standalone. It uses the sunspider test harness, modified to allow a "data" file to be loaded beforehand. Loading the data file is not included in the timings. The patch to be attached allows JSC's sunspider harness to run kraken tests. You can run it as ./sunspider --shell==..../jsc --suite=/absolute/path/to/kraken-1.1/. 497955 1 114042 wingo 2011-11-08 04:58:06 -0800 Created attachment 114042 Patch 497956 2 wingo 2011-11-08 04:59:22 -0800 Does not seem to affect sunspider-1.0 runs: TEST COMPARISON FROM TO DETAILS ============================================================================= ** TOTAL **: ?? 162.2ms +/- 1.0% 164.5ms +/- 1.5% not conclusive: might be *1.014x as slow* ============================================================================= 3d: ?? 25.3ms +/- 5.2% 25.9ms +/- 7.7% not conclusive: might be *1.024x as slow* cube: ?? 10.1ms +/- 12.7% 10.7ms +/- 18.9% not conclusive: might be *1.059x as slow* morph: - 7.2ms +/- 4.2% 7.1ms +/- 3.2% raytrace: ?? 8.0ms +/- 0.0% 8.1ms +/- 2.8% not conclusive: might be *1.012x as slow* access: - 21.2ms +/- 2.1% 20.9ms +/- 1.9% binary-trees: - 2.0ms +/- 0.0% 2.0ms +/- 0.0% fannkuch: - 10.9ms +/- 2.1% 10.8ms +/- 2.8% nbody: - 5.3ms +/- 6.5% 5.1ms +/- 4.4% nsieve: - 3.0ms +/- 0.0% 3.0ms +/- 0.0% bitops: ?? 14.3ms +/- 2.4% 14.8ms +/- 3.1% not conclusive: might be *1.035x as slow* 3bit-bits-in-byte: - 2.0ms +/- 0.0% 2.0ms +/- 0.0% bits-in-byte: - 5.2ms +/- 5.8% 5.2ms +/- 5.8% bitwise-and: - 3.0ms +/- 0.0% 3.0ms +/- 0.0% nsieve-bits: *1.122x as slow* 4.1ms +/- 5.5% 4.6ms +/- 8.0% significant controlflow: ?? 1.0ms +/- 0.0% 1.2ms +/- 25.1% not conclusive: might be *1.20x as slow* recursive: ?? 1.0ms +/- 0.0% 1.2ms +/- 25.1% not conclusive: might be *1.20x as slow* crypto: - 10.1ms +/- 2.2% 10.1ms +/- 2.2% aes: - 6.0ms +/- 0.0% 6.0ms +/- 0.0% md5: - 2.1ms +/- 10.8% 2.1ms +/- 10.8% sha1: - 2.0ms +/- 0.0% 2.0ms +/- 0.0% date: *1.033x as slow* 18.1ms +/- 1.2% 18.7ms +/- 2.6% significant format-tofte: ?? 10.1ms +/- 2.2% 10.4ms +/- 3.5% not conclusive: might be *1.030x as slow* format-xparb: ?? 8.0ms +/- 0.0% 8.3ms +/- 4.2% not conclusive: might be *1.038x as slow* math: *1.037x as slow* 16.1ms +/- 1.4% 16.7ms +/- 2.1% significant cordic: *1.118x as slow* 5.1ms +/- 4.4% 5.7ms +/- 6.1% significant partial-sums: - 8.0ms +/- 0.0% 8.0ms +/- 0.0% spectral-norm: - 3.0ms +/- 0.0% 3.0ms +/- 0.0% regexp: ?? 12.3ms +/- 2.8% 12.5ms +/- 3.0% not conclusive: might be *1.016x as slow* dna: ?? 12.3ms +/- 2.8% 12.5ms +/- 3.0% not conclusive: might be *1.016x as slow* string: - 43.8ms +/- 1.0% 43.7ms +/- 0.8% base64: - 4.0ms +/- 0.0% 4.0ms +/- 0.0% fasta: - 6.2ms +/- 4.9% 6.2ms +/- 4.9% tagcloud: - 11.0ms +/- 0.0% 10.9ms +/- 2.1% unpack-code: - 17.6ms +/- 2.1% 17.6ms +/- 2.1% validate-input: - 5.0ms +/- 0.0% 5.0ms +/- 0.0% 498182 3 ap 2011-11-08 10:05:37 -0800 Could you please describe what the rationale for making this change is? What makes running Kraken tests inside Sunspider useful? 498741 4 wingo 2011-11-09 02:07:01 -0800 (In reply to comment #3) > Could you please describe what the rationale for making this change is? What makes running Kraken tests inside Sunspider useful? Hi Alexey, The goal is to make it easier to benchmark changes to JSC. Kraken is a useful and widely-used benchmark, as seen in bug 71686. Also, SunSpider isn't terrible useful for testing the DFG JIT. Kraken's sunspider-compare-results is broken, but WebKit's sunspider can't run Kraken, so currently you have to use one tool from one and one from the other. Eventually, and for the same reasons that the V8 benchmarks can be run with the sunspider harness, I would like to import Kraken's benchmarks into the WebKit repo. This change to the driver would be necessary. However it is useful even if we do not import Kraken, and otherwise harmless. Andy. 499076 5 sam 2011-11-09 09:39:54 -0800 I believe the 'bencher' script is the intended replacement for the sunspider harness, and it can run SunSpider, V8, and Kraken. Phil, is that correct? 499138 6 wingo 2011-11-09 10:52:25 -0800 FWIW, I'm fine with using whatever works. However, (1) this patch might still be a good idea, and (2) whatever recommended benchmarking procedure is decided on should probably be documented somewhere so other people don't lose so much time. 499258 7 fpizlo 2011-11-09 12:35:02 -0800 (In reply to comment #6) > FWIW, I'm fine with using whatever works. However, (1) this patch might still be a good idea, and (2) whatever recommended benchmarking procedure is decided on should probably be documented somewhere so other people don't lose so much time. I think that this is still a good idea. 499262 8 fpizlo 2011-11-09 12:36:08 -0800 (In reply to comment #4) > (In reply to comment #3) > > Could you please describe what the rationale for making this change is? What makes running Kraken tests inside Sunspider useful? > > Hi Alexey, > > The goal is to make it easier to benchmark changes to JSC. Kraken is a useful and widely-used benchmark, as seen in bug 71686. Also, SunSpider isn't terrible useful for testing the DFG JIT. Kraken's sunspider-compare-results is broken, but WebKit's sunspider can't run Kraken, so currently you have to use one tool from one and one from the other. > > Eventually, and for the same reasons that the V8 benchmarks can be run with the sunspider harness, I would like to import Kraken's benchmarks into the WebKit repo. This change to the driver would be necessary. However it is useful even if we do not import Kraken, and otherwise harmless. > > Andy. As well, the -data.js feature of Kraken, which this patch adds to our harness, is generally useful. We might like to use it for any future benchmarks. 502974 9 114042 wingo 2011-11-15 09:39:28 -0800 Comment on attachment 114042 Patch Setting cq?. 503178 10 114042 webkit.review.bot 2011-11-15 13:04:08 -0800 Comment on attachment 114042 Patch Clearing flags on attachment: 114042 Committed r100313: <http://trac.webkit.org/changeset/100313> 503179 11 webkit.review.bot 2011-11-15 13:04:12 -0800 All reviewed patches have been landed. Closing bug. 114042 2011-11-08 04:58:06 -0800 2011-11-15 13:04:07 -0800 Patch bug-71799-20111108135804.patch text/plain 2411 wingo U3VidmVyc2lvbiBSZXZpc2lvbjogOTk1NDAKZGlmZiAtLWdpdCBhL1BlcmZvcm1hbmNlVGVzdHMv U3VuU3BpZGVyL0NoYW5nZUxvZyBiL1BlcmZvcm1hbmNlVGVzdHMvU3VuU3BpZGVyL0NoYW5nZUxv ZwppbmRleCBhY2U2NjVmNjUxOTdlZTE2YjIzYTkyZmVkZWUzYzhkYTQ5MzFmOTBjLi4yNzMyYjIy ZTY3MjNhODEyYzc5MGIzZDBlMGRhODNjNmVmODY0YTk4IDEwMDY0NAotLS0gYS9QZXJmb3JtYW5j ZVRlc3RzL1N1blNwaWRlci9DaGFuZ2VMb2cKKysrIGIvUGVyZm9ybWFuY2VUZXN0cy9TdW5TcGlk ZXIvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsMTQgQEAKKzIwMTEtMTEtMDggIEFuZHkgV2luZ28gIDx3 aW5nb0BpZ2FsaWEuY29tPgorCisgICAgICAgIEV4dGVuZCBzdW5zcGlkZXIgZHJpdmVyIHRvIGJl IGFibGUgdG8gcnVuIGtyYWtlbgorICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93 X2J1Zy5jZ2k/aWQ9NzE3OTkKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4K KworICAgICAgICAqIHJlc291cmNlcy9zdW5zcGlkZXItc3RhbmRhbG9uZS1kcml2ZXIuanM6IFRy eSB0byBsb2FkIGEgLWRhdGEKKyAgICAgICAgZmlsZS4gIElmIHRoYXQgc3VjY2VlZHMsIHdlIGhh dmUgYSBrcmFrZW4tbGlrZSB0ZXN0LCBzbyB3ZSB0aW1lCisgICAgICAgIHRoZSB0ZXN0IHVzaW5n IGBsb2FkJy4gIE90aGVyd2lzZSBmYWxsIGJhY2sgdG8gdXNpbmcgYHJ1bicuCisKIDIwMTEtMDgt MTEgIE1hcmsgUm93ZSAgPG1yb3dlQGFwcGxlLmNvbT4KIAogICAgICAgICBGaXggc3Vuc3BpZGVy LWNvbXBhcmUtcmVzdWx0cyB0byBkZWZhdWx0IHRvIHRoZSBzYW1lIHN1aXRlIHZlcnNpb24gYXMg dGhlIHN1bnNwaWRlciBzY3JpcHQuCmRpZmYgLS1naXQgYS9QZXJmb3JtYW5jZVRlc3RzL1N1blNw aWRlci9yZXNvdXJjZXMvc3Vuc3BpZGVyLXN0YW5kYWxvbmUtZHJpdmVyLmpzIGIvUGVyZm9ybWFu Y2VUZXN0cy9TdW5TcGlkZXIvcmVzb3VyY2VzL3N1bnNwaWRlci1zdGFuZGFsb25lLWRyaXZlci5q cwppbmRleCAxMzZlNmI3NjY3OTgyZTQ5MjQ0YmRlMGMyYjk3Y2Q5YWE1NDMxMTFhLi5lNjY4NmYx ZjYzODU2YjFjNDVlNTQxYzNkOWZjYTM5MmU0YzRlNzdkIDEwMDY0NAotLS0gYS9QZXJmb3JtYW5j ZVRlc3RzL1N1blNwaWRlci9yZXNvdXJjZXMvc3Vuc3BpZGVyLXN0YW5kYWxvbmUtZHJpdmVyLmpz CisrKyBiL1BlcmZvcm1hbmNlVGVzdHMvU3VuU3BpZGVyL3Jlc291cmNlcy9zdW5zcGlkZXItc3Rh bmRhbG9uZS1kcml2ZXIuanMKQEAgLTMyLDEyICszMiwyOCBAQCB2YXIgdGltZXMgPSBbXTsKIHRp bWVzLmxlbmd0aCA9IHRlc3RzLmxlbmd0aDsKIAogZm9yICh2YXIgaiA9IDA7IGogPCB0ZXN0cy5s ZW5ndGg7IGorKykgewotICAgIHZhciB0ZXN0TmFtZSA9IHN1aXRlUGF0aCArICIvIiArIHRlc3Rz W2pdICsgIi5qcyI7Ci0gICAgdmFyIHN0YXJ0VGltZSA9IG5ldyBEYXRlOwotICAgIGlmICh0ZXN0 TmFtZS5pbmRleE9mKCdwYXJzZS1vbmx5JykgPj0gMCkKKyAgICB2YXIgdGVzdEJhc2UgPSBzdWl0 ZVBhdGggKyAiLyIgKyB0ZXN0c1tqXTsKKyAgICB2YXIgdGVzdE5hbWUgPSB0ZXN0QmFzZSArICIu anMiOworICAgIHZhciB0ZXN0RGF0YSA9IHRlc3RCYXNlICsgIi1kYXRhLmpzIjsKKworICAgIGlm ICh0ZXN0TmFtZS5pbmRleE9mKCdwYXJzZS1vbmx5JykgPj0gMCkgewogICAgICAgICB0aW1lc1tq XSA9IGNoZWNrU3ludGF4KHRlc3ROYW1lKTsKLSAgICBlbHNlCi0gICAgICAgIHRpbWVzW2pdID0g cnVuKHRlc3ROYW1lKTsKKyAgICB9IGVsc2UgeworICAgICAgICAvLyBUZXN0cyBtYXkgb3IgbWF5 IG5vdCBoYXZlIGFzc29jaWF0ZWQgLWRhdGEgZmlsZXMgd2hvc2UgbG9hZGluZworICAgICAgICAv LyBzaG91bGQgbm90IGJlIHRpbWVkLgorICAgICAgICB0cnkgeworICAgICAgICAgICAgbG9hZCh0 ZXN0RGF0YSk7CisgICAgICAgICAgICAvLyBJZiBhIGZpbGUgZG9lcyBoYXZlIHRlc3QgZGF0YSwg dGhlbiB3ZSBjYW4ndCB1c2UgdGhlCisgICAgICAgICAgICAvLyBoaWdoZXItcHJlY2lzaW9uIGBy dW4nIHRpbWVyLCBiZWNhdXNlIGBydW4nIHVzZXMgYSBmcmVzaAorICAgICAgICAgICAgLy8gZ2xv YmFsIGVudmlyb25tZW50LCBzbyB3ZSBmYWxsIGJhY2sgdG8gYGxvYWQnLgorICAgICAgICAgICAg dmFyIHN0YXJ0VGltZSA9IG5ldyBEYXRlOworICAgICAgICAgICAgbG9hZCh0ZXN0TmFtZSk7Cisg ICAgICAgICAgICB0aW1lc1tqXSA9IG5ldyBEYXRlKCkgLSBzdGFydFRpbWU7CisgICAgICAgIH0g Y2F0Y2ggKGUpIHsKKyAgICAgICAgICAgIC8vIE5vIHRlc3QgZGF0YSwganVzdCB1c2UgYHJ1bicu CisgICAgICAgICAgICB0aW1lc1tqXSA9IHJ1bih0ZXN0TmFtZSk7CisgICAgICAgIH0KKyAgICB9 CiAgICAgZ2MoKTsKIH0KIAo=