114458 2013-04-11 12:41:54 -0700 Get rid of the 10ms delay between test executions 2013-04-11 13:01:20 -0700 1 1 1 Unclassified WebKit JavaScriptCore 528+ (Nightly build) All All RESOLVED FIXED InRadar P2 Normal --- 1 fpizlo fpizlo barraclough commit-queue ggaren mark.lam mhahnenberg msaboff oliver rniwa sam oldest_to_newest 874051 0 fpizlo 2013-04-11 12:41:54 -0700 I don't know the history of why the 10ms delay is there. But it causes the benchmark to exhibit some strange behaviors: - If your OS has any kind of power management smarts, and your machine is fast enough to run each individual test in significantly less than 10ms (like say, a test runs in 2ms), then the power management logic will think that you're mostly idle and reduce clock. We've seen this happen before on a variety of OSes, and it typically has the effect that a faster machine with the same OS and WebKit will appear to run _slower_ according to SunSpider output because to CPU is faster and hence makes us appear more idle. It's happening right now on my two MBP's, and as far as I can tell it's entirely because of that 10ms delay. Paradoxically, if your browser or some other thing on your system is simultaneously eating up CPU, your faster machine will appear to run faster in SunSpider because the power management thingy won't kick in. This is a double-whammy: if the power management does kick in, then you look slower; but worse, you have no guarantee that the power management thing *will* actually kick in, so you end up spending several days wondering why your SunSpider scores exhibit a bimodal distribution without any clear way of reproducing the "slow" runs. So the 10ms delay is pure evil: it rewards slower machines and noisy systems while making benchmark results wildly inconsistent. Getting rid of the delay means that the machine is consistently warm for the entire benchmark run, and this appears to get the power heuristics out of the way. (And yes, I did consider saying that you should only run SunSpider if you hack your machine into a no-power-management state but that seems like it would make the benchmark less practical for our purposes: we want to be able to run it in an ad-hoc casual way and rely on the fact that even without extensive setup we'll get useful results. This is crucial to how we do our benchmark-driven development.) - If your VM delays GC until idle time, SunSpider won't get to measure GC effects at all. I think this is strange. A benchmark for a GC'd language should include GC time in the reported run-time, and should not encourage people to build super-fast mutators with super-slow collectors. Anyway, eliminating this 10ms delay doesn't change results, except on quiet fast machines, where you now get consistent scores and those scores are indeed faster than the scores you'd get on slower machines. <rdar://problem/13631112> 874054 1 197656 fpizlo 2013-04-11 12:44:20 -0700 Created attachment 197656 the patch 874058 2 197656 ggaren 2013-04-11 12:48:56 -0700 Comment on attachment 197656 the patch r=me too 874072 3 fpizlo 2013-04-11 13:01:20 -0700 Landed in http://trac.webkit.org/changeset/148226 197656 2013-04-11 12:44:20 -0700 2013-04-11 12:48:56 -0700 the patch blah.patch text/plain 1393 fpizlo SW5kZXg6IFBlcmZvcm1hbmNlVGVzdHMvU3VuU3BpZGVyL0NoYW5nZUxvZwo9PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0t LSBQZXJmb3JtYW5jZVRlc3RzL1N1blNwaWRlci9DaGFuZ2VMb2cJKHJldmlzaW9uIDE0ODIyMykK KysrIFBlcmZvcm1hbmNlVGVzdHMvU3VuU3BpZGVyL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpA QCAtMSwzICsxLDE3IEBACisyMDEzLTA0LTExICBGaWxpcCBQaXpsbyAgPGZwaXpsb0BhcHBsZS5j b20+CisKKyAgICAgICAgR2V0IHJpZCBvZiB0aGUgMTBtcyBkZWxheSBiZXR3ZWVuIHRlc3QgZXhl Y3V0aW9ucworICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9 MTE0NDU4CisgICAgICAgIDxyZGFyOi8vcHJvYmxlbS8xMzYzMTExMj4KKworICAgICAgICBSZXZp ZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKyAgICAgICAgCisgICAgICAgIE1vcmUgZGV0YWlscyBv biB0aGlzIGluIHRoZSBidWcuIExvbmcgc3Rvcnkgc2hvcnQsIEkgZG9uJ3Qga25vdyB3aHkgdGhl IDEwbXMgZGVsYXkgaXMKKyAgICAgICAgdGhlcmUsIGFuZCBpdCBhcHBlYXJzIHRvIGJlIHRoZSBy b290IG9mIHNvbWUgbm9pc3luZXNzIGFuZCB1bnJlbGlhYmlsaXR5IGluIHRoZSByZXN1bHRzLgor ICAgICAgICBSZW1vdmluZyBpdCBtYWtlcyByZXN1bHRzIG1vcmUgY29uc2lzdGVudCwgd2hpY2gg aXMgYSBnb29kIHRoaW5nLgorCisgICAgICAgICogcmVzb3VyY2VzL2RyaXZlci1URU1QTEFURS5o dG1sOgorCiAyMDEzLTAxLTE2ICBFcmljIFNlaWRlbCAgPGVyaWNAd2Via2l0Lm9yZz4KIAogICAg ICAgICBSZW1vdmUgLS1zaGFyayogc3VwcG9ydCBmcm9tIHN1bnNwaWRlci9ydW4tc3Vuc3BpZGVy IG5vdyB0aGF0IFNoYXJrIGlzIGRlYWQKSW5kZXg6IFBlcmZvcm1hbmNlVGVzdHMvU3VuU3BpZGVy L3Jlc291cmNlcy9kcml2ZXItVEVNUExBVEUuaHRtbAo9PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBQZXJmb3JtYW5j ZVRlc3RzL1N1blNwaWRlci9yZXNvdXJjZXMvZHJpdmVyLVRFTVBMQVRFLmh0bWwJKHJldmlzaW9u IDE0ODEzNikKKysrIFBlcmZvcm1hbmNlVGVzdHMvU3VuU3BpZGVyL3Jlc291cmNlcy9kcml2ZXIt VEVNUExBVEUuaHRtbAkod29ya2luZyBjb3B5KQpAQCAtNTgsNyArNTgsNyBAQCBmdW5jdGlvbiBz dGFydCgpIAogCiBmdW5jdGlvbiBuZXh0KCkgCiB7Ci0gICAgd2luZG93LnNldFRpbWVvdXQocmVh bGx5TmV4dCwgMTApOworICAgIHdpbmRvdy5zZXRUaW1lb3V0KHJlYWxseU5leHQsIDApOwogfQog CiBmdW5jdGlvbiByZWFsbHlOZXh0KCkgCg==