Bug 114458 – Get rid of the 10ms delay between test executions

• Status: RESOLVED
• Resolution: FIXED
• Component: JavaScriptCore

[Filip Pizlo]

I don't know the history of why the 10ms delay is there.  But it causes the benchmark to exhibit some strange behaviors:

- If your OS has any kind of power management smarts, and your machine is fast enough to run each individual test in significantly less than 10ms (like say, a test runs in 2ms), then the power management logic will think that you're mostly idle and reduce clock.  We've seen this happen before on a variety of OSes, and it typically has the effect that a faster machine with the same OS and WebKit will appear to run _slower_ according to SunSpider output because to CPU is faster and hence makes us appear more idle.  It's happening right now on my two MBP's, and as far as I can tell it's entirely because of that 10ms delay.  Paradoxically, if your browser or some other thing on your system is simultaneously eating up CPU, your faster machine will appear to run faster in SunSpider because the power management thingy won't kick in.  This is a double-whammy: if the power management does kick in, then you look slower; but worse, you have no guarantee that the power management thing *will* actually kick in, so you end up spending several days wondering why your SunSpider scores exhibit a bimodal distribution without any clear way of reproducing the "slow" runs.  So the 10ms delay is pure evil: it rewards slower machines and noisy systems while making benchmark results wildly inconsistent.  Getting rid of the delay means that the machine is consistently warm for the entire benchmark run, and this appears to get the power heuristics out of the way.  (And yes, I did consider saying that you should only run SunSpider if you hack your machine into a no-power-management state but that seems like it would make the benchmark less practical for our purposes: we want to be able to run it in an ad-hoc casual way and rely on the fact that even without extensive setup we'll get useful results.  This is crucial to how we do our benchmark-driven development.)

- If your VM delays GC until idle time, SunSpider won't get to measure GC effects at all.  I think this is strange.  A benchmark for a GC'd language should include GC time in the reported run-time, and should not encourage people to build super-fast mutators with super-slow collectors.

Anyway, eliminating this 10ms delay doesn't change results, except on quiet fast machines, where you now get consistent scores and those scores are indeed faster than the scores you'd get on slower machines.

<rdar://problem/13631112>

[Filip Pizlo]

Created attachment 197656 [details]
the patch

[Geoffrey Garen]

Comment on attachment 197656 [details]
the patch

r=me too

[Filip Pizlo]

Landed in http://trac.webkit.org/changeset/148226