The largest part of the variability I mentioned in bug 43253 is that millisecond timing is insufficient for such short tests. I reduced the total variance in the tests by about 50% by using sub-millisecond timing. A few possible resolutions: - Using submillisecond timing. This would mean checking for the presence of a Date.timeInMicroSeconds() function, or similar, which would have to be created by the vendors. This is simple to add to Firefox, I don't know about others. - Increase the test lengths until they are no longer affected by this bug. The worst affected test is bitops-3bit-bits-in-byte, which typically runs in less than 1ms on my machine. bitops-bitwise-and, controlflow-recursive, crypto-sha1, math-spectral-norm, string-base64, and access-nbody all run in less than 10ms. The error due to this bug is highest in these tests. I think the second resolution is bad, because the test runs would have to be increased to perhaps 100ms each to really solve the problem, and that makes the test runs very long. It also means that if engines get much better, we run into the problem again. By contrast, checking for a Date.timeInMicroSeconds() function is non-invasive, allows vendors to opt-in, and doesn't change the experience for anyone.
IIRC data sets were cut down in 2007 to allow slow IE not to take too long, in particular for crypto-md5. Time to restore realistic data set sizes? I'm assuming there can be an "annual" SunSpider version so the old versions can stay up but new ones can appear. That would be great. SPEC does that, but too slowly for our crazy JS world. /be
I think the answer will have to be to make the tests longer. *** This bug has been marked as a duplicate of bug 61552 ***
Actually, this one got fixed. (Apologies, I should have resolved this at the time). The sunspider shell test harness was changed to call the run() function, which returns submillisecond timing, which fixed this. (Spidermonkey added the same run() function, so that fixes it on our end too). Here was the changeset (r72842) 2010-11-29 Geoffrey Garen <ggaren@apple.com> Reviewed by Gavin Barraclough. Improved accuracy of command-line SunSpider. * resources/sunspider-standalone-driver.js: Use the returned number of elapsed milliseconds from "run" and "checkSyntax" instead of measuring ourselves, for slightly more accurate numbers.