Bug 16849 - SunSpider should be rebased to run at least twice as long
Summary: SunSpider should be rebased to run at least twice as long
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Mac OS X 10.4
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2008-01-11 18:48 PST by Eric Seidel (no email)
Modified: 2011-07-02 12:42 PDT (History)
3 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Eric Seidel (no email) 2008-01-11 18:48:47 PST 
SunSpider should be rebased to run at least twice as long

SunSpider now runs in under 4 seconds on my (laptop) hardware.  The test should be rebased to take at least twice that long to help reduce error rates.  Yes, that means it would be *even slower* in other browsers, but my long term goal is to move all browsers to at least 10x their current JavaScript speed.  Rebasing the test to take 2x as long is a step in that direction.
Comment 1 Eric Seidel (no email) 2008-01-11 18:49:50 PST 
Error rates is not the sole motivator.  But certainly with more signal, small amounts of noise in the testing environment matter less.
Comment 2 Darin Fisher (:fishd, Google) 2008-01-17 22:43:19 PST 
Perhaps the tests should just be run multiple times, and then compute mean and variance?
Comment 3 Eric Seidel (no email) 2008-01-17 23:49:34 PST 
You can already do that with --runs=15
But that doesn't really address the precision question in my mind.