Bug 132784 - DYEBench should fail when the variance is more than 5%
Summary: DYEBench should fail when the variance is more than 5%
Status: RESOLVED WONTFIX
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Ryosuke Niwa
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-05-10 14:50 PDT by Ryosuke Niwa
Modified: 2014-05-10 21:09 PDT (History)
6 users (show)

See Also:

Attachments
Patch (3.52 KB, patch)
2014-05-10 15:36 PDT, Ryosuke Niwa 		no flags Details | Formatted Diff | Diff
Show Obsolete (1) View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Ryosuke Niwa 2014-05-10 14:50:48 PDT 
As far as I'm concerned, the variance on DYEBench should be no more than 5%.
e.g. Safari consistently reports the variance below 1% on many systems I've tested.

Since the variance of more than 5% indicates something went wrong,
we shouldn't even report the arithmetic mean.
Comment 1 Ryosuke Niwa 2014-05-10 15:36:27 PDT 
Created attachment 231239 [details]
Patch
Comment 2 Ryosuke Niwa 2014-05-10 15:50:40 PDT 
By the way, the "variance" I'm talking about here isn't really http://en.wikipedia.org/wiki/Variance.

I'm rejecting results with a confidence interval that is larger than 10% of the arithmetic mean.
Comment 3 Ryosuke Niwa 2014-05-10 21:03:12 PDT 
Comment on attachment 231239 [details]
Patch

Per email discussion with Phil, we're going to increase the number of iterations from 5 to 20 instead.
Comment 4 Ryosuke Niwa 2014-05-10 21:09:13 PDT 
Filed https://bugs.webkit.org/show_bug.cgi?id=132795 to track that.