Bug 132784 - DYEBench should fail when the variance is more than 5%
Summary: DYEBench should fail when the variance is more than 5%
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Ryosuke Niwa
Depends on:
Reported: 2014-05-10 14:50 PDT by Ryosuke Niwa
Modified: 2014-05-10 21:09 PDT (History)
6 users (show)

See Also:

Patch (3.52 KB, patch)
2014-05-10 15:36 PDT, Ryosuke Niwa
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Ryosuke Niwa 2014-05-10 14:50:48 PDT
As far as I'm concerned, the variance on DYEBench should be no more than 5%.
e.g. Safari consistently reports the variance below 1% on many systems I've tested.

Since the variance of more than 5% indicates something went wrong,
we shouldn't even report the arithmetic mean.
Comment 1 Ryosuke Niwa 2014-05-10 15:36:27 PDT
Created attachment 231239 [details]
Comment 2 Ryosuke Niwa 2014-05-10 15:50:40 PDT
By the way, the "variance" I'm talking about here isn't really http://en.wikipedia.org/wiki/Variance.

I'm rejecting results with a confidence interval that is larger than 10% of the arithmetic mean.
Comment 3 Ryosuke Niwa 2014-05-10 21:03:12 PDT
Comment on attachment 231239 [details]

Per email discussion with Phil, we're going to increase the number of iterations from 5 to 20 instead.
Comment 4 Ryosuke Niwa 2014-05-10 21:09:13 PDT
Filed https://bugs.webkit.org/show_bug.cgi?id=132795 to track that.