RESOLVED WONTFIX132784
DYEBench should fail when the variance is more than 5% 
https://bugs.webkit.org/show_bug.cgi?id=132784
Summary DYEBench should fail when the variance is more than 5%
Ryosuke Niwa
Reported 2014-05-10 14:50:48 PDT
As far as I'm concerned, the variance on DYEBench should be no more than 5%. e.g. Safari consistently reports the variance below 1% on many systems I've tested. Since the variance of more than 5% indicates something went wrong, we shouldn't even report the arithmetic mean.
Attachments
Patch (3.52 KB, patch)
2014-05-10 15:36 PDT, Ryosuke Niwa 		no flags
DetailsFormatted DiffDiff
Show Obsolete (1) View All Add attachment proposed patch, testcase, etc.
Ryosuke Niwa
Comment 1 2014-05-10 15:36:27 PDT
Created attachment 231239 [details] Patch
Ryosuke Niwa
Comment 2 2014-05-10 15:50:40 PDT
By the way, the "variance" I'm talking about here isn't really http://en.wikipedia.org/wiki/Variance. I'm rejecting results with a confidence interval that is larger than 10% of the arithmetic mean.
Ryosuke Niwa
Comment 3 2014-05-10 21:03:12 PDT
Comment on attachment 231239 [details] Patch Per email discussion with Phil, we're going to increase the number of iterations from 5 to 20 instead.
Ryosuke Niwa
Comment 4 2014-05-10 21:09:13 PDT
Filed https://bugs.webkit.org/show_bug.cgi?id=132795 to track that.
Note You need to log in before you can comment on or make changes to this bug.