Bug 132784

Summary: DYEBench should fail when the variance is more than 5%
Product: WebKit Reporter: Ryosuke Niwa <rniwa>
Component: Tools / TestsAssignee: Ryosuke Niwa <rniwa>
Status: RESOLVED WONTFIX    
Severity: Normal CC: barraclough, benjamin, darin, fpizlo, kling, mjs
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Patch none

Ryosuke Niwa
Reported 2014-05-10 14:50:48 PDT
As far as I'm concerned, the variance on DYEBench should be no more than 5%. e.g. Safari consistently reports the variance below 1% on many systems I've tested. Since the variance of more than 5% indicates something went wrong, we shouldn't even report the arithmetic mean.
Attachments
Patch (3.52 KB, patch)
2014-05-10 15:36 PDT, Ryosuke Niwa 		no flags
DetailsFormatted DiffDiff
Show Obsolete (1) View All Add attachment proposed patch, testcase, etc.
Ryosuke Niwa
Comment 1 2014-05-10 15:36:27 PDT
Created attachment 231239 [details] Patch
Ryosuke Niwa
Comment 2 2014-05-10 15:50:40 PDT
By the way, the "variance" I'm talking about here isn't really http://en.wikipedia.org/wiki/Variance. I'm rejecting results with a confidence interval that is larger than 10% of the arithmetic mean.
Ryosuke Niwa
Comment 3 2014-05-10 21:03:12 PDT
Comment on attachment 231239 [details] Patch Per email discussion with Phil, we're going to increase the number of iterations from 5 to 20 instead.
Ryosuke Niwa
Comment 4 2014-05-10 21:09:13 PDT
Filed https://bugs.webkit.org/show_bug.cgi?id=132795 to track that.
Note You need to log in before you can comment on or make changes to this bug.