Bug 132784

Summary:

DYEBench should fail when the variance is more than 5%

Product:

WebKit

Reporter:

Ryosuke Niwa <rniwa>

Component:

Tools / Tests

Assignee:

Ryosuke Niwa <rniwa>

Status:

RESOLVED WONTFIX

Severity:

Normal

CC:

barraclough, benjamin, darin, fpizlo, kling, mjs

Priority:

Version:

528+ (Nightly build)

Hardware:

Unspecified

OS:

Unspecified

Attachments:

Description	Flags
Patch	none

Description Ryosuke Niwa 2014-05-10 14:50:48 PDT

As far as I'm concerned, the variance on DYEBench should be no more than 5%.
e.g. Safari consistently reports the variance below 1% on many systems I've tested.

Since the variance of more than 5% indicates something went wrong,
we shouldn't even report the arithmetic mean.

Comment 1 Ryosuke Niwa 2014-05-10 15:36:27 PDT

Created attachment 231239 [details]
Patch

Comment 2 Ryosuke Niwa 2014-05-10 15:50:40 PDT

By the way, the "variance" I'm talking about here isn't really http://en.wikipedia.org/wiki/Variance.

I'm rejecting results with a confidence interval that is larger than 10% of the arithmetic mean.

Comment 3 Ryosuke Niwa 2014-05-10 21:03:12 PDT

Comment on attachment 231239 [details]
Patch

Per email discussion with Phil, we're going to increase the number of iterations from 5 to 20 instead.

Comment 4 Ryosuke Niwa 2014-05-10 21:09:13 PDT

Filed https://bugs.webkit.org/show_bug.cgi?id=132795 to track that.