Bug 143552 - The results of A/B testing should state statistical significance
Summary: The results of A/B testing should state statistical significance
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebKit Website (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Ryosuke Niwa
URL:
Keywords:
Depends on:
Blocks: 141443
  Show dependency treegraph
 
Reported: 2015-04-08 20:42 PDT by Ryosuke Niwa
Modified: 2015-04-08 21:59 PDT (History)
5 users (show)

See Also:


Attachments
Adds the feature (12.05 KB, patch)
2015-04-08 20:59 PDT, Ryosuke Niwa
cdumez: review+
Details | Formatted Diff | Diff
Screenshot 1 (statistically significant) (64.91 KB, image/png)
2015-04-08 21:05 PDT, Ryosuke Niwa
no flags Details
Screenshot 2 (statistically insignificant) (54.92 KB, image/png)
2015-04-08 21:05 PDT, Ryosuke Niwa
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Ryosuke Niwa 2015-04-08 20:42:55 PDT
When A/B testing completes, we should report whether differences are statistically significant or not.
Comment 1 Ryosuke Niwa 2015-04-08 20:59:18 PDT
Created attachment 250410 [details]
Adds the feature
Comment 2 Ryosuke Niwa 2015-04-08 21:05:09 PDT
Created attachment 250411 [details]
Screenshot 1 (statistically significant)
Comment 3 Ryosuke Niwa 2015-04-08 21:05:24 PDT
Created attachment 250412 [details]
Screenshot 2 (statistically insignificant)
Comment 4 Chris Dumez 2015-04-08 21:15:02 PDT
Comment on attachment 250410 [details]
Adds the feature

View in context: https://bugs.webkit.org/attachment.cgi?id=250410&action=review

rs=me

> Websites/perf.webkit.org/public/v2/app.js:1339
> +        for (var i = 0; i < configurations.length; i++) {

Looks like we only need to go up to configurations.length-1 ?

> Websites/perf.webkit.org/public/v2/app.js:1341
> +                var summary1 = configurations[i].summary;

should probably be outside this for loop as it doesn't change.

> Websites/perf.webkit.org/public/v2/app.js:1362
> +            return 'Not statistically significant' + details;

missing space after "significant".

> Websites/perf.webkit.org/public/v2/js/statistics.js:67
> +            return [null, null];

In the regular case you return a dict but here you are returning an array?
Comment 5 Ryosuke Niwa 2015-04-08 21:58:35 PDT
Comment on attachment 250410 [details]
Adds the feature

View in context: https://bugs.webkit.org/attachment.cgi?id=250410&action=review

Thanks for the review & nice bug catches!

>> Websites/perf.webkit.org/public/v2/app.js:1339
>> +        for (var i = 0; i < configurations.length; i++) {
> 
> Looks like we only need to go up to configurations.length-1 ?

Oops, nice catch!

>> Websites/perf.webkit.org/public/v2/app.js:1341
>> +                var summary1 = configurations[i].summary;
> 
> should probably be outside this for loop as it doesn't change.

Fixed.

>> Websites/perf.webkit.org/public/v2/app.js:1362
>> +            return 'Not statistically significant' + details;
> 
> missing space after "significant".

Oh, there is a leading space in "details".

>> Websites/perf.webkit.org/public/v2/js/statistics.js:67
>> +            return [null, null];
> 
> In the regular case you return a dict but here you are returning an array?

Oops, fixed.
Comment 6 Ryosuke Niwa 2015-04-08 21:59:12 PDT
Committed r182587: <http://trac.webkit.org/changeset/182587>