Bug 143552 – The results of A/B testing should state statistical significance

• Status: RESOLVED
• Resolution: FIXED
• Component: WebKit Website

[Ryosuke Niwa]

When A/B testing completes, we should report whether differences are statistically significant or not.

[Ryosuke Niwa]

Created attachment 250410 [details]
Adds the feature

[Ryosuke Niwa]

Created attachment 250411 [details]
Screenshot 1 (statistically significant)

[Ryosuke Niwa]

Created attachment 250412 [details]
Screenshot 2 (statistically insignificant)

[Chris Dumez]

Comment on attachment 250410 [details]
Adds the feature

View in context: https://bugs.webkit.org/attachment.cgi?id=250410&action=review

rs=me

> Websites/perf.webkit.org/public/v2/app.js:1339
> +        for (var i = 0; i < configurations.length; i++) {

Looks like we only need to go up to configurations.length-1 ?

> Websites/perf.webkit.org/public/v2/app.js:1341
> +                var summary1 = configurations[i].summary;

should probably be outside this for loop as it doesn't change.

> Websites/perf.webkit.org/public/v2/app.js:1362
> +            return 'Not statistically significant' + details;

missing space after "significant".

> Websites/perf.webkit.org/public/v2/js/statistics.js:67
> +            return [null, null];

In the regular case you return a dict but here you are returning an array?

[Ryosuke Niwa]

Comment on attachment 250410 [details]
Adds the feature

View in context: https://bugs.webkit.org/attachment.cgi?id=250410&action=review

Thanks for the review & nice bug catches!

>> Websites/perf.webkit.org/public/v2/app.js:1339
>> +        for (var i = 0; i < configurations.length; i++) {
> 
> Looks like we only need to go up to configurations.length-1 ?

Oops, nice catch!

>> Websites/perf.webkit.org/public/v2/app.js:1341
>> +                var summary1 = configurations[i].summary;
> 
> should probably be outside this for loop as it doesn't change.

Fixed.

>> Websites/perf.webkit.org/public/v2/app.js:1362
>> +            return 'Not statistically significant' + details;
> 
> missing space after "significant".

Oh, there is a leading space in "details".

>> Websites/perf.webkit.org/public/v2/js/statistics.js:67
>> +            return [null, null];
> 
> In the regular case you return a dict but here you are returning an array?

Oops, fixed.

[Ryosuke Niwa]

Committed r182587: <http://trac.webkit.org/changeset/182587>