When A/B testing completes, we should report whether differences are statistically significant or not.
Created attachment 250410 [details] Adds the feature
Created attachment 250411 [details] Screenshot 1 (statistically significant)
Created attachment 250412 [details] Screenshot 2 (statistically insignificant)
Comment on attachment 250410 [details] Adds the feature View in context: https://bugs.webkit.org/attachment.cgi?id=250410&action=review rs=me > Websites/perf.webkit.org/public/v2/app.js:1339 > + for (var i = 0; i < configurations.length; i++) { Looks like we only need to go up to configurations.length-1 ? > Websites/perf.webkit.org/public/v2/app.js:1341 > + var summary1 = configurations[i].summary; should probably be outside this for loop as it doesn't change. > Websites/perf.webkit.org/public/v2/app.js:1362 > + return 'Not statistically significant' + details; missing space after "significant". > Websites/perf.webkit.org/public/v2/js/statistics.js:67 > + return [null, null]; In the regular case you return a dict but here you are returning an array?
Comment on attachment 250410 [details] Adds the feature View in context: https://bugs.webkit.org/attachment.cgi?id=250410&action=review Thanks for the review & nice bug catches! >> Websites/perf.webkit.org/public/v2/app.js:1339 >> + for (var i = 0; i < configurations.length; i++) { > > Looks like we only need to go up to configurations.length-1 ? Oops, nice catch! >> Websites/perf.webkit.org/public/v2/app.js:1341 >> + var summary1 = configurations[i].summary; > > should probably be outside this for loop as it doesn't change. Fixed. >> Websites/perf.webkit.org/public/v2/app.js:1362 >> + return 'Not statistically significant' + details; > > missing space after "significant". Oh, there is a leading space in "details". >> Websites/perf.webkit.org/public/v2/js/statistics.js:67 >> + return [null, null]; > > In the regular case you return a dict but here you are returning an array? Oops, fixed.
Committed r182587: <http://trac.webkit.org/changeset/182587>