213219 2020-06-15 15:07:16 -0700 Multiple hypothesis testing should use False Discovery Rate instead of Bonferroni 2020-06-16 16:29:41 -0700 1 1 1 Unclassified WebKit JavaScriptCore WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED InRadar P2 Normal --- 1 saam saam ap darin mark.lam webkit-bug-importer oldest_to_newest 1662853 0 saam 2020-06-15 15:07:16 -0700 ... 1662922 1 401961 saam 2020-06-15 17:53:50 -0700 Created attachment 401961 patch 1663220 2 401961 mark.lam 2020-06-16 13:49:16 -0700 Comment on attachment 401961 patch r=me 1663230 3 ews-feeder 2020-06-16 14:09:33 -0700 Committed r263114: <https://trac.webkit.org/changeset/263114> All reviewed patches have been landed. Closing bug and clearing flags on attachment 401961. 1663232 4 webkit-bug-importer 2020-06-16 14:10:25 -0700 <rdar://problem/64421181> 1663288 5 401961 darin 2020-06-16 15:58:07 -0700 Comment on attachment 401961 patch View in context: https://bugs.webkit.org/attachment.cgi?id=401961&action=review > Tools/Scripts/compare-results:181 > + isSignifcant = computeMultipleHypothesesSignficance(a, b) typo: "signifcant" 1663296 6 saam 2020-06-16 16:22:38 -0700 (In reply to Darin Adler from comment #5) > Comment on attachment 401961 [details] > patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=401961&action=review > > > Tools/Scripts/compare-results:181 > > + isSignifcant = computeMultipleHypothesesSignficance(a, b) > > typo: "signifcant" Ah, I typod it twice! "Signficance" too Will fix 1663301 7 saam 2020-06-16 16:29:41 -0700 fixed in: https://trac.webkit.org/changeset/263124/webkit 401961 2020-06-15 17:53:50 -0700 2020-06-16 14:09:34 -0700 patch b-backup.diff text/plain 7067 saam SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDI2MzA2OSkKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDM3IEBACisyMDIwLTA2LTE1ICBTYWFtIEJhcmF0aSAgPHNiYXJhdGlAYXBwbGUuY29tPgor CisgICAgICAgIE11bHRpcGxlIGh5cG90aGVzaXMgdGVzdGluZyBzaG91bGQgdXNlIEZhbHNlIERp c2NvdmVyeSBSYXRlIGluc3RlYWQgb2YgQm9uZmVycm9uaQorICAgICAgICBodHRwczovL2J1Z3Mu d2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9MjEzMjE5CisKKyAgICAgICAgUmV2aWV3ZWQgYnkg Tk9CT0RZIChPT1BTISkuCisKKyAgICAgICAgTXkgcHJldmlvdXMgcGF0Y2ggaGVyZSB1c2VkIEJv bmZlcnJvbmkgdG8gZG8gbXVsdGlwbGUgaHlwb3RoZXNlcyBjb3JyZWN0aW9uLgorICAgICAgICBC b25mZXJyb25pIGRvZXMgYSBnb29kIGpvYiBhdCByZWR1Y2luZyBmYWxzZSBwb3NpdGl2ZSByYXRl ICh0eXBlIEkgZXJyb3IsCisgICAgICAgIHNheWluZyBzb21ldGhpbmcgaXMgc2lnbmlmaWNhbnQg d2hlbiBpdCdzIG5vdCwgZS5nLCByZWplY3RpbmcgdGhlIG51bGwKKyAgICAgICAgaHlwb3RoZXNp cyB3aGVuIHdlIHNob3VsZG4ndCksIGJ1dCBhdCB0aGUgZXhwZW5zZSBvZiBhbHNvIGhhdmluZyBh IGhpZ2gKKyAgICAgICAgZmFsc2UgbmVnYXRpdmUgcmF0ZSAodHlwZSBJSSBlcnJvciwgc2F5aW5n IHNvbWV0aGluZyBpcyBub3Qgc2lnbmlmaWNhbnQgd2hlbgorICAgICAgICBpdCBpcywgZS5nLCBu b3QgcmVqZWN0aW5nIHRoZSBudWxsIGh5cG90aGVzaXMgd2hlbiB3ZSBzaG91bGQpLgorICAgICAg ICAKKyAgICAgICAgQWZ0ZXIgZG9pbmcgYSBiaXQgbW9yZSByZWFkaW5nLCBpdCBzZWVtcyBsaWtl IHVzaW5nIGEgdGVjaG5pcXVlIGNhbGxlZCBGYWxzZQorICAgICAgICBEaXNjb3ZlcnkgUmF0ZSAo RkRSKSBpcyBiZXR0ZXIuIEl0IHN0aWxsIGRvZXMgYSBnb29kIGpvYiBhdCBjb250cm9sbGluZyB0 eXBlCisgICAgICAgIEkgZXJyb3IgcmF0ZSwgYnV0IHdpdGhvdXQgc2FjcmlmaWNpbmcgYSBoaWdo IHR5cGUgSUkgZXJyb3IgcmF0ZS4gRkRSIGlzIGJhc2VkCisgICAgICAgIG9uIGEgdGVjaG5pcXVl IGZvciAiZXN0aW1hdGluZyB0aGUgcHJvcG9ydGlvbiBvZiB3cm9uZ2x5IHJlamVjdGVkIG51bGwg aHlwb3RoZXNlcyIgWzJdLgorICAgICAgICAKKyAgICAgICAgVGhpcyBwYXRjaCB1c2VzIEJlbmph bWluaSBhbmQgSG9jaGJlcmcgZXN0aW1hdGlvbiBvZiBGRFIsIHdoaWNoIHJlbGllcworICAgICAg ICBvbiBoeXBvdGhlc2VzIGJlaW5nIGluZGVwZW5kZW50LiBXZSBrbm93IHN1YnRlc3RzIGluIGEg YmVuY2htYXJrIGFyZW4ndAorICAgICAgICBmdWxseSBpbmRlcGVuZGVudCB2YXJpYWJsZXMsIGJ1 dCBpdCdzIGEgZ29vZCBlbm91Z2ggYXBwcm94aW1hdGlvbi4KKyAgICAgICAgCisgICAgICAgIFlv dSBjYW4gcmVhZCBtb3JlIGFib3V0IEZEUjoKKyAgICAgICAgMS4gaHR0cHM6Ly93d3cuc3RhdC5i ZXJrZWxleS5lZHUvfm1nb2xkbWFuL1NlY3Rpb24wNDAyLnBkZgorICAgICAgICAyLiBodHRwczov L2Jlc2pvdXJuYWxzLm9ubGluZWxpYnJhcnkud2lsZXkuY29tL2RvaS8xMC4xMTExL2ouMjA0MS0y MTBYLjIwMTAuMDAwNjEueAorICAgICAgICAzLiBodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lr aS9GYWxzZV9kaXNjb3ZlcnlfcmF0ZQorCisgICAgICAgICogU2NyaXB0cy9jb21wYXJlLXJlc3Vs dHM6CisgICAgICAgIChkaXNwbGF5U3RyKToKKyAgICAgICAgKGNvbXB1dGVNdWx0aXBsZUh5cG90 aGVzZXNTaWduZmljYW5jZSk6CisgICAgICAgIChkdW1wQnJlYWtkb3ducyk6CisgICAgICAgICh3 cml0ZUNTVik6CisKIDIwMjAtMDYtMTUgIEpvbmF0aGFuIEJlZGFyZCAgPGpiZWRhcmRAYXBwbGUu Y29tPgogCiAgICAgICAgIEZvbGxvdy11cCBmaXg6IE1ha2UgQVBJIHRlc3RzIHRvbGVyYW50IG9m IG91ciByZWxhdGl2ZWx5IG5ldyB1c2Ugb2YgV2ViUGFnZVByb3h5OjpwcmVjb25uZWN0VG8KSW5k ZXg6IFRvb2xzL1NjcmlwdHMvY29tcGFyZS1yZXN1bHRzCj09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIFRvb2xzL1Nj cmlwdHMvY29tcGFyZS1yZXN1bHRzCShyZXZpc2lvbiAyNjMwNDQpCisrKyBUb29scy9TY3JpcHRz L2NvbXBhcmUtcmVzdWx0cwkod29ya2luZyBjb3B5KQpAQCAtMTIyLDIwICsxMjIsNjQgQEAgZGVm IHBsdDVCcmVha2Rvd24oanNvbk9iamVjdCk6CiBkZWYgZGlzcGxheVN0cih2YWx1ZSk6CiAgICAg cmV0dXJuICJ7Oi42Zn0iLmZvcm1hdChmbG9hdCh2YWx1ZSkpCiAKK2RlZiBjb21wdXRlTXVsdGlw bGVIeXBvdGhlc2VzU2lnbmZpY2FuY2UoYSwgYik6CisgICAgIyBUaGlzIGlzIHVzaW5nIHRoZSBC ZW5qYW1pbmktSG9jaGJlcmcgcHJvY2VkdXJlIGJhc2VkIG9uIEZhbHNlIERpc2NvdmVyeSBSYXRl CisgICAgIyBmb3IgY29tcHV0aW5nIHNpZ25pZmNhbmNlIGluIG11bHRpcGxlIGh5cG90aGVzaXMg dGVzdGluZyAKKyAgICAjIFJlYWQgbW9yZSBoZXJlOgorICAgICMgLSBodHRwczovL2VuLndpa2lw ZWRpYS5vcmcvd2lraS9GYWxzZV9kaXNjb3ZlcnlfcmF0ZQorICAgICMgLSBodHRwczovL3d3dy5z dGF0LmJlcmtlbGV5LmVkdS9+bWdvbGRtYW4vU2VjdGlvbjA0MDIucGRmCisgICAgIyBUaGlzIGlz IGJlc3QgdXNlZCBmb3IgaW5kZXBlbmRlbnQgdmFyaWFibGVzLiBXZSBrbm93IHN1YnRlc3RzIGFy ZW4ndAorICAgICMgZnVsbHkgaW5kZXBlbmRlbnQsIHRoaXMgaXQncyBhIHJlYXNvbmFibGUgYXBw cm94aW1hdGlvbi4KKyAgICAjIFdlIHVzZSB0aGlzIGluc3RlYWQgb2YgQm9uZmVycm9uaSBiZWNh dXNlIHdlIGNvbnRyb2wgZm9yIGFsbW9zdCB0aGUgc2FtZQorICAgICMgZmFsc2UgcG9zaXRpdmUg ZXJyb3IgcmF0ZSAobWFya2luZyBhcyBzaWduZmljYW50IHdoZW4gaXQncyBub3QpLCBidXQgd2l0 aCBhIG11Y2gKKyAgICAjIGxvd2VyIGZhbHNlIG5lZ2F0aXZlIGVycm9yIHJhdGUgKG5vdCBtYXJr aW5nIHNvbWV0aGluZyBhcyBzaWduZmljYW50IHdoZW4gaXQgaXMpLgorCisgICAgc29ydGVkUFZh bHVlcyA9IFtdCisgICAgcmV2ZXJzZVBWYWx1ZU1hcCA9IHt9CisKKyAgICBmb3Iga2V5IGluIGEu a2V5cygpOgorICAgICAgICBpZiBrZXkgPT0gdW5pdE1hcmtlcjoKKyAgICAgICAgICAgIGNvbnRp bnVlCisKKyAgICAgICAgKHRTdGF0aXN0aWMsIHBWYWx1ZSkgPSBzdGF0cy50dGVzdF9pbmQoYVtr ZXldLCBiW2tleV0sIGVxdWFsX3Zhcj1GYWxzZSkKKworICAgICAgICBzb3J0ZWRQVmFsdWVzLmFw cGVuZChwVmFsdWUpCisgICAgICAgIGlmIHBWYWx1ZSBub3QgaW4gcmV2ZXJzZVBWYWx1ZU1hcDoK KyAgICAgICAgICAgIHJldmVyc2VQVmFsdWVNYXBbcFZhbHVlXSA9IFtdCisgICAgICAgIHJldmVy c2VQVmFsdWVNYXBbcFZhbHVlXS5hcHBlbmQoa2V5KQorCisgICAgc29ydGVkUFZhbHVlcy5zb3J0 KCkKKyAgICBhc3NlcnQgc29ydGVkUFZhbHVlc1swXSA8PSBzb3J0ZWRQVmFsdWVzWy0xXQorCisg ICAgaXNTaWduaWZjYW50ID0gRmFsc2UKKyAgICByZXN1bHQgPSB7fQorICAgIHJhbmsgPSBmbG9h dChsZW4oc29ydGVkUFZhbHVlcykpCisgICAgZm9yIHBWYWx1ZSBpbiByZXZlcnNlZChzb3J0ZWRQ VmFsdWVzKToKKyAgICAgICAgYXNzZXJ0IHJhbmsgPj0gMS4wCisgICAgICAgIHRocmVzaG9sZCA9 IChyYW5rICogLjA1KSAvIGZsb2F0KGxlbihzb3J0ZWRQVmFsdWVzKSkKKyAgICAgICAgaWYgcFZh bHVlIDw9IHRocmVzaG9sZDoKKyAgICAgICAgICAgIGlzU2lnbmlmY2FudCA9IFRydWUKKyAgICAg ICAgCisgICAgICAgIGFzc2VydCBsZW4ocmV2ZXJzZVBWYWx1ZU1hcFtwVmFsdWVdKSA+IDAKKyAg ICAgICAgZm9yIHRlc3QgaW4gcmV2ZXJzZVBWYWx1ZU1hcFtwVmFsdWVdOgorICAgICAgICAgICAg cmVzdWx0W3Rlc3RdID0gaXNTaWduaWZjYW50CisKKyAgICAgICAgcmFuayA9IHJhbmsgLSAxLjAK KworICAgIHJldHVybiByZXN1bHQKKworCiBkZWYgZHVtcEJyZWFrZG93bnMoYSwgYik6CiAgICAg bmFtZUxlbmd0aCA9IGxlbigic3VidGVzdCIpCiAgICAgYUxlbmd0aCA9IGxlbihhW3VuaXRNYXJr ZXJdKQogICAgIGJMZW5ndGggPSBsZW4oYVt1bml0TWFya2VyXSkKICAgICByYXRpb0xlbmd0aCA9 IGxlbigiYiAvIGEiKQogCi0gICAgIyBTZWUgQm9uZmVycm9uaSBjb3JyZWN0aW9uOgotICAgICMg aHR0cHM6Ly9lbi5tLndpa2lwZWRpYS5vcmcvd2lraS9Cb25mZXJyb25pX2NvcnJlY3Rpb24KLSAg ICAjIElmIHdlJ3JlIHRlc3RpbmcgTiBoeXBvdGhlc2lzLCB3ZSBtYWtlIG91ciB0aHJlc2hvbGQs IGFscGhhLCAuMDUvTiBpbnN0ZWFkIG9mIC4wNS4KLSAgICB0ZXN0Q291bnQgPSBmbG9hdChsZW4o YS5rZXlzKCkpIC0gMSkKLSAgICBhbHBoYSA9IC4wNSAvIHRlc3RDb3VudAotICAgIHBWYWx1ZUhl YWRlciA9ICJwVmFsdWUsIGFscGhhID0ge30iLmZvcm1hdChkaXNwbGF5U3RyKGFscGhhKSkKKyAg ICBwVmFsdWVIZWFkZXIgPSAicFZhbHVlIChzaWduaWZpY2FuY2UgdXNpbmcgRmFsc2UgRGlzY292 ZXJ5IFJhdGUpIgogICAgIHBMZW5ndGggPSBsZW4ocFZhbHVlSGVhZGVyKQogCisgICAgaXNTaWdu aWZjYW50ID0gY29tcHV0ZU11bHRpcGxlSHlwb3RoZXNlc1NpZ25maWNhbmNlKGEsIGIpCisKICAg ICBmb3Iga2V5IGluIGEua2V5cygpOgogICAgICAgICBpZiBrZXkgPT0gdW5pdE1hcmtlcjoKICAg ICAgICAgICAgIGNvbnRpbnVlCkBAIC0xNDYsNyArMTkwLDcgQEAgZGVmIGR1bXBCcmVha2Rvd25z KGEsIGIpOgogCiAgICAgICAgICh0U3RhdGlzdGljLCBwVmFsdWUpID0gc3RhdHMudHRlc3RfaW5k KGFba2V5XSwgYltrZXldLCBlcXVhbF92YXI9RmFsc2UpCiAgICAgICAgIHNpZ25pZmljYW50U3Ry ID0gIiIKLSAgICAgICAgaWYgcFZhbHVlIDw9IGFscGhhOgorICAgICAgICBpZiBpc1NpZ25pZmNh bnRba2V5XToKICAgICAgICAgICAgIHNpZ25pZmljYW50U3RyID0gIiAoc2lnbmlmaWNhbnQpIgog ICAgICAgICBwTGVuZ3RoID0gbWF4KHBMZW5ndGgsIGxlbihkaXNwbGF5U3RyKHBWYWx1ZSkpICsg bGVuKHNpZ25pZmljYW50U3RyKSkKIApAQCAtMTY4LDcgKzIxMiw3IEBAIGRlZiBkdW1wQnJlYWtk b3ducyhhLCBiKToKICAgICAgICAgKHRTdGF0aXN0aWMsIHBWYWx1ZSkgPSBzdGF0cy50dGVzdF9p bmQoYVtrZXldLCBiW2tleV0sIGVxdWFsX3Zhcj1GYWxzZSkKIAogICAgICAgICBzaWduaWZpY2Fu dFN0ciA9ICIiCi0gICAgICAgIGlmIHBWYWx1ZSA8PSBhbHBoYToKKyAgICAgICAgaWYgaXNTaWdu aWZjYW50W2tleV06CiAgICAgICAgICAgICBzaWduaWZpY2FudFN0ciA9ICIgKHNpZ25pZmljYW50 KSIKIAogICAgICAgICBzdHJpbmdzLmFwcGVuZCgifCB7a2V5OntuYW1lTGVuZ3RofX18e2FTY29y ZTp7YUxlbmd0aH19IHx7YlNjb3JlOntiTGVuZ3RofX0gfHtjb21wYXJlOntyYXRpb0xlbmd0aH19 fCB7cFZhbHVlOjx7cExlbmd0aH19fCIuZm9ybWF0KGtleT1rZXksIGFTY29yZT1kaXNwbGF5U3Ry KGFTY29yZSksIGJTY29yZT1kaXNwbGF5U3RyKGJTY29yZSksIG5hbWVMZW5ndGg9bmFtZUxlbmd0 aCAtIDEsIGFMZW5ndGg9YUxlbmd0aCwgYkxlbmd0aD1iTGVuZ3RoLCByYXRpb0xlbmd0aD1yYXRp b0xlbmd0aCwgY29tcGFyZT1kaXNwbGF5U3RyKGJTY29yZSAvIGFTY29yZSksIHBWYWx1ZSA9IGRp c3BsYXlTdHIocFZhbHVlKSArIHNpZ25pZmljYW50U3RyLCBwTGVuZ3RoPXBMZW5ndGggLSAxKSkK QEAgLTE5MCwxMSArMjM0LDkgQEAgZGVmIGR1bXBCcmVha2Rvd25zKGEsIGIpOgogZGVmIHdyaXRl Q1NWKGEsIGIsIGZpbGVOYW1lKToKICAgICBzdHJpbmdzID0gW10KICAgICByZXN1bHQgPSAiIgot ICAgIHRlc3RDb3VudCA9IGZsb2F0KGxlbihhLmtleXMoKSkgLSAxKQotICAgIGFscGhhID0gLjA1 IC8gdGVzdENvdW50Ci0gICAgcFZhbHVlSGVhZGVyID0gInBWYWx1ZV93aXRoX2FscGhhX3t9Ii5m b3JtYXQoZGlzcGxheVN0cihhbHBoYSkpCi0gICAgcmVzdWx0ICs9ICJ0ZXN0X25hbWUsIHt9LCB7 fSwgYl9kaXZpZGVkX2J5X2EsIHt9LCBpc19zaWduaWZpY2FudFxuIi5mb3JtYXQoImFfaW5fIiAr IGFbdW5pdE1hcmtlcl0sICJiX2luXyIgKyBiW3VuaXRNYXJrZXJdLCBwVmFsdWVIZWFkZXIpCisg ICAgcmVzdWx0ICs9ICJ0ZXN0X25hbWUsIHt9LCB7fSwgYl9kaXZpZGVkX2J5X2EsIHBWYWx1ZSwg aXNfc2lnbmlmaWNhbnRfdXNpbmdfRmFsc2VfRGlzY292ZXJ5X1JhdGVcbiIuZm9ybWF0KCJhX2lu XyIgKyBhW3VuaXRNYXJrZXJdLCAiYl9pbl8iICsgYlt1bml0TWFya2VyXSkKIAorICAgIGlzU2ln bmlmY2FudCA9IGNvbXB1dGVNdWx0aXBsZUh5cG90aGVzZXNTaWduZmljYW5jZShhLCBiKQogCiAg ICAgZm9yIGtleSBpbiBhLmtleXMoKToKICAgICAgICAgaWYga2V5ID09IHVuaXRNYXJrZXI6CkBA IC0yMDQsOCArMjQ2LDkgQEAgZGVmIHdyaXRlQ1NWKGEsIGIsIGZpbGVOYW1lKToKICAgICAgICAg YlNjb3JlID0gbnVtcHkubWVhbihiW2tleV0pCiAKICAgICAgICAgKHRTdGF0aXN0aWMsIHBWYWx1 ZSkgPSBzdGF0cy50dGVzdF9pbmQoYVtrZXldLCBiW2tleV0sIGVxdWFsX3Zhcj1GYWxzZSkKLSAg ICAgICAgIyBTZWUgYWJvdmUgY29tbWVudCBhYm91dCBCb25mZXJyb25pIGNvcnJlY3Rpb24KICAg ICAgICAgc2lnbmlmaWNhbnRTdHIgPSAiTm8iCisgICAgICAgIGlmIGlzU2lnbmlmY2FudFtrZXld OgorICAgICAgICAgICAgc2lnbmlmaWNhbnRTdHIgPSAiWWVzIgogICAgICAgICByZXN1bHQgKz0g Int9LHt9LHt9LHt9LHt9LHt9XG4iLmZvcm1hdChrZXksIGRpc3BsYXlTdHIoYVNjb3JlKSwgZGlz cGxheVN0cihiU2NvcmUpLCBkaXNwbGF5U3RyKGJTY29yZSAvIGFTY29yZSksIGRpc3BsYXlTdHIo cFZhbHVlKSwgc2lnbmlmaWNhbnRTdHIpCiAKICAgICBmID0gb3BlbihmaWxlTmFtZSwgInciKQo=