132784 2014-05-10 14:50:48 -0700 DYEBench should fail when the variance is more than 5% 2014-05-10 21:09:13 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified RESOLVED WONTFIX P2 Normal --- 1 rniwa rniwa barraclough benjamin darin fpizlo kling mjs oldest_to_newest 1008307 0 rniwa 2014-05-10 14:50:48 -0700 As far as I'm concerned, the variance on DYEBench should be no more than 5%. e.g. Safari consistently reports the variance below 1% on many systems I've tested. Since the variance of more than 5% indicates something went wrong, we shouldn't even report the arithmetic mean. 1008316 1 231239 rniwa 2014-05-10 15:36:27 -0700 Created attachment 231239 Patch 1008328 2 rniwa 2014-05-10 15:50:40 -0700 By the way, the "variance" I'm talking about here isn't really http://en.wikipedia.org/wiki/Variance. I'm rejecting results with a confidence interval that is larger than 10% of the arithmetic mean. 1008369 3 231239 rniwa 2014-05-10 21:03:12 -0700 Comment on attachment 231239 Patch Per email discussion with Phil, we're going to increase the number of iterations from 5 to 20 instead. 1008371 4 rniwa 2014-05-10 21:09:13 -0700 Filed https://bugs.webkit.org/show_bug.cgi?id=132795 to track that. 231239 2014-05-10 15:36:27 -0700 2014-05-10 21:03:06 -0700 Patch bug-132784-20140510153604.patch text/plain 3607 rniwa SW5kZXg6IFBlcmZvcm1hbmNlVGVzdHMvQ2hhbmdlTG9nCj09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIFBlcmZvcm1h bmNlVGVzdHMvQ2hhbmdlTG9nCShyZXZpc2lvbiAxNjg1ODMpCisrKyBQZXJmb3JtYW5jZVRlc3Rz L0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwzICsxLDE5IEBACisyMDE0LTA1LTEwICBS eW9zdWtlIE5pd2EgIDxybml3YUB3ZWJraXQub3JnPgorCisgICAgICAgIERZRUJlbmNoIHNob3Vs ZCBmYWlsIHdoZW4gdGhlIHZhcmlhbmNlIGlzIG1vcmUgdGhhbiA1JQorICAgICAgICBodHRwczov L2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9MTMyNzg0CisKKyAgICAgICAgUmV2aWV3 ZWQgYnkgTk9CT0RZIChPT1BTISkuCisKKyAgICAgICAgU3BpdCBvdXQgYW4gZXJyb3IgbWVzc2Fn ZSBpbnN0ZWFkIG9mIHRoZSBhcml0aG1ldGljIG1lYW4gd2hlbiB0aGUgdmFyaWFuY2UgaXMgbW9y ZSB0aGFuIDUlCisgICAgICAgIHNpbmNlIHN1Y2ggYSBoaWdoIHZhcmlhbmNlIHVzdWFsbHkgaW5k aWNhdGVzIHNvbWV0aGluZyBoYXMgZ29uZSB3cm9uZy4KKyAgICAgICAgU2FmYXJpIGNvbnNpc3Rl bnRseSByZXBvcnRzIHRoZSB2YXJpYW5jZSBiZWxvdyAxJSBvbiBtYW55IHN5c3RlbXMgSSd2ZSB0 ZXN0ZWQuCisKKyAgICAgICAgQWxzbyBzcGl0IG91dCBhbiBlcnJvciBtZXNzYWdlIHdoZW4gdGhl IHNjcmVlbiBzaXplIGlzIHRvbyBzbWFsbC4KKworICAgICAgICAqIERvWW91RXZlbkJlbmNoL0Z1 bGwuaHRtbDoKKyAgICAgICAgKC5yZXBvcnRGaW5hbFJlc3VsdCk6CisKIDIwMTQtMDUtMDkgIFJ5 b3N1a2UgTml3YSAgPHJuaXdhQHdlYmtpdC5vcmc+CiAKICAgICAgICAgRFlFQmVuY2ggc2hvdWxk IHNob3cgOTV0aCBwZXJjZW50aWxlIHJpZ2h0IG5leHQgdG8gdGhlIG1lYW4gd2l0aCDCsQpJbmRl eDogUGVyZm9ybWFuY2VUZXN0cy9Eb1lvdUV2ZW5CZW5jaC9GdWxsLmh0bWwKPT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQot LS0gUGVyZm9ybWFuY2VUZXN0cy9Eb1lvdUV2ZW5CZW5jaC9GdWxsLmh0bWwJKHJldmlzaW9uIDE2 ODU2NCkKKysrIFBlcmZvcm1hbmNlVGVzdHMvRG9Zb3VFdmVuQmVuY2gvRnVsbC5odG1sCSh3b3Jr aW5nIGNvcHkpCkBAIC00Miw2ICs0MiwyMCBAQCB0ZCB7IHRleHQtYWxpZ246IGxlZnQ7IH0KICAg ICAgICAgcmVzdWx0Q29udGFpbmVyLmFwcGVuZENoaWxkKHJvdyk7CiAgICAgfQogCisgICAgZnVu Y3Rpb24gcmVwb3J0RmluYWxSZXN1bHQoKSB7CisgICAgICAgIHZhciBzdW0gPSB2YWx1ZXMucmVk dWNlKGZ1bmN0aW9uIChhLCBiKSB7IHJldHVybiBhICsgYjsgfSwgMCk7CisgICAgICAgIHZhciBh cml0aG1ldGljTWVhbiA9IHN1bSAvIHZhbHVlcy5sZW5ndGg7CisgICAgICAgIHZhciBtZWFuTGFi ZWwgPSBhcml0aG1ldGljTWVhbi50b0ZpeGVkKDIpICsgJyBtcyc7CisgICAgICAgIGlmICh3aW5k b3cuU3RhdGlzdGljcykgeworICAgICAgICAgICAgdmFyIGRlbHRhID0gU3RhdGlzdGljcy5jb25m aWRlbmNlSW50ZXJ2YWxEZWx0YSgwLjk1LCB2YWx1ZXMubGVuZ3RoLCBzdW0sIFN0YXRpc3RpY3Mu c3F1YXJlU3VtKHZhbHVlcykpOworICAgICAgICAgICAgdmFyIHBlcmNlbnREZWx0YSA9IGRlbHRh ICogMTAwIC8gYXJpdGhtZXRpY01lYW47CisgICAgICAgICAgICBpZiAocGVyY2VudERlbHRhID4g NS4wKQorICAgICAgICAgICAgICAgIHJldHVybiBhZGRSZXN1bHQoJ0Vycm9yJywgJ1RoZSB2YXJp YW5jZSB3YXMgdG9vIGhpZ2g6IFx4YjEgJyArIHBlcmNlbnREZWx0YS50b1ByZWNpc2lvbigyKSAr ICclIFJ1biBpdCBhZ2Fpbi4nKTsKKyAgICAgICAgICAgIG1lYW5MYWJlbCArPSAnIFx4YjEgJyAr IGRlbHRhLnRvRml4ZWQoMikgKyAnIG1zICgnICsgcGVyY2VudERlbHRhLnRvRml4ZWQoMikgKyAn JSknOworICAgICAgICB9CisgICAgICAgIGFkZFJlc3VsdCgnQXJpdGhtZXRpYyBNZWFuJywgbWVh bkxhYmVsKTsKKyAgICB9CisKICAgICB3aW5kb3cuYmVuY2htYXJrQ2xpZW50ID0gewogICAgICAg ICB3aWxsUnVuVGVzdDogZnVuY3Rpb24gKCkgewogICAgICAgICAgICAgaWYgKCFwcm9ncmVzcykg ewpAQCAtNjQsMjEgKzc4LDE4IEBAIHRkIHsgdGV4dC1hbGlnbjogbGVmdDsgfQogICAgICAgICAg ICAgYWRkUmVzdWx0KCdJdGVyYXRpb24gJyArIGl0ZXJhdGlvbk51bWJlciwgbWVhc3VyZWRWYWx1 ZXMudG90YWwudG9GaXhlZCgyKSArICcgbXMnKTsKICAgICAgICAgfSwKICAgICAgICAgZGlkRmlu aXNoTGFzdEl0ZXJhdGlvbjogZnVuY3Rpb24gKCkgewotICAgICAgICAgICAgdmFyIHN1bSA9IHZh bHVlcy5yZWR1Y2UoZnVuY3Rpb24gKGEsIGIpIHsgcmV0dXJuIGEgKyBiOyB9LCAwKTsKLSAgICAg ICAgICAgIHZhciBhcml0aG1ldGljTWVhbiA9IHN1bSAvIHZhbHVlcy5sZW5ndGg7Ci0gICAgICAg ICAgICB2YXIgbWVhbkxhYmVsID0gYXJpdGhtZXRpY01lYW4udG9GaXhlZCgyKSArICcgbXMnOwot ICAgICAgICAgICAgaWYgKHdpbmRvdy5TdGF0aXN0aWNzKSB7Ci0gICAgICAgICAgICAgICAgdmFy IGRlbHRhID0gU3RhdGlzdGljcy5jb25maWRlbmNlSW50ZXJ2YWxEZWx0YSgwLjk1LCB2YWx1ZXMu bGVuZ3RoLCBzdW0sIFN0YXRpc3RpY3Muc3F1YXJlU3VtKHZhbHVlcykpOwotICAgICAgICAgICAg ICAgIHZhciBwcmVjZW50RGVsdGEgPSBkZWx0YSAqIDEwMCAvIGFyaXRobWV0aWNNZWFuOwotICAg ICAgICAgICAgICAgIG1lYW5MYWJlbCArPSAnIFx4YjEgJyArIGRlbHRhLnRvRml4ZWQoMikgKyAn IG1zICgnICsgcHJlY2VudERlbHRhLnRvRml4ZWQoMikgKyAnJSknOwotICAgICAgICAgICAgfQot ICAgICAgICAgICAgYWRkUmVzdWx0KCdBcml0aG1ldGljIE1lYW4nLCBtZWFuTGFiZWwpOworICAg ICAgICAgICAgcmVwb3J0RmluYWxSZXN1bHQoKTsKICAgICAgICAgICAgIHByb2dyZXNzQ29udGFp bmVyLnBhcmVudE5vZGUucmVtb3ZlQ2hpbGQocHJvZ3Jlc3NDb250YWluZXIpOwogICAgICAgICB9 CiAgICAgfQogfSkoKTsKIAogZnVuY3Rpb24gc3RhcnRUZXN0KCkgeworICAgIGlmICh3aW5kb3cu aW5uZXJXaWR0aCA8IDgwMCB8fCB3aW5kb3cuaW5uZXJIZWlnaHQgPCA2MDApIHsKKyAgICAgICAg ZG9jdW1lbnQuYm9keS5pbm5lclRleHQgPSAnV2luZG93IGlzIHRvbyBzbWFsbCAoJyArIHdpbmRv dy5pbm5lcldpZHRoICsgJ3B4IGJ5ICcgKyB3aW5kb3cuaW5uZXJIZWlnaHQgKyAncHgpLiBNYWtl IHN1cmUgaXRcJ3MgYXQgbGVhc3QgODAwcHggYnkgNjAwcHguJzsKKyAgICAgICAgcmV0dXJuOwor ICAgIH0KKwogICAgIHZhciBpdGVyYXRpb25Db3VudCA9IDU7CiAgICAgYmVuY2htYXJrQ2xpZW50 LnRlc3RzQ291bnQgPSBpdGVyYXRpb25Db3VudCAqIFN1aXRlcy5yZWR1Y2UoZnVuY3Rpb24gKHRl c3RzQ291bnQsIHN1aXRlKSB7IHJldHVybiB0ZXN0c0NvdW50ICsgc3VpdGUudGVzdHMubGVuZ3Ro OyB9LCAwKTsKICAgICB2YXIgcnVubmVyID0gbmV3IEJlbmNobWFya1J1bm5lcihTdWl0ZXMsIGJl bmNobWFya0NsaWVudCk7Cg==