171617 2017-05-03 14:07:49 -0700 [iOS wk2] Layout Test perf/object-keys.html is flaky 2017-07-18 08:30:23 -0700 1 1 1 Unclassified WebKit JavaScriptCore WebKit Nightly Build Unspecified Unspecified REOPENED InRadar P2 Normal --- 1 jlewis3 saam ap caitp commit-queue joepeck mark.lam rniwa ryanhaddad saam webkit-bug-importer ysuzuki oldest_to_newest 1304234 0 jlewis3 2017-05-03 14:07:49 -0700 The following layout test is flaky on iOS-wk2 release perf/object-keys.html History: https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=perf%2Fobject-keys.html Build log: https://build.webkit.org/builders/Apple%20iOS%2010%20Simulator%20Release%20WK2%20(Tests)/builds/1133 Layout results: https://build.webkit.org/results/Apple%20iOS%2010%20Simulator%20Release%20WK2%20(Tests)/r216129%20(1133)/results.html First Failure started on 4/28/2017 @ 5:30 1304239 1 308952 jlewis3 2017-05-03 14:18:00 -0700 Created attachment 308952 Mark test as flaky 1304242 2 308952 ryanhaddad 2017-05-03 14:20:18 -0700 Comment on attachment 308952 Mark test as flaky Clearing flags on attachment: 308952 Committed r216141: <http://trac.webkit.org/changeset/216141> 1304379 3 ap 2017-05-03 16:53:04 -0700 There are lots of numbers in the failure output, I can't immediately tell what makes the test think that it failed. But I don't see how we can possibly have anything performance sensitive in layout tests. Tests that Object.keys is linear. FAIL: got >=O(n^2) expected O(n) DEBUG LOG: run iteration. magnitude 2 jsObjectCountBefore 48814 jsObjectCountAfter 48137 run iteration. magnitude 4 jsObjectCountBefore 48137 jsObjectCountAfter 48208 run iteration. magnitude 8 jsObjectCountBefore 48208 jsObjectCountAfter 48219 run iteration. magnitude 16 jsObjectCountBefore 48219 jsObjectCountAfter 48229 run iteration. magnitude 32 jsObjectCountBefore 48229 jsObjectCountAfter 48233 run iteration. magnitude 64 jsObjectCountBefore 48233 jsObjectCountAfter 48243 run iteration. magnitude 128 jsObjectCountBefore 48243 jsObjectCountAfter 48247 run iteration. magnitude 256 jsObjectCountBefore 48247 jsObjectCountAfter 48257 magnitudes: 2,4,8,16,32,64,128,256 iterations: 70102,53264,25294,15509,5991,3043,1285,469 numPoints 8 slope 0.00020386006481621003 intercept -0.0019242057485269765 rSquared 0.9794566908307417 numPoints 8 slope 0.0084124576373827 intercept -0.015167947434686677 rSquared 0.6134112576136115 numPoints 8 slope 1.047268969975335 intercept -9.02971987395735 rSquared 0.9846587163231699 rSquared 0.9794566908307417 rSquaredXLog 0.6134112576136115 rSquaredXYLog 0.9846587163231699 numTriesLeft: 2 run iteration. magnitude 2 jsObjectCountBefore 48257 jsObjectCountAfter 48308 run iteration. magnitude 4 jsObjectCountBefore 48308 jsObjectCountAfter 48315 run iteration. magnitude 8 jsObjectCountBefore 48315 jsObjectCountAfter 48321 run iteration. magnitude 16 jsObjectCountBefore 48321 jsObjectCountAfter 48331 run iteration. magnitude 32 jsObjectCountBefore 48331 jsObjectCountAfter 48337 run iteration. magnitude 64 jsObjectCountBefore 48337 jsObjectCountAfter 48347 run iteration. magnitude 128 jsObjectCountBefore 48347 jsObjectCountAfter 48351 run iteration. magnitude 256 jsObjectCountBefore 48351 jsObjectCountAfter 48361 magnitudes: 2,4,8,16,32,64,128,256 iterations: 69298,55624,27296,16397,6312,3196,1249,433 numPoints 8 slope 0.0002203330103331781 intercept -0.0024479191232797924 rSquared 0.9727058472458164 numPoints 8 slope 0.008975963694698264 intercept -0.016399177389588078 rSquared 0.5937034944864956 numPoints 8 slope 1.0631040977767605 intercept -9.09869214802648 rSquared 0.979064950229674 rSquared 0.9727058472458164 rSquaredXLog 0.5937034944864956 rSquaredXYLog 0.979064950229674 numTriesLeft: 1 run iteration. magnitude 2 jsObjectCountBefore 48361 jsObjectCountAfter 48403 run iteration. magnitude 4 jsObjectCountBefore 48403 jsObjectCountAfter 48410 run iteration. magnitude 8 jsObjectCountBefore 48410 jsObjectCountAfter 48416 run iteration. magnitude 16 jsObjectCountBefore 48416 jsObjectCountAfter 48428 run iteration. magnitude 32 jsObjectCountBefore 48428 jsObjectCountAfter 48432 run iteration. magnitude 64 jsObjectCountBefore 48432 jsObjectCountAfter 48442 run iteration. magnitude 128 jsObjectCountBefore 48442 jsObjectCountAfter 48446 run iteration. magnitude 256 jsObjectCountBefore 48446 jsObjectCountAfter 48457 magnitudes: 2,4,8,16,32,64,128,256 iterations: 68654,53887,28867,14640,5956,3187,1064,397 numPoints 8 slope 0.00024248818860952778 intercept -0.002719607248953147 rSquared 0.976910778052751 numPoints 8 slope 0.00990725037810828 intercept -0.01816330722518818 rSquared 0.5997427960173778 numPoints 8 slope 1.0855382556763826 intercept -9.117864568033017 rSquared 0.9791873624835375 rSquared 0.976910778052751 rSquaredXLog 0.5997427960173778 rSquaredXYLog 0.9791873624835375 1304390 4 joepeck 2017-05-03 16:59:16 -0700 There was just a recent change to Object.keys in bug 171291. Could be related. 1304432 5 webkit-bug-importer 2017-05-03 17:46:38 -0700 <rdar://problem/31977874> 1319707 6 saam 2017-06-15 14:23:57 -0700 I think this is just pure flakiness. For a few different reasons: 1. We run this alongside other things on the machine. 2. We use objects with very few keys, so there could just be a lot variance between runs. I'll try to make it more stable. 1319724 7 313011 saam 2017-06-15 14:38:46 -0700 Created attachment 313011 patch 1319728 8 313011 mark.lam 2017-06-15 14:39:36 -0700 Comment on attachment 313011 patch r=me 1319741 9 313011 commit-queue 2017-06-15 15:26:45 -0700 Comment on attachment 313011 patch Clearing flags on attachment: 313011 Committed r218364: <http://trac.webkit.org/changeset/218364> 1319742 10 commit-queue 2017-06-15 15:26:47 -0700 All reviewed patches have been landed. Closing bug. 1321337 11 jlewis3 2017-06-20 17:51:51 -0700 After the second patch: https://trac.webkit.org/changeset/218364/webkit Flakiness of the test did not improve and the test is still failing intermittently at a high rate on iOS Simulator according to the Dashboard. https://build.webkit.org/results/Apple%20iOS%2010%20Simulator%20Release%20WK2%20(Tests)/r218614%20(2428)/results.html new diff: --- /Volumes/Data/slave/ios-simulator-10-release-tests-wk2/build/layout-test-results/perf/object-keys-expected.txt +++ /Volumes/Data/slave/ios-simulator-10-release-tests-wk2/build/layout-test-results/perf/object-keys-actual.txt @@ -1,3 +1,44 @@ Tests that Object.keys is linear. -PASS +FAIL: got >=O(n^2) expected O(n) +DEBUG LOG: +run iteration. magnitude 256 jsObjectCountBefore 93783 jsObjectCountAfter 93108 +run iteration. magnitude 512 jsObjectCountBefore 93108 jsObjectCountAfter 93180 +run iteration. magnitude 1024 jsObjectCountBefore 93180 jsObjectCountAfter 93189 +run iteration. magnitude 2048 jsObjectCountBefore 93189 jsObjectCountAfter 93196 +run iteration. magnitude 4096 jsObjectCountBefore 93196 jsObjectCountAfter 93205 +run iteration. magnitude 8192 jsObjectCountBefore 93205 jsObjectCountAfter 93212 +magnitudes: 256,512,1024,2048,4096,8192 +iterations: 487,176,92,35,21,8 +numPoints 6 slope 0.0003802862605442898 intercept -0.10639593634290612 rSquared 0.9866968019879888 +numPoints 6 slope 0.781370446978062 intercept -4.7710360521047885 rSquared 0.7596636601354393 +numPoints 6 slope 1.1495574667742858 intercept -9.240963726164283 rSquared 0.9940214494669775 +rSquared 0.9866968019879888 rSquaredXLog 0.7596636601354393 rSquaredXYLog 0.9940214494669775 +numTriesLeft: 2 +run iteration. magnitude 256 jsObjectCountBefore 93212 jsObjectCountAfter 93265 +run iteration. magnitude 512 jsObjectCountBefore 93265 jsObjectCountAfter 93273 +run iteration. magnitude 1024 jsObjectCountBefore 93273 jsObjectCountAfter 93279 +run iteration. magnitude 2048 jsObjectCountBefore 93279 jsObjectCountAfter 93286 +run iteration. magnitude 4096 jsObjectCountBefore 93286 jsObjectCountAfter 93296 +run iteration. magnitude 8192 jsObjectCountBefore 93296 jsObjectCountAfter 93305 +magnitudes: 256,512,1024,2048,4096,8192 +iterations: 415,179,86,40,16,8 +numPoints 6 slope 0.0003918453220597 intercept -0.08609631857259037 rSquared 0.9980818570007993 +numPoints 6 slope 0.821371756110995 intercept -5.0107970207451435 rSquared 0.7997629788019797 +numPoints 6 slope 1.144017760650845 intercept -9.142490927567392 rSquared 0.9990380600703198 +rSquared 0.9980818570007993 rSquaredXLog 0.7997629788019797 rSquaredXYLog 0.9990380600703198 +numTriesLeft: 1 +run iteration. magnitude 256 jsObjectCountBefore 93305 jsObjectCountAfter 93349 +run iteration. magnitude 512 jsObjectCountBefore 93349 jsObjectCountAfter 93357 +run iteration. magnitude 1024 jsObjectCountBefore 93357 jsObjectCountAfter 93369 +run iteration. magnitude 2048 jsObjectCountBefore 93369 jsObjectCountAfter 93381 +run iteration. magnitude 4096 jsObjectCountBefore 93381 jsObjectCountAfter 93386 +run iteration. magnitude 8192 jsObjectCountBefore 93386 jsObjectCountAfter 93393 +magnitudes: 256,512,1024,2048,4096,8192 +iterations: 459,169,85,36,18,7 +numPoints 6 slope 0.0004386654990338807 intercept -0.15392039859634 rSquared 0.988900895680829 +numPoints 6 slope 0.8948014337703841 intercept -5.487192992472122 rSquared 0.7503849137821414 +numPoints 6 slope 1.1744944670284758 intercept -9.349381727854512 rSquared 0.99752152057426 +rSquared 0.988900895680829 rSquaredXLog 0.7503849137821414 rSquaredXYLog 0.99752152057426 + + 1322073 12 ryanhaddad 2017-06-22 15:01:27 -0700 Reopening per Matt's comment. 1322074 13 ryanhaddad 2017-06-22 15:04:20 -0700 Marked test as flaky in https://trac.webkit.org/r218722 308952 2017-05-03 14:18:00 -0700 2017-05-03 14:20:30 -0700 Mark test as flaky bug-171617-20170503141757.patch text/plain 1311 jlewis3 U3VidmVyc2lvbiBSZXZpc2lvbjogMjE2MTM5CmRpZmYgLS1naXQgYS9MYXlvdXRUZXN0cy9DaGFu Z2VMb2cgYi9MYXlvdXRUZXN0cy9DaGFuZ2VMb2cKaW5kZXggNGU2N2E0YWMxNWM1OGQ3ZjhiYzk3 ODg3OWQwYTY3NDk4YTVjYWQ2ZS4uZGVjZDIwOGVlNmIyYWUwYzFkM2E4MmQ1YWNhYzE5ZjdiMjQw NjY0NSAxMDA2NDQKLS0tIGEvTGF5b3V0VGVzdHMvQ2hhbmdlTG9nCisrKyBiL0xheW91dFRlc3Rz L0NoYW5nZUxvZwpAQCAtMSwzICsxLDEyIEBACisyMDE3LTA1LTAzICBNYXR0IExld2lzICA8amxl d2lzM0BhcHBsZS5jb20+CisKKyAgICAgICAgTWFyayBwZXJmL29iamVjdC1rZXlzLmh0bWwgYXMg Zmxha3kKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTE3 MTYxNworCisgICAgICAgIFVucmV2aWV3ZWQgdGVzdCBnYXJkZW5pbmcuCisKKyAgICAgICAgKiBw bGF0Zm9ybS9pb3Mtd2syL1Rlc3RFeHBlY3RhdGlvbnM6CisKIDIwMTctMDUtMDMgIEtlaXRoIE1p bGxlciAgPGtlaXRoX21pbGxlckBhcHBsZS5jb20+CiAKICAgICAgICAgRGlmZmVyZW50IGJlaGF2 aW91ciB3aXRoIHRoZSAuc29ydChjYWxsYmFjaykgbWV0aG9kICh1bmxpa2UgRmlyZWZveCAmIENo cm9tZSkKZGlmZiAtLWdpdCBhL0xheW91dFRlc3RzL3BsYXRmb3JtL2lvcy13azIvVGVzdEV4cGVj dGF0aW9ucyBiL0xheW91dFRlc3RzL3BsYXRmb3JtL2lvcy13azIvVGVzdEV4cGVjdGF0aW9ucwpp bmRleCA2NzA4YWMwZDA0NzA2YWUxOWVmYjYxOGU0MDMwNTA2ZDk3MDAwY2EyLi4wYjBjNmU1NmFh NmIyYjNmYmRlYzg5ZjJlYzRhYTMzYTliMzhlNWEwIDEwMDY0NAotLS0gYS9MYXlvdXRUZXN0cy9w bGF0Zm9ybS9pb3Mtd2syL1Rlc3RFeHBlY3RhdGlvbnMKKysrIGIvTGF5b3V0VGVzdHMvcGxhdGZv cm0vaW9zLXdrMi9UZXN0RXhwZWN0YXRpb25zCkBAIC0xOTYyLDMgKzE5NjIsNSBAQCB3ZWJraXQu b3JnL2IvMTcxNTU5IHN2Zy9hbmltYXRpb25zL2FuaW1hdGVkLXN2Zy1pbWFnZS1yZW1vdmVkLWZy b20tZG9jdW1lbnQtcGF1cwogd2Via2l0Lm9yZy9iLzE3MTU2NCBodHRwL3Rlc3RzL2xvY2FsL2Js b2Ivc2VuZC1zbGljZWQtZGF0YS1ibG9iLmh0bWwgWyBQYXNzIEZhaWx1cmUgVGltZW91dCBdCiAK IHdlYmtpdC5vcmcvYi8xNzE2MTMgaHR0cC90ZXN0cy94bWxodHRwcmVxdWVzdC9zdXBwb3J0ZWQt eG1sLWNvbnRlbnQtdHlwZXMuaHRtbCBbIFBhc3MgVGltZW91dCBdCisKK3dlYmtpdC5vcmcvYi8x NzE2MTcgWyBSZWxlYXNlIF0gcGVyZi9vYmplY3Qta2V5cy5odG1sIFsgUGFzcyBGYWlsdXJlIF0K 313011 2017-06-15 14:38:46 -0700 2017-06-15 15:26:45 -0700 patch b-backup.diff text/plain 2013 saam SW5kZXg6IExheW91dFRlc3RzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBMYXlvdXRUZXN0cy9D aGFuZ2VMb2cJKHJldmlzaW9uIDIxODM1NykKKysrIExheW91dFRlc3RzL0NoYW5nZUxvZwkod29y a2luZyBjb3B5KQpAQCAtMSwzICsxLDE3IEBACisyMDE3LTA2LTE1ICBTYWFtIEJhcmF0aSAgPHNi YXJhdGlAYXBwbGUuY29tPgorCisgICAgICAgIFtpT1Mgd2syXSBMYXlvdXQgVGVzdCBwZXJmL29i amVjdC1rZXlzLmh0bWwgaXMgZmxha3kKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcv c2hvd19idWcuY2dpP2lkPTE3MTYxNworICAgICAgICA8cmRhcjovL3Byb2JsZW0vMzE5Nzc4NzQ+ CisKKyAgICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISkuCisKKyAgICAgICAgSW5jcmVh c2UgdGhlIHN0YXJ0aW5nIG1hZ25pdHVkZSBvZiB0aGUgb2JqZWN0LWtleXMgdGVzdCB0bworICAg ICAgICByZW1vdmUgdmFyaWFuY2UgZnJvbSBoYXZpbmcgb2JqZWN0cyB3aXRoIGEgdmVyeSBmZXcg bnVtYmVyIG9mIGtleXMuCisKKyAgICAgICAgKiBwZXJmL29iamVjdC1rZXlzLmh0bWw6CisgICAg ICAgICogcGxhdGZvcm0vaW9zLXdrMi9UZXN0RXhwZWN0YXRpb25zOgorCiAyMDE3LTA2LTE1ICBN YXR0IExld2lzICA8amxld2lzM0BhcHBsZS5jb20+CiAKICAgICAgICAgUkVHUkVTU0lPTiAocjIx ODIyOCk6IExheW91dCBUZXN0IGVkaXRpbmcvZXhlY0NvbW1hbmQvc3RyaWtldGhyb3VnaC11c2Vz LXN0cmlrZS10YWcuaHRtbCBpcyBmYWlsaW5nCkluZGV4OiBMYXlvdXRUZXN0cy9wZXJmL29iamVj dC1rZXlzLmh0bWwKPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PQotLS0gTGF5b3V0VGVzdHMvcGVyZi9vYmplY3Qta2V5cy5o dG1sCShyZXZpc2lvbiAyMTgzNTUpCisrKyBMYXlvdXRUZXN0cy9wZXJmL29iamVjdC1rZXlzLmh0 bWwJKHdvcmtpbmcgY29weSkKQEAgLTEzLDYgKzEzLDkgQEAgZnVuY3Rpb24gdGVzdChtYWduaXR1 ZGUpCiAgICAgT2JqZWN0LmtleXMob2JqKTsKIH0KIAorTWFnbml0dWRlLmluaXRpYWxNYWduaXR1 ZGUgPSA4OworTWFnbml0dWRlLm51bVBvaW50cyA9IDY7CisKIE1hZ25pdHVkZS5kZXNjcmlwdGlv bigiVGVzdHMgdGhhdCBPYmplY3Qua2V5cyBpcyBsaW5lYXIuIik7CiBNYWduaXR1ZGUucnVuKHNl dHVwRnVuY3Rpb24sIHRlc3QsIE1hZ25pdHVkZS5MSU5FQVIpOwogPC9zY3JpcHQ+CkluZGV4OiBM YXlvdXRUZXN0cy9wbGF0Zm9ybS9pb3Mtd2syL1Rlc3RFeHBlY3RhdGlvbnMKPT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQot LS0gTGF5b3V0VGVzdHMvcGxhdGZvcm0vaW9zLXdrMi9UZXN0RXhwZWN0YXRpb25zCShyZXZpc2lv biAyMTgzNTUpCisrKyBMYXlvdXRUZXN0cy9wbGF0Zm9ybS9pb3Mtd2syL1Rlc3RFeHBlY3RhdGlv bnMJKHdvcmtpbmcgY29weSkKQEAgLTE5MjksOCArMTkyOSw2IEBAIHdlYmtpdC5vcmcvYi8xNzE1 NjQgaHR0cC90ZXN0cy9sb2NhbC9ibG8KIAogd2Via2l0Lm9yZy9iLzE3MTYxMyBodHRwL3Rlc3Rz L3htbGh0dHByZXF1ZXN0L3N1cHBvcnRlZC14bWwtY29udGVudC10eXBlcy5odG1sIFsgUGFzcyBU aW1lb3V0IF0KIAotd2Via2l0Lm9yZy9iLzE3MTYxNyBbIFJlbGVhc2UgXSBwZXJmL29iamVjdC1r ZXlzLmh0bWwgWyBQYXNzIEZhaWx1cmUgXQotCiB3ZWJraXQub3JnL2IvMTcxMzMxIGh0dHAvdGVz dHMvcHJlbG9hZC9zaW5nbGVfZG93bmxvYWRfcHJlbG9hZC5odG1sIFsgUGFzcyBGYWlsdXJlIF0K IAogd2Via2l0Lm9yZy9iLzE3MDY5OSBbIFJlbGVhc2UgXSBpbXBvcnRlZC93M2Mvd2ViLXBsYXRm b3JtLXRlc3RzL2h0bWwvd2ViYXBwYXBpcy90aW1lcnMvbmVnYXRpdmUtc2V0dGltZW91dC5odG1s IFsgUGFzcyBGYWlsdXJlIF0K