The following layout test is flaky on iOS-wk2 release perf/object-keys.html History: https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=perf%2Fobject-keys.html Build log: https://build.webkit.org/builders/Apple%20iOS%2010%20Simulator%20Release%20WK2%20(Tests)/builds/1133 Layout results: https://build.webkit.org/results/Apple%20iOS%2010%20Simulator%20Release%20WK2%20(Tests)/r216129%20(1133)/results.html First Failure started on 4/28/2017 @ 5:30
Created attachment 308952 [details] Mark test as flaky
Comment on attachment 308952 [details] Mark test as flaky Clearing flags on attachment: 308952 Committed r216141: <http://trac.webkit.org/changeset/216141>
There are lots of numbers in the failure output, I can't immediately tell what makes the test think that it failed. But I don't see how we can possibly have anything performance sensitive in layout tests. Tests that Object.keys is linear. FAIL: got >=O(n^2) expected O(n) DEBUG LOG: run iteration. magnitude 2 jsObjectCountBefore 48814 jsObjectCountAfter 48137 run iteration. magnitude 4 jsObjectCountBefore 48137 jsObjectCountAfter 48208 run iteration. magnitude 8 jsObjectCountBefore 48208 jsObjectCountAfter 48219 run iteration. magnitude 16 jsObjectCountBefore 48219 jsObjectCountAfter 48229 run iteration. magnitude 32 jsObjectCountBefore 48229 jsObjectCountAfter 48233 run iteration. magnitude 64 jsObjectCountBefore 48233 jsObjectCountAfter 48243 run iteration. magnitude 128 jsObjectCountBefore 48243 jsObjectCountAfter 48247 run iteration. magnitude 256 jsObjectCountBefore 48247 jsObjectCountAfter 48257 magnitudes: 2,4,8,16,32,64,128,256 iterations: 70102,53264,25294,15509,5991,3043,1285,469 numPoints 8 slope 0.00020386006481621003 intercept -0.0019242057485269765 rSquared 0.9794566908307417 numPoints 8 slope 0.0084124576373827 intercept -0.015167947434686677 rSquared 0.6134112576136115 numPoints 8 slope 1.047268969975335 intercept -9.02971987395735 rSquared 0.9846587163231699 rSquared 0.9794566908307417 rSquaredXLog 0.6134112576136115 rSquaredXYLog 0.9846587163231699 numTriesLeft: 2 run iteration. magnitude 2 jsObjectCountBefore 48257 jsObjectCountAfter 48308 run iteration. magnitude 4 jsObjectCountBefore 48308 jsObjectCountAfter 48315 run iteration. magnitude 8 jsObjectCountBefore 48315 jsObjectCountAfter 48321 run iteration. magnitude 16 jsObjectCountBefore 48321 jsObjectCountAfter 48331 run iteration. magnitude 32 jsObjectCountBefore 48331 jsObjectCountAfter 48337 run iteration. magnitude 64 jsObjectCountBefore 48337 jsObjectCountAfter 48347 run iteration. magnitude 128 jsObjectCountBefore 48347 jsObjectCountAfter 48351 run iteration. magnitude 256 jsObjectCountBefore 48351 jsObjectCountAfter 48361 magnitudes: 2,4,8,16,32,64,128,256 iterations: 69298,55624,27296,16397,6312,3196,1249,433 numPoints 8 slope 0.0002203330103331781 intercept -0.0024479191232797924 rSquared 0.9727058472458164 numPoints 8 slope 0.008975963694698264 intercept -0.016399177389588078 rSquared 0.5937034944864956 numPoints 8 slope 1.0631040977767605 intercept -9.09869214802648 rSquared 0.979064950229674 rSquared 0.9727058472458164 rSquaredXLog 0.5937034944864956 rSquaredXYLog 0.979064950229674 numTriesLeft: 1 run iteration. magnitude 2 jsObjectCountBefore 48361 jsObjectCountAfter 48403 run iteration. magnitude 4 jsObjectCountBefore 48403 jsObjectCountAfter 48410 run iteration. magnitude 8 jsObjectCountBefore 48410 jsObjectCountAfter 48416 run iteration. magnitude 16 jsObjectCountBefore 48416 jsObjectCountAfter 48428 run iteration. magnitude 32 jsObjectCountBefore 48428 jsObjectCountAfter 48432 run iteration. magnitude 64 jsObjectCountBefore 48432 jsObjectCountAfter 48442 run iteration. magnitude 128 jsObjectCountBefore 48442 jsObjectCountAfter 48446 run iteration. magnitude 256 jsObjectCountBefore 48446 jsObjectCountAfter 48457 magnitudes: 2,4,8,16,32,64,128,256 iterations: 68654,53887,28867,14640,5956,3187,1064,397 numPoints 8 slope 0.00024248818860952778 intercept -0.002719607248953147 rSquared 0.976910778052751 numPoints 8 slope 0.00990725037810828 intercept -0.01816330722518818 rSquared 0.5997427960173778 numPoints 8 slope 1.0855382556763826 intercept -9.117864568033017 rSquared 0.9791873624835375 rSquared 0.976910778052751 rSquaredXLog 0.5997427960173778 rSquaredXYLog 0.9791873624835375
There was just a recent change to Object.keys in bug 171291. Could be related.
<rdar://problem/31977874>
I think this is just pure flakiness. For a few different reasons: 1. We run this alongside other things on the machine. 2. We use objects with very few keys, so there could just be a lot variance between runs. I'll try to make it more stable.
Created attachment 313011 [details] patch
Comment on attachment 313011 [details] patch r=me
Comment on attachment 313011 [details] patch Clearing flags on attachment: 313011 Committed r218364: <http://trac.webkit.org/changeset/218364>
All reviewed patches have been landed. Closing bug.
After the second patch: https://trac.webkit.org/changeset/218364/webkit Flakiness of the test did not improve and the test is still failing intermittently at a high rate on iOS Simulator according to the Dashboard. https://build.webkit.org/results/Apple%20iOS%2010%20Simulator%20Release%20WK2%20(Tests)/r218614%20(2428)/results.html new diff: --- /Volumes/Data/slave/ios-simulator-10-release-tests-wk2/build/layout-test-results/perf/object-keys-expected.txt +++ /Volumes/Data/slave/ios-simulator-10-release-tests-wk2/build/layout-test-results/perf/object-keys-actual.txt @@ -1,3 +1,44 @@ Tests that Object.keys is linear. -PASS +FAIL: got >=O(n^2) expected O(n) +DEBUG LOG: +run iteration. magnitude 256 jsObjectCountBefore 93783 jsObjectCountAfter 93108 +run iteration. magnitude 512 jsObjectCountBefore 93108 jsObjectCountAfter 93180 +run iteration. magnitude 1024 jsObjectCountBefore 93180 jsObjectCountAfter 93189 +run iteration. magnitude 2048 jsObjectCountBefore 93189 jsObjectCountAfter 93196 +run iteration. magnitude 4096 jsObjectCountBefore 93196 jsObjectCountAfter 93205 +run iteration. magnitude 8192 jsObjectCountBefore 93205 jsObjectCountAfter 93212 +magnitudes: 256,512,1024,2048,4096,8192 +iterations: 487,176,92,35,21,8 +numPoints 6 slope 0.0003802862605442898 intercept -0.10639593634290612 rSquared 0.9866968019879888 +numPoints 6 slope 0.781370446978062 intercept -4.7710360521047885 rSquared 0.7596636601354393 +numPoints 6 slope 1.1495574667742858 intercept -9.240963726164283 rSquared 0.9940214494669775 +rSquared 0.9866968019879888 rSquaredXLog 0.7596636601354393 rSquaredXYLog 0.9940214494669775 +numTriesLeft: 2 +run iteration. magnitude 256 jsObjectCountBefore 93212 jsObjectCountAfter 93265 +run iteration. magnitude 512 jsObjectCountBefore 93265 jsObjectCountAfter 93273 +run iteration. magnitude 1024 jsObjectCountBefore 93273 jsObjectCountAfter 93279 +run iteration. magnitude 2048 jsObjectCountBefore 93279 jsObjectCountAfter 93286 +run iteration. magnitude 4096 jsObjectCountBefore 93286 jsObjectCountAfter 93296 +run iteration. magnitude 8192 jsObjectCountBefore 93296 jsObjectCountAfter 93305 +magnitudes: 256,512,1024,2048,4096,8192 +iterations: 415,179,86,40,16,8 +numPoints 6 slope 0.0003918453220597 intercept -0.08609631857259037 rSquared 0.9980818570007993 +numPoints 6 slope 0.821371756110995 intercept -5.0107970207451435 rSquared 0.7997629788019797 +numPoints 6 slope 1.144017760650845 intercept -9.142490927567392 rSquared 0.9990380600703198 +rSquared 0.9980818570007993 rSquaredXLog 0.7997629788019797 rSquaredXYLog 0.9990380600703198 +numTriesLeft: 1 +run iteration. magnitude 256 jsObjectCountBefore 93305 jsObjectCountAfter 93349 +run iteration. magnitude 512 jsObjectCountBefore 93349 jsObjectCountAfter 93357 +run iteration. magnitude 1024 jsObjectCountBefore 93357 jsObjectCountAfter 93369 +run iteration. magnitude 2048 jsObjectCountBefore 93369 jsObjectCountAfter 93381 +run iteration. magnitude 4096 jsObjectCountBefore 93381 jsObjectCountAfter 93386 +run iteration. magnitude 8192 jsObjectCountBefore 93386 jsObjectCountAfter 93393 +magnitudes: 256,512,1024,2048,4096,8192 +iterations: 459,169,85,36,18,7 +numPoints 6 slope 0.0004386654990338807 intercept -0.15392039859634 rSquared 0.988900895680829 +numPoints 6 slope 0.8948014337703841 intercept -5.487192992472122 rSquared 0.7503849137821414 +numPoints 6 slope 1.1744944670284758 intercept -9.349381727854512 rSquared 0.99752152057426 +rSquared 0.988900895680829 rSquaredXLog 0.7503849137821414 rSquaredXYLog 0.99752152057426 + +
Reopening per Matt's comment.
Marked test as flaky in https://trac.webkit.org/r218722