Bug 171617 – [iOS wk2] Layout Test perf/object-keys.html is flaky

• Status: REOPENED
• Component: JavaScriptCore

[Matt Lewis]

The following layout test is flaky on iOS-wk2 release

perf/object-keys.html

History:
https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=perf%2Fobject-keys.html

Build log:
https://build.webkit.org/builders/Apple%20iOS%2010%20Simulator%20Release%20WK2%20(Tests)/builds/1133

Layout results:
https://build.webkit.org/results/Apple%20iOS%2010%20Simulator%20Release%20WK2%20(Tests)/r216129%20(1133)/results.html

First Failure started on 4/28/2017 @ 5:30

[Matt Lewis]

Created attachment 308952 [details]
Mark test as flaky

[Ryan Haddad]

Comment on attachment 308952 [details]
Mark test as flaky

Clearing flags on attachment: 308952

Committed r216141: <http://trac.webkit.org/changeset/216141>

[Alexey Proskuryakov]

There are lots of numbers in the failure output, I can't immediately tell what makes the test think that it failed. But I don't see how we can possibly have anything performance sensitive in layout tests.


Tests that Object.keys is linear.
FAIL: got >=O(n^2) expected O(n)

DEBUG LOG:
run iteration. magnitude 2 jsObjectCountBefore 48814 jsObjectCountAfter 48137
run iteration. magnitude 4 jsObjectCountBefore 48137 jsObjectCountAfter 48208
run iteration. magnitude 8 jsObjectCountBefore 48208 jsObjectCountAfter 48219
run iteration. magnitude 16 jsObjectCountBefore 48219 jsObjectCountAfter 48229
run iteration. magnitude 32 jsObjectCountBefore 48229 jsObjectCountAfter 48233
run iteration. magnitude 64 jsObjectCountBefore 48233 jsObjectCountAfter 48243
run iteration. magnitude 128 jsObjectCountBefore 48243 jsObjectCountAfter 48247
run iteration. magnitude 256 jsObjectCountBefore 48247 jsObjectCountAfter 48257
magnitudes: 2,4,8,16,32,64,128,256
iterations: 70102,53264,25294,15509,5991,3043,1285,469
numPoints 8 slope 0.00020386006481621003 intercept -0.0019242057485269765 rSquared 0.9794566908307417
numPoints 8 slope 0.0084124576373827 intercept -0.015167947434686677 rSquared 0.6134112576136115
numPoints 8 slope 1.047268969975335 intercept -9.02971987395735 rSquared 0.9846587163231699
rSquared 0.9794566908307417 rSquaredXLog 0.6134112576136115 rSquaredXYLog 0.9846587163231699
numTriesLeft: 2
run iteration. magnitude 2 jsObjectCountBefore 48257 jsObjectCountAfter 48308
run iteration. magnitude 4 jsObjectCountBefore 48308 jsObjectCountAfter 48315
run iteration. magnitude 8 jsObjectCountBefore 48315 jsObjectCountAfter 48321
run iteration. magnitude 16 jsObjectCountBefore 48321 jsObjectCountAfter 48331
run iteration. magnitude 32 jsObjectCountBefore 48331 jsObjectCountAfter 48337
run iteration. magnitude 64 jsObjectCountBefore 48337 jsObjectCountAfter 48347
run iteration. magnitude 128 jsObjectCountBefore 48347 jsObjectCountAfter 48351
run iteration. magnitude 256 jsObjectCountBefore 48351 jsObjectCountAfter 48361
magnitudes: 2,4,8,16,32,64,128,256
iterations: 69298,55624,27296,16397,6312,3196,1249,433
numPoints 8 slope 0.0002203330103331781 intercept -0.0024479191232797924 rSquared 0.9727058472458164
numPoints 8 slope 0.008975963694698264 intercept -0.016399177389588078 rSquared 0.5937034944864956
numPoints 8 slope 1.0631040977767605 intercept -9.09869214802648 rSquared 0.979064950229674
rSquared 0.9727058472458164 rSquaredXLog 0.5937034944864956 rSquaredXYLog 0.979064950229674
numTriesLeft: 1
run iteration. magnitude 2 jsObjectCountBefore 48361 jsObjectCountAfter 48403
run iteration. magnitude 4 jsObjectCountBefore 48403 jsObjectCountAfter 48410
run iteration. magnitude 8 jsObjectCountBefore 48410 jsObjectCountAfter 48416
run iteration. magnitude 16 jsObjectCountBefore 48416 jsObjectCountAfter 48428
run iteration. magnitude 32 jsObjectCountBefore 48428 jsObjectCountAfter 48432
run iteration. magnitude 64 jsObjectCountBefore 48432 jsObjectCountAfter 48442
run iteration. magnitude 128 jsObjectCountBefore 48442 jsObjectCountAfter 48446
run iteration. magnitude 256 jsObjectCountBefore 48446 jsObjectCountAfter 48457
magnitudes: 2,4,8,16,32,64,128,256
iterations: 68654,53887,28867,14640,5956,3187,1064,397
numPoints 8 slope 0.00024248818860952778 intercept -0.002719607248953147 rSquared 0.976910778052751
numPoints 8 slope 0.00990725037810828 intercept -0.01816330722518818 rSquared 0.5997427960173778
numPoints 8 slope 1.0855382556763826 intercept -9.117864568033017 rSquared 0.9791873624835375
rSquared 0.976910778052751 rSquaredXLog 0.5997427960173778 rSquaredXYLog 0.9791873624835375

[Joseph Pecoraro]

There was just a recent change to Object.keys in bug 171291. Could be related.

[Radar WebKit Bug Importer]

<rdar://problem/31977874>

[Saam Barati]

I think this is just pure flakiness. For a few different reasons:
1. We run this alongside other things on the machine.
2. We use objects with very few keys, so there could just be a lot variance between runs.

I'll try to make it more stable.

[Saam Barati]

Created attachment 313011 [details]
patch

[Mark Lam]

Comment on attachment 313011 [details]
patch

r=me

[WebKit Commit Bot]

Comment on attachment 313011 [details]
patch

Clearing flags on attachment: 313011

Committed r218364: <http://trac.webkit.org/changeset/218364>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Matt Lewis]

After the second patch: https://trac.webkit.org/changeset/218364/webkit

Flakiness of the test did not improve and the test is still failing intermittently at a high rate on iOS Simulator according to the Dashboard.

https://build.webkit.org/results/Apple%20iOS%2010%20Simulator%20Release%20WK2%20(Tests)/r218614%20(2428)/results.html

new diff:
--- /Volumes/Data/slave/ios-simulator-10-release-tests-wk2/build/layout-test-results/perf/object-keys-expected.txt
+++ /Volumes/Data/slave/ios-simulator-10-release-tests-wk2/build/layout-test-results/perf/object-keys-actual.txt
@@ -1,3 +1,44 @@
 Tests that Object.keys is linear.
-PASS
+FAIL: got >=O(n^2) expected O(n)
 
+DEBUG LOG:
+run iteration. magnitude 256 jsObjectCountBefore 93783 jsObjectCountAfter 93108
+run iteration. magnitude 512 jsObjectCountBefore 93108 jsObjectCountAfter 93180
+run iteration. magnitude 1024 jsObjectCountBefore 93180 jsObjectCountAfter 93189
+run iteration. magnitude 2048 jsObjectCountBefore 93189 jsObjectCountAfter 93196
+run iteration. magnitude 4096 jsObjectCountBefore 93196 jsObjectCountAfter 93205
+run iteration. magnitude 8192 jsObjectCountBefore 93205 jsObjectCountAfter 93212
+magnitudes: 256,512,1024,2048,4096,8192
+iterations: 487,176,92,35,21,8
+numPoints 6 slope 0.0003802862605442898 intercept -0.10639593634290612 rSquared 0.9866968019879888
+numPoints 6 slope 0.781370446978062 intercept -4.7710360521047885 rSquared 0.7596636601354393
+numPoints 6 slope 1.1495574667742858 intercept -9.240963726164283 rSquared 0.9940214494669775
+rSquared 0.9866968019879888 rSquaredXLog 0.7596636601354393 rSquaredXYLog 0.9940214494669775
+numTriesLeft: 2
+run iteration. magnitude 256 jsObjectCountBefore 93212 jsObjectCountAfter 93265
+run iteration. magnitude 512 jsObjectCountBefore 93265 jsObjectCountAfter 93273
+run iteration. magnitude 1024 jsObjectCountBefore 93273 jsObjectCountAfter 93279
+run iteration. magnitude 2048 jsObjectCountBefore 93279 jsObjectCountAfter 93286
+run iteration. magnitude 4096 jsObjectCountBefore 93286 jsObjectCountAfter 93296
+run iteration. magnitude 8192 jsObjectCountBefore 93296 jsObjectCountAfter 93305
+magnitudes: 256,512,1024,2048,4096,8192
+iterations: 415,179,86,40,16,8
+numPoints 6 slope 0.0003918453220597 intercept -0.08609631857259037 rSquared 0.9980818570007993
+numPoints 6 slope 0.821371756110995 intercept -5.0107970207451435 rSquared 0.7997629788019797
+numPoints 6 slope 1.144017760650845 intercept -9.142490927567392 rSquared 0.9990380600703198
+rSquared 0.9980818570007993 rSquaredXLog 0.7997629788019797 rSquaredXYLog 0.9990380600703198
+numTriesLeft: 1
+run iteration. magnitude 256 jsObjectCountBefore 93305 jsObjectCountAfter 93349
+run iteration. magnitude 512 jsObjectCountBefore 93349 jsObjectCountAfter 93357
+run iteration. magnitude 1024 jsObjectCountBefore 93357 jsObjectCountAfter 93369
+run iteration. magnitude 2048 jsObjectCountBefore 93369 jsObjectCountAfter 93381
+run iteration. magnitude 4096 jsObjectCountBefore 93381 jsObjectCountAfter 93386
+run iteration. magnitude 8192 jsObjectCountBefore 93386 jsObjectCountAfter 93393
+magnitudes: 256,512,1024,2048,4096,8192
+iterations: 459,169,85,36,18,7
+numPoints 6 slope 0.0004386654990338807 intercept -0.15392039859634 rSquared 0.988900895680829
+numPoints 6 slope 0.8948014337703841 intercept -5.487192992472122 rSquared 0.7503849137821414
+numPoints 6 slope 1.1744944670284758 intercept -9.349381727854512 rSquared 0.99752152057426
+rSquared 0.988900895680829 rSquaredXLog 0.7503849137821414 rSquaredXYLog 0.99752152057426
+
+

[Ryan Haddad]

Reopening per Matt's comment.

[Ryan Haddad]

Marked test as flaky in https://trac.webkit.org/r218722