[JSC] Change some parameters based on a random search
Created attachment 280765 [details] Patch
We could probably get even better results by refining the search and running it for longer. The risk is we are gonna end up overfitting the benchmarks. Here are the results on my MBP (20 runs): Conf#1 Conf#2 SunSpider: 3d-cube 4.8645+-0.0160 ^ 4.4054+-0.0132 ^ definitely 1.1042x faster 3d-morph 4.9614+-0.0229 ? 4.9650+-0.0152 ? 3d-raytrace 5.2065+-0.0805 ^ 4.6210+-0.0541 ^ definitely 1.1267x faster access-binary-trees 1.9891+-0.0469 ^ 1.8490+-0.0051 ^ definitely 1.0757x faster access-fannkuch 5.7940+-0.0260 ? 5.8187+-0.0248 ? access-nbody 2.4748+-0.0310 ^ 2.4234+-0.0091 ^ definitely 1.0212x faster access-nsieve 3.0931+-0.0392 ^ 3.0126+-0.0313 ^ definitely 1.0267x faster bitops-3bit-bits-in-byte 1.0494+-0.0065 ! 1.0818+-0.0063 ! definitely 1.0309x slower bitops-bits-in-byte 2.6490+-0.0205 ? 2.6502+-0.0125 ? bitops-bitwise-and 1.9278+-0.0222 ^ 1.8795+-0.0060 ^ definitely 1.0257x faster bitops-nsieve-bits 2.9615+-0.0108 ! 3.0089+-0.0094 ! definitely 1.0160x slower controlflow-recursive 2.2855+-0.0398 ? 2.3338+-0.0151 ? might be 1.0212x slower crypto-aes 4.1901+-0.0085 ! 4.2768+-0.0418 ! definitely 1.0207x slower crypto-md5 2.3294+-0.0075 ! 2.3646+-0.0253 ! definitely 1.0151x slower crypto-sha1 2.2456+-0.0135 2.2272+-0.0142 date-format-tofte 6.3723+-0.0413 6.3714+-0.0486 date-format-xparb 4.6958+-0.1471 4.6712+-0.2151 math-cordic 2.7656+-0.0125 ? 2.7944+-0.0567 ? might be 1.0104x slower math-partial-sums 3.9678+-0.0120 ^ 3.8437+-0.0486 ^ definitely 1.0323x faster math-spectral-norm 1.9361+-0.0074 ^ 1.8146+-0.0048 ^ definitely 1.0669x faster regexp-dna 6.3831+-0.0688 6.3483+-0.0724 string-base64 3.8115+-0.0112 3.7881+-0.0288 string-fasta 5.5444+-0.0247 5.5409+-0.0237 string-tagcloud 8.3508+-0.0267 ? 8.4036+-0.0364 ? string-unpack-code 17.5749+-0.1711 ^ 17.1077+-0.0552 ^ definitely 1.0273x faster string-validate-input 4.0820+-0.0194 ^ 3.6538+-0.0183 ^ definitely 1.1172x faster <arithmetic> 4.3656+-0.0092 ^ 4.2791+-0.0120 ^ definitely 1.0202x faster Conf#1 Conf#2 LongSpider: 3d-cube 779.8235+-5.3602 776.8997+-5.4041 3d-morph 592.6016+-1.2196 592.1086+-1.8256 3d-raytrace 481.2230+-2.5263 477.3832+-2.1740 access-binary-trees 827.1957+-2.9611 825.7161+-2.6034 access-fannkuch 247.3705+-4.2828 244.0617+-3.2931 might be 1.0136x faster access-nbody 538.8888+-4.7767 ! 547.1318+-3.2908 ! definitely 1.0153x slower access-nsieve 289.3312+-5.0332 284.4443+-4.4292 might be 1.0172x faster bitops-3bit-bits-in-byte 32.6692+-0.0459 32.5745+-0.0760 bitops-bits-in-byte 94.1663+-0.7005 93.9790+-0.4733 bitops-nsieve-bits 388.4348+-2.1940 386.3802+-2.0631 controlflow-recursive 461.3027+-2.6779 459.5917+-0.9162 crypto-aes 580.4638+-1.3648 ^ 573.3677+-2.4718 ^ definitely 1.0124x faster crypto-md5 487.7357+-3.4744 487.6242+-2.8980 crypto-sha1 655.2313+-3.7394 ^ 645.5726+-2.3607 ^ definitely 1.0150x faster date-format-tofte 508.9334+-3.8576 ? 520.5671+-10.2484 ? might be 1.0229x slower date-format-xparb 624.6080+-1.1611 ? 625.3784+-3.4499 ? hash-map 141.5635+-0.5829 140.1171+-1.5694 might be 1.0103x faster math-cordic 441.8055+-2.3454 439.7765+-1.5205 math-partial-sums 284.1185+-0.7170 ? 294.3903+-23.8716 ? might be 1.0362x slower math-spectral-norm 553.4666+-2.1283 551.6168+-1.1638 string-base64 343.7146+-2.1778 343.6118+-1.7838 string-fasta 337.3581+-1.3954 ^ 333.4732+-0.9330 ^ definitely 1.0116x faster string-tagcloud 177.7166+-1.1628 177.6160+-1.1512 <geometric> 355.4752+-0.4329 354.6687+-0.9281 might be 1.0023x faster Conf#1 Conf#2 Octane: encrypt 0.15851+-0.00086 ? 0.15906+-0.00097 ? decrypt 2.80185+-0.00654 ? 2.81464+-0.01304 ? deltablue x2 0.12857+-0.00103 ! 0.13315+-0.00093 ! definitely 1.0356x slower earley 0.28549+-0.00109 0.28426+-0.00088 boyer 4.98780+-0.04328 4.98165+-0.01431 navier-stokes x2 4.93997+-0.01012 4.93250+-0.01401 raytrace x2 0.79310+-0.00325 ? 0.79766+-0.00191 ? richards x2 0.08288+-0.00074 ^ 0.08042+-0.00072 ^ definitely 1.0306x faster splay x2 0.33506+-0.00152 0.33321+-0.00119 regexp x2 16.93588+-0.17847 16.61988+-0.34030 might be 1.0190x faster pdfjs x2 39.23367+-0.20837 ^ 37.29290+-0.24211 ^ definitely 1.0520x faster mandreel x2 42.22440+-0.15400 ? 42.25369+-0.09947 ? gbemu x2 30.09149+-1.13352 ^ 27.31446+-0.07565 ^ definitely 1.1017x faster closure 0.48068+-0.00239 ? 0.48183+-0.00335 ? jquery 6.48689+-0.02070 ! 6.64552+-0.02984 ! definitely 1.0245x slower box2d x2 9.32962+-0.04321 9.24273+-0.04394 zlib x2 361.84243+-3.16570 357.48899+-3.86273 might be 1.0122x faster typescript x2 619.47997+-2.73034 ! 626.43856+-3.04624 ! definitely 1.0112x slower <geometric> 5.04150+-0.01237 ^ 4.98941+-0.01006 ^ definitely 1.0104x faster Conf#1 Conf#2 Kraken: ai-astar 86.829+-0.633 ^ 85.294+-0.781 ^ definitely 1.0180x faster audio-beat-detection 39.011+-0.277 ? 39.073+-0.191 ? audio-dft 98.980+-1.621 97.002+-0.914 might be 1.0204x faster audio-fft 30.942+-0.049 ? 30.960+-0.028 ? audio-oscillator 55.072+-0.162 ^ 48.887+-0.893 ^ definitely 1.1265x faster imaging-darkroom 61.519+-0.622 61.040+-0.056 imaging-desaturate 44.662+-0.822 43.818+-0.848 might be 1.0193x faster imaging-gaussian-blur 60.572+-1.195 59.107+-0.968 might be 1.0248x faster json-parse-financial 33.414+-0.502 ? 33.481+-0.575 ? json-stringify-tinderbox 22.697+-0.335 ? 22.887+-0.323 ? stanford-crypto-aes 37.841+-0.203 ^ 37.292+-0.063 ^ definitely 1.0147x faster stanford-crypto-ccm 33.269+-0.794 ^ 31.704+-0.660 ^ definitely 1.0494x faster stanford-crypto-pbkdf2 96.678+-0.334 ? 97.646+-0.743 ? might be 1.0100x slower stanford-crypto-sha256-iterative 31.539+-0.089 ^ 30.827+-0.277 ^ definitely 1.0231x faster <arithmetic> 52.359+-0.173 ^ 51.358+-0.133 ^ definitely 1.0195x faster Conf#1 Conf#2 AsmBench: bigfib.cpp 438.2704+-2.1964 ^ 423.7945+-1.5882 ^ definitely 1.0342x faster cray.c 366.5222+-1.5494 365.7688+-1.5055 dry.c 432.0049+-12.1917 426.0017+-11.7260 might be 1.0141x faster FloatMM.c 721.6651+-1.4404 ? 721.7654+-2.2090 ? gcc-loops.cpp 3696.7774+-15.9895 ! 3723.0834+-10.1815 ! definitely 1.0071x slower n-body.c 809.0939+-2.1459 ? 809.4533+-2.4112 ? Quicksort.c 392.0739+-1.0197 ? 393.2811+-1.9628 ? stepanov_container.cpp 3283.1602+-7.8905 3272.7343+-9.4546 Towers.c 272.5793+-0.4268 ? 273.1117+-0.2053 ? <geometric> 720.7329+-2.0867 717.5328+-2.1189 might be 1.0045x faster Conf#1 Conf#2 Geomean of preferred means: <scaled-result> 49.4346+-0.0483 ^ 48.8798+-0.0552 ^ definitely 1.0113x faster
Comment on attachment 280765 [details] Patch Awesome.
Comment on attachment 280765 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=280765&action=review > Source/JavaScriptCore/bytecode/CodeBlock.cpp:3619 > + // Those parameters where chosen by testing random values typo? "These parameters were chosen ..."
Comment on attachment 280765 [details] Patch Attachment 280765 [details] did not pass mac-debug-ews (mac): Output: http://webkit-queues.webkit.org/results/1463555 Number of test failures exceeded the failure limit.
Created attachment 280766 [details] Archive of layout-test-results from ews112 for mac-yosemite The attached test failures were seen while running run-webkit-tests on the mac-debug-ews. Bot: ews112 Port: mac-yosemite Platform: Mac OS X 10.10.5
Comment on attachment 280765 [details] Patch I forgot about something. Please make sure that NO_CJIT_OPTIONS in run-jsc-benchmarks locks in the *old* values. Otherwise our tests risk becoming meaningless.
Could we check in your script that does this search?
Created attachment 280838 [details] Patch
Comment on attachment 280838 [details] Patch That's so awesome!
Comment on attachment 280838 [details] Patch Clearing flags on attachment: 280838 Committed r201836: <http://trac.webkit.org/changeset/201836>
All reviewed patches have been landed. Closing bug.
This is 5% JSBench regression on Mac and 3% JSBench regression on iOS. This also appears to be 2% *progression* on MacBookPro but I see no other improvement elsewhere. Given the above, I recommend reverting this change.
This also looks like 1% PLT regression on iOS (iPhone 5).
Ben, maybe you can tune your parameter search for JSBench also? I suspect that the regression on JSBench is probably similar to PLT regression.
(In reply to comment #15) > Ben, maybe you can tune your parameter search for JSBench also? > > I suspect that the regression on JSBench is probably similar to PLT > regression. I think we should rollout this patch first. Keep adjusting parameters on the trunk would make it harder to judge how much of progression/regression this patch causes.
(In reply to comment #15) > Ben, maybe you can tune your parameter search for JSBench also? > > I suspect that the regression on JSBench is probably similar to PLT > regression. Yep, we can try this again including JSBench. It would be worth finding why JSBench is negatively affected, it is rather strange that it is going against Sunspider. AreWeFastYet looks pretty good though: https://arewefastyet.com/#machine=29 :)
Looks like a 1-2% PLT regression on all iOS hardware :(
Reverted r201836, r201845, and r201848 for reason: Looks like a 1-2% PLT regression on iOS Committed r201866: <http://trac.webkit.org/changeset/201866>
(In reply to comment #19) > Reverted r201836, r201845, and r201848 for reason: > > Looks like a 1-2% PLT regression on iOS > > Committed r201866: <http://trac.webkit.org/changeset/201866> Note that this was likely a PLT regression on Mac as well but more subtle.
(In reply to comment #20) > (In reply to comment #19) > > Reverted r201836, r201845, and r201848 for reason: > > > > Looks like a 1-2% PLT regression on iOS > > > > Committed r201866: <http://trac.webkit.org/changeset/201866> > > Note that this was likely a PLT regression on Mac as well but more subtle. There was also a 7% JSBench regression on iOS in this range as well, although I haven't proven yet it was caused by this change. I guess we'll see if we recover on JSBench as well from the roll out.
Created attachment 283814 [details] Patch
Created attachment 283825 [details] Patch
Comment on attachment 283825 [details] Patch Looks like it is trading a small Sunspider regression for big improvements everywhere else. Here is with 12 runs, in browser: Conf#1 Conf#2 SunSpider: 3d-cube 4.8830+-0.1203 4.7961+-0.0632 might be 1.0181x faster 3d-morph 5.3700+-0.3016 5.1531+-0.0397 might be 1.0421x faster 3d-raytrace 4.9310+-0.0864 ? 4.9614+-0.0696 ? access-binary-trees 2.4967+-0.0692 2.4884+-0.0308 access-fannkuch 5.5445+-0.0456 ? 5.5668+-0.0467 ? access-nbody 2.6198+-0.0342 ! 2.8439+-0.1773 ! definitely 1.0855x slower access-nsieve 3.6060+-0.2815 3.2723+-0.1423 might be 1.1020x faster bitops-3bit-bits-in-byte 1.0465+-0.0479 ? 1.0578+-0.0139 ? might be 1.0108x slower bitops-bits-in-byte 2.5893+-0.1329 ? 2.5953+-0.0881 ? bitops-bitwise-and 2.1785+-0.1584 2.0393+-0.0763 might be 1.0682x faster bitops-nsieve-bits 3.1296+-0.1468 3.0788+-0.1029 might be 1.0165x faster controlflow-recursive 2.2822+-0.0481 ? 2.2919+-0.0313 ? crypto-aes 4.2413+-0.0743 ! 4.9248+-0.1412 ! definitely 1.1611x slower crypto-md5 2.4638+-0.0387 ? 2.5348+-0.1337 ? might be 1.0288x slower crypto-sha1 2.3634+-0.0585 ! 2.6224+-0.0891 ! definitely 1.1096x slower date-format-tofte 8.3231+-0.1607 8.2034+-0.2274 might be 1.0146x faster date-format-xparb 5.9085+-0.1094 5.8491+-0.0687 might be 1.0102x faster math-cordic 2.7096+-0.0480 ? 2.7319+-0.0839 ? math-partial-sums 4.4115+-0.3906 ? 5.2021+-1.4577 ? might be 1.1792x slower math-spectral-norm 2.0176+-0.0719 1.9285+-0.0317 might be 1.0462x faster regexp-dna 7.9083+-0.5766 7.7104+-0.2724 might be 1.0257x faster string-base64 5.0283+-0.0885 4.9836+-0.0800 string-fasta 6.2235+-0.0568 ? 6.3348+-0.0668 ? might be 1.0179x slower string-tagcloud 9.8566+-0.5041 9.3895+-0.1980 might be 1.0498x faster string-unpack-code 20.2877+-0.5198 ? 20.7191+-0.6809 ? might be 1.0213x slower string-validate-input 5.3337+-0.0567 ? 5.4108+-0.0420 ? might be 1.0144x slower <arithmetic> 4.9136+-0.0682 ? 4.9496+-0.0730 ? might be 1.0073x slower Conf#1 Conf#2 Octane: encrypt 0.16050+-0.00168 0.16050+-0.00196 decrypt 2.78273+-0.00342 ! 2.80442+-0.00834 ! definitely 1.0078x slower deltablue x2 0.14251+-0.00495 0.13922+-0.00223 might be 1.0236x faster earley 0.30200+-0.00129 ? 0.30377+-0.00186 ? boyer 5.11307+-0.01753 ? 5.11972+-0.02473 ? navier-stokes x2 5.04453+-0.01047 ^ 5.01282+-0.01136 ^ definitely 1.0063x faster raytrace x2 0.93061+-0.00710 ^ 0.90873+-0.00808 ^ definitely 1.0241x faster richards x2 0.08450+-0.00139 0.08195+-0.00121 might be 1.0311x faster splay x2 0.34587+-0.00260 0.34269+-0.00160 regexp x2 20.87175+-0.39253 ! 22.45177+-0.45604 ! definitely 1.0757x slower pdfjs x2 37.28801+-0.19571 ^ 36.49137+-0.37266 ^ definitely 1.0218x faster mandreel x2 42.43480+-0.35898 ^ 41.32008+-0.19311 ^ definitely 1.0270x faster gbemu x2 29.76955+-0.31076 ^ 26.55283+-0.19594 ^ definitely 1.1211x faster closure 0.30490+-0.00133 ^ 0.30191+-0.00106 ^ definitely 1.0099x faster jquery 3.69938+-0.01367 3.69241+-0.01205 box2d x2 9.16650+-0.05385 ^ 8.95691+-0.05734 ^ definitely 1.0234x faster zlib x2 399.11329+-14.57122 ^ 325.90378+-1.82066 ^ definitely 1.2246x faster typescript x2 561.89688+-2.88146 ! 569.00385+-3.38915 ! definitely 1.0126x slower <geometric> 5.04297+-0.01848 ^ 4.91296+-0.00772 ^ definitely 1.0265x faster Conf#1 Conf#2 Kraken: ai-astar 92.906+-0.501 92.083+-1.036 audio-beat-detection 60.103+-0.351 ^ 58.487+-0.606 ^ definitely 1.0276x faster audio-dft 104.136+-1.823 ^ 98.039+-0.584 ^ definitely 1.0622x faster audio-fft 42.236+-0.060 ^ 40.140+-0.067 ^ definitely 1.0522x faster audio-oscillator 56.826+-0.407 ^ 55.186+-0.257 ^ definitely 1.0297x faster imaging-darkroom 75.830+-0.562 ^ 69.916+-0.549 ^ definitely 1.0846x faster imaging-desaturate 52.348+-1.117 ^ 48.747+-1.640 ^ definitely 1.0739x faster imaging-gaussian-blur 65.560+-1.127 ^ 61.832+-1.612 ^ definitely 1.0603x faster json-parse-financial 39.712+-0.153 39.349+-0.257 json-stringify-tinderbox 26.821+-0.147 26.627+-0.099 stanford-crypto-aes 49.002+-0.156 ^ 47.865+-0.282 ^ definitely 1.0238x faster stanford-crypto-ccm 50.679+-0.945 ? 51.467+-1.385 ? might be 1.0155x slower stanford-crypto-pbkdf2 102.160+-0.804 ^ 101.139+-0.173 ^ definitely 1.0101x faster stanford-crypto-sha256-iterative 40.544+-0.190 ^ 39.676+-0.138 ^ definitely 1.0219x faster <arithmetic> 61.347+-0.161 ^ 59.325+-0.189 ^ definitely 1.0341x faster Conf#1 Conf#2 JSBench: amazon-chrome 3.2897+-0.0340 3.2438+-0.0162 might be 1.0141x faster amazon-chrome-win 3.3697+-0.0455 ? 3.3883+-0.0397 ? amazon-firefox 2.8045+-0.2015 ? 2.8708+-0.1437 ? might be 1.0236x slower amazon-firefox-win 2.7466+-0.1715 2.6324+-0.2054 might be 1.0434x faster amazon-safari 3.9153+-0.0655 3.8130+-0.0492 might be 1.0268x faster facebook-chrome 36.6787+-0.2688 ^ 36.1957+-0.1646 ^ definitely 1.0133x faster facebook-chrome-win 79.1689+-0.6735 ^ 77.7271+-0.6665 ^ definitely 1.0185x faster facebook-firefox 21.4382+-0.1184 21.3267+-0.1158 facebook-firefox-win 14.6760+-0.2626 14.3616+-0.0849 might be 1.0219x faster facebook-safari 80.3112+-0.4966 80.1410+-0.4379 google-chrome 34.4195+-0.1928 34.0463+-0.2032 might be 1.0110x faster google-chrome-win 32.3465+-0.3334 31.9658+-0.5917 might be 1.0119x faster google-firefox 16.8966+-0.1156 ! 17.1268+-0.0908 ! definitely 1.0136x slower google-firefox-win 20.4265+-0.1841 ? 20.4397+-0.1501 ? google-safari 35.5818+-0.1765 35.4727+-0.4887 twitter-chrome 5.5514+-0.0730 5.4712+-0.0546 might be 1.0147x faster twitter-chrome-win 5.4917+-0.0370 5.4724+-0.0490 twitter-firefox 0.7926+-0.0175 0.7889+-0.0264 twitter-firefox-win 0.7733+-0.0456 0.7656+-0.0332 might be 1.0101x faster twitter-safari 5.9160+-0.0736 5.8360+-0.0619 might be 1.0137x faster yahoo-chrome 32.4908+-0.2439 ? 32.9385+-0.2583 ? might be 1.0138x slower yahoo-chrome-win 28.5546+-0.1245 ? 28.9519+-0.3663 ? might be 1.0139x slower yahoo-firefox 32.6455+-0.3819 ? 32.7986+-0.3393 ? yahoo-firefox-win 21.8593+-0.1633 ! 22.1806+-0.0738 ! definitely 1.0147x slower yahoo-safari 32.4324+-0.2292 ? 32.7938+-0.2821 ? might be 1.0111x slower <geometric> 11.9043+-0.0476 11.8507+-0.0497 might be 1.0045x faster Conf#1 Conf#2 AsmBench: bigfib.cpp 432.6618+-3.8810 ^ 423.8220+-1.1965 ^ definitely 1.0209x faster cray.c 389.7045+-1.8497 386.5185+-2.2558 dry.c 447.8438+-36.3144 ? 454.0457+-39.2907 ? might be 1.0138x slower FloatMM.c 729.5487+-12.8522 724.1023+-15.4406 gcc-loops.cpp 3650.9885+-25.4257 3636.1910+-28.5358 n-body.c 804.2384+-2.0716 802.9355+-1.7170 Quicksort.c 397.2989+-2.7595 393.8240+-2.4330 stepanov_container.cpp 3295.2933+-13.2986 3287.5590+-19.9495 Towers.c 267.7637+-0.7738 ! 273.0052+-0.6267 ! definitely 1.0196x slower <geometric> 726.4274+-5.5301 724.7086+-6.7170 might be 1.0024x faster Conf#1 Conf#2 CompressionBench: huffman 49.3873+-0.4197 ? 49.4794+-0.5845 ? arithmetic-simple 245.1948+-1.8406 ? 246.1661+-1.0660 ? arithmetic-precise 228.1144+-1.1609 ? 229.7283+-1.7235 ? arithmetic-complex-precise 229.2591+-1.5192 ? 229.7518+-1.4829 ? arithmetic-precise-order-0 254.7916+-1.5095 254.0297+-1.2381 arithmetic-precise-order-1 279.7392+-0.8041 ! 281.7750+-0.8336 ! definitely 1.0073x slower arithmetic-precise-order-2 334.2069+-2.3989 ^ 329.0390+-2.6054 ^ definitely 1.0157x faster arithmetic-simple-order-1 290.3063+-0.9266 ? 292.9446+-1.9940 ? arithmetic-simple-order-2 344.1967+-3.0825 339.8740+-2.9436 might be 1.0127x faster lz-string 280.5969+-1.6741 278.4226+-2.1999 <geometric> 230.4671+-0.3631 230.2875+-0.5272 might be 1.0008x faster Conf#1 Conf#2 Geomean of preferred means: <scaled-result> 38.0372+-0.0833 ^ 37.6584+-0.0945 ^ definitely 1.0101x faster
Comment on attachment 283825 [details] Patch r=me lets do it
Comment on attachment 283825 [details] Patch Attempt 2, let's hope PLT likes this one better!
Comment on attachment 283825 [details] Patch Clearing flags on attachment: 283825 Committed r203318: <http://trac.webkit.org/changeset/203318>
Reverted r203318 for reason: Regressed most JS Benchmarks on MacBook Air by ~2% (7% on SunSpider) Committed r203329: <http://trac.webkit.org/changeset/203329>
Thanks Chris. Looks like I'll have to account for dual-cores for the next attempt.