149061 2015-09-11 04:32:12 -0700 [ARM] REGRESSION(r189575): It made 2860 tests fail/crash on AArch64 Linux 2015-11-12 01:58:01 -0800 1 1 1 Unclassified WebKit JavaScriptCore Other Unspecified Unspecified RESOLVED DUPLICATE 150936 P2 Normal --- 108645 148666 1 ossy webkit-unassigned clopez fpizlo mark.lam msaboff ossy oldest_to_newest 1125166 0 ossy 2015-09-11 04:32:12 -0700 Unfortunately I can't add the details to the bug report, because https://build.webkit.org/waterfall is out of order again and again. :-/ 1125167 1 ossy 2015-09-11 04:37:37 -0700 Ah, build.webkit.org works again, so here is the link about this regression: https://build.webkit.org/builders/EFL%20Linux%20AArch64%20Release/builds/3270 I tested manually, everything works fine on r189574, but there are 2860 failures/crash on r189575 (with its buildfix r189588 I'm going to investigate this issue and try to provide debug build logs and/or other useful information. 1125175 2 ossy 2015-09-11 06:39:20 -0700 Unfortunately I can't reproduce this bug in debug mode. :( I will try to reproduce it on a relase build with debug symbols. 1125202 3 msaboff 2015-09-11 09:48:29 -0700 While debugging the callee saves work, I would run into failures on release builds that wouldn't reproduce with debug builds. Typically this was due to the optimizer making use of callee saves registers in the compiled C++ code. If JSC inadvertently stepped on one of those registers, it would only cause a problem on release builds. The first place I would look is in the FTL code. For example, I didn't test any of the changes to the Linux specific code in FTLUnwindInfo.cpp. See if failing tests work when the FTL is turned off. One technique that I used to track down these kinds of problems was to add back in the saving and restoring of callee saves to the pushCalleeSaves() / popCalleeSaves() macros in LowLevelInterpreter.asm and then in LowLEvelInterpreter64.asm:doVMEntry, write sentinel numeric values to the callee saves registers, e.g. 0x1019 to x19, 0x1020 to x20, ... After "makeCall()" in doVMEntry and at the beginning of _handleUncaughtException, compare the values with a breakpoint on mismatch. I made a macro to do the testing. That did 2 things, first it allowed building with debug. But probably more useful was that at any point executing in the JavaScript VMs I could look at the registers to see that they had the sentinel values were they should. I could also check the CallFrames that we saved the sentinel values where appropriate. I'll post a patch with this technique that I used for X86-64 debugging. 1125204 4 261007 msaboff 2015-09-11 09:50:48 -0700 Created attachment 261007 Patch used for X86-64 Callee Saves debugging 1125920 5 ossy 2015-09-15 02:59:46 -0700 (In reply to comment #3) > While debugging the callee saves work, I would run into failures on release > builds that wouldn't reproduce with debug builds. Typically this was due to > the optimizer making use of callee saves registers in the compiled C++ code. > If JSC inadvertently stepped on one of those registers, it would only cause > a problem on release builds. > > The first place I would look is in the FTL code. For example, I didn't test > any of the changes to the Linux specific code in FTLUnwindInfo.cpp. See if > failing tests work when the FTL is turned off. > > One technique that I used to track down these kinds of problems was to add > back in the saving and restoring of callee saves to the pushCalleeSaves() / > popCalleeSaves() macros in LowLevelInterpreter.asm and then in > LowLEvelInterpreter64.asm:doVMEntry, write sentinel numeric values to the > callee saves registers, e.g. 0x1019 to x19, 0x1020 to x20, ... After > "makeCall()" in doVMEntry and at the beginning of _handleUncaughtException, > compare the values with a breakpoint on mismatch. I made a macro to do the > testing. That did 2 things, first it allowed building with debug. But > probably more useful was that at any point executing in the JavaScript VMs I > could look at the registers to see that they had the sentinel values were > they should. I could also check the CallFrames that we saved the sentinel > values where appropriate. I'll post a patch with this technique that I used > for X86-64 debugging. Thanks for the ideas and the patch for debugging. I didn't check the FTL code yet, because it is disabled by default on Linux. I don't know if it works at all, I didn't check it in the latest 4-5 months. But it seems the bug is in the DFG tier somewhere, because tests pass with (build time) disabled DFG. (except ~20 tests) And I already managed to catch register mismatches with the idea you suggested. I'll continue debugging in the near future. 1141420 6 ossy 2015-11-12 01:58:01 -0800 https://trac.webkit.org/changeset/192352 already fixed this issue. Sometimes Linux failures point out real but hidden failure on iOS. ;) before: https://build.webkit.org/builders/EFL%20Linux%20AArch64%20Release/builds/4313 - 3227 failures after: https://build.webkit.org/builders/EFL%20Linux%20AArch64%20Release/builds/4314 - 52 failures The remaining failures might be related to this issue or can be a different issue, who knows what else happened in the latest 2 months. I'm going to file a new bug report for the remaining failures. *** This bug has been marked as a duplicate of bug 150936 *** 261007 2015-09-11 09:50:48 -0700 2015-09-14 03:47:50 -0700 Patch used for X86-64 Callee Saves debugging test-debug-callee-saves.patch text/plain 3828 msaboff SW5kZXg6IFNvdXJjZS9KYXZhU2NyaXB0Q29yZS9sbGludC9Mb3dMZXZlbEludGVycHJldGVyNjQu YXNtCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT0KLS0tIFNvdXJjZS9KYXZhU2NyaXB0Q29yZS9sbGludC9Mb3dMZXZlbElu dGVycHJldGVyNjQuYXNtCShyZXZpc2lvbiAxODg5MDMpCisrKyBTb3VyY2UvSmF2YVNjcmlwdENv cmUvbGxpbnQvTG93TGV2ZWxJbnRlcnByZXRlcjY0LmFzbQkod29ya2luZyBjb3B5KQpAQCAtMTE0 LDEwICsxMTQsMjQgQEAgbWFjcm8gY0NhbGw0KGZ1bmN0aW9uKQogICAgIGVuZAogZW5kCiAKK21h Y3JvIGNoZWNrUmVnVmFsdWUocmVnLCB2YWx1ZSwgdGVtcCkKKyAgICBtb3ZlIHZhbHVlLCB0ZW1w CisgICAgYnBlcSB0ZW1wLCByZWcsIC52YWx1ZU9LCisgICAgYnJlYWsKKyAgICBtb3ZlIHJlZywg dDEKKy52YWx1ZU9LOgorZW5kCisKIG1hY3JvIGRvVk1FbnRyeShtYWtlQ2FsbCkKICAgICBmdW5j dGlvblByb2xvZ3VlKCkKICAgICBwdXNoQ2FsbGVlU2F2ZXMoKQogCisgICAgbW92ZSAweDEwMCwg Y3NyMAorICAgIG1vdmUgMHgxMDEsIGNzcjEKKyAgICBtb3ZlIDB4MTAyLCBjc3IyCisgICAgbW92 ZSAweDEwMywgY3NyMworICAgIG1vdmUgMHgxMDQsIGNzcjQKKwogICAgIGNvbnN0IGVudHJ5ID0g YTAKICAgICBjb25zdCB2bSA9IGExCiAgICAgY29uc3QgcHJvdG9DYWxsRnJhbWUgPSBhMgpAQCAt MTcwLDYgKzE4NCwxMiBAQCBtYWNybyBkb1ZNRW50cnkobWFrZUNhbGwpCiAKICAgICBzdWJwIGNm ciwgQ2FsbGVlUmVnaXN0ZXJTYXZlU2l6ZSwgc3AKIAorICAgIGNoZWNrUmVnVmFsdWUoY3NyMCwg MHgxMDAsIHQ0KQorICAgIGNoZWNrUmVnVmFsdWUoY3NyMSwgMHgxMDEsIHQ0KQorICAgIGNoZWNr UmVnVmFsdWUoY3NyMiwgMHgxMDIsIHQ0KQorICAgIGNoZWNrUmVnVmFsdWUoY3NyMywgMHgxMDMs IHQ0KQorICAgIGNoZWNrUmVnVmFsdWUoY3NyNCwgMHgxMDQsIHQ0KQorCiAgICAgcG9wQ2FsbGVl U2F2ZXMoKQogICAgIGZ1bmN0aW9uRXBpbG9ndWUoKQogICAgIHJldApAQCAtMjM0LDYgKzI1NCwx MiBAQCBtYWNybyBkb1ZNRW50cnkobWFrZUNhbGwpCiAKICAgICBzdWJwIGNmciwgQ2FsbGVlUmVn aXN0ZXJTYXZlU2l6ZSwgc3AKIAorICAgIGNoZWNrUmVnVmFsdWUoY3NyMCwgMHgxMDAsIHQ0KQor ICAgIGNoZWNrUmVnVmFsdWUoY3NyMSwgMHgxMDEsIHQ0KQorICAgIGNoZWNrUmVnVmFsdWUoY3Ny MiwgMHgxMDIsIHQ0KQorICAgIGNoZWNrUmVnVmFsdWUoY3NyMywgMHgxMDMsIHQ0KQorICAgIGNo ZWNrUmVnVmFsdWUoY3NyNCwgMHgxMDQsIHQ0KQorCiAgICAgcG9wQ2FsbGVlU2F2ZXMoKQogICAg IGZ1bmN0aW9uRXBpbG9ndWUoKQogCkBAIC0yNzUsNiArMzAxLDEzIEBAIF9oYW5kbGVVbmNhdWdo dEV4Y2VwdGlvbjoKICAgICBhbmRwIE1hcmtlZEJsb2NrTWFzaywgdDMKICAgICBsb2FkcCBNYXJr ZWRCbG9jazo6bV93ZWFrU2V0ICsgV2Vha1NldDo6bV92bVt0M10sIHQzCiAgICAgcmVzdG9yZUNh bGxlZVNhdmVzRnJvbVZNQ2FsbGVlU2F2ZXNCdWZmZXIodDMsIHQwKQorCisgICAgY2hlY2tSZWdW YWx1ZShjc3IwLCAweDEwMCwgdDQpCisgICAgY2hlY2tSZWdWYWx1ZShjc3IxLCAweDEwMSwgdDQp CisgICAgY2hlY2tSZWdWYWx1ZShjc3IyLCAweDEwMiwgdDQpCisgICAgY2hlY2tSZWdWYWx1ZShj c3IzLCAweDEwMywgdDQpCisgICAgY2hlY2tSZWdWYWx1ZShjc3I0LCAweDEwNCwgdDQpCisKICAg ICBsb2FkcCBWTTo6Y2FsbEZyYW1lRm9yVGhyb3dbdDNdLCBjZnIKIAogICAgIGxvYWRwIENhbGxl ckZyYW1lW2Nmcl0sIGNmcgpJbmRleDogU291cmNlL0phdmFTY3JpcHRDb3JlL2xsaW50L0xvd0xl dmVsSW50ZXJwcmV0ZXIuYXNtCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIFNvdXJjZS9KYXZhU2NyaXB0Q29yZS9s bGludC9Mb3dMZXZlbEludGVycHJldGVyLmFzbQkocmV2aXNpb24gMTg4OTAzKQorKysgU291cmNl L0phdmFTY3JpcHRDb3JlL2xsaW50L0xvd0xldmVsSW50ZXJwcmV0ZXIuYXNtCSh3b3JraW5nIGNv cHkpCkBAIC00MDEsMTEgKzQwMSwxMSBAQCBtYWNybyBjaGVja1N0YWNrUG9pbnRlckFsaWdubWVu dCh0ZW1wUmVnCiAgICAgZW5kCiBlbmQKIAotaWYgQ19MT09QIG9yIEFSTTY0IG9yIFg4Nl82NCBv ciBYODZfNjRfV0lOCitpZiBDX0xPT1Agb3IgQVJNNjQgb3IgWDg2XzY0X1dJTgogICAgIGNvbnN0 IENhbGxlZVNhdmVSZWdpc3RlckNvdW50ID0gMAogZWxzaWYgQVJNIG9yIEFSTXY3X1RSQURJVElP TkFMIG9yIEFSTXY3CiAgICAgY29uc3QgQ2FsbGVlU2F2ZVJlZ2lzdGVyQ291bnQgPSA3Ci1lbHNp ZiBTSDQgb3IgTUlQUworZWxzaWYgU0g0IG9yIE1JUFMgb3IgWDg2XzY0CiAgICAgY29uc3QgQ2Fs bGVlU2F2ZVJlZ2lzdGVyQ291bnQgPSA1CiBlbHNpZiBYODYgb3IgWDg2X1dJTgogICAgIGNvbnN0 IENhbGxlZVNhdmVSZWdpc3RlckNvdW50ID0gMwpAQCAtNDE4LDcgKzQxOCw3IEBAIGNvbnN0IENh bGxlZVJlZ2lzdGVyU2F2ZVNpemUgPSBDYWxsZWVTYXYKIGNvbnN0IFZNRW50cnlUb3RhbEZyYW1l U2l6ZSA9IChDYWxsZWVSZWdpc3RlclNhdmVTaXplICsgc2l6ZW9mIFZNRW50cnlSZWNvcmQgKyBT dGFja0FsaWdubWVudCAtIDEpICYgflN0YWNrQWxpZ25tZW50TWFzawogCiBtYWNybyBwdXNoQ2Fs bGVlU2F2ZXMoKQotICAgIGlmIENfTE9PUCBvciBBUk02NCBvciBYODZfNjQgb3IgWDg2XzY0X1dJ TgorICAgIGlmIENfTE9PUCBvciBBUk02NCBvciBYODZfNjRfV0lOCiAgICAgZWxzaWYgQVJNIG9y IEFSTXY3X1RSQURJVElPTkFMCiAgICAgICAgIGVtaXQgInB1c2gge3I0LXIxMH0iCiAgICAgZWxz aWYgQVJNdjcKQEAgLTQ0MCw2ICs0NDAsMTIgQEAgbWFjcm8gcHVzaENhbGxlZVNhdmVzKCkKICAg ICAgICAgZW1pdCAicHVzaCAlZXNpIgogICAgICAgICBlbWl0ICJwdXNoICVlZGkiCiAgICAgICAg IGVtaXQgInB1c2ggJWVieCIKKyAgICBlbHNpZiBYODZfNjQKKyAgICAgICAgZW1pdCAicHVzaCAl cjE1IgorICAgICAgICBlbWl0ICJwdXNoICVyMTQiCisgICAgICAgIGVtaXQgInB1c2ggJXIxMyIK KyAgICAgICAgZW1pdCAicHVzaCAlcjEyIgorICAgICAgICBlbWl0ICJwdXNoICVyYngiCiAgICAg ZWxzaWYgWDg2X1dJTgogICAgICAgICBlbWl0ICJwdXNoIGVzaSIKICAgICAgICAgZW1pdCAicHVz aCBlZGkiCkBAIC00NDgsNyArNDU0LDcgQEAgbWFjcm8gcHVzaENhbGxlZVNhdmVzKCkKIGVuZAog CiBtYWNybyBwb3BDYWxsZWVTYXZlcygpCi0gICAgaWYgQ19MT09QIG9yIEFSTTY0IG9yIFg4Nl82 NCBvciBYODZfNjRfV0lOCisgICAgaWYgQ19MT09QIG9yIEFSTTY0IG9yIFg4Nl82NF9XSU4KICAg ICBlbHNpZiBBUk0gb3IgQVJNdjdfVFJBRElUSU9OQUwKICAgICAgICAgZW1pdCAicG9wIHtyNC1y MTB9IgogICAgIGVsc2lmIEFSTXY3CkBAIC00NzAsNiArNDc2LDEyIEBAIG1hY3JvIHBvcENhbGxl ZVNhdmVzKCkKICAgICAgICAgZW1pdCAicG9wICVlYngiCiAgICAgICAgIGVtaXQgInBvcCAlZWRp IgogICAgICAgICBlbWl0ICJwb3AgJWVzaSIKKyAgICBlbHNpZiBYODZfNjQKKyAgICAgICAgZW1p dCAicG9wICVyYngiCisgICAgICAgIGVtaXQgInBvcCAlcjEyIgorICAgICAgICBlbWl0ICJwb3Ag JXIxMyIKKyAgICAgICAgZW1pdCAicG9wICVyMTQiCisgICAgICAgIGVtaXQgInBvcCAlcjE1Igog ICAgIGVsc2lmIFg4Nl9XSU4KICAgICAgICAgZW1pdCAicG9wIGVieCIKICAgICAgICAgZW1pdCAi cG9wIGVkaSIK