190334 2018-10-06 21:34:38 -0700 Adding some temporary asserts with data logging to debug a mysterious ASAN bot crash. 2018-10-09 07:41:10 -0700 1 1 1 Unclassified WebKit JavaScriptCore WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED InRadar P2 Normal --- 1 mark.lam mark.lam benjamin cdumez cmarcelo dbates ddkilzer ews-watchlist keith_miller pnormand webkit-bug-importer ysuzuki oldest_to_newest 1466917 0 mark.lam 2018-10-06 21:34:38 -0700 Need some additional data to debug this issue. 1466918 1 webkit-bug-importer 2018-10-06 21:35:21 -0700 <rdar://problem/45071303> 1466919 2 351737 mark.lam 2018-10-06 21:38:57 -0700 Created attachment 351737 proposed patch. 1466946 3 351737 ysuzuki 2018-10-07 12:09:56 -0700 Comment on attachment 351737 proposed patch. View in context: https://bugs.webkit.org/attachment.cgi?id=351737&action=review r=me with a nit. > Source/WTF/wtf/StackBounds.cpp:152 > + else { I think using `if (stackDirection() == StackDirection::Downward)` here would be more readable. 1466952 4 351737 mark.lam 2018-10-07 13:04:20 -0700 Comment on attachment 351737 proposed patch. View in context: https://bugs.webkit.org/attachment.cgi?id=351737&action=review Thanks. >> Source/WTF/wtf/StackBounds.cpp:152 >> + else { > > I think using `if (stackDirection() == StackDirection::Downward)` here would be more readable. Since Upward and Downward are mutually exclusive, I'll just add an ASSERT(stackDirection() == StackDirection::Downward) in the else block here to document the condition. I feel like the "else if" would (wrongly) imply that a 3rd option is possible. 1466975 5 mark.lam 2018-10-07 19:04:12 -0700 Landed in r236909: <http://trac.webkit.org/r236909>. 1467103 6 351737 keith_miller 2018-10-08 10:22:37 -0700 Comment on attachment 351737 proposed patch. r=me too. 1467373 7 pnormand 2018-10-09 03:56:46 -0700 This broke the GTK Debug tests bot: Thread 1 (LWP 13935): #0 0x000055d80120a116 in WTFCrash () at /home/slave/webkitgtk/gtk-linux-64-debug/build/Source/WTF/wtf/Assertions.cpp:255 Backtrace stopped: Cannot access memory at address 0x7ffc49fab0a8 STDERR: BFD: Warning: /var/www/cores/bb-webkit-debug-test-64/core-pid_13935.dump is truncated: expected core file size >= 241209344, found: 6823936. STDERR: STDERR: warning: core file may not match specified executable file. STDERR: Cannot access memory at address 0x7f0e5bf25148 STDERR: Cannot access memory at address 0x7f0e5bf25140 STDERR: Failed to read a valid object file image from memory. STDERR: ASSERTION FAILED: newThreadStackBounds: currentPosition 0x7ffc49fab0c0 origin 0x7f0de7131000 bound 0x7f0de6930000 stackSize 8392704 STDERR: (currentPosition < origin && currentPosition > bound) STDERR: /home/slave/webkitgtk/gtk-linux-64-debug/build/Source/WTF/wtf/StackBounds.cpp(155) : static WTF::StackBounds WTF::StackBounds::newThreadStackBounds(WTF::PlatformThreadHandle) STDERR: 1 0x55d80120a111 /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTFCrash+0x9) [0x55d80120a111] STDERR: 2 0x55d80128f23f /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTF::StackBounds::newThreadStackBounds(unsigned long)+0x16f) [0x55d80128f23f] STDERR: 3 0x55d8012409be /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTF::Thread::create(char const*, WTF::Function<void ()>&&)+0x180) [0x55d8012409be] STDERR: 4 0x55d801289db2 /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTF::WorkQueue::platformInitialize(char const*, WTF::WorkQueue::Type, WTF::WorkQueue::QOS)+0x5c) [0x55d801289db2] STDERR: 5 0x55d8012478de /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTF::WorkQueue::WorkQueue(char const*, WTF::WorkQueue::Type, WTF::WorkQueue::QOS)+0x4a) [0x55d8012478de] STDERR: 6 0x55d801247879 /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTF::WorkQueue::create(char const*, WTF::WorkQueue::Type, WTF::WorkQueue::QOS)+0x39) [0x55d801247879] STDERR: 7 0x7f0e538e0f2b /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Debug/lib/libwebkit2gtk-4.0.so.37(WebKit::StorageManager::StorageManager(WTF::String const&)+0x4f) [0x7f0e538e0f2b] STDERR: 8 0x7f0e538e0ec1 /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Debug/lib/libwebkit2gtk-4.0.so.37(WebKit::StorageManager::create(WTF::String const&)+0x2d) [0x7f0e538e0ec1] STDERR: 9 0x7f0e538e5120 /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Debug/lib/libwebkit2gtk-4.0.so.37(WebKit::WebsiteDataStore::WebsiteDataStore(WebKit::WebsiteDataStore::Configuration, PAL::SessionID)+0xca) [0x7f0e538e5120] STDERR: 10 0x7f0e538e4f1a /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Debug/lib/libwebkit2gtk-4.0.so.37(WebKit::WebsiteDataStore::create(WebKit::WebsiteDataStore::Configuration, PAL::SessionID)+0x78) [0x7f0e538e4f1a] STDERR: 11 0x7f0e5372869e /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Debug/lib/libwebkit2gtk-4.0.so.37(API::WebsiteDataStore::WebsiteDataStore(WebKit::WebsiteDataStore::Configuration, PAL::SessionID)+0x86) [0x7f0e5372869e] STDERR: 12 0x7f0e53728595 /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Debug/lib/libwebkit2gtk-4.0.so.37(API::WebsiteDataStore::createLegacy(WebKit::WebsiteDataStore::Configuration)+0x6f) [0x7f0e53728595] STDERR: 13 0x7f0e53622d38 /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Debug/lib/libwebkit2gtk-4.0.so.37(WebKit::WebProcessPool::WebProcessPool(API::ProcessPoolConfiguration&)+0x812) [0x7f0e53622d38] STDERR: 14 0x7f0e53621eac /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Debug/lib/libwebkit2gtk-4.0.so.37(WebKit::WebProcessPool::create(API::ProcessPoolConfiguration&)+0x32) [0x7f0e53621eac] STDERR: 15 0x7f0e5372cb04 /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Debug/lib/libwebkit2gtk-4.0.so.37(WKContextCreateWithConfiguration+0x2b) [0x7f0e5372cb04] STDERR: 16 0x55d8011c01e2 /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTR::TestController::generatePageConfiguration(OpaqueWKContextConfiguration const*)+0x30) [0x55d8011c01e2] STDERR: 17 0x55d8011c085a /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTR::TestController::createWebViewWithOptions(WTR::TestOptions const&)+0x1ac) [0x55d8011c085a] STDERR: 18 0x55d8011c0dec /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTR::TestController::ensureViewSupportsOptionsForTest(WTR::TestInvocation const&)+0x14e) [0x55d8011c0dec] STDERR: 19 0x55d8011c3ba3 /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTR::TestController::configureViewForTest(WTR::TestInvocation const&)+0x23) [0x55d8011c3ba3] STDERR: 20 0x55d8011e23de /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTR::TestInvocation::invoke()+0x26) [0x55d8011e23de] STDERR: 21 0x55d8011c44c3 /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTR::TestController::runTest(char const*)+0x235) [0x55d8011c44c3] STDERR: 22 0x55d8011c4697 /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTR::TestController::runTestingServerLoop()+0xa5) [0x55d8011c4697] STDERR: 23 0x55d8011c46d8 /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTR::TestController::run()+0x24) [0x55d8011c46d8] STDERR: 24 0x55d8011beefa /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTR::TestController::TestController(int, char const**)+0x30c) [0x55d8011beefa] STDERR: 25 0x55d801209a77 /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(main+0xe2) [0x55d801209a77] STDERR: 26 0x7f0e3fbbc2b1 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f0e3fbbc2b1] STDERR: 27 0x55d8011ba53a /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(_start+0x2a) [0x55d8011ba53a] 1467374 8 pnormand 2018-10-09 03:59:39 -0700 AFAIK The GTK Debug bot doesn't have ASAN enabled. Any chance this assert could be raised only if ASAN is enabled? 1467375 9 ysuzuki 2018-10-09 04:07:21 -0700 (In reply to Philippe Normand from comment #8) > AFAIK The GTK Debug bot doesn't have ASAN enabled. Any chance this assert > could be raised only if ASAN is enabled? Ah, right. This assertion was wrong. This StackBounds::newThreadStackBounds can be called from the other thread. Typically, the caller thread (creating a thread) will call this function for the newly created thread. It is important since it removes one ping-pong when creating a new thread. 1467376 10 ysuzuki 2018-10-09 04:10:38 -0700 Ah, and I think this is the reason why ASAN bot intermittently fails. This `StackBounds::newThreadStackBounds` can be called from the caller thread. It always succeeds. But, internally, the thread would be in the middle of the initialization (start running?). Even in that case, when stack area is already allocated, so this function is OK. But these region would not be marked as ASAN-safe-region when calling this function! @Mark, what do you think of the above hypothesis? 1467380 11 ysuzuki 2018-10-09 04:45:01 -0700 (In reply to Yusuke Suzuki from comment #10) > Ah, and I think this is the reason why ASAN bot intermittently fails. > This `StackBounds::newThreadStackBounds` can be called from the caller > thread. > It always succeeds. But, internally, the thread would be in the middle of > the initialization (start running?). > Even in that case, when stack area is already allocated, so this function is > OK. > But these region would not be marked as ASAN-safe-region when calling this > function! > > @Mark, what do you think of the above hypothesis? I would like to know where the ASAN bot actually fails. If ASAN bot fails due to pthread_xxx call to retrieve StackBounds, it seems OK. But if ASAN crashes due to touching the stack memory early, it should not be done since these stack memory would not be safe if the region is not the above of the stack pointer of the thread. <origin> ====== safe ====== <stack pointer> ======= unsafe ====== <bound> 1467391 12 mark.lam 2018-10-09 07:41:10 -0700 (In reply to Philippe Normand from comment #7) > This broke the GTK Debug tests bot: Sorry about this. I've gotten the info I needed: the issue lies with the use of the detect_stack_use_after_return ASAB option. Will be reverting these asserts in https://bugs.webkit.org/show_bug.cgi?id=190396. 351737 2018-10-06 21:38:57 -0700 2018-10-08 10:23:13 -0700 proposed patch. bug-190334.patch text/plain 3522 mark.lam SW5kZXg6IFNvdXJjZS9XVEYvQ2hhbmdlTG9nCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIFNvdXJjZS9XVEYvQ2hh bmdlTG9nCShyZXZpc2lvbiAyMzY5MDMpCisrKyBTb3VyY2UvV1RGL0NoYW5nZUxvZwkod29ya2lu ZyBjb3B5KQpAQCAtMSwzICsxLDE5IEBACisyMDE4LTEwLTA2ICBNYXJrIExhbSAgPG1hcmsubGFt QGFwcGxlLmNvbT4KKworICAgICAgICBBZGRpbmcgc29tZSB0ZW1wb3JhcnkgYXNzZXJ0cyB3aXRo IGRhdGEgbG9nZ2luZyB0byBkZWJ1ZyBhIG15c3RlcmlvdXMgQVNBTiBib3QgY3Jhc2guCisgICAg ICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0xOTAzMzQKKyAgICAg ICAgPHJkYXI6Ly9wcm9ibGVtLzQ1MDcxMzAzPgorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9E WSAoT09QUyEpLgorCisgICAgICAgIFRoZXNlIGFzc2VydGlvbnMgYXJlIG5lZWRlZCBiZWNhdXNl IHdlIGNhbid0IHJlcHJvZHVjZSB0aGUgaXNzdWUgbG9jYWxseS4KKyAgICAgICAgV2UnbGwgcmVt b3ZlIHRoZXNlIGFzc2VydHMgYWZ0ZXIgdGhlIG5lZWRlZCBkYXRhIGhhcyBiZWVuIGNvbGxlY3Rl ZCBmcm9tIHRoZSBib3QuCisKKyAgICAgICAgKiB3dGYvU3RhY2tCb3VuZHMuY3BwOgorICAgICAg ICAoV1RGOjpTdGFja0JvdW5kczo6bmV3VGhyZWFkU3RhY2tCb3VuZHMpOgorICAgICAgICAqIHd0 Zi9TdGFja0JvdW5kcy5oOgorICAgICAgICAoV1RGOjpTdGFja0JvdW5kczo6Y2hlY2tDb25zaXN0 ZW5jeSBjb25zdCk6CisKIDIwMTgtMTAtMDYgIE1hcmsgTGFtICA8bWFyay5sYW1AYXBwbGUuY29t PgogCiAgICAgICAgIEFkZGluZyBzb21lIHRlbXBvcmFyeSBhc3NlcnRzIHRvIGRlYnVnIGEgbXlz dGVyaW91cyBBU0FOIGJvdCBjcmFzaC4KSW5kZXg6IFNvdXJjZS9XVEYvd3RmL1N0YWNrQm91bmRz LmNwcAo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09Ci0tLSBTb3VyY2UvV1RGL3d0Zi9TdGFja0JvdW5kcy5jcHAJKHJldmlz aW9uIDIzNjkwMykKKysrIFNvdXJjZS9XVEYvd3RmL1N0YWNrQm91bmRzLmNwcAkod29ya2luZyBj b3B5KQpAQCAtMTQ0LDYgKzE0NCwxOCBAQCBTdGFja0JvdW5kcyBTdGFja0JvdW5kczo6bmV3VGhy ZWFkU3RhY2tCCiAgICAgLy8gSWYgc3RhY2sgZ3Jvd3MgdXAsIG9yaWdpbiBhbmQgYm91bmQgaW4g dGhpcyBjb2RlIHNob3VsZCBiZSBzd2FwcGVkLgogICAgIGlmIChzdGFja0RpcmVjdGlvbigpID09 IFN0YWNrRGlyZWN0aW9uOjpVcHdhcmQpCiAgICAgICAgIHN0ZDo6c3dhcChvcmlnaW4sIGJvdW5k KTsKKworICAgIC8vIFRoZSBmb2xsb3dpbmcgc2V0IG9mIGFzc2VydGlvbnMgYXJlIG9ubHkgbmVl ZGVkIGZvciBkZWJ1Z2dpbmcgYQorICAgIC8vIG15c3RlcmlvdXMgY3Jhc2ggb24gYW4gQVNBTiBi b3QgdGhhdCBpcyBub3QgcmVwcm9kdWNpYmxlIG90aGVyd2lzZS4KKyAgICAvLyBXaWxsIHJlbW92 ZSBhZnRlciB0aGUgbmVlZGVkIGRhdGEgaGFzIGJlZW4gY29sbGVjdGVkLgorI2lmIE9TKExJTlVY KQorICAgIGVsc2UgeworICAgICAgICB2b2lkKiBjdXJyZW50UG9zaXRpb24gPSAmY3VycmVudFBv c2l0aW9uOworICAgICAgICBBU1NFUlRfV0lUSF9NRVNTQUdFX1VOVVNFRChjdXJyZW50UG9zaXRp b24sIChjdXJyZW50UG9zaXRpb24gPCBvcmlnaW4gJiYgY3VycmVudFBvc2l0aW9uID4gYm91bmQp LCAibmV3VGhyZWFkU3RhY2tCb3VuZHM6IGN1cnJlbnRQb3NpdGlvbiAlcCBvcmlnaW4gJXAgYm91 bmQgJXAgc3RhY2tTaXplICV6dSIsIGN1cnJlbnRQb3NpdGlvbiwgb3JpZ2luLCBib3VuZCwgc3Rh Y2tTaXplKTsKKyAgICB9CisjZW5kaWYKKyAgICAvLyBFbmQgb2YgQVNBTiBib3QgZGVidWdnaW5n IGFzc2VydGlvbnMuCisKICAgICByZXR1cm4gU3RhY2tCb3VuZHMgeyBvcmlnaW4sIGJvdW5kIH07 CiB9CiAKSW5kZXg6IFNvdXJjZS9XVEYvd3RmL1N0YWNrQm91bmRzLmgKPT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0g U291cmNlL1dURi93dGYvU3RhY2tCb3VuZHMuaAkocmV2aXNpb24gMjM2OTAzKQorKysgU291cmNl L1dURi93dGYvU3RhY2tCb3VuZHMuaAkod29ya2luZyBjb3B5KQpAQCAtMTUyLDE4ICsxNTIsMTAg QEAgcHJpdmF0ZToKICAgICAgICAgLy8gbXlzdGVyaW91cyBjcmFzaCBvbiBhbiBBU0FOIGJvdCB0 aGF0IGlzIG5vdCByZXByb2R1Y2libGUgb3RoZXJ3aXNlLgogICAgICAgICAvLyBXaWxsIHJlbW92 ZSBhZnRlciB0aGUgbmVlZGVkIGRhdGEgaGFzIGJlZW4gY29sbGVjdGVkLgogI2lmIE9TKExJTlVY KQotICAgICAgICBBU1NFUlQoY3VycmVudFBvc2l0aW9uKTsKLSAgICAgICAgQVNTRVJUKG1fb3Jp Z2luKTsKLSAgICAgICAgQVNTRVJUKG1fYm91bmQpOwotICAgICAgICBBU1NFUlQoY3VycmVudFBv c2l0aW9uICE9IG1fb3JpZ2luKTsKLSAgICAgICAgQVNTRVJUKGN1cnJlbnRQb3NpdGlvbiAhPSBt X2JvdW5kKTsKLSAgICAgICAgaWYgKGlzR3Jvd2luZ0Rvd253YXJkKCkpIHsKLSAgICAgICAgICAg IEFTU0VSVChjdXJyZW50UG9zaXRpb24gPCBtX29yaWdpbik7Ci0gICAgICAgICAgICBBU1NFUlQo Y3VycmVudFBvc2l0aW9uID4gbV9ib3VuZCk7Ci0gICAgICAgIH0gZWxzZSB7Ci0gICAgICAgICAg ICBBU1NFUlQoY3VycmVudFBvc2l0aW9uID4gbV9vcmlnaW4pOwotICAgICAgICAgICAgQVNTRVJU KGN1cnJlbnRQb3NpdGlvbiA8IG1fYm91bmQpOwotICAgICAgICB9CisgICAgICAgIGlmIChpc0dy b3dpbmdEb3dud2FyZCgpKQorICAgICAgICAgICAgQVNTRVJUX1dJVEhfTUVTU0FHRSgoY3VycmVu dFBvc2l0aW9uIDwgbV9vcmlnaW4gJiYgY3VycmVudFBvc2l0aW9uID4gbV9ib3VuZCksICJjaGVj a0NvbnNpc3RlbmN5OiBjdXJyZW50UG9zaXRpb24gJXAgbV9vcmlnaW4gJXAgbV9ib3VuZCAlcCBz dGFja1NpemUgJXp1IiwgY3VycmVudFBvc2l0aW9uLCBtX29yaWdpbiwgbV9ib3VuZCwgKHJlaW50 ZXJwcmV0X2Nhc3Q8dWludDhfdCo+KG1fb3JpZ2luKSAtIHJlaW50ZXJwcmV0X2Nhc3Q8dWludDhf dCo+KG1fYm91bmQpKSk7CisgICAgICAgIGVsc2UKKyAgICAgICAgICAgIEFTU0VSVF9XSVRIX01F U1NBR0UoKGN1cnJlbnRQb3NpdGlvbiA+IG1fb3JpZ2luICYmIGN1cnJlbnRQb3NpdGlvbiA8IG1f Ym91bmQpLCAiY2hlY2tDb25zaXN0ZW5jeTogY3VycmVudFBvc2l0aW9uICVwIG1fb3JpZ2luICVw IG1fYm91bmQgJXAgc3RhY2tTaXplICV6dSIsIGN1cnJlbnRQb3NpdGlvbiwgbV9vcmlnaW4sIG1f Ym91bmQsIChyZWludGVycHJldF9jYXN0PHVpbnQ4X3QqPihtX2JvdW5kKSAtIHJlaW50ZXJwcmV0 X2Nhc3Q8dWludDhfdCo+KG1fb3JpZ2luKSkpOwogI2VuZGlmIC8vIE9TKExJTlVYKQogICAgICAg ICAvLyBFbmQgb2YgQVNBTiBib3QgZGVidWdnaW5nIGFzc2VydGlvbnMuCiAK