Bug 190334 – Adding some temporary asserts with data logging to debug a mysterious ASAN bot crash.

• Status: RESOLVED
• Resolution: FIXED
• Component: JavaScriptCore

[Mark Lam]

Need some additional data to debug this issue.

[Radar WebKit Bug Importer]

<rdar://problem/45071303>

[Mark Lam]

Created attachment 351737 [details]
proposed patch.

[Yusuke Suzuki]

Comment on attachment 351737 [details]
proposed patch.

View in context: https://bugs.webkit.org/attachment.cgi?id=351737&action=review

r=me with a nit.

> Source/WTF/wtf/StackBounds.cpp:152
> +    else {

I think using `if (stackDirection() == StackDirection::Downward)` here would be more readable.

[Mark Lam]

Comment on attachment 351737 [details]
proposed patch.

View in context: https://bugs.webkit.org/attachment.cgi?id=351737&action=review

Thanks.

>> Source/WTF/wtf/StackBounds.cpp:152
>> +    else {
> 
> I think using `if (stackDirection() == StackDirection::Downward)` here would be more readable.

Since Upward and Downward are mutually exclusive, I'll just add an ASSERT(stackDirection() == StackDirection::Downward) in the else block here to document the condition.  I feel like the "else if" would (wrongly) imply that a 3rd option is possible.

[Mark Lam]

Landed in r236909: <http://trac.webkit.org/r236909>.

[Keith Miller]

Comment on attachment 351737 [details]
proposed patch.

r=me too.

[Philippe Normand]

This broke the GTK Debug tests bot:

Thread 1 (LWP 13935):
#0  0x000055d80120a116 in WTFCrash () at /home/slave/webkitgtk/gtk-linux-64-debug/build/Source/WTF/wtf/Assertions.cpp:255
Backtrace stopped: Cannot access memory at address 0x7ffc49fab0a8

STDERR: BFD: Warning: /var/www/cores/bb-webkit-debug-test-64/core-pid_13935.dump is truncated: expected core file size >= 241209344, found: 6823936.
STDERR: 
STDERR: warning: core file may not match specified executable file.
STDERR: Cannot access memory at address 0x7f0e5bf25148
STDERR: Cannot access memory at address 0x7f0e5bf25140
STDERR: Failed to read a valid object file image from memory.
STDERR: ASSERTION FAILED: newThreadStackBounds: currentPosition 0x7ffc49fab0c0 origin 0x7f0de7131000 bound 0x7f0de6930000 stackSize 8392704
STDERR: (currentPosition < origin && currentPosition > bound)
STDERR: /home/slave/webkitgtk/gtk-linux-64-debug/build/Source/WTF/wtf/StackBounds.cpp(155) : static WTF::StackBounds WTF::StackBounds::newThreadStackBounds(WTF::PlatformThreadHandle)
STDERR: 1   0x55d80120a111 /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTFCrash+0x9) [0x55d80120a111]
STDERR: 2   0x55d80128f23f /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTF::StackBounds::newThreadStackBounds(unsigned long)+0x16f) [0x55d80128f23f]
STDERR: 3   0x55d8012409be /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTF::Thread::create(char const*, WTF::Function<void ()>&&)+0x180) [0x55d8012409be]
STDERR: 4   0x55d801289db2 /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTF::WorkQueue::platformInitialize(char const*, WTF::WorkQueue::Type, WTF::WorkQueue::QOS)+0x5c) [0x55d801289db2]
STDERR: 5   0x55d8012478de /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTF::WorkQueue::WorkQueue(char const*, WTF::WorkQueue::Type, WTF::WorkQueue::QOS)+0x4a) [0x55d8012478de]
STDERR: 6   0x55d801247879 /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTF::WorkQueue::create(char const*, WTF::WorkQueue::Type, WTF::WorkQueue::QOS)+0x39) [0x55d801247879]
STDERR: 7   0x7f0e538e0f2b /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Debug/lib/libwebkit2gtk-4.0.so.37(WebKit::StorageManager::StorageManager(WTF::String const&)+0x4f) [0x7f0e538e0f2b]
STDERR: 8   0x7f0e538e0ec1 /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Debug/lib/libwebkit2gtk-4.0.so.37(WebKit::StorageManager::create(WTF::String const&)+0x2d) [0x7f0e538e0ec1]
STDERR: 9   0x7f0e538e5120 /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Debug/lib/libwebkit2gtk-4.0.so.37(WebKit::WebsiteDataStore::WebsiteDataStore(WebKit::WebsiteDataStore::Configuration, PAL::SessionID)+0xca) [0x7f0e538e5120]
STDERR: 10  0x7f0e538e4f1a /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Debug/lib/libwebkit2gtk-4.0.so.37(WebKit::WebsiteDataStore::create(WebKit::WebsiteDataStore::Configuration, PAL::SessionID)+0x78) [0x7f0e538e4f1a]
STDERR: 11  0x7f0e5372869e /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Debug/lib/libwebkit2gtk-4.0.so.37(API::WebsiteDataStore::WebsiteDataStore(WebKit::WebsiteDataStore::Configuration, PAL::SessionID)+0x86) [0x7f0e5372869e]
STDERR: 12  0x7f0e53728595 /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Debug/lib/libwebkit2gtk-4.0.so.37(API::WebsiteDataStore::createLegacy(WebKit::WebsiteDataStore::Configuration)+0x6f) [0x7f0e53728595]
STDERR: 13  0x7f0e53622d38 /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Debug/lib/libwebkit2gtk-4.0.so.37(WebKit::WebProcessPool::WebProcessPool(API::ProcessPoolConfiguration&)+0x812) [0x7f0e53622d38]
STDERR: 14  0x7f0e53621eac /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Debug/lib/libwebkit2gtk-4.0.so.37(WebKit::WebProcessPool::create(API::ProcessPoolConfiguration&)+0x32) [0x7f0e53621eac]
STDERR: 15  0x7f0e5372cb04 /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Debug/lib/libwebkit2gtk-4.0.so.37(WKContextCreateWithConfiguration+0x2b) [0x7f0e5372cb04]
STDERR: 16  0x55d8011c01e2 /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTR::TestController::generatePageConfiguration(OpaqueWKContextConfiguration const*)+0x30) [0x55d8011c01e2]
STDERR: 17  0x55d8011c085a /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTR::TestController::createWebViewWithOptions(WTR::TestOptions const&)+0x1ac) [0x55d8011c085a]
STDERR: 18  0x55d8011c0dec /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTR::TestController::ensureViewSupportsOptionsForTest(WTR::TestInvocation const&)+0x14e) [0x55d8011c0dec]
STDERR: 19  0x55d8011c3ba3 /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTR::TestController::configureViewForTest(WTR::TestInvocation const&)+0x23) [0x55d8011c3ba3]
STDERR: 20  0x55d8011e23de /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTR::TestInvocation::invoke()+0x26) [0x55d8011e23de]
STDERR: 21  0x55d8011c44c3 /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTR::TestController::runTest(char const*)+0x235) [0x55d8011c44c3]
STDERR: 22  0x55d8011c4697 /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTR::TestController::runTestingServerLoop()+0xa5) [0x55d8011c4697]
STDERR: 23  0x55d8011c46d8 /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTR::TestController::run()+0x24) [0x55d8011c46d8]
STDERR: 24  0x55d8011beefa /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(WTR::TestController::TestController(int, char const**)+0x30c) [0x55d8011beefa]
STDERR: 25  0x55d801209a77 /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(main+0xe2) [0x55d801209a77]
STDERR: 26  0x7f0e3fbbc2b1 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f0e3fbbc2b1]
STDERR: 27  0x55d8011ba53a /home/slave/webkitgtk/gtk-linux-64-debug-tests/build/WebKitBuild/Debug/bin/WebKitTestRunner(_start+0x2a) [0x55d8011ba53a]

[Philippe Normand]

AFAIK The GTK Debug bot doesn't have ASAN enabled. Any chance this assert could be raised only if ASAN is enabled?

[Yusuke Suzuki]

(In reply to Philippe Normand from comment #8)
> AFAIK The GTK Debug bot doesn't have ASAN enabled. Any chance this assert
> could be raised only if ASAN is enabled?

Ah, right. This assertion was wrong. This StackBounds::newThreadStackBounds can be called from the other thread. Typically, the caller thread (creating a thread) will call this function for the newly created thread. It is important since it removes one ping-pong when creating a new thread.

[Yusuke Suzuki]

Ah, and I think this is the reason why ASAN bot intermittently fails.
This `StackBounds::newThreadStackBounds` can be called from the caller thread.
It always succeeds. But, internally, the thread would be in the middle of the initialization (start running?).
Even in that case, when stack area is already allocated, so this function is OK.
But these region would not be marked as ASAN-safe-region when calling this function!

@Mark, what do you think of the above hypothesis?

[Yusuke Suzuki]

(In reply to Yusuke Suzuki from comment #10)
> Ah, and I think this is the reason why ASAN bot intermittently fails.
> This `StackBounds::newThreadStackBounds` can be called from the caller
> thread.
> It always succeeds. But, internally, the thread would be in the middle of
> the initialization (start running?).
> Even in that case, when stack area is already allocated, so this function is
> OK.
> But these region would not be marked as ASAN-safe-region when calling this
> function!
> 
> @Mark, what do you think of the above hypothesis?

I would like to know where the ASAN bot actually fails.
If ASAN bot fails due to pthread_xxx call to retrieve StackBounds, it seems OK.
But if ASAN crashes due to touching the stack memory early, it should not be done since these stack memory would not be safe if the region is not the above of the stack pointer of the thread.


<origin> ====== safe ====== <stack pointer> ======= unsafe ====== <bound>

[Mark Lam]

(In reply to Philippe Normand from comment #7)
> This broke the GTK Debug tests bot:

Sorry about this.  I've gotten the info I needed: the issue lies with the use of the detect_stack_use_after_return ASAB option.  Will be reverting these asserts in https://bugs.webkit.org/show_bug.cgi?id=190396.