Bug 221115 – RunLoop::threadWillExit is doing m_nextIteration.clear() without locking m_nextIterationLock

• Component: WPE WebKit
• Status: RESOLVED FIXED

[Lauro Moura]

From time to time, some tests hit some WTF::Deque::checkValidity assertions deep inside the WTR::TestController.

There are small variations in the stack up to checkValidity and inside it but they have some portions in common. Namely, the following snippet:

#6  0x0000561b672cf4ba in WTF::Deque<WTF::Function<void ()>, 0ul>::append(WTF::Function<void ()>&&) (this=0x7f58b0367040, value=...) at ../../Source/WTF/wtf/Deque.h:89
#7  0x0000561b672ce51b in WTF::RunLoop::dispatch(WTF::Function<void ()>&&) (this=0x7f58b0367000, function=...) at ../../Source/WTF/wtf/RunLoop.cpp:146
#8  0x0000561b67358329 in WTF::WorkQueue::platformInvalidate() (this=0x7f58b0396a50) at ../../Source/WTF/wtf/generic/WorkQueueGeneric.cpp:51
#9  0x0000561b6731376c in WTF::WorkQueue::~WorkQueue() (this=0x7f58b0396a50, __in_chrg=<optimized out>) at ../../Source/WTF/wtf/WorkQueue.cpp:54
#10 0x0000561b67313794 in WTF::WorkQueue::~WorkQueue() (this=0x7f58b0396a50, __in_chrg=<optimized out>) at ../../Source/WTF/wtf/WorkQueue.cpp:55
#11 0x0000561b66948e3e in WTF::ThreadSafeRefCounted<WTF::FunctionDispatcher, (WTF::DestructionThread)0>::deref() const::{lambda()#1}::operator()() const (this=0x7ffe9d297810) at DerivedSources/ForwardingHeaders/wtf/ThreadSafeRefCounted.h:117
#12 0x0000561b66948e85 in WTF::ThreadSafeRefCounted<WTF::FunctionDispatcher, (WTF::DestructionThread)0>::deref() const (this=0x7f58b0396a58) at DerivedSources/ForwardingHeaders/wtf/ThreadSafeRefCounted.h:135
#13 0x0000561b66a9f1b4 in WTF::Ref<WTF::WorkQueue, WTF::RawPtrTraits<WTF::WorkQueue> >::~Ref() (this=0x7f58b03972f0, __in_chrg=<optimized out>) at DerivedSources/ForwardingHeaders/wtf/Ref.h:61

The bottom of the stack also varies a bit starting from restStateToConsistentValues(). For example, starting either in ~ContentRuleListStore (TestController::resetContentExtensions()) or from ~Connection (TestController::clearIndexedDatabases()).

So far, no luck trying to reproduce locally, even getting the same list of tests of the worker that crashed.

Detailed crash logs in the following comments.

[Lauro Moura]

Created attachment 418697 [details]
svg/wicd/test-rightsizing-b.xhtml crash log from debug build 5743

svg/wicd/test-rightsizing-b.xhtml

Debug build 5743 (r272000)

Thread 1:

STDERR: ASSERTION FAILED: m_end < m_buffer.capacity()
STDERR: ../../Source/WTF/wtf/Deque.h(264) : void WTF::Deque<T, inlineCapacity>::checkValidity() const [with T = WTF::Function<void()>; long unsigned int inlineCapacity = 0]

Thread 1 (Thread 0x7f58b0c6e9c0 (LWP 90625)):
#0  WTFCrash() () at ../../Source/WTF/wtf/Assertions.cpp:295
#1  0x0000561b6564bcc8 in CRASH_WITH_INFO(...) () at DerivedSources/ForwardingHeaders/wtf/Assertions.h:713
#2  0x0000561b672cf9a7 in WTF::Deque<WTF::Function<void ()>, 0ul>::checkValidity() const (this=0x7f58b0367040) at ../../Source/WTF/wtf/Deque.h:264
#3  0x0000561b672d0381 in WTF::Deque<WTF::Function<void ()>, 0ul>::expandCapacity() (this=0x7f58b0367040) at ../../Source/WTF/wtf/Deque.h:444
#4  0x0000561b672cfa3c in WTF::Deque<WTF::Function<void ()>, 0ul>::expandCapacityIfNeeded() (this=0x7f58b0367040) at ../../Source/WTF/wtf/Deque.h:425
#5  0x0000561b672cfe6f in WTF::Deque<WTF::Function<void ()>, 0ul>::append<WTF::Function<void ()> >(WTF::Function<void ()>&&) (this=0x7f58b0367040, value=...) at ../../Source/WTF/wtf/Deque.h:475
#6  0x0000561b672cf4ba in WTF::Deque<WTF::Function<void ()>, 0ul>::append(WTF::Function<void ()>&&) (this=0x7f58b0367040, value=...) at ../../Source/WTF/wtf/Deque.h:89
#7  0x0000561b672ce51b in WTF::RunLoop::dispatch(WTF::Function<void ()>&&) (this=0x7f58b0367000, function=...) at ../../Source/WTF/wtf/RunLoop.cpp:146
#8  0x0000561b67358329 in WTF::WorkQueue::platformInvalidate() (this=0x7f58b0396a50) at ../../Source/WTF/wtf/generic/WorkQueueGeneric.cpp:51
#9  0x0000561b6731376c in WTF::WorkQueue::~WorkQueue() (this=0x7f58b0396a50, __in_chrg=<optimized out>) at ../../Source/WTF/wtf/WorkQueue.cpp:54
#10 0x0000561b67313794 in WTF::WorkQueue::~WorkQueue() (this=0x7f58b0396a50, __in_chrg=<optimized out>) at ../../Source/WTF/wtf/WorkQueue.cpp:55
#11 0x0000561b66948e3e in WTF::ThreadSafeRefCounted<WTF::FunctionDispatcher, (WTF::DestructionThread)0>::deref() const::{lambda()#1}::operator()() const (this=0x7ffe9d297810) at DerivedSources/ForwardingHeaders/wtf/ThreadSafeRefCounted.h:117
#12 0x0000561b66948e85 in WTF::ThreadSafeRefCounted<WTF::FunctionDispatcher, (WTF::DestructionThread)0>::deref() const (this=0x7f58b0396a58) at DerivedSources/ForwardingHeaders/wtf/ThreadSafeRefCounted.h:135
#13 0x0000561b66a9f1b4 in WTF::Ref<WTF::WorkQueue, WTF::RawPtrTraits<WTF::WorkQueue> >::~Ref() (this=0x7f58b03972f0, __in_chrg=<optimized out>) at DerivedSources/ForwardingHeaders/wtf/Ref.h:61
#14 0x00007f58c59ff942 in API::ContentRuleListStore::~ContentRuleListStore() (this=0x7f58b03972d8, __in_chrg=<optimized out>) at ../../Source/WebKit/UIProcess/API/APIContentRuleListStore.cpp:96
#15 0x00007f58c59ff97a in API::ContentRuleListStore::~ContentRuleListStore() (this=0x7f58b03972d8, __in_chrg=<optimized out>) at ../../Source/WebKit/UIProcess/API/APIContentRuleListStore.cpp:98
#16 0x00007f58c4bb3aa6 in WTF::ThreadSafeRefCounted<API::Object, (WTF::DestructionThread)0>::deref() const::{lambda()#1}::operator()() const (this=0x7ffe9d2978c0) at DerivedSources/ForwardingHeaders/wtf/ThreadSafeRefCounted.h:117
#17 0x00007f58c4bb3aed in WTF::ThreadSafeRefCounted<API::Object, (WTF::DestructionThread)0>::deref() const (this=0x7f58b03972e0) at DerivedSources/ForwardingHeaders/wtf/ThreadSafeRefCounted.h:135
#18 0x00007f58c57c18aa in WKRelease(WKTypeRef) (typeRef=0x7f58b03972d8) at ../../Source/WebKit/Shared/API/c/WKType.cpp:46
#19 0x0000561b6566f7ca in WebKit::WKRetainPtr<OpaqueWKUserContentExtensionStore const*>::~WKRetainPtr() (this=0x7ffe9d297948, __in_chrg=<optimized out>) at ../../Source/WebKit/UIProcess/API/cpp/WKRetainPtr.h:77
#20 0x0000561b656596d2 in WTR::TestController::resetContentExtensions() (this=0x7ffe9d298620) at ../../Tools/WebKitTestRunner/TestController.cpp:1427
#21 0x0000561b65657b23 in WTR::TestController::resetStateToConsistentValues(WTR::TestOptions const&, WTR::TestController::ResetStage) (this=0x7ffe9d298620, options=..., resetStage=WTR::TestController::ResetStage::AfterTest) at ../../Tools/WebKitTestRunner/TestController.cpp:1043
#22 0x0000561b6568bb5d in WTR::TestInvocation::invoke() (this=0x7f58701ec840) at ../../Tools/WebKitTestRunner/TestInvocation.cpp:180
#23 0x0000561b6565997e in WTR::TestController::runTest(char const*) (this=0x7ffe9d298620, inputLine=0x7ffe9d297d90 "/app/webkit/LayoutTests/svg/wicd/test-rightsizing-a.xhtml'--timeout'30000") at ../../Tools/WebKitTestRunner/TestController.cpp:1476
#24 0x0000561b65659b95 in WTR::TestController::runTestingServerLoop() (this=0x7ffe9d298620) at ../../Tools/WebKitTestRunner/TestController.cpp:1522
#25 0x0000561b65659bea in WTR::TestController::run() (this=0x7ffe9d298620) at ../../Tools/WebKitTestRunner/TestController.cpp:1530
#26 0x0000561b656543c3 in WTR::TestController::TestController(int, char const**) (this=0x7ffe9d298620, argc=2, argv=0x7ffe9d298ac8) at ../../Tools/WebKitTestRunner/TestController.cpp:193
#27 0x0000561b656bb7b4 in main(int, char**) (argc=2, argv=0x7ffe9d298ac8) at ../../Tools/WebKitTestRunner/wpe/main.cpp:35

[Lauro Moura]

Created attachment 418698 [details]
fast/dynamic/insertAdjacentHTML.html from debug build 5743

fast/dynamic/insertAdjacentHTML.html

Same build as the previous comment.

STDERR: ASSERTION FAILED: !m_end
STDERR: ../../Source/WTF/wtf/Deque.h(261) : void WTF::Deque<T, inlineCapacity>::checkValidity() const [with T = WTF::Function<void()>; long unsigned int inlineCapacity = 0]

Thread 1 (Thread 0x7f11fc6769c0 (LWP 34988)):
#0  WTFCrash() () at ../../Source/WTF/wtf/Assertions.cpp:295
#1  0x000055cb00efccc8 in CRASH_WITH_INFO(...) () at DerivedSources/ForwardingHeaders/wtf/Assertions.h:713
#2  0x000055cb02b80912 in WTF::Deque<WTF::Function<void ()>, 0ul>::checkValidity() const (this=0x7f11fbdbe040) at ../../Source/WTF/wtf/Deque.h:261
#3  0x000055cb02b80f19 in WTF::Deque<WTF::Function<void ()>, 0ul>::append<WTF::Function<void ()> >(WTF::Function<void ()>&&) (this=0x7f11fbdbe040, value=...) at ../../Source/WTF/wtf/Deque.h:481
#4  0x000055cb02b804ba in WTF::Deque<WTF::Function<void ()>, 0ul>::append(WTF::Function<void ()>&&) (this=0x7f11fbdbe040, value=...) at ../../Source/WTF/wtf/Deque.h:89
#5  0x000055cb02b7f51b in WTF::RunLoop::dispatch(WTF::Function<void ()>&&) (this=0x7f11fbdbe000, function=...) at ../../Source/WTF/wtf/RunLoop.cpp:146
#6  0x000055cb02c09329 in WTF::WorkQueue::platformInvalidate() (this=0x7f11fbd9ee10) at ../../Source/WTF/wtf/generic/WorkQueueGeneric.cpp:51
#7  0x000055cb02bc476c in WTF::WorkQueue::~WorkQueue() (this=0x7f11fbd9ee10, __in_chrg=<optimized out>) at ../../Source/WTF/wtf/WorkQueue.cpp:54
#8  0x000055cb02bc4794 in WTF::WorkQueue::~WorkQueue() (this=0x7f11fbd9ee10, __in_chrg=<optimized out>) at ../../Source/WTF/wtf/WorkQueue.cpp:55
#9  0x000055cb021f9e3e in WTF::ThreadSafeRefCounted<WTF::FunctionDispatcher, (WTF::DestructionThread)0>::deref() const::{lambda()#1}::operator()() const (this=0x7fffc5cca980) at DerivedSources/ForwardingHeaders/wtf/ThreadSafeRefCounted.h:117
#10 0x000055cb021f9e85 in WTF::ThreadSafeRefCounted<WTF::FunctionDispatcher, (WTF::DestructionThread)0>::deref() const (this=0x7f11fbd9ee18) at DerivedSources/ForwardingHeaders/wtf/ThreadSafeRefCounted.h:135
#11 0x000055cb023501b4 in WTF::Ref<WTF::WorkQueue, WTF::RawPtrTraits<WTF::WorkQueue> >::~Ref() (this=0x7f11fbdcc580, __in_chrg=<optimized out>) at DerivedSources/ForwardingHeaders/wtf/Ref.h:61
#12 0x00007f12110ccae9 in IPC::Connection::~Connection() (this=0x7f11fbdcc540, __in_chrg=<optimized out>) at ../../Source/WebKit/Platform/IPC/Connection.cpp:284
#13 0x00007f12105f91c5 in WTF::ThreadSafeRefCounted<IPC::Connection, (WTF::DestructionThread)2>::deref() const::{lambda()#1}::operator()() const (this=0x7f11fbdca798) at DerivedSources/ForwardingHeaders/wtf/ThreadSafeRefCounted.h:117
#14 0x00007f1210603a12 in WTF::Detail::CallableWrapper<WTF::ThreadSafeRefCounted<IPC::Connection, (WTF::DestructionThread)2>::deref() const::{lambda()#1}, void>::call() (this=0x7f11fbdca790) at DerivedSources/ForwardingHeaders/wtf/Function.h:52
#15 0x000055cb00f1dfdf in WTF::Function<void ()>::operator()() const (this=0x7fffc5ccaaa0) at DerivedSources/ForwardingHeaders/wtf/Function.h:83
#16 0x000055cb02b7f44d in WTF::RunLoop::performWork() (this=0x7f11fbdf9000) at ../../Source/WTF/wtf/RunLoop.cpp:128
#17 0x000055cb02c0ef00 in operator()(gpointer) const (__closure=0x0, userData=0x7f11fbdf9000) at ../../Source/WTF/wtf/glib/RunLoopGLib.cpp:80
#18 0x000055cb02c0ef24 in _FUN(gpointer) () at ../../Source/WTF/wtf/glib/RunLoopGLib.cpp:82
#19 0x000055cb02c0ee93 in operator()(GSource*, GSourceFunc, gpointer) const (__closure=0x0, source=0x55cb047a8b90, callback=0x55cb02c0ef07 <_FUN(gpointer)>, userData=0x7f11fbdf9000) at ../../Source/WTF/wtf/glib/RunLoopGLib.cpp:53
#20 0x000055cb02c0eee1 in _FUN(GSource*, GSourceFunc, gpointer) () at ../../Source/WTF/wtf/glib/RunLoopGLib.cpp:56
#21 0x00007f121ff3318f in g_main_dispatch (context=0x55cb047a79c0) at ../glib/gmain.c:3325
#22 g_main_context_dispatch (context=0x55cb047a79c0) at ../glib/gmain.c:4043
#23 0x00007f121ff33538 in g_main_context_iterate (context=0x55cb047a79c0, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4119
#24 0x00007f121ff33853 in g_main_loop_run (loop=0x55cb047a8b70) at ../glib/gmain.c:4317
#25 0x000055cb02c0f4aa in WTF::RunLoop::run() () at ../../Source/WTF/wtf/glib/RunLoopGLib.cpp:108
#26 0x000055cb00f68c95 in WTR::TestController::platformRunUntil(bool&, WTF::Seconds) (this=0x7fffc5ccbc80, done=@0x7fffc5ccadc8: false, timeout=...) at ../../Tools/WebKitTestRunner/wpe/TestControllerWPE.cpp:83
#27 0x000055cb00f0acbb in WTR::TestController::runUntil(bool&, WTF::Seconds) (this=0x7fffc5ccbc80, done=@0x7fffc5ccadc8: false, timeout=...) at ../../Tools/WebKitTestRunner/TestController.cpp:1546
#28 0x000055cb00f10842 in WTR::TestController::clearIndexedDatabases() (this=0x7fffc5ccbc80) at ../../Tools/WebKitTestRunner/TestController.cpp:2868
#29 0x000055cb00f08857 in WTR::TestController::resetStateToConsistentValues(WTR::TestOptions const&, WTR::TestController::ResetStage) (this=0x7fffc5ccbc80, options=..., resetStage=WTR::TestController::ResetStage::BeforeTest) at ../../Tools/WebKitTestRunner/TestController.cpp:986
#30 0x000055cb00f07978 in WTR::TestController::ensureViewSupportsOptionsForTest(WTR::TestInvocation const&) (this=0x7fffc5ccbc80, test=...) at ../../Tools/WebKitTestRunner/TestController.cpp:857
#31 0x000055cb00f09c87 in WTR::TestController::configureViewForTest(WTR::TestInvocation const&) (this=0x7fffc5ccbc80, test=...) at ../../Tools/WebKitTestRunner/TestController.cpp:1319
#32 0x000055cb00f3c965 in WTR::TestInvocation::invoke() (this=0x7f11bbbf0580) at ../../Tools/WebKitTestRunner/TestInvocation.cpp:144
#33 0x000055cb00f0a97e in WTR::TestController::runTest(char const*) (this=0x7fffc5ccbc80, inputLine=0x7fffc5ccb3f0 "/app/webkit/LayoutTests/fast/dynamic/insertAdjacentHTML.html'--timeout'30000") at ../../Tools/WebKitTestRunner/TestController.cpp:1476
#34 0x000055cb00f0ab95 in WTR::TestController::runTestingServerLoop() (this=0x7fffc5ccbc80) at ../../Tools/WebKitTestRunner/TestController.cpp:1522
#35 0x000055cb00f0abea in WTR::TestController::run() (this=0x7fffc5ccbc80) at ../../Tools/WebKitTestRunner/TestController.cpp:1530
#36 0x000055cb00f053c3 in WTR::TestController::TestController(int, char const**) (this=0x7fffc5ccbc80, argc=2, argv=0x7fffc5ccc128) at ../../Tools/WebKitTestRunner/TestController.cpp:193
#37 0x000055cb00f6c7b4 in main(int, char**) (argc=2, argv=0x7fffc5ccc128) at ../../Tools/WebKitTestRunner/wpe/main.cpp:35

[Lauro Moura]

This kind of failure is very rare, usually once for each test (<5 crashes per run, when it happens), so there is no need to garden them as flaky crash for now.

[Fujii Hironori]

*** Bug 222251 has been marked as a duplicate of this bug. ***

[Fujii Hironori]

GTK port is also observing this assertion failure. (Bug 222251)
I can reproduce this assertion failure with my local GTK Debug build.

RunLoop::threadWillExit() is calling m_nextIteration.clear() without locking m_nextIterationLock.

[Fujii Hironori]

r270496 (Bug 219232) seems the culprit.

[Fujii Hironori]

Created attachment 421160 [details]
Patch

[Yusuke Suzuki]

Comment on attachment 421160 [details]
Patch

r=me

[EWS]

Committed r273230: <https://commits.webkit.org/r273230>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 421160 [details].