Bug 195161

Summary:	Flaky API Test: TestWTF.WTF_SynchronizedFixedQueue.ProduceOnly
Product:	WebKit	Reporter:	Aakash Jain <aakash_jain>
Component:	Tools / Tests	Assignee:	Fujii Hironori <Hironori.Fujii>
Status:	RESOLVED FIXED
Severity:	Normal	CC:	aakash_jain, ap, Hironori.Fujii, lforschler, sabouhallawa, webkit-bot-watchers-bugzilla, webkit-bug-importer
Priority:	P2	Keywords:	InRadar
Version:	Other
Hardware:	Unspecified
OS:	Unspecified
See Also:	https://bugs.webkit.org/show_bug.cgi?id=176056

Description Aakash Jain 2019-02-28 05:58:23 PST

TestWTF.WTF_SynchronizedFixedQueue.ProduceOnly seems flaky. In https://ews-build.webkit-uat.org/#/builders/19/builds/1307, the test Failed in run-api-tests step. However, in the immediately next retry step (re-run-api-tests), it passed.

Comment 1 Radar WebKit Bug Importer 2019-05-06 13:16:21 PDT

<rdar://problem/50510054>

Comment 2 Fujii Hironori 2023-04-26 22:58:07 PDT

Only WinCairo port is randomly timing out these days.
https://results.webkit.org/?suite=api-tests&test=TestWTF.WTF_SynchronizedFixedQueue.ProduceOnly

Comment 3 Fujii Hironori 2023-04-26 22:59:22 PDT

Main thread

> ntdll.dll!00007fff6d970a74()	Unknown
> ntdll.dll!00007fff6d934131()	Unknown
> KernelBase.dll!00007fff6b1030e9()	Unknown
> WTF.dll!WTF::ThreadCondition::timedWait(WTF::Mutex & mutex, WTF::WallTime absoluteTime) Line 386	C++
> WTF.dll!WTF::ParkingLot::parkConditionallyImpl(const void * address, const WTF::ScopedLambda<bool __cdecl(void)> & validation, const WTF::ScopedLambda<void __cdecl(void)> & beforeSleep, const WTF::TimeWithDynamicClockType & timeout) Line 602	C++
> WTF.dll!WTF::ParkingLot::parkConditionally<`WTF::Condition::waitUntilUnchecked<WTF::Lock>'::`7'::<lambda_1>,`WTF::Condition::waitUntilUnchecked<WTF::Lock>'::`7'::<lambda_2>>(const void * address, const WTF::Condition::waitUntilUnchecked::__l7::<lambda_1> & validation, const WTF::Condition::waitUntilUnchecked::__l7::<lambda_2> & beforeSleep, const WTF::TimeWithDynamicClockType & timeout) Line 82	C++
> WTF.dll!WTF::Condition::waitUntilUnchecked<WTF::Lock>(WTF::Lock & lock, const WTF::TimeWithDynamicClockType & timeout) Line 192	C++
> WTF.dll!WTF::Condition::waitUntil(WTF::Lock & lock, const WTF::TimeWithDynamicClockType & timeout) Line 78	C++
> WTF.dll!WTF::Condition::waitUntilUnchecked<WTF::Lock,`WTF::BinarySemaphore::waitUntil'::`2'::<lambda_1>>(WTF::Lock & lock, const WTF::TimeWithDynamicClockType & timeout, const WTF::BinarySemaphore::waitUntil::__l2::<lambda_1> & predicate) Line 213	C++
> WTF.dll!WTF::Condition::waitUntil<`WTF::BinarySemaphore::waitUntil'::`2'::<lambda_1>>(WTF::Lock & lock, const WTF::TimeWithDynamicClockType & timeout, const WTF::BinarySemaphore::waitUntil::__l2::<lambda_1> & predicate) Line 92	C++
> WTF.dll!WTF::BinarySemaphore::waitUntil(const WTF::TimeWithDynamicClockType & absoluteTime) Line 41	C++
> TestWTFLib.dll!WTF::BinarySemaphore::wait() Line 52	C++
> TestWTFLib.dll!TestWebKitAPI::ToUpperConverter<4>::stopProducing() Line 127	C++
> TestWTFLib.dll!TestWebKitAPI::ToUpperConverter<4>::stop() Line 143	C++
> TestWTFLib.dll!TestWebKitAPI::WTF_SynchronizedFixedQueue_ProduceOnly_Test::TestBody() Line 198	C++
> TestWTFLib.dll!testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test,void>(testing::Test * object, void(testing::Test::*)() method, const char * location) Line 2592	C++
> TestWTFLib.dll!testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,void>(testing::Test * object, void(testing::Test::*)() method, const char * location) Line 2662	C++
> TestWTFLib.dll!testing::Test::Run() Line 2689	C++
> TestWTFLib.dll!testing::TestInfo::Run() Line 2864	C++
> TestWTFLib.dll!testing::TestSuite::Run() Line 3016	C++
> TestWTFLib.dll!testing::internal::UnitTestImpl::RunAllTests() Line 5852	C++
> TestWTFLib.dll!testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool>(testing::internal::UnitTestImpl * object, bool(testing::internal::UnitTestImpl::*)() method, const char * location) Line 2592	C++
> TestWTFLib.dll!testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool>(testing::internal::UnitTestImpl * object, bool(testing::internal::UnitTestImpl::*)() method, const char * location) Line 2662	C++
> TestWTFLib.dll!testing::UnitTest::Run() Line 5434	C++
> TestWTFLib.dll!RUN_ALL_TESTS() Line 2472	C++
> TestWTFLib.dll!TestWebKitAPI::TestsController::run(int argc, char * * argv) Line 89	C++
> TestWTFLib.dll!main(int argc, char * * argv) Line 41	C++
> TestWTFLib.dll!dllLauncherEntryPoint(int argc, char * * argv) Line 49	C++
> TestWTF.exe!main(int argc, const char * * argv) Line 149	C++
> [Inline Frame] TestWTF.exe!invoke_main() Line 78	C++
> TestWTF.exe!__scrt_common_main_seh() Line 288	C++
> kernel32.dll!00007fff6d4b7604()	Unknown
> ntdll.dll!00007fff6d9226a1()	Unknown

Produce thread

> ntdll.dll!00007fff6d970a74()	Unknown
> ntdll.dll!00007fff6d934131()	Unknown
> KernelBase.dll!00007fff6b1030e9()	Unknown
> WTF.dll!WTF::ThreadCondition::timedWait(WTF::Mutex & mutex, WTF::WallTime absoluteTime) Line 386	C++
> WTF.dll!WTF::ParkingLot::parkConditionallyImpl(const void * address, const WTF::ScopedLambda<bool __cdecl(void)> & validation, const WTF::ScopedLambda<void __cdecl(void)> & beforeSleep, const WTF::TimeWithDynamicClockType & timeout) Line 602	C++
> TestWTFLib.dll!WTF::ParkingLot::parkConditionally<`WTF::Condition::waitUntilUnchecked<WTF::Lock>'::`7'::<lambda_1>,`WTF::Condition::waitUntilUnchecked<WTF::Lock>'::`7'::<lambda_2>>(const void * address, const WTF::Condition::waitUntilUnchecked::__l7::<lambda_1> & validation, const WTF::Condition::waitUntilUnchecked::__l7::<lambda_2> & beforeSleep, const WTF::TimeWithDynamicClockType & timeout) Line 82	C++
> TestWTFLib.dll!WTF::Condition::waitUntilUnchecked<WTF::Lock>(WTF::Lock & lock, const WTF::TimeWithDynamicClockType & timeout) Line 192	C++
> TestWTFLib.dll!WTF::Condition::waitUntil(WTF::Lock & lock, const WTF::TimeWithDynamicClockType & timeout) Line 78	C++
> TestWTFLib.dll!WTF::Condition::wait(WTF::Lock & lock) Line 128	C++
> TestWTFLib.dll!WTF::Condition::wait<`WTF::SynchronizedFixedQueue<WTF::CString,4>::enqueue'::`2'::<lambda_1>>(WTF::Lock & lock, const WTF::SynchronizedFixedQueue<WTF::CString,4>::enqueue::__l2::<lambda_1> & predicate) Line 141	C++
> TestWTFLib.dll!WTF::SynchronizedFixedQueue<WTF::CString,4>::enqueue(const WTF::CString & value) Line 82	C++
> TestWTFLib.dll!TestWebKitAPI::ToUpperConverter<4>::startProducing::__l2::<lambda_1>::operator()() Line 86	C++
> TestWTFLib.dll!WTF::Detail::CallableWrapper<`TestWebKitAPI::ToUpperConverter<4>::startProducing'::`2'::<lambda_1>,void>::call() Line 53	C++
> WTF.dll!WTF::Function<void __cdecl(void)>::operator()() Line 83	C++
> WTF.dll!WTF::WorkQueueBase::dispatch::__l2::<lambda_1>::operator()() Line 70	C++
> WTF.dll!WTF::Detail::CallableWrapper<`WTF::WorkQueueBase::dispatch'::`2'::<lambda_1>,void>::call() Line 53	C++
> WTF.dll!WTF::Function<void __cdecl(void)>::operator()() Line 83	C++
> WTF.dll!WTF::RunLoop::performWork() Line 148	C++
> WTF.dll!WTF::RunLoop::wndProc(HWND__ * hWnd, unsigned int message, unsigned __int64 wParam, __int64 lParam) Line 57	C++
> WTF.dll!WTF::RunLoop::RunLoopWndProc(HWND__ * hWnd, unsigned int message, unsigned __int64 wParam, __int64 lParam) Line 39	C++
> user32.dll!00007fff6bc6e858()	Unknown
> user32.dll!00007fff6bc6e299()	Unknown
> WTF.dll!WTF::RunLoop::run() Line 74	C++
> WTF.dll!WTF::RunLoop::create::__l2::<lambda_1>::operator()() Line 115	C++
> WTF.dll!WTF::Detail::CallableWrapper<`WTF::RunLoop::create'::`2'::<lambda_1>,void>::call() Line 53	C++
> WTF.dll!WTF::Function<void __cdecl(void)>::operator()() Line 83	C++
> WTF.dll!WTF::Thread::entryPoint(WTF::Thread::NewThreadContext * newThreadContext) Line 251	C++
> WTF.dll!WTF::wtfThreadEntryPoint(void * data) Line 151	C++
> ucrtbase.dll!00007fff6b761bb2()	Unknown
> kernel32.dll!00007fff6d4b7604()	Unknown
> ntdll.dll!00007fff6d9226a1()	Unknown

Comment 4 Fujii Hironori 2023-04-26 23:39:26 PDT

Produce thread is blocked by enqueueing to m_upperQueue becuase it is full.
https://github.com/WebKit/WebKit/blob/7327aab70ea504499e0f1a14c78f55863233afd9/Tools/TestWebKitAPI/Tests/WTF/SynchronizedFixedQueue.cpp#L86

Main thread is blocked by m_produceCloseSemaphore.wait() because Produce thread doesn't signal.
https://github.com/WebKit/WebKit/blob/7327aab70ea504499e0f1a14c78f55863233afd9/Tools/TestWebKitAPI/Tests/WTF/SynchronizedFixedQueue.cpp#L126

How to fix this bug?

Comment 5 Fujii Hironori 2023-04-26 23:53:08 PDT

This dead lock can be reliably reproducible by increasing sleep time of enqueueLower.

diff --git a/Tools/TestWebKitAPI/Tests/WTF/SynchronizedFixedQueue.cpp b/Tools/TestWebKitAPI/Tests/WTF/SynchronizedFixedQueue.cpp
index f7a48db75f63..7cebd9fecaff 100644
--- a/Tools/TestWebKitAPI/Tests/WTF/SynchronizedFixedQueue.cpp
+++ b/Tools/TestWebKitAPI/Tests/WTF/SynchronizedFixedQueue.cpp
@@ -191,7 +191,7 @@ TEST(WTF_SynchronizedFixedQueue, ProduceOnly)
         converter.enqueueLower(item);
         ++count;
         
-        std::this_thread::sleep_for(std::chrono::milliseconds(1));
+        std::this_thread::sleep_for(std::chrono::milliseconds(100));
     }
 
     converter.stop();


192736@main (bug#176056) was the previous attempt to fix the dead lock.

Comment 6 Fujii Hironori 2023-04-27 00:13:16 PDT

Pull request: https://github.com/WebKit/WebKit/pull/13227

Comment 7 EWS 2023-05-03 23:22:53 PDT

Committed 263664@main (5b9202cb4147): <https://commits.webkit.org/263664@main>

Reviewed commits have been landed. Closing PR #13227 and removing active labels.