210955 2020-04-24 00:04:48 -0700 [Win] Deadlock in WTF::Thread::didExit() while WebKitNetworkProcess.exe is exiting 2021-03-03 13:26:39 -0800 1 1 1 Unclassified WebKit Web Template Framework WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=204192 https://bugs.webkit.org/show_bug.cgi?id=222682 P2 Normal --- 1 fujii fujii achristensen benjamin bfulgham cdumez cmarcelo don.olmstead ews-watchlist pvollan ross.kirsling webkit-bug-importer ysuzuki oldest_to_newest 1645107 0 fujii 2020-04-24 00:04:48 -0700 [WinCairo] Dead lock in WTF::Thread::didExit() while WebKitNetworkProcess.exe is exiting By running run-webkit-tests, sometimes WebKitNetworkProcess.exe process doesn't exit. > python ./Tools/Scripts/run-webkit-tests --wincairo --debug --no-new-test-results --no-retry-failures --no-show-results -f fast There is only the main thread, and here is its callstack. Callstack: > ntdll.dll!NtWaitForAlertByThreadId() Unknown > ntdll.dll!RtlSleepConditionVariableSRW() Unknown > KernelBase.dll!SleepConditionVariableSRW() Unknown > WTF.dll!WTF::ThreadCondition::timedWait(WTF::Mutex & mutex={...}, WTF::WallTime absoluteTime={...}) Line 383 C++ > WTF.dll!WTF::ParkingLot::parkConditionallyImpl(const void * address=0x00007ffc1ec325ac, const WTF::ScopedLambda<bool __cdecl(void)> & validation={...}, const WTF::ScopedLambda<void __cdecl(void)> & beforeSleep={...}, const WTF::TimeWithDynamicClockType & timeout={...}) Line 604 C++ > WTF.dll!WTF::ParkingLot::parkConditionally<bool <lambda>(void),void <lambda>(void)>(const void * address=0x00007ffc1ec325ac, const WTF::ParkingLot::compareAndPark::__l2::bool <lambda>(void) & validation=bool <lambda>(void){...}, const WTF::ParkingLot::compareAndPark::__l2::void <lambda>(void) & beforeSleep=void <lambda>(void){...}, const WTF::TimeWithDynamicClockType & timeout={...}) Line 82 C++ > WTF.dll!WTF::ParkingLot::compareAndPark<unsigned char,unsigned char>(const WTF::Atomic<unsigned char> * address=0x00007ffc1ec325ac, unsigned char expected='\x3') Line 94 C++ > WTF.dll!WTF::LockAlgorithm<unsigned char,1,2,WTF::EmptyLockHooks<unsigned char>>::lockSlow(WTF::Atomic<unsigned char> & lock={...}) Line 83 C++ > WTF.dll!WTF::Lock::lockSlow() Line 41 C++ > WTF.dll!WTF::Lock::lock() Line 60 C++ > WTF.dll!WTF::Locker<WTF::Lock>::lock() Line 114 C++ > WTF.dll!WTF::Locker<WTF::Lock>::Locker<WTF::Lock>(WTF::Lock & lockable={...}) Line 55 C++ > WTF.dll!WTF::Thread::didExit() Line 226 C++ > WTF.dll!WTF::Thread::ThreadHolder::~ThreadHolder() Line 274 C++ > WTF.dll!WTF::`dynamic atexit destructor for 's_threadHolder''() C++ > WTF.dll!__dyn_tls_dtor(void * __formal=0x000000007ffe0385, const unsigned long dwReason, void * __formal) Line 119 C++ > ntdll.dll!LdrpCallInitRoutine() Unknown > ntdll.dll!LdrpCallTlsInitializers() Unknown > ntdll.dll!LdrShutdownProcess() Unknown > ntdll.dll!RtlExitUserProcess() Unknown > kernel32.dll!ExitProcessImplementation() Unknown > ucrtbase.dll!exit_or_terminate_process() Unknown > ucrtbase.dll!common_exit() Unknown > WebKitNetworkProcess.exe!__scrt_common_main_seh() Line 295 C++ > kernel32.dll!BaseThreadInitThunk() Unknown > ntdll.dll!RtlUserThreadStart() Unknown I don't know when this issue started to happen, but not so long time ago. I fixed very similar issue in 5 months ago. Bug 204192 – [WinCairo] dead lock in Thread::destructTLS 1645849 1 397633 fujii 2020-04-26 16:27:22 -0700 Created attachment 397633 Patch to do quick_exit 1645850 2 fujii 2020-04-26 16:28:13 -0700 Using std::quick_exit doesn't solve the issue. The callstack looks almost same. > ntdll.dll!NtWaitForAlertByThreadId() Unknown > ntdll.dll!RtlSleepConditionVariableSRW() Unknown > KernelBase.dll!SleepConditionVariableSRW() Unknown > WTF.dll!WTF::ThreadCondition::timedWait(WTF::Mutex & mutex={...}, WTF::WallTime absoluteTime={...}) Line 383 C++ > WTF.dll!WTF::ParkingLot::parkConditionallyImpl(const void * address=0x00007ffc230755ac, const WTF::ScopedLambda<bool __cdecl(void)> & validation={...}, const WTF::ScopedLambda<void __cdecl(void)> & beforeSleep={...}, const WTF::TimeWithDynamicClockType & timeout={...}) Line 604 C++ > WTF.dll!WTF::ParkingLot::parkConditionally<bool <lambda>(void),void <lambda>(void)>(const void * address=0x00007ffc230755ac, const WTF::ParkingLot::compareAndPark::__l2::bool <lambda>(void) & validation=bool <lambda>(void){...}, const WTF::ParkingLot::compareAndPark::__l2::void <lambda>(void) & beforeSleep=void <lambda>(void){...}, const WTF::TimeWithDynamicClockType & timeout={...}) Line 82 C++ > WTF.dll!WTF::ParkingLot::compareAndPark<unsigned char,unsigned char>(const WTF::Atomic<unsigned char> * address=0x00007ffc230755ac, unsigned char expected='\x3') Line 94 C++ > WTF.dll!WTF::LockAlgorithm<unsigned char,1,2,WTF::EmptyLockHooks<unsigned char>>::lockSlow(WTF::Atomic<unsigned char> & lock={...}) Line 83 C++ > WTF.dll!WTF::Lock::lockSlow() Line 41 C++ > WTF.dll!WTF::Lock::lock() Line 60 C++ > WTF.dll!WTF::Locker<WTF::Lock>::lock() Line 114 C++ > WTF.dll!WTF::Locker<WTF::Lock>::Locker<WTF::Lock>(WTF::Lock & lockable={...}) Line 55 C++ > WTF.dll!WTF::holdLock<WTF::Lock>(WTF::Lock & lock={...}) Line 127 C++ > WTF.dll!WTF::Thread::didExit() Line 231 C++ > WTF.dll!WTF::Thread::ThreadHolder::~ThreadHolder() Line 274 C++ > WTF.dll!WTF::`dynamic atexit destructor for 's_threadHolder''() C++ > WTF.dll!__dyn_tls_dtor(void * __formal=0x000000007ffe0385, const unsigned long dwReason, void * __formal) Line 119 C++ > ntdll.dll!LdrpCallInitRoutine() Unknown > ntdll.dll!LdrpCallTlsInitializers() Unknown > ntdll.dll!LdrShutdownProcess() Unknown > ntdll.dll!RtlExitUserProcess() Unknown > kernel32.dll!ExitProcessImplementation() Unknown > ucrtbase.dll!exit_or_terminate_process() Unknown > ucrtbase.dll!common_exit() Unknown > WebKitNetworkProcess.exe!main(int argc=7, char * * argv=0x000001f2ec737650) Line 36 C++ > [Inline Frame] WebKitNetworkProcess.exe!invoke_main() Line 78 C++ > WebKitNetworkProcess.exe!__scrt_common_main_seh() Line 288 C++ > kernel32.dll!BaseThreadInitThunk() Unknown > ntdll.dll!RtlUserThreadStart() Unknown 1645881 3 397643 fujii 2020-04-26 22:01:26 -0700 Created attachment 397643 Patch to TerminateProcess Using TerminateProcess solves the issue. 1645910 4 397651 fujii 2020-04-27 00:25:04 -0700 Created attachment 397651 a test program to exit while other thread is locking a Lock I created a test program that the main thread exits while other thread is locking a Lock. This program is based on Bug 204192 Comment 26. > TestWTF.exe --gtest_filter=WTF_Lock.Test1 The deadlock happened with the following callstack. > ntdll.dll!NtWaitForAlertByThreadId() Unknown > ntdll.dll!RtlSleepConditionVariableSRW() Unknown > KernelBase.dll!SleepConditionVariableSRW() Unknown > WTF.dll!WTF::ThreadCondition::timedWait(WTF::Mutex & mutex={...}, WTF::WallTime absoluteTime={...}) Line 383 C++ > WTF.dll!WTF::ParkingLot::parkConditionallyImpl(const void * address=0x00007ffc1d0d1c28, const WTF::ScopedLambda<bool __cdecl(void)> & validation={...}, const WTF::ScopedLambda<void __cdecl(void)> & beforeSleep={...}, const WTF::TimeWithDynamicClockType & timeout={...}) Line 604 C++ > WTF.dll!WTF::ParkingLot::parkConditionally<bool <lambda>(void),void <lambda>(void)>(const void * address=0x00007ffc1d0d1c28, const WTF::ParkingLot::compareAndPark::__l2::bool <lambda>(void) & validation=bool <lambda>(void){...}, const WTF::ParkingLot::compareAndPark::__l2::void <lambda>(void) & beforeSleep=void <lambda>(void){...}, const WTF::TimeWithDynamicClockType & timeout={...}) Line 82 C++ > WTF.dll!WTF::ParkingLot::compareAndPark<unsigned char,unsigned char>(const WTF::Atomic<unsigned char> * address=0x00007ffc1d0d1c28, unsigned char expected='\x3') Line 94 C++ > WTF.dll!WTF::LockAlgorithm<unsigned char,1,2,WTF::EmptyLockHooks<unsigned char>>::lockSlow(WTF::Atomic<unsigned char> & lock={...}) Line 83 C++ > WTF.dll!WTF::Lock::lockSlow() Line 41 C++ > TestWTFLib.dll!WTF::Lock::lock() Line 60 C++ > TestWTFLib.dll!WTF::Locker<WTF::Lock>::lock() Line 114 C++ > TestWTFLib.dll!WTF::Locker<WTF::Lock>::Locker<WTF::Lock>(WTF::Lock & lockable={...}) Line 55 C++ > TestWTFLib.dll!WTF::holdLock<WTF::Lock>(WTF::Lock & lock={...}) Line 127 C++ > TestWTFLib.dll!`TestWebKitAPI::WTF_Lock_Test1_Test::TestBody'::`2'::ThreadExitCallback::~ThreadExitCallback() Line 206 C++ > TestWTFLib.dll!`TestWebKitAPI::WTF_Lock_Test1_Test::TestBody'::`2'::`dynamic atexit destructor for 'callback''() C++ > TestWTFLib.dll!__dyn_tls_dtor(void * __formal=0x000000007ffe0385, const unsigned long dwReason, void * __formal) Line 119 C++ > ntdll.dll!LdrpCallInitRoutine() Unknown > ntdll.dll!LdrpCallTlsInitializers() Unknown > ntdll.dll!LdrShutdownProcess() Unknown > ntdll.dll!RtlExitUserProcess() Unknown > kernel32.dll!ExitProcessImplementation() Unknown > ucrtbase.dll!exit_or_terminate_process() Unknown > ucrtbase.dll!common_exit() Unknown > TestWTF.exe!__scrt_common_main_seh() Line 295 C++ > kernel32.dll!BaseThreadInitThunk() Unknown > ntdll.dll!RtlUserThreadStart() Unknown 1646397 5 397810 fujii 2020-04-27 23:17:24 -0700 Created attachment 397810 Patch 1647121 6 397810 fujii 2020-04-29 13:06:53 -0700 Comment on attachment 397810 Patch Clearing flags on attachment: 397810 Committed r260911: <https://trac.webkit.org/changeset/260911> 1647122 7 fujii 2020-04-29 13:06:57 -0700 All reviewed patches have been landed. Closing bug. 1647129 8 ysuzuki 2020-04-29 13:19:16 -0700 Nice! 397633 2020-04-26 16:27:22 -0700 2020-04-27 23:17:10 -0700 Patch to do quick_exit quick-exit.diff text/plain 471 fujii ZGlmZiAtLWdpdCBhL1NvdXJjZS9XZWJLaXQvTmV0d29ya1Byb2Nlc3MvRW50cnlQb2ludC93aW4v TmV0d29ya1Byb2Nlc3NNYWluLmNwcCBiL1NvdXJjZS9XZWJLaXQvTmV0d29ya1Byb2Nlc3MvRW50 cnlQb2ludC93aW4vTmV0d29ya1Byb2Nlc3NNYWluLmNwcAppbmRleCA5ZmJmMzUwZmMwNS4uODk3 ZDNlN2ViNjUgMTAwNjQ0Ci0tLSBhL1NvdXJjZS9XZWJLaXQvTmV0d29ya1Byb2Nlc3MvRW50cnlQ b2ludC93aW4vTmV0d29ya1Byb2Nlc3NNYWluLmNwcAorKysgYi9Tb3VyY2UvV2ViS2l0L05ldHdv cmtQcm9jZXNzL0VudHJ5UG9pbnQvd2luL05ldHdvcmtQcm9jZXNzTWFpbi5jcHAKQEAgLTMyLDQg KzMyLDUgQEAKIGludCBtYWluKGludCBhcmdjLCBjaGFyKiogYXJndikKIHsKICAgICBXZWJLaXQ6 Ok5ldHdvcmtQcm9jZXNzTWFpbihhcmdjLCBhcmd2KTsKKyAgICBzdGQ6OnF1aWNrX2V4aXQoRVhJ VF9TVUNDRVNTKTsKIH0K 397643 2020-04-26 22:01:26 -0700 2020-04-27 23:17:12 -0700 Patch to TerminateProcess TerminateProcess.diff text/plain 559 fujii ZGlmZiAtLWdpdCBhL1NvdXJjZS9XZWJLaXQvTmV0d29ya1Byb2Nlc3MvRW50cnlQb2ludC93aW4v TmV0d29ya1Byb2Nlc3NNYWluLmNwcCBiL1NvdXJjZS9XZWJLaXQvTmV0d29ya1Byb2Nlc3MvRW50 cnlQb2ludC93aW4vTmV0d29ya1Byb2Nlc3NNYWluLmNwcAppbmRleCA5ZmJmMzUwZmMwNS4uNGNl OTgzYmQzMGQgMTAwNjQ0Ci0tLSBhL1NvdXJjZS9XZWJLaXQvTmV0d29ya1Byb2Nlc3MvRW50cnlQ b2ludC93aW4vTmV0d29ya1Byb2Nlc3NNYWluLmNwcAorKysgYi9Tb3VyY2UvV2ViS2l0L05ldHdv cmtQcm9jZXNzL0VudHJ5UG9pbnQvd2luL05ldHdvcmtQcm9jZXNzTWFpbi5jcHAKQEAgLTMyLDQg KzMyLDYgQEAKIGludCBtYWluKGludCBhcmdjLCBjaGFyKiogYXJndikKIHsKICAgICBXZWJLaXQ6 Ok5ldHdvcmtQcm9jZXNzTWFpbihhcmdjLCBhcmd2KTsKKyAgICAvLyBEZXN0cnVjdGluZyB0aHJl YWRfbG9jYWwgb2JqZWN0cyBvZiB0aGUgbWFpbiB0aHJlYWQgY2F1c2VkIGRlYWRsb2NrLgorICAg IFRlcm1pbmF0ZVByb2Nlc3MoR2V0Q3VycmVudFByb2Nlc3MoKSwgMCk7CiB9Cg== 397651 2020-04-27 00:25:04 -0700 2020-04-27 23:17:14 -0700 a test program to exit while other thread is locking a Lock test.diff text/plain 856 fujii ZGlmZiAtLWdpdCBhL1Rvb2xzL1Rlc3RXZWJLaXRBUEkvVGVzdHMvV1RGL0xvY2suY3BwIGIvVG9v bHMvVGVzdFdlYktpdEFQSS9UZXN0cy9XVEYvTG9jay5jcHAKaW5kZXggYThmMDNiMDdkNjguLmNj YjY1MTI1OWEyIDEwMDY0NAotLS0gYS9Ub29scy9UZXN0V2ViS2l0QVBJL1Rlc3RzL1dURi9Mb2Nr LmNwcAorKysgYi9Ub29scy9UZXN0V2ViS2l0QVBJL1Rlc3RzL1dURi9Mb2NrLmNwcApAQCAtMTg1 LDQgKzE4NSwyNiBAQCBURVNUKFdURl9Mb2NrLCBTZWN0aW9uQWRkcmVzc0NvbGxpc2lvbikKICAg ICBydW5Mb2NrVGVzdDxMb2NrPig0LCAyLCAxMDAwMCwgMjAwMCk7CiB9CiAKK1RFU1QoV1RGX0xv Y2ssIFRlc3QxKQoreworICAgIHN0YXRpYyBMb2NrIGxvY2s7CisgICAgVGhyZWFkOjpjcmVhdGUo CisgICAgICAgICJMb2NrIHRlc3QgdGhyZWFkIiwKKyAgICAgICAgICAgIFtdICgpIHsKKyAgICAg ICAgICAgICAgICBhdXRvIGxvY2tlciA9IGhvbGRMb2NrKGxvY2spOworICAgICAgICAgICAgICAg IFNsZWVwKDEwICogMTAwMCk7CisgICAgICAgICAgICB9KTsKKworICAgIHN0YXRpYyB0aHJlYWRf bG9jYWwgY2xhc3MgVGhyZWFkRXhpdENhbGxiYWNrIHsKKyAgICBwdWJsaWM6CisgICAgICAgIFRo cmVhZEV4aXRDYWxsYmFjayhMb2NrJiBsb2NrKSA6IG1fbG9jayhsb2NrKSB7IH0KKyAgICAgICAg TG9jayYgbV9sb2NrOworCisgICAgICAgIH5UaHJlYWRFeGl0Q2FsbGJhY2soKQorICAgICAgICB7 CisgICAgICAgICAgICBhdXRvIGxvY2tlciA9IGhvbGRMb2NrKG1fbG9jayk7CisgICAgICAgIH0K KyAgICB9IGNhbGxiYWNrKGxvY2spOworfQorCiB9IC8vIG5hbWVzcGFjZSBUZXN0V2ViS2l0QVBJ Cg== 397810 2020-04-27 23:17:24 -0700 2020-04-29 13:06:53 -0700 Patch bug-210955-20200428151722.patch text/plain 1827 fujii U3VidmVyc2lvbiBSZXZpc2lvbjogMjYwNzI5CmRpZmYgLS1naXQgYS9Tb3VyY2UvV1RGL0NoYW5n ZUxvZyBiL1NvdXJjZS9XVEYvQ2hhbmdlTG9nCmluZGV4IGRjNDEwNzIyZjNlMmQwMWRhZWRjZjk3 YmIxOGU2MTE1Mjk2NjdlZGYuLjc0NWQ1OWZlOTQzZDM0ZWU4ZmE0YWI4OWQyYTQzNzY1MjAyZjRi ODIgMTAwNjQ0Ci0tLSBhL1NvdXJjZS9XVEYvQ2hhbmdlTG9nCisrKyBiL1NvdXJjZS9XVEYvQ2hh bmdlTG9nCkBAIC0xLDMgKzEsMTggQEAKKzIwMjAtMDQtMjcgIEZ1amlpIEhpcm9ub3JpICA8SGly b25vcmkuRnVqaWlAc29ueS5jb20+CisKKyAgICAgICAgW1dpbl0gRGVhZGxvY2sgaW4gV1RGOjpU aHJlYWQ6OmRpZEV4aXQoKSB3aGlsZSBXZWJLaXROZXR3b3JrUHJvY2Vzcy5leGUgaXMgZXhpdGlu ZworICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9MjEwOTU1 CisKKyAgICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISkuCisKKyAgICAgICAgVGhlIHRo cmVhZF9sb2NhbCBvYmplY3Qgb2YgdGhlIG1haW4gdGhyZWFkIGlzIGRlc3RydWN0ZWQgYWZ0ZXIK KyAgICAgICAgV2luZG93cyB0ZXJtaW5hdGVzIG90aGVyIHRocmVhZHMuIElmIHRoZSB0ZXJtaW5h dGVkIHRocmVhZCB3YXMKKyAgICAgICAgaG9sZGluZyBhIG11dGV4LCB0cnlpbmcgdG8gbG9jayB0 aGUgbXV0ZXggY2F1c2VzIGRlYWRsb2NrLgorCisgICAgICAgICogd3RmL3dpbi9UaHJlYWRpbmdX aW4uY3BwOgorICAgICAgICAoV1RGOjpUaHJlYWQ6OlRocmVhZEhvbGRlcjo6flRocmVhZEhvbGRl cik6IERvIG5vdGhpbmcgaWYgdGhlCisgICAgICAgIHRocmVhZCBpcyB0aGUgbWFpbiB0aHJlYWQu CisKIDIwMjAtMDQtMjYgIEpvb25naHVuIFBhcmsgIDxqaDcxOC5wYXJrQHNhbXN1bmcuY29tPgog CiAgICAgICAgIFtXVEZdIFdvcmthcm91bmQgZ2NjIGJ1ZyBmb3IgdW5zaWduZWQgYml0ZmllbGQg cmVsYXRlZCB1c3VhbCBhcml0aG1ldGljIGNvbnZlcnNpb25zCmRpZmYgLS1naXQgYS9Tb3VyY2Uv V1RGL3d0Zi93aW4vVGhyZWFkaW5nV2luLmNwcCBiL1NvdXJjZS9XVEYvd3RmL3dpbi9UaHJlYWRp bmdXaW4uY3BwCmluZGV4IDEwMmE2NGU2YjRiMGVjNDQ2ZTIyOTU0NDQwMjQwYjQyMWQ2YmNlOTcu LmQ2MzJmZWQ4MTExNjkzYjQ3MzUxZmVjMzQ5Yzk0ZmM4ZTRjNTVkYTggMTAwNjQ0Ci0tLSBhL1Nv dXJjZS9XVEYvd3RmL3dpbi9UaHJlYWRpbmdXaW4uY3BwCisrKyBiL1NvdXJjZS9XVEYvd3RmL3dp bi9UaHJlYWRpbmdXaW4uY3BwCkBAIC0yNjcsNiArMjY3LDEyIEBAIHZvaWQgVGhyZWFkOjplc3Rh Ymxpc2hQbGF0Zm9ybVNwZWNpZmljSGFuZGxlKEhBTkRMRSBoYW5kbGUsIFRocmVhZElkZW50aWZp ZXIgdGhyCiBzdHJ1Y3QgVGhyZWFkOjpUaHJlYWRIb2xkZXIgewogICAgIH5UaHJlYWRIb2xkZXIo KQogICAgIHsKKyAgICAgICAgLy8gVGhlIHRocmVhZF9sb2NhbCBvYmplY3Qgb2YgdGhlIG1haW4g dGhyZWFkIGlzIGRlc3RydWN0ZWQKKyAgICAgICAgLy8gYWZ0ZXIgV2luZG93cyB0ZXJtaW5hdGVz IG90aGVyIHRocmVhZHMuIElmIHRoZSB0ZXJtaW5hdGVkCisgICAgICAgIC8vIHRocmVhZCB3YXMg aG9sZGluZyBhIG11dGV4LCB0cnlpbmcgdG8gbG9jayB0aGUgbXV0ZXggY2F1c2VzCisgICAgICAg IC8vIGRlYWRsb2NrLgorICAgICAgICBpZiAoaXNNYWluVGhyZWFkKCkpCisgICAgICAgICAgICBy ZXR1cm47CiAgICAgICAgIGlmICh0aHJlYWQpIHsKICAgICAgICAgICAgIHRocmVhZC0+c3BlY2lm aWNTdG9yYWdlKCkuZGVzdHJveVNsb3RzKCk7CiAgICAgICAgICAgICB0aHJlYWQtPmRpZEV4aXQo KTsK