175852 2017-08-22 14:22:02 -0700 Race condition in StartWebThread causing crash 2017-08-23 10:42:10 -0700 1 1 1 Unclassified WebKit WebKit API Other iPhone / iPad iOS 10.3 RESOLVED FIXED InRadar P2 Major --- 1 nham ysuzuki fpizlo ggaren kling mark.lam webkit-bug-importer ysuzuki oldest_to_newest 1341266 0 nham 2017-08-22 14:22:02 -0700 We have been seeing an increasing number of crashes in the Facebook app due to a race condition in StartWebThread. The crash involves a race between the main thread and the web thread. The main thread crashes in this RELEASE_ASSERT in ThreadIdentifierData: https://opensource.apple.com/source/WTF/WTF-7603.1.30.1.33/wtf/ThreadIdentifierDataPthreads.cpp.auto.html (line 78) The main thread's stack is: Thread #0 Crashed: 0 JavaScriptCore 0x1882aabf8 WTF::ThreadIdentifierData::initialize(unsigned int) + 72 1 JavaScriptCore 0x1882a9188 WTF::currentThread() + 48 2 JavaScriptCore 0x1882a9188 WTF::currentThread() + 48 3 JavaScriptCore 0x1882aac04 WTF::initializeApplicationUIThreadIdentifier() + 8 4 WebCore 0x188f76bec StartWebThread() + 504 5 libsystem_pthread.dylib 0x18348f418 __pthread_once_handler + 76 6 libsystem_platform.dylib 0x183484a44 _os_once + 48 7 libsystem_pthread.dylib 0x18348c2fc pthread_once + 64 8 WebKitLegacy 0x18a15ba54 +[WebView(WebPrivate) enableWebThread] + 268 9 WebKitLegacy 0x18a15b68c WebKitInitialize + 68 10 UIKit 0x18a781dd0 ___UIApplicationLoadWebKit_block_invoke + 212 11 libdispatch.dylib 0x18328299c _dispatch_client_callout + 12 12 libdispatch.dylib 0x1832836c8 dispatch_once_f + 52 13 libdispatch.dylib 0x18328299c _dispatch_client_callout + 12 14 libdispatch.dylib 0x1832937a4 _dispatch_barrier_sync_f_slow_invoke + 300 15 libdispatch.dylib 0x18328299c _dispatch_client_callout + 12 16 libdispatch.dylib 0x1832875e4 _dispatch_main_queue_callback_4CF + 992 17 CoreFoundation 0x1843790c4 __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 8 18 CoreFoundation 0x184376ce0 __CFRunLoopRun + 1568 19 CoreFoundation 0x1842a6da0 CFRunLoopRunSpecific + 420 20 GraphicsServices 0x185d11070 GSEventRunModal + 96 21 UIKit 0x18a561c98 UIApplicationMain + 204 22 Facebook 0x1000b5738 FBGenericMain (FBGenericMain.m:224) 23 Facebook 0x1000b5240 main (main.m:150) 24 libdyld.dylib 0x1832b5598 start + 0 This RELEASE_ASSERT fires if the pthread TSD key hasn't yet been created. The TSD key is created by WTF::initializeThreading, which is called by JSC::initializeThreading. So basically, the crash occurs because the MT called WTF::ThreadIdentifierData::initialize before another thread finished calling WTF::initializeThreading. The thread that's responsible for calling WTF::initializeThreading is the WebThread (which is forked off by StartWebThread). The WebThread has this call stack in the crash: Thread #63: 0 libsystem_kernel.dylib 0x1833c4f60 __psynch_rw_wrlock + 8 1 libsystem_pthread.dylib 0x18348b5c8 _pthread_rwlock_lock + 464 2 libobjc.A.dylib 0x182e35764 rwlock_tt<false>::write() + 100 3 libobjc.A.dylib 0x182e2ebc4 _class_getNonMetaClass + 36 4 libobjc.A.dylib 0x182e354ac lookUpImpOrForward + 220 5 libobjc.A.dylib 0x182e40474 _objc_msgSend_uncached + 52 6 libobjc.A.dylib 0x182e2d414 CALLING_SOME_+initialize_METHOD + 20 7 libobjc.A.dylib 0x182e2d680 _class_initialize + 608 8 libobjc.A.dylib 0x182e354b0 lookUpImpOrForward + 224 9 libobjc.A.dylib 0x182e40474 _objc_msgSend_uncached + 52 10 JavaScriptCore 0x1882a9958 WTF::initializeMainThreadPlatform() + 24 11 libsystem_pthread.dylib 0x18348f418 __pthread_once_handler + 76 12 libsystem_platform.dylib 0x183484a44 _os_once + 48 13 libsystem_pthread.dylib 0x18348c2fc pthread_once + 64 14 WebCore 0x188f78b94 RunWebThread(void*) + 32 15 libsystem_pthread.dylib 0x18348d688 _pthread_body + 236 16 libsystem_pthread.dylib 0x18348d598 _pthread_start + 280 17 libsystem_pthread.dylib 0x18348acb0 thread_start + 0 If you look at the code, the WebThread is stuck two lines *before* the call to JSC::initializeThreading (which is what calls WTF::initializeThreading)--it's executing WTF::initializeMainThreadPlatform: https://opensource.apple.com/source/WebCore/WebCore-7603.1.30.1.33/platform/ios/wak/WebCoreThread.mm.auto.html (line 653) In theory, there is some synchronization in the existing code that should prevent this from happening. In pseudo-code, the synchronization code looks like this: RunWebThread: (running on the web thread) JSC::initializeThreading() // other init... pthread_mutex_lock(&lock) pthread_cond_signal(&cond, &lock) pthread_mutex_unlock(&lock) StartWebThread: (running on the main thread) pthread_lock(&lock) pthread_cond_wait(&cond, &lock) pthread_unlock(&lock) WTF::initializeApplicationUIThreadIdentifier() // calls code which eventually asserts However, this synchronization code is buggy because pthread_cond_wait is allowed to spuriously wake up, and pthread_cond_wait is only called once. It is very easy to demonstrate that spurious wakeups exist with pthread condition variables, e.g. here is a simple test program: https://gist.github.com/bnham/f2ca3430a7c4f6f714a918b62702125e The fix is to add a boolean flag variable and to check whether that boolean variable is set while calling pthread_cond_wait in a loop, i.e. textbook usage of Mesa condition variables. A possible mitigation for affected apps is to call some JSC API that eventually calls JSC::initializeThreading early on in app initialization. 1341270 1 nham 2017-08-22 14:30:30 -0700 Also filed a radar: <rdar://34021573> 1341554 2 ap 2017-08-23 09:09:57 -0700 rdar://problem/34021570 1341555 3 ysuzuki 2017-08-23 09:11:56 -0700 Good catch! 1341566 4 ysuzuki 2017-08-23 09:37:58 -0700 Recently[1], I changed this code to call WTF::initailizeThreading() in StartWebThread before spawning WebThread :) This is OK because now WTF::initializeThreading does not have the limitation that requires WTF::initializeThreading() should be called in the main thread. So in the trunk, I think this issue is fixed. What do you think of Mark? [1]: http://trac.webkit.org/r219647 1341569 5 ysuzuki 2017-08-23 09:42:27 -0700 (In reply to Ben Nham from comment #0) > We have been seeing an increasing number of crashes in the Facebook app due > to a race condition in StartWebThread. The crash involves a race between the > main thread and the web thread. > > The main thread crashes in this RELEASE_ASSERT in ThreadIdentifierData: > > > https://opensource.apple.com/source/WTF/WTF-7603.1.30.1.33/wtf/ > ThreadIdentifierDataPthreads.cpp.auto.html (line 78) > > The main thread's stack is: > > Thread #0 Crashed: > 0 JavaScriptCore 0x1882aabf8 > WTF::ThreadIdentifierData::initialize(unsigned int) + 72 > 1 JavaScriptCore 0x1882a9188 WTF::currentThread() + 48 > 2 JavaScriptCore 0x1882a9188 WTF::currentThread() + 48 > 3 JavaScriptCore 0x1882aac04 > WTF::initializeApplicationUIThreadIdentifier() + 8 > 4 WebCore 0x188f76bec StartWebThread() + 504 > 5 libsystem_pthread.dylib 0x18348f418 __pthread_once_handler + 76 > 6 libsystem_platform.dylib 0x183484a44 _os_once + 48 > 7 libsystem_pthread.dylib 0x18348c2fc pthread_once + 64 > 8 WebKitLegacy 0x18a15ba54 +[WebView(WebPrivate) enableWebThread] + 268 > 9 WebKitLegacy 0x18a15b68c WebKitInitialize + 68 > 10 UIKit 0x18a781dd0 ___UIApplicationLoadWebKit_block_invoke + 212 > 11 libdispatch.dylib 0x18328299c _dispatch_client_callout + 12 > 12 libdispatch.dylib 0x1832836c8 dispatch_once_f + 52 > 13 libdispatch.dylib 0x18328299c _dispatch_client_callout + 12 > 14 libdispatch.dylib 0x1832937a4 _dispatch_barrier_sync_f_slow_invoke + 300 > 15 libdispatch.dylib 0x18328299c _dispatch_client_callout + 12 > 16 libdispatch.dylib 0x1832875e4 _dispatch_main_queue_callback_4CF + 992 > 17 CoreFoundation 0x1843790c4 > __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 8 > 18 CoreFoundation 0x184376ce0 __CFRunLoopRun + 1568 > 19 CoreFoundation 0x1842a6da0 CFRunLoopRunSpecific + 420 > 20 GraphicsServices 0x185d11070 GSEventRunModal + 96 > 21 UIKit 0x18a561c98 UIApplicationMain + 204 > 22 Facebook 0x1000b5738 FBGenericMain (FBGenericMain.m:224) > 23 Facebook 0x1000b5240 main (main.m:150) > 24 libdyld.dylib 0x1832b5598 start + 0 > > This RELEASE_ASSERT fires if the pthread TSD key hasn't yet been created. > The TSD key is created by WTF::initializeThreading, which is called by > JSC::initializeThreading. So basically, the crash occurs because the MT > called WTF::ThreadIdentifierData::initialize before another thread finished > calling WTF::initializeThreading. > > The thread that's responsible for calling WTF::initializeThreading is the > WebThread (which is forked off by StartWebThread). The WebThread has this > call stack in the crash: > > Thread #63: > 0 libsystem_kernel.dylib 0x1833c4f60 __psynch_rw_wrlock + 8 > 1 libsystem_pthread.dylib 0x18348b5c8 _pthread_rwlock_lock + 464 > 2 libobjc.A.dylib 0x182e35764 rwlock_tt<false>::write() + 100 > 3 libobjc.A.dylib 0x182e2ebc4 _class_getNonMetaClass + 36 > 4 libobjc.A.dylib 0x182e354ac lookUpImpOrForward + 220 > 5 libobjc.A.dylib 0x182e40474 _objc_msgSend_uncached + 52 > 6 libobjc.A.dylib 0x182e2d414 CALLING_SOME_+initialize_METHOD + 20 > 7 libobjc.A.dylib 0x182e2d680 _class_initialize + 608 > 8 libobjc.A.dylib 0x182e354b0 lookUpImpOrForward + 224 > 9 libobjc.A.dylib 0x182e40474 _objc_msgSend_uncached + 52 > 10 JavaScriptCore 0x1882a9958 WTF::initializeMainThreadPlatform() + 24 > 11 libsystem_pthread.dylib 0x18348f418 __pthread_once_handler + 76 > 12 libsystem_platform.dylib 0x183484a44 _os_once + 48 > 13 libsystem_pthread.dylib 0x18348c2fc pthread_once + 64 > 14 WebCore 0x188f78b94 RunWebThread(void*) + 32 > 15 libsystem_pthread.dylib 0x18348d688 _pthread_body + 236 > 16 libsystem_pthread.dylib 0x18348d598 _pthread_start + 280 > 17 libsystem_pthread.dylib 0x18348acb0 thread_start + 0 > > If you look at the code, the WebThread is stuck two lines *before* the call > to JSC::initializeThreading (which is what calls > WTF::initializeThreading)--it's executing WTF::initializeMainThreadPlatform: > > > https://opensource.apple.com/source/WebCore/WebCore-7603.1.30.1.33/platform/ > ios/wak/WebCoreThread.mm.auto.html (line 653) > > In theory, there is some synchronization in the existing code that should > prevent this from happening. In pseudo-code, the synchronization code looks > like this: > > RunWebThread: (running on the web thread) > JSC::initializeThreading() > // other init... > > pthread_mutex_lock(&lock) > pthread_cond_signal(&cond, &lock) > pthread_mutex_unlock(&lock) > > StartWebThread: (running on the main thread) > pthread_lock(&lock) > pthread_cond_wait(&cond, &lock) > pthread_unlock(&lock) > > WTF::initializeApplicationUIThreadIdentifier() // calls code which > eventually asserts > > However, this synchronization code is buggy because pthread_cond_wait is > allowed to spuriously wake up, and pthread_cond_wait is only called once. It > is very easy to demonstrate that spurious wakeups exist with pthread > condition variables, e.g. here is a simple test program: > https://gist.github.com/bnham/f2ca3430a7c4f6f714a918b62702125e > > The fix is to add a boolean flag variable and to check whether that boolean > variable is set while calling pthread_cond_wait in a loop, i.e. textbook > usage of Mesa condition variables. > > A possible mitigation for affected apps is to call some JSC API that > eventually calls JSC::initializeThreading early on in app initialization. BTW, I think your pointing is true that this pthread_cond_t use is broken due to the existence of the suspicious wakeup. We should fix this while the trunk do not show this problem. Our web threading's following code depends on that web thread is already initialized. 1341578 6 318880 ysuzuki 2017-08-23 09:57:40 -0700 Created attachment 318880 Patch 1341601 7 318880 mark.lam 2017-08-23 10:34:37 -0700 Comment on attachment 318880 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=318880&action=review r=me. > Source/WebCore/ChangeLog:9 > + by using pthread_cond_t. However, the main thread may be waken up due to the the existence of /waken/woken/. > Source/WebCore/ChangeLog:10 > + the suspicious wakeup of the pthread_cond_t. /suspicious/spurious/. > Source/WebCore/ChangeLog:14 > + does not have the suspicious wakup problem as described in Condition.h. /suspicious/spurious/. /wakup/wake up/. 1341605 8 318880 ysuzuki 2017-08-23 10:37:13 -0700 Comment on attachment 318880 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=318880&action=review Thank you! >> Source/WebCore/ChangeLog:9 >> + by using pthread_cond_t. However, the main thread may be waken up due to the the existence of > > /waken/woken/. Oops, thanks. Fixed. >> Source/WebCore/ChangeLog:10 >> + the suspicious wakeup of the pthread_cond_t. > > /suspicious/spurious/. Fixed. >> Source/WebCore/ChangeLog:14 >> + does not have the suspicious wakup problem as described in Condition.h. > > /suspicious/spurious/. > /wakup/wake up/. Fixed. 1341612 9 ysuzuki 2017-08-23 10:42:10 -0700 Committed r221083: <http://trac.webkit.org/changeset/221083> 318880 2017-08-23 09:57:40 -0700 2017-08-23 10:34:37 -0700 Patch bug-175852-20170824015739.patch text/plain 4045 ysuzuki U3VidmVyc2lvbiBSZXZpc2lvbjogMjIxMDc1CmRpZmYgLS1naXQgYS9Tb3VyY2UvV2ViQ29yZS9D aGFuZ2VMb2cgYi9Tb3VyY2UvV2ViQ29yZS9DaGFuZ2VMb2cKaW5kZXggZDNmYTRkNzM5N2Y5MWRi MTk3OGJkNGQxYWE0Zjc1NDBjMTAxYmMzOC4uYTM0NjEyZDUzYzM3YWYwZTYxM2VhNjlhMWY0MjA4 MWZiN2JiNTA4NyAxMDA2NDQKLS0tIGEvU291cmNlL1dlYkNvcmUvQ2hhbmdlTG9nCisrKyBiL1Nv dXJjZS9XZWJDb3JlL0NoYW5nZUxvZwpAQCAtMSwzICsxLDIyIEBACisyMDE3LTA4LTIzICBZdXN1 a2UgU3V6dWtpICA8dXRhdGFuZS50ZWFAZ21haWwuY29tPgorCisgICAgICAgIFJhY2UgY29uZGl0 aW9uIGluIFN0YXJ0V2ViVGhyZWFkIGNhdXNpbmcgY3Jhc2gKKyAgICAgICAgaHR0cHM6Ly9idWdz LndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTE3NTg1MgorCisgICAgICAgIFJldmlld2VkIGJ5 IE5PQk9EWSAoT09QUyEpLgorCisgICAgICAgIFdoZW4gc3RhcnRpbmcgd2ViIHRocmVhZCwgdGhl IG1haW4gdGhyZWFkIHdhaXRzIGZvciBjb21wbGV0aW9uIG9mIHRoZSB3ZWIgdGhyZWFkIGluaXRp YWxpemF0aW9uCisgICAgICAgIGJ5IHVzaW5nIHB0aHJlYWRfY29uZF90LiBIb3dldmVyLCB0aGUg bWFpbiB0aHJlYWQgbWF5IGJlIHdha2VuIHVwIGR1ZSB0byB0aGUgdGhlIGV4aXN0ZW5jZSBvZgor ICAgICAgICB0aGUgc3VzcGljaW91cyB3YWtldXAgb2YgdGhlIHB0aHJlYWRfY29uZF90LgorCisg ICAgICAgIEluc3RlYWQsIHdlIHNob3VsZCB1c2UgV1RGOjpMb2NrIGFuZCBXVEY6OkNvbmRpdGlv bi4gU2luY2Ugb3VyIFN0YXJ0V2ViVGhyZWFkIGFscmVhZHkgY2FsbHMKKyAgICAgICAgV1RGOjpp bml0aWFsaXplVGhyZWFkaW5nLCBpdCBpcyBzYWZlIHRvIHVzZSBXVEY6OkxvY2sgYW5kIFdURjo6 Q29uZGl0aW9uLiBBbmQgb3VyIFdURjo6Q29uZGl0aW9uCisgICAgICAgIGRvZXMgbm90IGhhdmUg dGhlIHN1c3BpY2lvdXMgd2FrdXAgcHJvYmxlbSBhcyBkZXNjcmliZWQgaW4gQ29uZGl0aW9uLmgu CisKKyAgICAgICAgKiBwbGF0Zm9ybS9pb3Mvd2FrL1dlYkNvcmVUaHJlYWQubW06CisgICAgICAg IChSdW5XZWJUaHJlYWQpOgorICAgICAgICAoU3RhcnRXZWJUaHJlYWQpOgorCiAyMDE3LTA4LTIz ICBDYXJsb3MgQWxiZXJ0byBMb3BleiBQZXJleiAgPGNsb3BlekBpZ2FsaWEuY29tPgogCiAgICAg ICAgIFtXUEVdIFNvbWUgZXZlbnQgaGFuZGxlcnMgbm90IHdvcmtpbmcuCmRpZmYgLS1naXQgYS9T b3VyY2UvV2ViQ29yZS9wbGF0Zm9ybS9pb3Mvd2FrL1dlYkNvcmVUaHJlYWQubW0gYi9Tb3VyY2Uv V2ViQ29yZS9wbGF0Zm9ybS9pb3Mvd2FrL1dlYkNvcmVUaHJlYWQubW0KaW5kZXggYzhmMDVhMWEw YWVkNjAyY2JkOGMwOTRlMGE5NzIwM2FmOWM2MDZiZi4uODBlNDU0MDg0YzgwZTA2YTIxOGJmN2Yw YjBlYWRjMjE2OTkxMzMzYyAxMDA2NDQKLS0tIGEvU291cmNlL1dlYkNvcmUvcGxhdGZvcm0vaW9z L3dhay9XZWJDb3JlVGhyZWFkLm1tCisrKyBiL1NvdXJjZS9XZWJDb3JlL3BsYXRmb3JtL2lvcy93 YWsvV2ViQ29yZVRocmVhZC5tbQpAQCAtMTI3LDggKzEyNyw4IEBAIHN0YXRpYyBCT09MIHNlbmRp bmdEZWxlZ2F0ZU1lc3NhZ2U7CiAKIHN0YXRpYyBDRlJ1bkxvb3BPYnNlcnZlclJlZiBtYWluUnVu TG9vcEF1dG9VbmxvY2tPYnNlcnZlcjsKIAotc3RhdGljIHB0aHJlYWRfbXV0ZXhfdCBzdGFydHVw TG9jayA9IFBUSFJFQURfTVVURVhfSU5JVElBTElaRVI7Ci1zdGF0aWMgcHRocmVhZF9jb25kX3Qg c3RhcnR1cENvbmRpdGlvbiA9IFBUSFJFQURfQ09ORF9JTklUSUFMSVpFUjsKK3N0YXRpYyBTdGF0 aWNMb2NrIHN0YXJ0dXBMb2NrOworc3RhdGljIFN0YXRpY0NvbmRpdGlvbiBzdGFydHVwQ29uZGl0 aW9uOwogCiBzdGF0aWMgV2ViVGhyZWFkQ29udGV4dCAqd2ViVGhyZWFkQ29udGV4dDsKIHN0YXRp YyBwdGhyZWFkX2tleV90IHRocmVhZENvbnRleHRLZXk7CkBAIC02ODEsMTQgKzY4MSwxMCBAQCB2 b2lkICpSdW5XZWJUaHJlYWQodm9pZCAqYXJnKQogICAgIFdlYlRocmVhZFJlbGVhc2VTb3VyY2Ug PSBDRlJ1bkxvb3BTb3VyY2VDcmVhdGUoTlVMTCwgLTEsICZSZWxlYXNlU291cmNlQ29udGV4dCk7 CiAgICAgQ0ZSdW5Mb29wQWRkU291cmNlKHdlYlRocmVhZFJ1bkxvb3AsIFdlYlRocmVhZFJlbGVh c2VTb3VyY2UsIGtDRlJ1bkxvb3BEZWZhdWx0TW9kZSk7CiAKLSAgICBpbnQgcmVzdWx0ID0gcHRo cmVhZF9tdXRleF9sb2NrKCZzdGFydHVwTG9jayk7Ci0gICAgQVNTRVJUX1dJVEhfTUVTU0FHRShy ZXN1bHQgPT0gMCwgInN0YXJ0dXAgbG9jayBmYWlsZWQgd2l0aCBjb2RlOiVkIiwgcmVzdWx0KTsK LQotICAgIHJlc3VsdCA9IHB0aHJlYWRfY29uZF9zaWduYWwoJnN0YXJ0dXBDb25kaXRpb24pOwot ICAgIEFTU0VSVF9XSVRIX01FU1NBR0UocmVzdWx0ID09IDAsICJzdGFydHVwIHNpZ25hbCBmYWls ZWQgd2l0aCBjb2RlOiVkIiwgcmVzdWx0KTsKLQotICAgIHJlc3VsdCA9IHB0aHJlYWRfbXV0ZXhf dW5sb2NrKCZzdGFydHVwTG9jayk7Ci0gICAgQVNTRVJUX1dJVEhfTUVTU0FHRShyZXN1bHQgPT0g MCwgInN0YXJ0dXAgdW5sb2NrIGZhaWxlZCB3aXRoIGNvZGU6JWQiLCByZXN1bHQpOworICAgIHsK KyAgICAgICAgTG9ja0hvbGRlciBsb2NrZXIoc3RhcnR1cExvY2spOworICAgICAgICBzdGFydHVw Q29uZGl0aW9uLm5vdGlmeU9uZSgpOworICAgIH0KIAogICAgIHdoaWxlICgxKQogICAgICAgICBD RlJ1bkxvb3BSdW5Jbk1vZGUoa0NGUnVuTG9vcERlZmF1bHRNb2RlLCBEaXN0YW50RnV0dXJlLCB0 cnVlKTsKQEAgLTc1OCwyMCArNzU0LDE3IEBAIHN0YXRpYyB2b2lkIFN0YXJ0V2ViVGhyZWFkKCkK ICAgICBwdGhyZWFkX2F0dHJfc2V0c2NoZWRwYXJhbSgmdGF0dHIsICZwYXJhbSk7CiAKICAgICAv LyBXYWl0IGZvciB0aGUgd2ViIHRocmVhZCB0byBzdGFydHVwIGNvbXBsZXRlbHkgYmVmb3JlIHdl IGNvbnRpbnVlLgotICAgIGludCByZXN1bHQgPSBwdGhyZWFkX211dGV4X2xvY2soJnN0YXJ0dXBM b2NrKTsKLSAgICBBU1NFUlRfV0lUSF9NRVNTQUdFKHJlc3VsdCA9PSAwLCAic3RhcnR1cCBsb2Nr IGZhaWxlZCB3aXRoIGNvZGU6JWQiLCByZXN1bHQpOworICAgIHsKKyAgICAgICAgTG9ja0hvbGRl ciBsb2NrZXIoc3RhcnR1cExvY2spOwogCi0gICAgLy8gUHJvcGFnYXRlIHRoZSBtYWluVGhyZWFk J3MgZmVudiB0byB3b3JrZXJzICYgdGhlIHdlYiB0aHJlYWQuCi0gICAgRmxvYXRpbmdQb2ludEVu dmlyb25tZW50OjpzaW5nbGV0b24oKS5zYXZlTWFpblRocmVhZEVudmlyb25tZW50KCk7CisgICAg ICAgIC8vIFByb3BhZ2F0ZSB0aGUgbWFpblRocmVhZCdzIGZlbnYgdG8gd29ya2VycyAmIHRoZSB3 ZWIgdGhyZWFkLgorICAgICAgICBGbG9hdGluZ1BvaW50RW52aXJvbm1lbnQ6OnNpbmdsZXRvbigp LnNhdmVNYWluVGhyZWFkRW52aXJvbm1lbnQoKTsKIAotICAgIHB0aHJlYWRfY3JlYXRlKCZ3ZWJU aHJlYWQsICZ0YXR0ciwgUnVuV2ViVGhyZWFkLCBOVUxMKTsKLSAgICBwdGhyZWFkX2F0dHJfZGVz dHJveSgmdGF0dHIpOworICAgICAgICBwdGhyZWFkX2NyZWF0ZSgmd2ViVGhyZWFkLCAmdGF0dHIs IFJ1bldlYlRocmVhZCwgTlVMTCk7CisgICAgICAgIHB0aHJlYWRfYXR0cl9kZXN0cm95KCZ0YXR0 cik7CiAKLSAgICByZXN1bHQgPSBwdGhyZWFkX2NvbmRfd2FpdCgmc3RhcnR1cENvbmRpdGlvbiwg JnN0YXJ0dXBMb2NrKTsKLSAgICBBU1NFUlRfV0lUSF9NRVNTQUdFKHJlc3VsdCA9PSAwLCAic3Rh cnR1cCB3YWl0IGZhaWxlZCB3aXRoIGNvZGU6JWQiLCByZXN1bHQpOwotCi0gICAgcmVzdWx0ID0g cHRocmVhZF9tdXRleF91bmxvY2soJnN0YXJ0dXBMb2NrKTsKLSAgICBBU1NFUlRfV0lUSF9NRVNT QUdFKHJlc3VsdCA9PSAwLCAic3RhcnR1cCB1bmxvY2sgZmFpbGVkIHdpdGggY29kZTolZCIsIHJl c3VsdCk7CisgICAgICAgIHN0YXJ0dXBDb25kaXRpb24ud2FpdChzdGFydHVwTG9jayk7CisgICAg fQogCiAgICAgaW5pdGlhbGl6ZUFwcGxpY2F0aW9uVUlUaHJlYWRJZGVudGlmaWVyKCk7CiB9Cg==