167876 2017-02-06 05:45:23 -0800 [Soup] Deadlock in NetworkProcess 2017-02-06 10:02:32 -0800 1 1 1 Unclassified WebKit WebKitGTK WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED https://bugzilla.redhat.com/show_bug.cgi?id=1418413 P2 Normal --- 1 tpopela webkit-unassigned bugs-noreply cgarcia oldest_to_newest 1273538 0 tpopela 2017-02-06 05:45:23 -0800 Loading big messages in Evolution could leave NetworkProcess in deadlock, causing Evolution to not load anything later. It was introduced in http://trac.webkit.org/changeset/210374 (bisected there). Thread 1 (Thread 0x7f0e6feb6fc0 (LWP 21043)): #0 0x00007f0e678dc460 in pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f0e6a9645eb in WTF::ThreadCondition::timedWait(WTF::Mutex&, double) () at /home/tpopela/dev/upstream/WebKit/WebKitBuild/Release/lib/libjavascriptcoregtk-4.0.so.18 #2 0x00007f0e6a93d963 in WTF::ParkingLot::parkConditionallyImpl(void const*, WTF::ScopedLambda<bool ()> const&, WTF::ScopedLambda<void ()> const&, WTF::TimeWithDynamicClockType const&) () at /home/tpopela/dev/upstream/WebKit/WebKitBuild/Release/lib/libjavascriptcoregtk-4.0.so.18 #3 0x00007f0e6a934234 in WTF::LockAlgorithm<unsigned char, (unsigned char)1, (unsigned char)2>::lockSlow(WTF::Atomic<unsigned char>&) () at /home/tpopela/dev/upstream/WebKit/WebKitBuild/Release/lib/libjavascriptcoregtk-4.0.so.18 #4 0x00007f0e6dd7dc3c in webkitSoupRequestInputStreamReadAsync(_GInputStream*, void*, unsigned long, int, _GCancellable*, void (*)(_GObject*, _GAsyncResult*, void*), void*) () at /home/tpopela/dev/upstream/WebKit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37 #5 0x00007f0e64c432d9 in g_input_stream_read_async (stream=0x10387e0 [WebKitSoupRequestInputStream], buffer=0x7f0e13df0000, count=8192, io_priority=0, cancellable=0x10d8cf0 [GCancellable], callback= 0x7f0e6dc502c0 <WebKit::NetworkDataTaskSoup::readCallback(_GInputStream*, _GAsyncResult*, WebKit::NetworkDataTaskSoup*)>, user_data=0x7f0e13df8480) at ginputstream.c:633 #6 0x00007f0e6dc4ef6f in WebKit::NetworkDataTaskSoup::read() () at /home/tpopela/dev/upstream/WebKit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37 #7 0x00007f0e6dc4f58d in WebKit::NetworkDataTaskSoup::didRead(long) () at /home/tpopela/dev/upstream/WebKit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37 #8 0x00007f0e6dc50428 in WebKit::NetworkDataTaskSoup::readCallback(_GInputStream*, _GAsyncResult*, WebKit::NetworkDataTaskSoup*) () at /home/tpopela/dev/upstream/WebKit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37 #9 0x00007f0e64c43f9a in async_ready_callback_wrapper (source_object=0x10387e0 [WebKitSoupRequestInputStream], res=0x119ae30, user_data=0x7f0e13df8480) at ginputstream.c:532 #10 0x00007f0e64c6b9d4 in g_task_return_now (task=0x119ae30 [GTask]) at gtask.c:1121 #11 0x00007f0e64c6c086 in g_task_return (task=0x119ae30 [GTask], type=<optimized out>) at gtask.c:1179 #12 0x00007f0e6dd7daf3 in webkitSoupRequestInputStreamReadAsyncResultComplete(_GTask*, void*, unsigned long) () at /home/tpopela/dev/upstream/WebKit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37 #13 0x00007f0e6dd7df6e in webkitSoupRequestInputStreamAddData () at /home/tpopela/dev/upstream/WebKit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37 #14 0x00007f0e6dda28c5 in WebKit::CustomProtocolManager::didReceiveMessage(IPC::Connection&, IPC::Decoder&) () at /home/tpopela/dev/upstream/WebKit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37 #15 0x00007f0e6da2e749 in IPC::MessageReceiverMap::dispatchMessage(IPC::Connection&, IPC::Decoder&) () at /home/tpopela/dev/upstream/WebKit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37 #16 0x00007f0e6dc060c6 in WebKit::NetworkProcess::didReceiveMessage(IPC::Connection&, IPC::Decoder&) () at /home/tpopela/dev/upstream/WebKit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37 #17 0x00007f0e6da2a50b in IPC::Connection::dispatchMessage(std::unique_ptr<IPC::Decoder, std::default_delete<IPC::Decoder> >) () at /home/tpopela/dev/upstream/WebKit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37 #18 0x00007f0e6da2b078 in IPC::Connection::dispatchOneMessage() () at /home/tpopela/dev/upstream/WebKit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37 #19 0x00007f0e6a93fb25 in WTF::RunLoop::performWork() () at /home/tpopela/dev/upstream/WebKit/WebKitBuild/Release/lib/libjavascriptcoregtk-4.0.so.18 #20 0x00007f0e6a965b79 in WTF::RunLoop::RunLoop()::{lambda(void*)#1}::_FUN(void*) () at /home/tpopela/dev/upstream/WebKit/WebKitBuild/Release/lib/libjavascriptcoregtk-4.0.so.18 #21 0x00007f0e644c3e42 in g_main_dispatch (context=0x98eb30) at gmain.c:3203 #22 0x00007f0e644c3e42 in g_main_context_dispatch (context=context@entry=0x98eb30) at gmain.c:3856 #23 0x00007f0e644c41c0 in g_main_context_iterate (context=0x98eb30, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at gmain.c:3929 #24 0x00007f0e644c44e2 in g_main_loop_run (loop=0x98ed20) at gmain.c:4125 #25 0x00007f0e6a966410 in WTF::RunLoop::run() () at /home/tpopela/dev/upstream/WebKit/WebKitBuild/Release/lib/libjavascriptcoregtk-4.0.so.18 #26 0x00007f0e6dc59706 in int WebKit::ChildProcessMain<WebKit::NetworkProcess, WebKit::ChildProcessMainBase>(int, char**) () at /home/tpopela/dev/upstream/WebKit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37 #27 0x00007f0e60d2a401 in __libc_start_main (main=0x400ac0 <main>, argc=2, argv=0x7ffc78222528, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc78222518) at ../csu/libc-start.c:289 #28 0x0000000000400b1a in _start () 1273541 1 cgarcia 2017-02-06 05:57:38 -0800 WebKitSoupRequestInputStream uses a read lock.What is happening is that webkitSoupRequestInputStreamAddData takes the lock, and it calls webkitSoupRequestInputStreamPendingReadAsyncComplete with the lock help. That causes webkitSoupRequestInputStreamReadAsync to be called again to read the next chunk, but in the same run loop operation. webkitSoupRequestInputStreamReadAsync also takes the read lock. I don't know why we are using that read lock, I don't think it's needed, at least now everything should happen in the main thread. But I'm going to look at it in more detail. If the lock is needed, then the solution is to release it before calling webkitSoupRequestInputStreamPendingReadAsyncComplete 1273542 2 cgarcia 2017-02-06 06:06:01 -0800 According to myself in https://bugs.webkit.org/show_bug.cgi?id=85880#c9 the mutex is needed, because glib will use a thread to implement read_async, and webkitSoupRequestInputStreamAddData can be called from other thread. Before r210374, that thread was the message work queue, that's why this never happened. Now it's the main thread, so I guess we just need to release the lock. 1273546 3 cgarcia 2017-02-06 06:19:53 -0800 (In reply to comment #2) > According to myself in https://bugs.webkit.org/show_bug.cgi?id=85880#c9 the > mutex is needed, because glib will use a thread to implement read_async, and > webkitSoupRequestInputStreamAddData can be called from other thread. Before > r210374, that thread was the message work queue, that's why this never > happened. Now it's the main thread, so I guess we just need to release the > lock. But I was wrong, because we are overriding GInputStreamClass::read_async, claiming we know how to handle the async request, and therefore not using the fallback implementation in GInputStream that runs the thread. It makes sense, indeed, because we are actually a GMemoryInputStream, there's no io in the read function, so we can handle it in the main thread. 1273548 4 300716 cgarcia 2017-02-06 06:29:49 -0800 Created attachment 300716 Patch 1273611 5 cgarcia 2017-02-06 10:02:32 -0800 Committed r211734: <http://trac.webkit.org/changeset/211734> 300716 2017-02-06 06:29:49 -0800 2017-02-06 07:31:37 -0800 Patch wk2-custom-protocols-deadlock.diff text/plain 3351 cgarcia ZGlmZiAtLWdpdCBhL1NvdXJjZS9XZWJLaXQyL0NoYW5nZUxvZyBiL1NvdXJjZS9XZWJLaXQyL0No YW5nZUxvZwppbmRleCAwOTA1YzdkYzYyNS4uMzhiYWE1NjA5NTggMTAwNjQ0Ci0tLSBhL1NvdXJj ZS9XZWJLaXQyL0NoYW5nZUxvZworKysgYi9Tb3VyY2UvV2ViS2l0Mi9DaGFuZ2VMb2cKQEAgLTEs MyArMSwyMCBAQAorMjAxNy0wMi0wNiAgQ2FybG9zIEdhcmNpYSBDYW1wb3MgIDxjZ2FyY2lhQGln YWxpYS5jb20+CisKKyAgICAgICAgW1NvdXBdIERlYWRsb2NrIGluIE5ldHdvcmtQcm9jZXNzCisg ICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0xNjc4NzYKKwor ICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICBXZWJLaXRTb3Vw UmVxdWVzdElucHV0U3RyZWFtIHVzZXMgYSByZWFkIGxvY2suIFdoYXQgaXMgaGFwcGVuaW5nIGlz IHRoYXQgd2Via2l0U291cFJlcXVlc3RJbnB1dFN0cmVhbUFkZERhdGEKKyAgICAgICAgdGFrZXMg dGhlIGxvY2ssIGFuZCBpdCBjYWxscyB3ZWJraXRTb3VwUmVxdWVzdElucHV0U3RyZWFtUGVuZGlu Z1JlYWRBc3luY0NvbXBsZXRlIHdpdGggdGhlIGxvY2sgaGVscC4gVGhhdAorICAgICAgICBjYXVz ZXMgd2Via2l0U291cFJlcXVlc3RJbnB1dFN0cmVhbVJlYWRBc3luYyB0byBiZSBjYWxsZWQgYWdh aW4gdG8gcmVhZCB0aGUgbmV4dCBjaHVuaywgYnV0IGluIHRoZSBzYW1lIHJ1biBsb29wCisgICAg ICAgIG9wZXJhdGlvbi4gV2UgZG9uJ3QgcmVhbGx5IG5lZWQgdGhlIHJlYWQgbG9jayBiZWNhdXNl IGJvdGggd2Via2l0U291cFJlcXVlc3RJbnB1dFN0cmVhbUFkZERhdGEgYW5kCisgICAgICAgIHdl YmtpdFNvdXBSZXF1ZXN0SW5wdXRTdHJlYW1SZWFkQXN5bmMgc2hvdWRsIGFsd2F5cyBiZSBjYWxs ZWQgZnJvbSB0aGUgbWFpbiB0aHJlYWQuCisKKyAgICAgICAgKiBXZWJQcm9jZXNzL3NvdXAvV2Vi S2l0U291cFJlcXVlc3RJbnB1dFN0cmVhbS5jcHA6CisgICAgICAgICh3ZWJraXRTb3VwUmVxdWVz dElucHV0U3RyZWFtUmVhZEFzeW5jKTogUmVtb3ZlIHRoZSByZWFkIGxvY2sgYW5kIGFzc2VydCBp ZiBjYWxsZWQgZnJvbSBhIHNlY29uZGFyeSB0aHJlYWQuCisgICAgICAgICh3ZWJraXRTb3VwUmVx dWVzdElucHV0U3RyZWFtQWRkRGF0YSk6IERpdHRvLgorCiAyMDE3LTAyLTA1ICBTaW1vbiBGcmFz ZXIgIDxzaW1vbi5mcmFzZXJAYXBwbGUuY29tPgogCiAgICAgICAgIFJlbW92ZSB1bnBhcmVudHNP ZmZzY3JlZW5UaWxlcyBsb2dpYyBpbiBUaWxlQ29udHJvbGxlcgpkaWZmIC0tZ2l0IGEvU291cmNl L1dlYktpdDIvV2ViUHJvY2Vzcy9zb3VwL1dlYktpdFNvdXBSZXF1ZXN0SW5wdXRTdHJlYW0uY3Bw IGIvU291cmNlL1dlYktpdDIvV2ViUHJvY2Vzcy9zb3VwL1dlYktpdFNvdXBSZXF1ZXN0SW5wdXRT dHJlYW0uY3BwCmluZGV4IGEzY2RjOGQwOWUxLi43OGQ5NzYxZWRhOCAxMDA2NDQKLS0tIGEvU291 cmNlL1dlYktpdDIvV2ViUHJvY2Vzcy9zb3VwL1dlYktpdFNvdXBSZXF1ZXN0SW5wdXRTdHJlYW0u Y3BwCisrKyBiL1NvdXJjZS9XZWJLaXQyL1dlYlByb2Nlc3Mvc291cC9XZWJLaXRTb3VwUmVxdWVz dElucHV0U3RyZWFtLmNwcApAQCAtMjAsOCArMjAsNyBAQAogI2luY2x1ZGUgImNvbmZpZy5oIgog I2luY2x1ZGUgIldlYktpdFNvdXBSZXF1ZXN0SW5wdXRTdHJlYW0uaCIKIAotI2luY2x1ZGUgPHd0 Zi9Mb2NrLmg+Ci0jaW5jbHVkZSA8d3RmL1RocmVhZGluZy5oPgorI2luY2x1ZGUgPHd0Zi9NYWlu VGhyZWFkLmg+CiAjaW5jbHVkZSA8d3RmL2dsaWIvR1JlZlB0ci5oPgogI2luY2x1ZGUgPHd0Zi9n bGliL0dVbmlxdWVQdHIuaD4KIApAQCAtNDUsNyArNDQsNiBAQCBzdHJ1Y3QgX1dlYktpdFNvdXBS ZXF1ZXN0SW5wdXRTdHJlYW1Qcml2YXRlIHsKIAogICAgIEdVbmlxdWVQdHI8R0Vycm9yPiBlcnJv cjsKIAotICAgIExvY2sgcmVhZExvY2s7CiAgICAgc3RkOjp1bmlxdWVfcHRyPEFzeW5jUmVhZERh dGE+IHBlbmRpbmdBc3luY1JlYWQ7CiB9OwogCkBAIC04NSwxMSArODMsMTAgQEAgc3RhdGljIGJv b2wgd2Via2l0U291cFJlcXVlc3RJbnB1dFN0cmVhbUlzV2FpdGluZ0ZvckRhdGEoV2ViS2l0U291 cFJlcXVlc3RJbnB1dFMKIAogc3RhdGljIHZvaWQgd2Via2l0U291cFJlcXVlc3RJbnB1dFN0cmVh bVJlYWRBc3luYyhHSW5wdXRTdHJlYW0qIGlucHV0U3RyZWFtLCB2b2lkKiBidWZmZXIsIGdzaXpl IGNvdW50LCBpbnQgLypwcmlvcml0eSovLCBHQ2FuY2VsbGFibGUqIGNhbmNlbGxhYmxlLCBHQXN5 bmNSZWFkeUNhbGxiYWNrIGNhbGxiYWNrLCBncG9pbnRlciB1c2VyRGF0YSkKIHsKKyAgICBBU1NF UlQoaXNNYWluVGhyZWFkKCkpOwogICAgIFdlYktpdFNvdXBSZXF1ZXN0SW5wdXRTdHJlYW0qIHN0 cmVhbSA9IFdFQktJVF9TT1VQX1JFUVVFU1RfSU5QVVRfU1RSRUFNKGlucHV0U3RyZWFtKTsKICAg ICBHUmVmUHRyPEdUYXNrPiB0YXNrID0gYWRvcHRHUmVmKGdfdGFza19uZXcoc3RyZWFtLCBjYW5j ZWxsYWJsZSwgY2FsbGJhY2ssIHVzZXJEYXRhKSk7CiAKLSAgICBMb2NrSG9sZGVyIGxvY2tlcihz dHJlYW0tPnByaXYtPnJlYWRMb2NrKTsKLQogICAgIGlmICghd2Via2l0U291cFJlcXVlc3RJbnB1 dFN0cmVhbUhhc0RhdGFUb1JlYWQoc3RyZWFtKSAmJiAhd2Via2l0U291cFJlcXVlc3RJbnB1dFN0 cmVhbUlzV2FpdGluZ0ZvckRhdGEoc3RyZWFtKSkgewogICAgICAgICBnX3Rhc2tfcmV0dXJuX2lu dCh0YXNrLmdldCgpLCAwKTsKICAgICAgICAgcmV0dXJuOwpAQCAtMTQ5LDExICsxNDYsMTEgQEAg R0lucHV0U3RyZWFtKiB3ZWJraXRTb3VwUmVxdWVzdElucHV0U3RyZWFtTmV3KHVpbnQ2NF90IGNv bnRlbnRMZW5ndGgpCiAKIHZvaWQgd2Via2l0U291cFJlcXVlc3RJbnB1dFN0cmVhbUFkZERhdGEo V2ViS2l0U291cFJlcXVlc3RJbnB1dFN0cmVhbSogc3RyZWFtLCBjb25zdCB2b2lkKiBkYXRhLCBz aXplX3QgZGF0YUxlbmd0aCkKIHsKKyAgICBBU1NFUlQoaXNNYWluVGhyZWFkKCkpOworCiAgICAg aWYgKHdlYmtpdFNvdXBSZXF1ZXN0SW5wdXRTdHJlYW1GaW5pc2hlZChzdHJlYW0pKQogICAgICAg ICByZXR1cm47CiAKLSAgICBMb2NrSG9sZGVyIGxvY2tlcihzdHJlYW0tPnByaXYtPnJlYWRMb2Nr KTsKLQogICAgIGlmIChkYXRhTGVuZ3RoKSB7CiAgICAgICAgIC8vIFRydW5jYXRlIHRoZSBkYXRh TGVuZ3RoIHRvIHRoZSBjb250ZW50TGVuZ3RoIGlmIGl0J3Mga25vd24uCiAgICAgICAgIGlmIChz dHJlYW0tPnByaXYtPmNvbnRlbnRMZW5ndGggJiYgc3RyZWFtLT5wcml2LT5ieXRlc1JlY2VpdmVk ICsgZGF0YUxlbmd0aCA+IHN0cmVhbS0+cHJpdi0+Y29udGVudExlbmd0aCkK