Bug 104684

Summary: [GTK] fast/xmlhttprequest/xmlhttprequest-recursive-sync-event.html is crashing
Product: WebKit Reporter: Zan Dobersek <zan>
Component: WebKitGTKAssignee: Nobody <webkit-unassigned>
Status: NEW ---    
Severity: Normal CC: bugs-noreply, danw, d-r, gustavo, mrobinson
Priority: P2 Keywords: Gtk, LayoutTestFailure
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   

Description Zan Dobersek 2012-12-11 10:35:35 PST
The test is crashing in a flaky manner. It started doing so after the libsoup Jhbuild dependency was bumped in r137286
http://trac.webkit.org/changeset/137286

The update bumped libsoup from the 2.40.1 tag to the 5651d18e commit.
http://git.gnome.org/browse/libsoup/commit/?id=91efda67678dbad28603ee8034a10cc6081f4b8d
http://git.gnome.org/browse/libsoup/commit/?id=5651d18eb8c9edff5961a32c488641bf6776ca6b

Glib was also updated in that patch, maybe the fault is there.

This test only crashes on the 64-bit release builder. That system has 24 cores so there are possibly other tests being run before this crasher which basically just set up the conditions in which the crash then occurs.

Here's the crash log, indicating file descriptor excess.
...
[Thread debugging using libthread_db enabled]
Core was generated by `/home/slave/webkitgtk/gtk-linux-64-release/build/WebKitBuild/Release/Programs/D'.
Program terminated with signal 5, Trace/breakpoint trap.
#0  0x00007f6f1de37978 in g_logv () from /home/slave/webkitgtk/gtk-linux-64-release/build/WebKitBuild/Dependencies/Root/lib64/libglib-2.0.so.0

...

Thread 1 (Thread 0x7f6f15046900 (LWP 3517)):
#0  0x00007f6f1de37978 in g_logv () from /home/slave/webkitgtk/gtk-linux-64-release/build/WebKitBuild/Dependencies/Root/lib64/libglib-2.0.so.0
#1  0x00007f6f1de37a75 in g_log () from /home/slave/webkitgtk/gtk-linux-64-release/build/WebKitBuild/Dependencies/Root/lib64/libglib-2.0.so.0
#2  0x00007f6f1de7bc13 in g_wakeup_new () from /home/slave/webkitgtk/gtk-linux-64-release/build/WebKitBuild/Dependencies/Root/lib64/libglib-2.0.so.0
#3  0x00007f6f1de2b25d in g_main_context_new () from /home/slave/webkitgtk/gtk-linux-64-release/build/WebKitBuild/Dependencies/Root/lib64/libglib-2.0.so.0
#4  0x00007f6f1fea84e1 in WebCore::WebCoreSynchronousLoader::WebCoreSynchronousLoader(WebCore::ResourceError&, WebCore::ResourceResponse&, _SoupSession*, WTF::Vector<char, 0ul>&) () from /home/slave/webkitgtk/gtk-linux-64-release/build/WebKitBuild/Release/.libs/libwebkitgtk-3.0.so.0
#5  0x00007f6f1fea565e in WebCore::ResourceHandle::loadResourceSynchronously(WebCore::NetworkingContext*, WebCore::ResourceRequest const&, WebCore::StoredCredentials, WebCore::ResourceError&, WebCore::ResourceResponse&, WTF::Vector<char, 0ul>&) () from /home/slave/webkitgtk/gtk-linux-64-release/build/WebKitBuild/Release/.libs/libwebkitgtk-3.0.so.0
#6  0x00007f6f1f741648 in WebCore::FrameLoader::loadResourceSynchronously(WebCore::ResourceRequest const&, WebCore::StoredCredentials, WebCore::ResourceError&, WebCore::ResourceResponse&, WTF::Vector<char, 0ul>&) () from /home/slave/webkitgtk/gtk-linux-64-release/build/WebKitBuild/Release/.libs/libwebkitgtk-3.0.so.0
#7  0x00007f6f1f72b53f in WebCore::DocumentThreadableLoader::loadRequest(WebCore::ResourceRequest const&, WebCore::SecurityCheckPolicy) () from /home/slave/webkitgtk/gtk-linux-64-release/build/WebKitBuild/Release/.libs/libwebkitgtk-3.0.so.0
#8  0x00007f6f1f72da30 in WebCore::DocumentThreadableLoader::DocumentThreadableLoader(WebCore::Document*, WebCore::ThreadableLoaderClient*, WebCore::DocumentThreadableLoader::BlockingBehavior, WebCore::ResourceRequest const&, WebCore::ThreadableLoaderOptions const&) () from /home/slave/webkitgtk/gtk-linux-64-release/build/WebKitBuild/Release/.libs/libwebkitgtk-3.0.so.0
#9  0x00007f6f1f72dbcc in WebCore::DocumentThreadableLoader::loadResourceSynchronously(WebCore::Document*, WebCore::ResourceRequest const&, WebCore::ThreadableLoaderClient&, WebCore::ThreadableLoaderOptions const&) () from /home/slave/webkitgtk/gtk-linux-64-release/build/WebKitBuild/Release/.libs/libwebkitgtk-3.0.so.0
#10 0x00007f6f1fa0900d in WebCore::XMLHttpRequest::createRequest(int&) () from /home/slave/webkitgtk/gtk-linux-64-release/build/WebKitBuild/Release/.libs/libwebkitgtk-3.0.so.0
#11 0x00007f6f1fa0d1cb in WebCore::XMLHttpRequest::send(WTF::String const&, int&) () from /home/slave/webkitgtk/gtk-linux-64-release/build/WebKitBuild/Release/.libs/libwebkitgtk-3.0.so.0
#12 0x00007f6f1fa0d5f8 in WebCore::XMLHttpRequest::send(int&) () from /home/slave/webkitgtk/gtk-linux-64-release/build/WebKitBuild/Release/.libs/libwebkitgtk-3.0.so.0
#13 0x00007f6f1f22bcde in WebCore::JSXMLHttpRequest::send(JSC::ExecState*) () from /home/slave/webkitgtk/gtk-linux-64-release/build/WebKitBuild/Release/.libs/libwebkitgtk-3.0.so.0
#14 0x00007f6f1fdae15b in WebCore::jsXMLHttpRequestPrototypeFunctionSend(JSC::ExecState*) () from /home/slave/webkitgtk/gtk-linux-64-release/build/WebKitBuild/Release/.libs/libwebkitgtk-3.0.so.0
#15 0x00007f6ed4410265 in ?? ()
#16 0x0000000000000000 in ?? ()

STDERR: 
STDERR: warning: Can't read pathname for load map: Input/output error.
STDERR: Creating pipes for GWakeup: Too many open files
STDERR:
Comment 1 Zan Dobersek 2012-12-12 02:30:16 PST
The 32-bit and 64-bit WK2 bots caught up and are reporting constant crashes:
http://test-results.appspot.com/dashboards/flakiness_dashboard.html#group=%40ToT%20-%20webkit.org&tests=fast%2Fxmlhttprequest%2Fxmlhttprequest-recursive-sync

No crashes on the debug builder, though.
Comment 2 Zan Dobersek 2013-01-17 04:08:15 PST
The crashes are now occurring on the debug builder as well.

Bug #100117 has detailed explanation of how this test works and why it can crash. Basically, on GTK port, the maximum call stack size is never exceeded so the file descriptors get out of hand in libsoup.

EFL and Qt ports are skipping the test, treating it like a WONTFIX.
IMO crashing is bad, so this test still needs attention on the GTK port.
Comment 3 Dan Winship 2013-01-17 08:39:43 PST
Eventually, we should switch to using the new "plain" SoupSession (rather than SoupSessionAsync), which lets you do both blocking and async I/O with the same session, and then we can just do blocking I/O here rather than async-I/O-on-a-new-GMainContext, so we wouldn't use an extra fd per sync request.

There might be other issues with that though that will take some time to iron out, so an easier short-term fix would be to add our own xmlhttprequest nesting limit and throw StackOverflowError earlier than we normally would.
Comment 4 Zan Dobersek 2013-01-19 11:17:10 PST
The test started failing on the 64-bit debug builder after the swap on that system was increased in size. It seems that caused an increase in the stack size as well, causing this test to crash but other test failures as well (bug #107257).

At the moment I don't have any clear idea of how to approach that. In the case of this test and the current soup behavior, this seems as an occurrence that any user could stumble upon.

(In reply to comment #3)
> Eventually, we should switch to using the new "plain" SoupSession (rather than SoupSessionAsync), which lets you do both blocking and async I/O with the same session, and then we can just do blocking I/O here rather than async-I/O-on-a-new-GMainContext, so we wouldn't use an extra fd per sync request.

That seems ideal.

> There might be other issues with that though that will take some time to iron out, so an easier short-term fix would be to add our own xmlhttprequest nesting limit and throw StackOverflowError earlier than we normally would.

I'm not sure if that is doable. The stack is controlled solely in JavaScriptCore, with (to me) no clear way of limiting the XHR nesting count from WebCore.
Comment 5 Dan Winship 2013-01-19 13:19:10 PST
(In reply to comment #4)
> > There might be other issues with that though that will take some time to iron out, so an easier short-term fix would be to add our own xmlhttprequest nesting limit and throw StackOverflowError earlier than we normally would.
> 
> I'm not sure if that is doable. The stack is controlled solely in JavaScriptCore, with (to me) no clear way of limiting the XHR nesting count from WebCore.

Well, I was thinking maybe you could return an appropriate error from loadResourceSynchronously that would get translated to StackOverflowError...

However... I notice now that loadResourceSynchronously asserts !loadingSynchronousRequest, which means that the previous WebCoreSynchronousLoader has to have already been destroyed at that point... so we shouldn't run out of fds, unless we're leaking GMainContexts or something?
Comment 6 Dominik Röttsches (drott) 2013-01-21 08:01:37 PST
(In reply to comment #3)
> Eventually, we should switch to using the new "plain" SoupSession (rather than SoupSessionAsync), which lets you do both blocking and async I/O with the same session, and then we can just do blocking I/O here rather than async-I/O-on-a-new-GMainContext, so we wouldn't use an extra fd per sync request.

I created bug 107451 to track that suggestion.