REGRESSION (r125126): Multiple crashes introduced in GTK debug builds

[Zan Dobersek]

After r125133[1] a couple of tests are crashing on the GTK builder. These tests are also flaky on the release build (as in they pass when rerun).
Test results server tells the story:
http://test-results.appspot.com/dashboards/flakiness_dashboard.html#group=%40ToT%20-%20webkit.org&tests=fast%2Fevents%2Fkeyevent-iframe-removed-crash.html%2Cfullscreen%2Ffull-screen-iframe-zIndex.html%2Cfullscreen%2Ffull-screen-iframe-allowed.html%2Cfullscreen%2Ffull-screen-iframe-not-allowed.html%2Cfullscreen%2Fexit-full-screen-iframe.html%2Csvg%2Fcustom%2Fuse-instanceRoot-as-event-target.xhtml

Here's the backtrace of the crash:
Crash log for DumpRenderTree (pid 28139):

[New LWP 28139]
[New LWP 28155]
[New LWP 28148]
[New LWP 28206]
[New LWP 28230]
[New LWP 28233]
[New LWP 28231]
[New LWP 28250]
[New LWP 28149]
[New LWP 28150]
[New LWP 28151]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Debug/Programs/DumpR'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f915a6b0973 in WebCore::JSEventListener::jsFunction (this=0xde9df70, scriptExecutionContext=0xfc6e6b8) at ../../Source/WebCore/bindings/js/JSEventListener.h:90
90	        ASSERT(m_wrapper || !m_jsFunction);

...

Thread 1 (Thread 0x7f914ede7900 (LWP 28139)):
#0  0x00007f915a6b0973 in WebCore::JSEventListener::jsFunction (this=0xde9df70, scriptExecutionContext=0xfc6e6b8) at ../../Source/WebCore/bindings/js/JSEventListener.h:90
#1  0x00007f915a6affec in WebCore::JSEventListener::handleEvent (this=0xde9df70, scriptExecutionContext=0xfc6e6b8, event=0xcd32f20) at ../../Source/WebCore/bindings/js/JSEventListener.cpp:80
#2  0x00007f915a9837ea in WebCore::EventTarget::fireEventListeners (this=0xde41340, event=0xcd32f20, d=0xde41730, entry=WTF::Vector of length 1, capacity 1 = {...}) at ../../Source/WebCore/dom/EventTarget.cpp:231
#3  0x00007f915a983648 in WebCore::EventTarget::fireEventListeners (this=0xde41340, event=0xcd32f20) at ../../Source/WebCore/dom/EventTarget.cpp:198
#4  0x00007f915a9ab01f in WebCore::Node::handleLocalEvents (this=0xde41340, event=0xcd32f20) at ../../Source/WebCore/dom/Node.cpp:2566
#5  0x00007f915a9750a9 in WebCore::EventContext::handleLocalEvents (this=0x118747e0, event=0xcd32f20) at ../../Source/WebCore/dom/EventContext.cpp:54
#6  0x00007f915a977cfb in WebCore::EventDispatcher::dispatchEventAtTarget (this=0x7fffc2754f10, event=...) at ../../Source/WebCore/dom/EventDispatcher.cpp:308
#7  0x00007f915a9770bd in WebCore::EventDispatcher::dispatchEvent (this=0x7fffc2754f10, prpEvent=...) at ../../Source/WebCore/dom/EventDispatcher.cpp:261
#8  0x00007f915a9926b0 in WebCore::MouseEventDispatchMediator::dispatchEvent (this=0x118a4580, dispatcher=0x7fffc2754f10) at ../../Source/WebCore/dom/MouseEvent.cpp:207
#9  0x00007f915a976162 in WebCore::EventDispatcher::dispatchEvent (node=0xde41340, mediator=...) at ../../Source/WebCore/dom/EventDispatcher.cpp:129
#10 0x00007f915a9ab948 in WebCore::Node::dispatchMouseEvent (this=0xde41340, event=..., eventType="mouseover", detail=0, relatedTarget=0x10488da0) at ../../Source/WebCore/dom/Node.cpp:2628
#11 0x00007f915ae03547 in WebCore::EventHandler::updateMouseEventTargetNode (this=0x1e703c8, targetNode=0xde41340, mouseEvent=..., fireMouseOverOut=true) at ../../Source/WebCore/page/EventHandler.cpp:2221
#12 0x00007f915ae0363f in WebCore::EventHandler::dispatchMouseEvent (this=0x1e703c8, eventType="mousemove", targetNode=0xde41340, clickCount=0, mouseEvent=..., setUnder=true) at ../../Source/WebCore/page/EventHandler.cpp:2235
#13 0x00007f915ae017d6 in WebCore::EventHandler::handleMouseMoveEvent (this=0x1e703c8, mouseEvent=..., hoveredNode=0x7fffc27553e0, onlyUpdateScrollbars=false) at ../../Source/WebCore/page/EventHandler.cpp:1821
#14 0x00007f915ae00f1f in WebCore::EventHandler::mouseMoved (this=0x1e703c8, event=...) at ../../Source/WebCore/page/EventHandler.cpp:1693
#15 0x00007f915a56be98 in webkit_web_view_motion_event (widget=0x1e44000, event=0x270cea0) at ../../Source/WebKit/gtk/webkit/webkitwebview.cpp:790
#16 0x00007f9158d51a14 in _gtk_marshal_BOOLEAN__BOXEDv () from /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Dependencies/Root/lib64/libgtk-3.so.0
#17 0x00007f9158566b02 in g_type_class_meta_marshalv () from /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Dependencies/Root/lib64/libgobject-2.0.so.0
#18 0x00007f91585666c5 in _g_closure_invoke_va () from /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Dependencies/Root/lib64/libgobject-2.0.so.0
#19 0x00007f9158582138 in g_signal_emit_valist () from /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Dependencies/Root/lib64/libgobject-2.0.so.0
#20 0x00007f91585832ec in g_signal_emit () from /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Dependencies/Root/lib64/libgobject-2.0.so.0
#21 0x00007f9158ee6da1 in gtk_widget_event_internal () from /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Dependencies/Root/lib64/libgtk-3.so.0
#22 0x00007f9158ee640b in gtk_widget_event () from /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Dependencies/Root/lib64/libgtk-3.so.0
#23 0x00007f9158d5136d in propagate_event_up () from /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Dependencies/Root/lib64/libgtk-3.so.0
#24 0x00007f9158d516cf in propagate_event () from /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Dependencies/Root/lib64/libgtk-3.so.0
#25 0x00007f9158d5179d in gtk_propagate_event () from /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Dependencies/Root/lib64/libgtk-3.so.0
#26 0x00007f9158d502b3 in gtk_main_do_event () from /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Dependencies/Root/lib64/libgtk-3.so.0
#27 0x000000000047f249 in dispatchEvent (event=0x270cea0) at ../../Tools/DumpRenderTree/gtk/EventSender.cpp:577
#28 0x000000000047f1d2 in sendOrQueueEvent (event=0x270cea0, shouldReplaySavedEvents=false) at ../../Tools/DumpRenderTree/gtk/EventSender.cpp:562
#29 0x000000000047e856 in mouseMoveToCallback (context=0x7f910a7c8088, function=0x7f910a77e1a0, thisObject=0x7f910a77dba0, argumentCount=2, arguments=0x7fffc2755db8, exception=0x7fffc2755e58) at ../../Tools/DumpRenderTree/gtk/EventSender.cpp:418
#30 0x00007f915f11c038 in JSC::JSCallbackFunction::call (exec=0x7f910a7c8088) at ../../Source/JavaScriptCore/API/JSCallbackFunction.cpp:73
#31 0x00007f915f31f543 in JSC::LLInt::handleHostCall (execCallee=0x7f910a7c8088, pc=0x11865c60, callee=..., kind=JSC::CodeForCall) at ../../Source/JavaScriptCore/llint/LLIntSlowPaths.cpp:1323
#32 0x00007f915f322263 in JSC::LLInt::setUpCall (execCallee=0x7f910a7c8088, pc=0x11865c60, kind=JSC::CodeForCall, calleeAsValue=..., callLinkInfo=0xd87a830) at ../../Source/JavaScriptCore/llint/LLIntSlowPaths.cpp:1367
#33 0x00007f915f3227de in JSC::LLInt::genericCall (exec=0x7f910a7c8038, pc=0x11865c60, kind=JSC::CodeForCall) at ../../Source/JavaScriptCore/llint/LLIntSlowPaths.cpp:1423
#34 0x00007f915f31faa6 in JSC::LLInt::llint_slow_path_call (exec=0x7f910a7c8038, pc=0x11865c60) at ../../Source/JavaScriptCore/llint/LLIntSlowPaths.cpp:1429
#35 0x00007f915f326192 in llint_op_call () from /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Debug/.libs/libjavascriptcoregtk-3.0.so.0
#36 0x00007fffc2756210 in ?? ()
#37 0x00007fffc2756240 in ?? ()
#38 0x00007f910a79c840 in ?? ()
#39 0x00007f915f226bd9 in JSC::Register::Register (this=0x0) at ../../Source/JavaScriptCore/interpreter/Register.h:105
#40 0x00007f915f2d31fe in JSC::JITCode::execute (this=0x7f910a6dc148, registerFile=0x1e97208, callFrame=0x7f910a7c8038, globalData=0x1efaa80) at ../../Source/JavaScriptCore/jit/JITCode.h:133
#41 0x00007f915f2cfb18 in JSC::Interpreter::executeCall (this=0x1e971f0, callFrame=0x7f910a75ee88, function=0x7f910a79c840, callType=JSC::CallTypeJS, callData=..., thisValue=..., args=...) at ../../Source/JavaScriptCore/interpreter/Interpreter.cpp:1322
#42 0x00007f915f39ce1d in JSC::call (exec=0x7f910a75ee88, functionObject=..., callType=JSC::CallTypeJS, callData=..., thisValue=..., args=...) at ../../Source/JavaScriptCore/runtime/CallData.cpp:39
#43 0x00007f915a680e73 in WebCore::JSMainThreadExecState::call (exec=0x7f910a75ee88, functionObject=..., callType=JSC::CallTypeJS, callData=..., thisValue=..., args=...) at ../../Source/WebCore/bindings/js/JSMainThreadExecState.h:56
#44 0x00007f915a6ef156 in WebCore::ScheduledAction::executeFunctionInContext (this=0x118af8b0, globalObject=0x7f910a75ec80, thisValue=..., context=0xfc6e6b8) at ../../Source/WebCore/bindings/js/ScheduledAction.cpp:115
#45 0x00007f915a6ef342 in WebCore::ScheduledAction::execute (this=0x118af8b0, document=0xfc6e590) at ../../Source/WebCore/bindings/js/ScheduledAction.cpp:137
#46 0x00007f915a6eeec6 in WebCore::ScheduledAction::execute (this=0x118af8b0, context=0xfc6e6b8) at ../../Source/WebCore/bindings/js/ScheduledAction.cpp:83
#47 0x00007f915ade645a in WebCore::DOMTimer::fired (this=0xde9f230) at ../../Source/WebCore/page/DOMTimer.cpp:149
#48 0x00007f915af8e698 in WebCore::ThreadTimers::sharedTimerFiredInternal (this=0x1e86540) at ../../Source/WebCore/platform/ThreadTimers.cpp:115
#49 0x00007f915af8e59f in WebCore::ThreadTimers::sharedTimerFired () at ../../Source/WebCore/platform/ThreadTimers.cpp:93
#50 0x00007f915ba0c8d2 in WebCore::timeout_cb () at ../../Source/WebCore/platform/gtk/SharedTimerGtk.cpp:49
#51 0x00007f9158461a42 in g_timeout_dispatch () from /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Dependencies/Root/lib64/libglib-2.0.so.0
#52 0x00007f915845fc91 in g_main_dispatch () from /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Dependencies/Root/lib64/libglib-2.0.so.0
#53 0x00007f9158460956 in g_main_context_dispatch () from /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Dependencies/Root/lib64/libglib-2.0.so.0
#54 0x00007f9158460b39 in g_main_context_iterate () from /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Dependencies/Root/lib64/libglib-2.0.so.0
#55 0x00007f9158460f69 in g_main_loop_run () from /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Dependencies/Root/lib64/libglib-2.0.so.0
#56 0x00007f9158d4f7de in gtk_main () from /home/slave/webkitgtk/gtk-linux-64-debug/build/WebKitBuild/Dependencies/Root/lib64/libgtk-3.so.0
#57 0x0000000000479dd5 in runTest (inputLine=...) at ../../Tools/DumpRenderTree/gtk/DumpRenderTree.cpp:752
#58 0x00000000004794a9 in runTestingServerLoop () at ../../Tools/DumpRenderTree/gtk/DumpRenderTree.cpp:539
#59 0x000000000047c434 in main (argc=2, argv=0x7fffc2757388) at ../../Tools/DumpRenderTree/gtk/DumpRenderTree.cpp:1442

[1] - http://trac.webkit.org/changeset/125133

[Hayato Ito]

Okay. Let me take a look.

[Hayato Ito]

It might take some time for me to set up GTK build environments...

[Hayato Ito]

I could reproduce some of them on my local environment. Let me investigate further.

(In reply to comment #2)
> It might take some time for me to set up GTK build environments...

[Hayato Ito]

GTK build takes too much time for me. Let me continue tomorrow. I've not found out the cause yet.

It's okay for me to revert r125133 if it is absolutely needed. 

(In reply to comment #3)
> I could reproduce some of them on my local environment. Let me investigate further.
> 
> (In reply to comment #2)
> > It might take some time for me to set up GTK build environments...

[Dominic Cooney]

use-instanceRoot-as-event-target.xhtml is probably related to r125251.

[Hayato Ito]

I confirmed that the crash in #1 happend on the revision before r125133 on my local environment.
I am afraid that we should bisect to find the cause.

(In reply to comment #4)
> GTK build takes too much time for me. Let me continue tomorrow. I've not found out the cause yet.
> 
> It's okay for me to revert r125133 if it is absolutely needed. 
> 
> (In reply to comment #3)
> > I could reproduce some of them on my local environment. Let me investigate further.
> > 
> > (In reply to comment #2)
> > > It might take some time for me to set up GTK build environments...

[Hayato Ito]

I'd like to note that crash is very flaky. Hard to reproduce on my local gtk build environment. It makes bisect very difficult.
Flakiness dashboard told me that limited recent results. Is there any way to see more previous results?

[Zan Dobersek]

(In reply to comment #7)
> I'd like to note that crash is very flaky. Hard to reproduce on my local gtk build environment. It makes bisect very difficult.
> Flakiness dashboard told me that limited recent results. Is there any way to see more previous results?

You can check the 'Show all runs' checkbox in the top right of the flakiness dashboard. That shows that on the 64-bit release bot, the tests first started failing in the revision range r125121-r125128. That range is covered by the following builds:
http://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release/builds/27330
http://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release/builds/27331

Unfortunately, the first buildbot cycle didn't make it through because of dependencies update failure.

Other than that, it seems the regression is hard to find because of unfortunate combination of bad patches and outside factors. At least the 64-bit debug builder is clear that the regression started occurring somewhere in between r12103 and r125133:
http://build.webkit.org/builders/GTK%20Linux%2064-bit%20Debug?numbuilds=200

This irritates me well enough that I'll take a look at it in the CET afternoon if you don't find the offending commit by then. Thanks for the effort though, much appreciated!

[Zan Dobersek]

The bisection outlined r125126 as the offending commit.
http://trac.webkit.org/changeset/125126

CC-ing proper people.

[Adam Barth]

This is on my list for today.

*** This bug has been marked as a duplicate of bug 93654 ***