Bug 119089 – REGRESSION(FTL): Most layout tests crashes

• Component: JavaScriptCore
• Status: RESOLVED FIXED

[Allan Sandfeld Jensen]

After the FTL tree was merged and the many build breaks were fixes, we now have most of the layout tests crashes. At least a large portion of them triggers asserts or 0 point access due to CodeBlocks with no jitCode.

[Allan Sandfeld Jensen]

Created attachment 207460 [details]
Patch

[Csaba Osztrogonác]

JSC guys: If the FTL merge was so important (keeping in top secret without any announcment e-mail), please fix the bugs your hundreds of patches caused as soon as possible. Thanks for your cooperation in advance.

[Mark Lam]

Comment on attachment 207460 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=207460&action=review

> Source/JavaScriptCore/llint/LLIntSlowPaths.cpp:295
> +
> +    if (!codeBlock->jitCode()) {
> +        if (Options::verboseOSR())
> +            dataLogF("    No code to compile.\n");
> +        return false;
> +    }
> +

This doesn't make sense.  The statements that follow below will attempt to compile the codeBlock, and should handle it appropriately if the compilation fails.  Can you please share your findings that led you to think that this is needed?  Perhaps the real issue lies elsewhere.  Thanks.

[Csaba Osztrogonác]

Hi Mark,

Thanks for looking into it. You can find more detailed infos on these bots:
http://build.webkit.org/builders/GTK%20Linux%2064-bit%20Debug%20WK1/builds/3235
http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Debug/builds/29786

[Allan Sandfeld Jensen]

A common backtrace:
#0  0x00007ffff34942f4 in WTFCrash () at /src/webkit/Source/WTF/wtf/Assertions.cpp:339
#1  0x00007ffff325d459 in JSC::CodeBlock::jitCompile (this=0x69ff50, exec=0x7fff9dbfa140) at /src/webkit/Source/JavaScriptCore/bytecode/CodeBlock.h:298
#2  0x00007ffff3335f3d in JSC::LLInt::jitCompileAndSetHeuristics (codeBlock=0x69ff50, exec=0x7fff9dbfa140) at /src/webkit/Source/JavaScriptCore/llint/LLIntSlowPaths.cpp:296
#3  0x00007ffff332cd18 in JSC::LLInt::llint_replace (exec=0x7fff9dbfa140, pc=0x6a0408) at /src/webkit/Source/JavaScriptCore/llint/LLIntSlowPaths.cpp:414
#4  0x00007ffff3339209 in llint_op_ret () from /src/webkit/WebKitBuild/Debug/lib/libQt5WebKit.so.5
#5  0x00007fffffffc670 in ?? ()
#6  0x00007ffff32e7bcf in JSC::JSStack::installTrapsAfterFrame (this=0x0, frame=0x0) at /src/webkit/Source/JavaScriptCore/interpreter/JSStackInlines.h:212
#7  0x00007ffff32f8e66 in JSC::JITCode::execute (this=0x6923d0, stack=0x65d808, callFrame=0x7fff9dbfa058, vm=0x64d060) at /src/webkit/Source/JavaScriptCore/jit/JITCode.cpp:46
#8  0x00007ffff32e4455 in JSC::Interpreter::execute (this=0x65d7f0, program=0x7fffe40ffef0, callFrame=0x7fffe41ef8e0, thisObj=0x7fffe409ff70)
    at /src/webkit/Source/JavaScriptCore/interpreter/Interpreter.cpp:856
#9  0x00007ffff33c6a5d in JSC::evaluate (exec=0x7fffe41ef8e0, source=..., thisValue=..., returnedException=0x7fffffffd350) at /src/webkit/Source/JavaScriptCore/runtime/Completion.cpp:83
#10 0x00007ffff30c413f in JSEvaluateScript (ctx=0x7fffe41ef8e0, script=0x4aa700, thisObject=0x7fffe409ff70, sourceURL=0x0, startingLineNumber=0, exception=0x7fffffffd428)
    at /src/webkit/Source/JavaScriptCore/API/JSBase.cpp:61
#11 0x0000000000424e07 in DumpRenderTree::initJSObjects (this=0x7fffffffdea0) at /src/webkit/Tools/DumpRenderTree/qt/DumpRenderTreeQt.cpp:789

[Allan Sandfeld Jensen]

(In reply to comment #3)
> (From update of attachment 207460 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=207460&action=review
> 
> > Source/JavaScriptCore/llint/LLIntSlowPaths.cpp:295
> > +
> > +    if (!codeBlock->jitCode()) {
> > +        if (Options::verboseOSR())
> > +            dataLogF("    No code to compile.\n");
> > +        return false;
> > +    }
> > +
> 
> This doesn't make sense.  The statements that follow below will attempt to compile the codeBlock, and should handle it appropriately if the compilation fails.  Can you please share your findings that led you to think that this is needed?  Perhaps the real issue lies elsewhere.  Thanks.

CodeBlock::jitCompile will ASSERT because jitType() returns JITCode::None, which it does becase m_jitCode is nullptr. Trying to let it compile will just result in later ASSERTs because or similar (jitType() == JITCode::InterpreterThunk) asserts later.

[Mark Lam]

(In reply to comment #6)
> CodeBlock::jitCompile will ASSERT because jitType() returns JITCode::None, which it does becase m_jitCode is nullptr. Trying to let it compile will just result in later ASSERTs because or similar (jitType() == JITCode::InterpreterThunk) asserts later.

Thanks.  I don't think it should ever return JITCode::None.  That sounds like something else has gone haywire.  Can you please provide a test case that reproduces this issue?  I've just run run-javascriptcore-tests, and unlike the results that Csaba shared, I'm not seeing any failures at all in my local build (unmodified from trunk).

[Oliver Hunt]

(In reply to comment #2)
> JSC guys: If the FTL merge was so important (keeping in top secret without any announcment e-mail), please fix the bugs your hundreds of patches caused as soon as possible. Thanks for your cooperation in advance.

Because there should not have been any dramatic fallout - I ran all tests without the FTL enabled, are you trying to enable it?

[Csaba Osztrogonác]

(In reply to comment #8)
> (In reply to comment #2)
> > JSC guys: If the FTL merge was so important (keeping in top secret without any announcment e-mail), please fix the bugs your hundreds of patches caused as soon as possible. Thanks for your cooperation in advance.
> 
> Because there should not have been any dramatic fallout - I ran all tests without the FTL enabled, are you trying to enable it?

No, jsc and layout tests assert/crash with disabled FTL 
on GTK and Qt too. On EFL the build is still broken.

[Geoffrey Garen]

> JSC guys: If the FTL merge was so important (keeping in top secret without any announcment e-mail), please fix the bugs your hundreds of patches caused as soon as possible.

Ossy, you're a great bot watcher, but this antagonistic mindset is not helping.

[Allan Sandfeld Jensen]

(In reply to comment #7)
> (In reply to comment #6)
> > CodeBlock::jitCompile will ASSERT because jitType() returns JITCode::None, which it does becase m_jitCode is nullptr. Trying to let it compile will just result in later ASSERTs because or similar (jitType() == JITCode::InterpreterThunk) asserts later.
> 
> Thanks.  I don't think it should ever return JITCode::None.  That sounds like something else has gone haywire.  Can you please provide a test case that reproduces this issue?  I've just run run-javascriptcore-tests, and unlike the results that Csaba shared, I'm not seeing any failures at all in my local build (unmodified from trunk).

It happens when running the DumpRenderTree, but apparently only for GTK and Qt. For instance WebKitBuild/Release/bin/DumpRenderTree LayoutTest/fast/css/001.html will assert as above.

It seems to happen most frequently during clean-up phases when the execution-context is recreated, but there are a number of different triggers above the common backtrace I posted earlier.

[Csaba Osztrogonác]

(In reply to comment #10)
> > JSC guys: If the FTL merge was so important (keeping in top secret without any announcment e-mail), please fix the bugs your hundreds of patches caused as soon as possible.
> 
> Ossy, you're a great bot watcher, but this antagonistic mindset is not helping.

I agree, it doesn't help, but my 5 buildfixes maybe helped 
a little bit. And the other 22 buildfixes made by others ...

It was only my rough but frank opinion about this situation. If we had got
a friendly reminder e-mail _before_ merging, I would have been friendlier ...

[Oliver Hunt]

What are the qt builds settings?  Which JIT's are enabled? Do you have the LLINT?  Is this 32 and/or 64 bit?

That's information that would be useful to know - are tests failing in run-javascriptore-tests?

[Csaba Osztrogonác]

(In reply to comment #13)
> What are the qt builds settings?  Which JIT's are enabled? Do you have the LLINT?  Is this 32 and/or 64 bit?
> 
> That's information that would be useful to know - are tests failing in run-javascriptore-tests?

LLINT, JIT and DFG-JIT are enabled on Qt and the problem occurs 
on 32 and 64 bit too, there are failing jsc and layout tests too

Here is the 32 bit relase Qt bot: http://build.webkit.org/builders/Qt%20Linux%20Release/builds/61619
Here is the 32 bit debug Qt bot: http://build.webkit.sed.hu/builders/x86-32%20Linux%20Qt%20Debug/builds/26602
Here is the 64 bit release Qt bot: http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release/builds/52568
Here is the 64 bit debug Qt bot: http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Debug/builds/29786

and you can see these fails on the GTK bots too:
- http://build.webkit.org/builders/GTK%20Linux%2064-bit%20Debug%20WK1/builds/3240
- http://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release/builds/39364
- http://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20WK2%20%28Tests%29/builds/7516

[Mark Lam]

(In reply to comment #11)
> (In reply to comment #7)
> > (In reply to comment #6)
> > > CodeBlock::jitCompile will ASSERT because jitType() returns JITCode::None, which it does becase m_jitCode is nullptr. Trying to let it compile will just result in later ASSERTs because or similar (jitType() == JITCode::InterpreterThunk) asserts later.
> > 
> > Thanks.  I don't think it should ever return JITCode::None.  That sounds like something else has gone haywire.  Can you please provide a test case that reproduces this issue?  I've just run run-javascriptcore-tests, and unlike the results that Csaba shared, I'm not seeing any failures at all in my local build (unmodified from trunk).
> 
> It happens when running the DumpRenderTree, but apparently only for GTK and Qt. For instance WebKitBuild/Release/bin/DumpRenderTree LayoutTest/fast/css/001.html will assert as above.
> 
> It seems to happen most frequently during clean-up phases when the execution-context is recreated, but there are a number of different triggers above the common backtrace I posted earlier.

I've just finished my build of DumpRenderTree and ran the test you suggested.  As expected, I'm not seeing the issue on Mac.  I don't think we'll be able to debug this on our side since it doesn't reproduce.  I'll do some more investigation and come back to provide you with some recommendations on how to debug the root cause of this issue.

[Oliver Hunt]

(In reply to comment #14)
> (In reply to comment #13)
> > What are the qt builds settings?  Which JIT's are enabled? Do you have the LLINT?  Is this 32 and/or 64 bit?
> > 
> > That's information that would be useful to know - are tests failing in run-javascriptore-tests?
> 
> LLINT, JIT and DFG-JIT are enabled on Qt and the problem occurs 
> on 32 and 64 bit too, there are failing jsc and layout tests too
> 
> Here is the 32 bit relase Qt bot: http://build.webkit.org/builders/Qt%20Linux%20Release/builds/61619
> Here is the 32 bit debug Qt bot: http://build.webkit.sed.hu/builders/x86-32%20Linux%20Qt%20Debug/builds/26602
> Here is the 64 bit release Qt bot: http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release/builds/52568
> Here is the 64 bit debug Qt bot: http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Debug/builds/29786
> 
> and you can see these fails on the GTK bots too:
> - http://build.webkit.org/builders/GTK%20Linux%2064-bit%20Debug%20WK1/builds/3240
> - http://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release/builds/39364
> - http://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20WK2%20%28Tests%29/builds/7516

Can you run jsc tests yourself?  If you see a test failing you can search the scroll back and see the command that lead to the failure.

You'll need to run the command in JavaScriptCore/tests/mozilla, run-jsc <list of files> _should_ do the right thing.

If that also crashes, then load up gdb, do attach -w jsc and then run the test that was failing.  GDB should connect, then do a continue and gdb should catch the fault

[Oliver Hunt]

(In reply to comment #15)
> (In reply to comment #11)
> > (In reply to comment #7)
> > > (In reply to comment #6)
> > > > CodeBlock::jitCompile will ASSERT because jitType() returns JITCode::None, which it does becase m_jitCode is nullptr. Trying to let it compile will just result in later ASSERTs because or similar (jitType() == JITCode::InterpreterThunk) asserts later.
> > > 
> > > Thanks.  I don't think it should ever return JITCode::None.  That sounds like something else has gone haywire.  Can you please provide a test case that reproduces this issue?  I've just run run-javascriptcore-tests, and unlike the results that Csaba shared, I'm not seeing any failures at all in my local build (unmodified from trunk).
> > 
> > It happens when running the DumpRenderTree, but apparently only for GTK and Qt. For instance WebKitBuild/Release/bin/DumpRenderTree LayoutTest/fast/css/001.html will assert as above.
> > 
> > It seems to happen most frequently during clean-up phases when the execution-context is recreated, but there are a number of different triggers above the common backtrace I posted earlier.
> 
> I've just finished my build of DumpRenderTree and ran the test you suggested.  As expected, I'm not seeing the issue on Mac.  I don't think we'll be able to debug this on our side since it doesn't reproduce.  I'll do some more investigation and come back to provide you with some recommendations on how to debug the root cause of this issue.

hmmm, try run-javascriptocre-tests --32-bit 

I had been running them, but i think i see failures now, so maybe one of the last patches broke things somehow?

[Csaba Osztrogonác]

Here is a gdb backtrace for you on Qt 64 bit debug on r153330:

$ gdb --args ../../../../WebKitBuild/Debug/bin/jsc  -s  -f ./ecma/shell.js -f ./ecma/Array/15.4-1.js
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /home/webkitbuildbot/oszi/WebKit/WebKitBuild/Debug/bin/jsc...done.
(gdb) run
Starting program: /home/webkitbuildbot/oszi/WebKit/WebKitBuild/Debug/bin/jsc -s -f ./ecma/shell.js -f ./ecma/Array/15.4-1.js
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffb4309700 (LWP 1862)]
[New Thread 0x7fffb3ae9700 (LWP 1863)]
[New Thread 0x7fffb32e8700 (LWP 1864)]
[New Thread 0x7fffb2ae7700 (LWP 1865)]
[New Thread 0x7fffb22e6700 (LWP 1866)]
[New Thread 0x7fffb1ae5700 (LWP 1867)]
[New Thread 0x7fffb12e4700 (LWP 1868)]
15.4-1 Array Objects
ASSERTION FAILED: jitType() == JITCode::BaselineJIT
/home/webkitbuildbot/oszi/WebKit/Source/JavaScriptCore/bytecode/CodeBlock.h(298) : JSC::CompilationResult JSC::CodeBlock::jitCompile(JSC::ExecState*)
1   0x841769 /home/webkitbuildbot/oszi/WebKit/WebKitBuild/Debug/bin/jsc() [0x841769]
2   0x5fc095 /home/webkitbuildbot/oszi/WebKit/WebKitBuild/Debug/bin/jsc() [0x5fc095]
3   0x6d6a3b /home/webkitbuildbot/oszi/WebKit/WebKitBuild/Debug/bin/jsc() [0x6d6a3b]
4   0x6cd7c3 /home/webkitbuildbot/oszi/WebKit/WebKitBuild/Debug/bin/jsc() [0x6cd7c3]
5   0x6d9ca9 /home/webkitbuildbot/oszi/WebKit/WebKitBuild/Debug/bin/jsc() [0x6d9ca9]

Program received signal SIGSEGV, Segmentation fault.
0x000000000084176e in WTFCrash () at /home/webkitbuildbot/oszi/WebKit/Source/WTF/wtf/Assertions.cpp:339
339         *(int *)(uintptr_t)0xbbadbeef = 0;
(gdb) bt
#0  0x000000000084176e in WTFCrash () at /home/webkitbuildbot/oszi/WebKit/Source/WTF/wtf/Assertions.cpp:339
#1  0x00000000005fc095 in JSC::CodeBlock::jitCompile (this=0x1010aa0, exec=0x7fffb06e41a0) at /home/webkitbuildbot/oszi/WebKit/Source/JavaScriptCore/bytecode/CodeBlock.h:298
#2  0x00000000006d6a3b in JSC::LLInt::jitCompileAndSetHeuristics (codeBlock=0x1010aa0, exec=0x7fffb06e41a0) at /home/webkitbuildbot/oszi/WebKit/Source/JavaScriptCore/llint/LLIntSlowPaths.cpp:290
#3  0x00000000006cd7c3 in JSC::LLInt::llint_replace (exec=0x7fffb06e41a0, pc=0x100ff70) at /home/webkitbuildbot/oszi/WebKit/Source/JavaScriptCore/llint/LLIntSlowPaths.cpp:403
#4  0x00000000006d9ca9 in llint_op_ret ()
#5  0x00007fffffffc9b0 in ?? ()
#6  0x0000000000687843 in JSC::JSStack::installTrapsAfterFrame (this=0x0, frame=0x0) at /home/webkitbuildbot/oszi/WebKit/Source/JavaScriptCore/interpreter/JSStackInlines.h:212
#7  0x00000000006997de in JSC::JITCode::execute (this=0xfdaf00, stack=0xfdb668, callFrame=0x7fffb06e4058, vm=0xfca730) at /home/webkitbuildbot/oszi/WebKit/Source/JavaScriptCore/jit/JITCode.cpp:46
#8  0x00000000006837c7 in JSC::Interpreter::execute (this=0xfdb650, program=0x7ffff7e3fe70, callFrame=0x7ffff7f7f8e0, thisObj=0x7ffff7e7feb0) at /home/webkitbuildbot/oszi/WebKit/Source/JavaScriptCore/interpreter/Interpreter.cpp:856
#9  0x000000000076e009 in JSC::evaluate (exec=0x7ffff7f7f8e0, source=..., thisValue=..., returnedException=0x7fffffffdfb0) at /home/webkitbuildbot/oszi/WebKit/Source/JavaScriptCore/runtime/Completion.cpp:83
#10 0x000000000040ffbc in runWithScripts (globalObject=0x7ffff7f7f870, scripts=..., dump=false) at /home/webkitbuildbot/oszi/WebKit/Source/JavaScriptCore/jsc.cpp:596
#11 0x0000000000410cc7 in jscmain (argc=6, argv=0x7fffffffe278) at /home/webkitbuildbot/oszi/WebKit/Source/JavaScriptCore/jsc.cpp:812
#12 0x000000000040fd98 in main (argc=6, argv=0x7fffffffe278) at /home/webkitbuildbot/oszi/WebKit/Source/JavaScriptCore/jsc.cpp:554

[Csaba Osztrogonác]

+info: If I set JSC_useDFGJIT=0, I got the same assertion.
But if I set JSC_useJIT=0, this test pass.

[Chris Dumez]

For what it's worth we also hit the following assertion on EFL:
crash log for WebProcess (pid <unknown>):
STDOUT: <empty>
STDERR: ASSERTION FAILED: jitType() == JITCode::BaselineJIT
STDERR: /home/chris/devel/WebKit/Source/JavaScriptCore/bytecode/CodeBlock.h(298) : JSC::CompilationResult JSC::CodeBlock::jitCompile(JSC::ExecState*)
STDERR: 1   0x7fd251aa27cf WTFCrash
STDERR: 2   0x7fd251807825 JSC::CodeBlock::jitCompile(JSC::ExecState*)
STDERR: 3   0x7fd251aa1329 JSC::LLInt::jitCompileAndSetHeuristics(JSC::CodeBlock*, JSC::ExecState*)
STDERR: 4   0x7fd251a9866d
STDERR: 5   0x7fd251e7714c

[Zan Dobersek]

As I've reported in #webkit, I get no regressions when running run-javascriptcore-tests when jsc is built with Clang, in debug mode, on the GTK port.

[Oliver Hunt]

Comment on attachment 207460 [details]
Patch

Zan has a fix

[Zan Dobersek]

Created attachment 207492 [details]
Patch

[Zan Dobersek]

Comment on attachment 207492 [details]
Patch

Clearing flags on attachment: 207492

Committed r153354: <http://trac.webkit.org/changeset/153354>

[Zan Dobersek]

All reviewed patches have been landed.  Closing bug.

[Geoffrey Garen]

> It was only my rough but frank opinion about this situation.

Rough yes, frank no.

frank 1 |fraNGk|
adjective
open, honest, and direct

Your bizarre musing about the FTL somehow being "top secret" was not frank.

[Csaba Osztrogonác]

(In reply to comment #26)
> > It was only my rough but frank opinion about this situation.
> 
> Rough yes, frank no.
> 
> frank 1 |fraNGk|
> adjective
> open, honest, and direct

- open - yes
- honest - yes
- direct - yes

honest
adj
1. not given to lying, cheating, stealing, etc.; trustworthy
2. not false or misleading; genuine
 
> Your bizarre musing about the FTL somehow being "top secret" was not frank.
I think you probably misinterpreted my sentence. I've never said that FTL
was kept top secret. But the merge was top secret and done in ~7 minutes:
- https://trac.webkit.org/changeset/153113 - 07/25/13 05:58:12 (CEST)
- https://trac.webkit.org/changeset/153296 - 07/25/13 06:05:36 (CEST)

and then http://webkitmemes.tumblr.com/post/18264800090/cool-developers-dont-look-at-the-build-bots ... when the european developers still slept ...

The missing communication of the merge was so bizarre, not my opinion.

[Geoffrey Garen]

(In reply to comment #27)

Look, I understand that you're skilled at being stubborn, so I'm not going to waste my time arguing with you. I'm just informing you: Your bad attitude is doing harm to your relationship with JSC developers, and the likely result will be a low-quality JS engine for Qt.

[Csaba Osztrogonác]

(In reply to comment #28)
> (In reply to comment #27)
> 
> Look, I understand that you're skilled at being stubborn, so I'm not going to waste my time arguing with you. I'm just informing you: Your bad attitude is doing harm to your relationship with JSC developers, and the likely result will be a low-quality JS engine for Qt.

I really don't understand this threat. :o I only criticezed the _missing_ communication/announcment of the _merge_. (not the FTL development) I didn't lie, I didn't indulge in personalities. But I and other developers landed many
buildfixes and then I _asked_ JSC developers to fix the regressions and said
"Thanks for your cooperation in advance." And generated backtrace in https://bugs.webkit.org/show_bug.cgi?id=119140 to help debugging. And I
was online at night in my free time and try to help Oliver figuring out
what happened. And tried to build QtWebKit with clang to check if it is
a gcc related or not (unfortunately without success). So please don't
consider me a hostile person.

And punishing Qt with low quality JS engine because of my opinion
would be a non-sense. I'm not the Qt port and Qt port isn't me and I didn't
work on Qt port at all since Digia acquired it last year. Apart from this, 
I try to help the community and sometimes fix GTK/EFL/Qt builds and cross
platform issues too on trunk. (Of course I referenced to Qt results in
bug119140, because only Qt port has public 32 bit x86 and ARM tester
bots on WebKit trunk.)