Bug 56553 – run-javascriptcore-tests almost always fails on Windows 7 SP1 due to some tests exiting with exit code 126

• Component: Tools / Tests
• Status: RESOLVED INVALID

[Adam Roben (:aroben)]

run-javascriptcore-tests is failing on Windows WebKit2 bots. See the URL for an example.

These tests aren't failing on any other bots. We should figure out what's different about these bots.

[Sam Weinig]

<rdar://problem/9149165>

[Adam Roben (:aroben)]

This is now the only thing keeping the Windows WebKit2 bots from being green!

[Adam Roben (:aroben)]

Geoff Garen helped me to discover that, at least in one case, one of the "failing" tests was actually crashing. I'm going to see if I can reproduce the crash in a debugger on one of the buildslaves.

[Adam Roben (:aroben)]

I can reproduce the failure by running run-javascriptcore-tests on the buildslave. I haven't yet been able to repro a crash, though.

[Adam Roben (:aroben)]

Created attachment 86236 [details]
Assertion failure in Heap::destroy

I wasn't able to get a crash in a Release build on the buildslave. But I was able to get an assertion failure in a Debug build on the buildslave.

[Adam Roben (:aroben)]

The assertion failure is the same one from bug 52156.

[Adam Roben (:aroben)]

You can see, for example, from <http://build.webkit.org/builders/Windows%207%20Release%20%28WebKit2%20Tests%29/builds/4322/steps/jscore-test/logs/actual.html>:

> Testcase ecma_2/LexicalConventions/regexp-literals-001.js failed 
> [ Previous Failure | Next Failure | Top of Page ]
> Expected exit code 0, got 126

Presumably this means the test crashed.

[Adam Roben (:aroben)]

Here's the backtrace of the assertion failure:

>	JavaScriptCore.dll!JSC::Heap::destroy()  Line 71 + 0x30 bytes	C++
 	jsc.exe!cleanupGlobalData(JSC::JSGlobalData * globalData=0x0299cd28)  Line 370	C++
 	jsc.exe!main(int argc=6, char * * argv=0x0299af60)  Line 362 + 0x9 bytes	C++
 	jsc.exe!__tmainCRTStartup()  Line 597 + 0x17 bytes	C
 	kernel32.dll!_BaseProcessStart@4()  + 0x23 bytes

[Adam Roben (:aroben)]

Some info from IRC:

<ggaren> aroben: what is the value of m_operationInProgress?
<aroben> ggaren: NoOperation
<ggaren> aroben: what are m_globalData->m_refCount and m_globalData->dynamicGlobalObject?
<aroben> ggaren: 2 and 0x022a0128
<ggaren> aroben: and is m_globalData->m_deletionHasBegun false?
<aroben> ggaren: yes
<ggaren> aroben: inside main, what is the value of res?
<aroben> ggaren: 3
<ggaren> aroben: i have a guess that res was set to 3 by the TRY / EXCEPT clause. if so, we've caught the bug too late.
<aroben> ggaren: makes sense
<aroben> ggaren: I can remove the try/except
<aroben> ggaren: I see a comment that say try/except is only there to block the normal crash report mechanisms
<aroben> ggaren: seems like eventually we'd like to save crash logs instead of just using a magic 3 return code
<ggaren> aroben: inside jsc, all access (save 1 irrelevant place) to dynamicGlobalObject is managed by an RAII object: DynamicGlobalObjectScope. so, now i'm even more sure of my guess.

[Adam Roben (:aroben)]

Removing the try/except resulted in a CRASH() with the following backtrace:


>	JavaScriptCore.dll!WTF::fastRealloc(void * p=0x7eb00040, unsigned int n=14785173)  Line 371 + 0x5 bytes	C++
 	JavaScriptCore.dll!JSC::AssemblerBuffer::grow(int extraCapacity=0)  Line 175 + 0x13 bytes	C++
 	JavaScriptCore.dll!JSC::AssemblerBuffer::ensureSpace(int space=16)  Line 61	C++
 	JavaScriptCore.dll!JSC::X86Assembler::X86InstructionFormatter::oneByteOp(JSC::X86Assembler::OneByteOpcodeID opcode=OP_CALL_rel32)  Line 1692	C++
 	JavaScriptCore.dll!JSC::X86Assembler::call()  Line 1221	C++
 	JavaScriptCore.dll!JSC::MacroAssemblerX86::call()  Line 125 + 0xe bytes	C++
 	JavaScriptCore.dll!JSC::JITStubCall::call()  Line 176	C++
 	JavaScriptCore.dll!JSC::JITStubCall::call(unsigned int dst=1)  Line 196	C++
 	JavaScriptCore.dll!JSC::JIT::emit_op_resolve_with_base(JSC::Instruction * currentInstruction=0x7bc495f0)  Line 1231	C++
 	JavaScriptCore.dll!JSC::JIT::privateCompileMainPass()  Line 300 + 0xc bytes	C++
 	JavaScriptCore.dll!JSC::JIT::privateCompile(JSC::MacroAssemblerCodePtr * functionEntryArityCheck=0x00000000)  Line 484	C++
 	JavaScriptCore.dll!JSC::JIT::compile(JSC::JSGlobalData * globalData=0x0299cd28, JSC::CodeBlock * codeBlock=0x57fb19c0, JSC::MacroAssemblerCodePtr * functionEntryArityCheck=0x00000000, void * offsetBase=0x00000000)  Line 183 + 0x26 bytes	C++
 	JavaScriptCore.dll!JSC::EvalExecutable::compileInternal(JSC::ExecState * exec=0x02b20038, JSC::ScopeChainNode * scopeChainNode=0x028c0128)  Line 121 + 0x20 bytes	C++
 	JavaScriptCore.dll!JSC::EvalExecutable::compile(JSC::ExecState * exec=0x02b20038, JSC::ScopeChainNode * scopeChainNode=0x028c0128)  Line 221 + 0x10 bytes	C++
 	JavaScriptCore.dll!JSC::EvalCodeCache::get(JSC::ExecState * exec=0x02b20038, JSC::ScriptExecutable * owner=0x03420180, bool inStrictContext=false, const JSC::UString & evalSource={f('0')
f('1')
f('2')
f('3')
f('4')
f('5')
f('6')
f('7')
f('8')
f('9')
f('10')
f('11')
f('12')
f('13')
f('14')
f('15')
f('16')
f('17')
f('18')
f('19')
f('20')
f('21')
f('22')
f('23')
f('24')
f('25')
f('26')
f('27')
f('28')
f('29')
f('30')
f('31')
f('32')
f('33')
f('34')
f('35')
f('36')
f('37')
f('38')
f('39')
f('40')
f('41')
f('42')
f('43')
f('44')
f('45')
f('46')
f('47')
f('48')
f('49')
f('50'), JSC::ScopeChainNode * scopeChain=0x028c0128, JSC::JSValue & exceptionValue={...})  Line 57 + 0x10 bytes	C++
 	JavaScriptCore.dll!JSC::Interpreter::callEval(JSC::ExecState * callFrame=0x02b20038, JSC::RegisterFile * registerFile=0x02ac8fcc, JSC::Register * argv=0x02b20040, int argc=2, int registerOffset=9)  Line 412 + 0x34 bytes	C++
 	JavaScriptCore.dll!cti_op_call_eval(void * * args=0x0012fc98)  Line 3118	C++
 	JavaScriptCore.dll!@cti_op_create_this@4()  + 0x1df bytes	C++
 	JavaScriptCore.dll!JSC::JITCode::execute(JSC::RegisterFile * registerFile=0x02ac8fcc, JSC::ExecState * callFrame=0x02b20038, JSC::JSGlobalData * globalData=0x0299cd28)  Line 77 + 0x22 bytes	C++
 	JavaScriptCore.dll!JSC::Interpreter::execute(JSC::ProgramExecutable * program=0x03420180, JSC::ExecState * callFrame=0x022a01a0, JSC::ScopeChainNode * scopeChain=0x028c0128, JSC::JSObject * thisObj=0x022a0128)  Line 773 + 0x25 bytes	C++
 	JavaScriptCore.dll!JSC::evaluate(JSC::ExecState * exec=0x022a01a0, JSC::ScopeChainNode * scopeChain=0x028c0128, const JSC::SourceCode & source={...}, JSC::JSValue thisValue={...})  Line 69	C++
 	jsc.exe!runWithScripts(GlobalObject * globalObject=0x022a0128, const WTF::Vector<Script,0> & scripts=[2]({isFile=true argument=0x0299afca "./js1_5/shell.js" },{isFile=true argument=0x0299afde "./js1_5/Regress/regress-159334.js" }), bool dump=false)  Line 402 + 0x52 bytes	C++
 	jsc.exe!jscmain(int argc=6, char * * argv=0x0299af60, JSC::JSGlobalData * globalData=0x0299cd28)  Line 540 + 0x11 bytes	C++
 	jsc.exe!main(int argc=6, char * * argv=0x0299af60)  Line 359 + 0x11 bytes	C++
 	jsc.exe!__tmainCRTStartup()  Line 597 + 0x17 bytes	C
 	kernel32.dll!_BaseProcessStart@4()  + 0x23 bytes

[Adam Roben (:aroben)]

The test that causes the crash is js1_5/Regress/regress-159334.js.

[Adam Roben (:aroben)]

The crashing call is in AssemblerBuffer::grow:

    m_buffer = static_cast<char*>(fastRealloc(m_buffer, m_capacity));

m_capacity is 14785173.

[Adam Roben (:aroben)]

In EvalCodeCache::get, evalSource's length is 1461754.

[Adam Roben (:aroben)]

I should note that I have full page heap enabled (which is basically the Windows equivalent of GuardMalloc).

[Adam Roben (:aroben)]

::GetLastError() says:

0x00000008 Not enough storage is available to process this command.

[Adam Roben (:aroben)]

Turning off full page heap makes the crash go away. Presumably the extra memory incurred by full page heap is causing the allocation to fail. Maybe this has all been a red herring.

[Adam Roben (:aroben)]

(In reply to comment #7)
> You can see, for example, from <http://build.webkit.org/builders/Windows%207%20Release%20%28WebKit2%20Tests%29/builds/4322/steps/jscore-test/logs/actual.html>:
> 
> > Testcase ecma_2/LexicalConventions/regexp-literals-001.js failed 
> > [ Previous Failure | Next Failure | Top of Page ]
> > Expected exit code 0, got 126
> 
> Presumably this means the test crashed.

Google seems to indicate that 126 means "specified module could not be found".

[Adam Roben (:aroben)]

Every failure I've seen on the bots recently has been due to exit code 126.

[Adam Roben (:aroben)]

It turns out this failure is specific to Windows 7 SP1. Windows XP, and Windows 7 without SP1 do not have this bug.

[Steve Falkenburg]

I don't see any crash dumps from jsc.exe on one of the failing bots, so I don't think this is manifesting as a crash. (Which corroborates what Adam indicated above about the crash he was seeing being caused by out of memory from full page heap).

[Csaba Osztrogonác]

There is no Windows WebKit2 port, and Windows bots are happy now.