Bug 56553 - run-javascriptcore-tests almost always fails on Windows 7 SP1 due to some tests exiting with exit code 126
Summary: run-javascriptcore-tests almost always fails on Windows 7 SP1 due to some tes...
Status: RESOLVED INVALID
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: PC Windows 7
: P2 Normal
Assignee: Nobody
URL: http://build.webkit.org/builders/Wind...
Keywords: InRadar, MakingBotsRed, PlatformOnly, ToolsHitList
Depends on:
Blocks: 56832
  Show dependency treegraph
 
Reported: 2011-03-17 01:48 PDT by Adam Roben (:aroben)
Modified: 2015-03-04 02:42 PST (History)
4 users (show)

See Also:


Attachments
Assertion failure in Heap::destroy (9.53 KB, text/plain)
2011-03-18 16:02 PDT, Adam Roben (:aroben)
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Adam Roben (:aroben) 2011-03-17 01:48:48 PDT
run-javascriptcore-tests is failing on Windows WebKit2 bots. See the URL for an example.

These tests aren't failing on any other bots. We should figure out what's different about these bots.
Comment 1 Sam Weinig 2011-03-17 10:42:40 PDT
<rdar://problem/9149165>
Comment 2 Adam Roben (:aroben) 2011-03-18 14:22:56 PDT
This is now the only thing keeping the Windows WebKit2 bots from being green!
Comment 3 Adam Roben (:aroben) 2011-03-18 14:33:48 PDT
Geoff Garen helped me to discover that, at least in one case, one of the "failing" tests was actually crashing. I'm going to see if I can reproduce the crash in a debugger on one of the buildslaves.
Comment 4 Adam Roben (:aroben) 2011-03-18 15:13:55 PDT
I can reproduce the failure by running run-javascriptcore-tests on the buildslave. I haven't yet been able to repro a crash, though.
Comment 5 Adam Roben (:aroben) 2011-03-18 16:02:16 PDT
Created attachment 86236 [details]
Assertion failure in Heap::destroy

I wasn't able to get a crash in a Release build on the buildslave. But I was able to get an assertion failure in a Debug build on the buildslave.
Comment 6 Adam Roben (:aroben) 2011-03-21 11:30:49 PDT
The assertion failure is the same one from bug 52156.
Comment 7 Adam Roben (:aroben) 2011-03-21 12:49:56 PDT
You can see, for example, from <http://build.webkit.org/builders/Windows%207%20Release%20%28WebKit2%20Tests%29/builds/4322/steps/jscore-test/logs/actual.html>:

> Testcase ecma_2/LexicalConventions/regexp-literals-001.js failed 
> [ Previous Failure | Next Failure | Top of Page ]
> Expected exit code 0, got 126

Presumably this means the test crashed.
Comment 8 Adam Roben (:aroben) 2011-03-21 14:11:12 PDT
Here's the backtrace of the assertion failure:

>	JavaScriptCore.dll!JSC::Heap::destroy()  Line 71 + 0x30 bytes	C++
 	jsc.exe!cleanupGlobalData(JSC::JSGlobalData * globalData=0x0299cd28)  Line 370	C++
 	jsc.exe!main(int argc=6, char * * argv=0x0299af60)  Line 362 + 0x9 bytes	C++
 	jsc.exe!__tmainCRTStartup()  Line 597 + 0x17 bytes	C
 	kernel32.dll!_BaseProcessStart@4()  + 0x23 bytes
Comment 9 Adam Roben (:aroben) 2011-03-21 14:18:33 PDT
Some info from IRC:

<ggaren> aroben: what is the value of m_operationInProgress?
<aroben> ggaren: NoOperation
<ggaren> aroben: what are m_globalData->m_refCount and m_globalData->dynamicGlobalObject?
<aroben> ggaren: 2 and 0x022a0128
<ggaren> aroben: and is m_globalData->m_deletionHasBegun false?
<aroben> ggaren: yes
<ggaren> aroben: inside main, what is the value of res?
<aroben> ggaren: 3
<ggaren> aroben: i have a guess that res was set to 3 by the TRY / EXCEPT clause. if so, we've caught the bug too late.
<aroben> ggaren: makes sense
<aroben> ggaren: I can remove the try/except
<aroben> ggaren: I see a comment that say try/except is only there to block the normal crash report mechanisms
<aroben> ggaren: seems like eventually we'd like to save crash logs instead of just using a magic 3 return code
<ggaren> aroben: inside jsc, all access (save 1 irrelevant place) to dynamicGlobalObject is managed by an RAII object: DynamicGlobalObjectScope. so, now i'm even more sure of my guess.
Comment 10 Adam Roben (:aroben) 2011-03-21 14:25:15 PDT
Removing the try/except resulted in a CRASH() with the following backtrace:


>	JavaScriptCore.dll!WTF::fastRealloc(void * p=0x7eb00040, unsigned int n=14785173)  Line 371 + 0x5 bytes	C++
 	JavaScriptCore.dll!JSC::AssemblerBuffer::grow(int extraCapacity=0)  Line 175 + 0x13 bytes	C++
 	JavaScriptCore.dll!JSC::AssemblerBuffer::ensureSpace(int space=16)  Line 61	C++
 	JavaScriptCore.dll!JSC::X86Assembler::X86InstructionFormatter::oneByteOp(JSC::X86Assembler::OneByteOpcodeID opcode=OP_CALL_rel32)  Line 1692	C++
 	JavaScriptCore.dll!JSC::X86Assembler::call()  Line 1221	C++
 	JavaScriptCore.dll!JSC::MacroAssemblerX86::call()  Line 125 + 0xe bytes	C++
 	JavaScriptCore.dll!JSC::JITStubCall::call()  Line 176	C++
 	JavaScriptCore.dll!JSC::JITStubCall::call(unsigned int dst=1)  Line 196	C++
 	JavaScriptCore.dll!JSC::JIT::emit_op_resolve_with_base(JSC::Instruction * currentInstruction=0x7bc495f0)  Line 1231	C++
 	JavaScriptCore.dll!JSC::JIT::privateCompileMainPass()  Line 300 + 0xc bytes	C++
 	JavaScriptCore.dll!JSC::JIT::privateCompile(JSC::MacroAssemblerCodePtr * functionEntryArityCheck=0x00000000)  Line 484	C++
 	JavaScriptCore.dll!JSC::JIT::compile(JSC::JSGlobalData * globalData=0x0299cd28, JSC::CodeBlock * codeBlock=0x57fb19c0, JSC::MacroAssemblerCodePtr * functionEntryArityCheck=0x00000000, void * offsetBase=0x00000000)  Line 183 + 0x26 bytes	C++
 	JavaScriptCore.dll!JSC::EvalExecutable::compileInternal(JSC::ExecState * exec=0x02b20038, JSC::ScopeChainNode * scopeChainNode=0x028c0128)  Line 121 + 0x20 bytes	C++
 	JavaScriptCore.dll!JSC::EvalExecutable::compile(JSC::ExecState * exec=0x02b20038, JSC::ScopeChainNode * scopeChainNode=0x028c0128)  Line 221 + 0x10 bytes	C++
 	JavaScriptCore.dll!JSC::EvalCodeCache::get(JSC::ExecState * exec=0x02b20038, JSC::ScriptExecutable * owner=0x03420180, bool inStrictContext=false, const JSC::UString & evalSource={f('0')
f('1')
f('2')
f('3')
f('4')
f('5')
f('6')
f('7')
f('8')
f('9')
f('10')
f('11')
f('12')
f('13')
f('14')
f('15')
f('16')
f('17')
f('18')
f('19')
f('20')
f('21')
f('22')
f('23')
f('24')
f('25')
f('26')
f('27')
f('28')
f('29')
f('30')
f('31')
f('32')
f('33')
f('34')
f('35')
f('36')
f('37')
f('38')
f('39')
f('40')
f('41')
f('42')
f('43')
f('44')
f('45')
f('46')
f('47')
f('48')
f('49')
f('50'), JSC::ScopeChainNode * scopeChain=0x028c0128, JSC::JSValue & exceptionValue={...})  Line 57 + 0x10 bytes	C++
 	JavaScriptCore.dll!JSC::Interpreter::callEval(JSC::ExecState * callFrame=0x02b20038, JSC::RegisterFile * registerFile=0x02ac8fcc, JSC::Register * argv=0x02b20040, int argc=2, int registerOffset=9)  Line 412 + 0x34 bytes	C++
 	JavaScriptCore.dll!cti_op_call_eval(void * * args=0x0012fc98)  Line 3118	C++
 	JavaScriptCore.dll!@cti_op_create_this@4()  + 0x1df bytes	C++
 	JavaScriptCore.dll!JSC::JITCode::execute(JSC::RegisterFile * registerFile=0x02ac8fcc, JSC::ExecState * callFrame=0x02b20038, JSC::JSGlobalData * globalData=0x0299cd28)  Line 77 + 0x22 bytes	C++
 	JavaScriptCore.dll!JSC::Interpreter::execute(JSC::ProgramExecutable * program=0x03420180, JSC::ExecState * callFrame=0x022a01a0, JSC::ScopeChainNode * scopeChain=0x028c0128, JSC::JSObject * thisObj=0x022a0128)  Line 773 + 0x25 bytes	C++
 	JavaScriptCore.dll!JSC::evaluate(JSC::ExecState * exec=0x022a01a0, JSC::ScopeChainNode * scopeChain=0x028c0128, const JSC::SourceCode & source={...}, JSC::JSValue thisValue={...})  Line 69	C++
 	jsc.exe!runWithScripts(GlobalObject * globalObject=0x022a0128, const WTF::Vector<Script,0> & scripts=[2]({isFile=true argument=0x0299afca "./js1_5/shell.js" },{isFile=true argument=0x0299afde "./js1_5/Regress/regress-159334.js" }), bool dump=false)  Line 402 + 0x52 bytes	C++
 	jsc.exe!jscmain(int argc=6, char * * argv=0x0299af60, JSC::JSGlobalData * globalData=0x0299cd28)  Line 540 + 0x11 bytes	C++
 	jsc.exe!main(int argc=6, char * * argv=0x0299af60)  Line 359 + 0x11 bytes	C++
 	jsc.exe!__tmainCRTStartup()  Line 597 + 0x17 bytes	C
 	kernel32.dll!_BaseProcessStart@4()  + 0x23 bytes
Comment 11 Adam Roben (:aroben) 2011-03-21 14:26:10 PDT
The test that causes the crash is js1_5/Regress/regress-159334.js.
Comment 12 Adam Roben (:aroben) 2011-03-21 14:27:23 PDT
The crashing call is in AssemblerBuffer::grow:

    m_buffer = static_cast<char*>(fastRealloc(m_buffer, m_capacity));

m_capacity is 14785173.
Comment 13 Adam Roben (:aroben) 2011-03-21 14:28:57 PDT
In EvalCodeCache::get, evalSource's length is 1461754.
Comment 14 Adam Roben (:aroben) 2011-03-21 14:30:23 PDT
I should note that I have full page heap enabled (which is basically the Windows equivalent of GuardMalloc).
Comment 15 Adam Roben (:aroben) 2011-03-21 14:31:16 PDT
::GetLastError() says:

0x00000008 Not enough storage is available to process this command.
Comment 16 Adam Roben (:aroben) 2011-03-21 14:36:34 PDT
Turning off full page heap makes the crash go away. Presumably the extra memory incurred by full page heap is causing the allocation to fail. Maybe this has all been a red herring.
Comment 17 Adam Roben (:aroben) 2011-03-21 14:41:03 PDT
(In reply to comment #7)
> You can see, for example, from <http://build.webkit.org/builders/Windows%207%20Release%20%28WebKit2%20Tests%29/builds/4322/steps/jscore-test/logs/actual.html>:
> 
> > Testcase ecma_2/LexicalConventions/regexp-literals-001.js failed 
> > [ Previous Failure | Next Failure | Top of Page ]
> > Expected exit code 0, got 126
> 
> Presumably this means the test crashed.

Google seems to indicate that 126 means "specified module could not be found".
Comment 18 Adam Roben (:aroben) 2011-03-24 11:55:01 PDT
Every failure I've seen on the bots recently has been due to exit code 126.
Comment 19 Adam Roben (:aroben) 2011-04-19 10:59:24 PDT
It turns out this failure is specific to Windows 7 SP1. Windows XP, and Windows 7 without SP1 do not have this bug.
Comment 20 Steve Falkenburg 2011-04-19 14:06:51 PDT
I don't see any crash dumps from jsc.exe on one of the failing bots, so I don't think this is manifesting as a crash. (Which corroborates what Adam indicated above about the crash he was seeing being caused by out of memory from full page heap).
Comment 21 Csaba Osztrogonác 2015-03-04 02:42:54 PST
There is no Windows WebKit2 port, and Windows bots are happy now.