Bug 30835 – REGRESSION (??? - r50171): inspector tests crashing at JSC::TypeInfo::type()

• Status: RESOLVED
• Resolution: WORKSFORME
• Component: Web Inspector (Deprecated)

[Eric Seidel (no email)]

inspector/console-format.html crashed on the Leopard Debug Bot

http://build.webkit.org/results/Leopard%20Intel%20Debug%20(Tests)/r50171%20(6519)/results.html

Unfortunately I don't have a crash log for you.  I've only seen it happen once so far.

[Eric Seidel (no email)]

I wonder if this could be related to the crash just seen on the Tiger bot in inspector/console- tests:
http://build.webkit.org/results/Tiger%20Intel%20Release/r50171%20(5612)/results.html

[Eric Seidel (no email)]

Looks like it's crashing again this morning.  This is a real bug:
http://build.webkit.org/results/Leopard%20Intel%20Debug%20(Tests)/r50217%20(6552)/results.html

[Timothy Hatcher]

Where can we get the crash logs? Why isn't the stderr output file a 404? (That would help, since this is likely an ASSERT.)

[Timothy Hatcher]

"why is"

[Eric Seidel (no email)]

We don't have an easy way to get crash logs off the bots yet. :(  bug 14861.

[Eric Seidel (no email)]

I expect that
run-webkit-tests --iterations 100 inspector/console-format.html
might reproduce it in a local debug build.  Since this crash seems flakey, it doesn't crash every time.

[Eric Seidel (no email)]

This console failure (internal timeout):
http://build.webkit.org/results/Leopard%20Intel%20Debug%20(Tests)/r50239%20(6569)/inspector/console-format-collections-pretty-diff.html
may also be related.

I ran the inspector tests 100 times (--iterations 100) locally in debug mode and saw no crashes.

[Eric Seidel (no email)]

My --iterations 100 run was with a rather old build of WebKit.  I'm going to update and run the tests again since these failures only recently started on the leopard bot.

Sadly inspector/ is currently the most flakey set of tests on the leopard debug bot. :(

[Pavel Feldman]

There are two things that we can try:
1) Change DRT so that inspector is enabled for LayoutTests/inspectors tests
2) Replace setTimeout(0) with the direct call in tests.

[Eric Seidel (no email)]

I worry this might be an http-induced crasher leaking into the inspector tests.  I remember we've had some flakiness with XHR tests in the past causing random crashes.  I'll dig up the bugs.

I'm right now running:
run-webkit-tests --iterations 100 http inspector
to see if this could be related to being run after the http tests.

[Eric Seidel (no email)]

bug 29344, bug 30726, bug 30519, bug 30392 and bug 29090 are all about flakey http tests, some of which involve unexplained crashers.  It is possible the inspector tests are just the most recent victim here.

[Eric Seidel (no email)]

Just had console-dirxml.html fail on the Tiger bot:
http://build.webkit.org/results/Tiger%20Intel%20Release/r50240%20(5657)/results.html
Clearly something is wrong here that's causing flakey inspector/ tests on multiple bots. :(

[Eric Seidel (no email)]

Created attachment 42064 [details]
crash report from running "run-webkit-tests --iterations 100 http inspector"

I guess I'll CC some of the JSC guys.

[Eric Seidel (no email)]

CCing a couple JIT guys in case this crash dump looks familiar to them.

[Eric Seidel (no email)]

Created attachment 42082 [details]
13 crash reports from another run of  "run-webkit-tests --iterations 100 http inspector"

I let the run complete.  Here are 13 crash reports from the run.  Obviously this bug is reproducible . :)  Now I guess we just need a reduction...

[Eric Seidel (no email)]

The 13 reports don't all look the same, but I think there are only two separate stack traces.

[Eric Seidel (no email)]

I'm currently trying to reduce the number of tests required to cause this to fail.  Right now I have the set down to about 140.  I'll post the list when I have it down to a more reasonable number.  Right now that 140 is a subset of the http tests and all of the inspector tests.

[Eric Seidel (no email)]

If I'm correctly reading the stack traces correctly, it looks like something is trying to toString() a bad JSValue pointer?  Is that a correct reading?

[Pavel Feldman]

There has been a bug where quarantine wrappers were not holding wrapped objects and those were collected on the go. That was causing random crap to take place including the one you describe. Quarantine code seemed to be right though - at least it had appropriate mark methods. Could we run this with GC disabled? Or do some stessful GC on inspector tests only? Quarantined objects are only used in inspector and should soon go away.

[Eric Seidel (no email)]

Created attachment 42147 [details]
180 tests when run which are known to crash.

I'm still trying to reduce this set.

cat known_to_crash.txt | xargs run-webkit-tests --iterations 100

will lead to the crash.

[Eric Seidel (no email)]

When this crashes, it seems to crash on the very first inspector test.  It's possible that a GC is triggered during an inspector test and that's the reason why it's crashing.  I guess I could try sprinkling gc() calls in inspector/console-dir.html and see what happens.

[Eric Seidel (no email)]

Alexey suggested I try using COLLECT_ON_EVERY_ALLOCATION from Collector.cpp.

I built a copy of WebKit with that, and ran all of the inspector/*.html tests under DumpRenderTree.  I was not able to produce a crash.

I wonder if one of the http tests is smashing memory in some way?  It's strange that all of the crashes seem to have very similar crash points:

0x00000000fffffff0
0x0000000000000001
0x0000000000000fe4
0x0000000000000002
0x00000000fffffff6

Do these values look familiar to anyone in JIT land?  The crash point is always:
0   com.apple.JavaScriptCore      	0x0052bb81 JSC::TypeInfo::type() const + 9 (JSTypeInfo.h:60)

[Eric Seidel (no email)]

Why do all of those crash points only have the low 8 bytes set?

[Oliver Hunt]

Whether either getting passed in bogus values or (and this seems more likely) we're truncating that tag bits from an jsvalue.  It would be good to see if we can find exactly what revision this started at.

[Oliver Hunt]

Eric are you running on leopard or snowleopard?

[Eric Seidel (no email)]

I'm running Leopard.  So are the bots we've seen this crash on.  I believe I've only ever seen this crash on Leopard Debug, although it's possible it crashes on other configurations.

[Oliver Hunt]

As yet i have been unable to repro -- if you can get a narrower revision range that would be great

[David Levin]

Created attachment 42230 [details]
crash log

[Pavel Feldman]

(In reply to comment #27)
> As yet i have been unable to repro -- if you can get a narrower revision range
> that would be great

The tests (as well as the testing harness) for inspector were introduced not so long ago, so I think it might be hard to narrow the revision range. It might have been there for a while.

[Oliver Hunt]

I've added a myriad of assertions but have yet to hit anything prior to the actual crash.  It's really bizarre.  I may start adding assertions looking for these specific bad values.

[Eric Seidel (no email)]

This seems to be less common on the bots today, but is not gone.  I suspect that xmlhttprequest tests were added and thus changed what objects were live at the time gc() ended up being called during the crashing console tests.

[Mark Rowe (bdash)]

<rdar://problem/7363589>

[Eric Seidel (no email)]

inspector/console-format-collections.html is crashing consistently no the Leopard Debug Test Bot this evening.  I assume it's just this bug.  I assume that the stars aligned with the addition of some new test such that the gc timing is correct to trigger this bug more frequently again.  Or at least that my (uninformed) theory. :)

[Pavel Feldman]

Have not seen Leopard bots failing since I queued things carefully in https://bugs.webkit.org/show_bug.cgi?id=30884. Or was it failing?

[Eric Seidel (no email)]

I haven't seen the bots crash due to this in a while either.  But I haven't been paying super-close attention.  If it's fixed, do we have any idea what could have fixed it?

[Pavel Feldman]

I've queued all the interaction between the inspected page and frontend more carefully. In particular this excluded re-enterability from withing the timer fire.

[Pavel Feldman]

Created attachment 43152 [details]
another crash log on r50935

[Eric Seidel (no email)]

Created attachment 43183 [details]
crash report from console-dir.html when trying to land bug 31474

https://bugs.webkit.org/show_bug.cgi?id=31474#c5 saw this crash again.
There have been a rash of GC-related crashes the last few days, so this may not be related to this particular bug, but is the same test.  bug 31460 is one example of the other JSC crashes seen in the last 48 hours.

[Eric Seidel (no email)]

Created attachment 43186 [details]
Crash report from console-dir.html when trying to land bug 31456

[Eric Seidel (no email)]

Created attachment 43188 [details]
Another crash report from console-dir.html when trying to land bug 31406

[Eric Seidel (no email)]

I'm not sure that:
https://bugs.webkit.org/attachment.cgi?id=43183
https://bugs.webkit.org/attachment.cgi?id=43186
https://bugs.webkit.org/attachment.cgi?id=43188
Are actually related to the original bug in question.  They just happen to be console-dir.html crashes of the last couple days.  They may be of a different origin.

[Eric Seidel (no email)]

Looks like we're seeing console-dir.html crashes on the build bots too:
http://build.webkit.org/results/Leopard%20Intel%20Release%20(Tests)/r50956%20(7276)/results.html
http://build.webkit.org/results/Leopard%20Intel%20Release%20(Tests)/r50933%20(7258)/results.html

[Eric Seidel (no email)]

Attempting to reduce the set of tests required to produce a crash.

[Eric Seidel (no email)]

I've plugged the set of test cases into the automated test minimizer "tmin": http://code.google.com/p/tmin/wiki/TminManual and we're just gonna hope. :)

[Eric Seidel (no email)]

Created attachment 43383 [details]
27 tests which are known to crash when run together

cat known_to_crash.txt | xargs run-webkit-tests --iterations 10 --no-launch-safari --debug

is the command I'm using.

[Eric Seidel (no email)]

OK.  I've reduced it to 4 tests required to produce the crash:

http/tests/xmlhttprequest/workers/shared-worker-close.html http/tests/xmlhttprequest/workers/shared-worker-methods.html
inspector/console-dir.html
inspector/console-format.html

This command:
run-webkit-tests --debug --iterations 20 --no-launch-safari http/tests/xmlhttprequest/workers/shared-worker-close.html http/tests/xmlhttprequest/workers/shared-worker-methods.html inspector/console-dir.html inspector/console-format.html

Reliably produces a crash for me.

Looking now to see if I can condense this down into a single test case.

[Eric Seidel (no email)]

Looks like this is caused by Shared Workers + gc()

This command crashes reliably for me:
run-webkit-tests --debug --iterations 100 --no-launch-safari  http/tests/xmlhttprequest/workers/shared-worker-methods.html

I'll see if I can reduce that single test further.  Although at this point I would expect one of the JSC experts should be able to give some theories as to what's going wrong here. :)

[Dmitry Titov]

I think I have a fix for this. Will create a patch later today.

This particular test (xhr in shared workers) fails because of this change: http://trac.webkit.org/changeset/50919 (landed 11/12)

[Oliver Hunt]

(In reply to comment #47)
> Looks like this is caused by Shared Workers + gc()
> 
> This command crashes reliably for me:
> run-webkit-tests --debug --iterations 100 --no-launch-safari 
> http/tests/xmlhttprequest/workers/shared-worker-methods.html
> 
> I'll see if I can reduce that single test further.  Although at this point I
> would expect one of the JSC experts should be able to give some theories as to
> what's going wrong here. :)

Based on that i think the crash we're currently looking at is a different issue from the one this bug refers to (the revision you refer to is after the date this bug was filed)

[Dmitry Titov]

Perhaps there are more then one cause. One of them is bug 31615. Lets see what remains after that one will land.

[Dmitry Titov]

With patch for bug 31615 applied, this command does not crash (before it did):
run-webkit-tests --iterations 1000 --no-launch-safari http/tests/xmlhttprequest/workers/shared-worker-methods.html

[Eric Seidel (no email)]

Inspired by the diagnosis made in bug 31615 I looked back through the changes just before r50171 again.  I wonder if http://trac.webkit.org/changeset/50167 could be related to this at all?  XHRs are used from multiple threads, no?  Is it safe to call those inspector methods from XHR?

[Pavel Feldman]

Timeline can only receive events on main thread. I can see timeline being called from within callReadyStateChangeListener only. This one is presumably dispatching events on the main thread for Document context. Workers' contexts should have no timeline agent instances due to the logic in InspectorTimelineAgent::retrieve. So things should be ok, unless marshalling of these events from XHR to main thread is happening later.

[Adam Roben (:aroben)]

*** Bug 31999 has been marked as a duplicate of this bug. ***

[Timothy Hatcher]

Does not seem to be there anymore.