Bug 37342 – new-run-webkit-tests no longer prints out stack traces

• Status: RESOLVED
• Resolution: WORKSFORME
• Component: Tools / Tests

[Ojan Vafai]

On Linux, it doesn't print it to stdio or to the local file. It says it's printing the stacktrace, but then nothing prints. And the -stack.txt files exist, but are empty. See http://build.chromium.org/buildbot/waterfall/builders/Webkit%20Linux%20(dbg)(1)/builds/18823/steps/webkit_tests/logs/stdio as an example.

On Windows it doesn't even say it's printing the stacktrace. See svg/text/text-text-06-t.svg on http://build.chromium.org/buildbot/waterfall/builders/Webkit/builds/20702/steps/webkit_tests/logs/stdio.

On Mac, it seems to be working fine. See svg/custom/transform-ignore-after-invalid.svg at http://build.chromium.org/buildbot/waterfall/builders/Webkit%20Mac10.5%20(dbg)(3)/builds/12532/steps/webkit_tests/logs/stdio.

Marking as a P1 since this is tooling that's holding up progress (e.g. the webkit merge).

[Dirk Pranke]

Okay, there seem to be several different issues here. 

The behavior is different on different ports, and differs between Debug and Release builds, and differs between a CHECK() firing and an actual crash. We also differ as to whether or not we get the assert string, the backtrace, or both.

I tested this all by instrumenting test_shell with a CHECK(false) and an explicit write to 0x0, and then fed tests to test_shell on the command line like so:

% echo "/src/third_party/WebKit/LayoutTests/fast/html/article-element.html" | test_shell --layout-tests

First, let's look at handling of CHECK()s firing. 

1) On Mac and Linux, when you pass --layout-tests to test_shell, CHECK() is not treated as fatal - the test shell continues, in both debug and release builds. On Windows, CHECK() is fatal in both configurations.

2) On all platforms, in Release builds, we don't ever get a stack trace for a CHECK(). In Debug builds, in all situations, we get both the assert string and the stack trace.

Second, for "actual crashes" (a write to 0x0):

1) We get a stack trace to stderr, but nothing to stdout.

Note that if we call abort() instead of writing to 0x0, we get nothing (no stack trace). I think this is actually okay.

Now, I believe the new-run-webkit-tests code doesn't handle reading from stderr on Chromium ports well at all, so whatever is getting logged to stderr in a crash is getting discarded.

[Dirk Pranke]

It seems like CHECK() should be fatal in all cases, but --layout-tests seems to disable it on the Mac and Linux. Someone must have done this intentionally, presumably under the theory that the test shell should "keep going", but this doesn't seem like a good thing to me.

[Dirk Pranke]

I have filed http://code.google.com/p/chromium/issues/detail?id=41120 to get test_shell to do the same consistent thing with CHECK() across all platforms.

I will continue to use this bug to track that, when we do get crashes to stdout and stderr, new-run-webkit-tests actually does the right thing.

[Dirk Pranke]

Also, when a CHECK() fails, we actually get output to both stdout and stderr. I'm not sure if there's a good reason for this, but life would be simpler if it only went one place.

new-run-webkit-tests doesn't have the concept of tracking both stdout and stderr output for a given test (I think run-webkit-tests does). 

For the moment, I will append any stderr output to the stdout output for a test (which means that any test that produces stderr output will produce a failure). Speak up if you think we should do something different.

[Dirk Pranke]

interestingly, I don't appear to get stack traces at all running the WebKit Mac port; there's probably a separate bug here.

[Dirk Pranke]

Hmm, I seem to be reliably getting stack traces on all of the platforms using my current checkout now for some reason. I may need to re-sync the tree and/or check the bots to see if this is still an issue.

[Dirk Pranke]

I'm going to close this as WORKSFORME ; I'm not sure if we're still seeing this. If we are, please point to something and I'll take another look.