I'm running new-run-webkit-tests fast/ruby with current ToT. There is currently an assertion failure occurring in four of theses tests (bug 67007), so these have been added to expectations as crashing ones. How can I know if a Crash Reporter dialog that just popped up on my screen is due to an expectedly crashing test, or because the crash happens in-between tests, so NRWT doesn't attribute it to any test? The former is just a nuisance (lots of time wasted for CrashReporter to create its log, and a window for me to dismiss), while the latter requires immediate action, as it's an unknown crasher bug in WebKit. My suggestion is to print out a line when an expected crash occurs, so that one could see whether there is a reason to panic when seeing a CrashReporter dialog during tests.
I suspect this is an artifact of the fact that Chromium (where NRWT came from) uses their own custom crash reporter, which behaves more consistently than CrashReporter does. CrashReporter, for all of its benefits, doesn't fit very well with our testing model, as you can't really ask it if a process crashed or if we should wait for it to finish running, etc. In any case, I'm sure we can come up with a way to make this more pleasant in NRWT. Thank you for the report!
Since ORWT and NRWT do their own handling of crash logs, it would be nice if they could suppress the Crash Reporter window from showing. It would be cool if NRWT knew how to detect that the crash that just occurred was the "expected" crash, as opposed to some other crash that it wasn't expecting. It would probably have to compare stack traces or something. (This might not work very well for crashes due to race conditions.)
Or perhaps we should just skip crashing tests on Mac instead of adding crashing expectations.
(In reply to comment #3) > Or perhaps we should just skip crashing tests on Mac instead of adding crashing expectations. I think that's a very reasonable solution. Just convert all crashing expectations to skip expectations in mac.py.
Alexey, can you refresh me on what you'd currently like to have happen here? I'm not sure what you mean by a Crash Reporter dialog; I don't get dialogs on my screen, but maybe I have them disabled somewhere?
Yes, it's a preference (not sure what the default is). You can get CrashReporterPrefs app by installing Auxiliary Xcode Tools from <https://developer.apple.com/downloads/index.action?name=for%20Xcode%20->. You may also already have it on your system if you use an older Xcode.
Okay, I'm going to close this under the assumption that you'll just skip expected crashes that you don't want crashreports for. If you do want a warning, please reopen and I'll add it. A new warning for such things could be a little jarring to people used to not seeing it, but I'm of the opinion that we shouldn't be suppressing crashes with TestExpectations for any period of time, so that's fine by me.