I had an image failure for the snowleopard mac port (i.e. not --chromium). The image diff was below the default 0.1 tolerance. Here's what I got: new-run-webkit-tests --experimental-fully-parallel --child-processes 10 --no-show-results --time-out-ms 1 --no-retry-failures fullscreen -p fullscreen/full-screen-zIndex.html -> unexpected image and text mismatch 9 tests ran as expected, 1 didn't: Regressions: Unexpected image and text mismatch : (1) fullscreen/full-screen-zIndex.html = IMAGE+TEXT What I should have gotten is something like the following: new-run-webkit-tests --experimental-fully-parallel --child-processes 10 --no-show-results --time-out-ms 1 --no-retry-failures fullscreen -p fullscreen/full-screen-zIndex.html -> unexpected text mismatch and image hash failure (below 0.1% tolerance) 9 tests ran as expected, 1 didn't: Regressions: Unexpected text mismatch : (1) fullscreen/full-screen-zIndex.html =TEXT 2 things went wrong: -Should show an image failure below tolerance -Should not count as an IMAGE+TEXT, but just a TEXT Also, it didn't write out a -diff.png file. I'm not sure whether we should or not. We should do whatever old-run-webkit-tests did in this case. When I run the above with tolerance=0, everything works out as expected (e.g. we print the same thing and we write out a -diff.png file).
clearing ownership; I may not get to this any time soon, so someone else can feel free to work on it.
I think we now match ORWT's behavior. Let me know if you still want something on this.