Bug 58804 – new-run-webkit-tests doesn't correctly handle image failures below tolerance

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Ojan Vafai]

I had an image failure for the snowleopard mac port (i.e. not --chromium). The image diff was below the default 0.1 tolerance. Here's what I got:
new-run-webkit-tests --experimental-fully-parallel --child-processes 10 --no-show-results --time-out-ms 1 --no-retry-failures fullscreen -p
  fullscreen/full-screen-zIndex.html -> unexpected image and text mismatch
9 tests ran as expected, 1 didn't:                 

Regressions: Unexpected image and text mismatch : (1)
  fullscreen/full-screen-zIndex.html = IMAGE+TEXT


What I should have gotten is something like the following:
new-run-webkit-tests --experimental-fully-parallel --child-processes 10 --no-show-results --time-out-ms 1 --no-retry-failures fullscreen -p
  fullscreen/full-screen-zIndex.html -> unexpected text mismatch and image hash failure (below 0.1% tolerance)
9 tests ran as expected, 1 didn't:                 

Regressions: Unexpected text mismatch : (1)
  fullscreen/full-screen-zIndex.html =TEXT


2 things went wrong:
-Should show an image failure below tolerance
-Should not count as an IMAGE+TEXT, but just a TEXT

Also, it didn't write out a -diff.png file. I'm not sure whether we should or not. We should do whatever old-run-webkit-tests did in this case.

When I run the above with tolerance=0, everything works out as expected (e.g. we print the same thing and we write out a -diff.png file).

[Dirk Pranke]

clearing ownership; I may not get to this any time soon, so someone else can feel free to work on it.

[Dirk Pranke]

I think we now match ORWT's behavior. Let me know if you still want something on this.