if the ImageDiff tool fails, then the diff_image function (in ChromiumPort, Tools/Scripts/webkitpy/layout_tests/port/layoutchromium.py) returns "None" to indicate the test passed. It does print an error in this case, but it is too easy to overlook it.
It is probably important to find a cleaner way to handle this. Some options (thanks to dpranke):
- the ideal option is to modify NRWT to handle internal errors so that tests are marked as "internal error, could not run test" instead of passed or failed.
- modify the code to cause a test failure and outputs a diff, then the user can see that the diff is actually correct (false negatives may be safer than false positives)
- optionally, have NRWT output a diff, with a hardcoded .png that literally says there was an internal error in the image.
I wonder if this is a dupe of or at least related to bug 47240.
On further reading, no. :)
whoops, this is now a dup of bug 92934, since I attached a patch there ...
*** This bug has been marked as a duplicate of bug 92934 ***