66581 2011-08-19 13:53:07 -0700 new-run-webkit-tests returns false-positives for chromium port if ImageDiff fails 2012-08-01 19:27:37 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified RESOLVED DUPLICATE 92934 NRWT P2 Normal --- 64491 0 shawnsingh dpranke dpranke eric ojan simon.fraser tony oldest_to_newest 454051 0 shawnsingh 2011-08-19 13:53:07 -0700 if the ImageDiff tool fails, then the diff_image function (in ChromiumPort, Tools/Scripts/webkitpy/layout_tests/port/layoutchromium.py) returns "None" to indicate the test passed. It does print an error in this case, but it is too easy to overlook it. It is probably important to find a cleaner way to handle this. Some options (thanks to dpranke): - the ideal option is to modify NRWT to handle internal errors so that tests are marked as "internal error, could not run test" instead of passed or failed. - modify the code to cause a test failure and outputs a diff, then the user can see that the diff is actually correct (false negatives may be safer than false positives) - optionally, have NRWT output a diff, with a hardcoded .png that literally says there was an internal error in the image. 481852 1 eric 2011-10-11 13:15:56 -0700 I wonder if this is a dupe of or at least related to bug 47240. 481853 2 eric 2011-10-11 13:16:31 -0700 On further reading, no. :) 684898 3 dpranke 2012-08-01 19:27:37 -0700 whoops, this is now a dup of bug 92934, since I attached a patch there ... *** This bug has been marked as a duplicate of bug 92934 ***