Bug 66581 - new-run-webkit-tests returns false-positives for chromium port if ImageDiff fails
Summary: new-run-webkit-tests returns false-positives for chromium port if ImageDiff f...
Status: RESOLVED DUPLICATE of bug 92934
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Dirk Pranke
URL:
Keywords: NRWT
Depends on:
Blocks: 64491
  Show dependency tree / graph
 
Reported: 2011-08-19 13:53 PDT by Shawn Singh
Modified: 2012-08-01 19:27 PDT (History)
5 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Shawn Singh 2011-08-19 13:53:07 PDT 
if the ImageDiff tool fails, then the diff_image function (in ChromiumPort, Tools/Scripts/webkitpy/layout_tests/port/layoutchromium.py) returns "None" to indicate the test passed.  It does print an error in this case, but it is too easy to overlook it.

It is probably important to find a cleaner way to handle this.  Some options (thanks to dpranke):
 - the ideal option is to modify NRWT to handle internal errors so that tests are marked as "internal error, could not run test" instead of passed or failed.
 - modify the code to cause a test failure and outputs a diff, then the user can see that the diff is actually correct (false negatives may be safer than false positives)
 - optionally, have NRWT output a diff, with a hardcoded .png that literally says there was an internal error in the image.
Comment 1 Eric Seidel (no email) 2011-10-11 13:15:56 PDT 
I wonder if this is a dupe of or at least related to bug 47240.
Comment 2 Eric Seidel (no email) 2011-10-11 13:16:31 PDT 
On further reading, no. :)
Comment 3 Dirk Pranke 2012-08-01 19:27:37 PDT 
whoops, this is now a dup of bug 92934, since I attached a patch there ...

*** This bug has been marked as a duplicate of bug 92934 ***