After r123893 and the image diffing code refactoring, the default tolerance is set to 0.1 for the port when diffing images. With the tolerance for diffing reftest output being 0, the expression on the next line evaluates to 0.1. This means the ImageDiff operates with 0.1 tolerance, which can lead to false positives (i.e. ImageDiff reports that the two images' differences are within the allowed boundaries).
I guess it should explicitly be checked whether the tolerance argument passed into base.diff_image is None and act accordingly. I don't know why the default value for the tolerance option is being overridden, though (if it is actually overridden rather than just ensured it exists).
 - http://trac.webkit.org/changeset/123893
 - http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py?rev=123893#L325
 - http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py?rev=123893#L326
Created attachment 155940 [details]
whoops. good catch.
Comment on attachment 155940 [details]
Clearing flags on attachment: 155940
Committed r124401: <http://trac.webkit.org/changeset/124401>
All reviewed patches have been landed. Closing bug.