Bug 94704

Summary: REGRESSION(r126189): Reftest mismatches are (again) run through ImageDiff with 0.1 tolerance
Product: WebKit Reporter: Zan Dobersek <zan>
Component: New BugsAssignee: Zan Dobersek <zan>
Severity: Normal CC: abarth, beidson, dpranke, eric, ojan, simon.fraser, thorton, webkit.review.bot
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Description Flags
Patch none

Description Zan Dobersek 2012-08-22 06:36:09 PDT
REGRESSION(r126189): Reftest mismatches are (again) run through ImageDiff with 0.1 tolerance
Comment 1 Zan Dobersek 2012-08-22 06:37:06 PDT
Created attachment 159923 [details]
Comment 2 Zan Dobersek 2012-08-22 06:38:25 PDT
Perhaps a reincarnation of bug #92847, this caused some tests to pop up as passing. Too bad.
Comment 3 Dirk Pranke 2012-08-22 09:19:12 PDT
Oh, that's my fault. Will look into it in a bit ...
Comment 4 Eric Seidel (no email) 2012-08-22 13:24:49 PDT
How do we test this so it doesn't happen again?
Comment 5 Dirk Pranke 2012-08-22 13:44:14 PDT
Actually, to clarify ... this is not a regression of bug #92847. This was caused by the fix I landed for bug 94540 (r126189). At some point, we stopped calling diff_image() for reftests and were just comparing checksums. This seemed to be bad (at least for the Apple Mac port) and so I added the diff_image as a sanity check.

It wasn't clear to me at the time (and still isn't) whether we should use the default tolerance for the port, or use 0 (and thus ref tests and pixel tests would potentially be compared with different tolerances). I can see arguments for either way, and perhaps this should be a separate default setting.

In the absence of agreeing over what we want to do here, it's not obvious what the test should be testing ;).

That said, I'll R+ this as I *think* the Apple Mac port doesn't actually need fuzzy diffs and so this shouldn't break things. I guess we can find out.
Comment 6 Dirk Pranke 2012-08-22 13:44:46 PDT
Comment on attachment 159923 [details]

Also, we should add some sort of a test for this (per eric's comment), but I don't want to hold up this fix for that. I'll file a separate bug.
Comment 7 WebKit Review Bot 2012-08-22 14:42:15 PDT
Comment on attachment 159923 [details]

Clearing flags on attachment: 159923

Committed r126353: <http://trac.webkit.org/changeset/126353>
Comment 8 WebKit Review Bot 2012-08-22 14:42:19 PDT
All reviewed patches have been landed.  Closing bug.