Bug 21458 – Improve image comparison algorithm in ImageDiff

• Component: Tools / Tests
• Status: RESOLVED WONTFIX

[Pierre-Olivier Latour]

We would like to improve the image comparison algorithm in ImageDiff:
- provide a new comparison algorithm that is more tolerant to "acceptable" failures (i.e. very small differences in font rendering, which --threshold is not really good at handling)
- generate normalized intensity only diff images
- have this new behavior be opt-in through a new option for backward compatibility

(In reply to comment #0)
> - generate normalized intensity only diff images

I think we should consider dropping diff image generation and do it on demand using <canvas> imageData API. This would allow the user to adjust contrast, invert, threshold etc. after the fact. See also bug 11959.

[Pierre-Olivier Latour]

> I think we should consider dropping diff image generation and do it on demand
> using <canvas> imageData API. This would allow the user to adjust contrast,
> invert, threshold etc. after the fact. See also bug 11959.

This makes a lot of sense, but seems to be quite a bit of work. This specific bug is just to improve the current situation so we can move ahead with using pixel testing in our workflow.

I'll drop the patch soon.

[Pierre-Olivier Latour]

Current behavior:
WebKitTools/Scripts/run-webkit-tests --pixel --threshold 0 LayoutTests/fast/repaint
21.45s total testing time
44 test cases (44%) succeeded
56 test cases (56%) had incorrect layout

New behavior (faster):
WebKitTools/Scripts/run-webkit-tests --pixel --alternate-difference --threshold 0 LayoutTests/fast/repaint
18.60s total testing time
44 test cases (44%) succeeded
56 test cases (56%) had incorrect layout

New behavior taking advantage of more flexible tolerance (ignore differences less than "threshold" percents):
WebKitTools/Scripts/run-webkit-tests --pixel --alternate-difference --threshold 0.1 LayoutTests/fast/repaint
17.16s total testing time
91 test cases (91%) succeeded
9 test cases (9%) had incorrect layout

In the latest usage mode, we managed to avoid almost all "false negatives" (tests that failed because of font rendering), although we have a few "false positives" (tests that have differences not due to font rendering but passed because they were really small):
fast/repaint/box-shadow-dynamic.html (0.04%)
fast/repaint/overflow-scroll-delete-diffs.html (0.10%)
fast/repaint/selection-gap-overflow-scroll-diffs.html (0.04%)
fast/repaint/table-cell-collapsed-border.html (0.01%)

Suggestions needed for:
- better name than "--alternate-difference" for the new option in the script
- better name than "--alternate" for the new option in the tool
- better phrasing for updated usage of the script

[Pierre-Olivier Latour]

Created attachment 24212 [details]
Revision 1

[Pierre-Olivier Latour]

Created attachment 24272 [details]
Revision 2

After discussing with Darin, he pointed out this behavior, if better, shouldn't be opt-in, but the default. So here's a new patch with the old behavior gone and cleaned up code as well.

[Pierre-Olivier Latour]

Because of interdependencies, this patch will be merged with the master one at https://bugs.webkit.org/show_bug.cgi?id=21322