Bug 94746 – decide what (and how) we should set the tolerance for ref test pixel compares and test for that

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Dirk Pranke]

See comments in https://bugs.webkit.org/show_bug.cgi?id=94704 for background ... it's perhaps not clear if we should use the default tolerance the port specifies for pixel tests also for ref tests, or if ref tests should always require an exact match (tolerance == 0 or just hash compares), or if we should have a separate default-tolerance-for-ref-tests parameter.

Once we agree on this, we should make sure this is implemented and write a test to check the behavior (we don't have a test today).

[Ojan Vafai]

I don't see why we'd ever want a tolerance other than 0 for reftests. The point of tolerance is to handle things like different text anti-aliasing across OS releases. None of these sort of things apply to reftests.

[Ryosuke Niwa]

Yeah, it seems like we shouldn't need any tolerance for ref tests.

[Dirk Pranke]

I don't have any real argument against using 0. Simon, anyone else from Apple, any objections?

[Simon Fraser (smfr)]

No objection, but I wouldn't be surprised if some ref tests are "close but not exact" and fail with a zero tolerance. We'll have to try it and see.

[Ryosuke Niwa]

I guess 1px rounding differences are common (e.g. inline-block vs. inline) in text rendering.

[Dirk Pranke]

it would seem a bit odd to me to have reftests that attempt to render the same string of text two different ways and hope that they would be the same, but maybe I'm not being creative enough. Do we have many of those?

[Ryosuke Niwa]

(In reply to comment #6)
> it would seem a bit odd to me to have reftests that attempt to render the same string of text two different ways and hope that they would be the same, but maybe I'm not being creative enough. Do we have many of those?

I've hit that when I was converting a test for bdi because inline-block and inline have a different semantics in UBA.

[Dirk Pranke]

Created attachment 161534 [details]
Patch

[Dirk Pranke]

Looks like there's general consensus that we should require exact matches. I'm adding an assertion to the test cases to enforce this.

[WebKit Review Bot]

Comment on attachment 161534 [details]
Patch

Clearing flags on attachment: 161534

Committed r127180: <http://trac.webkit.org/changeset/127180>

[WebKit Review Bot]

All reviewed patches have been landed.  Closing bug.