Bug 221618 – Drop pixel test support

• Status: RESOLVED
• Resolution: WONTFIX
• Component: Tools / Tests

[Sam Sneddon [:gsnedders]]

With bug 121112 never having been resolved, ever since Blink forked the pixel tests (i.e., -expected.png) haven't been run.

Given this is now nearly eight years ago, it would be nice to acknowledge we aren't running the pixel tests (and that the expectations haven't been updated in equally long) and drop support for running them from run-webkit-tests.

[Simon Fraser (smfr)]

I think I'm OK with removing expected.png files, but I'm hesitant about removing support for run-webkit-tests --pixel, because there might be times where a developer wants to do before/after pixel testing of a change by rebasing pixel results first, then running tests.

[Alexey Proskuryakov]

What's been stopping us before was that even though pixel tests couldn't meaningfully run, referring to an ancient -expected.png file provided insight into what a test was meant to do.

[Sam Sneddon [:gsnedders]]

(In reply to Simon Fraser (smfr) from comment #1)
> I think I'm OK with removing expected.png files, but I'm hesitant about
> removing support for run-webkit-tests --pixel, because there might be times
> where a developer wants to do before/after pixel testing of a change by
> rebasing pixel results first, then running tests.

FWIW, I have some concern about our ability to keep all that code working when it isn't running on any bot, hence I somewhat question the value of keeping it around if we're never actually running it. There's enough scattered around various parts of the codebase it feels somewhat fragile.

If we _do_ want to keep it, I'd rather we started running at the very least a small number of pixel tests in CI. (Though I don't much like the idea of running them at all, given how fragile they are!)

(In reply to Alexey Proskuryakov from comment #2)
> What's been stopping us before was that even though pixel tests couldn't
> meaningfully run, referring to an ancient -expected.png file provided
> insight into what a test was meant to do.

My proposal here was to just remove the support from run-webkit-tests, and not necessarily do anything about the -expected.png files (which IIRC Myles saw value in, for the reason you say).

[Fujii Hironori]

Even though no ports enable pixel tests as default, they are still used by enabling -p switch explicitly on developers PC.
For example, r272361, r270756 and r269072.

[Sam Sneddon [:gsnedders]]

Okay, if people are still definitely using them, then WONTFIX.