Bug 148512 – fast/css/object-fit/object-fit-embed.html isn't testing anything

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Alexey Proskuryakov]

Positively bisected the flakiness to http://trac.webkit.org/r188807 - when the test fails, expected result seems to be dumped too early, before load finishes.

I can reproduce on a debug build like this:

run-webkit-tests fast/css/object-fit/object-fit-embed.html --repeat 50 -f --no-retry --no-show

[Tim Horton]

Interesting! I'll look at this tomorrow.

[Tim Horton]

It's fairly amazing that only one (or three or so) tests would be made flaky by this, if we really are snapshotting at the wrong time...

[Tim Horton]

Even upping it to 500, this doesn't repro for me.

[Tim Horton]

Alexey and I have learned a lot (like, for example, this test was previously not working correctly, but previously both the expected and actual would fail in the same way).

It would appear that we are trying to use the "WebKit Test PlugIn" to load our PNG instead of an ImageDocument. And indeed, the plugin claims to load PNGs.

Not yet sure why it ever sometimes works correctly, though.

[Tim Horton]

The test similarly behaves wrongly in DRT.

[Tim Horton]

And doing a:

testRunner.setPluginsEnabled(false);

makes it behave correctly.

This does not, however, explain why the test became flaky (it should *always* behave incorrectly, consistently) with my patch on the bots and on Alexey's machine.

[Tim Horton]

Patch came from Blink, smfr merged it. That probably explains how we ended up with a broken test.

[Tim Horton]

(In reply to comment #7)
> Patch came from Blink, smfr merged it. That probably explains how we ended
> up with a broken test.

Also, this is one of the bizarre downsides of reftests.

[Tim Horton]

(In reply to comment #4)
> Not yet sure why it ever sometimes works correctly, though.

Alexey has another bug about sometimes not loading plugins when we should, and is going to investigate that part separately. I'm going to post a patch to disable plugins in the few tests that are affected by the test plugin's handling of PNGs.

[Tim Horton]

Created attachment 260096 [details]
Patch

[Tim Horton]

http://trac.webkit.org/changeset/189073

Alexey, can you relate the bug you're working on?