Positively bisected the flakiness to http://trac.webkit.org/r188807 - when the test fails, expected result seems to be dumped too early, before load finishes. I can reproduce on a debug build like this: run-webkit-tests fast/css/object-fit/object-fit-embed.html --repeat 50 -f --no-retry --no-show
Interesting! I'll look at this tomorrow.
It's fairly amazing that only one (or three or so) tests would be made flaky by this, if we really are snapshotting at the wrong time...
Even upping it to 500, this doesn't repro for me.
Alexey and I have learned a lot (like, for example, this test was previously not working correctly, but previously both the expected and actual would fail in the same way). It would appear that we are trying to use the "WebKit Test PlugIn" to load our PNG instead of an ImageDocument. And indeed, the plugin claims to load PNGs. Not yet sure why it ever sometimes works correctly, though.
The test similarly behaves wrongly in DRT.
And doing a: testRunner.setPluginsEnabled(false); makes it behave correctly. This does not, however, explain why the test became flaky (it should *always* behave incorrectly, consistently) with my patch on the bots and on Alexey's machine.
Patch came from Blink, smfr merged it. That probably explains how we ended up with a broken test.
(In reply to comment #7) > Patch came from Blink, smfr merged it. That probably explains how we ended > up with a broken test. Also, this is one of the bizarre downsides of reftests.
(In reply to comment #4) > Not yet sure why it ever sometimes works correctly, though. Alexey has another bug about sometimes not loading plugins when we should, and is going to investigate that part separately. I'm going to post a patch to disable plugins in the few tests that are affected by the test plugin's handling of PNGs.
Created attachment 260096 [details] Patch
http://trac.webkit.org/changeset/189073 Alexey, can you relate the bug you're working on?