Bug 148512

Summary: fast/css/object-fit/object-fit-embed.html isn't testing anything
Product: WebKit Reporter: Alexey Proskuryakov <ap>
Component: Tools / TestsAssignee: Tim Horton <thorton>
Status: RESOLVED FIXED    
Severity: Normal CC: aestes, simon.fraser, thorton
Priority: P2    
Version: Other   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=143743
Attachments:
Description Flags
Patch aestes: review+

Description Alexey Proskuryakov 2015-08-26 21:42:22 PDT
Positively bisected the flakiness to http://trac.webkit.org/r188807 - when the test fails, expected result seems to be dumped too early, before load finishes.

I can reproduce on a debug build like this:

run-webkit-tests fast/css/object-fit/object-fit-embed.html --repeat 50 -f --no-retry --no-show
Comment 1 Tim Horton 2015-08-26 21:53:35 PDT
Interesting! I'll look at this tomorrow.
Comment 2 Tim Horton 2015-08-26 21:58:06 PDT
It's fairly amazing that only one (or three or so) tests would be made flaky by this, if we really are snapshotting at the wrong time...
Comment 3 Tim Horton 2015-08-27 10:56:16 PDT
Even upping it to 500, this doesn't repro for me.
Comment 4 Tim Horton 2015-08-27 15:55:22 PDT
Alexey and I have learned a lot (like, for example, this test was previously not working correctly, but previously both the expected and actual would fail in the same way).

It would appear that we are trying to use the "WebKit Test PlugIn" to load our PNG instead of an ImageDocument. And indeed, the plugin claims to load PNGs.

Not yet sure why it ever sometimes works correctly, though.
Comment 5 Tim Horton 2015-08-27 15:57:52 PDT
The test similarly behaves wrongly in DRT.
Comment 6 Tim Horton 2015-08-27 16:04:38 PDT
And doing a:

testRunner.setPluginsEnabled(false);

makes it behave correctly.

This does not, however, explain why the test became flaky (it should *always* behave incorrectly, consistently) with my patch on the bots and on Alexey's machine.
Comment 7 Tim Horton 2015-08-27 16:06:34 PDT
Patch came from Blink, smfr merged it. That probably explains how we ended up with a broken test.
Comment 8 Tim Horton 2015-08-27 16:06:54 PDT
(In reply to comment #7)
> Patch came from Blink, smfr merged it. That probably explains how we ended
> up with a broken test.

Also, this is one of the bizarre downsides of reftests.
Comment 9 Tim Horton 2015-08-27 16:41:56 PDT
(In reply to comment #4)
> Not yet sure why it ever sometimes works correctly, though.

Alexey has another bug about sometimes not loading plugins when we should, and is going to investigate that part separately. I'm going to post a patch to disable plugins in the few tests that are affected by the test plugin's handling of PNGs.
Comment 10 Tim Horton 2015-08-27 16:46:31 PDT
Created attachment 260096 [details]
Patch
Comment 11 Tim Horton 2015-08-27 16:53:04 PDT
http://trac.webkit.org/changeset/189073

Alexey, can you relate the bug you're working on?