Bug 148512 - fast/css/object-fit/object-fit-embed.html isn't testing anything
Summary: fast/css/object-fit/object-fit-embed.html isn't testing anything
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: Other
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Tim Horton
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-08-26 21:42 PDT by Alexey Proskuryakov
Modified: 2015-08-27 16:53 PDT (History)
3 users (show)

See Also:


Attachments
Patch (3.92 KB, patch)
2015-08-27 16:46 PDT, Tim Horton
aestes: review+
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Alexey Proskuryakov 2015-08-26 21:42:22 PDT
Positively bisected the flakiness to http://trac.webkit.org/r188807 - when the test fails, expected result seems to be dumped too early, before load finishes.

I can reproduce on a debug build like this:

run-webkit-tests fast/css/object-fit/object-fit-embed.html --repeat 50 -f --no-retry --no-show
Comment 1 Tim Horton 2015-08-26 21:53:35 PDT
Interesting! I'll look at this tomorrow.
Comment 2 Tim Horton 2015-08-26 21:58:06 PDT
It's fairly amazing that only one (or three or so) tests would be made flaky by this, if we really are snapshotting at the wrong time...
Comment 3 Tim Horton 2015-08-27 10:56:16 PDT
Even upping it to 500, this doesn't repro for me.
Comment 4 Tim Horton 2015-08-27 15:55:22 PDT
Alexey and I have learned a lot (like, for example, this test was previously not working correctly, but previously both the expected and actual would fail in the same way).

It would appear that we are trying to use the "WebKit Test PlugIn" to load our PNG instead of an ImageDocument. And indeed, the plugin claims to load PNGs.

Not yet sure why it ever sometimes works correctly, though.
Comment 5 Tim Horton 2015-08-27 15:57:52 PDT
The test similarly behaves wrongly in DRT.
Comment 6 Tim Horton 2015-08-27 16:04:38 PDT
And doing a:

testRunner.setPluginsEnabled(false);

makes it behave correctly.

This does not, however, explain why the test became flaky (it should *always* behave incorrectly, consistently) with my patch on the bots and on Alexey's machine.
Comment 7 Tim Horton 2015-08-27 16:06:34 PDT
Patch came from Blink, smfr merged it. That probably explains how we ended up with a broken test.
Comment 8 Tim Horton 2015-08-27 16:06:54 PDT
(In reply to comment #7)
> Patch came from Blink, smfr merged it. That probably explains how we ended
> up with a broken test.

Also, this is one of the bizarre downsides of reftests.
Comment 9 Tim Horton 2015-08-27 16:41:56 PDT
(In reply to comment #4)
> Not yet sure why it ever sometimes works correctly, though.

Alexey has another bug about sometimes not loading plugins when we should, and is going to investigate that part separately. I'm going to post a patch to disable plugins in the few tests that are affected by the test plugin's handling of PNGs.
Comment 10 Tim Horton 2015-08-27 16:46:31 PDT
Created attachment 260096 [details]
Patch
Comment 11 Tim Horton 2015-08-27 16:53:04 PDT
http://trac.webkit.org/changeset/189073

Alexey, can you relate the bug you're working on?