Bug 163362

Summary: REGRESSION (r207162): [ios-simulator] LayoutTest platform/ios-simulator/ios/plugin/youtube-flash-plugin-iframe.html became more flaky
Product: WebKit Reporter: Ryan Haddad <ryanhaddad>
Component: New BugsAssignee: Nobody <webkit-unassigned>
Status: NEW ---    
Severity: Normal CC: achristensen, aestes, ap, cdumez, dbates, jlewis3, rniwa, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=163127
Attachments:
Description Flags
Patch achristensen: review-

Description Ryan Haddad 2016-10-12 15:13:31 PDT
LayoutTest platform/ios-simulator/ios/plugin/youtube-flash-plugin-iframe.html failing

https://build.webkit.org/results/Apple%20iOS%2010%20Simulator%20Debug%20WK2%20(Tests)/r207162%20(707)/results.html

http://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=platform%2Fios-simulator%2Fios%2Fplugin%2Fyoutube-flash-plugin-iframe.html

--- /Volumes/Data/slave/ios-simulator-10-debug-tests-wk2/build/layout-test-results/platform/ios-simulator/ios/plugin/youtube-flash-plugin-iframe-expected.txt
+++ /Volumes/Data/slave/ios-simulator-10-debug-tests-wk2/build/layout-test-results/platform/ios-simulator/ios/plugin/youtube-flash-plugin-iframe-actual.txt
@@ -2,7 +2,6 @@
 Blocked access to external URL http://www.youtube.com/embed/N0gb9v4LI4o
 Blocked access to external URL http://www.youtube.com/embed/N0gb9v4LI4o?version=3&hl=en_US
 Blocked access to external URL http://www.youtube.com/embed/N0gb9v4LI4o?version=3&hl=en_US
-Blocked access to external URL http://www.youtube.com/embed/N0gb9v4LI4o
 Test transforming youtube flash plugin to youtube iFrame based player. < rdar://problem/11201356>
 
 On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
@@ -19,8 +18,8 @@
 PASS objectEmbedShadowRoot.firstChild.firstChild.tagName is "IFRAME"
 PASS internals.shadowPseudoId(objectNoEmbedShadowRoot.firstChild) is "-webkit-plugin-replacement"
 PASS objectNoEmbedShadowRoot.firstChild.firstChild.tagName is "IFRAME"
-PASS internals.shadowPseudoId(normalEmbedInIframeShadowRoot.firstChild) is "-webkit-plugin-replacement"
-PASS normalEmbedInIframeShadowRoot.firstChild.firstChild.tagName is "IFRAME"
+FAIL internals.shadowPseudoId(normalEmbedInIframeShadowRoot.firstChild) should be -webkit-plugin-replacement. Threw exception TypeError: null is not an object (evaluating 'normalEmbedInIframeShadowRoot.firstChild')
+FAIL normalEmbedInIframeShadowRoot.firstChild.firstChild.tagName should be IFRAME. Threw exception TypeError: null is not an object (evaluating 'normalEmbedInIframeShadowRoot.firstChild')
 PASS successfullyParsed is true
 
 TEST COMPLETE
Comment 1 Ryan Haddad 2016-10-12 15:14:10 PDT
Flakiness dashboard seems to indicate that this may be fallout from http://trac.webkit.org/changeset/207162
Comment 2 Ryan Haddad 2016-10-17 16:03:54 PDT
Marked as flaky in http://trac.webkit.org/projects/webkit/changeset/207440
Comment 3 Alexey Proskuryakov 2016-11-15 11:49:08 PST
Alex, is this something that you are looking into?
Comment 4 Alex Christensen 2016-11-15 12:52:13 PST
Not actively.
Comment 5 Radar WebKit Bug Importer 2016-11-15 13:32:30 PST
<rdar://problem/29274536>
Comment 6 Alex Christensen 2016-12-08 23:26:42 PST
I verified that this test works fine for me locally.

run-webkit-tests platform/ios-simulator/ios/plugin/youtube-flash-plugin-iframe.html --ios-simulator --iterations=10 --repeat-each=10

I think we should just unskip it.  What do you think, Alexey?
Comment 7 Alex Christensen 2016-12-08 23:34:56 PST
Created attachment 296647 [details]
Patch
Comment 8 Alex Christensen 2016-12-08 23:42:32 PST
Comment on attachment 296647 [details]
Patch

Nope, I can reproduce this about 10% of the time.  URLParser enabled or not, doesn't matter.  This means it's probably not related to the URLParser being enabled.
Comment 9 Alex Christensen 2016-12-08 23:46:10 PST
I'm not familiar with this test, or why these URLs should be blocked.  Dan, what is going on here?
Comment 10 Daniel Bates 2016-12-09 09:51:10 PST
(In reply to comment #9)
> I'm not familiar with this test, or why these URLs should be blocked.  Dan,
> what is going on here?

The "Blocked access to external URL" messages are because we do not allow external connections when running a test in DRT/WKTR. They can be ignored for the purposes of this test.

> +FAIL internals.shadowPseudoId(normalEmbedInIframeShadowRoot.firstChild) should be -webkit-plugin-replacement. Threw exception TypeError: null is not an object (evaluating 'normalEmbedInIframeShadowRoot.firstChild')

This means that that the plugin had not been not replaced by the time we checked. Plugin/plugin replacement loading is asynchronous with respect to the initial parsing of an HTML object/embed. We need to find a better way to know when a plugin/plugin-replacement has loaded.
Comment 11 Daniel Bates 2016-12-09 09:52:10 PST
(In reply to comment #10)
> We need to find a better way to know when a plugin/plugin-replacement has loaded.

I meant to write:

We need to find a better way to know when a plugin/plugin-replacement has loaded than using a zero timer.
Comment 12 Alexey Proskuryakov 2016-12-09 10:08:01 PST
Checking internal test history, I see this test being flaky on ASan and GuardMalloc bots even in September. Also, WebKit1 was a flaky failure.

But debug WK2 bots did indeed see a very sharp increase in flakiness around r207162.
Comment 13 Alex Christensen 2016-12-12 12:12:43 PST
Having concluded that this was not related to the URLParser change because of the equal flakiness rate with or without the URLParser, I'm removing myself as the assignee and I do not intend to look into this further.
Comment 14 Alexey Proskuryakov 2016-12-12 12:50:10 PST
Alex and I looked at the flakiness rate, and it has very clearly increased dramatically around r207162. There isn't really another change close enough that can be blamed.

There were no failures for ~200 revisions before r207162. Here is the history around it:

207158 oooo (this revision was tested 4 times, with 4 passes)
207159 o
207160 (this exact revision hasn't been tested)
207161 o
207162 x (tested once, failed)
207163 o (tested once, passed)
207164 xxxxoo (this revision was tested 6 times, with 4 failures and 2 passes)
207165 x
207166 xo
207167
207168 xx
207169 xx

r207161 is a contributors.json change
r207160 is purely a WebKit1 change
r207159 is a change to iOS 9 built apps that could theoretically have a bug, but it just uses standard settings machinery, unlikely to be wrong.
Comment 15 Alexey Proskuryakov 2016-12-12 12:56:54 PST
It seems likely that it's not an obvious outright bug in the URL parser causing this, but some kind of subtle change (perhaps timing), but we still need to make the test work again.
Comment 16 Ryan Haddad 2017-11-29 09:50:59 PST
This test now fails 100% of the time on debug and it is flaky on release.

Here is the diff from a release build:
--- /Volumes/Data/slave/ios-simulator-11-release-tests-wk2/build/layout-test-results/platform/ios/ios/plugin/youtube-flash-plugin-iframe-expected.txt
+++ /Volumes/Data/slave/ios-simulator-11-release-tests-wk2/build/layout-test-results/platform/ios/ios/plugin/youtube-flash-plugin-iframe-actual.txt
@@ -15,8 +15,8 @@
 PASS document.querySelectorAll("iframe").length is 1
 PASS internals.shadowPseudoId(normalEmbedShadowRoot.firstChild) is "-webkit-plugin-replacement"
 PASS normalEmbedShadowRoot.firstChild.firstChild.tagName is "IFRAME"
-PASS internals.shadowPseudoId(objectEmbedShadowRoot.firstChild) is "-webkit-plugin-replacement"
-PASS objectEmbedShadowRoot.firstChild.firstChild.tagName is "IFRAME"
+FAIL internals.shadowPseudoId(objectEmbedShadowRoot.firstChild) should be -webkit-plugin-replacement. Threw exception TypeError: null is not an object (evaluating 'objectEmbedShadowRoot.firstChild')
+FAIL objectEmbedShadowRoot.firstChild.firstChild.tagName should be IFRAME. Threw exception TypeError: null is not an object (evaluating 'objectEmbedShadowRoot.firstChild')
 PASS internals.shadowPseudoId(objectNoEmbedShadowRoot.firstChild) is "-webkit-plugin-replacement"
 PASS objectNoEmbedShadowRoot.firstChild.firstChild.tagName is "IFRAME"
 PASS internals.shadowPseudoId(normalEmbedInIframeShadowRoot.firstChild) is "-webkit-plugin-replacement"
Comment 17 Matt Lewis 2017-12-13 12:55:07 PST
This is still happening as of 12/13/2017

Updated Expectations:
https://trac.webkit.org/changeset/225867/webkit