Bug 163362 – REGRESSION (r207162): [ios-simulator] LayoutTest platform/ios-simulator/ios/plugin/youtube-flash-plugin-iframe.html became more flaky

• Status: NEW
• Component: New Bugs

[Ryan Haddad]

LayoutTest platform/ios-simulator/ios/plugin/youtube-flash-plugin-iframe.html failing

https://build.webkit.org/results/Apple%20iOS%2010%20Simulator%20Debug%20WK2%20(Tests)/r207162%20(707)/results.html

http://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=platform%2Fios-simulator%2Fios%2Fplugin%2Fyoutube-flash-plugin-iframe.html

--- /Volumes/Data/slave/ios-simulator-10-debug-tests-wk2/build/layout-test-results/platform/ios-simulator/ios/plugin/youtube-flash-plugin-iframe-expected.txt
+++ /Volumes/Data/slave/ios-simulator-10-debug-tests-wk2/build/layout-test-results/platform/ios-simulator/ios/plugin/youtube-flash-plugin-iframe-actual.txt
@@ -2,7 +2,6 @@
 Blocked access to external URL http://www.youtube.com/embed/N0gb9v4LI4o
 Blocked access to external URL http://www.youtube.com/embed/N0gb9v4LI4o?version=3&hl=en_US
 Blocked access to external URL http://www.youtube.com/embed/N0gb9v4LI4o?version=3&hl=en_US
-Blocked access to external URL http://www.youtube.com/embed/N0gb9v4LI4o
 Test transforming youtube flash plugin to youtube iFrame based player. < rdar://problem/11201356>
 
 On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
@@ -19,8 +18,8 @@
 PASS objectEmbedShadowRoot.firstChild.firstChild.tagName is "IFRAME"
 PASS internals.shadowPseudoId(objectNoEmbedShadowRoot.firstChild) is "-webkit-plugin-replacement"
 PASS objectNoEmbedShadowRoot.firstChild.firstChild.tagName is "IFRAME"
-PASS internals.shadowPseudoId(normalEmbedInIframeShadowRoot.firstChild) is "-webkit-plugin-replacement"
-PASS normalEmbedInIframeShadowRoot.firstChild.firstChild.tagName is "IFRAME"
+FAIL internals.shadowPseudoId(normalEmbedInIframeShadowRoot.firstChild) should be -webkit-plugin-replacement. Threw exception TypeError: null is not an object (evaluating 'normalEmbedInIframeShadowRoot.firstChild')
+FAIL normalEmbedInIframeShadowRoot.firstChild.firstChild.tagName should be IFRAME. Threw exception TypeError: null is not an object (evaluating 'normalEmbedInIframeShadowRoot.firstChild')
 PASS successfullyParsed is true
 
 TEST COMPLETE

[Ryan Haddad]

Flakiness dashboard seems to indicate that this may be fallout from http://trac.webkit.org/changeset/207162

[Ryan Haddad]

Marked as flaky in http://trac.webkit.org/projects/webkit/changeset/207440

[Alexey Proskuryakov]

Alex, is this something that you are looking into?

[Alex Christensen]

Not actively.

[Radar WebKit Bug Importer]

<rdar://problem/29274536>

[Alex Christensen]

I verified that this test works fine for me locally.

run-webkit-tests platform/ios-simulator/ios/plugin/youtube-flash-plugin-iframe.html --ios-simulator --iterations=10 --repeat-each=10

I think we should just unskip it.  What do you think, Alexey?

[Alex Christensen]

Created attachment 296647 [details]
Patch

[Alex Christensen]

Comment on attachment 296647 [details]
Patch

Nope, I can reproduce this about 10% of the time.  URLParser enabled or not, doesn't matter.  This means it's probably not related to the URLParser being enabled.

[Alex Christensen]

I'm not familiar with this test, or why these URLs should be blocked.  Dan, what is going on here?

[Daniel Bates]

(In reply to comment #9)
> I'm not familiar with this test, or why these URLs should be blocked.  Dan,
> what is going on here?

The "Blocked access to external URL" messages are because we do not allow external connections when running a test in DRT/WKTR. They can be ignored for the purposes of this test.

> +FAIL internals.shadowPseudoId(normalEmbedInIframeShadowRoot.firstChild) should be -webkit-plugin-replacement. Threw exception TypeError: null is not an object (evaluating 'normalEmbedInIframeShadowRoot.firstChild')

This means that that the plugin had not been not replaced by the time we checked. Plugin/plugin replacement loading is asynchronous with respect to the initial parsing of an HTML object/embed. We need to find a better way to know when a plugin/plugin-replacement has loaded.

[Daniel Bates]

(In reply to comment #10)
> We need to find a better way to know when a plugin/plugin-replacement has loaded.

I meant to write:

We need to find a better way to know when a plugin/plugin-replacement has loaded than using a zero timer.

[Alexey Proskuryakov]

Checking internal test history, I see this test being flaky on ASan and GuardMalloc bots even in September. Also, WebKit1 was a flaky failure.

But debug WK2 bots did indeed see a very sharp increase in flakiness around r207162.

[Alex Christensen]

Having concluded that this was not related to the URLParser change because of the equal flakiness rate with or without the URLParser, I'm removing myself as the assignee and I do not intend to look into this further.

[Alexey Proskuryakov]

Alex and I looked at the flakiness rate, and it has very clearly increased dramatically around r207162. There isn't really another change close enough that can be blamed.

There were no failures for ~200 revisions before r207162. Here is the history around it:

207158 oooo (this revision was tested 4 times, with 4 passes)
207159 o
207160 (this exact revision hasn't been tested)
207161 o
207162 x (tested once, failed)
207163 o (tested once, passed)
207164 xxxxoo (this revision was tested 6 times, with 4 failures and 2 passes)
207165 x
207166 xo
207167
207168 xx
207169 xx

r207161 is a contributors.json change
r207160 is purely a WebKit1 change
r207159 is a change to iOS 9 built apps that could theoretically have a bug, but it just uses standard settings machinery, unlikely to be wrong.

[Alexey Proskuryakov]

It seems likely that it's not an obvious outright bug in the URL parser causing this, but some kind of subtle change (perhaps timing), but we still need to make the test work again.

[Ryan Haddad]

This test now fails 100% of the time on debug and it is flaky on release.

Here is the diff from a release build:
--- /Volumes/Data/slave/ios-simulator-11-release-tests-wk2/build/layout-test-results/platform/ios/ios/plugin/youtube-flash-plugin-iframe-expected.txt
+++ /Volumes/Data/slave/ios-simulator-11-release-tests-wk2/build/layout-test-results/platform/ios/ios/plugin/youtube-flash-plugin-iframe-actual.txt
@@ -15,8 +15,8 @@
 PASS document.querySelectorAll("iframe").length is 1
 PASS internals.shadowPseudoId(normalEmbedShadowRoot.firstChild) is "-webkit-plugin-replacement"
 PASS normalEmbedShadowRoot.firstChild.firstChild.tagName is "IFRAME"
-PASS internals.shadowPseudoId(objectEmbedShadowRoot.firstChild) is "-webkit-plugin-replacement"
-PASS objectEmbedShadowRoot.firstChild.firstChild.tagName is "IFRAME"
+FAIL internals.shadowPseudoId(objectEmbedShadowRoot.firstChild) should be -webkit-plugin-replacement. Threw exception TypeError: null is not an object (evaluating 'objectEmbedShadowRoot.firstChild')
+FAIL objectEmbedShadowRoot.firstChild.firstChild.tagName should be IFRAME. Threw exception TypeError: null is not an object (evaluating 'objectEmbedShadowRoot.firstChild')
 PASS internals.shadowPseudoId(objectNoEmbedShadowRoot.firstChild) is "-webkit-plugin-replacement"
 PASS objectNoEmbedShadowRoot.firstChild.firstChild.tagName is "IFRAME"
 PASS internals.shadowPseudoId(normalEmbedInIframeShadowRoot.firstChild) is "-webkit-plugin-replacement"

[Matt Lewis]

This is still happening as of 12/13/2017

Updated Expectations:
https://trac.webkit.org/changeset/225867/webkit