[ Mac WK1 ] Layout Test media/modern-media-controls/compact-media-controls/compact-media-controls-layout.html is a flaky text diff failure

• Component: Media
• Status: RESOLVED FIXED

[Shawn Roberts]

The following layout test is a flaky failure on Mac WK1

media/modern-media-controls/compact-media-controls/compact-media-controls-layout.html

Probable cause:

Bisecting down to build 230006. Changes were made to media control layout tests and now is a flaky text diff failure

Reproducible with:

run-webkit-tests t230006 media/modern-media-controls/compact-media-controls/compact-media-controls-layout.html --iterations 1000 -f -1

Flakiness Dashboard:

http://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=media%2Fmodern-media-controls%2Fcompact-media-controls%2Fcompact-media-controls-layout.html

Diff: 

--- /Volumes/Data/slave/mojave-release-tests-wk1/build/layout-test-results/media/modern-media-controls/compact-media-controls/compact-media-controls-layout-expected.txt
+++ /Volumes/Data/slave/mojave-release-tests-wk1/build/layout-test-results/media/modern-media-controls/compact-media-controls/compact-media-controls-layout-actual.txt
@@ -1,5 +1,5 @@
+CONSOLE MESSAGE: line 140: Button failed to load, iconName = InvalidCompact, layoutTraits = 8, src = InvalidCompact.pdf
 CONSOLE MESSAGE: line 140: Button failed to load, iconName = PlayCompact, layoutTraits = 8, src = PlayCompact.pdf
-CONSOLE MESSAGE: line 140: Button failed to load, iconName = InvalidCompact, layoutTraits = 8, src = InvalidCompact.pdf
 CONSOLE MESSAGE: line 140: Button failed to load, iconName = ActivityIndicatorSpriteCompact, layoutTraits = 8, src = [e-mail]
 Testing the CompactMediaControls layout.

[Radar WebKit Bug Importer]

<rdar://problem/48012947>

[Shawn Roberts]

Created attachment 362443 [details]
Patch

[EWS Watchlist]

Comment on attachment 362443 [details]
Patch

Attachment 362443 [details] did not pass win-ews (win):
Output: https://webkit-queues.webkit.org/results/11213240

New failing tests:
js/dom/custom-constructors.html

[EWS Watchlist]

Created attachment 362472 [details]
Archive of layout-test-results from ews202 for win-future

The attached test failures were seen while running run-webkit-tests on the win-ews.
Bot: ews202  Port: win-future  Platform: CYGWIN_NT-6.1-2.10.0-0.325-5-3-x86_64-64bit

[Shawn Roberts]

Created attachment 362536 [details]
Update TestExpectations

[Truitt Savell]

Comment on attachment 362536 [details]
Update TestExpectations

Clearing flags on attachment: 362536

Committed r241835: <https://trac.webkit.org/changeset/241835>

[Shawn Roberts]

Additional repro steps:

Searching on Trac I found the test was originally created in r228718 and was 100% failure. Test was modified 230895 and was a flaky failure. Continued to bisect to find exact regression point as follows.


242640 --iterations 1000 --exit-after-n-failures=5: 49 tests ran as expected, 5 didn't (946 didn't run)
230895 4 out of 1000 failures
230894 3 out of 1000 failures
230350 3 out of 1000 failures
230078 2 out of 1000 failures
230006 3 out of 1000 failures

230005 100% Failure
230004 100% Failure
230002 100% Failure
229998 100% Failure
229990 100% Failure
229974 100% Failure
229931 100% Failure
229931 100% Failure 
229806 100% Failure
228718 100% Failure

[Shawn Roberts]

Correction, is not a regression. Code changed in 230006.

Test was added in https://trac.webkit.org/changeset/228718/webkit and likely has been a flaky failure since then.

[Antoine Quint]

Created attachment 393885 [details]
Patch

[Antoine Quint]

Committed r258656: <https://trac.webkit.org/changeset/258656>

[Aakash Jain]

(In reply to Antoine Quint from comment #10)
> Committed r258656: <https://trac.webkit.org/changeset/258656>

media/modern-media-controls/compact-media-controls/compact-media-controls-layout.html seems to be consistently failing on iOS after r258656.

History: https://results.webkit.org/?suite=layout-tests&test=media%2Fmodern-media-controls%2Fcompact-media-controls%2Fcompact-media-controls-layout.html

[Antoine Quint]

The iOS-specific test expectations have been removed in r258688.