Bug 194309 - [ Mac WK1 ] Layout Test media/modern-media-controls/compact-media-controls/compact-media-controls-layout.html is a flaky text diff failure
Summary: [ Mac WK1 ] Layout Test media/modern-media-controls/compact-media-controls/co...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Media (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Antoine Quint
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2019-02-05 15:11 PST by Shawn Roberts
Modified: 2020-03-19 02:13 PDT (History)
16 users (show)

See Also:


Attachments
Patch (1.50 KB, patch)
2019-02-19 15:18 PST, Shawn Roberts
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ews202 for win-future (12.86 MB, application/zip)
2019-02-19 19:18 PST, EWS Watchlist
no flags Details
Update TestExpectations (1.57 KB, patch)
2019-02-20 13:59 PST, Shawn Roberts
no flags Details | Formatted Diff | Diff
Patch (3.75 KB, patch)
2020-03-18 12:16 PDT, Antoine Quint
dino: review+
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Shawn Roberts 2019-02-05 15:11:45 PST
The following layout test is a flaky failure on Mac WK1

media/modern-media-controls/compact-media-controls/compact-media-controls-layout.html

Probable cause:

Bisecting down to build 230006. Changes were made to media control layout tests and now is a flaky text diff failure

Reproducible with:

run-webkit-tests t230006 media/modern-media-controls/compact-media-controls/compact-media-controls-layout.html --iterations 1000 -f -1

Flakiness Dashboard:

http://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=media%2Fmodern-media-controls%2Fcompact-media-controls%2Fcompact-media-controls-layout.html

Diff: 

--- /Volumes/Data/slave/mojave-release-tests-wk1/build/layout-test-results/media/modern-media-controls/compact-media-controls/compact-media-controls-layout-expected.txt
+++ /Volumes/Data/slave/mojave-release-tests-wk1/build/layout-test-results/media/modern-media-controls/compact-media-controls/compact-media-controls-layout-actual.txt
@@ -1,5 +1,5 @@
+CONSOLE MESSAGE: line 140: Button failed to load, iconName = InvalidCompact, layoutTraits = 8, src = InvalidCompact.pdf
 CONSOLE MESSAGE: line 140: Button failed to load, iconName = PlayCompact, layoutTraits = 8, src = PlayCompact.pdf
-CONSOLE MESSAGE: line 140: Button failed to load, iconName = InvalidCompact, layoutTraits = 8, src = InvalidCompact.pdf
 CONSOLE MESSAGE: line 140: Button failed to load, iconName = ActivityIndicatorSpriteCompact, layoutTraits = 8, src = ActivityIndicatorSpriteCompact@1x.png
 Testing the CompactMediaControls layout.
Comment 1 Radar WebKit Bug Importer 2019-02-12 13:33:21 PST
<rdar://problem/48012947>
Comment 2 Shawn Roberts 2019-02-19 15:18:09 PST
Created attachment 362443 [details]
Patch
Comment 3 EWS Watchlist 2019-02-19 19:18:35 PST
Comment on attachment 362443 [details]
Patch

Attachment 362443 [details] did not pass win-ews (win):
Output: https://webkit-queues.webkit.org/results/11213240

New failing tests:
js/dom/custom-constructors.html
Comment 4 EWS Watchlist 2019-02-19 19:18:46 PST
Created attachment 362472 [details]
Archive of layout-test-results from ews202 for win-future

The attached test failures were seen while running run-webkit-tests on the win-ews.
Bot: ews202  Port: win-future  Platform: CYGWIN_NT-6.1-2.10.0-0.325-5-3-x86_64-64bit
Comment 5 Shawn Roberts 2019-02-20 13:59:51 PST
Created attachment 362536 [details]
Update TestExpectations
Comment 6 Truitt Savell 2019-02-20 14:04:09 PST
Comment on attachment 362536 [details]
Update TestExpectations

Clearing flags on attachment: 362536

Committed r241835: <https://trac.webkit.org/changeset/241835>
Comment 7 Shawn Roberts 2019-03-08 15:19:44 PST
Additional repro steps:

Searching on Trac I found the test was originally created in r228718 and was 100% failure. Test was modified 230895 and was a flaky failure. Continued to bisect to find exact regression point as follows.


242640 --iterations 1000 --exit-after-n-failures=5: 49 tests ran as expected, 5 didn't (946 didn't run)
230895 4 out of 1000 failures
230894 3 out of 1000 failures
230350 3 out of 1000 failures
230078 2 out of 1000 failures
230006 3 out of 1000 failures

230005 100% Failure
230004 100% Failure
230002 100% Failure
229998 100% Failure
229990 100% Failure
229974 100% Failure
229931 100% Failure
229931 100% Failure 
229806 100% Failure
228718 100% Failure
Comment 8 Shawn Roberts 2019-04-03 14:04:05 PDT
Correction, is not a regression. Code changed in 230006.

Test was added in https://trac.webkit.org/changeset/228718/webkit and likely has been a flaky failure since then.
Comment 9 Antoine Quint 2020-03-18 12:16:56 PDT
Created attachment 393885 [details]
Patch
Comment 10 Antoine Quint 2020-03-18 12:22:37 PDT
Committed r258656: <https://trac.webkit.org/changeset/258656>
Comment 11 Aakash Jain 2020-03-18 18:18:48 PDT
(In reply to Antoine Quint from comment #10)
> Committed r258656: <https://trac.webkit.org/changeset/258656>

media/modern-media-controls/compact-media-controls/compact-media-controls-layout.html seems to be consistently failing on iOS after r258656.

History: https://results.webkit.org/?suite=layout-tests&test=media%2Fmodern-media-controls%2Fcompact-media-controls%2Fcompact-media-controls-layout.html
Comment 12 Antoine Quint 2020-03-19 02:13:55 PDT
The iOS-specific test expectations have been removed in r258688.