Bug 187557 – REGRESSION: [ MacOS WK1 & iOS ] Layout Tests in media/media-fragments/ are flaky

• Status: RESOLVED
• Resolution: WORKSFORME
• Component: Tools / Tests

[Truitt Savell]

The following layout tests are flaky on MacOS WK1 in:

media/media-fragments/

Probable cause:

All the tests have been flaky for quite some time. They are highly intermittent and all fail in very similar ways. 

Example of failure output:
--- /Volumes/Data/slave/highsierra-debug-tests-wk1/build/layout-test-results/media/media-fragments/TC0078-expected.txt
+++ /Volumes/Data/slave/highsierra-debug-tests-wk1/build/layout-test-results/media/media-fragments/TC0078-actual.txt
@@ -7,6 +7,6 @@
 EXPECTED (video.currentTime == '3') OK
 RUN(video.play())
 EVENT(pause)
-EXPECTED (video.currentTime - fragmentEndTime <= '0.5') OK
+EXPECTED (video.currentTime - fragmentEndTime <= '0.5'), OBSERVED '0.66' FAIL
 END OF TEST
 

Flakiness Dashboard:

http://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=media%2Fmedia-fragments%2F

[Truitt Savell]

*** Bug 187522 has been marked as a duplicate of this bug. ***

[Dawei Fenton (:realdawei)]

*** Bug 187659 has been marked as a duplicate of this bug. ***

[Truitt Savell]

Looking further me and David have found about 12 tests within the media/media-fragments/ are flaky failures for MacOS Debug WK1

media/media-fragments/TC0005.html
media/media-fragments/TC0024.html
media/media-fragments/TC0037.html
media/media-fragments/TC0078.html
media/media-fragments/TC0079.html
media/media-fragments/TC0085.html
media/media-fragments/TC0086.html
media/media-fragments/TC0087.html
media/media-fragments/TC0088.html
media/media-fragments/TC0089.html
media/media-fragments/TC0090.html
media/media-fragments/TC0091.html

[Truitt Savell]

Created attachment 344984 [details]
Update TestExpectations

[WebKit Commit Bot]

Comment on attachment 344984 [details]
Update TestExpectations

Clearing flags on attachment: 344984

Committed r233819: <https://trac.webkit.org/changeset/233819>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Truitt Savell]

Reopening to track issue

[WebKit Commit Bot]

Comment on attachment 344984 [details]
Update TestExpectations

Rejecting attachment 344984 [details] from commit-queue.

Failed to run "['/Volumes/Data/EWS/WebKit/Tools/Scripts/webkit-patch', '--status-host=webkit-queues.webkit.org', '--bot-id=webkit-cq-03', 'apply-attachment', '--no-update', '--non-interactive', 344984, '--port=mac']" exit_code: 2 cwd: /Volumes/Data/EWS/WebKit

Logging in as [e-mail]...
Fetching: https://bugs.webkit.org/attachment.cgi?id=344984&action=edit
Fetching: https://bugs.webkit.org/show_bug.cgi?id=187557&ctype=xml&excludefield=attachmentdata
Processing 1 patch from 1 bug.
Processing patch 344984 from bug 187557.
Fetching: https://bugs.webkit.org/attachment.cgi?id=344984
Failed to run "[u'/Volumes/Data/EWS/WebKit/Tools/Scripts/svn-apply', '--force']" exit_code: 1 cwd: /Volumes/Data/EWS/WebKit

Parsed 2 diffs from patch file(s).
patching file LayoutTests/ChangeLog
Hunk #1 succeeded at 1 with fuzz 1.
patching file LayoutTests/platform/mac-wk1/TestExpectations
Hunk #1 FAILED at 567.
1 out of 1 hunk FAILED -- saving rejects to file LayoutTests/platform/mac-wk1/TestExpectations.rej

Failed to run "[u'/Volumes/Data/EWS/WebKit/Tools/Scripts/svn-apply', '--force']" exit_code: 1 cwd: /Volumes/Data/EWS/WebKit

Full output: https://webkit-queues.webkit.org/results/8529954

[Radar WebKit Bug Importer]

<rdar://problem/42184800>

[Truitt Savell]

Added test expectations to iOS as these tests are flakey on iOS 12

Committed in: https://trac.webkit.org/changeset/236408/webkit

[Truitt Savell]

Additional tests found to be flakey failures on iOS 12:
media/media-fragments/TC0011.html
media/media-fragments/TC0053.html
media/media-fragments/TC0036.html
media/media-fragments/TC0039.html
media/media-fragments/TC0072.html
media/media-fragments/TC0006.html
media/media-fragments/TC0038.html
media/media-fragments/TC0052.html
media/media-fragments/TC0051.html
media/media-fragments/TC0054.html
media/media-fragments/TC0055.html
media/media-fragments/TC0011.html
media/media-fragments/TC0059.html
media/media-fragments/TC0035.html

[Truitt Savell]

marked remaining media-fragment/ tests as flakey in https://trac.webkit.org/changeset/236426/webkit

[Truitt Savell]

Marked two more tests as flakey in https://trac.webkit.org/changeset/236504/webkit

Tests Marked:
media/media-fragments/TC0014.html
media/media-fragments/TC0009.html

[Jason Lawrence]

I have included media/media-fragments/TC0036.html in the list of flaky failures on Mac wk1. 

History:
https://results.webkit.org/?suite=layout-tests&test=media%2Fmedia-fragments%2FTC0036.html&version_name=Mojave&style=debug&flavor=wk2&limit=50000

Expectations: 
https://trac.webkit.org/changeset/259881/webkit

[Ben Schwartz]

This no longer appears to be occurring on iOS. macOS wk1 has also experienced a major reduction of cases, with one in the past few months on only two of the tests.

For this reason, I will consider this infrastructure flake and remove test expectations. Please file a new bug if issues persist. Tracking test expectations removal in bug #274990 (rdar://129080660).