Bug 187557 - REGRESSION: [ MacOS WK1 & iOS ] Layout Tests in media/media-fragments/ are flaky
Summary: REGRESSION: [ MacOS WK1 & iOS ] Layout Tests in media/media-fragments/ are flaky
Status: REOPENED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
: 187522 187659 (view as bug list)
Depends on:
Blocks:
 
Reported: 2018-07-11 10:57 PDT by Truitt Savell
Modified: 2020-04-10 10:16 PDT (History)
8 users (show)

See Also:


Attachments
Update TestExpectations (2.18 KB, patch)
2018-07-13 14:44 PDT, Truitt Savell
commit-queue: commit-queue-
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Truitt Savell 2018-07-11 10:57:33 PDT
The following layout tests are flaky on MacOS WK1 in:

media/media-fragments/

Probable cause:

All the tests have been flaky for quite some time. They are highly intermittent and all fail in very similar ways. 

Example of failure output:
--- /Volumes/Data/slave/highsierra-debug-tests-wk1/build/layout-test-results/media/media-fragments/TC0078-expected.txt
+++ /Volumes/Data/slave/highsierra-debug-tests-wk1/build/layout-test-results/media/media-fragments/TC0078-actual.txt
@@ -7,6 +7,6 @@
 EXPECTED (video.currentTime == '3') OK
 RUN(video.play())
 EVENT(pause)
-EXPECTED (video.currentTime - fragmentEndTime <= '0.5') OK
+EXPECTED (video.currentTime - fragmentEndTime <= '0.5'), OBSERVED '0.66' FAIL
 END OF TEST
 

Flakiness Dashboard:

http://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=media%2Fmedia-fragments%2F
Comment 1 Truitt Savell 2018-07-13 14:29:43 PDT
*** Bug 187522 has been marked as a duplicate of this bug. ***
Comment 2 Dawei Fenton (:realdawei) 2018-07-13 14:30:37 PDT
*** Bug 187659 has been marked as a duplicate of this bug. ***
Comment 3 Truitt Savell 2018-07-13 14:35:30 PDT
Looking further me and David have found about 12 tests within the media/media-fragments/ are flaky failures for MacOS Debug WK1

media/media-fragments/TC0005.html
media/media-fragments/TC0024.html
media/media-fragments/TC0037.html
media/media-fragments/TC0078.html
media/media-fragments/TC0079.html
media/media-fragments/TC0085.html
media/media-fragments/TC0086.html
media/media-fragments/TC0087.html
media/media-fragments/TC0088.html
media/media-fragments/TC0089.html
media/media-fragments/TC0090.html
media/media-fragments/TC0091.html
Comment 4 Truitt Savell 2018-07-13 14:44:43 PDT
Created attachment 344984 [details]
Update TestExpectations
Comment 5 WebKit Commit Bot 2018-07-13 15:26:16 PDT
Comment on attachment 344984 [details]
Update TestExpectations

Clearing flags on attachment: 344984

Committed r233819: <https://trac.webkit.org/changeset/233819>
Comment 6 WebKit Commit Bot 2018-07-13 15:26:18 PDT
All reviewed patches have been landed.  Closing bug.
Comment 7 Truitt Savell 2018-07-13 15:27:26 PDT
Reopening to track issue
Comment 8 WebKit Commit Bot 2018-07-13 15:28:52 PDT
Comment on attachment 344984 [details]
Update TestExpectations

Rejecting attachment 344984 [details] from commit-queue.

Failed to run "['/Volumes/Data/EWS/WebKit/Tools/Scripts/webkit-patch', '--status-host=webkit-queues.webkit.org', '--bot-id=webkit-cq-03', 'apply-attachment', '--no-update', '--non-interactive', 344984, '--port=mac']" exit_code: 2 cwd: /Volumes/Data/EWS/WebKit

Logging in as commit-queue@webkit.org...
Fetching: https://bugs.webkit.org/attachment.cgi?id=344984&action=edit
Fetching: https://bugs.webkit.org/show_bug.cgi?id=187557&ctype=xml&excludefield=attachmentdata
Processing 1 patch from 1 bug.
Processing patch 344984 from bug 187557.
Fetching: https://bugs.webkit.org/attachment.cgi?id=344984
Failed to run "[u'/Volumes/Data/EWS/WebKit/Tools/Scripts/svn-apply', '--force']" exit_code: 1 cwd: /Volumes/Data/EWS/WebKit

Parsed 2 diffs from patch file(s).
patching file LayoutTests/ChangeLog
Hunk #1 succeeded at 1 with fuzz 1.
patching file LayoutTests/platform/mac-wk1/TestExpectations
Hunk #1 FAILED at 567.
1 out of 1 hunk FAILED -- saving rejects to file LayoutTests/platform/mac-wk1/TestExpectations.rej

Failed to run "[u'/Volumes/Data/EWS/WebKit/Tools/Scripts/svn-apply', '--force']" exit_code: 1 cwd: /Volumes/Data/EWS/WebKit

Full output: https://webkit-queues.webkit.org/results/8529954
Comment 9 Radar WebKit Bug Importer 2018-07-13 15:53:27 PDT
<rdar://problem/42184800>
Comment 10 Truitt Savell 2018-09-24 10:20:53 PDT
Added test expectations to iOS as these tests are flakey on iOS 12

Committed in: https://trac.webkit.org/changeset/236408/webkit
Comment 11 Truitt Savell 2018-09-24 13:31:52 PDT
Additional tests found to be flakey failures on iOS 12:
media/media-fragments/TC0011.html
media/media-fragments/TC0053.html
media/media-fragments/TC0036.html
media/media-fragments/TC0039.html
media/media-fragments/TC0072.html
media/media-fragments/TC0006.html
media/media-fragments/TC0038.html
media/media-fragments/TC0052.html
media/media-fragments/TC0051.html
media/media-fragments/TC0054.html
media/media-fragments/TC0055.html
media/media-fragments/TC0011.html
media/media-fragments/TC0059.html
media/media-fragments/TC0035.html
Comment 12 Truitt Savell 2018-09-24 13:41:46 PDT
marked remaining media-fragment/ tests as flakey in https://trac.webkit.org/changeset/236426/webkit
Comment 13 Truitt Savell 2018-09-26 09:13:49 PDT
Marked two more tests as flakey in https://trac.webkit.org/changeset/236504/webkit

Tests Marked:
media/media-fragments/TC0014.html
media/media-fragments/TC0009.html
Comment 14 Jason Lawrence 2020-04-10 10:16:38 PDT
I have included media/media-fragments/TC0036.html in the list of flaky failures on Mac wk1. 

History:
https://results.webkit.org/?suite=layout-tests&test=media%2Fmedia-fragments%2FTC0036.html&version_name=Mojave&style=debug&flavor=wk2&limit=50000

Expectations: 
https://trac.webkit.org/changeset/259881/webkit