142152 2015-03-01 12:37:24 -0800 media/track/track-in-band-cues-added-once.html flakily fails 2019-12-20 14:55:19 -0800 1 1 1 Unclassified WebKit Media 528+ (Nightly build) Unspecified Unspecified NEW P2 Normal --- 1 ap webkit-unassigned darin eric.carlson jer.noble jlewis3 ryanhaddad oldest_to_newest 1073182 0 ap 2015-03-01 12:37:24 -0800 media/track/track-in-band-cues-added-once.html frequently fails on bots: @@ -18,7 +18,6 @@ EVENT(change) RUN(video.play()) -RUN(video.pause()) RUN(video.play()) RUN(video.pause()) http://webkit-test-results.appspot.com/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=media%2Ftrack%2Ftrack-in-band-cues-added-once.html 1073183 1 ap 2015-03-01 12:38:43 -0800 I can reproduce locally with: run-webkit-tests --no-retry -2 media/track/track-in-band-cues-added-once.html --repeat 100 -f 1073185 2 ap 2015-03-01 12:46:24 -0800 The test is quite complex, I'm not sure if this failure discovers a real bug, or a test shortcoming. Marked as flaky in http://trac.webkit.org/r180868. 1073363 3 ap 2015-03-02 10:16:18 -0800 A different type of failure: @@ -18,6 +18,8 @@ EVENT(change) RUN(video.play()) +RUN(video.play()) + RUN(video.pause()) RUN(video.play()) 1217292 4 ryanhaddad 2016-08-03 18:29:00 -0700 (In reply to comment #3) > A different type of failure: > > @@ -18,6 +18,8 @@ > EVENT(change) > RUN(video.play()) > > +RUN(video.play()) > + > RUN(video.pause()) > RUN(video.play()) The test was recently re-enabled on iOS, and it mirrors this flaky failure https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=media%2Ftrack%2Ftrack-in-band-cues-added-once.html Marked as flaky on ios-simulator-wk2 in http://trac.webkit.org/projects/webkit/changeset/204109 1316346 5 jlewis3 2017-06-06 17:55:39 -0700 The test is now starting to Time out according to the Flakiness Dashboard on all macOS Debug. Diff: --- /Volumes/Data/slave/sierra-debug-tests-wk2/build/layout-test-results/media/track/track-in-band-cues-added-once-expected.txt +++ /Volumes/Data/slave/sierra-debug-tests-wk2/build/layout-test-results/media/track/track-in-band-cues-added-once-actual.txt @@ -1,26 +1,5 @@ +#PID UNRESPONSIVE - com.apple.WebKit.WebContent.Development (pid 48403) +FAIL: Timed out waiting for notifyDone to be called -In-band text tracks. - -EVENT(addtrack) -EXPECTED (event.track == 'video.textTracks[0]') OK - -EVENT(canplaythrough) - -** Check initial in-band track states -EXPECTED (video.textTracks.length == '1') OK -RUN(inbandTrack1 = video.textTracks[0]) -EXPECTED (inbandTrack1.mode == 'disabled') OK -EXPECTED (inbandTrack1.cues == 'null') OK -EXPECTED (inbandTrack1.language == 'en') OK -EXPECTED (inbandTrack1.kind == 'captions') OK - -RUN(inbandTrack1.mode = 'showing') -EVENT(change) -RUN(video.play()) - -RUN(video.pause()) -RUN(video.play()) - -RUN(video.pause()) -END OF TEST - +#EOF +#EOF 1320132 6 jlewis3 2017-06-16 13:42:42 -0700 According to the flakiness dashboard, this test has not passed as far back as the data goes. The expectations were set to [ Pass Failure ]. I adjusted the test expectations in r218410 https://trac.webkit.org/changeset/218410/webkit 1600773 7 darin 2019-12-20 09:51:36 -0800 media/track/track-in-band-cues-added-once.html is passing on my Catalina machine, not failing nor timing out 1600786 8 ap 2019-12-20 10:05:36 -0800 It's still failing a lot on bots: https://results.webkit.org/?suite=layout-tests&test=media%2Ftrack%2Ftrack-in-band-cues-added-once.html 1600884 9 darin 2019-12-20 12:55:26 -0800 That bot result looks like [ Pass Failure ]. The TestExpectations file says [ Failure Timeout ]. My comment wasn’t an effort to get "Failure" removed. 1600956 10 ap 2019-12-20 14:52:55 -0800 Updated TestExpectations in http://trac.webkit.org/r253846 1600960 11 darin 2019-12-20 14:55:19 -0800 Thanks, that's great. I would have done it myself if I was a little more sure.