media/track/track-in-band-cues-added-once.html flakily fails

[Alexey Proskuryakov]

media/track/track-in-band-cues-added-once.html frequently fails on bots:

@@ -18,7 +18,6 @@
 EVENT(change)
 RUN(video.play())
 
-RUN(video.pause())
 RUN(video.play())
 
 RUN(video.pause())

http://webkit-test-results.appspot.com/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=media%2Ftrack%2Ftrack-in-band-cues-added-once.html

[Alexey Proskuryakov]

I can reproduce locally with:

run-webkit-tests --no-retry -2 media/track/track-in-band-cues-added-once.html --repeat 100 -f

[Alexey Proskuryakov]

The test is quite complex, I'm not sure if this failure discovers a real bug, or a test shortcoming.

Marked as flaky in http://trac.webkit.org/r180868.

[Alexey Proskuryakov]

A different type of failure:

@@ -18,6 +18,8 @@
 EVENT(change)
 RUN(video.play())
 
+RUN(video.play())
+
 RUN(video.pause())
 RUN(video.play())

[Ryan Haddad]

(In reply to comment #3)
> A different type of failure:
> 
> @@ -18,6 +18,8 @@
>  EVENT(change)
>  RUN(video.play())
>  
> +RUN(video.play())
> +
>  RUN(video.pause())
>  RUN(video.play())

The test was recently re-enabled on iOS, and it mirrors this flaky failure https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=media%2Ftrack%2Ftrack-in-band-cues-added-once.html

Marked as flaky on ios-simulator-wk2 in http://trac.webkit.org/projects/webkit/changeset/204109

[Matt Lewis]

The test is now starting to Time out according to the Flakiness Dashboard on all macOS Debug.

Diff:
--- /Volumes/Data/slave/sierra-debug-tests-wk2/build/layout-test-results/media/track/track-in-band-cues-added-once-expected.txt
+++ /Volumes/Data/slave/sierra-debug-tests-wk2/build/layout-test-results/media/track/track-in-band-cues-added-once-actual.txt
@@ -1,26 +1,5 @@
+#PID UNRESPONSIVE - com.apple.WebKit.WebContent.Development (pid 48403)
+FAIL: Timed out waiting for notifyDone to be called
 
-In-band text tracks.
-
-EVENT(addtrack)
-EXPECTED (event.track == 'video.textTracks[0]') OK
-
-EVENT(canplaythrough)
-
-** Check initial in-band track states
-EXPECTED (video.textTracks.length == '1') OK
-RUN(inbandTrack1 = video.textTracks[0])
-EXPECTED (inbandTrack1.mode == 'disabled') OK
-EXPECTED (inbandTrack1.cues == 'null') OK
-EXPECTED (inbandTrack1.language == 'en') OK
-EXPECTED (inbandTrack1.kind == 'captions') OK
-
-RUN(inbandTrack1.mode = 'showing')
-EVENT(change)
-RUN(video.play())
-
-RUN(video.pause())
-RUN(video.play())
-
-RUN(video.pause())
-END OF TEST
-
+#EOF
+#EOF

[Matt Lewis]

According to the flakiness dashboard, this test has not passed as far back as the data goes. The expectations were set to [ Pass Failure ]. I adjusted the test expectations in r218410 https://trac.webkit.org/changeset/218410/webkit

[Darin Adler]

media/track/track-in-band-cues-added-once.html is passing on my Catalina machine, not failing nor timing out

[Alexey Proskuryakov]

It's still failing a lot on bots: https://results.webkit.org/?suite=layout-tests&test=media%2Ftrack%2Ftrack-in-band-cues-added-once.html

[Darin Adler]

That bot result looks like [ Pass Failure ].

The TestExpectations file says [ Failure Timeout ].

My comment wasn’t an effort to get "Failure" removed.

[Alexey Proskuryakov]

Updated TestExpectations in http://trac.webkit.org/r253846

[Darin Adler]

Thanks, that's great. I would have done it myself if I was a little more sure.