Bug 251771 – Overlapping VTTCues carried by InbandTextTrack (LegibleOutput) does not handle correctly

• Status: RESOLVED
• Resolution: FIXED
• Component: Media

[Taichiro Ishikawa]

Created attachment 464857 [details]
Sample Stream for Multiple Cue

Background:
LegibleOutput sends multiple vttc atom (VTTCue in isobmff) in one nativeSample. (nativeSamples length = 1, but multiple vttc in the nativeSample)
But, InbandTextTrackPrivateAVF::processNativeSamples handles only first vttc each callback and rest vttc still in buffer.
So, rest cues are not handles in same callback and process in next callback and use next timing information, that wrongly.

Issue:
Overlapping Timing VTTCues in InbandTextTrack does not handle correctly

Reproduction steps:
Play m3u8 in attachmend zipfile.

Actual behavior:
VTTCue[0]: startTime=0, endTIme=2
VTTCue[1]: startTime=0.5786666666666667, endTime=0.5786666666666667

Expected behavior:
VTTCue[0]: startTime=0, endTIme=2
VTTCue[1]: startTime=0, endTime=2

Impact:
Video Streaming Service not capable of using vertical splitted (top and bottom) cues.

[Taichiro Ishikawa]

Created attachment 464859 [details]
patch for handle all vttc atom in nativeSamples

I wrote a patch for handle all vttc atom in InbandTextTrackPrivateAVF::processNativeSamples (Simply, add while loop)
It is an example, might be memory issue.

This patch works as expected for me.

[Taichiro Ishikawa]

I've hosted same sample stream on https://taichiro-ishikawa-play.github.io/hls-sample-stream/multiple-cue-for-webkit/master.m3u8
Please check startTime and endTime for each cue.

[Taichiro Ishikawa]

I also raised feedback in Feedback Assistant. (https://feedbackassistant.apple.com/feedback/11985197).

And, add another sample stream. describe current webkit behavior below.
https://taichiro-ishikawa-play.github.io/hls-sample-stream/multiple-cue-for-webkit2/master.m3u8

Original VTT Content in sample HLS VTT Fragment:
VTTCue[0]: text="line 1 (0)", startTime=0, endTime=2
VTTCue[1]: text="line 2 (1)", startTime=0, endTime=2
VTTCue[2]: text="line 1 (2)", startTime=2, endTime=4
VTTCue[3]: text="line 2 (3)", startTime=2.5, endTime=3.5
VTTCue[4]: text="line 3 (4)", startTime=3, endTime=4

Current webkit HLS TextTrack VTTCue result:
VTTCue[0]: text="line 1 (0)", startTime=0, endTime=2
VTTCue[1]: text="line 2 (1)", startTime=1.9996666666666667, endTime=2.4996666666666667
VTTCue[2]: text="line 1 (2)", startTime=2.4996666666666667, endTime=2.9996666666666667
VTTCue[3]: text="line 1 (2)", startTime=2.9996666666666667, endTime=3.4996666666666667
VTTCue[4]: text="line 2 (3)", startTime=3.4996666666666667, endTime=3.9996666666666667
VTTCue[5]: text="line 1 (2)", startTime=3.9996666666666667, endTime=3.9996666666666667

It is not same as Original VTT Content.
Because InbandTextTrackPrivateAVF::processNativeSamples handles only first vttc atom in legibleoutput callback.

Patched webkit HLS TextTrack VTTCue result:
VTTCue[0]: text="line 1 (0)", startTime=0, endTime=2
VTTCue[1]: text="line 2 (1)", startTime=0, endTime=2
VTTCue[2]: text="line 1 (2)", startTime=1.9996666666666667, endTime=2.4996666666666667
VTTCue[3]: text="line 1 (2)", startTime=2.4996666666666667, endTime=2.9996666666666667
VTTCue[4]: text="line 2 (3)", startTime=2.4996666666666667, endTime=2.9996666666666667
VTTCue[5]: text="line 1 (2)", startTime=2.9996666666666667, endTime=3.4996666666666667
VTTCue[6]: text="line 2 (3)", startTime=2.9996666666666667, endTime=3.4996666666666667
VTTCue[7]: text="line 3 (4)", startTime=2.9996666666666667, endTime=3.4996666666666667
VTTCue[8]: text="line 1 (2)", startTime=3.4996666666666667, endTime=3.9996666666666667
VTTCue[9]: text="line 3 (4)", startTime=3.4996666666666667, endTime=3.9996666666666667

It is fragmented, so concatenate, it same as Original VTT Content.

[Radar WebKit Bug Importer]

<rdar://problem/105372930>

[Taichiro Ishikawa]

Created attachment 464991 [details]
Sample Stream for Multiple Cue (5 cue overlapping)

I attach another sample stream in zip (same as above m3u8). 

This incorrect behavior since over 5 years ago. 
https://developer.apple.com/forums/thread/75821

I hope to fix this incorrect behavior and wish to use separate positioning cue in webkit.

[Taichiro Ishikawa]

This issue is still reproducible on Version 16.4.
How is the progress?

[Eric Carlson]

Pull request: https://github.com/WebKit/WebKit/pull/15982

[EWS]

Committed 266260@main (6f89f7891c5c): <https://commits.webkit.org/266260@main>

Reviewed commits have been landed. Closing PR #15982 and removing active labels.