Bug 222260 – [MSE] Media segment is incorrectly dropped when using negative timestampOffset

• Component: Media
• Status: RESOLVED FIXED

[Jean-Yves Avenard [:jya]]

Here is a small test:
https://jyavenard.github.io/htmltests/tests/mse_mp4/test_audiooffset.html

This adds roughly 1s of audio (975ms), the media segment is made of 21 samples ranging from [0.9752380952380952, 1.9504761904761905); each sample is 46.44ms long.

When adding them to an audio source buffer with timestampOffset set to -1; you would expect to see [0, 0.9504761904761905] (as both Firefox and Chrome do) now being buffered, but you get instead: [0.4860770975056689, 0.9504761904761905)


Per spec:
https://w3c.github.io/media-source/#sourcebuffer-coded-frame-processing
"8. If presentation timestamp is less than appendWindowStart, then set the need random access point flag to true, drop the coded frame, and jump to the top of the loop to start processing the next coded frame."
"Some implementations MAY choose to collect coded frames with presentation timestamp less than appendWindowEnd and frame end timestamp greater than appendWindowEnd and use them to generate a splice across the portion of the collected coded frames within the append window at time of collection, and the beginning portion of later processed frames which only partially overlap the end of the collected coded frames. Supporting this requires multiple decoders or faster than real-time decoding so for now this behavior will not be a normative requirement. In conjunction with collecting coded frames that span appendWindowStart, implementations MAY thus support gapless audio splicing. "

Now, even if audio splicing isn't supported; the only audio frame that should be dropped per the description above is the one with time [-0.024762,0.021678], but webkit drops 10 samples from 0.021678s to 0.486077s that shouldn't be.

[Jean-Yves Avenard [:jya]]

SourceBufferPrivateAVFObjC seems to pack the samples in block of 10 samples. So the first block of 10 samples is dropped.

[Jean-Yves Avenard [:jya]]

Created attachment 421288 [details]
Patch

[Jean-Yves Avenard [:jya]]

Created attachment 421300 [details]
Patch

[Eric Carlson]

Comment on attachment 421300 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=421300&action=review

> Source/WebCore/ChangeLog:11
> +        CoreMedia packs multiple audio frames together into a single CMSampleBuffer, this allows for faster processing and easier insertion into the track buffer tree.
> +        However, per mediasoure spec [1], a frame is to be dropped according to its start time and duration. So if only the beginning of the MediaSample was to be dropped, we would have incorrectly dropped the lot.

Since there is a typo to fix, please wrap these overly long lines people won't need to scroll to read the ChangeLog.

> Source/WebCore/ChangeLog:13
> +        Audio splicing isn't done yet, but this gets us closeer to it.

s/closeer/closer/

[Jean-Yves Avenard [:jya]]

Created attachment 421477 [details]
Patch

[EWS]

Committed r273461: <https://commits.webkit.org/r273461>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 421477 [details].

[Radar WebKit Bug Importer]

<rdar://problem/74721039>