Bug 157539 – Media Source Extensions - SourceBuffer "mode" support

• Component: Media
• Status: NEW

Created attachment 278535 [details]
Test files and screenshots.

https://developer.mozilla.org/en-US/docs/Web/API/SourceBuffer/mode
https://w3c.github.io/media-source/#idl-def-AppendMode

When playing a MIME that would default to "segments" (eg AAC-LC, 'audio/mp4; codecs="mp4a.40.2"') changing the mode to "sequence" does not have the intended effect.

I can buffer a 4 minute AAC-LC file just fine if I do not touch the mode attribute; if I change it to "sequence" before buffering any data then somehow only about a half second gets buffered (according to the SourceBuffer itself). I would expect the entire file to be buffered exactly the same as if it was in "segments" mode (for a single song).

Note that the difference in behavior comes from buffering two files. Say I buffer two AAC-LC songs, one is 4 minutes and the second is 3 minutes. Also assume I do not change timestampOffset.
- In sequence mode the data will "overlap" and the total amount buffered will be 4 minutes. The first 3 minutes will be the second song, the last minute will be the last minute of the first song.
- In segments mode the data will not overlap and the total amount buffered will be 7 minutes. The first 4 minutes are the first song, the last 3 minutes are the second song.


This works as intended on Chrome (50) and FireFox.


Why is this useful?

For developers who are migrating audio formats and who's code is built around the "sequence" mode. Note that MP3/MPEG always acts as if it is in "sequence" mode. So if a developer then wants to switch to AAC playback the easiest thing to do on their end is make sure mode is always "sequence" rather then "segments". As this is not properly supported it is more work for developers to abstract this away in their JavaScript themselves if they wish to support Safari.


Attached is a zip with example code and some screenshots of the issue in Chrome, FireFox, and Safari (note it is working in Chrome + FF as stated before). The example screenshots used two audio files as described above, though you can test with just one file and still not get anything buffered.

Tried both Safari 9.1 11601.5.17.1 and Webkit Nightly on MacOS. Both do not work (see screenshot of Safari in the zip - its the same on both). The source buffer says there is a half second buffered when in segments mode despite giving over 3 minutes of audio data.

Bumping to see if there's an update and also to note that my example is backwards:

 In sequence mode the data will "overlap" and the total amount buffered will be 4 minutes. The first 3 minutes will be the second song, the last minute will be the last minute of the first song.
- In segments mode the data will not overlap and the total amount buffered will be 7 minutes. The first 4 minutes are the first song, the last 3 minutes are the second song.


Should reverse sequence and segments mode:

 In *segments* mode the data will "overlap" and the total amount buffered will be 4 minutes. The first 3 minutes will be the second song, the last minute will be the last minute of the first song.
- In *sequence* mode the data will not overlap and the total amount buffered will be 7 minutes. The first 4 minutes are the first song, the last 3 minutes are the second song.

https://www.w3.org/TR/media-source/#idl-def-AppendMode

[Tom]

Any update on this? We have a similar issue.


We want to ignore the timestamps that are inside the mp4 fragments and append each fragment directly after the previous fragment in the buffer, but it seems safari is always appending at the internal fragment timestamp, with the timestampOffset being added to this.

The timestampOffset should be used instead of the internal one

[Radar WebKit Bug Importer]

<rdar://problem/42080313>

[Dustin Kerstein]

Tom, I ran into this issue as well. As an ugly workaround (for video in my use-case), I discovered that by setting:

sourceBuffer = ms.addSourceBuffer('audio/mpeg'); 
sourceBuffer.mode = 'sequence';

I was able to trick Safari into setting m_shouldGenerateTimestamps (https://github.com/WebKit/webkit/blob/29271ffbec500cd9c92050fcc0e613adffd0ce6a/Source/WebCore/Modules/mediasource/SourceBuffer.cpp)

Chrome and Firefox seem to handle 'sequence' mode + setting timestampOffset differently. Here's a demo that works (and I believe is spec-compliant) in Chrome and Firefox but is broken in Safari - https://jsfiddle.net/1qLsbtfw

And here's a demo using that Safari workaround described above - https://jsfiddle.net/wsm8gh7e

WebKit devs, is the timestampOffset being correctly handled in 'sequence' mode?