Bug 165394 - [GStreamer][MSE] Support multiple streams per SourceBuffer and dynamic audio/video/text type changes
Summary: [GStreamer][MSE] Support multiple streams per SourceBuffer and dynamic audio/...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Platform (show other bugs)
Version: WebKit Local Build
Hardware: Unspecified Unspecified
: P2 Enhancement
Assignee: Nobody
URL:
Keywords:
: 217580 (view as bug list)
Depends on: 229072
Blocks: 167107
  Show dependency treegraph
 
Reported: 2016-12-05 11:46 PST by Enrique Ocaña
Modified: 2022-08-05 14:43 PDT (History)
6 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Enrique Ocaña 2016-12-05 11:46:32 PST
The current GStreamer MSE platform implementation doesn't support more than one stream per SourceBuffer.

Caps changes are managed as stream quality changes in order to support adaptive streaming, but that's all. An audio/video/text stream can't be reconverted to other type. This feature is required for LayoutTests/media/media-source/media-source-seek-detach-crash.html to work properly.
Comment 1 Alicia Boya García 2017-10-10 15:12:25 PDT
> An audio/video/text stream can't be reconverted to other type. This feature
> is required for
> LayoutTests/media/media-source/media-source-seek-detach-crash.html to work
> properly.

Note that MSE not only does not require support for converting streams to other
formats, but even forbids it per spec:

https://www.w3.org/TR/media-source/#sourcebuffer-init-segment-received

3. If the first initialization segment received flag is true, then run the
following steps:

  1. Verify the following properties. If any of the checks fail then run the
append error algorithm and abort these steps.

     * The number of audio, video, and text tracks match what was in the first
       initialization segment.

     * The codecs for each track, match what was specified in the first
       initialization segment.

     * If more than one track for a single type are present (e.g., 2 audio
       tracks), then the Track IDs match the ones in the first initialization
       segment.


What the test is checking is that if that any of these conditions is met the
error is handled gracefully, which is not currently the case.

The test expectation should be changed from [ Skip ] to [ Crash ] and the bug
eventually fixed.
Comment 2 Philippe Normand 2020-10-15 05:10:50 PDT
At least one stream in the wild hitting this issue: https://peer.tube/videos/watch/5a370009-1868-46de-a5c2-2b34405fc5c7
Comment 3 Adrian Perez 2020-10-15 07:23:24 PDT
*** Bug 217580 has been marked as a duplicate of this bug. ***
Comment 4 Enrique Ocaña 2021-11-16 12:37:22 PST
The multiple streams per SourceBuffer has been implemented in r281440. Peertube videos work now, and I ensured that they have multiple streams per SourceBuffer. Resolving this bug as fixed.
Comment 5 Enrique Ocaña 2021-11-16 12:42:03 PST
...but the stream type reconversion (MSE changeType()[1]) is still missing.

[1] https://w3c.github.io/media-source/#dom-sourcebuffer-changetype
Comment 6 Alicia Boya García 2022-08-05 14:43:05 PDT
There is no reason to have this bug set as REOPENED.

(In reply to Enrique Ocaña from comment #5)
> ...but the stream type reconversion (MSE changeType()[1]) is still missing.
> 
> [1] https://w3c.github.io/media-source/#dom-sourcebuffer-changetype

A changeType implementation is a completely different effort. Note it wasn't even in the spec when this original bug was open.

Also, as I explained "An audio/video/text stream can't be reconverted to other type" is a requirement of the spec. Even with changeType, SourceBuffers can't change from video into audio. The number and type of tracks must much, but container and codecs can differ. https://w3c.github.io/media-source/#ref-for-dfn-first-initialization-segment-received-flag-2

Feel free to open a separate bug for a changeType implementation in MSE GStreamer. That is a feature that is in the cooking pot, but to which it's hard to get to while being overwhelmed with bug fixing.