Bug 186391 – [GStreamer] Media capabilities configuration support

• Status: RESOLVED
• Resolution: FIXED
• Component: Platform

[Philippe Normand]

The use-case I have is that the decoder I use theoretically supports 1080p h.264 but in practice it works only with 720p currently. Also VPx is not supported yet. So the idea I had was to have a websetting specifying (perhaps in caps template syntax?) the formats supported for the MSE player. the ::supportsType method could use this to correctly negotiate the codecs.

Any thoughts?

[Alicia Boya García]

If VPx capable GStreamer plugins are not installed, MediaSource.isTypeSupported() will reject VP9.

Regarding video resolution... AFAIK that is up to the application. By the time GStreamer knows the resolution of the video it must have been already been demuxed (and therefore partially downloaded) so at most it could fail, but that's hardly useful. A possible exception would be some adaptive elements, like dashdemux, that can choose themselves the desired resolution.

What I've seen is applications querying the window size in order to determine the resolution they will demand; e.g. if you have a 720p screen they will not bother downloading 1080p video. This is what YouTube does for instance.

Decoder elements, if they are smart enough, can manage to work slightly above their actual capabilities by listening to QOS events that advise them to drop non-sync frames here and there until decoding becomes faster than run time.

[Philippe Normand]

The current hacks I have:

https://github.com/Igalia/webkit/commit/609e0f8dbd33fb39f359764f1e142c468868a875
https://github.com/Igalia/webkit/commit/c7c4969badac7b5a8e9bf8c57932ab09eb7f040a

work, but I was hoping to have a better solution.

[Philippe Normand]

(In reply to Alicia Boya García from comment #1)
> If VPx capable GStreamer plugins are not installed,
> MediaSource.isTypeSupported() will reject VP9.
> 

True. I'm not sure vpx support can easily be disabled at build time in the v4l2 plugin though. I'll have a look...

> Regarding video resolution... AFAIK that is up to the application. By the
> time GStreamer knows the resolution of the video it must have been already
> been demuxed (and therefore partially downloaded) so at most it could fail,
> but that's hardly useful. A possible exception would be some adaptive
> elements, like dashdemux, that can choose themselves the desired resolution.
> 

I'm talking about MSE here, not DASH :)

> What I've seen is applications querying the window size in order to
> determine the resolution they will demand; e.g. if you have a 720p screen
> they will not bother downloading 1080p video. This is what YouTube does for
> instance.
> 

True, my screen is 1080p though.

> Decoder elements, if they are smart enough, can manage to work slightly
> above their actual capabilities by listening to QOS events that advise them
> to drop non-sync frames here and there until decoding becomes faster than
> run time.

I don't know about the reasons why my platform doesn't have hardware-accelerated support for vp8 yet. For sure it doesn't have vp9 support at all.

[Enrique Ocaña]

(In reply to Philippe Normand from comment #2)
> The current hacks I have:
> 
> https://github.com/Igalia/webkit/commit/
> 609e0f8dbd33fb39f359764f1e142c468868a875
> https://github.com/Igalia/webkit/commit/
> c7c4969badac7b5a8e9bf8c57932ab09eb7f040a
> 
> work, but I was hoping to have a better solution.

This is basically what I do downstream when I want to disable vpx and opus temporarily. Disabling vpx and opus on platforms having the right plugins installed looks like a customization and I'm not sure if WebKit should support it. After all, the current behaviour comes from a request to autodetect the installed codecs and not exposing anything not already present: https://bugs.webkit.org/show_bug.cgi?id=167120

If there's some problem in the installed plugins (eg: no hardware acceleration support), then maybe the system/distro maintainer for the specific device should reflect about if vpx should actually be exposed in the first place or if it should be disabled at build time in case it's not production ready on that device.

Anyway, if anybody writes a patch to disable features on demand, I won't oppose as long as the default behaviour (enable the autodetected features) still works.

[Philippe Normand]

See also https://wicg.github.io/media-capabilities/

[Philippe Normand]

(In reply to Philippe Normand from comment #5)
> See also https://wicg.github.io/media-capabilities/

Eric, is it too early to implement this spec in WebKit?

[Eric Carlson]

(In reply to Philippe Normand from comment #6)
> (In reply to Philippe Normand from comment #5)
> > See also https://wicg.github.io/media-capabilities/
> 
> Eric, is it too early to implement this spec in WebKit?

Not at all, I think it is a good time to implement the spec

[Philippe Normand]

Ok, I'll start working on it.

[Philippe Normand]

(In reply to Philippe Normand from comment #8)
> Ok, I'll start working on it.

Actually it was already started :-) https://bugs.webkit.org/show_bug.cgi?id=181064

[Philippe Normand]

There's basic support now for media-capabilities probing, let's close this bug and open new specific issues as they pop-up.