Bug 191257 – [GStreamer][MSE] pixelated video when MSE is enabled

• Status: RESOLVED
• Resolution: INVALID
• Component: WebKitGTK

[Jim Mason]

Created attachment 353850 [details]
MiniBrowser Screenshot

When MediaSource extensions are enabled in 2.22.3, I am getting pixelated video in YouTube whenever webm/vp9 streams are offered.  If webm is not available (for example, someone has just uploaded a video and only mp4 streams are available -- including HD 1920x1080 mp4), the videos play fine, even with MSE enabled, and in this case, all resolutions are available, including 1080p.

When webm is present, all resolutions (including SD) pixelate in YouTube when MSE is enabled.  If I disable MSE, then all videos play fine in YouTube, whether they include webm streams or not.

However, webm test videos at https://www.webmfiles.org/demo-files/ play fine in all cases.

I am attaching a screenshot from the MiniBrowser that illustrates the pixelation.  The video will from time to time display clearly, then it resumes pixelation.

Steps to reproduce:
    1. Find a YouTube video that has webm/vp9 streams.  I use youtube-dl for this purpose.
    2. Enable MediaSource extensions in WebKit
    3. Open to the video from step 1.  It will be pixelated in all resolutions.

Expected behaviour:
    All videos should display without pixelation.

Environment:
    WebKit: 2.22.3
    GStreamer: 1.14.4 with plugins base, good, ugly, bad, and libav
    GPU: NVIDIA GeForce GT 610
    GL libraries provided by: NVIDIA driver 346.35
    OS: Solaris 11.3

[Jim Mason]

I can confirm the problem is specific to webm and MSE.  If I comment out the following one line in the source, all YouTube videos display without pixelation when MSE is enabled:


--- a/Source/WebCore/platform/graphics/gstreamer/mse/MediaPlayerPrivateGStreamerMSE.cpp	2018-10-27 22:20:58.000000000 +0000
+++ b/Source/WebCore/platform/graphics/gstreamer/mse/MediaPlayerPrivateGStreamerMSE.cpp	2018-11-05 21:18:54.717811557 +0000
@@ -713,7 +713,9 @@
         const char* mimeTypes[] = {
             "video/mp4",
             "audio/mp4",
+#ifdef ENABLE_MSE_WEBM_VIDEO
             "video/webm",
+#endif
             "audio/webm"
         };
         for (auto& type : mimeTypes)



It seems the problem is specific to MSE, as I can play other webm vp8 and vp9 streams in WebKit, such as these: https://base-n.de/webm/VP9%20Sample.html

[Alicia Boya García]

The matrix would be something like this, right?

             +------------------+------------+
             | Regular playback |    MSE     |
+------------+------------------+------------+
| h264 (MP4) |        OK        |     OK     |
| VP9 (WebM) |        OK        | Pixelation |
+------------+------------------+------------+

Can you check whether Opus audio (audio/webm) playback works correctly? You can check if you are using it by looking for "stats for nerds" in the YT context menu.

Does the pixelation occur since the first frame is played or does it happen later (e.g. in response to a seek). Does it clear at times (e.g. refresh frames) or is it constant?

What CPU architecture are you using?

Do you have gstreamer1-vaapi installed? If so, is there any change after uninstalling it?

Also, try to play a VP9 webm video file with this command and see if you can reproduce the issue:

$ gst-play-1.0 pushfile:///home/user/yourvideo.webm

[Jim Mason]

Hello Sra. Alicia, good afternoon, thank you for your prompt reply!

My comments follow in-line below.


(In reply to Alicia Boya García from comment #2)
> The matrix would be something like this, right?
> 
>              +------------------+------------+
>              | Regular playback |    MSE     |
> +------------+------------------+------------+
> | h264 (MP4) |        OK        |     OK     |
> | VP9 (WebM) |        OK        | Pixelation |
> +------------+------------------+------------+

Yes, this is correct.


> Can you check whether Opus audio (audio/webm) playback works correctly? You
> can check if you are using it by looking for "stats for nerds" in the YT
> context menu.

Yes, this seems working.  In the nerd display, I see (for example):

    Codecs  avc1.4d401f (136) / opus (251)

and the sound sounds OK.


> Does the pixelation occur since the first frame is played or does it happen
> later (e.g. in response to a seek). Does it clear at times (e.g. refresh
> frames) or is it constant?

Usually this occurs since the beginning of the video, although also again after a seek.  It will in some moments play clearly, then spontaneously pixelate.

 
> What CPU architecture are you using?

x86_64


> Do you have gstreamer1-vaapi installed? If so, is there any change after
> uninstalling it?

No, it is not installed.


> Also, try to play a VP9 webm video file with this command and see if you can
> reproduce the issue:
> 
> $ gst-play-1.0 pushfile:///home/user/yourvideo.webm

This command plays OK a local VP9 webm file.

Thank you again, enjoy what remains of the afternoon, and of course, please let me know if you need more information!

[Alicia Boya García]

Created attachment 353968 [details]
[MSE][Debug] Add debug option to dump appends

I'm attaching a debug patch that can be used to dump the MSE appends. Please build WebKit with it and then run with WEBKIT_MSE_DUMP_APPENDS=1 set as an environment variable and play one of the offending videos without seeking.

You should get a file like /tmp/SourceBuffer-2-video-0.webm. This file contains the MSE video byte to byte as YouTube pushes it to WebKit.

Then play it like this (I'm assuming a Bash shell). This replicates all the stages of playback with the same elements WebKit uses.

$ gst-launch-1.0 pushfilesrc location=<(cat /tmp/SourceBuffer-2-video-0.webm) ! matroskademux ! decodebin ! glupload ! glcolorconvert ! glimagesink

If you are running WebKit inside jhbuild (which is the default if you are using `run-minibrowser`), you should run it in jhbuild as well, e.g. by prepending `jhbuild-wrapper --gtk run` to the previous command, i.e:

$ jhbuild-wrapper --gtk run gst-launch-1.0 pushfilesrc location=<(cat /tmp/SourceBuffer-2-video-0.webm) ! matroskademux ! decodebin ! glupload ! glcolorconvert ! glimagesink

(The point is to make sure you are using the same GStreamer instance in both cases.)

I would expect this command to reproduce the issue. Then we could switch elements until finding which one is causing the problem in your system.

If the above command does not reproduce the issue then it's difficult that I am able to debug the issue without having a system where it is reproducible.

[Jim Mason]

(In reply to Alicia Boya García from comment #4)
> Then play it like this (I'm assuming a Bash shell). This replicates all the
> stages of playback with the same elements WebKit uses.
> 
> $ gst-launch-1.0 pushfilesrc location=<(cat
> /tmp/SourceBuffer-2-video-0.webm) ! matroskademux ! decodebin ! glupload !
> glcolorconvert ! glimagesink
> 
> I would expect this command to reproduce the issue. Then we could switch
> elements until finding which one is causing the problem in your system.

YES this indeed reproduces the issue!

Please advise me which elements to switch in and out.

Thank you!

[Alicia Boya García]

I was going to send you a list, but while I was trying it, I found the element causing pixelation bug. It's the VP9 decoder from libav.

Using the decoder from libav (avdec_*), it pixelates:

$ gst-launch-1.0 pushfilesrc location=<(cat /tmp/SourceBuffer-2-video-0.webm) ! matroskademux ! avdec_vp9 ! autovideosink

Using vp9 elements from GStreamer, it works without issues:

$ gst-launch-1.0 pushfilesrc location=<(cat /tmp/SourceBuffer-2-video-0.webm) ! matroskademux ! vp9dec ! autovideosink

vp9dec has a higher rank than avdec_vp9, but you probably built gst-plugins-good without libvpx so it had to go with avdec_vp9 anyway. Reconfiguring gst-plugins-good with libvpx and rebuilding should fix the issue.

It's probably worth it to report this as a bug in avdec_vp9 anyway (which may easily end up being a bug in libav).

That does not explain why this did not happen on regular WebM playback, but it may be something different about the way YT files are encoded that the other WebM test files you tried may do differently... Indeed, just checking it now, the test file from https://www.webmfiles.org/demo-files/ is actually encoded in VP8, not VP9, so it's a completely different video format even if the container format is also WebM.

[Jim Mason]

Thank you!!

Yes, I just uninstalled gst-libav and now get this:

    no suitable plugins found:
    Missing decoder: VP9 (video/x-vp9, width=(int)1920, height=(int)1080, framerate=(fraction)25/1)

So indeed, 1) it was using the decoder from libav, and 2) it appears my gst-plugins-good was built without vpx support.

I am now working on rebuilding gst-plugins-good...

In the interim, can you please report the bug in avdec_vp9?  I am not an expert in streaming media; I think you will be able to communicate more clearly than I what the problem is.

[Jim Mason]

I can confirm, it is now working in WebKit with the rebuilt gst-plugins-good.

YT nerd box reports:

   Codecs vp9 (248) / opus (251)

No more distortion in the video.  Seek works also without issue.

In addition, I reinstalled gst-libav.  GStreamer is correctly picking up the vp9 decoder from gst-plugins-good instead of the buggy libav one.

If you would be so kind as to follow up and report the bug in avdec_vp9, that would be appreciated.  I think either that bug needs fixing, or WebKit should explicitly avoid using that decoder.

Thank you again!

[Alicia Boya García]

The bug is already reported to gst-libav. Closing here:

https://gitlab.freedesktop.org/gstreamer/gst-libav/issues/32