Bug 252581 – [GStreamer][VP9] HFR Youtube playback stutters with vavp9dec

• Status: RESOLVED
• Resolution: FIXED
• Component: Platform

[Philippe Normand]

Tested in Ephy TP.
vavp9dec doesn't seem to like this one: https://www.youtube.com/watch?v=LXb3EKWsInQ
With vp9dec rendering is OK.

[Philippe Normand]

To reliably reproduce this you need to force 1080p60 in the playback settings.

[Philippe Normand]

0:00:42.705813692     2 0x7f2118013da0 INFO                vavp9dec gstvavp9dec.c:613:gst_va_vp9_dec_negotiate:<vavp9dec1> Negotiated caps video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, framerate=(fraction)60000/1001
0:00:42.726240724     2 0x7f2118013da0 DEBUG               vavp9dec gstvabasedec.c:502:gst_va_base_dec_decide_allocation:<vavp9dec1> may need other pool for copy frames <glbufferpool2>
0:00:42.726414094     2 0x7f2118013da0 DEBUG               vamemory gstvaallocator.c:1071:_update_image_info:<vaallocator2> Created surface 0 [1920x1080]
0:00:42.728023874     2 0x7f2118013da0 INFO                vamemory gstvaallocator.c:1519:gst_va_allocator_try:<vaallocator2> va allocator info, surface format: NV12, image format: NV12, use derived: true, rt format: 0x1, fourcc: NV12
0:00:42.728064582     2 0x7f2118013da0 INFO                  vapool gstvapool.c:161:gst_va_pool_set_config:<vapool2> Video meta is required in buffer.
0:00:44.781351937     2 0x7f21180134d0 WARN                basesink gstbasesink.c:3143:gst_base_sink_is_too_late:<webkit-gl-video-appsink> warning: A lot of buffers are being dropped.
0:00:44.781373585     2 0x7f21180134d0 WARN                basesink gstbasesink.c:3143:gst_base_sink_is_too_late:<webkit-gl-video-appsink> warning: There may be a timestamping problem, or this computer is too slow.
0:00:44.781448319     2 0x557d149ca370 WARN       webkitmediaplayer MediaPlayerPrivateGStreamer.cpp:1748:handleMessage:<MSE-media-player-5> A lot of buffers are being dropped. (url=mediasourceblob:https://www.youtube.com/07922544-1177-468f-8ebf-c98c245ca02b) (code=13)

[Philippe Normand]

Created attachment 465091 [details]
screenshot with dmabuf sink enabled

Oh nice, I can reproduce the broken rendering when the dmabuf sink is enabled.

[Víctor M. Jáquez L.]

IIUC you're reporting two different problems

1. A lot of drops when using gl sink. Perhaps something (decoding, upload, color conversion, etc.) is slugging the rendering.
2. the dma's drm format is not negotiated so the image is not well rendered using the dmabuf sink. For this issue we need to finish https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/3524

For both cases, perhaps adding a vapostproc after the decoder as video filter might help.

[Víctor M. Jáquez L.]

(In reply to Víctor M. Jáquez L. from comment #4)
> 2. the dma's drm format is not negotiated so the image is not well rendered
-------------------^ modifier

[Philippe Normand]

(In reply to Víctor M. Jáquez L. from comment #4)
> IIUC you're reporting two different problems
> 

Correct.

> 1. A lot of drops when using gl sink. Perhaps something (decoding, upload,
> color conversion, etc.) is slugging the rendering.
> 2. the dma's drm format is not negotiated so the image is not well rendered
> using the dmabuf sink. For this issue we need to finish
> https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/3524
> 
> For both cases, perhaps adding a vapostproc after the decoder as video
> filter might help.

It might help in GL sink case but it definitely does not with the dmabuf sink. We hardcode modifiers to 0 anyway when handling the buffers from the sink.

[Víctor M. Jáquez L.]

(In reply to Philippe Normand from comment #6)
> 
> It might help in GL sink case but it definitely does not with the dmabuf
> sink. We hardcode modifiers to 0 anyway when handling the buffers from the
> sink.

decoders might produce buffers with a particular modifier, but vapostproc could convert that modifier to 0  if a color space transformation is done. I don't remember if dma sink accepts only RGB or also NV12?

[Philippe Normand]

(In reply to Víctor M. Jáquez L. from comment #7)
> (In reply to Philippe Normand from comment #6)
> > 
> > It might help in GL sink case but it definitely does not with the dmabuf
> > sink. We hardcode modifiers to 0 anyway when handling the buffers from the
> > sink.
> 
> decoders might produce buffers with a particular modifier, but vapostproc
> could convert that modifier to 0  if a color space transformation is done. I
> don't remember if dma sink accepts only RGB or also NV12?

iiuc it should work for both.

Anyways, I tried https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/3524
Doesn't work, videoconvertscale errors out, among other things. See attached log.

[Philippe Normand]

Created attachment 465112 [details]
gst log

[Philippe Normand]

Naive patch for using the new API:

diff --git a/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp b/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp
index 364c699ca072..cfeaf295f7b8 100644
--- a/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp
+++ b/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp
@@ -129,6 +129,7 @@
 #include "TextureMapperPlatformLayerProxyDMABuf.h"
 #include <gbm.h>
 #include <gst/allocators/gstdmabuf.h>
+#include <gst/video/video-info-dma.h>
 #endif // USE(TEXTURE_MAPPER_DMABUF)
 
 #if USE(WPE_VIDEO_PLANE_DISPLAY_DMABUF)
@@ -2856,6 +2857,9 @@ void MediaPlayerPrivateGStreamer::createGSTPlayBin(const URL& url)
 
     g_object_set(m_pipeline.get(), "audio-sink", m_audioSink.get(), "video-sink", createVideoSink(), nullptr);
 
+    auto* filter = makeGStreamerElement("vapostproc", nullptr);
+    //g_object_set(m_pipeline.get(), "video-filter", filter, nullptr);
+
     if (m_shouldPreservePitch && !isMediaStream) {
         if (auto* scale = makeGStreamerElement("scaletempo", nullptr))
             g_object_set(m_pipeline.get(), "audio-filter", scale, nullptr);
@@ -3219,6 +3223,13 @@ void MediaPlayerPrivateGStreamer::pushDMABufToCompositor()
         if (!proxy.isActive())
             return;
 
+        GstVideoInfoDmaDrm dmaDrmInfo;
+        uint64_t modifier = 0;
+        if (gst_video_info_dma_drm_from_caps(&dmaDrmInfo, caps)) {
+            modifier = dmaDrmInfo.drm_modifier;
+        }
+        gst_printerrln("Modifier: %" G_GUINT64_FORMAT, modifier);
+
         // Provide the DMABufObject with a relevant handle (memory address). When provided for the first time,
         // the lambda will be invoked and all dmabuf data is filled in.
         downcast<TextureMapperPlatformLayerProxyDMABuf>(proxy).pushDMABuf(
@@ -3256,7 +3267,7 @@ void MediaPlayerPrivateGStreamer::pushDMABufToCompositor()
                     gst_video_format_info_component(videoInfo.finfo, i, comp);
                     object.offset[i] = offset;
                     object.stride[i] = GST_VIDEO_INFO_PLANE_STRIDE(&videoInfo, i);
-                    object.modifier[i] = 0;
+                    object.modifier[i] = modifier;
                 }
                 return WTFMove(object);
             });

[Philippe Normand]

(In reply to Víctor M. Jáquez L. from comment #4)
> IIUC you're reporting two different problems
> 
> 1. A lot of drops when using gl sink. Perhaps something (decoding, upload,
> color conversion, etc.) is slugging the rendering.
> 2. the dma's drm format is not negotiated so the image is not well rendered
> using the dmabuf sink. For this issue we need to finish
> https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/3524
> 

Both cases work now.