Bug 253807 – [GTK][GStreamer] VA+DMABuf videos flicker

• Component: WebKitGTK
• Status: RESOLVED FIXED

[Kdwk]

With VA decoders ranked to the max and the DMABuf sink turned on, videos sometimes flicker with frames from +/- a couple of seconds. This can be observed wherever videos can be played, including YouTube and Apple product pages.

WebKitGTK 2.39.91, GStreamer 1.20.5, Epiphany Technology Preview 44.rc-19-ga80fcd0ab+

[Kdwk]

I am unable to attach a screencast as an attachment here (413), so I hosted it here instead http://kdwk.tplinkdns.com/index.php/s/bP8gHAsLym5Nbr6

[Philippe Normand]

I noticed this too, thanks for the bug report!

[Zan Dobersek]

I think this happens because of the GstMemory being released (or rather disposed) for decoder's reuse before the sampling of dmabufs from such GstMemory is complete.

The GstMemory is disposed as soon as the sample object is updated in MediaPlayerPrivateGStreamer::triggerRepaint(), releasing the last reference on the memory-containing buffer object. What I think is happening is the decoder can end up taking the released memory and decode into the associated dambuf even when the dmabuf is still being used for sampling on our side.

The DMABufObject we use internally has an eventfd-based mechanism for indicating release from the presentation layer, meaning we have a pretty good idea of when a given dmabuf is not sampled anymore. When this is associated somehow with the GstMemory/GstBuffer lifetime, I think the flickering should go away.

[Kdwk]

This issue doesn't seem to occur when using the vaav1dec decoder. I tested this on my laptop with AV1 hardware decoding, then set YouTube to serve videos in AV1.

[Kdwk]

Ah nevermind. It's still using CPU decode

[Philippe Normand]

(In reply to kdwkleung from comment #5)
> Ah nevermind. It's still using CPU decode

HW AV1 decoding should work though, I guess you're the first to test this. Can you file a separate bug please?

[Kdwk]

I thought org.gnome.Platform doesn’t ship vaav1dec because it’s still on GStreamer 1.20…

I’m not actually sure if it’s hardware-decoded or software-decoded anymore. Clapper the video player can hardware decode (it explicitly says vaav1dec) an AV1 video with about 2-3% CPU usage. 

The WebKitWebProcess, for a similar video, needs around 7% for (suspected) VA+DMABuf (but I’ve seen it do it at 2-3% at the same WebKitGTK+GStreamer version before so not sure what changed). This is much higher than confirmed hardware decoding but also significantly lower than confirmed software decoding — right about in the middle, so I’m very confused what’s happening.

[Kdwk]

Specifically, this is happening on YouTube. Videos on https://wpewebkit.org/blog/05-new-svg-engine.html still work as expected with 2-3% CPU usage. 

Additional information: after retesting the CPU usage seems to be 5% instead of 7%. The 5% figure doesn’t seem to scale with video resolution. 1080p24 is 5%, 4K60 still 5%. This is also only happening on AV1. VP9 works as expected at 2-3%.

I should probably open a new bug report for this…

[Zan Dobersek]

Created attachment 465676 [details]
WIP patch

Provisional idea. I was able to reproduce the problem, but not that consistently. With the patch I wasn't able to reproduce it, but not sure cause it wasn't common before then either.

This is the dumb approach of storing references and dropping them upon new samples. It still has flaws since the is-released check is not necessarily consistent and can lead to retained GstMemory references, draining the decoder of available dmabuf targets for decoding.

[Philippe Normand]

This works quite well here already, I can confirm it works on my laptop, 4K VP9@60FPS with vavp9dec YT renders smoothly, I can't notice flickering anymore. https://www.youtube.com/watch?v=LXb3EKWsInQ

[Kdwk]

I'm on WebKitGTK 2.41.2, and it seems the issue can no longer be observed.

[Philippe Normand]

(In reply to kdwkleung from comment #11)
> I'm on WebKitGTK 2.41.2, and it seems the issue can no longer be observed.

Without this WIP patch you mean?

[Kdwk]

Indeed.

[Philippe Normand]

In 2.41 (so upcoming 2.42) the dmabuf sink is on by default btw, unless this env var is set: WEBKIT_GST_DMABUF_SINK_DISABLED=1

[Philippe Normand]

Here with 2.41, Ephy TP emits this warning:

WebKit wasn't able to find the DMABuf video sink dependencies. Hardware-accelerated zero-copy video rendering won't be achievable with this p
lugin.

And I suspect the va decoders are no longer in use either...

[Kdwk]

Ah, so that's why CPU usage is through the roof again. I'll open another bug report

[Víctor M. Jáquez L.]

(In reply to Philippe Normand from comment #15)
> Here with 2.41, Ephy TP emits this warning:
> 
> WebKit wasn't able to find the DMABuf video sink dependencies.
> Hardware-accelerated zero-copy video rendering won't be achievable with this
> plugin.
> 
> And I suspect the va decoders are no longer in use either...

[📦 org.gnome.Epiphany.Devel ~]$ GST_REGISTRY= GST_DEBUG=va*:8 gst-inspect-1.0 /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstva.so
0:00:00.047993566    12 0x55ae7026d780 INFO               vadisplay gstvadisplay_drm.c:154:gst_va_display_drm_create_va_display:<vadisplaydrm0> DRM render node with kernel driver i915
0:00:00.048022485    12 0x55ae7026d780 INFO               vadisplay gstvadisplay.c:299:_va_info:<vadisplaydrm0> VA info: VA-API version 1.15.0
0:00:00.048034108    12 0x55ae7026d780 INFO               vadisplay gstvadisplay.c:299:_va_info:<vadisplaydrm0> VA info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
0:00:00.048051344    12 0x55ae7026d780 INFO               vadisplay gstvadisplay.c:299:_va_info:<vadisplaydrm0> VA info: Trying to open /usr/lib/x86_64-linux-gnu/dri/intel-vaapi-driver/iHD_drv_video.so
0:00:00.048857978    12 0x55ae7026d780 WARN               vadisplay gstvadisplay.c:287:_va_warning:<vadisplaydrm0> VA error: /usr/lib/x86_64-linux-gnu/dri/intel-vaapi-driver/iHD_drv_video.so has no function __vaDriverInit_1_0
0:00:00.048867446    12 0x55ae7026d780 INFO               vadisplay gstvadisplay.c:299:_va_info:<vadisplaydrm0> VA info: Trying to open /usr/lib/x86_64-linux-gnu/GL/lib/dri/iHD_drv_video.so
0:00:00.048890511    12 0x55ae7026d780 INFO               vadisplay gstvadisplay.c:299:_va_info:<vadisplaydrm0> VA info: va_openDriver() returns -1
0:00:00.048898563    12 0x55ae7026d780 INFO               vadisplay gstvadisplay.c:299:_va_info:<vadisplaydrm0> VA info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
0:00:00.048911836    12 0x55ae7026d780 INFO               vadisplay gstvadisplay.c:299:_va_info:<vadisplaydrm0> VA info: Trying to open /usr/lib/x86_64-linux-gnu/dri/intel-vaapi-driver/i965_drv_video.so
0:00:00.049297301    12 0x55ae7026d780 WARN               vadisplay gstvadisplay.c:287:_va_warning:<vadisplaydrm0> VA error: /usr/lib/x86_64-linux-gnu/dri/intel-vaapi-driver/i965_drv_video.so has no function __vaDriverInit_1_0
0:00:00.049304197    12 0x55ae7026d780 INFO               vadisplay gstvadisplay.c:299:_va_info:<vadisplaydrm0> VA info: Trying to open /usr/lib/x86_64-linux-gnu/GL/lib/dri/i965_drv_video.so
0:00:00.049321443    12 0x55ae7026d780 INFO               vadisplay gstvadisplay.c:299:_va_info:<vadisplaydrm0> VA info: va_openDriver() returns -1
0:00:00.049325583    12 0x55ae7026d780 WARN               vadisplay gstvadisplay.c:347:gst_va_display_initialize:<vadisplaydrm0> vaInitialize: unknown libva error
0:00:00.370719171    11 0x56034ada7680 INFO               vadisplay gstvadisplay_drm.c:154:gst_va_display_drm_create_va_display:<vadisplaydrm0> DRM render node with kernel driver i915
0:00:00.370764331    11 0x56034ada7680 INFO               vadisplay gstvadisplay.c:299:_va_info:<vadisplaydrm0> VA info: VA-API version 1.15.0
0:00:00.370777935    11 0x56034ada7680 INFO               vadisplay gstvadisplay.c:299:_va_info:<vadisplaydrm0> VA info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
0:00:00.370787742    11 0x56034ada7680 INFO               vadisplay gstvadisplay.c:299:_va_info:<vadisplaydrm0> VA info: Trying to open /usr/lib/x86_64-linux-gnu/dri/intel-vaapi-driver/iHD_drv_video.so
0:00:00.372541070    11 0x56034ada7680 WARN               vadisplay gstvadisplay.c:287:_va_warning:<vadisplaydrm0> VA error: /usr/lib/x86_64-linux-gnu/dri/intel-vaapi-driver/iHD_drv_video.so has no function __vaDriverInit_1_0
0:00:00.372548829    11 0x56034ada7680 INFO               vadisplay gstvadisplay.c:299:_va_info:<vadisplaydrm0> VA info: Trying to open /usr/lib/x86_64-linux-gnu/GL/lib/dri/iHD_drv_video.so
0:00:00.372563074    11 0x56034ada7680 INFO               vadisplay gstvadisplay.c:299:_va_info:<vadisplaydrm0> VA info: va_openDriver() returns -1
0:00:00.372569358    11 0x56034ada7680 INFO               vadisplay gstvadisplay.c:299:_va_info:<vadisplaydrm0> VA info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
0:00:00.372576603    11 0x56034ada7680 INFO               vadisplay gstvadisplay.c:299:_va_info:<vadisplaydrm0> VA info: Trying to open /usr/lib/x86_64-linux-gnu/dri/intel-vaapi-driver/i965_drv_video.so
0:00:00.373103367    11 0x56034ada7680 WARN               vadisplay gstvadisplay.c:287:_va_warning:<vadisplaydrm0> VA error: /usr/lib/x86_64-linux-gnu/dri/intel-vaapi-driver/i965_drv_video.so has no function __vaDriverInit_1_0
0:00:00.373110688    11 0x56034ada7680 INFO               vadisplay gstvadisplay.c:299:_va_info:<vadisplaydrm0> VA info: Trying to open /usr/lib/x86_64-linux-gnu/GL/lib/dri/i965_drv_video.so
0:00:00.373125681    11 0x56034ada7680 INFO               vadisplay gstvadisplay.c:299:_va_info:<vadisplaydrm0> VA info: va_openDriver() returns -1
0:00:00.373142625    11 0x56034ada7680 WARN               vadisplay gstvadisplay.c:347:gst_va_display_initialize:<vadisplaydrm0> vaInitialize: unknown libva error
Plugin Details:
  Name                     va
  Description              VA-API codecs plugin
  Filename                 /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstva.so
  Version                  1.20.5
  License                  LGPL
  Source module            gst-plugins-bad
  Source release date      2022-12-19
  Binary package           GStreamer Bad Plug-ins source release
  Origin URL               freedesktop-sdk


  0 features:

it seems that org.gnome.Epihany.Devel doesn't "talk" with org.freedesktop.Platform.VAAPI.Intel runtime.

[Philippe Normand]

That's https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/issues/1579 that requires a GNOME runtime re-build, which IIUC, is somehow, in-progress.

[Philippe Normand]

(In reply to Zan Dobersek from comment #9)
> Created attachment 465676 [details]
> WIP patch
> 
> Provisional idea. I was able to reproduce the problem, but not that
> consistently. With the patch I wasn't able to reproduce it, but not sure
> cause it wasn't common before then either.
> 
> This is the dumb approach of storing references and dropping them upon new
> samples. It still has flaws since the is-released check is not necessarily
> consistent and can lead to retained GstMemory references, draining the
> decoder of available dmabuf targets for decoding.

Have you observed this draining in practice?

[Philippe Normand]

Before this patch bitrots too much, can we mainline it?
Back then I did check it was addressing the rendering issue reported here :)

[Kdwk]

This can still be observed in the latest WebKit main. Would really appreciate if the fix can be merged :)

[Philippe Normand]

Pull request: https://github.com/WebKit/WebKit/pull/14548

[Kdwk]

With this PR the vast majority of flickers are gone, including YouTube. However, one video on https://www.apple.com/airpods-3rd-generation/ still flickers (the video under the paragraph that starts with "A case full of energy.")

[Kdwk]

After full reloading the page for several more times, this seems to have gone away, but it seems some videos on apple.com product pages still flicker very occassionally

[Kdwk]

Videos on https://www.apple.com/mac-studio/ still flicker severely (sometimes)

[Philippe Normand]

When that happens do you get warnings or errors in the gst log? Checking "videodec*:6,va*:6" ?

[Kdwk]

Created attachment 466569 [details]
GStreamer log of the problem still appearing after applying the patch

Here are the requested logs. The problematic event should occur towards the end.

[Kdwk]

Created attachment 466570 [details]
GStreamer logs with correct env var set

Oops. Env var were not correctly set last time. Here is the correct log.

[Xi Ruoyao]

I'm getting a segfault in WebCore::MediaPlayerPrivateGStreamer::triggerRepaint with webkitgtk-2.41.5 when I try to play videos on Bilibili.  I guess it's related to this issue but I'm not sure.

[Xi Ruoyao]

(In reply to Xi Ruoyao from comment #29)
> I'm getting a segfault in
> WebCore::MediaPlayerPrivateGStreamer::triggerRepaint with webkitgtk-2.41.5
> when I try to play videos on Bilibili.  I guess it's related to this issue
> but I'm not sure.

Nope, my crash seems related to my custom user-agent setting and not this one.

[Philippe Normand]

(In reply to kdwkleung from comment #25)
> Videos on https://www.apple.com/mac-studio/ still flicker severely
> (sometimes)

I can reproduce this indeed, enabling the native-video flag in playbin3 helps a bit but there's still a queue in playsink, holding a maximum of 3 buffers. I wonder if that could prevent memory recycling? Not sure...

[Kdwk]

I wonder if the patch can be landed as a partial fix? It already solves most of the most annoying flickering, especially on YouTube which would be completely unwatchable on my machine otherwise, and it doesn't cause any regressions...

[Kdwk]

The flickering on apple.com doesn't matter for the moment because the tab freezes due to MSE anyway

[EWS]

Committed 270971@main (2f3793730f22): <https://commits.webkit.org/270971@main>

Reviewed commits have been landed. Closing PR #14548 and removing active labels.

[Philippe Normand]

This still happens... a bit less that before, but still.

[Philippe Normand]

Pull request: https://github.com/WebKit/WebKit/pull/29980

[EWS]

Committed 280210@main (9d5844679af8): <https://commits.webkit.org/280210@main>

Reviewed commits have been landed. Closing PR #29980 and removing active labels.