Bug 259643 – [GStreamer][MSE] Tab hangs with any YouTube video on Nvidia hardware

• Component: WebKitGTK
• Status: RESOLVED FIXED

[Kdwk]

Created attachment 467160 [details]
gdb (bt full; c) output.txt

Tested with: Gnome Web 44.6/ WebKitGTK 2.40.4, GStreamer 1.20.6; Nvidia RTX 4070 (driver version 535, via Universal Blue Nvidia Image)
Tested on: YouTube videos, apple.com/macbook-pro
On YouTube, play a video for a few seconds: tab hangs
On apple.com/macbook-pro, page never loads
Happens both with and without nvdec elements max-ranked.

Console output when playing videos on YouTube:
[kdwk@fedora ~]$ flatpak run org.gnome.Epiphany

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4677:gst_pad_push_data:<nvvp9dec0:src> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<video_0:proxypad75> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4677:gst_pad_push_data:<decodebin3-1:video_0> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<video_0:proxypad76> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4677:gst_pad_push_data:<uridecodebin3-1:video_0> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<video-concat:sink_0> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4677:gst_pad_push_data:<video-concat:src> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<playsink:video_sink> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4677:gst_pad_push_data:<video_sink:proxypad65> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<streamsynchronizer5:sink_0> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4677:gst_pad_push_data:<streamsynchronizer5:src_0> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<vdbin:sink> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4677:gst_pad_push_data:<sink:proxypad80> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<vdconv:sink> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4677:gst_pad_push_data:<vdconv:src> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<deinterlace:sink> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4677:gst_pad_push_data:<deinterlace:src> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<src:proxypad81> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4677:gst_pad_push_data:<vdbin:src> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<vbin:sink> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.646: ../gst/gstpad.c:4677:gst_pad_push_data:<sink:proxypad79> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<vqueue:sink> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4677:gst_pad_push_data:<vqueue:src> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<vconv:sink> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4677:gst_pad_push_data:<sink:proxypad77> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<conv:sink> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4677:gst_pad_push_data:<conv:src> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<scale:sink> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4677:gst_pad_push_data:<scale:src> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<videobalance:sink> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-CRITICAL **: 20:09:13.647: gst_segment_to_stream_time: assertion 'segment->format == format' failed

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4677:gst_pad_push_data:<videobalance:src> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<conv2:sink> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4677:gst_pad_push_data:<conv2:src> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<src:proxypad78> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4677:gst_pad_push_data:<vconv:src> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<webkitglvideosink1:sink> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4677:gst_pad_push_data:<sink:proxypad62> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<gluploadelement1:sink> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4677:gst_pad_push_data:<gluploadelement1:src> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<glcolorconvertelement1:sink> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4677:gst_pad_push_data:<glcolorconvertelement1:src> Got data flow before segment event

(WebKitWebProcess:2): GStreamer-WARNING **: 20:09:13.647: ../gst/gstpad.c:4416:gst_pad_chain_data_unchecked:<webkit-gl-video-appsink:sink> Got data flow before segment event
KMS: DRM_IOCTL_MODE_CREATE_DUMB failed: Permission denied

[Kdwk]

Created attachment 467163 [details]
gstreamer logs.txt

[Philippe Normand]

nvcodec gstnvrtcloader.c:116:gst_nvrtc_load_library: Could not open library libnvrtc.so, libnvrtc.so: cannot open shared object file: No such file or directory

[Kdwk]

Created attachment 467164 [details]
gstfoo.log

GStreamer log

[Michael Catanzaro]

So we just need to mount them in the sandbox... seriously? That's going to be the end result of so many years of complaints from NVIDIA users? Something so simple?

Well, OK. So where are they installed on the host system? Must be someplace weird?

And wherever it is, can we please find out if it a standard location that's the same for all distros that package the NVIDIA driver?

[Philippe Normand]

Can you check if the same issue happens with a standalone pipeline?

Start a shell in the ephy runtime with --command=bash

and then GST_DEBUG="3,nv*:6" gst-launch-1.0 playbin uri=".../foo.mp4"

[Kdwk]

> And wherever it is, can we please find out if it a standard location that's
> the same for all distros that package the NVIDIA driver?

Universal Blue recommends these environment variables for hardware-accelerated decode in Firefox: https://universal-blue.org/images/nvidia/#video-playback

I can confirm that this works in Firefox.

[Philippe Normand]

what happens if you --filesystem=host-os for ephy?

[Kdwk]

(In reply to Philippe Normand from comment #7)
> what happens if you --filesystem=host-os for ephy?

That did not appear to have any effect

[Kdwk]

(In reply to Philippe Normand from comment #5)
> Can you check if the same issue happens with a standalone pipeline?
> 
> Start a shell in the ephy runtime with --command=bash
> 
> and then GST_DEBUG="3,nv*:6" gst-launch-1.0 playbin uri=".../foo.mp4"

That works just fine. It is able to use nvdec to play the video normally

[Philippe Normand]

Some comparison of pipeline graphs will be needed...

GST_DEBUG_DUMP_DOT_DIR=$HOME/dots
mkdir $HOME/dots

set the env var in both cases, compress the dots dir, upload here.

[Kdwk]

Upon further investigation, I've discovered that viewing the video at https://upload.wikimedia.org/wikipedia/commons/transcoded/a/a7/How_to_make_video.webm/How_to_make_video.webm.720p.vp9.webm does not cause the tab to hang. Only videos from sites like YouTube and Reddit do.

[Kdwk]

However, even though https://upload.wikimedia.org/wikipedia/commons/transcoded/a/a7/How_to_make_video.webm/How_to_make_video.webm.720p.vp9.webm works, software decoding is used despite max-ranking nvdec GStreamer elements

[Michael Catanzaro]

The problem is surely that libnvrtc.so is not available in the web process sandbox. We need to know where is it installed?

[Michael Catanzaro]

(In reply to kdwkleung from comment #6)
> > And wherever it is, can we please find out if it a standard location that's
> > the same for all distros that package the NVIDIA driver?
> 
> Universal Blue recommends these environment variables for
> hardware-accelerated decode in Firefox:
> https://universal-blue.org/images/nvidia/#video-playback
> 
> I can confirm that this works in Firefox.

Those instructions are trying to use the NVIDIA driver installed on your host system. That's weird and I'm not interested in trying to make it work. Only the driver provided by the NVIDIA flatpak runtime extension is expected to work.

(In reply to Michael Catanzaro from comment #13)
> The problem is surely that libnvrtc.so is not available in the web process
> sandbox. We need to know where is it installed?

Sorry, this was wrong because you're not using bubblewrap sandbox: you're using flatpak and the flatpak-spawn subsandbox. It seems that the subsandbox doesn't have libnvrtc.so, but I don't know why. So that's probably what we need to figure out. I still want to know where it's installed to, but what matters is where it's installed inside the flatpak sandbox ('flatpak run --command=/bin/bash org.gnome.Epiphany'), not where it's installed on your host system.

(Please do also attach the pipeline graphs that Phil requested.)

[Kdwk]

Created attachment 467178 [details]
epiphany-dots.zip

[Kdwk]

Created attachment 467179 [details]
playbin-dots.zip

[Philippe Normand]

I'm confused, the webkit gst pipeline looks good. Nothing plays? What is the issue exactly...?

The GL sink is used, maybe you try the dmabuf sink too...

[Kdwk]

The above dots are for the video that plays correctly: https://upload.wikimedia.org/wikipedia/commons/transcoded/a/a7/How_to_make_video.webm/How_to_make_video.webm.720p.vp9.webm

Only videos on sites like YouTube and Reddit cause the tab to hang, and for those I can't find a direct URL to the actual video file

[Philippe Normand]

Look, convert 0.00.35.384073642-MSE-media-player-6_PAUSED_PLAYING.dot to a SVG file and open it, you will see among other things:

MSE-media-player-6
[>]
current-uri="mediasourceblob:https://www.youtube.com/ef37beb4-b848-45d6-90f6-12147c1b4fbf"

That's a YT playback pipeline and it looks fine.

[Kdwk]

Oh right, I recorded epiphany-dots with YouTube but playbin-dots with the Wikimedia video.

> That's a YT playback pipeline and it looks fine.

GStreamer writes a lot of warnings to the console (comment #1)

[Philippe Normand]

flatpak run --env="GST_DEBUG=3,webkit*:9" --env=GST_DEBUG_FILE=$HOME/tmp/gstfoo.log --env=WEBKIT_DISABLE_SANDBOX_THIS_IS_DANGEROUS=1  --filesystem=home  org.gnome.Epiphany -p youtube.com

[Kdwk]

Created attachment 467181 [details]
gstfoo2.log

[Philippe Normand]

WARN   [00m [00m               nvdec gstnvdec.c:777:parser_display_callback:<nvvp9dec0>[00m no frame for picture index 0

Something goes wrong when the decoder receives its first buffer, the log don't say much though. Sorry I can't debug this.

[Michael Catanzaro]

(In reply to Philippe Normand from comment #2)
> nvcodec gstnvrtcloader.c:116:gst_nvrtc_load_library: Could not open library
> libnvrtc.so, libnvrtc.so: cannot open shared object file: No such file or
> directory

I don't understand why you're ignoring this. Isn't this a smoking gun?

[Philippe Normand]

I know next to nothing about this nvdec stuff, and I don't have a nvidia GPU. So you'll need someone familiar with this and with the required HW.

[Michael Catanzaro]

I don't think it makes sense to spend time debugging the multimedia pipeline side of this when we are still failing to load a shared library. Don't need any special hardware for that. Just need to wait for the bug reporter to tell us where the library is installed.

[Kdwk]

How do I find this location? find / libnvrtc.so did not return anything

[Michael Catanzaro]

I'm afraid I never learned how to use 'find' :(

[Philippe Normand]

find / -name libnvrtc.so

[Kdwk]

(In reply to Philippe Normand from comment #29)
> find / -name libnvrtc.so

When run inside Epiphany's runtime, I get

find: ‘/proc/tty/driver’: Permission denied
find: ‘/run/host/usr/etc/audit’: Permission denied
find: ‘/run/host/usr/etc/credstore’: Permission denied
find: ‘/run/host/usr/etc/credstore.encrypted’: Permission denied
find: ‘/run/host/usr/etc/cups/ssl’: Permission denied
find: ‘/run/host/usr/etc/dhcp’: Permission denied
find: ‘/run/host/usr/etc/firewalld’: Permission denied
find: ‘/run/host/usr/etc/grub.d’: Permission denied
find: ‘/run/host/usr/etc/lvm/archive’: Permission denied
find: ‘/run/host/usr/etc/lvm/backup’: Permission denied
find: ‘/run/host/usr/etc/lvm/cache’: Permission denied
find: ‘/run/host/usr/etc/nftables’: Permission denied
find: ‘/run/host/usr/etc/openvpn/client’: Permission denied
find: ‘/run/host/usr/etc/openvpn/server’: Permission denied
find: ‘/run/host/usr/etc/polkit-1/localauthority’: Permission denied
find: ‘/run/host/usr/etc/polkit-1/rules.d’: Permission denied
find: ‘/run/host/usr/etc/selinux/final’: Permission denied
find: ‘/run/host/usr/etc/selinux/targeted/active’: Permission denied
find: ‘/run/host/usr/etc/sos/cleaner’: Permission denied
find: ‘/run/host/usr/etc/ssh/sshd_config.d’: Permission denied
find: ‘/run/host/usr/etc/sssd’: Permission denied
find: ‘/run/host/usr/etc/sudoers.d’: Permission denied
find: ‘/run/host/usr/etc/wireguard’: Permission denied
find: ‘/run/host/usr/lib/ostree-boot/efi’: Permission denied
find: ‘/run/host/usr/lib/ostree-boot/grub2’: Permission denied
find: ‘/run/host/usr/libexec/initscripts/legacy-actions/auditd’: Permission denied
find: ‘/run/host/usr/share/empty.sshd’: Permission denied

sudo does not exist inside the runtime

[Michael Catanzaro]

Well it's not going to be in any of those locations, so sounds like libnvrtc.so is not provided by the flatpak extension at all. So then why is something trying to load it?

This needs developers familiar with NVIDIA. I'm afraid no developers on this Bugzilla have much experience with NVIDIA. I'll point a couple people at this and see if we can find somebody who understands what's happening....

[Kdwk]

This file might not be on my computer at all. I just searched for the file in Files from / 

No result.

[Kdwk]

I'm pleased to report that videos on Reddit work fine after all. It might 'just' be the YouTube bug from https://bugs.webkit.org/show_bug.cgi?id=248224

[Kdwk]

(In reply to kdwkleung from comment #33)
> I'm pleased to report that videos on Reddit work fine after all. It might
> 'just' be the YouTube bug from https://bugs.webkit.org/show_bug.cgi?id=248224

Yeah I don't think so. Even embedded YouTube videos cause the tab to hang

[Kdwk]

The tabs don't hang but videos are still green on Nvidia