[GStreamer] Won't play media and video in GtkOffscreenWindow

• Component: Media
• Status: NEW ---

[Po Lu]

I can't get video to play in a WebKitWebView attached to a
GtkOffscreenWindow.  After setting up the logic to pass through button
events, and upon clicking the "play" button in, for instance, the
video at the front page of gnu.org, the playback area flickers white
for a second, then does nothing.

Media set to autoplay doesn't play either, with
`media-playback-requires-user-gesture' set to TRUE in WebKitSettings.

Following the instructions in the WebKitGTK/Debugging wiki page, no
gst.log is produced with the following environment variables upon
trying to play a video:

   export GST_DEBUG="3,webkit*:6" GST_DEBUG_FILE="$HOME/gst.log" \
          GST_DEBUG_NO_COLOR=1 WEBKIT_FORCE_SANDBOX=0

With a GST build with logs enabled.

With media logs enabled, I see the following output in the journal:

-- Journal begins at Mon 2021-11-01 18:10:02 CST. --
Nov 08 19:55:02 hostname WebKitWebProcess[658126]: MediaElementSession::clientDataBufferingTimerFired(8FE379B8B31B3CE5) visible = false
Nov 08 19:55:05 hostname WebKitWebProcess[658126]: HTMLMediaElement::play(8FE379B8B31B3CE5)
Nov 08 19:55:05 hostname WebKitWebProcess[658126]: MediaElementSession::playbackStateChangePermitted(8FE379B8B31B3CE5) state = 0
Nov 08 19:55:05 hostname WebKitWebProcess[658126]: MediaElementSession::removeBehaviorRestriction(8FE379B8B31B3CE5) removed RequireUserGestureForAudioRateChange, RequireUserGestureForFullscreen, RequireUserGestureToControlControlsManager
Nov 08 19:55:05 hostname WebKitWebProcess[658126]: HTMLMediaElement::playInternal(8FE379B8B31B3CE5)
Nov 08 19:55:05 hostname WebKitWebProcess[658126]: PlatformMediaSessionManager::addSession(0) (8FE379B8B31B3CE5)
Nov 08 19:55:05 hostname WebKitWebProcess[658126]: MediaElementSession::clientWillBeginPlayback(8FE379B8B31B3CE5) state = Autoplaying
Nov 08 19:55:05 hostname WebKitWebProcess[658126]: PlatformMediaSessionManager::setCurrentSession(0) (8FE379B8B31B3CE5)
Nov 08 19:55:05 hostname WebKitWebProcess[658126]: PlatformMediaSessionManager::sessionWillBeginPlayback(0) (8FE379B8B31B3CE5) returning true
Nov 08 19:55:05 hostname WebKitWebProcess[658126]: MediaElementSession::setState(8FE379B8B31B3CE5) Playing
Nov 08 19:55:05 hostname WebKitWebProcess[658126]: HTMLMediaElement::setShouldDelayLoadEvent(8FE379B8B31B3CE5) true
Nov 08 19:55:05 hostname WebKitWebProcess[658126]: MediaElementSession::pageAllowsDataLoading(8FE379B8B31B3CE5) returning FALSE
Nov 08 19:55:05 hostname WebKitWebProcess[658126]: HTMLMediaElement::selectMediaResource(8FE379B8B31B3CE5) not allowed to load in background, waiting
Nov 08 19:55:05 hostname WebKitWebProcess[658126]: HTMLMediaElement::setShouldDelayLoadEvent(8FE379B8B31B3CE5) false
Nov 08 19:55:05 hostname WebKitWebProcess[658126]: HTMLMediaElement::setShowPosterFlag(8FE379B8B31B3CE5) false
Nov 08 19:55:05 hostname WebKitWebProcess[658126]: HTMLMediaElement::setAutoplayEventPlaybackState(8FE379B8B31B3CE5) StartedWithUserGesture
Nov 08 19:55:05 hostname WebKitWebProcess[658126]: HTMLMediaElement::dispatchEvent(8FE379B8B31B3CE5) play
Nov 08 19:55:05 hostname WebKitWebProcess[658126]: HTMLMediaElement::dispatchEvent(8FE379B8B31B3CE5) waiting

I remember this working fine in an offscreen window about a year or so
ago, so this is probably a regression.  If it helps, I tested this on
Fedora 34, with WebKitGTK 3.34.1.  Thanks in advance.

[Xabier Rodríguez Calvar]

There should be a gst.log file in your home if you applied those env vars.

[Po Lu]

(In reply to Xabier Rodríguez Calvar from comment #1)
> There should be a gst.log file in your home if you applied those env vars.

There was no such log.  But running `GST_DEBUG=5 gst-launch-1.0
fakesrc ! fakesink' with those environment variables set did produce a
large log, so the environment variables are correct.

[Liang-jie Lee]

Is there any update on this bug? If this can be fixed, then we can have Emacs playing videos with webkit, which is a feature wished by the community for many years.

[Philippe Normand]

Well, we still need gst logs...

[Liang-jie Lee]

Created attachment 467047 [details]
GST log

[Liang-jie Lee]

Comment on attachment 467047 [details]
GST log

Hi Po Lu and the Webkit developers,

TL;DR, I tried to reproduce the issue and the attachment is my gst.log.

Environment: Debian 10, webkit-2.38.4, Emacs 30.0.50

Steps to reproduce: (Since I know nothing about webkit and Emacs internal, bear with me that I might be too verbose here)

For the webkit part, this is how I compiled the library:
1. wget https://webkitgtk.org/releases/webkitgtk-2.38.4.tar.xz
2. tar -xvf webkitgtk-2.38.4.tar.xz && cd webkitgtk-2.38.4
3. cmake -DPORT=GTK -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_JOURNALD_LOG=ON -GNinja
4. ninja && sudo ninja install

For the Emacs part:
1. Compile Emacs with xwidget support (e.g. ./configure --with-xwidget)
2. Start Emacs with GST log (e.g. GST_DEBUG="3,webkit*:6" GST_DEBUG_FILE="$HOME/gst.log" GST_DEBUG_NO_COLOR=1 WEBKIT_DISABLE_SANDBOX_THIS_IS_DANGEROUS=1 emacs)
3. M-x xwidget-webkit-browse-url, and visit https://yotube.com
4. Click some videos and the video playback is shown. But no video is played.

Thanks.

[Philippe Normand]

Are you really playing anything? The log has no mediaplayer logs...
Please try something simpler, like any basic static html5 video, no YT.

[Liang-jie Lee]

Created attachment 467049 [details]
GST log (gnu.org)

How about this one? I'm visiting gnu.org which I believe their website should be really simple. BTW, what should the mediaplayer log looks like?

[Philippe Normand]

Did you ... hit the play button?

[Liang-jie Lee]

Yes, I did. For both the youtube and gnu cases, when I hit the play button, the video started loading and stuck there infinitely. I'm sure I'm not the only person encounter this. For example, on reddit (https://www.reddit.com/r/emacs/comments/wm7hz8/videos_on_webkit_stuck_in_infinite_load/), there are users reporting same problems.

Po Lu is it possible to reproduce this problem with a simpler demo program without involving Emacs? We need your help to add more information about how Emacs utilize these libraries. Thanks.

[Michael Catanzaro]

Try disconnecting emacs's decide-policy callback webkit_decide_policy_cb(). Does that make any difference?