Bug 39474 – [GStreamer] GTK XOverlay support in GStreamerGWorld

• Component: Media
• Status: RESOLVED FIXED

[Philippe Normand]

GStreamerGWorld should be able to capture messages coming from the platform video sink and use the gst_x_overlay API to configure a native window that can then later on be used for fullscreen video rendering.

[Philippe Normand]

Created attachment 57336 [details]
proposed patch

[Philippe Normand]

Created attachment 57564 [details]
proposed patch

[Philippe Normand]

Sebastian, can you have a look at the patch when you get some time please? :)

[Sebastian Dröge (slomo)]

<slomo> philn-tp: also, the video-sink of playbin2 is not always a GstBin, what you do there is not a good idea
<slomo> philn-tp: create your own bin and use it as video-sink
<slomo> philn-tp: i mean, *always* use your own GstBin as videosink, which has a tee in the beginning and then splits this to the webkit cairo sink and if requested to the fullscreen videosink

The video-sink of playbin2 is not always a GstBin and you really shouldn't do things like that with a bin where you don't know what it is ;)

What you should do is the following:
- create a GstBin, put a tee in there and plug after that the webkit cairo video sink (and queue, ffmpegcolorspace, videoscale)
- If fullscreen is requested, request a new pad from the tee and link your fullscreen stuff to it

*Always* do that, create this video sink bin together with playbin2 and set it as video-sink.

[Philippe Normand]

(In reply to comment #4)
> <slomo> philn-tp: also, the video-sink of playbin2 is not always a GstBin, what you do there is not a good idea
> <slomo> philn-tp: create your own bin and use it as video-sink
> <slomo> philn-tp: i mean, *always* use your own GstBin as videosink, which has a tee in the beginning and then splits this to the webkit cairo sink and if requested to the fullscreen videosink
> 
> The video-sink of playbin2 is not always a GstBin and you really shouldn't do things like that with a bin where you don't know what it is ;)
> 
> What you should do is the following:
> - create a GstBin, put a tee in there and plug after that the webkit cairo video sink (and queue, ffmpegcolorspace, videoscale)
> - If fullscreen is requested, request a new pad from the tee and link your fullscreen stuff to it
> 
> *Always* do that, create this video sink bin together with playbin2 and set it as video-sink.

This is what i do in the patch of bug 39472. Sorry about the confusion ;)

[Sebastian Dröge (slomo)]

Oh, sorry. I saw the two patches in the wrong order. The patch looks good :)

[Philippe Normand]

Created attachment 57745 [details]
proposed patch

No feature change compared to previous patch. I only removed cruft and
renamed variables.

[Philippe Normand]

Created attachment 57776 [details]
proposed patch

This new version improves the new-segment event sending code when
entering fullscreen. Previous patch had hard-coded values, which is
bad ;)

[Philippe Normand]

Created attachment 57849 [details]
proposed patch

This version fixes some memory leaks spotted by Sebastian. Thanks!

[Gustavo Noronha (kov)]

Comment on attachment 57849 [details]
proposed patch

WebCore/platform/graphics/gstreamer/GStreamerGWorld.cpp:113
 +      tee = gst_bin_get_by_name(GST_BIN(videoSink.get()), "t");
Variables should be declared at first usage in cases like this.

WebCore/platform/graphics/gstreamer/GStreamerGWorld.cpp:142
 +      // See https://bugzilla.gnome.org/show_bug.cgi?id=620490.
Perhaps you can just do a check, conditional on having a new enough gstreamer, so we have the feature in builds with newer versions, without the need to require them?

 165     GOwnPtr<GstElement> videoSink;
 166     GstElement* tee;
 167     GstElement* colorspace;
 168     GstElement* platformVideoSink;
 169     GstElement* queue;
 170     GstElement* videoScale;
 171     GstPad* srcPad;
 172     GstPad* sinkPad;

Same thing here, many of these declarations should be done at first usage - I know some of them need to be pre-declared, like videoSink, but they can be pre-declared in the line immediately above the function that gets them - g_object_get in this case.

I can't find what makes the m_fullscreenWindow go fullscreen (like a call to gtk_window_fullscreen), is this planned to be handled elsewhere? I also did not find who calls enterFullscreen, I think you're planning to land this first, and then wiring it up? I'd prefer to have the wiring up in this patch, otherwise we're landing dead code. I'll mark this r- so we can work on the issues I mentioned, thanks for working on this! =)

[Philippe Normand]

(In reply to comment #10)
> 
> I can't find what makes the m_fullscreenWindow go fullscreen (like a call to gtk_window_fullscreen), is this planned to be handled elsewhere? I also did not find who calls enterFullscreen, I think you're planning to land this first, and then wiring it up? I'd prefer to have the wiring up in this patch, otherwise we're landing dead code. I'll mark this r- so we can work on the issues I mentioned, thanks for working on this! =)

Thanks for the review.
I didn't include the "wiring" in this patch because it would have been to big. Like you guessed I plan to submit another patch including a FullscreenVideoController and the missing bits for enabling fullscreen video in the gtk+ port.

I'll send a new patch fixing the issues you spotted

[Philippe Normand]

Created attachment 59740 [details]
proposed patch

[Philippe Normand]

Created attachment 61149 [details]
gtk xoverlay support in GStreamerGWorld

[Philippe Normand]

Created attachment 61150 [details]
fullscreen video support in WebKit/gtk

[Philippe Normand]

Created attachment 61151 [details]
tests rebaselined

[Philippe Normand]

Created attachment 61153 [details]
fullscreen video support in WebKit/gtk

style fixes

[Philippe Normand]

*** Bug 32798 has been marked as a duplicate of this bug. ***

[Philippe Normand]

Created attachment 61771 [details]
fullscreen support in WebKit/gtk

[Philippe Normand]

Comment on attachment 61771 [details]
fullscreen support in WebKit/gtk

Err, incomplete patch uploaded

[WebKit Review Bot]

Attachment 61771 [details] did not build on gtk:
Build output: http://webkit-commit-queue.appspot.com/results/3576049

[Philippe Normand]

Created attachment 61773 [details]
fullscreen video support in WebKit/gtk

[WebKit Review Bot]

Attachment 61773 [details] did not build on gtk:
Build output: http://webkit-commit-queue.appspot.com/results/3558056

[Philippe Normand]

Created attachment 61777 [details]
fullscreen video support in WebKit/gtk

[Gustavo Noronha (kov)]

Comment on attachment 61777 [details]
fullscreen video support in WebKit/gtk

 79 #if PLATFORM(GTK)
 80         m_fullscreenWindow = gtk_window_new(GTK_WINDOW_TOPLEVEL);
 81         gtk_widget_set_events(m_fullscreenWindow, GDK_POINTER_MOTION_MASK | GDK_KEY_PRESS_MASK | GDK_FOCUS_CHANGE_MASK);
 82 
 83         m_videoWindow = gtk_drawing_area_new();
 84         gtk_widget_set_double_buffered(m_videoWindow, FALSE);
 85         gtk_container_add(GTK_CONTAINER(m_fullscreenWindow), m_videoWindow);
 86 
 87         gtk_widget_show_all(m_fullscreenWindow);
 88         gtk_widget_realize(m_fullscreenWindow);
 89         gtk_widget_hide_all(m_fullscreenWindow);

This sequence looks very weird to me. What exactly do you intend to do here? Do you need both m_fullscreenWindow, and m_videoWindow realized? Why not just call realize on both? But, what comes later is a bit more weird:

 90 
 91 #ifdef GDK_WINDOWING_X11
 92         gdk_threads_enter();
 93         gdk_display_sync(gdk_display_get_default());
 94         m_videoWindowId = GDK_WINDOW_XID(gtk_widget_get_window(m_fullscreenWindow));
 95         gdk_threads_leave();
 96 #endif
 97 #endif
 98     }

If you think gdk_threads_enter() is required here, it should also be used above in the gtk calls, why are you not using it there? For the future, to avoid having a lot of #ifdefing, we may want to replace these with calls to, say, platformEnterFullScreen, that would be implemented in a platform/graphics/GStreamerGWorldGtk.cpp, or maybe platform/graphics/gstreamer/gtk/GStreamerGWorldGtk.cpp, similar to how, say, platform-specific parts of ResourceRequest are_factory_make("queue", "queue"); Sorry for not spotting these in earlier patches =(. I'm still looking at the patch, let me checkpoint here, though.

[Gustavo Noronha (kov)]

Comment on attachment 61777 [details]
fullscreen video support in WebKit/gtk

 286     gtk_widget_show_all(m_window);
 287     gtk_widget_realize(m_window);

No point in showing, then realizing. Showing triggers realizing. You seem to only need realize at this point, but it seems you want it shown anyway, so just show it.

 290     GdkCursor* cursor = gdk_cursor_new(GDK_BLANK_CURSOR);
 291     m_cursor = gdk_window_get_cursor(window);
 292     gdk_window_set_cursor(window, cursor);

I think you need to ref this cursor. Since the cursor is owned by the window, the pointer you got will not necessarily (and more likely not) exist after you set the blank cursor on it.

 298     gtk_window_fullscreen(GTK_WINDOW(m_window));
 299     gtk_widget_show_all(m_hudWindow);
 300     showHud(true);

You are already showing the m_hudWindow inside showHud, why do you need this call to gtk_widget_show_all() here?

 319     // Scale the hud to the full width of the screen.
 320     gtk_window_resize(GTK_WINDOW(m_hudWindow), fullscreenRectangle.width, hudHeight);

Scale -> resize. Scale gives the wrong idea.

Some general comments: we have a number of callbacks taking the video controller object as the data. This has been, I think, one of the top three causes of crashes in webkitgtk code as far as I remember. mrobinson figured out a way to make this kind of code more robust, using GClosures. Check this change out, I'd recommend doing something similar for these: http://trac.webkit.org/changeset/57901. Also, you don't seem to have many checks in place for the case where enterFullScreen fails, and returns 0.

There are two things I really don't like: the code uses the media element directly for many things, and seems to duplicate some code, I think that might be something we'll have to accept, though, I haven't been able to assess if you could use the player instead of messing with the element. I really dislike how enterFullscreen returns a gulong, though. That feels really dirty heh. I would prefer having another method, platformWindow(), say, that would return a typedef'ed type, having each implementation return its window type, with no need for ugly reinterpret_cast'ing, and stuff. enterFullScreen could return a boolean to let the caller know things went through or not. So you'd have, instead of:

 266     m_window = reinterpret_cast<GtkWidget*>(m_mediaElement->platformMedia().media.gstreamerGWorld->enterFullscreen());

this:

    if (!m_mediaElement->platformMedia().media.gstreamerGWorld->enterFullscreen())
        return;

    m_window = m_mediaElement->platformMedia().media.gstreamerGWorld->platformWindow();

I don't really like these huge chains, too, btw, can we just hold a pointer to gstreamerGWorld? Finally, I think this design is a good enough one for a first iteration, but I think we should consider factoring out the "interfacing with the media element/player" into a MediaPlayerClient so that we could provide API for browsers to know about/control the media elements in a way that is higher level than using the DOM bindings. I'll say r- for now so we can get these comments worked on, thanks for working on this, again =).

[Philippe Normand]

(In reply to comment #25)
> (From update of attachment 61777 [details])
>  286     gtk_widget_show_all(m_window);
>  287     gtk_widget_realize(m_window);
> 
> No point in showing, then realizing. Showing triggers realizing. You seem to only need realize at this point, but it seems you want it shown anyway, so just show it.

I just need it realized to access the window, just below. Removed the show_all call.

> 
>  290     GdkCursor* cursor = gdk_cursor_new(GDK_BLANK_CURSOR);
>  291     m_cursor = gdk_window_get_cursor(window);
>  292     gdk_window_set_cursor(window, cursor);
> 
> I think you need to ref this cursor. Since the cursor is owned by the window, the pointer you got will not necessarily (and more likely not) exist after you set the blank cursor on it.
> 

I guess you're right although I didn't observe what you states ;) I now use GRefPtr for the cursor, just to be safe.

>  298     gtk_window_fullscreen(GTK_WINDOW(m_window));
>  299     gtk_widget_show_all(m_hudWindow);
>  300     showHud(true);
> 
> You are already showing the m_hudWindow inside showHud, why do you need this call to gtk_widget_show_all() here?
> 

Remove show_all call ;)

>  319     // Scale the hud to the full width of the screen.
>  320     gtk_window_resize(GTK_WINDOW(m_hudWindow), fullscreenRectangle.width, hudHeight);
> 
> Scale -> resize. Scale gives the wrong idea.
> 

ok

> Some general comments: we have a number of callbacks taking the video controller object as the data. This has been, I think, one of the top three causes of crashes in webkitgtk code as far as I remember. mrobinson figured out a way to make this kind of code more robust, using GClosures. Check this change out, I'd recommend doing something similar for these: http://trac.webkit.org/changeset/57901. 

Sorry but I don't understand this. Where exactly should I use GClosures? g_signal_connect already internally uses GClosures. Would you please explain a bit more?

> Also, you don't seem to have many checks in place for the case where enterFullScreen fails, and returns 0.
> 

Made it return a bool and adapted the controller's enterFullscreen accordingly.

> There are two things I really don't like: the code uses the media element directly for many things, and seems to duplicate some code, I think that might be something we'll have to accept, though, I haven't been able to assess if you could use the player instead of messing with the element. 

Using the private player you mean? or the MediaPlayer instance?
It'd be nice if I could use the MediaPlayer instance but using the media element here is better (I think) because it's the highest-level API. So by using it we are sure our controller is using the high level APIs as defined in the spec.

> I really dislike how enterFullscreen returns a gulong, though. That feels really dirty heh. I would prefer having another method, platformWindow(), say, that would return a typedef'ed type, having each implementation return its window type, with no need for ugly reinterpret_cast'ing, and stuff. enterFullScreen could return a boolean to let the caller know things went through or not. So you'd have, instead of:
> 
>  266     m_window = reinterpret_cast<GtkWidget*>(m_mediaElement->platformMedia().media.gstreamerGWorld->enterFullscreen());
> 
> this:
> 
>     if (!m_mediaElement->platformMedia().media.gstreamerGWorld->enterFullscreen())
>         return;
> 
>     m_window = m_mediaElement->platformMedia().media.gstreamerGWorld->platformWindow();
> 

Done that, I think :)
A PlatformVideoWindow class with a Gtk implementation.

> I don't really like these huge chains, too, btw, can we just hold a pointer to gstreamerGWorld?

Sure!

> Finally, I think this design is a good enough one for a first iteration, but I think we should consider factoring out the "interfacing with the media element/player" into a MediaPlayerClient so that we could provide API for browsers to know about/control the media elements in a way that is higher level than using the DOM bindings. I'll say r- for now so we can get these comments worked on, thanks for working on this, again =).

Yes that'd be really great. The MediaPlayer would need some changes to accept multiple clients instead of only one though.

Will attach the updated patch once I tested it in my debug build

[Philippe Normand]

Created attachment 62047 [details]
fullscreen video support in WebKit/gtk

[Gustavo Noronha (kov)]

Comment on attachment 62047 [details]
fullscreen video support in WebKit/gtk

WebKit/gtk/WebCoreSupport/FullscreenVideoController.cpp:571
 +      g_object_set(m_exitFullscreenAction, "icon-name", "view-restore-symbolic", NULL);
symbolic icons are quite new, are we going to have to shift our dependencies for this?

 294     if (!node->hasTagName(HTMLNames::videoTag))
 295         return;
 296 
 297 #if ENABLE(VIDEO)

Might as well only do the check if video is enabled? Otherwise it's just useless work.

311 void webkit_web_view_exit_fullscreen(WebKitWebView* webView)
 312 {
 313 #if ENABLE(VIDEO)
 314     WebKitWebViewPrivate* priv = webView->priv;
 315     if (priv->fullscreenVideoController)
 316         priv->fullscreenVideoController->exitFullscreen();
 317 #endif

Certainly a good idea to NULLify the fullscreenVideoController here. Also, I believe you need to exit fullscreen in case a fullscreenVideoController exists during WebKitWebView's destruction? The callbacks receiving C++ objects as user data still concern me, I'm pretty sure we'll get crashes from the object going away and the callback being called right after that.

These are my last concerns, I think, the rest of the patch looks good, so let's try to clear them up and land =D

[Martin Robinson]

Comment on attachment 61151 [details]
tests rebaselined

You shouldn't need a review for rebaselined tests. I'm not sure if these are valid any longer, but if they aren't, you can commit them without review when they are ready.

[Philippe Normand]

Created attachment 64202 [details]
fullscreen video support in WebKit/gtk

Adressed the changes asked by Gustavo. Enabled symbolici icons only if
we build with GTK+3 so the GTK+2 requirement doesn't need to be
bumped, feel free to comment on that :)

About the signal callbacks I still fail to understand how to use
GClosures. Will poke Martin about it.

[Philippe Normand]

Created attachment 64203 [details]
tests rebaselined

[Gustavo Noronha (kov)]

(In reply to comment #29)
> (From update of attachment 61151 [details])
> You shouldn't need a review for rebaselined tests. I'm not sure if these are valid any longer, but if they aren't, you can commit them without review when they are ready.

Indeed, and they should very likely go in the same patch as the one that requires the rebaselines, to avoid an unnecessary failure->fixed cycle =) I'll look at the new patch in a few hours.

[Eric Seidel (no email)]

Comment on attachment 61151 [details]
tests rebaselined

Cleared Martin Robinson's review+ from obsolete attachment 61151 [details] so that this bug does not appear in http://webkit.org/pending-commit.

[Martin Robinson]

Comment on attachment 64203 [details]
tests rebaselined

No need to rebaseline these tests in a separate patch. Keep
them with your fullscreen change, so that they bots don't
become red between the patches.

[Philippe Normand]

Created attachment 64557 [details]
fullscreen video support in WebKit/gtk

[Philippe Normand]

Created attachment 64717 [details]
fullscreen video support in WebKit/gtk

Removed some spurious pointer casts in the signal disconnection calls
of exitFullscreen().

About the GClosure usage in this context, Martin said it shouldn't be
needed as long as the signal handlers are disconnected during the
instance destruction, which is the case.

[Gustavo Noronha (kov)]

Comment on attachment 64717 [details]
fullscreen video support in WebKit/gtk

WebCore/platform/graphics/gstreamer/PlatformVideoWindow.h:46
 +          gulong m_videoWindowId;
Is gulong portable enough? Should we use a typedef similar to the PlatformWindowType here?

 418     void webkit_web_view_enter_fullscreen(WebKitWebView* webView, WebCore::Node* node);
 419     void webkit_web_view_exit_fullscreen(WebKitWebView* webView);

Make these follow the standard WebKit naming, since they are not really API, I think. webkitWebViewEnterFullscreen, for instance.

WebKit/gtk/WebCoreSupport/FullscreenVideoController.cpp:298
 +  
Needless empty line.

WebKit/gtk/WebCoreSupport/FullscreenVideoController.cpp:590
 +  
One more needless empty line =).

Looks good to me, thanks for clarifying the GClosure point, btw, let's get this thing on the wild!

[Philippe Normand]

(In reply to comment #37)
> (From update of attachment 64717 [details])
> WebCore/platform/graphics/gstreamer/PlatformVideoWindow.h:46
>  +          gulong m_videoWindowId;
> Is gulong portable enough? Should we use a typedef similar to the PlatformWindowType here?

I believe gulong is fine. This is what gst_x_overlay_set_xwindow_id() requires anyway.

On Mac OS X it will be a cast of a NSView pointer and on Windows a cast of a HWND variable. I think it'd be fine to have those casts in the platform-specific implementations.

I adressed the last requested changes. I also cleaned up the signal and g_source callbacks in the controller to avoid the casts, thanks to the wise help of Martin.

Will land the patch, thanks for the long-standing review ;)

[Philippe Normand]

Landed in http://trac.webkit.org/changeset/65658

[Philippe Normand]

Comment on attachment 64717 [details]
fullscreen video support in WebKit/gtk

Clearing r+ flag of committed patch