RESOLVED FIXED 39474
[GStreamer] GTK XOverlay support in GStreamerGWorld
https://bugs.webkit.org/show_bug.cgi?id=39474
Summary [GStreamer] GTK XOverlay support in GStreamerGWorld
Philippe Normand
Reported 2010-05-20 23:54:38 PDT
GStreamerGWorld should be able to capture messages coming from the platform video sink and use the gst_x_overlay API to configure a native window that can then later on be used for fullscreen video rendering.
Attachments
proposed patch (10.41 KB, patch)
2010-05-28 09:54 PDT, Philippe Normand
no flags
proposed patch (10.90 KB, patch)
2010-06-01 10:35 PDT, Philippe Normand
no flags
proposed patch (10.94 KB, patch)
2010-06-03 01:39 PDT, Philippe Normand
no flags
proposed patch (11.48 KB, patch)
2010-06-03 09:02 PDT, Philippe Normand
no flags
proposed patch (12.01 KB, patch)
2010-06-04 00:47 PDT, Philippe Normand
gustavo: review-
proposed patch (12.14 KB, patch)
2010-06-25 00:41 PDT, Philippe Normand
no flags
gtk xoverlay support in GStreamerGWorld (12.33 KB, patch)
2010-07-10 04:29 PDT, Philippe Normand
no flags
fullscreen video support in WebKit/gtk (32.41 KB, patch)
2010-07-10 04:29 PDT, Philippe Normand
no flags
tests rebaselined (8.37 KB, patch)
2010-07-10 04:30 PDT, Philippe Normand
no flags
fullscreen video support in WebKit/gtk (32.62 KB, patch)
2010-07-10 04:38 PDT, Philippe Normand
no flags
fullscreen support in WebKit/gtk (5.27 KB, patch)
2010-07-16 00:32 PDT, Philippe Normand
no flags
fullscreen video support in WebKit/gtk (33.04 KB, patch)
2010-07-16 01:04 PDT, Philippe Normand
no flags
fullscreen video support in WebKit/gtk (44.60 KB, patch)
2010-07-16 01:33 PDT, Philippe Normand
gustavo: review-
fullscreen video support in WebKit/gtk (49.86 KB, patch)
2010-07-20 02:47 PDT, Philippe Normand
gustavo: review-
fullscreen video support in WebKit/gtk (50.30 KB, patch)
2010-08-12 03:12 PDT, Philippe Normand
no flags
tests rebaselined (12.33 KB, patch)
2010-08-12 03:13 PDT, Philippe Normand
mrobinson: review-
fullscreen video support in WebKit/gtk (62.39 KB, patch)
2010-08-17 00:40 PDT, Philippe Normand
no flags
fullscreen video support in WebKit/gtk (62.21 KB, patch)
2010-08-18 09:09 PDT, Philippe Normand
no flags
Philippe Normand
Comment 1 2010-05-28 09:54:41 PDT
Created attachment 57336 [details] proposed patch
Philippe Normand
Comment 2 2010-06-01 10:35:55 PDT
Created attachment 57564 [details] proposed patch
Philippe Normand
Comment 3 2010-06-01 10:38:00 PDT
Sebastian, can you have a look at the patch when you get some time please? :)
Sebastian Dröge (slomo)
Comment 4 2010-06-01 11:01:52 PDT
<slomo> philn-tp: also, the video-sink of playbin2 is not always a GstBin, what you do there is not a good idea <slomo> philn-tp: create your own bin and use it as video-sink <slomo> philn-tp: i mean, *always* use your own GstBin as videosink, which has a tee in the beginning and then splits this to the webkit cairo sink and if requested to the fullscreen videosink The video-sink of playbin2 is not always a GstBin and you really shouldn't do things like that with a bin where you don't know what it is ;) What you should do is the following: - create a GstBin, put a tee in there and plug after that the webkit cairo video sink (and queue, ffmpegcolorspace, videoscale) - If fullscreen is requested, request a new pad from the tee and link your fullscreen stuff to it *Always* do that, create this video sink bin together with playbin2 and set it as video-sink.
Philippe Normand
Comment 5 2010-06-01 11:18:20 PDT
(In reply to comment #4) > <slomo> philn-tp: also, the video-sink of playbin2 is not always a GstBin, what you do there is not a good idea > <slomo> philn-tp: create your own bin and use it as video-sink > <slomo> philn-tp: i mean, *always* use your own GstBin as videosink, which has a tee in the beginning and then splits this to the webkit cairo sink and if requested to the fullscreen videosink > > The video-sink of playbin2 is not always a GstBin and you really shouldn't do things like that with a bin where you don't know what it is ;) > > What you should do is the following: > - create a GstBin, put a tee in there and plug after that the webkit cairo video sink (and queue, ffmpegcolorspace, videoscale) > - If fullscreen is requested, request a new pad from the tee and link your fullscreen stuff to it > > *Always* do that, create this video sink bin together with playbin2 and set it as video-sink. This is what i do in the patch of bug 39472. Sorry about the confusion ;)
Sebastian Dröge (slomo)
Comment 6 2010-06-03 01:11:50 PDT
Oh, sorry. I saw the two patches in the wrong order. The patch looks good :)
Philippe Normand
Comment 7 2010-06-03 01:39:09 PDT
Created attachment 57745 [details] proposed patch No feature change compared to previous patch. I only removed cruft and renamed variables.
Philippe Normand
Comment 8 2010-06-03 09:02:00 PDT
Created attachment 57776 [details] proposed patch This new version improves the new-segment event sending code when entering fullscreen. Previous patch had hard-coded values, which is bad ;)
Philippe Normand
Comment 9 2010-06-04 00:47:30 PDT
Created attachment 57849 [details] proposed patch This version fixes some memory leaks spotted by Sebastian. Thanks!
Gustavo Noronha (kov)
Comment 10 2010-06-24 07:05:56 PDT
Comment on attachment 57849 [details] proposed patch WebCore/platform/graphics/gstreamer/GStreamerGWorld.cpp:113 + tee = gst_bin_get_by_name(GST_BIN(videoSink.get()), "t"); Variables should be declared at first usage in cases like this. WebCore/platform/graphics/gstreamer/GStreamerGWorld.cpp:142 + // See https://bugzilla.gnome.org/show_bug.cgi?id=620490. Perhaps you can just do a check, conditional on having a new enough gstreamer, so we have the feature in builds with newer versions, without the need to require them?  165 GOwnPtr<GstElement> videoSink;  166 GstElement* tee;  167 GstElement* colorspace;  168 GstElement* platformVideoSink;  169 GstElement* queue;  170 GstElement* videoScale;  171 GstPad* srcPad;  172 GstPad* sinkPad; Same thing here, many of these declarations should be done at first usage - I know some of them need to be pre-declared, like videoSink, but they can be pre-declared in the line immediately above the function that gets them - g_object_get in this case. I can't find what makes the m_fullscreenWindow go fullscreen (like a call to gtk_window_fullscreen), is this planned to be handled elsewhere? I also did not find who calls enterFullscreen, I think you're planning to land this first, and then wiring it up? I'd prefer to have the wiring up in this patch, otherwise we're landing dead code. I'll mark this r- so we can work on the issues I mentioned, thanks for working on this! =)
Philippe Normand
Comment 11 2010-06-24 23:21:57 PDT
(In reply to comment #10) > > I can't find what makes the m_fullscreenWindow go fullscreen (like a call to gtk_window_fullscreen), is this planned to be handled elsewhere? I also did not find who calls enterFullscreen, I think you're planning to land this first, and then wiring it up? I'd prefer to have the wiring up in this patch, otherwise we're landing dead code. I'll mark this r- so we can work on the issues I mentioned, thanks for working on this! =) Thanks for the review. I didn't include the "wiring" in this patch because it would have been to big. Like you guessed I plan to submit another patch including a FullscreenVideoController and the missing bits for enabling fullscreen video in the gtk+ port. I'll send a new patch fixing the issues you spotted
Philippe Normand
Comment 12 2010-06-25 00:41:33 PDT
Created attachment 59740 [details] proposed patch
Philippe Normand
Comment 13 2010-07-10 04:29:35 PDT
Created attachment 61149 [details] gtk xoverlay support in GStreamerGWorld
Philippe Normand
Comment 14 2010-07-10 04:29:55 PDT
Created attachment 61150 [details] fullscreen video support in WebKit/gtk
Philippe Normand
Comment 15 2010-07-10 04:30:19 PDT
Created attachment 61151 [details] tests rebaselined
Philippe Normand
Comment 16 2010-07-10 04:38:39 PDT
Created attachment 61153 [details] fullscreen video support in WebKit/gtk style fixes
Philippe Normand
Comment 17 2010-07-11 23:39:37 PDT
*** Bug 32798 has been marked as a duplicate of this bug. ***
Philippe Normand
Comment 18 2010-07-16 00:32:26 PDT
Created attachment 61771 [details] fullscreen support in WebKit/gtk
Philippe Normand
Comment 19 2010-07-16 00:36:16 PDT
Comment on attachment 61771 [details] fullscreen support in WebKit/gtk Err, incomplete patch uploaded
WebKit Review Bot
Comment 20 2010-07-16 00:46:29 PDT
Philippe Normand
Comment 21 2010-07-16 01:04:55 PDT
Created attachment 61773 [details] fullscreen video support in WebKit/gtk
WebKit Review Bot
Comment 22 2010-07-16 01:27:18 PDT
Philippe Normand
Comment 23 2010-07-16 01:33:22 PDT
Created attachment 61777 [details] fullscreen video support in WebKit/gtk
Gustavo Noronha (kov)
Comment 24 2010-07-16 07:54:03 PDT
Comment on attachment 61777 [details] fullscreen video support in WebKit/gtk  79 #if PLATFORM(GTK)  80 m_fullscreenWindow = gtk_window_new(GTK_WINDOW_TOPLEVEL);  81 gtk_widget_set_events(m_fullscreenWindow, GDK_POINTER_MOTION_MASK | GDK_KEY_PRESS_MASK | GDK_FOCUS_CHANGE_MASK);  82  83 m_videoWindow = gtk_drawing_area_new();  84 gtk_widget_set_double_buffered(m_videoWindow, FALSE);  85 gtk_container_add(GTK_CONTAINER(m_fullscreenWindow), m_videoWindow);  86  87 gtk_widget_show_all(m_fullscreenWindow);  88 gtk_widget_realize(m_fullscreenWindow);  89 gtk_widget_hide_all(m_fullscreenWindow); This sequence looks very weird to me. What exactly do you intend to do here? Do you need both m_fullscreenWindow, and m_videoWindow realized? Why not just call realize on both? But, what comes later is a bit more weird:  90  91 #ifdef GDK_WINDOWING_X11  92 gdk_threads_enter();  93 gdk_display_sync(gdk_display_get_default());  94 m_videoWindowId = GDK_WINDOW_XID(gtk_widget_get_window(m_fullscreenWindow));  95 gdk_threads_leave();  96 #endif  97 #endif  98 } If you think gdk_threads_enter() is required here, it should also be used above in the gtk calls, why are you not using it there? For the future, to avoid having a lot of #ifdefing, we may want to replace these with calls to, say, platformEnterFullScreen, that would be implemented in a platform/graphics/GStreamerGWorldGtk.cpp, or maybe platform/graphics/gstreamer/gtk/GStreamerGWorldGtk.cpp, similar to how, say, platform-specific parts of ResourceRequest are_factory_make("queue", "queue"); Sorry for not spotting these in earlier patches =(. I'm still looking at the patch, let me checkpoint here, though.
Gustavo Noronha (kov)
Comment 25 2010-07-16 10:47:53 PDT
Comment on attachment 61777 [details] fullscreen video support in WebKit/gtk  286 gtk_widget_show_all(m_window);  287 gtk_widget_realize(m_window); No point in showing, then realizing. Showing triggers realizing. You seem to only need realize at this point, but it seems you want it shown anyway, so just show it.  290 GdkCursor* cursor = gdk_cursor_new(GDK_BLANK_CURSOR);  291 m_cursor = gdk_window_get_cursor(window);  292 gdk_window_set_cursor(window, cursor); I think you need to ref this cursor. Since the cursor is owned by the window, the pointer you got will not necessarily (and more likely not) exist after you set the blank cursor on it.  298 gtk_window_fullscreen(GTK_WINDOW(m_window));  299 gtk_widget_show_all(m_hudWindow);  300 showHud(true); You are already showing the m_hudWindow inside showHud, why do you need this call to gtk_widget_show_all() here?  319 // Scale the hud to the full width of the screen.  320 gtk_window_resize(GTK_WINDOW(m_hudWindow), fullscreenRectangle.width, hudHeight); Scale -> resize. Scale gives the wrong idea. Some general comments: we have a number of callbacks taking the video controller object as the data. This has been, I think, one of the top three causes of crashes in webkitgtk code as far as I remember. mrobinson figured out a way to make this kind of code more robust, using GClosures. Check this change out, I'd recommend doing something similar for these: http://trac.webkit.org/changeset/57901. Also, you don't seem to have many checks in place for the case where enterFullScreen fails, and returns 0. There are two things I really don't like: the code uses the media element directly for many things, and seems to duplicate some code, I think that might be something we'll have to accept, though, I haven't been able to assess if you could use the player instead of messing with the element. I really dislike how enterFullscreen returns a gulong, though. That feels really dirty heh. I would prefer having another method, platformWindow(), say, that would return a typedef'ed type, having each implementation return its window type, with no need for ugly reinterpret_cast'ing, and stuff. enterFullScreen could return a boolean to let the caller know things went through or not. So you'd have, instead of:  266 m_window = reinterpret_cast<GtkWidget*>(m_mediaElement->platformMedia().media.gstreamerGWorld->enterFullscreen()); this: if (!m_mediaElement->platformMedia().media.gstreamerGWorld->enterFullscreen()) return; m_window = m_mediaElement->platformMedia().media.gstreamerGWorld->platformWindow(); I don't really like these huge chains, too, btw, can we just hold a pointer to gstreamerGWorld? Finally, I think this design is a good enough one for a first iteration, but I think we should consider factoring out the "interfacing with the media element/player" into a MediaPlayerClient so that we could provide API for browsers to know about/control the media elements in a way that is higher level than using the DOM bindings. I'll say r- for now so we can get these comments worked on, thanks for working on this, again =).
Philippe Normand
Comment 26 2010-07-20 00:32:31 PDT
(In reply to comment #25) > (From update of attachment 61777 [details]) >  286 gtk_widget_show_all(m_window); >  287 gtk_widget_realize(m_window); > > No point in showing, then realizing. Showing triggers realizing. You seem to only need realize at this point, but it seems you want it shown anyway, so just show it. I just need it realized to access the window, just below. Removed the show_all call. > >  290 GdkCursor* cursor = gdk_cursor_new(GDK_BLANK_CURSOR); >  291 m_cursor = gdk_window_get_cursor(window); >  292 gdk_window_set_cursor(window, cursor); > > I think you need to ref this cursor. Since the cursor is owned by the window, the pointer you got will not necessarily (and more likely not) exist after you set the blank cursor on it. > I guess you're right although I didn't observe what you states ;) I now use GRefPtr for the cursor, just to be safe. >  298 gtk_window_fullscreen(GTK_WINDOW(m_window)); >  299 gtk_widget_show_all(m_hudWindow); >  300 showHud(true); > > You are already showing the m_hudWindow inside showHud, why do you need this call to gtk_widget_show_all() here? > Remove show_all call ;) >  319 // Scale the hud to the full width of the screen. >  320 gtk_window_resize(GTK_WINDOW(m_hudWindow), fullscreenRectangle.width, hudHeight); > > Scale -> resize. Scale gives the wrong idea. > ok > Some general comments: we have a number of callbacks taking the video controller object as the data. This has been, I think, one of the top three causes of crashes in webkitgtk code as far as I remember. mrobinson figured out a way to make this kind of code more robust, using GClosures. Check this change out, I'd recommend doing something similar for these: http://trac.webkit.org/changeset/57901. Sorry but I don't understand this. Where exactly should I use GClosures? g_signal_connect already internally uses GClosures. Would you please explain a bit more? > Also, you don't seem to have many checks in place for the case where enterFullScreen fails, and returns 0. > Made it return a bool and adapted the controller's enterFullscreen accordingly. > There are two things I really don't like: the code uses the media element directly for many things, and seems to duplicate some code, I think that might be something we'll have to accept, though, I haven't been able to assess if you could use the player instead of messing with the element. Using the private player you mean? or the MediaPlayer instance? It'd be nice if I could use the MediaPlayer instance but using the media element here is better (I think) because it's the highest-level API. So by using it we are sure our controller is using the high level APIs as defined in the spec. > I really dislike how enterFullscreen returns a gulong, though. That feels really dirty heh. I would prefer having another method, platformWindow(), say, that would return a typedef'ed type, having each implementation return its window type, with no need for ugly reinterpret_cast'ing, and stuff. enterFullScreen could return a boolean to let the caller know things went through or not. So you'd have, instead of: > >  266 m_window = reinterpret_cast<GtkWidget*>(m_mediaElement->platformMedia().media.gstreamerGWorld->enterFullscreen()); > > this: > > if (!m_mediaElement->platformMedia().media.gstreamerGWorld->enterFullscreen()) > return; > > m_window = m_mediaElement->platformMedia().media.gstreamerGWorld->platformWindow(); > Done that, I think :) A PlatformVideoWindow class with a Gtk implementation. > I don't really like these huge chains, too, btw, can we just hold a pointer to gstreamerGWorld? Sure! > Finally, I think this design is a good enough one for a first iteration, but I think we should consider factoring out the "interfacing with the media element/player" into a MediaPlayerClient so that we could provide API for browsers to know about/control the media elements in a way that is higher level than using the DOM bindings. I'll say r- for now so we can get these comments worked on, thanks for working on this, again =). Yes that'd be really great. The MediaPlayer would need some changes to accept multiple clients instead of only one though. Will attach the updated patch once I tested it in my debug build
Philippe Normand
Comment 27 2010-07-20 02:47:24 PDT
Created attachment 62047 [details] fullscreen video support in WebKit/gtk
Gustavo Noronha (kov)
Comment 28 2010-08-02 15:44:04 PDT
Comment on attachment 62047 [details] fullscreen video support in WebKit/gtk WebKit/gtk/WebCoreSupport/FullscreenVideoController.cpp:571 + g_object_set(m_exitFullscreenAction, "icon-name", "view-restore-symbolic", NULL); symbolic icons are quite new, are we going to have to shift our dependencies for this?  294 if (!node->hasTagName(HTMLNames::videoTag))  295 return;  296  297 #if ENABLE(VIDEO) Might as well only do the check if video is enabled? Otherwise it's just useless work. 311 void webkit_web_view_exit_fullscreen(WebKitWebView* webView)  312 {  313 #if ENABLE(VIDEO)  314 WebKitWebViewPrivate* priv = webView->priv;  315 if (priv->fullscreenVideoController)  316 priv->fullscreenVideoController->exitFullscreen();  317 #endif Certainly a good idea to NULLify the fullscreenVideoController here. Also, I believe you need to exit fullscreen in case a fullscreenVideoController exists during WebKitWebView's destruction? The callbacks receiving C++ objects as user data still concern me, I'm pretty sure we'll get crashes from the object going away and the callback being called right after that. These are my last concerns, I think, the rest of the patch looks good, so let's try to clear them up and land =D
Martin Robinson
Comment 29 2010-08-11 18:15:37 PDT
Comment on attachment 61151 [details] tests rebaselined You shouldn't need a review for rebaselined tests. I'm not sure if these are valid any longer, but if they aren't, you can commit them without review when they are ready.
Philippe Normand
Comment 30 2010-08-12 03:12:29 PDT
Created attachment 64202 [details] fullscreen video support in WebKit/gtk Adressed the changes asked by Gustavo. Enabled symbolici icons only if we build with GTK+3 so the GTK+2 requirement doesn't need to be bumped, feel free to comment on that :) About the signal callbacks I still fail to understand how to use GClosures. Will poke Martin about it.
Philippe Normand
Comment 31 2010-08-12 03:13:02 PDT
Created attachment 64203 [details] tests rebaselined
Gustavo Noronha (kov)
Comment 32 2010-08-12 05:56:45 PDT
(In reply to comment #29) > (From update of attachment 61151 [details]) > You shouldn't need a review for rebaselined tests. I'm not sure if these are valid any longer, but if they aren't, you can commit them without review when they are ready. Indeed, and they should very likely go in the same patch as the one that requires the rebaselines, to avoid an unnecessary failure->fixed cycle =) I'll look at the new patch in a few hours.
Eric Seidel (no email)
Comment 33 2010-08-13 21:33:03 PDT
Comment on attachment 61151 [details] tests rebaselined Cleared Martin Robinson's review+ from obsolete attachment 61151 [details] so that this bug does not appear in http://webkit.org/pending-commit.
Martin Robinson
Comment 34 2010-08-16 11:02:59 PDT
Comment on attachment 64203 [details] tests rebaselined No need to rebaseline these tests in a separate patch. Keep them with your fullscreen change, so that they bots don't become red between the patches.
Philippe Normand
Comment 35 2010-08-17 00:40:00 PDT
Created attachment 64557 [details] fullscreen video support in WebKit/gtk
Philippe Normand
Comment 36 2010-08-18 09:09:36 PDT
Created attachment 64717 [details] fullscreen video support in WebKit/gtk Removed some spurious pointer casts in the signal disconnection calls of exitFullscreen(). About the GClosure usage in this context, Martin said it shouldn't be needed as long as the signal handlers are disconnected during the instance destruction, which is the case.
Gustavo Noronha (kov)
Comment 37 2010-08-18 15:40:17 PDT
Comment on attachment 64717 [details] fullscreen video support in WebKit/gtk WebCore/platform/graphics/gstreamer/PlatformVideoWindow.h:46 + gulong m_videoWindowId; Is gulong portable enough? Should we use a typedef similar to the PlatformWindowType here?  418 void webkit_web_view_enter_fullscreen(WebKitWebView* webView, WebCore::Node* node);  419 void webkit_web_view_exit_fullscreen(WebKitWebView* webView); Make these follow the standard WebKit naming, since they are not really API, I think. webkitWebViewEnterFullscreen, for instance. WebKit/gtk/WebCoreSupport/FullscreenVideoController.cpp:298 + Needless empty line. WebKit/gtk/WebCoreSupport/FullscreenVideoController.cpp:590 + One more needless empty line =). Looks good to me, thanks for clarifying the GClosure point, btw, let's get this thing on the wild!
Philippe Normand
Comment 38 2010-08-19 00:47:39 PDT
(In reply to comment #37) > (From update of attachment 64717 [details]) > WebCore/platform/graphics/gstreamer/PlatformVideoWindow.h:46 > + gulong m_videoWindowId; > Is gulong portable enough? Should we use a typedef similar to the PlatformWindowType here? I believe gulong is fine. This is what gst_x_overlay_set_xwindow_id() requires anyway. On Mac OS X it will be a cast of a NSView pointer and on Windows a cast of a HWND variable. I think it'd be fine to have those casts in the platform-specific implementations. I adressed the last requested changes. I also cleaned up the signal and g_source callbacks in the controller to avoid the casts, thanks to the wise help of Martin. Will land the patch, thanks for the long-standing review ;)
Philippe Normand
Comment 39 2010-08-19 01:06:46 PDT
Philippe Normand
Comment 40 2010-08-19 01:07:19 PDT
Comment on attachment 64717 [details] fullscreen video support in WebKit/gtk Clearing r+ flag of committed patch
Note You need to log in before you can comment on or make changes to this bug.