Bug 169811 – [GStreamer] Add support for canvas captureStream, tests failing since they were added in r213598

• Component: WebKitGTK
• Status: RESOLVED FIXED

[Carlos Garcia Campos]

fast/mediacapturefromelement/CanvasCaptureMediaStream-capture-out-of-DOM-element.html [ Crash ]
  fast/mediacapturefromelement/CanvasCaptureMediaStream-webgl-events.html [ Crash ]
  fast/mediastream/captureStream/canvas2d-heavy-drawing.html [ Crash ]
  fast/mediastream/captureStream/canvas2d.html [ Crash ]

[youenn fablet]

Would be cool if GTK would support that.
The port work might be limited to updating Source/WebCore/platform/graphics/ImageBuffer.cpp as done in r213598 for the Mac platform.

[Alejandro G. Castro]

*** Bug 172954 has been marked as a duplicate of this bug. ***

[Michael Catanzaro]

So unimplemented web-exposed WebRTC features cause web process crashes instead of JS-level failures?

[Alejandro G. Castro]

(In reply to Michael Catanzaro from comment #3)
> So unimplemented web-exposed WebRTC features cause web process crashes
> instead of JS-level failures?

In this case it is what is happening, you can read my comment in the other bug to understand it:

https://bugs.webkit.org/show_bug.cgi?id=172954#c4

Anyway, this is an API from a different spec that defines mediaCapture of DOM elements such as canvas that can be consumed by WebRTC or WebAudio, check the spec here:

https://w3c.github.io/mediacapture-fromelement/#

If we do not have time to check it before the next release we need to guard it with something different than MEDIA_STREAM.

[Fujii Hironori]

(In reply to Carlos Garcia Campos from comment #0)
>   fast/mediacapturefromelement/CanvasCaptureMediaStream-capture-out-of-DOM-element.html [ Crash ]
>   fast/mediacapturefromelement/CanvasCaptureMediaStream-webgl-events.html [ Crash ]

This doesn't crash now.

>   fast/mediastream/captureStream/canvas2d-heavy-drawing.html [ Crash ]
>   fast/mediastream/captureStream/canvas2d.html [ Crash ]

Still crashes.

> STDERR: warning: core file may not match specified executable file.
> STDERR: 58	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> STDERR: g_object_get: assertion 'G_IS_OBJECT (object)' failed

Callstack:

> Thread 1 (Thread 0x7ffff7f25040 (LWP 72572)):
> #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58
> #1  0x00007fffeb8ee37a in __GI_abort () at abort.c:89
> #2  0x00007fffeb930090 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7fffeba4396c "*** %s ***: %s terminated\n") at ../sysdeps/posix/libc_fatal.c:175
> #3  0x00007fffeb9d1384 in __GI___fortify_fail (msg=<optimized out>, msg@entry=0x7fffeba4394e "stack smashing detected") at fortify_fail.c:37
> #4  0x00007fffeb9d1330 in __stack_chk_fail () at stack_chk_fail.c:28
> #5  0x0000555556e71612 in WebCore::MediaPlayerPrivateGStreamerOwr::trackEnded(WebCore::MediaStreamTrackPrivate&) ()
> #6  0x000055555694f500 in WebCore::MediaStreamTrackPrivate::endTrack() ()
> #7  0x0000555555f3a42b in WebCore::MediaStreamTrack::stopTrack(WebCore::MediaStreamTrack::StopMode) ()
> #8  0x0000555555f3a45e in WebCore::MediaStreamTrack::stop() ()
> #9  0x00005555562fa33b in WebCore::ScriptExecutionContext::stopActiveDOMObjects() ()
> #10 0x000055555627ef80 in WebCore::Document::stopActiveDOMObjects() ()
> #11 0x000055555627f087 in WebCore::Document::prepareForDestruction() ()
> #12 0x000055555676c1d2 in WebCore::Frame::setView(WTF::RefPtr<WebCore::FrameView>&&) ()
> #13 0x000055555676efbc in WebCore::Frame::createView(WebCore::IntSize const&, WebCore::Color const&, bool, WebCore::IntSize const&, WebCore::IntRect const&, bool, WebCore::ScrollbarMode, bool, WebCore::ScrollbarMode, bool) ()
> #14 0x00007ffff578c62b in WebKit::WebFrameLoaderClient::transitionToCommittedForNewPage() () from /home/fujii/work/webkit/ga/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37
> #15 0x00005555566357ca in WebCore::FrameLoader::transitionToCommitted(WebCore::CachedPage*) ()
> #16 0x0000555556635913 in WebCore::FrameLoader::commitProvisionalLoad() ()
> #17 0x000055555660ed17 in WebCore::DocumentLoader::finishedLoading() ()
> #18 0x000055555660f309 in WebCore::DocumentLoader::maybeLoadEmpty() ()
> #19 0x0000555556612362 in WebCore::DocumentLoader::startLoadingMainResource() ()
> #20 0x0000555556636348 in WebCore::FrameLoader::continueLoadAfterNavigationPolicy(WebCore::ResourceRequest const&, WebCore::FormState*, bool, WebCore::AllowNavigationToInvalidURL) ()
> #21 0x000055555666306b in WebCore::PolicyCallback::call(bool) ()
> #22 0x000055555666c020 in WebCore::PolicyChecker::continueAfterNavigationPolicy(WebCore::PolicyAction) ()
> #23 0x00007ffff5794a81 in WebKit::WebFrameLoaderClient::dispatchDecidePolicyForNavigationAction(WebCore::NavigationAction const&, WebCore::ResourceRequest const&, WebCore::FormState*, std::function<void (WebCore::PolicyAction)>) () from /home/fujii/work/webkit/ga/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37
> #24 0x000055555666b00d in WebCore::PolicyChecker::checkNavigationPolicy(WebCore::ResourceRequest const&, bool, WebCore::DocumentLoader*, WebCore::FormState*, WTF::Function<void (WebCore::ResourceRequest const&, WebCore::FormState*, bool)>) ()
> #25 0x0000555556636752 in WebCore::FrameLoader::loadWithDocumentLoader(WebCore::DocumentLoader*, WebCore::FrameLoadType, WebCore::FormState*, WebCore::AllowNavigationToInvalidURL) ()
> #26 0x0000555556639392 in WebCore::FrameLoader::load(WebCore::FrameLoadRequest const&) ()
> #27 0x00007ffff57c21f8 in WebKit::WebPage::loadRequest(WebKit::LoadParameters const&) () from /home/fujii/work/webkit/ga/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37
> #28 0x00007ffff59aad97 in void IPC::handleMessage<Messages::WebPage::LoadAlternateHTMLString, WebKit::WebPage, void (WebKit::WebPage::*)(WebKit::LoadParameters const&)>(IPC::Decoder&, WebKit::WebPage*, void (WebKit::WebPage::*)(WebKit::LoadParameters const&)) () from /home/fujii/work/webkit/ga/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37
> #29 0x00007ffff59a88b9 in WebKit::WebPage::didReceiveWebPageMessage(IPC::Connection&, IPC::Decoder&) () from /home/fujii/work/webkit/ga/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37
> #30 0x00007ffff558f2e9 in IPC::MessageReceiverMap::dispatchMessage(IPC::Connection&, IPC::Decoder&) () from /home/fujii/work/webkit/ga/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37
> #31 0x00007ffff56ffd36 in WebKit::WebProcess::didReceiveMessage(IPC::Connection&, IPC::Decoder&) () from /home/fujii/work/webkit/ga/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37
> #32 0x00007ffff558ab3b in IPC::Connection::dispatchMessage(std::unique_ptr<IPC::Decoder, std::default_delete<IPC::Decoder> >) () from /home/fujii/work/webkit/ga/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37
> #33 0x00007ffff558bb28 in IPC::Connection::dispatchOneMessage() () from /home/fujii/work/webkit/ga/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37
> #34 0x00007ffff3bc017d in WTF::RunLoop::performWork() () from /home/fujii/work/webkit/ga/WebKitBuild/Release/lib/libjavascriptcoregtk-4.0.so.18
> #35 0x00007ffff3bf7249 in WTF::RunLoop::RunLoop()::{lambda(void*)#1}::_FUN(void*) () from /home/fujii/work/webkit/ga/WebKitBuild/Release/lib/libjavascriptcoregtk-4.0.so.18
> #36 0x00007fffeeea96ea in g_main_dispatch () at /home/fujii/work/webkit/ga/WebKitBuild/DependenciesGTK/Source/glib-2.52.1/glib/gmain.c:3212
> #37 g_main_context_dispatch () at /home/fujii/work/webkit/ga/WebKitBuild/DependenciesGTK/Source/glib-2.52.1/glib/gmain.c:3865
> #38 0x00007fffeeea9aa0 in g_main_context_iterate () at /home/fujii/work/webkit/ga/WebKitBuild/DependenciesGTK/Source/glib-2.52.1/glib/gmain.c:3938
> #39 0x00007fffeeea9dc2 in g_main_loop_run () at /home/fujii/work/webkit/ga/WebKitBuild/DependenciesGTK/Source/glib-2.52.1/glib/gmain.c:4134
> #40 0x00007ffff3bf7c70 in WTF::RunLoop::run() () from /home/fujii/work/webkit/ga/WebKitBuild/Release/lib/libjavascriptcoregtk-4.0.so.18
> #41 0x00007ffff5952a2a in int WebKit::ChildProcessMain<WebKit::WebProcess, WebKit::WebProcessMain>(int, char**) () from /home/fujii/work/webkit/ga/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37
> #42 0x00007fffeb8d73f1 in __libc_start_main (main=0x555555e2c5a0 <main>, argc=2, argv=0x7fffffffdca8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdc98) at ../csu/libc-start.c:291
> #43 0x0000555555e2caea in _start ()

[Charlie Turner]

The stderr above seems like an environmental issues clopez?

fast/mediastream/captureStream/canvas2d-heavy-drawing.html [ Crash ]
fast/mediastream/captureStream/canvas2d.html [ Crash ]

These are now flaky tests. heavy-drawing for example gave me this on stderr,

invalid uninstantiatable type '(null)' in cast to 'OwrMediaSource'
owr_media_renderer_set_source: assertion '!source || OWR_IS_MEDIA_SOURCE(source)' failed

Shouldn't everything under fast/mediastream be skipped, since we don't support it IIUC?

[Alejandro G. Castro]

(In reply to Charlie Turner from comment #6)
> The stderr above seems like an environmental issues clopez?
> 
> fast/mediastream/captureStream/canvas2d-heavy-drawing.html [ Crash ]
> fast/mediastream/captureStream/canvas2d.html [ Crash ]
> 
> These are now flaky tests. heavy-drawing for example gave me this on stderr,
> 
> invalid uninstantiatable type '(null)' in cast to 'OwrMediaSource'
> owr_media_renderer_set_source: assertion '!source ||
> OWR_IS_MEDIA_SOURCE(source)' failed
> 
> Shouldn't everything under fast/mediastream be skipped, since we don't
> support it IIUC?

We support mediastream but we do not support captureStream for the moment.

[Charlie Turner]

Yes, sorry I missed the "captureStream" of the path there.

[Philippe Normand]

I suppose this will work with the libwebrtc backend?

[Thibault Saunier]

(In reply to Philippe Normand from comment #9)
> I suppose this will work with the libwebrtc backend?

Not really, we are still missing something for the CaptureStream API.

[Tyler Larson]

Can we get an update on the status of this feature? And while we wait is there any other supported way to capture the canvas that currently works in iOS?

[Alessandro]

Any update on this? Is there a workaround to capture the canvas stream that works on iOS?

[Philippe Normand]

(In reply to Alessandro from comment #12)
> Any update on this? Is there a workaround to capture the canvas stream that
> works on iOS?

This specific bug is for the GTK port of WebKit, not iOS. Can you open a new bug please?

[Philippe Normand]

*** Bug 186845 has been marked as a duplicate of this bug. ***

[Philippe Normand]

Created attachment 404677 [details]
Patch

[Darin Adler]

Comment on attachment 404677 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=404677&action=review

> Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.cpp:114
> +    GstCaps* caps = gst_sample_get_caps(m_sample.get());

auto?

> Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.h:40
> +    static RefPtr<MediaSampleGStreamer> createImageSample(Vector<uint8_t>&&, unsigned long width, unsigned long height);

This should return Ref, not RefPtr.

This should take unsigned, not unsigned long.

> Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.h:58
> +    RefPtr<JSC::Uint8ClampedArray> getRGBAImageData() const override;

final instead of override?

[Philippe Normand]

Comment on attachment 404677 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=404677&action=review

>> Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.h:40
>> +    static RefPtr<MediaSampleGStreamer> createImageSample(Vector<uint8_t>&&, unsigned long width, unsigned long height);
> 
> This should return Ref, not RefPtr.
> 
> This should take unsigned, not unsigned long.

The AVF MediaSample class would then need to be updated as well, for consistency sake, at least.

[Philippe Normand]

*** Bug 214541 has been marked as a duplicate of this bug. ***

[Philippe Normand]

Created attachment 404705 [details]
Patch for landing

[Philippe Normand]

Created attachment 404706 [details]
Patch for landing

[Philippe Normand]

Comment on attachment 404706 [details]
Patch for landing

View in context: https://bugs.webkit.org/attachment.cgi?id=404706&action=review

> Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm:49
> -RefPtr<MediaSampleAVFObjC> MediaSampleAVFObjC::createImageSample(Vector<uint8_t>&& array, unsigned long width, unsigned long height)
> +Ref<MediaSampleAVFObjC> MediaSampleAVFObjC::createImageSample(Vector<uint8_t>&& array, unsigned width, unsigned height)

Ah, returning a Ref here isn't possible. :(

[Philippe Normand]

Created attachment 404708 [details]
Patch for landing

[EWS]

Committed r264596: <https://trac.webkit.org/changeset/264596>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 404708 [details].

[Yusuke Suzuki]

Re-opened since this is blocked by bug 214551

[Philippe Normand]

Thread 1 (Thread 0x7f731b5512c0 (LWP 64688)):
#0  0x00007f731d797ee5 in _g_log_abort (breakpoint=1) at ../glib/gmessages.c:554
#1  0x00007f731d7991c9 in g_logv (log_domain=0x7f731e11a5e0 <g_log_domain_gstreamer> "GStreamer", log_level=G_LOG_LEVEL_CRITICAL, format=<optimized out>, args=args@entry=0x7fffe84e84a0) at ../glib/gmessages.c:1373
#2  0x00007f731d799393 in g_log (log_domain=<optimized out>, log_level=log_level@entry=G_LOG_LEVEL_CRITICAL, format=format@entry=0x7f731d7ea77f "%s: assertion '%s' failed") at ../glib/gmessages.c:1415
#3  0x00007f731d799b8d in g_return_if_fail_warning (log_domain=<optimized out>, pretty_function=pretty_function@entry=0x7f731e11b060 <__func__.22596> "gst_memory_new_wrapped", expression=expression@entry=0x7f731e11d416 "data != NULL") at ../glib/gmessages.c:2771
#4  0x00007f731e0713ad in gst_memory_new_wrapped (flags=flags@entry=GST_MEMORY_FLAG_READONLY, data=data@entry=0x0, maxsize=maxsize@entry=180000, offset=offset@entry=0, size=size@entry=180000, user_data=user_data@entry=0x7f72ab035000, notify=0x7f732668b6d0 <WebCore::MediaSampleGStreamer::createImageSample(WTF::Vector<unsigned char, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, unsigned int, unsigned int)::{lambda(void*)#1}::_FUN(void*)>) at ../gst/gstallocator.c:639
#5  0x00007f731e07c2bd in gst_buffer_new_wrapped_full (flags=GST_MEMORY_FLAG_READONLY, data=0x0, maxsize=180000, offset=0, size=180000, user_data=0x7f72ab035000, notify=0x7f732668b6d0 <WebCore::MediaSampleGStreamer::createImageSample(WTF::Vector<unsigned char, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, unsigned int, unsigned int)::{lambda(void*)#1}::_FUN(void*)>) at ../gst/gstbuffer.c:984
#6  0x00007f732668c86f in WebCore::MediaSampleGStreamer::createImageSample(WTF::Vector<unsigned char, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, unsigned int, unsigned int) () at /app/webkit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37
#7  0x00007f73258f2fb3 in WebCore::HTMLCanvasElement::toMediaSample() () at /app/webkit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37
#8  0x00007f732529fd58 in WebCore::CanvasCaptureMediaStreamTrack::Source::captureCanvas() () at /app/webkit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37
#9  0x00007f7321c276e8 in WTF::RunLoop::performWork() () at /app/webkit/WebKitBuild/Release/lib/libjavascriptcoregtk-4.0.so.18
#10 0x00007f7321c8cc89 in WTF::RunLoop::RunLoop()::{lambda(void*)#1}::_FUN(void*) () at /app/webkit/WebKitBuild/Release/lib/libjavascriptcoregtk-4.0.so.18
#11 0x00007f731d791c3e in g_main_dispatch (context=0x56436d368f50) at ../glib/gmain.c:3309
#12 0x00007f731d791c3e in g_main_context_dispatch (context=context@entry=0x56436d368f50) at ../glib/gmain.c:3974
#13 0x00007f731d791ff0 in g_main_context_iterate (context=0x56436d368f50, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4047
#14 0x00007f731d7922e3 in g_main_loop_run (loop=0x56436d396d30) at ../glib/gmain.c:4241
#15 0x00007f7321c8d770 in WTF::RunLoop::run() () at /app/webkit/WebKitBuild/Release/lib/libjavascriptcoregtk-4.0.so.18
#16 0x00007f7324935b2f in WebKit::WebProcessMain(int, char**) () at /app/webkit/WebKitBuild/Release/lib/libwebkit2gtk-4.0.so.37
#17 0x00007f731c3c5183 in __libc_start_main (main=0x56436c92bc00 <main>, argc=4, argv=0x7fffe84e89f8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffe84e89e8) at ../csu/libc-start.c:308
#18 0x000056436c92bc8e in _start () at ../sysdeps/x86_64/start.S:120

STDERR: 
STDERR: warning: core file may not match specified executable file.
STDERR: 
STDERR: (WebKitWebProcess:64688): GStreamer-CRITICAL **: 06:43:32.526: gst_memory_new_wrapped: assertion 'data != NULL' failed

[Philippe Normand]

I did run the tests ;) What's interesting is that in clang builds, they pass, but in gcc builds, they crash... Stay tuned for more fun facts about toolchains.

[Philippe Normand]

Created attachment 404719 [details]
Patch

[Darin Adler]

What was the gcc/clang difference?

[Philippe Normand]

Comment on attachment 404708 [details]
Patch for landing

View in context: https://bugs.webkit.org/attachment.cgi?id=404708&action=review

> Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.cpp:103
> +    auto buffer = adoptGRef(gst_buffer_new_wrapped_full(GST_MEMORY_FLAG_READONLY, bgraData.data(), size, 0, size, bgraData.releaseBuffer().leakPtr(), [](gpointer data) {

The issue with this code (I think) is that we were passing the bgra.data() and immediately after, releasing it using the leakPtr() call.

[Darin Adler]

Comment on attachment 404708 [details]
Patch for landing

View in context: https://bugs.webkit.org/attachment.cgi?id=404708&action=review

>> Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.cpp:103
>> +    auto buffer = adoptGRef(gst_buffer_new_wrapped_full(GST_MEMORY_FLAG_READONLY, bgraData.data(), size, 0, size, bgraData.releaseBuffer().leakPtr(), [](gpointer data) {
> 
> The issue with this code (I think) is that we were passing the bgra.data() and immediately after, releasing it using the leakPtr() call.

Yes, that makes sense. I see now that this is indeed incorrect. We can’t call both data() and releaseBuffer() for two different arguments and count on the data() call happening first.

[Darin Adler]

Comment on attachment 404719 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=404719&action=review

> Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm:52
>      auto status = CVPixelBufferCreateWithBytes(kCFAllocatorDefault, width, height, kCVPixelFormatType_32BGRA, array.data(), width * 4, releaseUint8Vector, array.releaseBuffer().leakPtr(), NULL, &pixelBuffer);

This code has the same bug as the mistake in GStreamer, calling data() and releaseBuffer() and depending on order of evaluation, and needs the same kind of fix.

[EWS]

Committed r264607: <https://trac.webkit.org/changeset/264607>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 404719 [details].