Bug 259735 – [PipeWire] Crash, hang in gst_pipewire_device_provider_stop called by WebCore::maximumNumberOfOutputChannels

• Component: Media
• Status: RESOLVED FIXED

[Michael Catanzaro]

Created attachment 467182 [details]
Full backtrace

I just saw two different web processes crash in the same way at the same time:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fa310764437 in gst_pipewire_device_provider_stop (provider=0x5648a681e9c0)
    at ../src/gst/gstpipewiredeviceprovider.c:668
668	  pw_thread_loop_lock (self->core->loop);
[Current thread is 1 (Thread 0x7fa39775cf40 (LWP 2))]
(gdb) bt
#0  0x00007fa310764437 in gst_pipewire_device_provider_stop
    (provider=0x5648a681e9c0 [GstPipeWireDeviceProvider|pipewiredeviceprovider0])
    at ../src/gst/gstpipewiredeviceprovider.c:668
#1  0x00007fa39bf70d3f in gst_device_provider_stop
    (provider=0x5648a681e9c0 [GstPipeWireDeviceProvider|pipewiredeviceprovider0]) at ../gst/gstdeviceprovider.c:536
#2  0x00007fa39bf86e63 in gst_device_monitor_stop (monitor=0x5648a68174f0 [GstDeviceMonitor|devicemonitor0])
    at ../gst/gstdevicemonitor.c:586
#3  gst_device_monitor_stop (monitor=0x5648a68174f0 [GstDeviceMonitor|devicemonitor0])
    at ../gst/gstdevicemonitor.c:563
#4  0x00007fa3a1b815de in WebCore::maximumNumberOfOutputChannels()::$_0::operator()() const (this=<optimized out>)
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebCore/platform/audio/gstreamer/AudioDestinationGStreamer.cpp:85
#5  std::__invoke_impl<void, WebCore::maximumNumberOfOutputChannels()::$_0>(std::__invoke_other, WebCore::maximumNumberOfOutputChannels()::$_0&&) (__f=<optimized out>)
    at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.1.0/../../../../include/c++/13.1.0/bits/invoke.h:61
#6  std::__invoke<WebCore::maximumNumberOfOutputChannels()::$_0>(WebCore::maximumNumberOfOutputChannels()::$_0&&)
    (__fn=<optimized out>)
    at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.1.0/../../../../include/c++/13.1.0/bits/invoke.h:96
#7  std::call_once<WebCore::maximumNumberOfOutputChannels()::$_0>(std::once_flag&, WebCore::maximumNumberOfOutputChannels()::$_0&&)::{lambda()#1}::operator()() const (this=<optimized out>)
    at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.1.0/../../../../include/c++/13.1.0/mutex:900
#8  std::once_flag::_Prepare_execution::_Prepare_execution<std::call_once<WebCore::maximumNumberOfOutputChannels()::$_0>(std::once_flag&, WebCore::maximumNumberOfOutputChannels()::$_0&&)::{lambda()#1}>(WebCore::maximumNumberOfOutputChannels()::$_0&)::{lambda()#1}::operator()() const (this=<optimized out>)
    at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.1.0/../../../../include/c++/13.1.0/mutex:836
#9  std::once_flag::_Prepare_execution::_Prepare_execution<std::call_once<WebCore::maximumNumberOfOutputChannels()::$_0>(std::once_flag&, WebCore::maximumNumberOfOutputChannels()::$_0&&)::{lambda()#1}>(WebCore::maximumNumberOfOutputChannels()::$_0&)::{lambda()#1}::__invoke() ()
    at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.1.0/../../../../include/c++/13.1.0/mutex:836
#10 0x00007fa39f4a7103 in __pthread_once_slow
    (once_control=0x7fa3a38eec60 <WebCore::maximumNumberOfOutputChannels()::onceFlag>, init_routine=0x7fa39aeefd70 <std::__once_proxy()>) at pthread_once.c:116
#11 0x00007fa3a1b7f354 in __gthread_once(int*, void (*)()) (__once=0x5648a681e9c0, __func=0x0)
    at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.1.0/../../../../include/c++/13.1.0/x86_64-unknown-linux-gnu/bits/gthr-default.h:700
#12 std::call_once<WebCore::maximumNumberOfOutputChannels()::$_0>(std::once_flag&, WebCore::maximumNumberOfOutputChannels()::$_0&&) (__once=..., __f=...)
    at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.1.0/../../../../include/c++/13.1.0/mutex:907
#13 WebCore::maximumNumberOfOutputChannels() ()
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebCore/platform/audio/gstreamer/AudioDestinationGStreamer.cpp:62
#14 WebCore::AudioDestination::maxChannelCount() ()
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebCore/platform/audio/gstreamer/AudioDestinationGStreamer.cpp:110
#15 0x00007fa3a0553ed5 in WebCore::jsAudioDestinationNode_maxChannelCountGetter(JSC::JSGlobalObject&, WebCore::JSAudioDestinationNode&) (thisObject=..., lexicalGlobalObject=<optimized out>)
    at WebCore/DerivedSources/JSAudioDestinationNode.cpp:165
#16 WebCore::IDLAttribute<WebCore::JSAudioDestinationNode>::get<&WebCore::jsAudioDestinationNode_maxChannelCountGetter, (WebCore::CastedThisErrorBehavior)3>(JSC::JSGlobalObject&, long, JSC::PropertyName)
    (thisValue=0, lexicalGlobalObject=<optimized out>, attributeName=...)
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebCore/bindings/js/JSDOMAttribute.h:89
#17 WebCore::jsAudioDestinationNode_maxChannelCount(JSC::JSGlobalObject*, long, JSC::PropertyName)
    (lexicalGlobalObject=<optimized out>, thisValue=0, attributeName=...)
    at WebCore/DerivedSources/JSAudioDestinationNode.cpp:170
--Type <RET> for more, q to quit, c to continue without paging--c
#18 0x00007fa39e7f4bc2 in JSC::PropertySlot::getValue(JSC::JSGlobalObject*, JSC::PropertyName) const
    (this=0x7ffccf1aaef8, globalObject=0x7fa1e3ca1990, propertyName=...)
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/JavaScriptCore/runtime/PropertySlot.h:407
#19 JSC::JSValue::get(JSC::JSGlobalObject*, JSC::PropertyName, JSC::PropertySlot&) const
    (this=0x7ffccf1aaef0, globalObject=0x7fa1e3ca1990, propertyName=..., slot=...)
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/JavaScriptCore/runtime/JSCJSValueInlines.h:1038
#20 JSC::JSValue::get(JSC::JSGlobalObject*, JSC::PropertyName) const
    (this=0x7ffccf1aaef0, globalObject=0x7fa1e3ca1990, propertyName=...)
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/JavaScriptCore/runtime/JSCJSValueInlines.h:1028
#21 JSC::CommonSlowPaths::opEnumeratorGetByVal(JSC::JSGlobalObject*, JSC::JSValue, JSC::JSValue, unsigned int, JSC::JSPropertyNameEnumerator::Flag, JSC::JSPropertyNameEnumerator*, JSC::ArrayProfile*, unsigned char*)
    (globalObject=0x7fa1e3ca1990, baseValue=..., propertyNameValue=..., index=<optimized out>, mode=<optimized out>, enumerator=<optimized out>, arrayProfile=0x7fa0dee741b0, enumeratorMetadata=0x7fa0dee741bc "\004")
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/JavaScriptCore/runtime/CommonSlowPaths.h:91
#22 0x00007fa39e7e41e0 in slow_path_enumerator_get_by_val(JSC::CallFrame*, JSC::JSInstruction const*)
    (callFrame=0x7ffccf1ab110, pc=0x7fa1de434116)
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/JavaScriptCore/runtime/CommonSlowPaths.cpp:957
#23 0x00007fa39db949e4 in llint_op_enumerator_get_by_val_wide32 ()
    at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-6.0.so.1
#24 0x0000000000000000 in  ()

Full backtrace attached.

[Michael Catanzaro]

At the same time that these two web processes crashed, a third web process hung in the same code:

Thread 2 (Thread 0x7f699767bf40 (LWP 2)):
#0  0x00007f699f49e6d9 in __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=179, futex_word=0x7f4ffffff990) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f4ffffff990, expected=179, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at futex-internal.c:87
#2  0x00007f699f49e75f in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f4ffffff990, expected=<optimized out>, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at futex-internal.c:139
#3  0x00007f699f4a3b43 in __pthread_clockjoin_ex (threadid=139981574108864, thread_return=0x0, clockid=0, abstime=0x0, block=<optimized out>) at pthread_join_common.c:102
#4  0x00007f68ec0d63a0 in spa_thread_utils_join (retval=0x0, thread=<optimized out>, o=<optimized out>) at ../spa/include/spa/support/thread.h:73
#5  pw_data_loop_stop (loop=0x5559fdb07a80) at ../src/pipewire/data-loop.c:226
#6  0x00007f68ec0d69d6 in pw_context_destroy (context=0x5559fdb06180) at ../src/pipewire/context.c:417
#7  0x00007f68ed631849 in make_core (fd=<optimized out>) at ../src/gst/gstpipewirecore.c:93
#8  gst_pipewire_core_get (fd=<optimized out>) at ../src/gst/gstpipewirecore.c:128
#9  0x00007f68ed6348eb in gst_pipewire_device_provider_start (provider=0x5559fdaf8330 [GstPipeWireDeviceProvider|pipewiredeviceprovider0]) at ../src/gst/gstpipewiredeviceprovider.c:623
#10 0x00007f699be9bb7c in gst_device_provider_start (provider=0x5559fdaf8330 [GstPipeWireDeviceProvider|pipewiredeviceprovider0]) at ../gst/gstdeviceprovider.c:478
#11 0x00007f699beb0416 in gst_device_monitor_start (monitor=0x5559fdaf0ef0 [GstDeviceMonitor|devicemonitor0]) at ../gst/gstdevicemonitor.c:533
#12 0x00007f69a1b814c8 in WebCore::maximumNumberOfOutputChannels()::$_0::operator()() const (this=<optimized out>) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebCore/platform/audio/gstreamer/AudioDestinationGStreamer.cpp:66
#13 std::__invoke_impl<void, WebCore::maximumNumberOfOutputChannels()::$_0>(std::__invoke_other, WebCore::maximumNumberOfOutputChannels()::$_0&&) (__f=<optimized out>) at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.1.0/../../../../include/c++/13.1.0/bits/invoke.h:61
#14 std::__invoke<WebCore::maximumNumberOfOutputChannels()::$_0>(WebCore::maximumNumberOfOutputChannels()::$_0&&) (__fn=<optimized out>) at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.1.0/../../../../include/c++/13.1.0/bits/invoke.h:96
#15 std::call_once<WebCore::maximumNumberOfOutputChannels()::$_0>(std::once_flag&, WebCore::maximumNumberOfOutputChannels()::$_0&&)::{lambda()#1}::operator()() const (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.1.0/../../../../include/c++/13.1.0/mutex:900
#16 std::once_flag::_Prepare_execution::_Prepare_execution<std::call_once<WebCore::maximumNumberOfOutputChannels()::$_0>(std::once_flag&, WebCore::maximumNumberOfOutputChannels()::$_0&&)::{lambda()#1}>(WebCore::maximumNumberOfOutputChannels()::$_0&)::{lambda()#1}::operator()() const (this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.1.0/../../../../include/c++/13.1.0/mutex:836
#17 std::once_flag::_Prepare_execution::_Prepare_execution<std::call_once<WebCore::maximumNumberOfOutputChannels()::$_0>(std::once_flag&, WebCore::maximumNumberOfOutputChannels()::$_0&&)::{lambda()#1}>(WebCore::maximumNumberOfOutputChannels()::$_0&)::{lambda()#1}::__invoke() () at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.1.0/../../../../include/c++/13.1.0/mutex:836
#18 0x00007f699f4a7103 in __pthread_once_slow (once_control=0x7f69a38eec60 <WebCore::maximumNumberOfOutputChannels()::onceFlag>, init_routine=0x7f699acefd70 <std::__once_proxy()>) at pthread_once.c:116
#19 0x00007f69a1b7f354 in __gthread_once(int*, void (*)()) (__once=0x7f4ffffff990, __func=0x109) at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.1.0/../../../../include/c++/13.1.0/x86_64-unknown-linux-gnu/bits/gthr-default.h:700
#20 std::call_once<WebCore::maximumNumberOfOutputChannels()::$_0>(std::once_flag&, WebCore::maximumNumberOfOutputChannels()::$_0&&) (__once=..., __f=...) at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.1.0/../../../../include/c++/13.1.0/mutex:907
#21 WebCore::maximumNumberOfOutputChannels() () at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebCore/platform/audio/gstreamer/AudioDestinationGStreamer.cpp:62
#22 WebCore::AudioDestination::maxChannelCount() () at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebCore/platform/audio/gstreamer/AudioDestinationGStreamer.cpp:110
#23 0x00007f69a0553ed5 in WebCore::jsAudioDestinationNode_maxChannelCountGetter(JSC::JSGlobalObject&, WebCore::JSAudioDestinationNode&) (thisObject=..., lexicalGlobalObject=<optimized out>) at WebCore/DerivedSources/JSAudioDestinationNode.cpp:165
#24 WebCore::IDLAttribute<WebCore::JSAudioDestinationNode>::get<&WebCore::jsAudioDestinationNode_maxChannelCountGetter, (WebCore::CastedThisErrorBehavior)3>(JSC::JSGlobalObject&, long, JSC::PropertyName) (thisValue=265, lexicalGlobalObject=<optimized out>, attributeName=...) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebCore/bindings/js/JSDOMAttribute.h:89
#25 WebCore::jsAudioDestinationNode_maxChannelCount(JSC::JSGlobalObject*, long, JSC::PropertyName) (lexicalGlobalObject=<optimized out>, thisValue=265, attributeName=...) at WebCore/DerivedSources/JSAudioDestinationNode.cpp:170
#26 0x00007f699e5f4bc2 in JSC::PropertySlot::getValue(JSC::JSGlobalObject*, JSC::PropertyName) const (this=0x7fffe6b6bec8, globalObject=0x7f6936243068, propertyName=...) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/JavaScriptCore/runtime/PropertySlot.h:407
#27 JSC::JSValue::get(JSC::JSGlobalObject*, JSC::PropertyName, JSC::PropertySlot&) const (this=0x7fffe6b6bec0, globalObject=0x7f6936243068, propertyName=..., slot=...) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/JavaScriptCore/runtime/JSCJSValueInlines.h:1038
#28 JSC::JSValue::get(JSC::JSGlobalObject*, JSC::PropertyName) const (this=0x7fffe6b6bec0, globalObject=0x7f6936243068, propertyName=...) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/JavaScriptCore/runtime/JSCJSValueInlines.h:1028
#29 JSC::CommonSlowPaths::opEnumeratorGetByVal(JSC::JSGlobalObject*, JSC::JSValue, JSC::JSValue, unsigned int, JSC::JSPropertyNameEnumerator::Flag, JSC::JSPropertyNameEnumerator*, JSC::ArrayProfile*, unsigned char*) (globalObject=0x7f6936243068, baseValue=..., propertyNameValue=..., index=<optimized out>, mode=<optimized out>, enumerator=<optimized out>, arrayProfile=0x7f68c8da1840, enumeratorMetadata=0x7f68c8da184c "\004") at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/JavaScriptCore/runtime/CommonSlowPaths.h:91
#30 0x00007f699e5e41e0 in slow_path_enumerator_get_by_val(JSC::CallFrame*, JSC::JSInstruction const*) (callFrame=0x7fffe6b6c0e0, pc=0x7f6822b0f136) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/JavaScriptCore/runtime/CommonSlowPaths.cpp:957
#31 0x00007f699d9949e4 in llint_op_enumerator_get_by_val_wide32 () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-6.0.so.1
#32 0x0000000000000000 in  ()

This third web process was then crashed by its watchdog thread:

Thread 1 (Thread 0x7f6913fff6c0 (LWP 182)):
#0  g_log_structured_array (log_level=<optimized out>, fields=0x7f6913ffe6f0, n_fields=3) at ../glib/gmessages.c:555
#1  0x00007f699b2c16fc in g_log_default_handler (log_domain=log_domain@entry=0x0, log_level=log_level@entry=6, message=message@entry=0x7f69a27f07f0 "WebProcess didn't exit as expected after the UI process connection was closed", unused_data=unused_data@entry=0x0) at ../glib/gmessages.c:3284
#2  0x00007f6984102242 in trap_handler (log_domain=log_domain@entry=0x0, log_level=log_level@entry=6, message=message@entry=0x7f69a27f07f0 "WebProcess didn't exit as expected after the UI process connection was closed", user_data=user_data@entry=0x0) at ../lib/ephy-debug.c:104
#3  0x00007f699b2c19a6 in g_logv (log_domain=0x0, log_level=G_LOG_LEVEL_ERROR, format=<optimized out>, args=args@entry=0x7f6913ffe870) at ../glib/gmessages.c:1391
#4  0x00007f699b2c1c93 in g_log (log_domain=<optimized out>, log_level=<optimized out>, format=<optimized out>) at ../glib/gmessages.c:1460
#5  0x00007f69a0347a9d in WebKit::crashAfter10Seconds(IPC::Connection*)::$_0::operator()() const (this=<optimized out>) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebKit/WebProcess/WebProcess.cpp:282

I will attach an all-threads backtrace for this third web process.

[Michael Catanzaro]

Created attachment 467183 [details]
All threads backtrace for third web process

[Philippe Normand]

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fa310764437 in gst_pipewire_device_provider_stop (provider=0x5648a681e9c0)
    at ../src/gst/gstpipewiredeviceprovider.c:668
668	  pw_thread_loop_lock (self->core->loop);


The bt doesn't say... is self->core null or what?
Seems to me this is a Pipewire bug, introduced by https://gitlab.freedesktop.org/pipewire/pipewire/-/commit/b0a7e4a267f36bf7d727aba5f31e33f0111e331e

[Michael Catanzaro]

(In reply to Philippe Normand from comment #3)
> The bt doesn't say... is self->core null or what?
> Seems to me this is a Pipewire bug, introduced by
> https://gitlab.freedesktop.org/pipewire/pipewire/-/commit/
> b0a7e4a267f36bf7d727aba5f31e33f0111e331e

Yes:

(gdb) print self->core
$2 = (GstPipeWireCore *) 0x0

At least it's not a dangling pointer. Reminds me of unrelated bug https://gitlab.gnome.org/GNOME/glib-networking/-/issues/214#note_1773704 which we haven't fixed yet....

[Philippe Normand]

https://gitlab.freedesktop.org/pipewire/pipewire/-/merge_requests/1673

[Philippe Normand]

(In reply to Philippe Normand from comment #5)
> https://gitlab.freedesktop.org/pipewire/pipewire/-/merge_requests/1673

Well, that one landed. I don't think we can do much more here. Let's close?