Bug 303919 – [GStreamer] Flaky web process deadlock when running imported/w3c/web-platform-tests/webrtc-stats/hardware-capability-stats.https.html

• Component: WebKitGTK
• Status: RESOLVED FIXED

[Claudio Saavedra]

It started around these two commits, I can't tell which one is to blame:

- https://github.com/WebKit/WebKit/commit/6ed6b6a4c121b050e54bedea8db4a811aa6fa323
- https://github.com/WebKit/WebKit/commit/49b71e2d05c099f27f5b75a61bac0a1ea39daade

See https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fwebrtc-stats%2Fhardware-capability-stats.https.html&platform=GTK&platform=WPE

Full trace: https://build.webkit.org/results/GTK-Linux-64-bit-Release-Tests/304194@main%20(22435)/imported/w3c/web-platform-tests/webrtc-stats/hardware-capability-stats.https-crash-log.txt

Thread 1 (Thread 0x7f01817fa6c0 (LWP 519720)):
#0  0x00007f044a2306b1 in g_logv () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#1  0x00007f044a230963 in g_log () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f045040442d in WTF::Detail::CallableWrapper<WebKit::crashAfter10Seconds(IPC::Connection*)::{lambda()#1}, void>::call() () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libwebkitgtk-6.0.so.4
#3  0x00007f044c86fe13 in WTF::Detail::CallableWrapper<WTF::RunLoop::dispatchAfter(WTF::Seconds, WTF::Function<void ()>&&)::{lambda()#1}, void>::call() () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libjavascriptcoregtk-6.0.so.1
#4  0x00007f044c95c3a1 in WTF::RunLoop::TimerBase::TimerBase(WTF::Ref<WTF::RunLoop, WTF::RawPtrTraits<WTF::RunLoop>, WTF::DefaultRefDerefTraits<WTF::RunLoop> >&&, WTF::ASCIILiteral)::{lambda(void*)#1}::_FUN(void*) () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libjavascriptcoregtk-6.0.so.1
#5  0x00007f044c95fc5d in WTF::RunLoop::{lambda(_GSource*, int (*)(void*), void*)#1}::_FUN(_GSource*, int (*)(void*), void*) () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libjavascriptcoregtk-6.0.so.1
#6  0x00007f044a22b49e in ??? () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#7  0x00007f044a22b710 in g_main_context_dispatch () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#8  0x00007f044c95e107 in WTF::RunLoop::runGLibMainLoopIteration(WTF::RunLoop::MayBlock) () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libjavascriptcoregtk-6.0.so.1
#9  0x00007f044c95e512 in WTF::RunLoop::run() () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libjavascriptcoregtk-6.0.so.1
#10 0x00007f044c8e402d in WTF::Thread::entryPoint(WTF::Thread::NewThreadContext*) () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libjavascriptcoregtk-6.0.so.1
#11 0x00007f044c9633ad in WTF::wtfThreadEntryPoint(void*) () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libjavascriptcoregtk-6.0.so.1
#12 0x00007f044dc8aaa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#13 0x00007f044dd17c6c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

[Claudio Saavedra]

Debug crash: 

Thread 1 (Thread 0x7f43f8ff96c0 (LWP 528843)):
#0  WTFCrash () at ../../../Source/WTF/wtf/Assertions.cpp:380
#1  0x00007f474d4537ec in WTF::RefCountDebugger::applyRefDerefThreadingCheck (this=0x7f4726e60550, refCount=2, isThreadSafe=WTF::RefCountDebugger::RefCountIsThreadSafe::No) at WTF/Headers/wtf/RefCountDebugger.h:114
#2  0x00007f474d453849 in WTF::RefCountDebugger::willDeref (this=0x7f4726e60550, refCount=2, isThreadSafe=WTF::RefCountDebugger::RefCountIsThreadSafe::No) at WTF/Headers/wtf/RefCountDebugger.h:146
#3  0x00007f474d455a36 in WTF::RefCountedBase::derefBase (this=0x7f4726e60550) at WTF/Headers/wtf/RefCounted.h:53
#4  0x00007f474fdb8352 in WTF::RefCounted<WebCore::RiceBackendClient>::deref (this=0x7f4726e60550) at WTF/Headers/wtf/RefCounted.h:75
#5  0x00007f474fdb27cb in WTF::DefaultRefDerefTraits<WebCore::RiceBackendClient>::derefIfNotNull (ptr=0x7f4726e60550) at WTF/Headers/wtf/Ref.h:64
#6  0x00007f474fdaea2c in WTF::RefPtr<WebCore::RiceBackendClient, WTF::RawPtrTraits<WebCore::RiceBackendClient>, WTF::DefaultRefDerefTraits<WebCore::RiceBackendClient> >::~RefPtr (this=0x55f9cb6883d0, __in_chrg=<optimized out>) at WTF/Headers/wtf/RefPtr.h:66
#7  0x00007f47534f4d30 in _WebKitGstIceAgentPrivate::~_WebKitGstIceAgentPrivate (this=0x55f9cb6883d0, __in_chrg=<optimized out>) at ../../../Source/WebCore/Modules/mediastream/gstreamer/GStreamerIceAgent.cpp:80
#8  0x00007f47534f4d79 in webkit_gst_webrtc_ice_backend_finalize (object=0x55f9cb6884a0) at ../../../Source/WebCore/Modules/mediastream/gstreamer/GStreamerIceAgent.cpp:121
#9  0x00007f473df8e61d in g_object_unref () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0
#10 0x00007f46d4f7b587 in gst_webrtc_bin_dispose (object=0x55f9cb427690) at ../../../../../jhbuild/checkout/gstreamer/subprojects/gst-plugins-bad/ext/webrtc/gstwebrtcbin.c:8783
#11 0x00007f473df8e3fe in g_object_unref () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0
#12 0x00007f473766759f in gst_bin_remove_func (bin=0x55f9cb67ee90, element=<optimized out>) at ../../../../../jhbuild/checkout/gstreamer/subprojects/gstreamer/gst/gstbin.c:1817
#13 0x00007f47376666af in gst_bin_remove (bin=bin@entry=0x55f9cb67ee90, element=0x55f9cb427690) at ../../../../../jhbuild/checkout/gstreamer/subprojects/gstreamer/gst/gstbin.c:1868
#14 0x00007f47376669b3 in gst_bin_dispose (object=0x55f9cb67ee90) at ../../../../../jhbuild/checkout/gstreamer/subprojects/gstreamer/gst/gstbin.c:527
#15 0x00007f473df8e3fe in g_object_unref () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0
#16 0x00007f473768a469 in gst_element_call_async_func (data=0x55f9cb658550, user_data=<optimized out>) at ../../../../../jhbuild/checkout/gstreamer/subprojects/gstreamer/gst/gstelement.c:3842
#17 0x00007f4737ffa532 in ??? () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#18 0x00007f4737ff4d92 in ??? () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#19 0x00007f473f28aaa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#20 0x00007f473f317c6c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78


https://build.webkit.org/results/GTK-Linux-64-bit-Debug-Tests/304173@main%20(17716)/imported/w3c/web-platform-tests/webrtc-stats/hardware-capability-stats.https-crash-log.txt

[Philippe Normand]

There are multiple issues here, and those commits are not the culprit because they are libwebrtc-related.

There's some Skia crash here, reported elsewhere.

There's a deadlock too:

Thread 129 (Thread 0x7f0180ff96c0 (LWP 511621)):
#0  0x00007f044dc86d71 in __futex_abstimed_wait_common64 (private=32516, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f0439e20260) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=32516, abstime=0x0, clockid=0, expected=0, futex_word=0x7f0439e20260) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f0439e20260, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f044dc897ed in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f0439e20210, cond=0x7f0439e20238) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7f0439e20238, mutex=0x7f0439e20210) at ./nptl/pthread_cond_wait.c:627
#5  0x00007f044c963d02 in WTF::ThreadCondition::timedWait(WTF::Mutex&, WTF::WallTime) () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libjavascriptcoregtk-6.0.so.1
#6  0x00007f044c86814b in WTF::ParkingLot::parkConditionallyImpl(void const*, WTF::ScopedLambda<bool ()> const&, WTF::ScopedLambda<void ()> const&, WTF::TimeWithDynamicClockType const&) () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libjavascriptcoregtk-6.0.so.1
#7  0x00007f044c957a05 in WTF::BinarySemaphore::waitUntil(WTF::TimeWithDynamicClockType const&) () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libjavascriptcoregtk-6.0.so.1
#8  0x00007f044c856612 in WTF::callOnMainRunLoopAndWait(WTF::Function<void ()>&&) () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libjavascriptcoregtk-6.0.so.1
#9  0x00007f045061817b in non-virtual thunk to WebKit::RiceBackendProxy::gatherSocketAddresses(unsigned int) () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libwebkitgtk-6.0.so.4
#10 0x00007f045196d24f in webkitGstWebRTCIceStreamGatherCandidates(_GstWebRTCICEStream*) () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libwebkitgtk-6.0.so.4
#11 0x00007f045196dc50 in webkitGstWebRTCIceAgentGatherCandidates(_GstWebRTCICE*, _GstWebRTCICEStream*) () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libwebkitgtk-6.0.so.4
#12 0x00007f03581411d3 in _set_description_task (webrtc=<optimized out>, sd=0x55696c5d9b50) at ../../../../../jhbuild/checkout/gstreamer/subprojects/gst-plugins-bad/ext/webrtc/gstwebrtcbin.c:6946
#13 0x00007f035812eddc in _execute_op (op=0x55697004e530) at ../../../../../jhbuild/checkout/gstreamer/subprojects/gst-plugins-bad/ext/webrtc/gstwebrtcbin.c:1212
#14 0x00007f044a22b49e in ??? () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#15 0x00007f044a28a737 in ??? () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#16 0x00007f044a22bf87 in g_main_loop_run () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#17 0x00007f0358131d6d in _gst_pc_thread (webrtc=0x55696fe29b60) at ../../../../../jhbuild/checkout/gstreamer/subprojects/gst-plugins-bad/ext/webrtc/gstwebrtcbin.c:1137
#18 0x00007f044a259d92 in ??? () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#19 0x00007f044dc8aaa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#20 0x00007f044dd17c6c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Thread 30 (Thread 0x7f0442d9d440 (LWP 503380)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f044a28436a in ??? () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f035813198c in gst_webrtc_bin_get_property (object=0x55696fe29b60, prop_id=5, value=0x7ffd88c5ad20, pspec=0x55696fe37050) at ../../../../../jhbuild/checkout/gstreamer/subprojects/gst-plugins-bad/ext/webrtc/gstwebrtcbin.c:8673
#3  0x00007f044cadcdde in g_object_get_valist () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0
#4  0x00007f044cadd27d in g_object_get () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0
#5  0x00007f04519661eb in WTF::Detail::CallableWrapper<WebCore::GStreamerMediaEndpoint::onIceCandidate(unsigned int, char*)::{lambda()#1}, void>::call() () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libwebkitgtk-6.0.so.4
#6  0x00007f044c8712d5 in WTF::RunLoop::performWork() () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libjavascriptcoregtk-6.0.so.1
#7  0x00007f044c95c2cd in WTF::RunLoop::RunLoop()::{lambda(void*)#1}::_FUN(void*) () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libjavascriptcoregtk-6.0.so.1
#8  0x00007f044c95fc5d in WTF::RunLoop::{lambda(_GSource*, int (*)(void*), void*)#1}::_FUN(_GSource*, int (*)(void*), void*) () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libjavascriptcoregtk-6.0.so.1
#9  0x00007f044a22b49e in ??? () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#10 0x00007f044a22b710 in g_main_context_dispatch () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#11 0x00007f044c95e107 in WTF::RunLoop::runGLibMainLoopIteration(WTF::RunLoop::MayBlock) () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libjavascriptcoregtk-6.0.so.1
#12 0x00007f044c95e512 in WTF::RunLoop::run() () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libjavascriptcoregtk-6.0.so.1
#13 0x00007f045074e68e in int WebKit::AuxiliaryProcessMain<WebKit::WebProcessMainGtk>(int, char**) () at /home/buildbot-worker/GTK-Linux-64-bit-Release-Build/build/WebKitBuild/GTK/Release/lib/libwebkitgtk-6.0.so.4
#14 0x00007f044dc181ca in __libc_start_call_main (main=main@entry=0x55695f3f6800 <main>, argc=argc@entry=4, argv=argv@entry=0x7ffd88c5b318) at ../sysdeps/nptl/libc_start_call_main.h:58
#15 0x00007f044dc1828b in __libc_start_main_impl (main=0x55695f3f6800 <main>, argc=4, argv=0x7ffd88c5b318, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd88c5b308) at ../csu/libc-start.c:360
#16 0x000055695f3f6735 in _start ()

The webrtcbin lock is taken in thread 129 and thread 30 is blocked, attempting to also take it...

[Philippe Normand]

Pull request: https://github.com/WebKit/WebKit/pull/55175

[Philippe Normand]

I couldn't reproduce the issue here, so this is a prospective fix, sorry...

[Claudio Saavedra]

> There's some Skia crash here, reported elsewhere.

Yes, I saw that one too, but it looks like that was already happening before with other tests. What I noticed too for example is that around 303625@main webrtc/captureCanvas-webrtc-with-video-scaling-adaptation.html (which was previously a flaky crash reported in https://bugs.webkit.org/show_bug.cgi?id=286859) started failing with a text failure:

https://build.webkit.org/results/GTK-Linux-64-bit-Release-Tests/304227@main%20(22441)/webrtc/captureCanvas-webrtc-with-video-scaling-adaptation-pretty-diff.html

So some regressions might have not been noticed when that landed because the tests were already marked as flakies :-/

[Claudio Saavedra]

(In reply to Claudio Saavedra from comment #5)
>  around 303625@main
> 

Actually I meant https://commits.webkit.org/303624@main and https://commits.webkit.org/303623@main: 

303608@main                   PASS (Expected: PASS CRASH)
[303609@main-303612@main]     UNKNOWN
303613@main                   PASS (Expected: PASS CRASH)
[303614@main-303617@main]     UNKNOWN
[303618@main-303619@main]     PASS (Expected: PASS CRASH)
303620@main                   UNKNOWN
[303621@main-303622@main]     PASS (Expected: PASS CRASH)
303623@main                   UNKNOWN
303624@main                   CRASH (Expected: PASS CRASH)
303625@main                   TEXT CRASH (Expected: PASS CRASH)
303626@main                   UNKNOWN
303627@main                   TEXT CRASH (Expected: PASS CRASH)
303628@main                   UNKNOWN
303629@main                   CRASH (Expected: PASS CRASH)
303630@main                   TEXT CRASH (Expected: PASS CRASH)
[303631@main-303632@main]     CRASH (Expected: PASS CRASH)
[303633@main-303634@main]     TEXT CRASH (Expected: PASS CRASH)
303635@main                   TEXT (Expected: PASS CRASH)
303636@main                   UNKNOWN
303637@main                   TEXT CRASH (Expected: PASS CRASH)
303638@main                   TEXT (Expected: PASS CRASH)
303639@main                   UNKNOWN
303640@main                   TEXT (Expected: PASS CRASH)

[Philippe Normand]

*** Bug 303372 has been marked as a duplicate of this bug. ***

[EWS]

Committed 304290@main (5019b64ef279): <https://commits.webkit.org/304290@main>

Reviewed commits have been landed. Closing PR #55175 and removing active labels.