Bug 257551 – [GStreamer] Crash after 10 seconds on watchdog thread due to hang in gst_deinit()

• Status: RESOLVED
• Resolution: FIXED
• Component: Media

[Michael Catanzaro]

Here's another case of us hitting the WebKit::crashAfter10Seconds crash that we added in bug #249272 (WebProcess.cpp:281). Instead of posting the backtrace for the watchdog thread where the crash occurs, which is useless, I'm instead going to post the backtrace for the main thread since that's the thread where we have a problem:

Thread 29 (Thread 0x7f649d287a00 (LWP 2)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f64a0c8d78e in g_cond_wait (cond=cond@entry=0x563e7ac1bd10, mutex=0x563e7ac1b670) at ../glib/gthread-posix.c:1475
#2  0x00007f64a0c5ff3d in g_thread_pool_free (pool=0x563e7ac1bcf0, immediate=0, wait_=<optimized out>) at ../glib/gthreadpool.c:931
#3  0x00007f64a1502645 in gst_task_cleanup_all () at ../gst/gsttask.c:439
#4  0x00007f64a147860d in gst_deinit () at ../gst/gst.c:1123
#5  gst_deinit () at ../gst/gst.c:1101
#6  0x00007f64a58523b1 in WebKit::AuxiliaryProcessMainBase<WebKit::WebProcess, true>::run(int, char**) (this=0x7fff97241480, argc=3, argv=0x7fff97241618) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebKit/Shared/AuxiliaryProcessMain.h:73
#7  WebKit::AuxiliaryProcessMain<WebKit::WebProcessMainGtk>(int, char**) (argc=3, argv=0x7fff97241618) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebKit/Shared/AuxiliaryProcessMain.h:98
#8  0x00007f64a462954a in __libc_start_call_main (main=main@entry=0x563e78ff4150 <main>, argc=argc@entry=3, argv=argv@entry=0x7fff97241618) at ../sysdeps/nptl/libc_start_call_main.h:58
#9  0x00007f64a462960b in __libc_start_main_impl (main=0x563e78ff4150 <main>, argc=3, argv=0x7fff97241618, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:389
#10 0x0000563e78ff4085 in _start (

We hang when freeing a thread pool in gst_deinit(). Although the wait_ argument here is optimized out, I checked gstthreadpool.c to confirm that it's TRUE, causing GLib to wait until all threads finish before returning. So one of the GStreamer threads has deadlocked. There are only two such threads:

Thread 18 (Thread 0x7f625e7fc640 (LWP 102)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f64a0c8d78e in g_cond_wait (cond=cond@entry=0x563e7ad4b320, mutex=mutex@entry=0x563e7ad4b330) at ../glib/gthread-posix.c:1475
#2  0x00007f64a8cc8578 in gst_app_src_create (bsrc=0x563e7ad4b5f0 [GstAppSrc|appsrc3], offset=<optimized out>, size=<optimized out>, buf=<optimized out>) at ../gst-libs/gst/app/gstappsrc.c:1785
#3  0x00007f64a15c4b58 in gst_base_src_get_range (src=src@entry=0x563e7ad4b5f0 [GstAppSrc|appsrc3], offset=361643, length=<optimized out>, buf=buf@entry=0x7f625e7fba70) at ../libs/gst/base/gstbasesrc.c:2587
#4  0x00007f64a15cab09 in gst_base_src_loop (pad=0x563e7ad4b8f0 [GstPad|src]) at ../libs/gst/base/gstbasesrc.c:2911
#5  0x00007f64a14fa211 in gst_task_func (task=0x563e7ab40c70 [GstTask|appsrc3:src]) at ../gst/gsttask.c:384
#6  0x00007f64a0c5f452 in g_thread_pool_thread_proxy (data=<optimized out>) at ../glib/gthreadpool.c:350
#7  0x00007f64a0c5ea19 in g_thread_proxy (data=0x7f6490004ea0) at ../glib/gthread.c:831
#8  0x00007f64a468f1da in start_thread (arg=<optimized out>) at pthread_create.c:442
#9  0x00007f64a4717f44 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Warning: the current language does not match this frame.

Thread 10 (Thread 0x7f625effd640 (LWP 101)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f64a0c8d78e in g_cond_wait (cond=cond@entry=0x563e7ac80390, mutex=mutex@entry=0x563e7ac803a0) at ../glib/gthread-posix.c:1475
#2  0x00007f64a8cc8578 in gst_app_src_create (bsrc=0x563e7ac80660 [GstAppSrc|appsrc2], offset=<optimized out>, size=<optimized out>, buf=<optimized out>) at ../gst-libs/gst/app/gstappsrc.c:1785
#3  0x00007f64a15c4b58 in gst_base_src_get_range (src=src@entry=0x563e7ac80660 [GstAppSrc|appsrc2], offset=3903, length=<optimized out>, buf=buf@entry=0x7f625effca70) at ../libs/gst/base/gstbasesrc.c:2587
#4  0x00007f64a15cab09 in gst_base_src_loop (pad=0x563e7ac7ed70 [GstPad|src]) at ../libs/gst/base/gstbasesrc.c:2911
#5  0x00007f64a14fa211 in gst_task_func (task=0x563e7acb7170 [GstTask|appsrc2:src]) at ../gst/gsttask.c:384
#6  0x00007f64a0c5f452 in g_thread_pool_thread_proxy (data=<optimized out>) at ../glib/gthreadpool.c:350
#7  0x00007f64a0c5ea19 in g_thread_proxy (data=0x7f6490004bf0) at ../glib/gthread.c:831
#8  0x00007f64a468f1da in start_thread (arg=<optimized out>) at pthread_create.c:442
#9  0x00007f64a4717f44 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Warning: the current language does not match this frame.

Is something going wrong in GstAppSrc?

[Michael Catanzaro]

Created attachment 466552 [details]
All-threads backtrace

[Philippe Normand]

I'm going to guess you were browsing some site with videos?

We use appsrc for:

- mediastream/webrtc: I guess this is out of scope
- MSE
- webaudio

[Michael Catanzaro]

(In reply to Philippe Normand from comment #2)
> I'm going to guess you were browsing some site with videos?

Safe bet. Probably reddit.

It might be related to https://gitlab.gnome.org/GNOME/gnome-build-meta/-/issues/683 (but GStreamer should of course be robust to any such issues).

[Philippe Normand]

Looking at AppendPipeline, I wonder if we have a ref cycle between SourceBufferPrivateGStreamer and AppendPipeline?

SourceBufferPrivateGStreamer keeps a UniqueRef to AppendPipeline, and AppendPipeline keeps a ref of SourceBufferPrivateGStreamer...

[Michael Catanzaro]

More recent example: https://gitlab.gnome.org/GNOME/epiphany/-/issues/2257

[Philippe Normand]

Pull request: https://github.com/WebKit/WebKit/pull/22122

[EWS]

Committed 272552@main (6c5b045e794c): <https://commits.webkit.org/272552@main>

Reviewed commits have been landed. Closing PR #22122 and removing active labels.

[Radar WebKit Bug Importer]

<rdar://problem/120340688>