Bug 303336 – [GTK] Watchdog crash after 10 seconds due to deadlock when destroying two different GLContexts on separate threads

• Status: NEW
• Component: WebKitGTK

[Michael Catanzaro]

Created attachment 477575 [details]
Full backtrace

We have yet another problem with destroying the EGL display in an exit handler. Thread 1 crashes in WebKit::crashAfter10Seconds. At the time, thread 17 is waiting on a mutex while trying to destroy GLContext 0x7f32393f00c0. Meanwhile, thread 25 is doing the same while trying to destroy GLContext 0x7f323931c2c0. Since they are different GLContexts, I would expect this to be safe, but apparently not. Could it be a bug in mesa?

We've been plagued by this class of bug for at least a decade now. It is impossible on Apple platforms because WebKit does not allow exit-time destructors in cross-platform code or on Apple platforms. But we still allow it in the Linux-specific code. Using NeverDestroyed would eliminate this entire class of bug.

Thread 17 (Thread 0x7f31ad7fa6c0 (LWP 35)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f31cd9bef0a in futex_wait () at /usr/lib/x86_64-linux-gnu/GL/default/lib/libgallium-25.2.6.so
#2  0x00007f31cd9cc170 in do_futex_fence_wait () at /usr/lib/x86_64-linux-gnu/GL/default/lib/libgallium-25.2.6.so
#3  0x00007f31cd9cd34c in util_queue_finish () at /usr/lib/x86_64-linux-gnu/GL/default/lib/libgallium-25.2.6.so
#4  0x00007f31cde3d4ea in si_set_debug_callback () at /usr/lib/x86_64-linux-gnu/GL/default/lib/libgallium-25.2.6.so
#5  0x00007f31cde3dd58 in si_destroy_context () at /usr/lib/x86_64-linux-gnu/GL/default/lib/libgallium-25.2.6.so
#6  0x00007f31cdc08031 in tc_destroy () at /usr/lib/x86_64-linux-gnu/GL/default/lib/libgallium-25.2.6.so
#7  0x00007f31cd590220 in st_destroy_context_priv () at /usr/lib/x86_64-linux-gnu/GL/default/lib/libgallium-25.2.6.so
#8  0x00007f31cd591d0d in st_destroy_context () at /usr/lib/x86_64-linux-gnu/GL/default/lib/libgallium-25.2.6.so
#9  0x00007f31cd4f2ffb in dri_destroy_context () at /usr/lib/x86_64-linux-gnu/GL/default/lib/libgallium-25.2.6.so
#10 0x00007f31de72a329 in dri2_destroy_context () at /usr/lib/x86_64-linux-gnu/GL/default/lib/libEGL_mesa.so.0
#11 0x00007f31de71d04d in eglDestroyContext () at /usr/lib/x86_64-linux-gnu/GL/default/lib/libEGL_mesa.so.0
#12 0x00007f3251346b91 in WebCore::GLContext::~GLContext (this=0x7f32393f00c0) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WebCore/platform/graphics/egl/GLContext.cpp:335
#13 0x00007f32513fc337 in std::default_delete<WebCore::GLContext>::operator() (this=0x7f32393c80c8, __ptr=0x7f32393f00c0) at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/unique_ptr.h:93
#14 std::unique_ptr<WebCore::GLContext, std::default_delete<WebCore::GLContext> >::~unique_ptr (this=0x7f32393c80c8) at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/unique_ptr.h:399
#15 WebCore::SkiaGLContext::~SkiaGLContext (this=0x7f32393c80c0) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WebCore/platform/graphics/skia/PlatformDisplaySkia.cpp:238
#16 WTF::ThreadSafeWeakPtrControlBlock::strongDeref<WebCore::SkiaGLContext, (WTF::DestructionThread)0>() const::{lambda()#1}::operator()() const (this=<optimized out>) at WTF/Headers/wtf/ThreadSafeWeakPtr.h:93
#17 WTF::ThreadSafeWeakPtrControlBlock::strongDeref<WebCore::SkiaGLContext, (WTF::DestructionThread)0> (this=<optimized out>) at WTF/Headers/wtf/ThreadSafeWeakPtr.h:108
#18 0x00007f324e04585f in __GI___call_tls_dtors () at cxa_thread_atexit_impl.c:156
#19 0x00007f324e09b40a in start_thread (arg=<optimized out>) at pthread_create.c:453
#20 0x00007f324e11ee54 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 25 (Thread 0x7f3243ffde40 (LWP 2)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f31de73787a in futex_wait () at /usr/lib/x86_64-linux-gnu/GL/default/lib/libEGL_mesa.so.0
#2  0x00007f31de71b8ef in _eglLockDisplay () at /usr/lib/x86_64-linux-gnu/GL/default/lib/libEGL_mesa.so.0
#3  0x00007f31de71d1c4 in eglMakeCurrent () at /usr/lib/x86_64-linux-gnu/GL/default/lib/libEGL_mesa.so.0
#4  0x00007f3247eac6ad in InternalLoseCurrent () at ../src/EGL/libegl.c:551
#5  0x00007f3251346b81 in WebCore::GLContext::~GLContext (this=0x7f323931c2c0) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WebCore/platform/graphics/egl/GLContext.cpp:334
#6  0x00007f32513fc337 in std::default_delete<WebCore::GLContext>::operator() (this=0x7f323939c4c8, __ptr=0x7f323931c2c0) at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/unique_ptr.h:93
#7  std::unique_ptr<WebCore::GLContext, std::default_delete<WebCore::GLContext> >::~unique_ptr (this=0x7f323939c4c8) at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/unique_ptr.h:399
#8  WebCore::SkiaGLContext::~SkiaGLContext (this=0x7f323939c4c0) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WebCore/platform/graphics/skia/PlatformDisplaySkia.cpp:238
#9  WTF::ThreadSafeWeakPtrControlBlock::strongDeref<WebCore::SkiaGLContext, (WTF::DestructionThread)0>() const::{lambda()#1}::operator()() const (this=<optimized out>) at WTF/Headers/wtf/ThreadSafeWeakPtr.h:93
#10 WTF::ThreadSafeWeakPtrControlBlock::strongDeref<WebCore::SkiaGLContext, (WTF::DestructionThread)0> (this=<optimized out>) at WTF/Headers/wtf/ThreadSafeWeakPtr.h:108
#11 0x00007f3251336723 in WebCore::PlatformDisplay::clearGLContexts (this=0x55f63fc52c98) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WebCore/platform/graphics/PlatformDisplay.cpp:130
#12 0x00007f324f469484 in WebKit::WebProcess::stopRunLoop (this=0x7f3238030110) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WebKit/WebProcess/glib/WebProcessGLib.cpp:124
#13 0x00007f324eee78f2 in WebKit::AuxiliaryProcess::terminate (this=0x7f3238030110) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WebKit/Shared/AuxiliaryProcess.cpp:242
#14 0x00007f324f19a6c0 in WebKit::WebProcess::removeWebPage (this=0x7f3238030110, pageID=...) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WebKit/WebProcess/WebProcess.cpp:1056
#15 0x00007f324f4169b3 in WebKit::WebPage::close (this=0x7f32380e4170) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WebKit/WebProcess/WebPage/WebPage.cpp:2048
#16 0x00007f324f469429 in WebKit::WebProcess::stopRunLoop (this=0x7f3238030110) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WebKit/WebProcess/glib/WebProcessGLib.cpp:121
#17 0x00007f324eeda120 in IPC::Connection::dispatchDidCloseAndInvalidate()::$_0::operator()() const (this=0x7f3238038118) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WebKit/Platform/IPC/Connection.cpp:1311
#18 WTF::Detail::CallableWrapper<IPC::Connection::dispatchDidCloseAndInvalidate()::$_0, void>::call (this=0x7f3238038110) at WTF/Headers/wtf/Function.h:59
#19 0x00007f324c61ee55 in WTF::Function<void()>::operator() (this=0x7ffe00cb4810) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/Function.h:103
#20 WTF::RunLoop::performWork (this=0x7f3238014180) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/RunLoop.cpp:148
#21 0x00007f324c6e566d in WTF::RunLoop::RunLoop()::$_0::operator()(void*) const (userData=0x55f63fc52c98, userData@entry=0x7f3238014180, this=<optimized out>) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:80
#22 WTF::RunLoop::RunLoop()::$_0::__invoke(void*) (userData=0x55f63fc52c98) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:79
#23 0x00007f324c6e3d11 in WTF::RunLoop::$_1::operator() (source=0x55f63fa1b460, callback=0x7f324c6e5660 <WTF::RunLoop::RunLoop()::$_0::__invoke(void*)>, userData=0x7f3238014180, this=<optimized out>) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:57
#24 WTF::RunLoop::$_1::__invoke (source=0x55f63fa1b460, callback=0x7f324c6e5660 <WTF::RunLoop::RunLoop()::$_0::__invoke(void*)>, userData=0x7f3238014180) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:49
#25 0x00007f324ce7d7cb in g_main_dispatch (context=context@entry=0x55f63fa1b2c0) at ../glib/gmain.c:3565
#26 0x00007f324ce81380 in g_main_context_dispatch_unlocked (context=0x55f63fa1b2c0) at ../glib/gmain.c:4425
#27 g_main_context_dispatch (context=0x55f63fa1b2c0) at ../glib/gmain.c:4413
#28 0x00007f324c6e42f3 in WTF::RunLoop::runGLibMainLoopIteration (this=0x7f3238014180, mayBlock=WTF::RunLoop::MayBlock::Yes) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:118
#29 0x00007f324c6e468a in WTF::RunLoop::runGLibMainLoop (this=0x7f3238014180) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:127
#30 WTF::RunLoop::run () at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:140
#31 0x00007f324f469d94 in WebKit::AuxiliaryProcessMainBase<WebKit::WebProcess, true>::run (this=0x7ffe00cb49f0, argc=<optimized out>, argv=<optimized out>) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WebKit/Shared/AuxiliaryProcessMain.h:77
#32 WebKit::AuxiliaryProcessMain<WebKit::WebProcessMainGtk> (argc=3, argv=<optimized out>) at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WebKit/Shared/AuxiliaryProcessMain.h:103
#33 0x00007f324e02c975 in __libc_start_call_main (main=main@entry=0x55f628e04150 <main(int, char**)>, argc=argc@entry=3, argv=argv@entry=0x7ffe00cb4b88) at ../sysdeps/nptl/libc_start_call_main.h:58
#34 0x00007f324e02ca28 in __libc_start_main_impl (main=0x55f628e04150 <main(int, char**)>, argc=3, argv=0x7ffe00cb4b88, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe00cb4b78) at ../csu/libc-start.c:360
#35 0x000055f628e04085 in _start () at ../sysdeps/x86_64/start.S:115

Full backtrace attached.

[Carlos Garcia Campos]

Where is the exit handler? I don't find it in the bt. And where is the egl display being destroyed? I only see threads destroying their GL contexts.

[Michael Catanzaro]

Thread 17 is in an exit handler (__GI___call_tls_dtors), but you're right: the EGL display is not being destroyed. I just assumed it was.

[Michael Catanzaro]

Well, I suppose thread-local storage destruction is not the same as an exit handler.

[[e-mail]]

We are hitting what looks like the same defect as #303336 on a newer
WebKitGTK, but it surfaces as heap corruption + SIGABRT at process exit rather
than as the 10-second watchdog deadlock. Two coredumps captured, both with the
identical signature. Full backtraces attached.

## Version / build
- **WebKitGTK 2.52.4**, API/ABI 6.0 (`libwebkitgtk-6.0.so.4`,
  `libjavascriptcoregtk-6.0.so.1`). Skia rendering backend.
- Build source: **NixOS 26.05** (nixpkgs), stock webkitgtk derivation.
  This build already contains the 2.52.1 heap-corruption fix (RedHat
  bug 2450634), so this is not that issue.
- GTK 4.22.4, GLib 2.88.1, glibc 2.42.
- **Mesa 26.1.4**, `libEGL_mesa` / `libgallium-26.1.4`.
- GPU: AMD integrated (amdgpu, PCI 1002:1900). Real hardware EGL on Wayland.
- Kernel Linux 6.18.38 x86_64.
- Session: KDE Plasma 6.6.6 / KWin 6.6.6 **Wayland**.

## Host application
The WebKitGTK view is embedded in a first-party GTK4 / WebKitGTK-6.0
application (built with Wails v3: a Go backend hosting a
`WebKitWebView` in a GTK4 window). It runs natively on the live Plasma
Wayland session. The app processes only local mock/test data.

## Steps to reproduce
1. Run a GTK4 application embedding a WebKitGTK 6.0 `WebKitWebView`, on a real
   Wayland session with hardware EGL (here: Plasma/KWin Wayland, Mesa, amdgpu).
2. Close the top-level window normally (KWin delivers `xdg_toplevel` close);
   the app shuts down and the main process exits.
3. Repeat. In roughly **2 of 20** window-close teardowns (~6-10%), after the
   main process has already exited cleanly (status 0), the **WebKitWebProcess**
   renderer child aborts during its own exit.

This is a non-deterministic teardown race, so it takes repeated runs.

## Actual result
`WebKitWebProcess` prints `free(): corrupted unsorted chunks` to stderr and
aborts (SIGABRT). The core shows two teardown activities racing:

- A secondary thread is destroying a `SkiaGLContext` from a **libc TLS
  destructor** at exit, on the wrong/secondary thread:

  ```
  #0  syscall
  #1  futex_wait                       (libEGL_mesa.so.0)
  #2  _eglLockDisplay                  (libEGL_mesa.so.0)
  #3  eglDestroyContext                (libEGL_mesa.so.0)
  #4  WebCore::GLContext::~GLContext() (libwebkitgtk-6.0.so.4)
  #5  WTF::ThreadSafeWeakPtrControlBlock::strongDeref<WebCore::SkiaGLContext, (WTF::DestructionThread)0>() const
  #6  __call_tls_dtors                 (libc.so.6)
  #7  start_thread
  #8  clone
  ```

  This is the same `eglDestroyContext` -> `_eglLockDisplay` +
  `SkiaGLContext strongDeref<DestructionThread(0)>` + `__call_tls_dtors`
  pattern described in #303336, blocked on Mesa's EGL display lock.

- Meanwhile another exiting worker thread aborts while releasing its per-thread
  malloc arena — glibc's checker finds a corrupted chunk:

  ```
  #0  abort
  #1  __libc_message_impl.cold
  #2  malloc_printerr
  #3  _int_free_create_chunk
  #4  _int_free_merge_chunk
  #5  _int_free_chunk
  #6  __malloc_arena_thread_freeres
  #7  start_thread
  #8  clone
  ```

- The main thread is in normal `exit()`, running Mesa's atexit handler
  (`atexit_handler` -> `util_queue_kill_threads` -> `thrd_join`).

So at `exit()`, GL contexts are being torn down on secondary threads via TLS
destructors while other threads free their arenas; the heap is left corrupted
and the corruption is tripped during arena cleanup.

**Note on MALLOC_CHECK_:** the process ran with `MALLOC_CHECK_=3` (and
`MALLOC_PERTURB_=170`) set. That does not cause the corruption — it makes the
pre-existing heap damage surface **deterministically as an immediate abort**
instead of a silent/late failure. This is likely why we see a clean SIGABRT
where #303336 saw a watchdog deadlock: the same unsynchronized teardown, made
visible earlier by glibc malloc checking.

## Expected result
The WebKitWebProcess should exit cleanly after the parent closes its window;
GL/Skia contexts should be torn down without cross-thread races or heap
corruption.

## Environment specificity
- Reproduces only on the real EGL/Wayland GPU path (Mesa, amdgpu).
- **Never** observed under X11/Xvfb software GL across 30+ historical runs.
- The parent app is always unaffected (exit 0; captured data intact); only the
  renderer child aborts, after the parent is gone.

## What we have NOT done (honesty)
- We have **not** produced a minimal MiniBrowser/standalone reproducer. Given
  the crash window is purely teardown-after-close and lives entirely in
  WebKit's Skia GL-context destruction + libc TLS-destructor path, we expect it
  is reproducible with essentially any GTK WebKit view destroyed on a
  Wayland/EGL session under repeated open/close — but we have not verified that
  claim, so we are not asserting it.
- We did not bisect or test a patch. We are on 2.52.4; we did not test 2.52.5
  or trunk (public 2.52.5 notes do not call out a fix here).

## Attachments
- `backtraces/key-threads.md` — the relevant threads, annotated.
- `backtraces/gdb-bt-520750.txt`, `backtraces/gdb-bt-523858.txt` — full
  `thread apply all bt` from gdb, symbolized against the nixpkgs webkitgtk
  build (function names + library paths).
- `backtraces/coredumpctl-info-520750.txt`,
  `backtraces/coredumpctl-info-523858.txt` — systemd-coredump metadata +
  per-thread stacks + full loaded-module list.

Raw `.core` files (~15.6M each, contain only mock/test data) are retained
locally and can be provided privately to a maintainer on request.

[[e-mail]]

Created attachment 480700 [details]
Key-treads

[[e-mail]]

Created attachment 480701 [details]
Full symbolized 'thread apply all bt' — coredump 1

[[e-mail]]

Created attachment 480702 [details]
Full symbolized 'thread apply all bt' — coredump 2

[[e-mail]]

Created attachment 480703 [details]
systemd-coredump metadata + module list — coredump 1

[[e-mail]]

Created attachment 480704 [details]
systemd-coredump metadata + module list — coredump 2

[Michael Catanzaro]

> `WebKitWebProcess` prints `free(): corrupted unsorted chunks` to stderr and aborts (SIGABRT).

I also see this routinely with Epiphany, and I agree it's probably the same underlying problem as the deadlock that I reported here.