Bug 204715

Summary: [GTK] WebKitGTK build hangs on g-ir-scanner
Product: WebKit Reporter: Jim Mason <jmason>
Component: WebKitGTKAssignee: Nobody <webkit-unassigned>
Status: RESOLVED FIXED    
Severity: Normal CC: bugs-noreply, cgarcia, commit-queue, ews-watchlist, hi, joepeck, keith_miller, mark.lam, msaboff, saam, tzagallo
Priority: P2    
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Patch
none
Patch
none
Patch
none
Patch none

Description Jim Mason 2019-11-30 06:08:00 PST 
The patch for Bug 204503 ([GTK][WPE] RemoteInspector: use sockets instead of DBus) seems to have introduced a hang in the build process.  Specifically, in my environment, the build hangs whilst running g-ir-scanner.

There are two hung `g-ir-scanner` processes and two `WebKit2-4.0` children.  gdb reveals the latter are spinning on a synchronization primitive.  The gdb backtrace is as follows:

  #0  0x00007ff7ef9814fa in yield () at /lib/64/libc.so.1
  #1  0x00007ff7ef95d281 in sched_yield () at /lib/64/libc.so.1
  #2  0x00007ff7e579f2e5 in bmalloc::Mutex::lockSlowCase() ()
      at /build/rtutils/components/desktop/webkitgtk4-dev/build/amd64/lib/libjavascriptcoregtk-4.0.so.18
  #3  0x00007ff7e57982fd in bmalloc::StaticPerProcess<bmalloc::Environment>::getSlowCase() ()
      at /build/rtutils/components/desktop/webkitgtk4-dev/build/amd64/lib/libjavascriptcoregtk-4.0.so.18
  #4  0x00007ff7e5797cdd in bmalloc::Cache::allocateSlowCaseNullCache(bmalloc::HeapKind, unsigned long) ()
      at /build/rtutils/components/desktop/webkitgtk4-dev/build/amd64/lib/libjavascriptcoregtk-4.0.so.18
  #5  0x00007ff7e576fb44 in WTF::CStringBuffer::createUninitialized(unsigned long) ()
      at /build/rtutils/components/desktop/webkitgtk4-dev/build/amd64/lib/libjavascriptcoregtk-4.0.so.18
  #6  0x00007ff7e576fb82 in WTF::CString::init(char const*, unsigned long) ()
      at /build/rtutils/components/desktop/webkitgtk4-dev/build/amd64/lib/libjavascriptcoregtk-4.0.so.18
  #7  0x00007ff7ee91ac16 in _GLOBAL__sub_I_UnifiedSource_88d1702b_25.cpp ()
      at /build/rtutils/components/desktop/webkitgtk4-dev/build/amd64/lib/libwebkit2gtk-4.0.so.37
  #8  0x00007ff7efa4237f in call_array () at /lib/amd64/ld.so.1
  #9  0x00007ff7efa424ee in call_init () at /lib/amd64/ld.so.1
  #10 0x00007ff7efa42bba in load_completion () at /lib/amd64/ld.so.1
  #11 0x00007ff7efa482ab in dlmopen_check () at /lib/amd64/ld.so.1
  #12 0x00007ff7efa4838c in dlopen () at /lib/amd64/ld.so.1
  #13 0x00007ff7e57991df in bmalloc::Environment::computeIsDebugHeapEnabled() ()
      at /build/rtutils/components/desktop/webkitgtk4-dev/build/amd64/lib/libjavascriptcoregtk-4.0.so.18
  #14 0x00007ff7e5799241 in bmalloc::Environment::Environment(std::lock_guard<bmalloc::Mutex>&) ()
      at /build/rtutils/components/desktop/webkitgtk4-dev/build/amd64/lib/libjavascriptcoregtk-4.0.so.18
  #15 0x00007ff7e579831f in bmalloc::StaticPerProcess<bmalloc::Environment>::getSlowCase() ()
      at /build/rtutils/components/desktop/webkitgtk4-dev/build/amd64/lib/libjavascriptcoregtk-4.0.so.18
  #16 0x00007ff7e5797cdd in bmalloc::Cache::allocateSlowCaseNullCache(bmalloc::HeapKind, unsigned long) ()
      at /build/rtutils/components/desktop/webkitgtk4-dev/build/amd64/lib/libjavascriptcoregtk-4.0.so.18
  #17 0x00007ff7e576fb44 in WTF::CStringBuffer::createUninitialized(unsigned long) ()
      at /build/rtutils/components/desktop/webkitgtk4-dev/build/amd64/lib/libjavascriptcoregtk-4.0.so.18
  #18 0x00007ff7e576fb82 in WTF::CString::init(char const*, unsigned long) ()
      at /build/rtutils/components/desktop/webkitgtk4-dev/build/amd64/lib/libjavascriptcoregtk-4.0.so.18
  #19 0x00007ff7e5061828 in __static_initialization_and_destruction_0(int, int) [clone .constprop.0] ()
      at /build/rtutils/components/desktop/webkitgtk4-dev/build/amd64/lib/libjavascriptcoregtk-4.0.so.18
  #20 0x00007ff7efa4237f in call_array () at /lib/amd64/ld.so.1
  #21 0x00007ff7efa424ee in call_init () at /lib/amd64/ld.so.1
  #22 0x00007ff7efa42bba in load_completion () at /lib/amd64/ld.so.1
  #23 0x00007ff7efa52300 in elf_bndr () at /lib/amd64/ld.so.1
  #24 0x00007ff7efa33703 in elf_rtbndr () at /lib/amd64/ld.so.1
  #25 0x00007ff7ef9e10e8 in  ()
  #26 0x0000000000000035 in  ()
  #27 0x0000000000406187 in invoke_get_type ()
  #28 0x0000000000407044 in dump_irepository ()
  #29 0x000000000040730b in main ()

After backing out the patch from Bug 204503, the build completes normally.

Building on Solaris x86_64 with glib 2.52.0.  If you need more information from the environment, please advise.
Comment 1 Jim Mason 2019-11-30 12:10:47 PST 
FWIW, the failure mode is very much like Bug 125651.
Comment 2 Jim Mason 2019-12-05 02:48:44 PST 
Rebuilt with debug symbols.  Problem is in initialization of a static, s_messageHandlers, which was added in the Bug 204503 patch:


(gdb) frame 34
#34 __static_initialization_and_destruction_0 (__initialize_p=<optimized out>, 
    __priority=65535)
    at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/JavaScriptCore/inspector/remote/glib/RemoteInspectorGlib.cpp:95
95	const SocketConnection::MessageHandlers RemoteInspector::s_messageHandlers = {
(


The complete debug backtrace of the deadlocked thread:


(gdb) bt
#0  0x00007ffa718c14fa in yield () at /lib/64/libc.so.1
#1  0x00007ffa7189d281 in sched_yield () at /lib/64/libc.so.1
#2  0x00007ffa6775f975 in bmalloc::Mutex::lockSlowCase()
    (this=0x7ffa677f1339 <bmalloc::StaticPerProcessStorageTraits<bmalloc::Environment>::Storage::s_mutex>)
    at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/bmalloc/bmalloc/Mutex.cpp:52
#3  0x00007ffa67757dbd in bmalloc::Mutex::lock() (this=<optimized out>)
    at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/bmalloc/bmalloc/Mutex.h:82
#4  0x00007ffa67757dbd in std::lock_guard<bmalloc::Mutex>::lock_guard(bmalloc::Mutex&) (__m=..., this=0xffff80c792ce4eb8)
    at /usr/gcc/9/include/c++/9.2.0/bits/std_mutex.h:159
#5  0x00007ffa67757dbd in bmalloc::StaticPerProcess<bmalloc::Environment>::getSlowCase() ()
    at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/bmalloc/bmalloc/StaticPerProcess.h:82
#6  0x00007ffa6775779d in bmalloc::StaticPerProcess<bmalloc::Environment>::get() ()
    at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/bmalloc/bmalloc/DebugHeap.h:76
#7  0x00007ffa6775779d in bmalloc::DebugHeap::tryGet() ()
    at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/bmalloc/bmalloc/DebugHeap.h:74
#8  0x00007ffa6775779d in bmalloc::Cache::allocateSlowCaseNullCache(bmalloc::HeapKind, unsigned long) (heapKind=heapKind@entry=bmalloc::HeapKind::Primary, size=size@entry=25) at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/bmalloc/bmalloc/Cache.cpp:63
#9  0x00007ffa676f4771 in bmalloc::Cache::allocate(bmalloc::HeapKind, unsigned long) (size=25, heapKind=bmalloc::HeapKind::Primary) at /build/rtutils/components/desktop/webkitgtk4-dev/build/amd64/DerivedSources/ForwardingHeaders/bmalloc/Cache.h:81
#10 0x00007ffa676f4771 in bmalloc::api::malloc(unsigned long, bmalloc::HeapKind) (kind=bmalloc::HeapKind::Primary, size=25) at /build/rtutils/components/desktop/webkitgtk4-dev/build/amd64/DerivedSources/ForwardingHeaders/bmalloc/bmalloc.h:49
#11 0x00007ffa676f4771 in WTF::fastMalloc(unsigned long) (size=size@entry=25) at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/WTF/wtf/FastMalloc.cpp:279
#12 0x00007ffa67737c74 in WTF::CStringBuffer::createUninitialized(unsigned long) (length=length@entry=8) at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/WTF/wtf/text/CString.cpp:41
#13 0x00007ffa67737cb2 in WTF::CString::init(char const*, unsigned long) (this=0xffff80c792ce4ff0, str=0x7ffa6e5a910e "DidClose", length=8) at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/WTF/wtf/text/CString.cpp:67
#14 0x00007ffa7086c556 in _GLOBAL__sub_I_UnifiedSource_88d1702b_25.cpp () at /build/rtutils/components/desktop/webkitgtk4-dev/build/amd64/lib/libwebkit2gtk-4.0.so.37
#15 0x00007ffa7197d37f in call_array () at /lib/amd64/ld.so.1
#16 0x00007ffa7197d4ee in call_init () at /lib/amd64/ld.so.1
#17 0x00007ffa7197dbba in load_completion () at /lib/amd64/ld.so.1
#18 0x00007ffa719832ab in dlmopen_check () at /lib/amd64/ld.so.1
#19 0x00007ffa7198338c in dlopen () at /lib/amd64/ld.so.1
#20 0x00007ffa67758c4f in bmalloc::isSanitizerEnabled () at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/bmalloc/bmalloc/Environment.cpp:141
#21 0x00007ffa67758c4f in bmalloc::Environment::computeIsDebugHeapEnabled() (this=<optimized out>) at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/bmalloc/bmalloc/Environment.cpp:141
#22 0x00007ffa67758c4f in bmalloc::Environment::computeIsDebugHeapEnabled() (this=<optimized out>) at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/bmalloc/bmalloc/Environment.cpp:135
#23 0x00007ffa67758cb1 in bmalloc::Environment::Environment(std::lock_guard<bmalloc::Mutex>&) (this=0x7ffa677f1338 <bmalloc::StaticPerProcessStorageTraits<bmalloc::Environment>::Storage::s_memory>) at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/bmalloc/bmalloc/Environment.cpp:130
#24 0x00007ffa67757ddf in bmalloc::StaticPerProcess<bmalloc::Environment>::getSlowCase() () at /usr/gcc/9/include/c++/9.2.0/new:174
#25 0x00007ffa6775779d in bmalloc::StaticPerProcess<bmalloc::Environment>::get() () at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/bmalloc/bmalloc/DebugHeap.h:76
#26 0x00007ffa6775779d in bmalloc::DebugHeap::tryGet() () at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/bmalloc/bmalloc/DebugHeap.h:74
#27 0x00007ffa6775779d in bmalloc::Cache::allocateSlowCaseNullCache(bmalloc::HeapKind, unsigned long) (heapKind=heapKind@entry=bmalloc::HeapKind::Primary, size=size@entry=25) at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/bmalloc/bmalloc/Cache.cpp:63
#28 0x00007ffa676f4771 in bmalloc::Cache::allocate(bmalloc::HeapKind, unsigned long) (size=25, heapKind=bmalloc::HeapKind::Primary) at /build/rtutils/components/desktop/webkitgtk4-dev/build/amd64/DerivedSources/ForwardingHeaders/bmalloc/Cache.h:81
#29 0x00007ffa676f4771 in bmalloc::api::malloc(unsigned long, bmalloc::HeapKind) (kind=bmalloc::HeapKind::Primary, size=25) at /build/rtutils/components/desktop/webkitgtk4-dev/build/amd64/DerivedSources/ForwardingHeaders/bmalloc/bmalloc.h:49
#30 0x00007ffa676f4771 in WTF::fastMalloc(unsigned long) (size=size@entry=25) at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/WTF/wtf/FastMalloc.cpp:279
#31 0x00007ffa67737c74 in WTF::CStringBuffer::createUninitialized(unsigned long) (length=length@entry=8) at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/WTF/wtf/text/CString.cpp:41
#32 0x00007ffa67737cb2 in WTF::CString::init(char const*, unsigned long) (this=0xffff80c792ce5460, str=0x7ffa667c724b "DidClose", length=8) at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/WTF/wtf/text/CString.cpp:67
#33 0x00007ffa67071958 in WTF::KeyValuePair<WTF::CString, std::pair<WTF::CString, void (*)(WTF::SocketConnection&, _GVariant*, void*)> >::KeyValuePair<char const (&) [9], std::pair<WTF::CString, void (*)(WTF::SocketConnection&, _GVariant*, void*)> >(char const (&) [9], std::pair<WTF::CString, void (*)(WTF::SocketConnection&, _GVariant*, void*)>&&) (value=..., key=..., this=0x7ffa631978a0) at /usr/gcc/9/include/c++/9.2.0/bits/move.h:74
#34 0x00007ffa67071958 in __static_initialization_and_destruction_0(int, int) (__initialize_p=<optimized out>, __priority=65535) at /build/rtutils/components/desktop/webkitgtk4-dev/webkit/Source/JavaScriptCore/inspector/remote/glib/RemoteInspectorGlib.cpp:95
#35 0x00007ffa7197d37f in call_array () at /lib/amd64/ld.so.1
#36 0x00007ffa7197d4ee in call_init () at /lib/amd64/ld.so.1
#37 0x00007ffa7197dbba in load_completion () at /lib/amd64/ld.so.1
#38 0x00007ffa7198d300 in elf_bndr () at /lib/amd64/ld.so.1
#39 0x00007ffa7196e703 in elf_rtbndr () at /lib/amd64/ld.so.1
#40 0x00007ffa719210e8 in  ()
#41 0x0000000000000035 in  ()
#42 0x0000000000406187 in invoke_get_type ()
#43 0x0000000000407044 in dump_irepository ()
#44 0x000000000040730b in main ()


Suggestions are welcome.  I am quite surprised that I am the only one who is apparently hitting the problem caused by the changes of Bug 204503...
Comment 3 Jim Mason 2019-12-06 05:06:18 PST 
Created attachment 385005 [details]
Patch
Comment 4 Jim Mason 2019-12-06 05:29:01 PST 
Comment on attachment 385005 [details]
Patch

It seems the static data members introduced by Bug 204503 can elicit a static initialization order problem.

The attached patch replaces the static data members with statics that are constructed only upon first access (i.e., the 'construct on first use' idiom).
Comment 5 Carlos Garcia Campos 2019-12-06 05:52:43 PST 
Comment on attachment 385005 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=385005&action=review

> Source/JavaScriptCore/inspector/remote/RemoteInspector.h:181
> -    static const SocketConnection::MessageHandlers s_messageHandlers;
> +    static const SocketConnection::MessageHandlers& getMessageHandlers();

getMessageHandlers() -> messageHandlers()

> Source/JavaScriptCore/inspector/remote/glib/RemoteInspectorGlib.cpp:97
> +    static SocketConnection::MessageHandlers *messageHandlers = new SocketConnection::MessageHandlers({

Use NeverDestroyed.
Comment 6 Jim Mason 2019-12-06 08:30:48 PST 
Created attachment 385014 [details]
Patch
Comment 7 Jim Mason 2019-12-06 08:44:14 PST 
Created attachment 385016 [details]
Patch
Comment 8 Jim Mason 2019-12-06 08:50:08 PST 
Created attachment 385017 [details]
Patch
Comment 9 Jim Mason 2019-12-06 08:53:04 PST 
(In reply to Carlos Garcia Campos from comment #5)
> Comment on attachment 385005 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=385005&action=review
> 
> > Source/JavaScriptCore/inspector/remote/RemoteInspector.h:181
> > -    static const SocketConnection::MessageHandlers s_messageHandlers;
> > +    static const SocketConnection::MessageHandlers& getMessageHandlers();
> 
> getMessageHandlers() -> messageHandlers()
> 
> > Source/JavaScriptCore/inspector/remote/glib/RemoteInspectorGlib.cpp:97
> > +    static SocketConnection::MessageHandlers *messageHandlers = new SocketConnection::MessageHandlers({
> 
> Use NeverDestroyed.

Thanks Carlos, I have revised the patch with your suggestions.
Comment 10 WebKit Commit Bot 2019-12-13 08:22:41 PST 
The commit-queue encountered the following flaky tests while processing attachment 385017 [details]:

imported/w3c/web-platform-tests/content-security-policy/unsafe-eval/eval-scripts-setTimeout-blocked.sub.html bug 203973 (author: dbates@webkit.org)
The commit-queue is continuing to process your patch.
Comment 11 WebKit Commit Bot 2019-12-13 08:23:36 PST 
Comment on attachment 385017 [details]
Patch

Clearing flags on attachment: 385017

Committed r253475: <https://trac.webkit.org/changeset/253475>
Comment 12 WebKit Commit Bot 2019-12-13 08:23:38 PST 
All reviewed patches have been landed.  Closing bug.