Bug 309296 – REGRESSION(306838@main): [GTK] Crash in LayerTreeHost::updateRendering

• Component: WebKitGTK
• Status: NEW

[Michael Catanzaro]

I have three of these within a 10 minute period, but, as usual, no reproducer. Sorry. :(

The stack trace is very confusing. I don't know what is wrong.

Core was generated by `/usr/libexec/webkitgtk-6.0/WebKitWebProcess 456 623'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  std::__atomic_base<unsigned int>::load (this=0x0, __m=std::memory_order::seq_cst)
    at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/atomic_base.h:501
501		return __atomic_load_n(&_M_i, int(__m));
[Current thread is 1 (Thread 0x7f3f829f1d80 (LWP 2))]
(gdb) bt full
#0  std::__atomic_base<unsigned int>::load (this=0x0, __m=std::memory_order::seq_cst)
    at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/atomic_base.h:501
#1  std::__atomic_base<unsigned int>::operator unsigned int (this=0x0)
    at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/atomic_base.h:361
#2  WTF::ThreadSafeRefCountedBase::hasOneRef (this=0x0) at WTF/Headers/wtf/ThreadSafeRefCounted.h:47
#3  WebKit::LayerTreeHost::updateRendering()::$_0::operator()<WTF::KeyValuePair<unsigned long, WTF::Ref<WebCore::CoordinatedImageBackingStore, WTF::RawPtrTraits<WebCore::CoordinatedImageBackingStore>, WTF::DefaultRefDerefTraits<WebCore::CoordinatedImageBackingStore> > > >(WTF::KeyValuePair<unsigned long, WTF::Ref<WebCore::CoordinatedImageBackingStore, WTF::RawPtrTraits<WebCore::CoordinatedImageBackingStore>, WTF::DefaultRefDerefTraits<WebCore::CoordinatedImageBackingStore> > >&) const (this=<optimized out>, it=<optimized out>)
    at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WebKit/WebProcess/WebPage/CoordinatedGraphics/LayerTreeHost.cpp:241
#4  removeIf<(lambda at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebKit/WebProcess/WebPage/CoordinatedGraphics/LayerTreeHost.cpp:240:35)> (this=<optimized out>, functor=<optimized out>) at WTF/Headers/wtf/HashTable.h:1178
        i = <optimized out>
        bucket = <optimized out>
        removedBucketCount = 0
        table = 0x7f3d4e4004b0
#5  removeIf<(lambda at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebKit/WebProcess/WebPage/CoordinatedGraphics/LayerTreeHost.cpp:240:35)> (this=<optimized out>, functor=<optimized out>) at WTF/Headers/wtf/HashMap.h:564
#6  WebKit::LayerTreeHost::updateRendering (this=0x7f3f7600c600)
    at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WebKit/WebProcess/WebPage/CoordinatedGraphics/LayerTreeHost.cpp:240
        reentrancyProtector = {m_scopedVariable = @0x7f3f7600c643, m_valueToRestore = <optimized out>}
        traceScope = {m_exitCode = LayerTreeHostRenderingUpdateEnd}
        page = {static isRef = <optimized out>, m_ptr = 0x7f3f760a8640}
        flags = {m_storage = <optimized out>}
        didChangeSceneState = <optimized out>
#7  0x00007f3f8ad787ea in WTF::Function<void()>::operator() (this=0x7f3d2e4e1ee0)
    at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/Function.h:103
#8  WTF::ActivityObserver::notify (this=0x7f3d2e4e1ec0)
    at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/ActivityObserver.h:78
#9  WTF::RunLoop::notifyActivity (this=0x7f3f76014180, activity=WTF::RunLoop::Activity::BeforeWaiting)
    at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:293
        observer = {static isRef = <optimized out>, m_ptr = 0x7f3d2e4e1ec0}
        observersToBeNotified = Python Exception <class 'gdb.error'>: value has been optimized out

#10 0x00007f3f8ad78547 in WTF::RunLoop::runGLibMainLoopIteration (this=0x7f3f76014180, mayBlock=WTF::RunLoop::MayBlock::Yes)
    at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:180
        maxPriority = 100
        timeoutInMilliseconds = 0
        numFDs = <optimized out>
#11 0x00007f3f8ad7896a in WTF::RunLoop::runGLibMainLoop (this=0x7f3f76014180)
    at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:200
#12 WTF::RunLoop::run () at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:213
        runLoop = {static isRef = <optimized out>, m_ptr = 0x7f3f76014180}
#13 0x00007f3f8daad624 in WebKit::AuxiliaryProcessMainBase<WebKit::WebProcess, true>::run
    (this=0x7fff7330eae0, argc=<optimized out>, argv=<optimized out>)
    at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WebKit/Shared/AuxiliaryProcessMain.h:77
#14 WebKit::AuxiliaryProcessMain<WebKit::WebProcessMainGtk> (argc=3, argv=<optimized out>)
    at /usr/lib/debug/source/sdk/webkitgtk-6.0.bst/Source/WebKit/Shared/AuxiliaryProcessMain.h:103
        auxiliaryMain = {m_storage = {m_storage = {data = {0, 48, 225, 145, 63, 127, 0 <repeats 26 times>, 200, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 255, 255, 255, 255, 0 <repeats 20 times>}}}}
#15 0x00007f3f8c62c975 in __libc_start_call_main
    (main=main@entry=0x55d57ac8f150 <main(int, char**)>, argc=argc@entry=3, argv=argv@entry=0x7fff7330ec78)
    at ../sysdeps/nptl/libc_start_call_main.h:58
        self = <optimized out>
        result = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, 7330086118910019935, 140735125974136, 3, 139911011565568, 94375376395680, 7330086118924699999, 7221606498975697247}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x7fff7330ec78}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
--Type <RET> for more, q to quit, c to continue without paging--c
        not_first_call = <optimized out>
#16 0x00007f3f8c62ca28 in __libc_start_main_impl
    (main=0x55d57ac8f150 <main(int, char**)>, argc=3, argv=0x7fff7330ec78, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff7330ec68) at ../csu/libc-start.c:360
#17 0x000055d57ac8f085 in _start () at ../sysdeps/x86_64/start.S:115

[Michael Catanzaro]

So the code here is:

    // Eject any backing stores whose only reference is held in the HashMap cache.
    m_imageBackingStores.removeIf([](auto& it) {
        return it.value->hasOneRef(); // <-- crash occurs here
    });

This code looks safe: m_imageBackingStores holds Ref to a CoordinatedImageBackingStore, which inherits from ThreadSafeRefCounted.

I think the LayerTreeHost itself is still valid:

(gdb) print *this
$5 = {<WTF::CanMakeCheckedPtr<WebKit::LayerTreeHost, (WTF::DefaultedOperatorEqual)0, (WTF::CheckedPtrDeleteCheckException)0>> = {<WTF::CanMakeCheckedPtrBase<WTF::SingleThreadIntegralWrapper<unsigned int>, unsigned int, bool, (WTF::CheckedPtrDeleteCheckException)0>> = {
      m_checkedPtrCount = {m_value = 1}}, <No data fields>}, <WebCore::GraphicsLayerFactory> = {
    _vptr$GraphicsLayerFactory = 0x7fc19f016f98 <vtable for WebKit::LayerTreeHost+16>}, <WebCore::CoordinatedPlatformLayer::Client> = {
    _vptr$Client = 0x7fc19f017018 <vtable for WebKit::LayerTreeHost+144>}, m_webPage = @0x7fc1830a4640, m_layerTreeContext = {
    contextID = 1}, m_sceneState = {static isRef = <optimized out>, m_ptr = 0x7fc183108440}, m_rootCompositingLayer = 0x7fc1851e0c00, 
  m_overlayCompositingLayer = 0x0, m_layerTreeStateIsFrozen = false, m_pendingResize = false, m_pendingForceRepaint = false, 
  m_isUpdatingRendering = true, m_waitUntilPaintingComplete = false, m_isSuspended = false, m_isWaitingForRenderer = true, 
  m_scheduledWhileWaitingForRenderer = false, m_forceFrameSync = false, m_compositionRequired = false, 
  m_compositionRequiredInScrollingThread = true, m_compositor = {static isRefPtr = <optimized out>, m_ptr = 0x7fc183074830}, 
  m_forceRepaintAsync = {callback = {m_function = {m_callableWrapper = std::unique_ptr<WTF::Detail::CallableWrapperBase<void>> = {
          get() = 0x0}}, m_callThread = {<No data fields>}}, compositionRequestID = std::optional [no contained value]}, 
  m_renderingUpdateRunLoopObserver = std::unique_ptr<WebCore::RunLoopObserver> = {get() = 0x7fc183106200}, 
  m_skiaPaintingEngine = std::unique_ptr<WebCore::SkiaPaintingEngine> = {get() = 0x7fc183001010}, m_imageBackingStores = {m_impl = {
      static smallMaxLoadNumerator = 3, static smallMaxLoadDenominator = 4, static largeMaxLoadNumerator = 1, 
      static largeMaxLoadDenominator = 2, static maxSmallTableCapacity = 1024, static minLoad = 6, static tableSizeOffset = -1, 
      static tableSizeMaskOffset = -2, static keyCountOffset = -3, static deletedCountOffset = -4, static metadataSize = 16, 
      m_table = 0x7fc183f528f0}}, m_transientZoom = false, m_transientZoomScale = 1, m_transientZoomOrigin = {m_x = 0, m_y = 0}, 
  m_frameDamageHistoryForTestingLock = {static isHeldBit = 1 '\001', static hasParkedBit = 2 '\002', m_byte = {
      value = std::atomic<unsigned char> = { 0 '\000' }}}, m_frameDamageHistoryForTesting = WTF::Vector of length 0, capacity 0, 
  m_damageInGlobalCoordinateSpace = std::shared_ptr<WebCore::Damage> (use count 23, weak count 0) = {get() = 0x5582d2f78f90}, 
  static _vtable$ = <optimized out>}

And the HashTable:

(gdb) print m_imageBackingStores
$2 = {m_impl = {static smallMaxLoadNumerator = 3, static smallMaxLoadDenominator = 4, static largeMaxLoadNumerator = 1, 
    static largeMaxLoadDenominator = 2, static maxSmallTableCapacity = 1024, static minLoad = 6, static tableSizeOffset = -1, 
    static tableSizeMaskOffset = -2, static keyCountOffset = -3, static deletedCountOffset = -4, static metadataSize = 16, 
    m_table = 0x7fc183f528f0}}

[Michael Catanzaro]

Luckily, I have found a reproducer! Visit https://www.employeefiduciary.com/blog/solo-401k-vs.-sep-ira and scroll the page up or down. Normally it crashes within 5-10 seconds. Sometimes it takes longer, but it should eventually crash. So that should help a lot.

[Claudio Saavedra]

I can't reproduce this in main, is this main or a particular wkgtk release? If so, can you reproduce in main too?

[Michael Catanzaro]

I'll find out. I had been testing WebKitGTK 2.51.93. Since it's reproducible, I should be able to bisect... hopefully, as long as the bug occurs in my jhbuild environment.

[Michael Catanzaro]

Reportedly this bug can also be triggered by scrolling on Jupyter notebooks.

[Michael Catanzaro]

Well bad news: as of today, the bug is no longer reproducible. I still see it occasionally crashing in Tech Preview (2 so far in the past hour), but my reproducer from yesterday no longer triggers the crash at all.

Since it's no longer reproducible, I won't be able to guess whether the issue is broken in main or only on the 2.52 branch, and I certainly won't be able to bisect.

[Xi Ruoyao]

I reproduced it several times reviewing larger GitHub pull requests with 2.52.0.

[Xi Ruoyao]

(In reply to Xi Ruoyao from comment #7)
> I reproduced it several times reviewing larger GitHub pull requests with
> 2.52.0.

Scrolling https://github.com/AOSC-Dev/aosc-os-abbs/pull/15249/changes likely triggers the crash but I don't know if it only works for me.

[Michael Catanzaro]

I think it's a "stateful" bug, like bug #263208. Under some unknown condition, it's going to crash a lot, reproducibly. I had no difficulty reproducing this bug on Wednesday; in fact, it kept interrupting my work, because it would happen again and again at random. But if the unknown condition is not present, then it does not crash often, or at all. No luck today: I just cannot get either page to crash.

[Xi Ruoyao]

FWIW I use MESA_LOADER_DRIVER_OVERRIDE=zink and I can easily get the crash but I'm unsure if they are related.

[Michael Catanzaro]

I've hit this crash 4 times in the past 20 minutes, using 2.52.0.

I'm currently compiling WebKit. Guess: the aforementioned unknown condition is just heavy system load. Things go slower, and that's when it crashes. Boring but plausible.

[Michael Catanzaro]

I hit it twice just now on https://github.com/WebKit/WebKit/pull/61521 when scrolling the page.

[Claudio Saavedra]

*** Bug 310973 has been marked as a duplicate of this bug. ***

[Alberto Garcia]

I can reproduce this easily browsing elpais.com and scrolling up and down randomly, the suspicion is that m_imageBackingStores is not thread safe.

[Fujii Hironori]

306838@main is the regression point.
Disabling USE_COORDINATED_GRAPHICS_ASYNC_SCROLLBAR should be able to work around.

[Alberto Garcia]

I think Carlos is working of a fix

[Alberto Garcia]

Created attachment 478833 [details]
Workaround for the crash

So the problem seems to be that m_imageBackingStores is being accessed from different threads, which is not supposed to happen. I've been talking to Carlos and Nikolas, and a proper solution is being worked on.

In the meantime, this workaround should solve the crash, I confirm that it works for me.