Bug 227075 – [WPE][GTK] Every web process crash loop caused by ContentExtensions

• Component: WebKitGTK
• Status: NEW

[Michael Catanzaro]

Currently WebKit can somehow get into a bad state where every active web process enters a crash loop. This affects several web processes all at once. Idle web processes don't seem to be affected, but they will start crashing if you try to do something with them. When a web process crashes, it is immediately respawns and then immediately crashes again. The crash occurs in WebCore::ContentExtensions::ContentExtensionsBackend::actionsForResourceLoad.

I don't know how to reproduce it, but I hit it fairly regularly. It's pretty obvious in coredumpctl because a huge number of crashes get recorded all at once. Looks like I hit it once today, once yesterday, once on Friday, once last Wednesday, once last Tuesday, and once last Monday. That's a weird time for WebKit to have started crashing since it doesn't correspond with any recent WebKit update to the GNOME runtime.

This backtrace is using WebKitGTK 2.33.1 because we're having some trouble building 2.33.2, but Carlos Garcia has a newer build that is also affected. I'll attach a full backtrace as well.

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f83916eb855 in __GI_abort () at abort.c:79
#2  0x00007f8391edb4eb in  () at /usr/lib/x86_64-linux-gnu/libwebkit2gtk-4.0.so.37
#3  0x00007f83930b1223 in WebCore::ContentExtensions::ContentExtensionsBackend::actionsForResourceLoad(WebCore::ContentExtensions::ResourceLoadInfo const&) const (this=<optimized out>, resourceLoadInfo=...)
    at ../Source/WebCore/contentextensions/ContentExtensionsBackend.cpp:125
#4  0x00007f83930b5b73 in WebCore::ContentExtensions::ContentExtensionsBackend::processContentRuleListsForLoad(WebCore::Page&, WTF::URL const&, WTF::OptionSet<WebCore::ContentExtensions::ResourceType>, WebCore::DocumentLoader&, WTF::URL const&) (this=0x7f8389289310, page=..., url=..., resourceType=..., resourceType@entry=..., initiatingDocumentLoader=
    ..., redirectFrom=...) at ../Source/WebCore/contentextensions/ContentExtensionsBackend.cpp:199
#5  0x00007f839391ad9b in WebCore::UserContentProvider::processContentRuleListsForLoad(WebCore::Page&, WTF::URL const&, WTF::OptionSet<WebCore::ContentExtensions::ResourceType>, WebCore::DocumentLoader&, WTF::URL const&)
    (this=this@entry=0x7f83892892c0, page=..., url=..., resourceType=resourceType@entry=..., initiatingDocumentLoader=..., redirectFrom=...) at ../Source/WebCore/page/UserContentController.h:59
#6  0x00007f8393824d2d in WebCore::CachedResourceLoader::requestResource(WebCore::CachedResource::Type, WebCore::CachedResourceRequest&&, WebCore::CachedResourceLoader::ForPreload, WebCore::ImageLoading) (this=this@entry=
    0x7f81ae4d60d0, type=type@entry=WebCore::CachedResource::Type::Beacon, request=..., forPreload=forPreload@entry=WebCore::CachedResourceLoader::ForPreload::No, imageLoading=imageLoading@entry=WebCore::ImageLoading::Immediate)
    at ../Source/WebCore/platform/network/ResourceRequestBase.h:169
#7  0x00007f8393827d49 in WebCore::CachedResourceLoader::requestBeaconResource(WebCore::CachedResourceRequest&&)
    (this=this@entry=0x7f81ae4d60d0, request=...) at ../Source/WebCore/loader/cache/CachedResourceLoader.cpp:356
#8  0x00007f8392d05bba in WebCore::NavigatorBeacon::sendBeacon(WebCore::Document&, WTF::String const&, WTF::Optional<WTF::Variant<WTF::RefPtr<WebCore::Blob, WTF::RawPtrTraits<WebCore::Blob>, WTF::DefaultRefDerefTraits<WebCore::Blob> >, WTF::RefPtr<JSC::ArrayBufferView, WTF::RawPtrTraits<JSC::ArrayBufferView>, WTF::DefaultRefDerefTraits<JSC::ArrayBufferView> >, WTF::RefPtr<JSC::ArrayBuffer, WTF::RawPtrTraits<JSC::ArrayBuffer>, WTF::DefaultRefDerefTraits<JSC::ArrayBuffer> >, WTF::RefPtr<WebCore::DOMFormData, WTF::RawPtrTraits<WebCore::DOMFormData>, WTF::DefaultRefDerefTraits<WebCore::DOMFormData> >, WTF::RefPtr<WebCore::URLSearchParams, WTF::RawPtrTraits<WebCore::URLSearchParams>, WTF::DefaultRefDerefTraits<WebCore::URLSearchParams> >, WTF::RefPtr<WebCore::ReadableStream, WTF::RawPtrTraits<WebCore::ReadableStream>, WTF::DefaultRefDerefTraits<WebCore::ReadableStream> >, WTF::String> >&&) (this=0x7f81a594dd98, document=
    ..., url=..., body=...) at WTF/Headers/wtf/RefPtr.h:62
#9  0x00007f8392d05fb7 in WebCore::NavigatorBeacon::sendBeacon(WebCore::Navigator&, WebCore::Document&, WTF::String const&, WTF::Optional<WTF::Variant<WTF::RefPtr<WebCore::Blob, WTF::RawPtrTraits<WebCore::Blob>, WTF::DefaultRefDerefTraits<WebCore::Blob> >, WTF::RefPtr<JSC::ArrayBufferView, WTF::RawPtrTraits<JSC::ArrayBufferView>, WTF::DefaultRefDerefTraits<JSC::ArrayBufferView> >, WTF::RefPtr<JSC::ArrayBuffer, WTF::RawPtrTraits<JSC::ArrayBuffer>, WTF::DefaultRefDerefTraits<JSC::ArrayBuffer> >, WTF::RefPtr<WebCore::DOMFormData, WTF::RawPtrTraits<WebCore::DOMFormData>, WTF::DefaultRefDerefTraits<WebCore::DOMFormData> >, WTF::RefPtr<WebCore::URLSearchParams, WTF::RawPtrTraits<WebCore::URLSearchParams>, WTF::DefaultRefDerefTraits<WebCore::URLSearchParams> >, WTF::RefPtr<WebCore::ReadableStream, WTF::RawPtrTraits<WebCore::ReadableStream>, WTF::DefaultRefDerefTraits<WebCore::ReadableStream> >, WTF::String> >&&) (navigator=
    ..., document=..., url=..., body=...) at ../Source/WebCore/Modules/beacon/NavigatorBeacon.cpp:164
#10 0x00007f8392948c88 in WebCore::jsNavigatorPrototypeFunction_sendBeaconBody
    (castedThis=<optimized out>, callFrame=<optimized out>, lexicalGlobalObject=0x7f83892c2068)
    at WebCore/DerivedSources/JSNavigator.cpp:947
#11 WebCore::IDLOperation<WebCore::JSNavigator>::call<WebCore::jsNavigatorPrototypeFunction_sendBeaconBody>
    (operationName=0x7f839442b5ee "sendBeacon", callFrame=..., lexicalGlobalObject=...)
    at ../Source/WebCore/bindings/js/JSDOMOperation.h:55
#12 WebCore::jsNavigatorPrototypeFunction_sendBeacon(JSC::JSGlobalObject*, JSC::CallFrame*)
    (lexicalGlobalObject=0x7f83892c2068, callFrame=<optimized out>) at WebCore/DerivedSources/JSNavigator.cpp:952
#13 0x00007f837bffebd8 in  ()
#14 0x00007ffde3fb4040 in  ()
#15 0x00007f838fe118bd in llint_op_call ()
    at /usr/lib/debug/source/sdk/webkitgtk.bst/Source/JavaScriptCore/llint/LowLevelInterpreter.asm:1097
#16 0x0000000000000000 in  ()

[Michael Catanzaro]

Created attachment 431552 [details]
bt full

[Michael Catanzaro]

Created attachment 431553 [details]
thread apply all bt

[Michael Catanzaro]

I notice that my adblock .filterinfo under ~/.var/app/org.gnome.Epiphany.Devel/cache/epiphany/adblock was last modified at 09∶50∶16 AM AM today. At 09:50:20 CDT, my web processes entered their seemingly-daily crash loop.

Adrian also notes:

#define ADBLOCK_FILTER_UPDATE_FREQUENCY 24 * 60 * 60 /* In seconds */

It doesn't seem to be *quite* that regular for me, though.

[Michael Catanzaro]

I adjusted the filter update frequency from 24 hours to 24 seconds and build an Epiphany flatpak locally using the GNOME master runtime, to ensure an environment as close to Tech Preview as possible. Unfortunately it does not crash or misbehave when updating the adblock filters.

However, this crash really is still affecting Tech Preview, and it definitely happens at the same time as the daily adblock filter update.

[Michael Catanzaro]

Got a gargantuan amount of complaints from valgrind. I'm going to attach only three (it is too much, and three is enough to go on).

[Michael Catanzaro]

Created attachment 441432 [details]
Some valgrind hints