Bug 260962 – Network process crash in WebKit::CacheStorageDiskStore::readRecords

• Status: NEW
• Component: WebCore Misc.

[Michael Catanzaro]

Created attachment 467506 [details]
Full backtrace

I've just seen this network process crash twice in the past hour. The short backtrace is pasted below. Notably, we tried to memcpy 18446744073709551615 (2^64-1) bytes, which is not good.

Program terminated with signal SIGBUS, Bus error.
#0  memcpy (__src=0x7f36cc554299, __len=8, __dest=<optimized out>)
    at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:29
29	  return __builtin___memcpy_chk (__dest, __src, __len,
[Current thread is 1 (Thread 0x7f366bfff6c0 (LWP 31))]
(gdb) bt
#0  memcpy (__src=0x7f36cc554299, __len=8, __dest=<optimized out>)
    at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:29
#1  WTF::Persistence::Decoder::decodeNumber<unsigned long>(std::optional<unsigned long>&)
    (this=0x7f366bffdcd0, optional=<optimized out>)
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/persistence/PersistentDecoder.cpp:85
#2  WTF::Persistence::Decoder::operator>>(std::optional<unsigned long>&)
    (this=0x7f366bffdcd0, result=std::optional<unsigned long> [no contained value])
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/persistence/PersistentDecoder.cpp:120
#3  0x00007f36cf806865 in WTF::Persistence::Decoder::operator>><WebCore::FetchOptionsDestination, (void*)0>(std::optional<WebCore::FetchOptionsDestination>&) (this=0x7f366bffdcd0, result=<optimized out>)
    at WTF/Headers/wtf/persistence/PersistentDecoder.h:75
#4  WebCore::FetchOptions::decodePersistent<WTF::Persistence::Decoder>(WTF::Persistence::Decoder&, WebCore::FetchOptions&) (decoder=..., options=...) at WebCore/PrivateHeaders/WebCore/FetchOptions.h:215
#5  0x00007f36cf854351 in WebKit::decodeRecordHeader(std::span<unsigned char const, 18446744073709551615ul>)
    (headerData=Python Exception <class 'gdb.error'>: value has been optimized out
std::span of length 1967)
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebKit/NetworkProcess/storage/CacheStorageDiskStore.cpp:234
#6  WebKit::readRecordInfoFromFileData(std::array<unsigned char, 8ul> const&, std::span<unsigned char const, 18446744073709551615ul>) (salt=..., fileData=Python Exception <class 'gdb.error'>: value has been optimized out
)
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebKit/NetworkProcess/storage/CacheStorageDiskStore.cpp:294
#7  0x00007f36cf8536a7 in WebKit::CacheStorageDiskStore::readRecordFromFileData(std::span<unsigned char const, 18446744073709551615ul>, WTF::FileSystemImpl::MappedFileData&&)
    (this=<optimized out>, buffer=std::span of length 1528 = {...}, blobBuffer=...)
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebKit/NetworkProcess/storage/CacheStorageDiskStore.cpp:305
#8  0x00007f36cf8627e6 in WebKit::CacheStorageDiskStore::readRecords(WTF::Vector<WebKit::CacheStorageRecordInformation, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc> const&, WTF::CompletionHandler<void (WTF::Vector<std::optional<WebKit::CacheStorageRecord>, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&)>&&)::$_1::operator()<WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc> >(WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>)
    (this=0x7f36bd0b91a8, fileDatas=WTF::Vector of length 0, capacity 0, blobDatas=WTF::Vector of length 0, capacity 0) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebKit/NetworkProcess/storage/CacheStorageDiskStore.cpp:412
#9  WTF::Detail::CallableWrapper<WebKit::CacheStorageDiskStore::readRecords(WTF::Vector<WebKit::CacheStorageRecordInformation, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc> const&, WTF::CompletionHandler<void (WTF::Vector<std::optional<WebKit::CacheStorageRecord>, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&)>&&)::$_1, void, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&>::call(WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&) (this=0x7f36bd0b91a0, in=<optimized out>, in=<optimized out>)
    at WTF/Headers/wtf/Function.h:53
#10 0x00007f36cf862547 in WTF::Function<void (WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&)>::operator()(WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&) const (in=..., in=..., this=<optimized out>) at WTF/Headers/wtf/Function.h:82
#11 WTF::CompletionHandler<void (WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&)>::operator()(WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&)
    (this=<optimized out>, in=..., in=...) at WTF/Headers/wtf/CompletionHandler.h:75
#12 WebKit::CacheStorageDiskStore::readRecordsInternal(WTF::Vector<WTF::String, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::CompletionHandler<void (WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow--Type <RET> for more, q to quit, c to continue without paging--c
, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&)>&&)::$_0::operator()()::{lambda()#1}::operator()() (this=<optimized out>)
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebKit/NetworkProcess/storage/CacheStorageDiskStore.cpp:395
#13 WTF::Detail::CallableWrapper<WebKit::CacheStorageDiskStore::readRecordsInternal(WTF::Vector<WTF::String, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::CompletionHandler<void (WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&)>&&)::$_0::operator()()::{lambda()#1}, void>::call() (this=<optimized out>)
    at WTF/Headers/wtf/Function.h:53
#14 0x00007f36ce2f834b in WTF::Function<void ()>::operator()() const (this=<optimized out>)
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/Function.h:82
#15 WTF::RunLoop::performWork() (this=0x7f36bd03c180)
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/RunLoop.cpp:147
#16 0x00007f36ce3520fd in WTF::RunLoop::RunLoop()::$_0::operator()(void*) const (userData=0x7f366bffdcd0, 
    userData@entry=0x7f36bd03c180, this=<optimized out>)
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:80
#17 WTF::RunLoop::RunLoop()::$_0::__invoke(void*) (userData=0x7f366bffdcd0)
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:79
#18 0x00007f36ce351501 in WTF::RunLoop::$_0::operator()(_GSource*, int (*)(void*), void*) const
    (source=0x7f3660000dc0, callback=0x7f36ce3520f0 <WTF::RunLoop::RunLoop()::$_0::__invoke(void*)>, userData=0x7f36bd03c180, this=<optimized out>) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:53
#19 WTF::RunLoop::$_0::__invoke(_GSource*, int (*)(void*), void*)
    (source=0x7f3660000dc0, callback=0x7f36ce3520f0 <WTF::RunLoop::RunLoop()::$_0::__invoke(void*)>, userData=0x7f36bd03c180) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:45
#20 0x00007f36ca8b6c97 in g_main_dispatch (context=context@entry=0x7f3660000b70) at ../glib/gmain.c:3476
#21 0x00007f36ca8b8da7 in g_main_context_dispatch_unlocked (context=0x7f3660000b70) at ../glib/gmain.c:4284
#22 g_main_context_iterate_unlocked
    (context=0x7f3660000b70, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>)
    at ../glib/gmain.c:4349
#23 0x00007f36ca8b9757 in g_main_loop_run (loop=0x7f3660000da0) at ../glib/gmain.c:4551
#24 0x00007f36ce351ad1 in WTF::RunLoop::run() ()
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:108
#25 0x00007f36ce2fb9a7 in WTF::Function<void ()>::operator()() const (this=<optimized out>)
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/Function.h:82
#26 WTF::Thread::entryPoint(WTF::Thread::NewThreadContext*) (newThreadContext=0x7f36bd0340e0)
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/Threading.cpp:250
#27 0x00007f36ce35582d in WTF::wtfThreadEntryPoint(void*) (context=0x7f366bffdcd0)
    at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/posix/ThreadingPOSIX.cpp:242
#28 0x00007f36cea8ee39 in start_thread (arg=<optimized out>) at pthread_create.c:444
#29 0x00007f36ceb16cc4 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

[Michael Catanzaro]

For fun, 18446744073709551615 bytes is roughly 16,777,216 TiB. I don't have that much RAM. :(

[Michael Catanzaro]

> Notably, we tried to memcpy 18446744073709551615 (2^64-1) bytes, which is not good.

Well, actually, that's what WebKit thinks is the size of the record (see frame 5), but the memcpy is only __len=8. Oops, sorry.

[Michael Catanzaro]

Hm, I just hit a third network process crash today, plus one more yesterday, which was the first time I ever hit this crash. But there has been no WebKitGTK update since August 19 (2.41.91). That is suspicious; I would normally expect a crash that occurs with such high frequency to be more clearly associated with a particular update.

All three crashes today occurred when using https://discourse.gnome.org/

[Michael Catanzaro]

I hit two more network process crashes since my previous comment, again both on discourse.gnome.org, so this is happening often enough that I would *expect* to eventually figure out how to reproduce it. The crash seems to occur when writing comments, but so far it only happens when I'm not trying to reproduce and never when I am. If I could reproduce, then I can add prints to verify where things are going wrong....

[Radar WebKit Bug Importer]

<rdar://problem/115113736>

[Michael Catanzaro]

Hit again today. I somehow got a better backtrace this time and unfortunately it is a buffer overread. In frame 6, we have:

(gdb) print fileDatas
$2 = WTF::Vector of length 0, capacity 0
(gdb) print blobDatas
$3 = WTF::Vector of length 0, capacity 0

So recordInfos is also required to be 0, but:

(gdb) print recordInfos
$1 = WTF::Vector of length 1, capacity 1 = {{key = {m_partition = "record", 
      m_type = "c13dab6b-1248-4ed8-818a-66e7d5c489f4", m_identifier = "1e1f03fa-d804-45c7-b65f-ee68cd9a0421", 
      m_range = "(null)", m_hash = {_M_elems = ",H\023\214~\362\230\027\216Έ\256LH}\347\357/\214D"}, 
      m_partitionHash = {_M_elems = "\"%7H\237\262Y\343\"(&1\201\025\366\366\3776\\\230"}}, 
    insertionTime = 23529902.664000001, identifier = 43, updateResponseCounter = 0, size = 90, url = {
      m_string = "https://globalnews.ca/wp-content/themes/shaw-globalnews/assets/dist/images/pattern-greystripe.png", m_isValid = 1, m_protocolIsInHTTPFamily = 1, m_hasOpaquePath = 0, m_portLength = 0, static maxPortLength = 7, 
      static maxSchemeLength = 67108863, m_schemeEnd = 5, m_userStart = 8, m_userEnd = 8, m_passwordEnd = 8, 
      m_hostEnd = 21, m_pathAfterLastSlash = 75, m_pathEnd = 97, m_queryEnd = 97}, hasVaryStar = false, 
    varyHeaders = {m_impl = {static smallMaxLoadNumerator = 3, static smallMaxLoadDenominator = 4, 
        static largeMaxLoadNumerator = 1, static largeMaxLoadDenominator = 2, static maxSmallTableCapacity = 1024, 
        static minLoad = 6, static tableSizeOffset = -1, static tableSizeMaskOffset = -2, 
        static keyCountOffset = -3, static deletedCountOffset = -4, static metadataSize = 16, {m_table = 0x0, 
          m_tableForLLDB = 0x0}}}}}

There are assertions at the top of the function that would catch this in debug builds:

ASSERT(recordInfos.size() == fileDatas.size());
ASSERT(recordInfos.size() == blobDatas.size());

but they are unfortunately not enabled in release builds.

I'm not sure how this happened. Looking in CacheStorageDiskStore::readRecordsInternal, it really looks like the sizes should always match.

[Michael Catanzaro]

I just hit this crash 10 times in a row with WebKitGTK 2.43.3. i.e. the network process would restart itself, then immediately crash again. This time it was caused by one of the product pages under https://www.corsair.com/us/en/c/psu though I'm not sure exactly which one. After 10 network process crashes, the UI process crashed, and after restarting the crashes no longer occur.