Bug 295780 – [JSC] Reproducible JSC segfault at allocateMoreOutOfLineStorage on 32-bit ARM

• Status: NEW
• Component: JavaScriptCore

We're using WPEWebKit 2.46.7 via cog 0.18.4 on an embedded i.MX6DL board with wayland backend, provided by yocto scarthgap.

The problem is that occasionally there's the following:

    (cog:924): Cog-Core-WARNING **: 15:21:59.504: Renderer process terminated, restarting (attempt 1/5)

    Thread 1 "WPEWebProcess" received signal SIGSEGV, Segmentation fault.
    0xb42ed044 in JSC::JSObject::allocateMoreOutOfLineStorage(JSC::VM&, unsigned int, unsigned int) ()
    from /usr/lib/libWPEWebKit-2.0.so.1
    0 0xb42ed044 in JSC::JSObject::allocateMoreOutOfLineStorage(JSC::VM&, unsigned int, unsigned int) ()
    from /usr/lib/libWPEWebKit-2.0.so.1
    1 0xb3e05136 in WTF::ASCIILiteral JSC::JSObject::putDirectInternal<(JSC::JSObject::PutMode)0>(JSC::VM&, JSC::PropertyName, JSC::JSValue, unsigned int, JSC::PutPropertySlot&) () from /usr/lib/libWPEWebKit-2.0.so.1
    2 0xb405f2ec in JSC::putByVal(JSC::JSGlobalObject*, JSC::JSValue, JSC::JSValue, JSC::JSValue, JSC::ArrayProfile*, JSC::ECMAMode)
    () from /usr/lib/libWPEWebKit-2.0.so.1
    3 0xb4060760 in operationPutByValSloppyGeneric () from /usr/lib/libWPEWebKit-2.0.so.1
    4 0xac8107cc in ?? ()
    Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Happens in a very specific scenario that involves switching between heavy HTML/JS pages, quite infrequently.

However, the same fault (similar stack trace) is easily reproducible by just opening https://html5gamepad.com/ and waiting.

With very high probability it crashes within 7 minutes, sometimes also hitting OOM (5-10% of the crashes).

useDFGJIT=0 makes it crash much faster

JSC_useJIT=0 makes it crash much slower

If not fixing it, any advice to toggle some settings to make it crash even less frequently is appreciated.

[Alexey Proskuryakov]

> However, the same fault (similar stack trace) is easily reproducible by just opening https://html5gamepad.com/ and waiting.

Is this website functional currently? It just redirects me to a different one.

This failure looks like running out of memory possibly. Can you please watch process memory use before it crashes? I'm not seeing memory growth in Safari on the redirected website.

Yes, the website is correct and is functioning. I found it in an older bug, and luckily it worked. Not sure what's so special about it though.

The bug is apparently related to memory management, but it's not simply running out of memory. OOM condition (when kernel kills the browser) is reached very infrequently, most of the time it crashes much sooner.

The bug might be specific to i.MX, no wonder Safari works fine -- would be quite wild for it to crash on a valid website.

Using debug build (DEBUG_BUILD = "1" in yocto) makes it last 3-4 times longer until it crashes, and OOM is hit in the majority (but not all) of cases.

[Alexey Proskuryakov]

FWIW, it still redirects me to https://hardwaretester.com/gamepad

(In reply to Alexey Proskuryakov from comment #4)
> FWIW, it still redirects me to https://hardwaretester.com/gamepad

Either URL works for reproducing (on i.MX6, doubt it would reproduce with Safari), I should have probably specified https://hardwaretester.com/gamepad from the start to avoid confusion.

[Wouter Vanhauwaert]

Hitting same on imx6q, rdk-vivante backend (no cog)

Core was generated by `/usr/libexec/wpe-webkit-1.1/WPEWebProcess 23 27 31'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x7419c8ec in JSC::JSObject::allocateMoreOutOfLineStorage(JSC::VM&, unsigned int, unsigned int) () from ./usr/lib/libWPEWebKit-1.1.so.0
[Current thread is 1 (LWP 700)]
(gdb) bt
#0  0x7419c8ec in JSC::JSObject::allocateMoreOutOfLineStorage(JSC::VM&, unsigned int, unsigned int) () from ./usr/lib/libWPEWebKit-1.1.so.0
#1  0x742741c6 in JSC::LiteralParser<unsigned char, (JSC::JSONReviverMode)0>::parseRecursively(JSC::VM&, unsigned char*) () from ./usr/lib/libWPEWebKit-1.1.so.0
#2  0x7427093c in JSC::LiteralParser<unsigned char, (JSC::JSONReviverMode)0>::parseRecursively(JSC::VM&, unsigned char*) () from ./usr/lib/libWPEWebKit-1.1.so.0
#3  0x7427093c in JSC::LiteralParser<unsigned char, (JSC::JSONReviverMode)0>::parseRecursively(JSC::VM&, unsigned char*) () from ./usr/lib/libWPEWebKit-1.1.so.0
#4  0x7426fc60 in JSC::LiteralParser<unsigned char, (JSC::JSONReviverMode)0>::parseRecursively(JSC::VM&, unsigned char*) () from ./usr/lib/libWPEWebKit-1.1.so.0
#5  0x7427093c in JSC::LiteralParser<unsigned char, (JSC::JSONReviverMode)0>::parseRecursively(JSC::VM&, unsigned char*) () from ./usr/lib/libWPEWebKit-1.1.so.0
#6  0x7427093c in JSC::LiteralParser<unsigned char, (JSC::JSONReviverMode)0>::parseRecursively(JSC::VM&, unsigned char*) () from ./usr/lib/libWPEWebKit-1.1.so.0
#7  0x7427a0a0 in JSC::LiteralParser<unsigned char, (JSC::JSONReviverMode)0>::parseRecursivelyEntry(JSC::VM&) () from ./usr/lib/libWPEWebKit-1.1.so.0
#8  0x74192484 in JSC::jsonProtoFuncParse(JSC::JSGlobalObject*, JSC::CallFrame*) () from ./usr/lib/libWPEWebKit-1.1.so.0
#9  0x6c2ff148 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) 

And hit it last year on imx53 too (cog):

https://bugs.webkit.org/show_bug.cgi?id=275099

Both running our same webapp

[Adrian Perez]

*** Bug 275099 has been marked as a duplicate of this bug. ***

[Adrian Perez]

While I do not currently have an idea about how to go about fixing this, I wonder if the higher amount of JSC operations needed to manage a constant stream of gamepad inputs (or periodic WebSocket data as reported in bug #275099) might be making a latent issue easier to reproduce 🤔

[Adrian Perez]

Potentially related to bug #279285 -- that one shows a similar backtrace but on RISC-V, and also mentions that there is no such issue when building with Clang.

I've built wpewebkit with Clang by adding
TOOLCHAIN = “clang”
to wpewebkit recipe (haven't changed TC_CXX_RUNTIME or LIBCPLUSPLUS) and it didn't help (log.do_compile indeed shows that clang was used).
It ran into out of memory condition most of the time, which is similar behavior to GCC debug build. FWIW, I think it ran into OOM faster than when built with GCC.

[Radar WebKit Bug Importer]

<rdar://problem/161319742>

FWIW, wpewebkit is also crashing on https://browserbench.org/Speedometer3.1/, haven't checked the stack trace though.

Chromium passes it fine (just saying).

[Wouter Vanhauwaert]

The memory on epiphany (webkitgtk/2.48.5) is even going through the roof on that page...

[Carlos Alberto Lopez Perez]

How much RAM does have the test device?

1 GB, however in the original test case it rarely hit out-of-memory condition, most of the time it crashed before.

[Wouter Vanhauwaert]

I did some further investigation by narrowing down webkit versions
webkit 2.38.6 does not run into the issue
webkit 2.40.5 does

[Wouter Vanhauwaert]

After some further investigation:
2.40.4 was still ok, so issue introduced leading towards 2.40.5
I git bisected in between them and turned out commit 
https://github.com/WebKit/WebKit/commit/d784299e55fa84c267ca35ed257ef08a500571f7
is introducing the malicious behavior.
I'm not really into the internals of what's really happening. My understanding that a choice is added to go for a fast or slow path...

[Wouter Vanhauwaert]

(In reply to Wouter Vanhauwaert from comment #17)
> After some further investigation:
> 2.40.4 was still ok, so issue introduced leading towards 2.40.5
> I git bisected in between them and turned out commit 
> https://github.com/WebKit/WebKit/commit/
> d784299e55fa84c267ca35ed257ef08a500571f7
> is introducing the malicious behavior.
> I'm not really into the internals of what's really happening. My
> understanding that a choice is added to go for a fast or slow path...

Sorry. It did crash in the end, only took a lot longer then I anticipated. +1h30 in my tests where it's mostly +- 30min on average. Now I doubt my git bisect in its whole. Will investigate further and come back to it

FWIW, I don't think that commit has ever been picked into specifically WPEWebKit, at least it doesn't show up in git log it seems.

[Michael Catanzaro]

That's a stable branch commit. On the main branch it landed as 266439@main.

[Michael Catanzaro]

Ah, looks like the main branch commit excludes all the code changes. The problem must have been resolved differently on main.