Bug 277400 – [WASM] WASM python crashes on JSC sometimes when run JSC with JIT(BBQ, OMG).

• Status: NEW
• Component: JavaScriptCore

[Song Juhyun]

Created attachment 472023 [details]
Even though same execution command, somtimes crash and sometimes run as well.

I am a student currently working on building a differential fuzzing harness (https://github.com/UsQuake/wasi_pyrb_diff_test) capable of executing WASM (WASI) Python/Ruby in the debug shell of browser JS engines.

During my fuzzing tests, I observed that the JSC shell occasionally crashes when using the default options --useOMGJIT=1 --useBBQJIT=1.
The crashes occur at least once every 100 test cycles, and the crash messages indicate out-of-memory access problems.

However, when I use the options --useOMGJIT=0 --useBBQJIT=0 (which only utilizes WASM-LLInt), the JSC shell does not crash during 100 test cycles.

Furthermore, I did not encounter any crashes when I executed the same WASM and JavaScript polyfill code in V8 or SpiderMonkey.

Based on these findings, I suspect that there may be a bug in the WASM JIT for JSC, such as OMG-JIT or BBQ-JIT.

To address this issue, I am reporting this issue to the community for insights or suggestions on how to solve or debug this problem.
Even a small opinion would be greatly appreciated. Thank you!

[Radar WebKit Bug Importer]

<rdar://problem/132910348>

[Radar WebKit Bug Importer]

<rdar://problem/132910866>

[Mark Lam]

Please provide:
1. a reproducible test case
2. details of the command line you use to invoke the jsc shell
3. details of what OS (and version) and platform you're running this on
4. details of how you got your jsc shell: did you build it yourself?  If you built it yourself, what commit of WebKit did you build it from, and how did you build it?

Please provide all the details.  Thanks.

[Mark Lam]

rdar://132910348

[Song Juhyun]

(In reply to Mark Lam from comment #3)
> Please provide:
> 1. a reproducible test case
> 2. details of the command line you use to invoke the jsc shell
> 3. details of what OS (and version) and platform you're running this on
> 4. details of how you got your jsc shell: did you build it yourself?  If you
> built it yourself, what commit of WebKit did you build it from, and how did
> you build it?
> 
> Please provide all the details.  Thanks.

Umm.... How about using a docker?
I think sending you docker image(that I built for test python wasm in JavascriptCore) can solved all of the above questions.
(https://github.com/UsQuake/wasi_pyrb_diff_test/blob/master/sandbox_imgs/jsc_python_wasi/Dockerfile).

I understand that you got a lot of curious about what I did, That's reason why I used docker image and container to do test.
  
But If you need, I can still answer to your all above questions instead of sending docker image to execute wasm python interpreter on JavascriptCore.
Even though I should tell you a lot of details of my experiment.

[Song Juhyun]

I really appreciate your replication efforts to this bug.

[Song Juhyun]

1. It happens flasky to all the testcase python file.

2. I used command that "./jsc module-ready-wasi-py.js -- none"
or  "./jsc module-ready-wasi-py.js --useBBQJIT=1 --useOMGJIT=1 -- none"

2-1.module-ready-wasi-py.js is polyfill codes that includes async threadpool, wasi polyfill and mapped files of python/ruby interpreter's dependencies.

2-2. --useBBQJIT=1, --useOMGJIT=1 option means that I want to use WASM-JIT features(as you know). But It's default options so, you don't need to specify like this.

2-3. "-- none" is dummy command line. To use argument object, inside of polyfill javascript code.

3. ubuntu 22.04.

4. WebKit, JSCOnly build  release version. Sorry. I cannot specify commit hash information disappeared.

If you need more information to replicate the case, ask me more or request more thing you need. I really sorry for lost  commit hash information. I'll test in latest committed WebKit and share you commit hash.

[Mark Lam]

Since you're running ubuntu 22.04, I presume you're running on an x86_64 machine?  Just confirming.

[David Degazio]

Do you think you could attach the generated module-ready-wasi-py.js (and any other JS/WASM sources it depends on) to this bug?

[Song Juhyun]

(In reply to Mark Lam from comment #8)
> Since you're running ubuntu 22.04, I presume you're running on an x86_64
> machine?  Just confirming.

yes. exactly!

[Song Juhyun]

(In reply to David Degazio from comment #9)
> Do you think you could attach the generated module-ready-wasi-py.js (and any
> other JS/WASM sources it depends on) to this bug?

No. If it was the problem, I guess it should have crashed in part of javascript and it should have crashed even though I turn off the JIT feature of WASM(BBQ, OMG).
 But all of the crashes that I saw were related to WASM. The thing that module-ready-wasi-py.js does is just replace WebAPI and map HTTP request of module files into real file-system access function which is supported to debug shell of all the browser javascript engines like readbuffer("/path/to/file"), read("/path/to/file").. etc.

[Song Juhyun]

(In reply to Song Juhyun from comment #10)
> (In reply to Mark Lam from comment #8)
> > Since you're running ubuntu 22.04, I presume you're running on an x86_64
> > machine?  Just confirming.
> 
> yes. exactly!

commit hash of webkit: d1264f8d514b7edbd8bccbdb56b3ff247974cd9c

[Yusuke Suzuki]

Can you upload the generated module-ready-wasi-py.js for easy testing?

[Song Juhyun]

Created attachment 472164 [details]
module-ready-wasi-py.js, needed modules, python.wasm and testcase.py

Here is the module-ready-wasi-py.js, needed modules, python.wasm and testcase.py. 
You should extract the tar file and copy paste a debug shell(jsc) into extracted folder(which name is py-wasi-sandbox). 
command is
(presume you already opened the ./py-wasi-sandbox)
"./jsc module-ready-wasi-py.js -- none"
Then, you could see the debug shell crashes flasky.

[Yusuke Suzuki]

(In reply to Song Juhyun from comment #14)
> Created attachment 472164 [details]
> module-ready-wasi-py.js, needed modules, python.wasm and testcase.py
> 
> Here is the module-ready-wasi-py.js, needed modules, python.wasm and
> testcase.py. 
> You should extract the tar file and copy paste a debug shell(jsc) into
> extracted folder(which name is py-wasi-sandbox). 
> command is
> (presume you already opened the ./py-wasi-sandbox)
> "./jsc module-ready-wasi-py.js -- none"
> Then, you could see the debug shell crashes flasky.

I ran the above code, however I got different results (but this is consistent on ARM64 macOS / x64 macOS). Is it expected?

$ VM=~/dev/OpenSource/WebKitBuild/Release; DYLD_FRAMEWORK_PATH=$VM $VM/jsc module-ready-wasi-py.js -- none
171
-3.0
[6, 6, 259, 78856]
11.3
[4, 8, 255, 471, 5333]
11.3
[4, 8, 255, 471, 5333]
[4, 8, 255, 471, 5333]
[4, 8, 255, 471, 5333]

[Song Juhyun]

Yes. absolutely. You successfully replicate my case just once.
Did you ran a 50 times? 
During 50 times run, 
It sometimes crashed and sometimes ran clearly like your case.
"Flasky" means sometimes run clearly and sometime crashes.

[Yusuke Suzuki]

(In reply to Song Juhyun from comment #16)
> Yes. absolutely. You successfully replicate my case just once.
> Did you ran a 50 times? 
> During 50 times run, 
> It sometimes crashed and sometimes ran clearly like your case.
> "Flasky" means sometimes run clearly and sometime crashes.

So far, I ran 20 times, but I'm always observing the above results (so not flaky). Is it expected?

[Song Juhyun]

No, It's not expected result.
As you said, I think it's not flasky case(in mac).

I ran case in x86_64(ubuntu 24.04).
So, I think it has a no problem in macos.

Do you have docker with a rosetta2(which is supports arm<->x86_64 conversion)?
I guess if it only crashes in the docker container(ubuntu 24.04), it only occurs in linux(or debian only).

[Yusuke Suzuki]

I ran 100 times on x64 macOS, but I'm seeing consistent results. So this looks like Linux only issue.

[Song Juhyun]

I seriously thanks for your replication :)
If the case works correctly on mac, then I guess it's not serious problem.

Then, Can you share commit hash of WebKit which you used for test?
I think I should figure which is the problem(OS or difference of commit).

[Yusuke Suzuki]

(In reply to Song Juhyun from comment #20)
> I seriously thanks for your replication :)
> If the case works correctly on mac, then I guess it's not serious problem.
> 
> Then, Can you share commit hash of WebKit which you used for test?
> I think I should figure which is the problem(OS or difference of commit).

Thanks! This is current ToT, 63415d14935cfbfc7e77866707034882c7b9a162.
Also, @Justin is looking into it to check whether it can reproduce on Linux machine.

[Song Juhyun]

With the given commit hash(release build of debug shell),
It still crashes flaky. 
So, I guess commit hash difference is not the problem. 
It's just crashes on linux environment.

[Song Juhyun]

(In reply to Yusuke Suzuki from comment #21)
> Thanks! This is current ToT, 63415d14935cfbfc7e77866707034882c7b9a162.
> Also, @Justin is looking into it to check whether it can reproduce on Linux
> machine.

Sorry, I think ToT df419b7ac5fae718ea1dce64c347a87c9ca8e849 is the problem.
Can you test it in mac..?

[Mark Lam]

df419b7ac5fae718ea1dce64c347a87c9ca8e849 is 282335@main, which landed on Aug 15, 2024.

Using a Release build of 1eba2ca13db0a4949ad07a1053cb29a5152bbbb0 which is 287329@main, more recently landed on Dec 3, 2024 ... and running the attached repro test case 100 times, I was not able to get a single crash.

This was tested on Mac.  Either this is a Linux specific issue or it is already fixed.