Bug 305926
| Summary: | [WPE][ARM 32-bits]:REGRESSION(305815@main-305896@main): All subtests of JetStream2 timeout and WebProcess uses 100% of CPU. | ||
|---|---|---|---|
| Product: | WebKit | Reporter: | Carlos Alberto Lopez Perez <clopez> |
| Component: | WPE WebKit | Assignee: | Nobody <webkit-unassigned> |
| Status: | RESOLVED DUPLICATE | ||
| Severity: | Normal | CC: | angelos, annevk, bugs-noreply, jmichaud |
| Priority: | P2 | ||
| Version: | WebKit Nightly Build | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| See Also: |
https://bugs.webkit.org/show_bug.cgi?id=305723 https://bugs.webkit.org/show_bug.cgi?id=306062 |
||
Carlos Alberto Lopez Perez
This has been detected in the WPE RPi4 perf bots for ARM32.
* Last good run at 305815@main https://build.webkit.org/#/builders/892/builds/18952
* First bad run at 305896@main https://build.webkit.org/#/builders/892/builds/18952
As you can see in the bad run the test step got stuck running JetStream2. It is taking 19 hours and counting. Previously it "only" took 9 hours to run everything.
Previously only a few of the JetStream2 test were giving timeotus
$ curl -s https://build.webkit.org/api/v2/logs/47895223/raw_inline | grep Timeout
2026-01-19 14:09:45,163 - ERROR - subtest OfflineAssembler of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-19 16:47:27,587 - ERROR - subtest hello_world-LJF of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-19 17:20:46,921 - ERROR - subtest list_search-LJF of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-19 17:54:06,208 - ERROR - subtest lists-LJF of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-19 18:27:25,527 - ERROR - subtest string_lists-LJF of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
But now **all** of the jetstream2 subtests are timing out
$ curl -s https://build.webkit.org/api/v2/logs/47918864/raw_inline | grep Timeout
2026-01-20 13:44:17,613 - ERROR - subtest Air of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 14:17:36,926 - ERROR - subtest Basic of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 14:50:56,198 - ERROR - subtest ML of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 15:24:15,462 - ERROR - subtest Babylon of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 15:57:34,622 - ERROR - subtest cdjs of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 16:30:53,807 - ERROR - subtest first-inspector-code-load of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 17:04:13,009 - ERROR - subtest multi-inspector-code-load of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 17:37:32,276 - ERROR - subtest Box2D of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 18:10:51,491 - ERROR - subtest octane-code-load of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 18:44:10,683 - ERROR - subtest crypto of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 19:17:29,911 - ERROR - subtest delta-blue of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 19:50:49,121 - ERROR - subtest earley-boyer of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 20:24:08,363 - ERROR - subtest gbemu of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 20:57:27,531 - ERROR - subtest mandreel of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 21:30:46,756 - ERROR - subtest navier-stokes of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 22:04:05,981 - ERROR - subtest pdfjs of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 22:37:25,198 - ERROR - subtest raytrace of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 23:10:44,549 - ERROR - subtest regexp of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 23:44:03,903 - ERROR - subtest richards of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 00:17:23,152 - ERROR - subtest splay of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 00:50:42,340 - ERROR - subtest typescript of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 01:24:01,615 - ERROR - subtest octane-zlib of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 01:57:20,910 - ERROR - subtest FlightPlanner of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 02:30:40,216 - ERROR - subtest OfflineAssembler of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 03:03:59,434 - ERROR - subtest UniPoker of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 03:37:18,709 - ERROR - subtest async-fs of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 04:10:37,911 - ERROR - subtest float-mm.c of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 04:43:57,147 - ERROR - subtest hash-map of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 05:17:16,523 - ERROR - subtest ai-astar of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 05:50:35,707 - ERROR - subtest gaussian-blur of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 06:23:54,912 - ERROR - subtest stanford-crypto-aes of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 06:57:14,047 - ERROR - subtest stanford-crypto-pbkdf2 of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 07:30:33,361 - ERROR - subtest stanford-crypto-sha256 of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 08:03:52,585 - ERROR - subtest json-stringify-inspector of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 08:37:11,861 - ERROR - subtest json-parse-inspector of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
[...still suite test-run unfinished as of writing this...]
I entered into the board and I can see WPEWebProcess using 100% of the CPU, 2% of the RAM (4GB total).
This are the commits related to JSC in the range
$ git log --oneline 3a8fc06e2e25f38b2051ca2dd6c4a78bec5756c6..0b1706ea5580e9617b324814f95a476ad2d31bf1 Source/JavaScriptCore/
d21256503ee6 [JSC] `RegExp.prototype [ %Symbol.split% ]` Incorrectly Uses Fast Path when `limit` is not a Number https://bugs.webkit.org/show_bug.cgi?id=305192
b932c71d5e15 Address some unsafe cast warning in Internals.cpp https://bugs.webkit.org/show_bug.cgi?id=305808
bbf8d344a5bb [JSC] Use `hasSpecialProperties` bit for cheking `constructor` properties https://bugs.webkit.org/show_bug.cgi?id=305801
c0deda0b0cc5 [JSC] Handle `String.fromCharCode` in abstract interpreter https://bugs.webkit.org/show_bug.cgi?id=194308
38411ab91e01 [JSC] Optimize `[...set]` https://bugs.webkit.org/show_bug.cgi?id=305446
986543b9d0e8 [JSC] Handle `StringCodePointAt` in Abastract Interpreter https://bugs.webkit.org/show_bug.cgi?id=305629
4211cd058e41 [JSC][WASM][Debugger] Fix instance registration race and use InstanceAnchor for lifecycle management rdar://168389785 https://bugs.webkit.org/show_bug.cgi?id=305714
3e93d47f3f58 Allow implicit conversion from RefPtr<T> to T* https://bugs.webkit.org/show_bug.cgi?id=305754
| Attachments | ||
|---|---|---|
| Add attachment proposed patch, testcase, etc. |
Carlos Alberto Lopez Perez
This has been detected in the WPE RPi4 perf bots for ARM32.
* Last good run at 305815@main https://build.webkit.org/#/builders/892/builds/18952
* First bad run at 305896@main https://build.webkit.org/#/builders/892/builds/18960
As you can see in the bad run the test step got stuck running JetStream2. It is taking 19 hours and counting. Previously it "only" took 9 hours to run everything.
Previously only a few of the JetStream2 test were giving timeotus
$ curl -s https://build.webkit.org/api/v2/logs/47895223/raw_inline | grep Timeout
2026-01-19 14:09:45,163 - ERROR - subtest OfflineAssembler of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-19 16:47:27,587 - ERROR - subtest hello_world-LJF of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-19 17:20:46,921 - ERROR - subtest list_search-LJF of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-19 17:54:06,208 - ERROR - subtest lists-LJF of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-19 18:27:25,527 - ERROR - subtest string_lists-LJF of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
But now **all** of the jetstream2 subtests are timing out
$ curl -s https://build.webkit.org/api/v2/logs/47918864/raw_inline | grep Timeout
2026-01-20 13:44:17,613 - ERROR - subtest Air of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 14:17:36,926 - ERROR - subtest Basic of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 14:50:56,198 - ERROR - subtest ML of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 15:24:15,462 - ERROR - subtest Babylon of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 15:57:34,622 - ERROR - subtest cdjs of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 16:30:53,807 - ERROR - subtest first-inspector-code-load of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 17:04:13,009 - ERROR - subtest multi-inspector-code-load of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 17:37:32,276 - ERROR - subtest Box2D of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 18:10:51,491 - ERROR - subtest octane-code-load of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 18:44:10,683 - ERROR - subtest crypto of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 19:17:29,911 - ERROR - subtest delta-blue of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 19:50:49,121 - ERROR - subtest earley-boyer of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 20:24:08,363 - ERROR - subtest gbemu of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 20:57:27,531 - ERROR - subtest mandreel of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 21:30:46,756 - ERROR - subtest navier-stokes of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 22:04:05,981 - ERROR - subtest pdfjs of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 22:37:25,198 - ERROR - subtest raytrace of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 23:10:44,549 - ERROR - subtest regexp of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-20 23:44:03,903 - ERROR - subtest richards of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 00:17:23,152 - ERROR - subtest splay of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 00:50:42,340 - ERROR - subtest typescript of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 01:24:01,615 - ERROR - subtest octane-zlib of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 01:57:20,910 - ERROR - subtest FlightPlanner of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 02:30:40,216 - ERROR - subtest OfflineAssembler of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 03:03:59,434 - ERROR - subtest UniPoker of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 03:37:18,709 - ERROR - subtest async-fs of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 04:10:37,911 - ERROR - subtest float-mm.c of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 04:43:57,147 - ERROR - subtest hash-map of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 05:17:16,523 - ERROR - subtest ai-astar of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 05:50:35,707 - ERROR - subtest gaussian-blur of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 06:23:54,912 - ERROR - subtest stanford-crypto-aes of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 06:57:14,047 - ERROR - subtest stanford-crypto-pbkdf2 of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 07:30:33,361 - ERROR - subtest stanford-crypto-sha256 of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 08:03:52,585 - ERROR - subtest json-stringify-inspector of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
2026-01-21 08:37:11,861 - ERROR - subtest json-parse-inspector of benchmark plan jetstream2 failed with exception: Timeout alarm was triggered
[...still suite test-run unfinished as of writing this...]
I entered into the board and I can see WPEWebProcess using 100% of the CPU, 2% of the RAM (4GB total).
This are the commits related to JSC in the range
$ git log --oneline 3a8fc06e2e25f38b2051ca2dd6c4a78bec5756c6..0b1706ea5580e9617b324814f95a476ad2d31bf1 Source/JavaScriptCore/
d21256503ee6 [JSC] `RegExp.prototype [ %Symbol.split% ]` Incorrectly Uses Fast Path when `limit` is not a Number https://bugs.webkit.org/show_bug.cgi?id=305192
b932c71d5e15 Address some unsafe cast warning in Internals.cpp https://bugs.webkit.org/show_bug.cgi?id=305808
bbf8d344a5bb [JSC] Use `hasSpecialProperties` bit for cheking `constructor` properties https://bugs.webkit.org/show_bug.cgi?id=305801
c0deda0b0cc5 [JSC] Handle `String.fromCharCode` in abstract interpreter https://bugs.webkit.org/show_bug.cgi?id=194308
38411ab91e01 [JSC] Optimize `[...set]` https://bugs.webkit.org/show_bug.cgi?id=305446
986543b9d0e8 [JSC] Handle `StringCodePointAt` in Abastract Interpreter https://bugs.webkit.org/show_bug.cgi?id=305629
4211cd058e41 [JSC][WASM][Debugger] Fix instance registration race and use InstanceAnchor for lifecycle management rdar://168389785 https://bugs.webkit.org/show_bug.cgi?id=305714
3e93d47f3f58 Allow implicit conversion from RefPtr<T> to T* https://bugs.webkit.org/show_bug.cgi?id=305754
Carlos Alberto Lopez Perez
Angelos comments that the JSCOnly post-commit bot also appears to be extremely slow, timing out after 20h (normally takes less than half an hour)
We can see there this:
- First bad build at 305865@main (20 hours, timeout) https://build.webkit.org/#/builders/24/builds/60635
- Last bad build at 305856@main (19 minutes) https://build.webkit.org/#/builders/24/builds/60634
So that narrows the commit range to [305856@main-305865@main] which gives us 9 commits, but no one of them touches Source/JavaScriptCore (??)
$ git log --oneline b932c71d5e15603ac2cd43c7552d918ed161a64e..54b3bb5bb0669034734b7bf49a584693165945b7
54b3bb5bb066 Facebook video call web page is promoted to now playing info application rdar://168095616 https://bugs.webkit.org/show_bug.cgi?id=305469
fbbd9a28caa2 Convert HashMap<..., RefPtr to Ref in NetworkProcess https://bugs.webkit.org/show_bug.cgi?id=305723
89ad6512372c Reduce unsafeness in WebCore/css/query https://bugs.webkit.org/show_bug.cgi?id=304399 rdar://167168414
4b75ae7073f0 [WPE] Further cleanup after removal of the Qt5 embedding API https://bugs.webkit.org/show_bug.cgi?id=305602
1d511d6cf0ce Address some unsafe cast warnings in Source/WebCore/layout https://bugs.webkit.org/show_bug.cgi?id=305810
c5f94bb4b981 Address unsafe cast issues in Source/WebCore/bindings https://bugs.webkit.org/show_bug.cgi?id=305798
d21d3038fbce WPT service-worker/tentative/static-router/static-router-resource-timing.https.html is no longer flaky in cocoa bots rdar://168444917 https://bugs.webkit.org/show_bug.cgi?id=305766
e27ac61ee273 getUserMedia with constraints where max frameRate bigger than 16 does not work rdar://115713922 https://bugs.webkit.org/show_bug.cgi?id=261461
8123c8a002e5 Address some unsafe cast warnings in Source/WebCore/svg https://bugs.webkit.org/show_bug.cgi?id=305805
I'm puzzled.
Carlos Alberto Lopez Perez
Angelos comments that the JSCOnly post-commit bot also appears to be extremely slow, timing out after 20h (normally takes less than half an hour)
We can see there this:
- First bad build at 305865@main (20 hours, timeout) https://build.webkit.org/#/builders/24/builds/60635
- Last good build at 305856@main (19 minutes) https://build.webkit.org/#/builders/24/builds/60634
So that narrows the commit range to [305856@main-305865@main] which gives us 9 commits, but no one of them touches Source/JavaScriptCore (??)
$ git log --oneline b932c71d5e15603ac2cd43c7552d918ed161a64e..54b3bb5bb0669034734b7bf49a584693165945b7
54b3bb5bb066 Facebook video call web page is promoted to now playing info application rdar://168095616 https://bugs.webkit.org/show_bug.cgi?id=305469
fbbd9a28caa2 Convert HashMap<..., RefPtr to Ref in NetworkProcess https://bugs.webkit.org/show_bug.cgi?id=305723
89ad6512372c Reduce unsafeness in WebCore/css/query https://bugs.webkit.org/show_bug.cgi?id=304399 rdar://167168414
4b75ae7073f0 [WPE] Further cleanup after removal of the Qt5 embedding API https://bugs.webkit.org/show_bug.cgi?id=305602
1d511d6cf0ce Address some unsafe cast warnings in Source/WebCore/layout https://bugs.webkit.org/show_bug.cgi?id=305810
c5f94bb4b981 Address unsafe cast issues in Source/WebCore/bindings https://bugs.webkit.org/show_bug.cgi?id=305798
d21d3038fbce WPT service-worker/tentative/static-router/static-router-resource-timing.https.html is no longer flaky in cocoa bots rdar://168444917 https://bugs.webkit.org/show_bug.cgi?id=305766
e27ac61ee273 getUserMedia with constraints where max frameRate bigger than 16 does not work rdar://115713922 https://bugs.webkit.org/show_bug.cgi?id=261461
8123c8a002e5 Address some unsafe cast warnings in Source/WebCore/svg https://bugs.webkit.org/show_bug.cgi?id=305805
I'm puzzled.
Angelos Oikonomopoulos
After some bisecting, I can confirm that the extreme slowdown is a result of this change to `HashTraits.h`:
https://github.com/WebKit/WebKit/commit/fbbd9a28caa2a79f3bcb40a184cab7c63c0e23f7#diff-6d87d8d923fe599f98c8af6f8dea0fa7f1eff31fc577cdccefa015176ccb54f2
Carlos Alberto Lopez Perez
305864@main got reverted in 306040@main
So I'm closing this issue.
*** This bug has been marked as a duplicate of bug 306062 ***