259773 2023-08-03 03:23:45 -0700 REGRESSION(264514@main): [WPE] Lot of crashes when running layout tests on machines without a GPU 2023-08-04 04:04:03 -0700 1 1 1 Unclassified WebKit WPE WebKit WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=257321 P2 Normal --- 1 clopez philn bugs-noreply philn oldest_to_newest 1969418 0 clopez 2023-08-03 03:23:45 -0700 264514@main updated several recipes on the Flatpak SDK. Among them it updated the FDO junction from freedesktop-sdk-22.08.11-24-g67b3605b402cffed2db0344f1da3a6acb5b6c914 to freedesktop-sdk-22.08.11-127-g7bed8a0d05bfd13939862b30e7080d1ffd5f635b And that it turn updated Mesa from mesa-23.0.3-0-g77661a60228061aa5d78107e0de5c58c803a57a4 to mesa-23.1.0-0-gbe4f7fb656180ab55a50eff01f36672b0bf5f146 And there is a regression with the new version of Mesa that is causing massive crashes on the WPE bots that run without a GPU (which happen to be all the EWS machines). You can see how they are crashing with more than 500 crashes (exit early) since then: https://ews-build.webkit.org/#/builders/34 I have been able to reproduce the issue by building Mesa with jhbuild (for faster iteration) and I'm now bisecting Mesa to find out which Mesa commit caused this 1969424 1 clopez 2023-08-03 04:00:45 -0700 Bisect finished. This is the Mesa commit causing the issue: - b8da022da410519c95d5726fc92a9ee5731de5e8 is the first bad commit - https://gitlab.freedesktop.org/mesa/mesa/-/commit/b8da022da410519c95d5726fc92a9ee5731de5e8 Is not obvious to me why this Mesa commit is causing this crashes. Will try to debug it with GDB. 1969425 2 clopez 2023-08-03 04:07:47 -0700 The issue (crashes) is not reproducible with Mesa main so it seems this has been fixed on the main branch. I will do a reverse bisect to find the commit fixing it 1969464 3 philn 2023-08-03 10:28:00 -0700 Thanks for investigating this, Carlos! Do you know if Mesa 23.1.5 is also affected by this? 1969465 4 philn 2023-08-03 10:30:01 -0700 Do you have a bt of the crash? The EWS doesn't seem to log them... 1969573 5 clopez 2023-08-03 18:21:18 -0700 I had trouble bisecting the main branch as I'm not able to build several commits in the range. However I tested the stable branches and: - 23.1.5 : works fine - 23.1.4 : crashes Bisecting between both tags: - The commit fixing the issue is: c795abedd2c8d293ad5e9bfac1f8b362260e5bf8 https://gitlab.freedesktop.org/mesa/mesa/-/commit/c795abedd2c8d293ad5e9bfac1f8b362260e5bf8 Related MR: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24347 So we need to update Mesa to 23.1.5 to fix this issue on the bots. I didn't bothered to get a GDB back-trace but I can get one easily if you think is useful, just let me know if you want it. 1969617 6 philn 2023-08-04 01:22:42 -0700 (In reply to Carlos Alberto Lopez Perez from comment #5) > So we need to update Mesa to 23.1.5 to fix this issue on the bots. > I can submit a PR. > > I didn't bothered to get a GDB back-trace but I can get one easily if you > think is useful, just let me know if you want it. I was just asking out of curiosity. 1969620 7 clopez 2023-08-04 02:15:38 -0700 (In reply to Philippe Normand from comment #6) > (In reply to Carlos Alberto Lopez Perez from comment #5) > > So we need to update Mesa to 23.1.5 to fix this issue on the bots. > > > > I can submit a PR. > Great. I appreciate that. Thanks! > > > > I didn't bothered to get a GDB back-trace but I can get one easily if you > > think is useful, just let me know if you want it. > > I was just asking out of curiosity. Ok. Will try to get some backtrace. 1969627 8 pnormand 2023-08-04 03:08:09 -0700 Pull request: https://github.com/WebKit/WebKit/pull/16388 1969631 9 clopez 2023-08-04 03:19:58 -0700 (In reply to Carlos Alberto Lopez Perez from comment #7) > (In reply to Philippe Normand from comment #6) > > (In reply to Carlos Alberto Lopez Perez from comment #5) > > > So we need to update Mesa to 23.1.5 to fix this issue on the bots. > > > > > > > I can submit a PR. > > > > Great. I appreciate that. Thanks! > > > > > > > I didn't bothered to get a GDB back-trace but I can get one easily if you > > > think is useful, just let me know if you want it. > > > > I was just asking out of curiosity. > > Ok. Will try to get some backtrace. I tried to debug this but is a bit complicated. There isn't a crash as such. And the issue is not on the WebProcess but on the UIProcess. I can reproduce this both with WebKitTestRunner or with Cog. On stderr you can see this: libEGL warning: failed to get driver name for fd 0 libEGL warning: MESA-LOADER: failed to retrieve device information But I couldn't get much more useful information. Seems some OpenGL/EGL API call that is not expected to fail is failing and then everything goes wrong. 1969633 10 ews-feeder 2023-08-04 03:26:54 -0700 Committed 266576@main (87a213be3f95): <https://commits.webkit.org/266576@main> Reviewed commits have been landed. Closing PR #16388 and removing active labels. 1969634 11 clopez 2023-08-04 03:31:37 -0700 (In reply to Carlos Alberto Lopez Perez from comment #9) > (In reply to Carlos Alberto Lopez Perez from comment #7) > > (In reply to Philippe Normand from comment #6) > > > (In reply to Carlos Alberto Lopez Perez from comment #5) > > > > So we need to update Mesa to 23.1.5 to fix this issue on the bots. > > > > > > > > > > I can submit a PR. > > > > > > > Great. I appreciate that. Thanks! > > > > > > > > > > I didn't bothered to get a GDB back-trace but I can get one easily if you > > > > think is useful, just let me know if you want it. > > > > > > I was just asking out of curiosity. > > > > Ok. Will try to get some backtrace. > > > I tried to debug this but is a bit complicated. > > There isn't a crash as such. > > And the issue is not on the WebProcess but on the UIProcess. > > I can reproduce this both with WebKitTestRunner or with Cog. > > On stderr you can see this: > > libEGL warning: failed to get driver name for fd 0 > libEGL warning: MESA-LOADER: failed to retrieve device information > > But I couldn't get much more useful information. > > Seems some OpenGL/EGL API call that is not expected to fail is failing and > then everything goes wrong. Also the issue is much more easy to trigger if you run several WebKitTestRunners in parallel. 1969638 12 clopez 2023-08-04 04:04:03 -0700 (In reply to Carlos Alberto Lopez Perez from comment #9) > On stderr you can see this: > > libEGL warning: failed to get driver name for fd 0 > libEGL warning: MESA-LOADER: failed to retrieve device information > Another important thing is that the log is also full of warnings like this: EGLDisplay Initialization failed: EGL_NOT_INITIALIZED But that warning (EGL_NOT_INITIALIZED) also happens with this new Mesa version that fixes the issue and was also happening before this regression. So it is not new. Things work despite that EGL_NOT_INITIALIZED warning, but maybe we should look at it as it doesn't seem normal this warning.