Bug 246206 - [GTK] 2.38.0 breaks lightdm-webkit2-greeter (can't log in)
Summary: [GTK] 2.38.0 breaks lightdm-webkit2-greeter (can't log in)
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebKitGTK (show other bugs)
Version: Other
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-10-07 02:03 PDT by Dave
Modified: 2022-10-09 09:00 PDT (History)
4 users (show)

See Also:


Attachments
gdb backtrace of lightdm-webkit2-greeter (21.79 KB, text/plain)
2022-10-08 09:25 PDT, Dave
no flags Details
full backtrace (24.38 KB, text/plain)
2022-10-09 00:45 PDT, Dave
no flags Details
gdb bt full debuginfod (36.34 KB, text/plain)
2022-10-09 06:14 PDT, Dave
no flags Details
second gdb bt full debuginfod (39.09 KB, text/plain)
2022-10-09 06:45 PDT, Dave
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Dave 2022-10-07 02:03:59 PDT
My environment: Linux Manjaro, lightdm-webkit2-greeter setup in /etc/lightdm/lightdm.conf.

Yesterday I received an update of webkit2gtk, from 2.36.7 to 2.38.0.

What happens/the issue which I encountered: when the login greeter appears, I type the password but I can't login: the enter key nor the login button works.
I tried to change the greeter's theme but doesn't help.

I downgraded webkit2gtk to 2.36.7 and the mentioned issue went away.

Please ask if more info are needed.
Comment 2 Michael Catanzaro 2022-10-07 04:54:49 PDT
Note: this has also been reported at https://github.com/Litarvan/lightdm-webkit-theme-litarvan/issues/186 and https://github.com/NoiSek/Aether/issues/115.

(In reply to Dave from comment #0)
> Please ask if more info are needed.

The other users who noticed this had a bunch of warnings and criticals in the console output, e.g. from https://github.com/Litarvan/lightdm-webkit-theme-litarvan/issues/186#issuecomment-1257277177 we see:

(WebKitWebProcess:9872): GLib-WARNING **: 20:34:52.728: ../glib/glib/giounix.c:412Error while getting flags for FD: Bad file descriptor (9)

(WebKitWebProcess:9872): GLib-WARNING **: 20:34:52.732: Invalid file descriptor.
Connected: 0
Error: Failed to write to daemon: Bad file descriptor

To investigate this, we'd want to see a gdb backtrace to the very first warning, taken with G_DEBUG=fatal-warnings. That will point directly to the problematic file descriptor. Otherwise, no way to know where it's coming from.
Comment 3 Dave 2022-10-07 05:16:31 PDT
(In reply to Michael Catanzaro from comment #2)
> we'd want to see a gdb backtrace to the very first
> warning, taken with G_DEBUG=fatal-warnings. That will point directly to the
> problematic file descriptor. Otherwise, no way to know where it's coming
> from.

Can you tell/explain me how to achieve this?
Thank you.
Comment 4 Michael Catanzaro 2022-10-07 07:48:29 PDT
I can explain the gdb part: https://blogs.gnome.org/mcatanzaro/2021/09/18/creating-quality-backtraces-for-crash-reports/

But since lightdm-webkit2-greeter is a system component, someone who uses it will have to figure out how to run it with the G_DEBUG environment variable to make it crash. If you can run it from command line and see the same error, that would be easy and ideal. If that doesn't work, you *could* try setting it in /etc/environment, but that will probably cause other system components to crash too and could stop you from getting far enough to even launch lightdm-webkit2-greeter, so that might not work. A surefire way would be to replace the lightdm-webkit2-greeter binary with a script that sets the environment variable and then execs the binary. That will almost definitely work.
Comment 5 Dave 2022-10-07 12:06:52 PDT
I am not able to get a gdb backtrace: when I launch lightdm-webkit2-greeter (with webkit2gtk 2.38.0 - and lightdm-webkit2-greeter as shell script with the env variable) through gdb, obviously I got stuck, since I can't login: the only way to back to desktop and so into the terminal where to give commands to gdb (run, bt and quit) is by login in tty2 and here run startx, but here all is lost from the session, the terminal is no longer opened/present.
Comment 6 Michael Catanzaro 2022-10-07 13:52:52 PDT
You don't need to run it under gdb. You can use 'coredumpctl gdb' to get a backtrace after it crashes.
Comment 7 Dave 2022-10-08 00:55:33 PDT
coredumpctl gdb results in:
No journal files were opened due to insufficient permissions.

sudo coredumpctl gdb results in:
No match found.
Comment 8 Michael Catanzaro 2022-10-08 09:02:14 PDT
You shouldn't need to use sudo. Not sure what's wrong with your coredumpctl. You will need to figure out how to fix that and get a backtrace before we can proceed further here. Good luck....
Comment 9 Michael Catanzaro 2022-10-08 09:07:06 PDT
Actually based on https://github.com/WebKit/WebKit/pull/4868#issuecomment-1272333602 I'm going to suggest you try that PR and see if it fixes the issue. I suspect lightdm-webkit2-greeter is expecting file descriptors to leak into the child process, which is wrong.
Comment 10 Dave 2022-10-08 09:23:55 PDT
(In reply to Michael Catanzaro from comment #8)
> You shouldn't need to use sudo. Not sure what's wrong with your coredumpctl.
> You will need to figure out how to fix that and get a backtrace before we
> can proceed further here. Good luck....

Ok: I've been able to catch the coredump, I attach it to this ticket.
Comment 11 Dave 2022-10-08 09:25:16 PDT
Created attachment 462883 [details]
gdb backtrace of lightdm-webkit2-greeter

Output of coredumpctl gdb.
Comment 12 Dave 2022-10-08 09:34:14 PDT
(In reply to Michael Catanzaro from comment #9)
> Actually based on
> https://github.com/WebKit/WebKit/pull/4868#issuecomment-1272333602 I'm going
> to suggest you try that PR and see if it fixes the issue

I don't know how to deal with github (apply a PR and then what I have to compile)
Comment 13 Adrian Perez 2022-10-08 12:01:20 PDT
(In reply to Dave from comment #12)
> (In reply to Michael Catanzaro from comment #9)
> > Actually based on
> > https://github.com/WebKit/WebKit/pull/4868#issuecomment-1272333602 I'm going
> > to suggest you try that PR and see if it fixes the issue
> 
> I don't know how to deal with github (apply a PR and then what I have to
> compile)

If you append .patch to the URL of a pull request, you can download
the patch directly and apply it locally to your copy of the WebKit
source tree:

  cd path/to/WebKit
  curl -sL https://github.com/WebKit/WebKit/pull/4868.patch | patch -p1

=)
Comment 14 Michael Catanzaro 2022-10-08 13:58:57 PDT
(In reply to Dave from comment #11)
> Created attachment 462883 [details]
> gdb backtrace of lightdm-webkit2-greeter
> 
> Output of coredumpctl gdb.

Hmm that's actually pointing to WebKit, rather than lightdm, which is not what I was expecting to see. So it is possible that this is a WebKit bug. But you don't have any debuginfo installed, so there's no way to know where it's coming from. If you install debuginfo and try again, then we'll know for sure. That will fill in all the n/a frames:

#0  0x00007ffff7e4dfe8 g_log_structured_array (libglib-2.0.so.0 + 0x5cfe8)
                #1  0x00007ffff7e4e623 g_log_default_handler (libglib-2.0.so.0 + 0x5d623)
                #2  0x00007ffff7e4f785 g_logv (libglib-2.0.so.0 + 0x5e785)
                #3  0x00007ffff7e4fa34 g_log (libglib-2.0.so.0 + 0x5ea34)
                #4  0x00007ffff4936671 n/a (libwebkit2gtk-4.0.so.37 + 0xa46671)
                #5  0x00007ffff47f3642 n/a (libwebkit2gtk-4.0.so.37 + 0x903642)
                #6  0x00007ffff495d421 n/a (libwebkit2gtk-4.0.so.37 + 0xa6d421)
                #7  0x00007ffff49666c3 n/a (libwebkit2gtk-4.0.so.37 + 0xa766c3)
                #8  0x00007ffff4966adb n/a (libwebkit2gtk-4.0.so.37 + 0xa76adb)
                #9  0x00007ffff384329c n/a (libjavascriptcoregtk-4.0.so.18 + 0xfea29c)
                #10 0x00007ffff385ec46 n/a (libjavascriptcoregtk-4.0.so.18 + 0x1005c46)
                #11 0x00007ffff7e4681b g_main_context_dispatch (libglib-2.0.so.0 + 0x5581b)
                #12 0x00007ffff7e9cec9 n/a (libglib-2.0.so.0 + 0xabec9)
                #13 0x00007ffff7e45d7f g_main_loop_run (libglib-2.0.so.0 + 0x54d7f)
                #14 0x00007ffff7806e9f gtk_main (libgtk-3.so.0 + 0x1d8e9f)
                #15 0x00005555555576c3 main (lightdm-webkit2-greeter + 0x36c3)
                #16 0x00007ffff2553290 n/a (libc.so.6 + 0x23290)
                #17 0x00007ffff255334a __libc_start_main (libc.so.6 + 0x2334a)
                #18 0x0000555555557725 _start (lightdm-webkit2-greeter + 0x3725)

Also, please use the 'bt full' command to take the backtrace.
Comment 15 Dave 2022-10-09 00:25:07 PDT
(In reply to Michael Catanzaro from comment #14)
> Also, please use the 'bt full' command to take the backtrace.

After executing coredumpctl gdb it back to the terminal prompt, not to gdb (I expect "(gdb)" prompt), am I miss something?.

Anyway I installed debuginfod: which command and where/how execute it?
Comment 16 Dave 2022-10-09 00:44:53 PDT
(In reply to Dave from comment #15)
> (In reply to Michael Catanzaro from comment #14)
> > Also, please use the 'bt full' command to take the backtrace.
> 
> After executing coredumpctl gdb it back to the terminal prompt, not to gdb
> (I expect "(gdb)" prompt), am I miss something?.
> 
> Anyway I installed debuginfod: which command and where/how execute it?

I got a full coredump with bt full. I attach it to this ticket.
Comment 17 Dave 2022-10-09 00:45:34 PDT
Created attachment 462889 [details]
full backtrace
Comment 18 Dave 2022-10-09 00:59:22 PDT
(In reply to Dave from comment #16)
> I got a full coredump with bt full. I attach it to this ticket.

Anyway: It didn't prompted to enable debuginfod (with "y"), but debuginfod is installed.
Comment 19 Michael Catanzaro 2022-10-09 05:57:53 PDT
Again, the WebKit frames are all empty, so we can't do anything with this. Once you install debuginfo for WebKitGTK -- if your distro does not enable debuginfod you'll have to figure out how to do that manually -- and retake the backtrace, then we'll know where the file descriptor in question is.
Comment 20 Dave 2022-10-09 06:01:02 PDT
(In reply to Michael Catanzaro from comment #19)
> Again, the WebKit frames are all empty, so we can't do anything with this.
> Once you install debuginfo for WebKitGTK -- if your distro does not enable
> debuginfod you'll have to figure out how to do that manually -- and retake
> the backtrace, then we'll know where the file descriptor in question is.

debuginfod is installed, these are the files on the system:

/etc/debuginfod/archlinux.urls
/etc/profile.d/debuginfod.csh
/etc/profile.d/debuginfod.sh
/usr/bin/debuginfod
/usr/bin/debuginfod-find
/usr/share/doc/debuginfod/AUTHORS
/usr/share/doc/debuginfod/ChangeLog
/usr/share/doc/debuginfod/NEWS
/usr/share/doc/debuginfod/NOTES
/usr/share/doc/debuginfod/README
/usr/share/man/man1/debuginfod-find.1.gz
/usr/share/man/man3/debuginfod_add_http_header.3.gz
/usr/share/man/man3/debuginfod_begin.3.gz
/usr/share/man/man3/debuginfod_end.3.gz
/usr/share/man/man3/debuginfod_find_debuginfo.3.gz
/usr/share/man/man3/debuginfod_find_executable.3.gz
/usr/share/man/man3/debuginfod_find_source.3.gz
/usr/share/man/man3/debuginfod_get_url.3.gz
/usr/share/man/man3/debuginfod_get_user_data.3.gz
/usr/share/man/man3/debuginfod_set_progressfn.3.gz
/usr/share/man/man3/debuginfod_set_user_data.3.gz
/usr/share/man/man7/debuginfod-client-config.7.gz
/usr/share/man/man8/debuginfod.8.gz

I have to execute something else?
Comment 21 Dave 2022-10-09 06:14:12 PDT
(In reply to Michael Catanzaro from comment #19)
> Again, the WebKit frames are all empty, so we can't do anything with this.
> Once you install debuginfo for WebKitGTK -- if your distro does not enable
> debuginfod you'll have to figure out how to do that manually -- and retake
> the backtrace, then we'll know where the file descriptor in question is.

Ok, I had to export DEBUGINFOD_URLS="https://debuginfod.archlinux.org/"; now has worked. I attach it.
Comment 22 Dave 2022-10-09 06:14:52 PDT
Created attachment 462890 [details]
gdb bt full debuginfod
Comment 23 Dave 2022-10-09 06:45:12 PDT
Created attachment 462891 [details]
second gdb bt full debuginfod

The second coredump is longer in terms of lines, I attach it to be sure.
Comment 24 Dave 2022-10-09 07:08:10 PDT
However, under /var/lib/systemd/coredump/ two coredump files are generated, one for lightdm-webkit2 and the other one for WebKitNetwork
Comment 25 Michael Catanzaro 2022-10-09 09:00:47 PDT
Well that wasn't what I was expecting to see... you've simply run out of memory. Normally this means you've messed with your overcommit settings, which will cause fork() to fail. 255325@main will "fix" this by switching back to posix_spawn(), but the problem could recur in the future if we switch back to using fork() again.

Note your problem is totally different from what other users of lightdm-webkit2-greeter were complaining about, but I will close this anyway because I suspect the same commit will "fix" the invalid file descriptor issue as well. Same warning applies, though: lightdm-webkit2-greeter should find a different way to do its IPC if it wants to be robust to future process launching changes.