Bug 246206 – [GTK] 2.38.0 breaks lightdm-webkit2-greeter (can't log in)

• Status: RESOLVED
• Resolution: FIXED
• Component: WebKitGTK

[Dave]

My environment: Linux Manjaro, lightdm-webkit2-greeter setup in /etc/lightdm/lightdm.conf.

Yesterday I received an update of webkit2gtk, from 2.36.7 to 2.38.0.

What happens/the issue which I encountered: when the login greeter appears, I type the password but I can't login: the enter key nor the login button works.
I tried to change the greeter's theme but doesn't help.

I downgraded webkit2gtk to 2.36.7 and the mentioned issue went away.

Please ask if more info are needed.

[Dave]

Also other users are affected by webkit2gto 2.38.0:

https://forum.manjaro.org/t/lightdm-webkit2-greeter-no-longer-accepts-the-password-submission-due-to-webkit2gtk/123470/

https://forum.manjaro.org/t/can-no-longer-log-in-after-most-recent-update/123546

https://forum.manjaro.org/t/lightdm-hangs-on-login/123410

[Michael Catanzaro]

Note: this has also been reported at https://github.com/Litarvan/lightdm-webkit-theme-litarvan/issues/186 and https://github.com/NoiSek/Aether/issues/115.

(In reply to Dave from comment #0)
> Please ask if more info are needed.

The other users who noticed this had a bunch of warnings and criticals in the console output, e.g. from https://github.com/Litarvan/lightdm-webkit-theme-litarvan/issues/186#issuecomment-1257277177 we see:

(WebKitWebProcess:9872): GLib-WARNING **: 20:34:52.728: ../glib/glib/giounix.c:412Error while getting flags for FD: Bad file descriptor (9)

(WebKitWebProcess:9872): GLib-WARNING **: 20:34:52.732: Invalid file descriptor.
Connected: 0
Error: Failed to write to daemon: Bad file descriptor

To investigate this, we'd want to see a gdb backtrace to the very first warning, taken with G_DEBUG=fatal-warnings. That will point directly to the problematic file descriptor. Otherwise, no way to know where it's coming from.

[Dave]

(In reply to Michael Catanzaro from comment #2)
> we'd want to see a gdb backtrace to the very first
> warning, taken with G_DEBUG=fatal-warnings. That will point directly to the
> problematic file descriptor. Otherwise, no way to know where it's coming
> from.

Can you tell/explain me how to achieve this?
Thank you.

[Michael Catanzaro]

I can explain the gdb part: https://blogs.gnome.org/mcatanzaro/2021/09/18/creating-quality-backtraces-for-crash-reports/

But since lightdm-webkit2-greeter is a system component, someone who uses it will have to figure out how to run it with the G_DEBUG environment variable to make it crash. If you can run it from command line and see the same error, that would be easy and ideal. If that doesn't work, you *could* try setting it in /etc/environment, but that will probably cause other system components to crash too and could stop you from getting far enough to even launch lightdm-webkit2-greeter, so that might not work. A surefire way would be to replace the lightdm-webkit2-greeter binary with a script that sets the environment variable and then execs the binary. That will almost definitely work.

[Dave]

I am not able to get a gdb backtrace: when I launch lightdm-webkit2-greeter (with webkit2gtk 2.38.0 - and lightdm-webkit2-greeter as shell script with the env variable) through gdb, obviously I got stuck, since I can't login: the only way to back to desktop and so into the terminal where to give commands to gdb (run, bt and quit) is by login in tty2 and here run startx, but here all is lost from the session, the terminal is no longer opened/present.

[Michael Catanzaro]

You don't need to run it under gdb. You can use 'coredumpctl gdb' to get a backtrace after it crashes.

[Dave]

coredumpctl gdb results in:
No journal files were opened due to insufficient permissions.

sudo coredumpctl gdb results in:
No match found.

[Michael Catanzaro]

You shouldn't need to use sudo. Not sure what's wrong with your coredumpctl. You will need to figure out how to fix that and get a backtrace before we can proceed further here. Good luck....

[Michael Catanzaro]

Actually based on https://github.com/WebKit/WebKit/pull/4868#issuecomment-1272333602 I'm going to suggest you try that PR and see if it fixes the issue. I suspect lightdm-webkit2-greeter is expecting file descriptors to leak into the child process, which is wrong.

[Dave]

(In reply to Michael Catanzaro from comment #8)
> You shouldn't need to use sudo. Not sure what's wrong with your coredumpctl.
> You will need to figure out how to fix that and get a backtrace before we
> can proceed further here. Good luck....

Ok: I've been able to catch the coredump, I attach it to this ticket.

[Dave]

Created attachment 462883 [details]
gdb backtrace of lightdm-webkit2-greeter

Output of coredumpctl gdb.

[Dave]

(In reply to Michael Catanzaro from comment #9)
> Actually based on
> https://github.com/WebKit/WebKit/pull/4868#issuecomment-1272333602 I'm going
> to suggest you try that PR and see if it fixes the issue

I don't know how to deal with github (apply a PR and then what I have to compile)

[Adrian Perez]

(In reply to Dave from comment #12)
> (In reply to Michael Catanzaro from comment #9)
> > Actually based on
> > https://github.com/WebKit/WebKit/pull/4868#issuecomment-1272333602 I'm going
> > to suggest you try that PR and see if it fixes the issue
> 
> I don't know how to deal with github (apply a PR and then what I have to
> compile)

If you append .patch to the URL of a pull request, you can download
the patch directly and apply it locally to your copy of the WebKit
source tree:

  cd path/to/WebKit
  curl -sL https://github.com/WebKit/WebKit/pull/4868.patch | patch -p1

=)

[Michael Catanzaro]

(In reply to Dave from comment #11)
> Created attachment 462883 [details]
> gdb backtrace of lightdm-webkit2-greeter
> 
> Output of coredumpctl gdb.

Hmm that's actually pointing to WebKit, rather than lightdm, which is not what I was expecting to see. So it is possible that this is a WebKit bug. But you don't have any debuginfo installed, so there's no way to know where it's coming from. If you install debuginfo and try again, then we'll know for sure. That will fill in all the n/a frames:

#0  0x00007ffff7e4dfe8 g_log_structured_array (libglib-2.0.so.0 + 0x5cfe8)
                #1  0x00007ffff7e4e623 g_log_default_handler (libglib-2.0.so.0 + 0x5d623)
                #2  0x00007ffff7e4f785 g_logv (libglib-2.0.so.0 + 0x5e785)
                #3  0x00007ffff7e4fa34 g_log (libglib-2.0.so.0 + 0x5ea34)
                #4  0x00007ffff4936671 n/a (libwebkit2gtk-4.0.so.37 + 0xa46671)
                #5  0x00007ffff47f3642 n/a (libwebkit2gtk-4.0.so.37 + 0x903642)
                #6  0x00007ffff495d421 n/a (libwebkit2gtk-4.0.so.37 + 0xa6d421)
                #7  0x00007ffff49666c3 n/a (libwebkit2gtk-4.0.so.37 + 0xa766c3)
                #8  0x00007ffff4966adb n/a (libwebkit2gtk-4.0.so.37 + 0xa76adb)
                #9  0x00007ffff384329c n/a (libjavascriptcoregtk-4.0.so.18 + 0xfea29c)
                #10 0x00007ffff385ec46 n/a (libjavascriptcoregtk-4.0.so.18 + 0x1005c46)
                #11 0x00007ffff7e4681b g_main_context_dispatch (libglib-2.0.so.0 + 0x5581b)
                #12 0x00007ffff7e9cec9 n/a (libglib-2.0.so.0 + 0xabec9)
                #13 0x00007ffff7e45d7f g_main_loop_run (libglib-2.0.so.0 + 0x54d7f)
                #14 0x00007ffff7806e9f gtk_main (libgtk-3.so.0 + 0x1d8e9f)
                #15 0x00005555555576c3 main (lightdm-webkit2-greeter + 0x36c3)
                #16 0x00007ffff2553290 n/a (libc.so.6 + 0x23290)
                #17 0x00007ffff255334a __libc_start_main (libc.so.6 + 0x2334a)
                #18 0x0000555555557725 _start (lightdm-webkit2-greeter + 0x3725)

Also, please use the 'bt full' command to take the backtrace.

[Dave]

(In reply to Michael Catanzaro from comment #14)
> Also, please use the 'bt full' command to take the backtrace.

After executing coredumpctl gdb it back to the terminal prompt, not to gdb (I expect "(gdb)" prompt), am I miss something?.

Anyway I installed debuginfod: which command and where/how execute it?

[Dave]

(In reply to Dave from comment #15)
> (In reply to Michael Catanzaro from comment #14)
> > Also, please use the 'bt full' command to take the backtrace.
> 
> After executing coredumpctl gdb it back to the terminal prompt, not to gdb
> (I expect "(gdb)" prompt), am I miss something?.
> 
> Anyway I installed debuginfod: which command and where/how execute it?

I got a full coredump with bt full. I attach it to this ticket.

[Dave]

Created attachment 462889 [details]
full backtrace

[Dave]

(In reply to Dave from comment #16)
> I got a full coredump with bt full. I attach it to this ticket.

Anyway: It didn't prompted to enable debuginfod (with "y"), but debuginfod is installed.

[Michael Catanzaro]

Again, the WebKit frames are all empty, so we can't do anything with this. Once you install debuginfo for WebKitGTK -- if your distro does not enable debuginfod you'll have to figure out how to do that manually -- and retake the backtrace, then we'll know where the file descriptor in question is.

[Dave]

(In reply to Michael Catanzaro from comment #19)
> Again, the WebKit frames are all empty, so we can't do anything with this.
> Once you install debuginfo for WebKitGTK -- if your distro does not enable
> debuginfod you'll have to figure out how to do that manually -- and retake
> the backtrace, then we'll know where the file descriptor in question is.

debuginfod is installed, these are the files on the system:

/etc/debuginfod/archlinux.urls
/etc/profile.d/debuginfod.csh
/etc/profile.d/debuginfod.sh
/usr/bin/debuginfod
/usr/bin/debuginfod-find
/usr/share/doc/debuginfod/AUTHORS
/usr/share/doc/debuginfod/ChangeLog
/usr/share/doc/debuginfod/NEWS
/usr/share/doc/debuginfod/NOTES
/usr/share/doc/debuginfod/README
/usr/share/man/man1/debuginfod-find.1.gz
/usr/share/man/man3/debuginfod_add_http_header.3.gz
/usr/share/man/man3/debuginfod_begin.3.gz
/usr/share/man/man3/debuginfod_end.3.gz
/usr/share/man/man3/debuginfod_find_debuginfo.3.gz
/usr/share/man/man3/debuginfod_find_executable.3.gz
/usr/share/man/man3/debuginfod_find_source.3.gz
/usr/share/man/man3/debuginfod_get_url.3.gz
/usr/share/man/man3/debuginfod_get_user_data.3.gz
/usr/share/man/man3/debuginfod_set_progressfn.3.gz
/usr/share/man/man3/debuginfod_set_user_data.3.gz
/usr/share/man/man7/debuginfod-client-config.7.gz
/usr/share/man/man8/debuginfod.8.gz

I have to execute something else?

[Dave]

(In reply to Michael Catanzaro from comment #19)
> Again, the WebKit frames are all empty, so we can't do anything with this.
> Once you install debuginfo for WebKitGTK -- if your distro does not enable
> debuginfod you'll have to figure out how to do that manually -- and retake
> the backtrace, then we'll know where the file descriptor in question is.

Ok, I had to export DEBUGINFOD_URLS="https://debuginfod.archlinux.org/"; now has worked. I attach it.

[Dave]

Created attachment 462890 [details]
gdb bt full debuginfod

[Dave]

Created attachment 462891 [details]
second gdb bt full debuginfod

The second coredump is longer in terms of lines, I attach it to be sure.

[Dave]

However, under /var/lib/systemd/coredump/ two coredump files are generated, one for lightdm-webkit2 and the other one for WebKitNetwork

[Michael Catanzaro]

Well that wasn't what I was expecting to see... you've simply run out of memory. Normally this means you've messed with your overcommit settings, which will cause fork() to fail. 255325@main will "fix" this by switching back to posix_spawn(), but the problem could recur in the future if we switch back to using fork() again.

Note your problem is totally different from what other users of lightdm-webkit2-greeter were complaining about, but I will close this anyway because I suspect the same commit will "fix" the invalid file descriptor issue as well. Same warning applies, though: lightdm-webkit2-greeter should find a different way to do its IPC if it wants to be robust to future process launching changes.