Bug 181695

Summary: Possible to build both libjavascriptcoregtk and libwebkit2gtk as static libs only?
Product: WebKit Reporter: stefan.froberg
Component: WebKitGTKAssignee: Nobody <webkit-unassigned>
Status: RESOLVED FIXED    
Severity: Normal CC: bugs-noreply, didier, mcatanzaro, sfroberg13
Priority: P2    
Version: WebKit Local Build   
Hardware: PC   
OS: Linux   

Description stefan.froberg 2018-01-16 11:48:34 PST
Hello.

Is it somehow possible to instruct cmake to build static libraries instead of shared ones of both
libjavascriptcoregtk and libwebkit2gtk ?


I need to deploy my app as a one, big static blob but I need static versions of those two libs.


Very Best Regards
Stefan Fröberg
Comment 1 Michael Catanzaro 2018-01-16 12:01:51 PST
You can certainly try, and it just work better than using shared libs. ;) But it's definitely not supported, because WebKitGTK+ is a system library.

Here's what I would try:

 * In the toplevel CMakeLists.txt, change WebKit_LIBRARY_TYPE and remove the line below that setting CMAKE_POSITION_INDEPENDENT_CODE.
 * In Source/cmake/OptionsGTK.cmake, remove the line "set(JavaScriptCore_LIBRARY_TYPE SHARED)"
 * Build with -DENABLE_INTROSPECTION=OFF. Since you're hacking at the CMake files anyway, might as well change the default value of that option in OptionsGTK.cmake.

Good luck....

P.S. You might consider using flatpak to distribute your app, that is working well for us.
Comment 2 Michael Catanzaro 2018-01-16 12:02:21 PST
(In reply to Michael Catanzaro from comment #1)
> You can certainly try, and it just work better than using shared libs. ;)

"just might work better"

(Shared JavaScriptCore has caused a lot of problems in the past couple weeks....)
Comment 3 stefan.froberg 2018-01-16 12:11:55 PST
Okay, Thanks. 

I try to hack the source and see if I get somewhere.

Here's what I have been feeding sofar to cmake from separate build directory (Im trying to keep it simple at this point and disabling lot's of stuff for now):

CFLAGS=-Wno-expansion-to-defined CXXFLAGS=-Wno-expansion-to-defined cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_SKIP_RPATH=ON -DPORT=GTK -DUSE_LIBNOTIFY=OFF -DENABLE_SPELLCHECK=OFF -DENABLE_INTROSPECTION=OFF -DENABLE_API_TESTS=OFF -DLIB_INSTALL_DIR=/usr/lib -DUSE_LIBHYPHEN=OFF -DENABLE_MINIBROWSER=ON -DUSE_SYSTEM_MALLOC=ON -DENABLE_GEOLOCATION=OFF -DENABLE_GTKDOC=OFF -DENABLE_PLUGIN_PROCESS_GTK2=OFF -DENABLE_NETSCAPE_PLUGIN_API=OFF -DENABLE_OPENGL=OFF -DENABLE_SAMPLING_PROFILER=OFF -DENABLE_TOUCH_EVENTS=OFF -DENABLE_VIDEO=OFF -DENABLE_WEBDRIVER=OFF -DENABLE_WEB_AUDIO=OFF -DUSE_LD_GOLD=ON -DCMAKE_CXX_COMPILER_AR=/usr/bin/gcc-ar -DCMAKE_C_COMPILER_AR=/usr/bin/gcc-ar -DCMAKE_CXX_COMPILER_RANLIB=/usr/bin/gcc-ranlib -DCMAKE_C_COMPILER_RANLIB=/usr/bin/gcc-ranlib -DPAL_LIBRARY_TYPE=STATIC -DJavaScriptCore_LIBRARY_TYPE=STATIC -DWebKit_LIBRARY_TYPE=STATIC WTF_LIBRARY_TYPE=STATIC -DWebCoreTestSupport_LIBRARY_TYPE=STATIC -DWebCore_LIBRARY_TYPE=STATIC -DWebKit2_LIBRARY_TYPE=STATIC -DENABLE_WEBASSEMBLY=OFF -DUSE_THIN_ARCHIVES=OFF -Wno-dev ..
Comment 4 stefan.froberg 2018-01-16 13:19:54 PST
Hmmm... I probably also should mention that previously, I did manage to build
my app completely statically against webkit. 

It was against Qt port of webkit and instead of glibc I used musl.
It ran fine but whenever I tried to access any page with JavaScript it spit out the following error:

Error compiling builtin:
Fatal error compiling builtin function 'apply': Segmentation fault

It took me a while to figure out that disabling JavaScript from my app did absolutely nothing because Webkit has (since 2014?) used builting JavaScript functions.

Anyway, Qt-port uses still old WebKit1 and I want to switch to WebKit2 so that's why I want to try my luck with static linking but with GTK port this time.
Comment 5 stefan.froberg 2018-01-16 16:10:31 PST
I have now managed to build both libjavascriptcoregtk and libwebkit2gtk as static libs. 

However there is still a thing called libwebkit2gtkinjectedbundle.so ?

What is it? Do I need it? Can it be disabled?
Comment 6 Michael Catanzaro 2018-01-16 19:23:04 PST
(In reply to stefan.froberg from comment #5)
> I have now managed to build both libjavascriptcoregtk and libwebkit2gtk as
> static libs. 
> 
> However there is still a thing called libwebkit2gtkinjectedbundle.so ?
> 
> What is it? Do I need it? Can it be disabled?

That is the web process extension loader... it's certainly not supposed to be disabled, but you're not supposed to be building a static library either, so don't let that stop you from trying! If you don't need to inject code into the web process, then you don't need it. I'd try sabotaging the webkit2gtkinjectedbundle build target (comment it out, and comment out anything that uses it) in Source/WebKit/PlatformGTK.cmake.
Comment 7 Stefan Fröberg 2018-01-20 08:40:14 PST
Thank you very much.

I have now both the libjavcascriptcoregtk and libwevkit2gtk as static library and skipped building libwebkit2gtkinjectedbundle.so and tried to comment out any  references to it.

So, I have the headers and libs but when I try to static link my app against them it complains about duplicate _cashes variable in libxdgmime.a and libgio-2.0.a.

I tried to pass -DUSE_XDGMIME=OFF to cmake but without luck.
Comment 8 Stefan Fröberg 2018-01-20 09:17:46 PST
Hmmm.... would symbol hidding maybe solve that issue? If I understanded right, only the xdgmime.h has stuff that should be a public API and everything else could be hidden ? Including the extern XdgMimeCache **_caches; variable from xdgmimecache.h ?
Comment 9 Michael Catanzaro 2018-01-20 09:28:31 PST
My recommendation would be to just delete Source/ThirdParty/xdg-mime so that you get glib's copy of it... then try moving the xdgmime.h header somewhere into WebKit's include path. I think it should work.

(In reply to Stefan Fröberg from comment #8)
> Hmmm.... would symbol hidding maybe solve that issue? If I understanded
> right, only the xdgmime.h has stuff that should be a public API and
> everything else could be hidden ? Including the extern XdgMimeCache
> **_caches; variable from xdgmimecache.h ?

I imagine static linking to two different copies of xdgmime will break somehow, and the caches variable looks like a pretty good candidate....
Comment 10 Stefan Fröberg 2018-01-22 13:17:25 PST
Thank You very very much.

I followed your guide and have finally managed to build and link my test app against static versions of the libs :-)

But starting it brings one more problem:

** (test:8298): ERROR **: Unable to fork a new child process: Failed to execute child process "/usr/libexec/webkit2gtk-4.0/WebKitNetworkProcess" (No such file or directory)

So WebKit2 needs that helper binary (/usr/libexec/webkit2gtk-4.0/WebKitNetworkProcess) to start apps linked against it?


backtrace of my test app:

#0  _g_log_abort (breakpoint=1) at gmessages.c:509
#1  0x00000000035a63fa in g_log_default_handler (log_domain=log_domain@entry=0x0, log_level=log_level@entry=6, 
    message=message@entry=0x662a540 "Unable to fork a new child process: Failed to execute child process \"/usr/libexec/webkit2gtk-4.0/WebKitNetworkProcess\" (No such file or directory)", unused_data=unused_data@entry=0x0) at gmessages.c:2896
#2  0x00000000035a674d in g_logv (log_domain=0x0, log_level=G_LOG_LEVEL_ERROR, format=<optimized out>, 
    args=args@entry=0x7fffffffd400) at gmessages.c:1297
#3  0x00000000035a68bf in g_log (log_domain=<optimized out>, log_level=<optimized out>, format=<optimized out>)
    at gmessages.c:1359
#4  0x00000000006310e2 in WebKit::ProcessLauncher::launchProcess() ()
#5  0x00000000005d7c66 in WebKit::ChildProcessProxy::connect() ()
#6  0x00000000004f43fe in WebKit::NetworkProcessProxy::create(WebKit::WebProcessPool&) ()
#7  0x00000000004b184a in WebKit::WebProcessPool::ensureNetworkProcess(WebKit::WebsiteDataStore*) [clone .part.379] ()
#8  0x00000000004b273f in WebKit::WebProcessPool::createNewWebProcess(WebKit::WebsiteDataStore&) ()
#9  0x00000000004b2dd9 in WebKit::WebProcessPool::createWebPage(WebKit::PageClient&, WTF::Ref<API::PageConfiguration>&&) ()
#10 0x000000000043d491 in webkitWebViewBaseCreateWebPage(_WebKitWebViewBase*, WTF::Ref<API::PageConfiguration>&&) ()
#11 0x0000000000567299 in webkitWebContextCreatePageForWebView(_WebKitWebContext*, _WebKitWebView*, _WebKitUserContentManager*, _WebKitWebView*) ()
#12 0x000000000041dab7 in webkitWebViewConstructed(_GObject*) ()
#13 0x0000000000dc3e80 in g_object_new_internal (class=class@entry=0x66b9620, params=params@entry=0x7fffffffdce0, 
    n_params=n_params@entry=2) at gobject.c:1823
#14 0x0000000000dc5a5d in g_object_new_valist (object_type=object_type@entry=106907488, 
    first_property_name=first_property_name@entry=0x38ead0e "is-ephemeral", var_args=var_args@entry=0x7fffffffde30)
    at gobject.c:2042
#15 0x0000000000dc5ea1 in g_object_new (object_type=106907488, first_property_name=0x38ead0e "is-ephemeral")
    at gobject.c:1626
Comment 11 Michael Catanzaro 2018-01-22 17:49:47 PST
(In reply to Stefan Fröberg from comment #10) 
> So WebKit2 needs that helper binary
> (/usr/libexec/webkit2gtk-4.0/WebKitNetworkProcess) to start apps linked
> against it?

Also WebKitWebProcess and WebKitStorageProcess. (There's also WebKitPluginProcess and WebkitPluginProcess2, but you've disabled those already.) So three that you definitely need. (If you need a single-process web engine, your best bet would be to try creating your own port of WebKitLegacy... hopefully you do not need a single-process web engine!)

You probably already have all these binaries static linked to libwebkit2gtk after making the above changes (did I mention static linking is a bad idea ;) but I guess you're not running the CMake install stage. You'll need to ensure they get installed to disk one way or another. Of course, you surely don't want them installed under /usr. You can either set -DCMAKE_INSTALL_FULL_LIBEXECDIR to wherever you want at build time, or hack up ProcessExecutablePathGtk.cpp as desired. (It looks for the binaries under PKGLIBEXECDIR, which is set under CMAKE_INSTALL_FULL_LIBEXECDIR in OptionsGTK.cmake.)

P.S. No promises that what you're trying to do is possible, but I think it ought to work!
Comment 12 Stefan Fröberg 2018-01-24 10:26:40 PST
Thanks Michael :-)

I have now my test app and all the three Process binaries also in the same directory and opening HTTP site works :-)

However, HTTPS does not work :-(

Looking into it, it seems that WebKit uses libsoup and that uses glib-networking which in turn seems to dynamically load /usr/lib/gio/modules/libgiognutls.so with Glib's GIO infrastructure. 

Now, I could make a static version, /usr/lib/gio/modules/libgiognutls.a and link into my app without problem but I have no clue how to make libsoup+glib-networking combo to use it...

What exactly are the roles of libsoup and glib-networking ? 
Does libsoup handle the http or is it glib-networks job? 

Could I maybe directly inject the https handling code (gnutls) to either one, without needing to use dlopen and shared GIO module libgiognutls.so ?
Comment 13 Michael Catanzaro 2018-01-24 11:54:06 PST
Ah, more problems....

libsoup handles HTTP, glib-networking handles TLS (notably the S in HTTPS) and proxy settings. libsoup just uses GLib sockets and the GTlsConnection interface; glib-networking provides a GIO extension point that implements GTlsConnection.

You have lucky timing, because glib-networking just grew support for being used as a static library a couple weeks ago. I don't know how exactly it works, but I'll try to point you in the right direction. You'll need to use glib 2.55.1. You'll also need glib-networking 2.55.90, which has not been released yet, so just use git master. (The first stable releases with this feature will be 2.56.0, in March.) And you'll need to build glib-networking with -Dstatic_modules=true (pass that to meson, the new build system). Then that should (hopefully) spit out giognomeproxy.a, giolibproxy.a, and giognutls.a. The last one is what you need to make HTTPS work.

I'm not completely sure how you're supposed to use it from there, but *maybe* it will just work automatically if you link giognutls.a into your app? If you have trouble, you'll want to investigate https://bugzilla.gnome.org/show_bug.cgi?id=791100 and https://bugzilla.gnome.org/show_bug.cgi?id=684282.

Keep in mind that most of these dependencies are security-critical, so if you care about that, you'll want to stay on top of libsoup, glib-networking, and GnuTLS updates, in addition to WebKitGTK+ updates. Be careful with GnuTLS because its version numbers are confusing: the highest version number is usually an unstable development release, and the recommended stable version will be somewhat lower. Then GnuTLS itself has more security-critical dependencies with fun copyleft licenses; if those are problematic, then I'd recommend looking into the glib-openssl project as an alternative to glib-networking.
Comment 14 Stefan Fröberg 2018-01-24 16:00:48 PST
Okay I have now given it a test run.

I have now compiled glib 2.55.1 and latest git of glib-networking.
Both compiled succesfully and I have now /usr/lib/gio/modules/libgiognutls.a.

I then tried to link against /usr/lib/gio/modules/libgiognutls.a and run the app but it still gives "TLS/SSL support not available; install glib-networking".

nm gives the following:

nm -s /usr/lib/gio/modules/libgiognutls.a | grep g_io_gnutls_load

g_io_gnutls_load in gnutls-module.c.o
0000000000000000 T g_io_gnutls_load


The https://bugzilla.gnome.org/attachment.cgi?id=366306&action=edit
mentions:

"A module can rename its g_io_module_load() function to
g_io_<modulename>_load(), and then an application which links statically
against that module can call g_io_<modulename>_load(NULL) to register
types and extension points from the module. If a module is loaded
dynamically, its load() function will continue to be called with a
non-NULL GIOModule instance."

So I tried also adding g_io_gnutls_load(NULL) (after adding it's function definition) in the app, relinked and run but still the same.


Gosh...I guess I have to wait till march and Glib 2.56 ...

In the mean time I try to hack with the webgitgtk-2.18.5 sources and try to make the libs smaller :-) (after stripping 94 MB and after UPX 33 MB)
Comment 15 Michael Catanzaro 2018-01-24 18:13:52 PST
(In reply to Stefan Fröberg from comment #14)
> Gosh...I guess I have to wait till march and Glib 2.56 ...

That won't help as you already have all the relevant commits, we just don't know how exactly it's supposed to work. You'll have to debug it. Maybe Xavier could point you in the right direction.
Comment 16 Stefan Fröberg 2018-01-25 16:25:47 PST
I managed to make it work with HTTPS :-)

I checked these old stuff


https://gitlab.com/thiblahute/cerbero/commit/1b800edcbbabc44ede7b4e57749384e37f38f80a

https://mail.gnome.org/archives/libsoup-list/2013-September/msg00011.html

Then modified the gtlsbackend-gnutls.c like said above and added

g_tls_backend_gnutls_register() to

Source/WebKit/CMakeFiles/NetworkProcess.dir/NetworkProcess/EntryPoint/unix/NetworkProcessMain.cpp

I try to make a prober patch and reproduce it soon but now I have one final problem: JavaScript.

It does not run. Not even after making sure that     webkit_settings_set_enable_javascript(settings,TRUE);
 is set.


Did I disable too much from the webkit in that cmake configuration stage?
Is jsc binary needed for JavaScript in WebKit?
Comment 17 Michael Catanzaro 2018-01-26 05:20:46 PST
(In reply to Stefan Fröberg from comment #16)
> Did I disable too much from the webkit in that cmake configuration stage?

It looks OK to me. I'd be cautious about -DUSE_SYSTEM_MALLOC=ON since that will probably tank performance, but it shouldn't hurt JavaScript.

> Is jsc binary needed for JavaScript in WebKit?

No, it's definitely not needed. I don't have any guesses here, sorry.
Comment 18 Stefan Fröberg 2018-01-30 11:13:37 PST
Could the JavaScript not working on musl be related to stack size?


https://github.com/voidlinux/void-packages/issues/5360

Is there some global variable or #define somewhere in the source that determines the maximum stack size used by JavaScriptCore ?
Comment 19 Michael Catanzaro 2018-01-30 12:31:48 PST
(In reply to Stefan Fröberg from comment #18)
> Could the JavaScript not working on musl be related to stack size?

Possibly; I don't know. There's probably loads of glibc assumptions throughout WebKit just waiting for you to discover.... 

> https://github.com/voidlinux/void-packages/issues/5360
> 
> Is there some global variable or #define somewhere in the source that
> determines the maximum stack size used by JavaScriptCore ?

I don't know, but from that issue I see you already found the ones in VMEntryScope.cpp. I would start by testing those.