Bug 32921 - [GTK] Build time must be reduced
Summary: [GTK] Build time must be reduced
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebKitGTK (show other bugs)
Version: 528+ (Nightly build)
Hardware: PC OS X 10.5
: P2 Normal
Assignee: Nobody
URL:
Keywords: Gtk
Depends on:
Blocks:
 
Reported: 2009-12-24 11:54 PST by Gustavo Noronha (kov)
Modified: 2010-09-10 08:18 PDT (History)
8 users (show)

See Also:


Attachments
Fix warning during generation of DocTypeStrings.cpp (1.10 KB, patch)
2009-12-24 11:54 PST, Gustavo Noronha (kov)
no flags Details | Formatted Diff | Diff
Implement Evan Martin's idea - link libwebkit directly, without going through libWebCore (3.23 KB, patch)
2009-12-24 11:56 PST, Gustavo Noronha (kov)
no flags Details | Formatted Diff | Diff
Make debug builds based on many small shared libraries (15.74 KB, patch)
2009-12-24 12:03 PST, Gustavo Noronha (kov)
no flags Details | Formatted Diff | Diff
Fix warnings for gperf calls (1.60 KB, patch)
2009-12-28 14:50 PST, Gustavo Noronha (kov)
no flags Details | Formatted Diff | Diff
Remove libWebCore intermediate library, to improve link time. (4.07 KB, patch)
2009-12-28 15:07 PST, Gustavo Noronha (kov)
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Gustavo Noronha (kov) 2009-12-24 11:54:26 PST
Created attachment 45470 [details]
Fix warning during generation of DocTypeStrings.cpp

During the hackfest we explored a bunch of strategies to make the build take less time, specially when doing incremental builds after touching a small number of files. We should push some proper changes implementing some of the ideas without breaking the build system =). My proposal is to make two changes:

1. adopt Evan Martin's awesome idea of linking all files only once, at the end, for release builds
2. make debug builds be many small shared libraries, to guarantee small build times when touching some files only

I am posting patches that implement both. The first patch is actually a build fix for a gperf call that doesn't add the appropriate include. I am not marking the patches for review as of yet - I would like people to test them/agree to the changes before posting proper patches.
Comment 1 Gustavo Noronha (kov) 2009-12-24 11:56:06 PST
Created attachment 45471 [details]
Implement Evan Martin's idea - link libwebkit directly, without going through libWebCore
Comment 2 Gustavo Noronha (kov) 2009-12-24 12:03:39 PST
Created attachment 45472 [details]
Make debug builds based on many small shared libraries

Some numbers:

I made some measurements running this command: ccache -C; sync; sleep 2; time env WebKitMakeArguments='-j2' ./WebKitTools/Scripts/build-webkit --gtk --enable-introspection --debug. Here are the numbers:

No patch at all

Full build: 40:34.24elapsed
Touching WebCore/platform/network/soup/ResourceHandleSoup.cpp: 8:19.97elapsed

With the patch that removes libWebCore:

Full build: 39:02.65elapsed
Touching WebCore/platform/network/soup/ResourceHandleSoup.cpp: 5:08.00elapsed

With the many shared libs patch:

Full build: 38:23.59elapsed
Touching WebCore/platform/network/soup/ResourceHandleSoup.cpp: 1:34.33elapsed

There was no swapping at all in any of those (I closed most apps to not interfere with the tests) - I believe the shared libs patch may be a big help if normal debug builds are currently swapping (which is the case for me, usually), though.
Comment 3 Evan Martin 2009-12-26 07:00:56 PST
I am a little surprised the third patch works -- I would think all the .so would have missing symbols pointing into each other.

+webcore_built_objects = := $(patsubst %.cpp,%.lo,$(webcore_built_nosources))
I wonder if there are any .c files (perhaps generated ones?) that need to be in this list.

But all of these patches look good to me if they work for you.
Comment 4 Benjamin Otte 2009-12-26 08:18:40 PST
I dislike the last patch. It introduces a separate build system for debug builds and due to that shares all the problems of a separate build system, among them:
- more code to manage
- more bugs
- different bugs
- different behavior (think performance here)
I also think that a debug build should not deviate from a default build too much. And my gut feeling says that the last patch does too much of this.

Did anybody ever talk to the gcc people about how to improve build times? Or talk to other people working on large C++ projects with similar issues (Open Office, Mozilla etc)? THey might have better ideas than splitting the build system.
Comment 5 Evan Martin 2009-12-26 08:36:07 PST
Chrome shared-link builds put all of webkit into a single .so.  I think it was only about 1 min for incremental links once we removed all of the .la files (not just libwebcore, but also libwebcorejs and libjavascriptcore).  Maybe that's an acceptable compromise.
Comment 6 Gustavo Noronha (kov) 2009-12-27 11:04:56 PST
(In reply to comment #3)
> I am a little surprised the third patch works -- I would think all the .so
> would have missing symbols pointing into each other.
> 
> +webcore_built_objects = := $(patsubst %.cpp,%.lo,$(webcore_built_nosources))
> I wonder if there are any .c files (perhaps generated ones?) that need to be in
> this list.
> 
> But all of these patches look good to me if they work for you.

Yeah, I was a bit surprised as well. Only libwebcore_svg needed special handling there. All tests pass, and I was able to run Epiphany for a while with it, with no issues.
Comment 7 Gustavo Noronha (kov) 2009-12-27 11:09:17 PST
(In reply to comment #4)
> I dislike the last patch. It introduces a separate build system for debug
> builds and due to that shares all the problems of a separate build system,
> among them:
> - more code to manage
> - more bugs
> - different bugs
> - different behavior (think performance here)
> I also think that a debug build should not deviate from a default build too
> much. And my gut feeling says that the last patch does too much of this.

Kinda. It does not really introduce a separate build system. It just changes how the linking happens. All the file lists are the same. About different behavior: debug buils are already heavily different in behavior, specially where performance is concerned. Having said that, I am also a bit uneasy about introducing different linkage rules for Debug builds, but given the amount of time it saves in incremental builds, I think it may be worth a try.

I plan to also get rid of the libWebCoreJS, and do something to libJavaScriptCore - specially because we would like to have it shipping as a separate library - and maybe that would give us the same amount of benefits, regarding link time, but *shrug*.
Comment 8 Gustavo Noronha (kov) 2009-12-28 14:50:34 PST
Created attachment 45575 [details]
Fix warnings for gperf calls
Comment 9 WebKit Review Bot 2009-12-28 14:55:21 PST
style-queue ran check-webkit-style on attachment 45575 [details] without any errors.
Comment 10 Gustavo Noronha (kov) 2009-12-28 14:58:45 PST
(In reply to comment #3)
> I am a little surprised the third patch works -- I would think all the .so
> would have missing symbols pointing into each other.
> 
> +webcore_built_objects = := $(patsubst %.cpp,%.lo,$(webcore_built_nosources))
> I wonder if there are any .c files (perhaps generated ones?) that need to be in
> this list.

I don't think there are. $(webcore_built_nosources) comes from this:

 webcore_built_nosources += $(patsubst %.idl,DerivedSources/JS%.cpp,$(notdir
$(IDL_BINDINGS_JS)))

So we should be safe.
Comment 11 Ariya Hidayat 2009-12-28 15:05:49 PST
Comment on attachment 45575 [details]
Fix warnings for gperf calls

LGTM. The Qt port also uses --include (same as -I) for invoking gpref.
Comment 12 Gustavo Noronha (kov) 2009-12-28 15:07:27 PST
Created attachment 45576 [details]
Remove libWebCore intermediate library, to improve link time.
Comment 13 WebKit Review Bot 2009-12-28 15:11:25 PST
style-queue ran check-webkit-style on attachment 45576 [details] without any errors.
Comment 14 Gustavo Noronha (kov) 2009-12-28 16:00:26 PST
Comment on attachment 45575 [details]
Fix warnings for gperf calls

Landed as 52603.
Comment 15 Xan Lopez 2010-01-05 09:06:09 PST
Comment on attachment 45576 [details]
Remove libWebCore intermediate library, to improve link time.

Makes sense to me, can't think of any downside to this and no reason to do it in the first place.
Comment 16 Gustavo Noronha (kov) 2010-01-05 11:09:30 PST
Comment on attachment 45576 [details]
Remove libWebCore intermediate library, to improve link time.

Landed as r52813.
Comment 17 Martin Robinson 2010-09-10 08:18:36 PDT
I've done a lot of work on this recently. For instance I've removed the remaining two intermediate libraries and removed the need to call autogen.sh for every build-webkit invocation. Our null build is about 14 seconds on my local machine. There's still a bit of work to do to remove the GNUmake lag (the time it takes to read the Makefiles and build the dependency graph), but I think they can be addressed in smaller bugs. So I'll close this bug for now. If anyone objects, please reopen.