| Summary: | [GTK] Build time must be reduced | ||
|---|---|---|---|
| Product: | WebKit | Reporter: | Gustavo Noronha (kov) <gustavo> |
| Component: | WebKitGTK | Assignee: | Nobody <webkit-unassigned> |
| Status: | RESOLVED FIXED | ||
| Severity: | Normal | CC: | christian, evan, gustavo, mrobinson, otte, tevaum, webkit.review.bot, xan.lopez |
| Priority: | P2 | Keywords: | Gtk |
| Version: | 528+ (Nightly build) | ||
| Hardware: | PC | ||
| OS: | OS X 10.5 | ||
| Attachments: | |||
Created attachment 45471 [details]
Implement Evan Martin's idea - link libwebkit directly, without going through libWebCore
Created attachment 45472 [details]
Make debug builds based on many small shared libraries
Some numbers:
I made some measurements running this command: ccache -C; sync; sleep 2; time env WebKitMakeArguments='-j2' ./WebKitTools/Scripts/build-webkit --gtk --enable-introspection --debug. Here are the numbers:
No patch at all
Full build: 40:34.24elapsed
Touching WebCore/platform/network/soup/ResourceHandleSoup.cpp: 8:19.97elapsed
With the patch that removes libWebCore:
Full build: 39:02.65elapsed
Touching WebCore/platform/network/soup/ResourceHandleSoup.cpp: 5:08.00elapsed
With the many shared libs patch:
Full build: 38:23.59elapsed
Touching WebCore/platform/network/soup/ResourceHandleSoup.cpp: 1:34.33elapsed
There was no swapping at all in any of those (I closed most apps to not interfere with the tests) - I believe the shared libs patch may be a big help if normal debug builds are currently swapping (which is the case for me, usually), though.
I am a little surprised the third patch works -- I would think all the .so would have missing symbols pointing into each other. +webcore_built_objects = := $(patsubst %.cpp,%.lo,$(webcore_built_nosources)) I wonder if there are any .c files (perhaps generated ones?) that need to be in this list. But all of these patches look good to me if they work for you. I dislike the last patch. It introduces a separate build system for debug builds and due to that shares all the problems of a separate build system, among them: - more code to manage - more bugs - different bugs - different behavior (think performance here) I also think that a debug build should not deviate from a default build too much. And my gut feeling says that the last patch does too much of this. Did anybody ever talk to the gcc people about how to improve build times? Or talk to other people working on large C++ projects with similar issues (Open Office, Mozilla etc)? THey might have better ideas than splitting the build system. Chrome shared-link builds put all of webkit into a single .so. I think it was only about 1 min for incremental links once we removed all of the .la files (not just libwebcore, but also libwebcorejs and libjavascriptcore). Maybe that's an acceptable compromise. (In reply to comment #3) > I am a little surprised the third patch works -- I would think all the .so > would have missing symbols pointing into each other. > > +webcore_built_objects = := $(patsubst %.cpp,%.lo,$(webcore_built_nosources)) > I wonder if there are any .c files (perhaps generated ones?) that need to be in > this list. > > But all of these patches look good to me if they work for you. Yeah, I was a bit surprised as well. Only libwebcore_svg needed special handling there. All tests pass, and I was able to run Epiphany for a while with it, with no issues. (In reply to comment #4) > I dislike the last patch. It introduces a separate build system for debug > builds and due to that shares all the problems of a separate build system, > among them: > - more code to manage > - more bugs > - different bugs > - different behavior (think performance here) > I also think that a debug build should not deviate from a default build too > much. And my gut feeling says that the last patch does too much of this. Kinda. It does not really introduce a separate build system. It just changes how the linking happens. All the file lists are the same. About different behavior: debug buils are already heavily different in behavior, specially where performance is concerned. Having said that, I am also a bit uneasy about introducing different linkage rules for Debug builds, but given the amount of time it saves in incremental builds, I think it may be worth a try. I plan to also get rid of the libWebCoreJS, and do something to libJavaScriptCore - specially because we would like to have it shipping as a separate library - and maybe that would give us the same amount of benefits, regarding link time, but *shrug*. Created attachment 45575 [details]
Fix warnings for gperf calls
style-queue ran check-webkit-style on attachment 45575 [details] without any errors.
(In reply to comment #3) > I am a little surprised the third patch works -- I would think all the .so > would have missing symbols pointing into each other. > > +webcore_built_objects = := $(patsubst %.cpp,%.lo,$(webcore_built_nosources)) > I wonder if there are any .c files (perhaps generated ones?) that need to be in > this list. I don't think there are. $(webcore_built_nosources) comes from this: webcore_built_nosources += $(patsubst %.idl,DerivedSources/JS%.cpp,$(notdir $(IDL_BINDINGS_JS))) So we should be safe. Comment on attachment 45575 [details]
Fix warnings for gperf calls
LGTM. The Qt port also uses --include (same as -I) for invoking gpref.
Created attachment 45576 [details]
Remove libWebCore intermediate library, to improve link time.
style-queue ran check-webkit-style on attachment 45576 [details] without any errors.
Comment on attachment 45575 [details]
Fix warnings for gperf calls
Landed as 52603.
Comment on attachment 45576 [details]
Remove libWebCore intermediate library, to improve link time.
Makes sense to me, can't think of any downside to this and no reason to do it in the first place.
Comment on attachment 45576 [details] Remove libWebCore intermediate library, to improve link time. Landed as r52813. I've done a lot of work on this recently. For instance I've removed the remaining two intermediate libraries and removed the need to call autogen.sh for every build-webkit invocation. Our null build is about 14 seconds on my local machine. There's still a bit of work to do to remove the GNUmake lag (the time it takes to read the Makefiles and build the dependency graph), but I think they can be addressed in smaller bugs. So I'll close this bug for now. If anyone objects, please reopen. |
Created attachment 45470 [details] Fix warning during generation of DocTypeStrings.cpp During the hackfest we explored a bunch of strategies to make the build take less time, specially when doing incremental builds after touching a small number of files. We should push some proper changes implementing some of the ideas without breaking the build system =). My proposal is to make two changes: 1. adopt Evan Martin's awesome idea of linking all files only once, at the end, for release builds 2. make debug builds be many small shared libraries, to guarantee small build times when touching some files only I am posting patches that implement both. The first patch is actually a build fix for a gperf call that doesn't add the appropriate include. I am not marking the patches for review as of yet - I would like people to test them/agree to the changes before posting proper patches.