Bug 55484 – [Qt][WK2] Memory exhausted when building qtwebkit2 on linux

• Status: RESOLVED
• Resolution: FIXED
• Component: WebKit Qt

Getting a memory exhausted error when building qtwebkit2 on linux desktop.
Error occurs during the linking phase. 

./build-webkit --qt --debug  --qmakearg=CONFIG+=webkit2 --makeargs="-j10"

/usr/bin/ld: failed to set dynamic section sizes: Memory exhausted
collect2: ld returned 1 exit status
make[1]: *** [../lib/libQtWebKit.so.4.9.0] Error 1

[Benjamin Poulain]

And what exactly are you expecting from this bug report???

[Csaba Osztrogonác]

AFAIK you need minimum 3-4Gb of free memory for debug linking.

Created attachment 84276 [details]
Proposed Patch

Tried to build webkit2 on a 32 bit machine with 4G RAM. It is failing during the linking phase.
(probably hitting the per process memory limit). 

Using the "AllInOne" files from WebCore/rendering/svg/ and WebCore/svg/ we can get it to work. 
Modified webkit.pro to include RenderSVGAllInOne.cpp and SVGAllInOne.cpp.

[Benjamin Poulain]

Comment on attachment 84276 [details]
Proposed Patch

Best case make that an option. You should not break everyone debug build.

[Csaba Osztrogonác]

(In reply to comment #3)
> Created an attachment (id=84276) [details]
> Proposed Patch
> 
> Tried to build webkit2 on a 32 bit machine with 4G RAM. It is failing during the linking phase.
> (probably hitting the per process memory limit). 
> 
> Using the "AllInOne" files from WebCore/rendering/svg/ and WebCore/svg/ we can get it to work. 
> Modified webkit.pro to include RenderSVGAllInOne.cpp and SVGAllInOne.cpp.

It might solve linking problem for a while, but we will run into similar problem in the near future because of growing project. :(

Additionally all-in-one file will cause longer building time. Because 
if you touch only one svg file, all svg source will be rebuilt.

Reopen to discuss which one is more important. What do you think if we add a command line option to build-webkit for all-in-one file enabled or disabled.

[Chang Shu]

> Reopen to discuss which one is more important. What do you think if we add a command line option to build-webkit for all-in-one file enabled or disabled.

It seems to me you guys don't see this build problem in your env. But in Boston, people who use a 32-bit Ubuntu with at least 3G RAM are having trouble with the latest code. Having an option for build-webkit would be nice.

[Yael]

(In reply to comment #5)
> Reopen to discuss which one is more important. What do you think if we add a command line option to build-webkit for all-in-one file enabled or disabled.

It would be really nice to have this build option. ATM, the only I can make a debug build is if I disable svg and video. (Even with gold linker).

[Andras Becsi]

(In reply to comment #7)
> (In reply to comment #5)
> > Reopen to discuss which one is more important. What do you think if we add a command line option to build-webkit for all-in-one file enabled or disabled.
> 
> It would be really nice to have this build option. ATM, the only I can make a debug build is if I disable svg and video. (Even with gold linker).

Debug linking needs almost 4G of memory, but is not hitting the per process memory limit. If you have a build farm (icecc for ex.) mounting swap-files (or partitions) is a feasible work-around, so the build lasts approximately twice the time of a release build on my 32bit machine using gold linker.

However I also support the usage of opt-in all-in-one files, which would make the things a bit easier at least for a while.

[Benjamin Poulain]

I am not too fan of an option, I think that should work out of the box.

Only a few devs are still using 32 bits machines. I think it is reasonable to use the SVGAllInOne for linux+debug+32bits (you can check that in the .pro file). The other config would keep the regular SOURCES and enjoy the benefit of having those files separated :)

[Csaba Osztrogonác]

(In reply to comment #9)
> I am not too fan of an option, I think that should work out of the box.
> 
> Only a few devs are still using 32 bits machines. I think it is reasonable to use the SVGAllInOne for linux+debug+32bits (you can check that in the .pro file). The other config would keep the regular SOURCES and enjoy the benefit of having those files separated :)

Hm, it is a good idea. I'd like to see a fix like this:

# Comment: why we need it ...
linux-g++:CONFIG(debug, debug|release):isEqual(QT_ARCH,i386) {
    # all-in-one-file
} else {
    # separated files
}

Created attachment 84447 [details]
Patch

Using AllInOne files to avoid memory exhaustion for debug builds on 32 bit linux machines.

[Benjamin Poulain]

Comment on attachment 84447 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=84447&action=review

> Source/WebCore/ChangeLog:9
> +        builds on 32 bit liiux machines.

liiux -> Linux.

> Source/WebCore/ChangeLog:11
> +        No new tests.Fixing a build issue.

Space missing after the dot.

> Source/WebCore/WebCore.pro:3091
> +

No need for space here.

Created attachment 84449 [details]
Patch

[Benjamin Poulain]

Comment on attachment 84449 [details]
Patch

LGTM.
I can't r+ just yet since the bot won't take it into account.

[Csaba Osztrogonác]

Comment on attachment 84449 [details]
Patch

LGTM, r=me. cq-, because it seems the patch can't be applied. I'll land it manually.

[Csaba Osztrogonác]

Landed in http://trac.webkit.org/changeset/80169

[Laszlo Gombos]

Just out of curiosity, wouldn't this build option might result in more efficient (less binary size/faster code) code as the complier can optimize a set of files together instead of a file by file basis ? 

If that is the case (and we can perhaps even measure the difference) this might be a preferred build option for productization even though this option is not preferred during development. 

Anyone ?

[Csaba Osztrogonác]

linux-g++ check was incorrect, we have to use linux-g++* , 
fix landed in: http://trac.webkit.org/changeset/80212

[Csaba Osztrogonác]

(In reply to comment #17)
> Just out of curiosity, wouldn't this build option might result in more efficient (less binary size/faster code) code as the complier can optimize a set of files together instead of a file by file basis ? 
> 
> If that is the case (and we can perhaps even measure the difference) this might be a preferred build option for productization even though this option is not preferred during development. 
> 
> Anyone ?

You're right, it might be good for code size and/or performance.
But we can be sure if any volunteer do some measurements.

[Csaba Osztrogonác]

(In reply to comment #17)
> Just out of curiosity, wouldn't this build option might result in more efficient (less binary size/faster code) code as the complier can optimize a set of files together instead of a file by file basis ? 
> 
> If that is the case (and we can perhaps even measure the difference) this might be a preferred build option for productization even though this option is not preferred during development. 
> 
> Anyone ?

File size of stripped libQtWebKit.so in release mode on 32 bit:
-   with all-in-one SVG files : 26,899,312 bytes
- without all-in-one SVG files: 26,910,468 bytes
-                  difference :     11,156 bytes (0.041 %)

All-in-one file is the winner, but the difference is negligible.

Unfortunately we don't know any SVG performance benchmark, 
so I don't know if this change has performance impact or not.