149161 2015-09-15 06:47:39 -0700 [cmake] Enable debug fission only if it is supported 2015-09-17 08:18:21 -0700 1 1 1 Unclassified WebKit New Bugs Other Unspecified Unspecified RESOLVED FIXED P2 Normal --- 148639 1 ossy ossy calvaris clopez commit-queue mrobinson ossy rhodovan.u-szeged oldest_to_newest 1125945 0 ossy 2015-09-15 06:47:39 -0700 https://trac.webkit.org/changeset/189179 broke the debug build with icecc, because debug fission isn't supported with icecc. We should enable debug fission only if the compiler accepts -gsplit-dwarf option. 1125946 1 261190 ossy 2015-09-15 06:49:27 -0700 Created attachment 261190 Patch 1125949 2 clopez 2015-09-15 07:10:28 -0700 (In reply to comment #0) > https://trac.webkit.org/changeset/189179 broke the debug build > with icecc, because debug fission isn't supported with icecc. > > We should enable debug fission only if the compiler accepts -gsplit-dwarf > option. I'm using succesfully icecc with clang and debug fission (also with ccache). The trick is to send your toolchain to the build slaves. You have to include /usr/bin/objcopy in the toolchain. Check the documentation here: https://trac.webkit.org/wiki/WebKitGTK/SpeedUpBuild#Usingiceccwithclang The interesting part is on 4.1 But, there is a bug on icecc that will cause that the buildslaves run out of disk space quickly. https://github.com/icecc/icecream/issues/140 1125950 3 ossy 2015-09-15 07:18:15 -0700 (In reply to comment #2) > (In reply to comment #0) > > https://trac.webkit.org/changeset/189179 broke the debug build > > with icecc, because debug fission isn't supported with icecc. > > > > We should enable debug fission only if the compiler accepts -gsplit-dwarf > > option. > > I'm using succesfully icecc with clang and debug fission (also with ccache). > > The trick is to send your toolchain to the build slaves. You have to include > /usr/bin/objcopy in the toolchain. > > Check the documentation here: > https://trac.webkit.org/wiki/WebKitGTK/SpeedUpBuild#Usingiceccwithclang The > interesting part is on 4.1 > > But, there is a bug on icecc that will cause that the buildslaves run out of > disk space quickly. https://github.com/icecc/icecream/issues/140 I experienced with GCC only and it doesn't work at all and won't in the near future. (until icecc and/or gcc is fixed) Copying objcopy isn't enough, you have to made icecc to send back dwo files. And it is still not enough, because object files contains hard coded paths ( eg: /tmp/... ) to the responsible dwo files. 1125951 4 clopez 2015-09-15 07:30:35 -0700 (In reply to comment #3) > (In reply to comment #2) > > (In reply to comment #0) > > > https://trac.webkit.org/changeset/189179 broke the debug build > > > with icecc, because debug fission isn't supported with icecc. > > > > > > We should enable debug fission only if the compiler accepts -gsplit-dwarf > > > option. > > > > I'm using succesfully icecc with clang and debug fission (also with ccache). > > > > The trick is to send your toolchain to the build slaves. You have to include > > /usr/bin/objcopy in the toolchain. > > > > Check the documentation here: > > https://trac.webkit.org/wiki/WebKitGTK/SpeedUpBuild#Usingiceccwithclang The > > interesting part is on 4.1 > > > > But, there is a bug on icecc that will cause that the buildslaves run out of > > disk space quickly. https://github.com/icecc/icecream/issues/140 > > I experienced with GCC only and it doesn't work at all and > won't in the near future. (until icecc and/or gcc is fixed) > > Copying objcopy isn't enough, you have to made icecc to send > back dwo files. And it is still not enough, because object files > contains hard coded paths ( eg: /tmp/... ) to the responsible > dwo files. I have just tried it. It works fine for me also with GCC. The .dwo objects are copied back just fine. Please try to build your toolchain like this: /usr/lib/icecc/icecc-create-env --gcc /usr/bin/gcc /usr/bin/g++ --addfile /usr/bin/objcopy and before running the compilation export the env variable ICECC_VERSION to the full path of the tar.gz that is created. You can do this in one shot with something like: export ICECC_VERSION="$(pwd)/$(/usr/lib/icecc/icecc-create-env --gcc /usr/bin/gcc /usr/bin/g++ --addfile /usr/bin/objcopy|tee /dev/stderr|grep creating|awk '{print $2}')" I'm using GCC 4.9.2 and and icecc 1.0.1 1125957 5 261190 commit-queue 2015-09-15 08:01:24 -0700 Comment on attachment 261190 Patch Clearing flags on attachment: 261190 Committed r189807: <http://trac.webkit.org/changeset/189807> 1125958 6 commit-queue 2015-09-15 08:01:37 -0700 All reviewed patches have been landed. Closing bug. 1126395 7 ossy 2015-09-16 10:35:32 -0700 (In reply to comment #4) > (In reply to comment #3) > > (In reply to comment #2) > > > (In reply to comment #0) > > > > https://trac.webkit.org/changeset/189179 broke the debug build > > > > with icecc, because debug fission isn't supported with icecc. > > > > > > > > We should enable debug fission only if the compiler accepts -gsplit-dwarf > > > > option. > > > > > > I'm using succesfully icecc with clang and debug fission (also with ccache). > > > > > > The trick is to send your toolchain to the build slaves. You have to include > > > /usr/bin/objcopy in the toolchain. > > > > > > Check the documentation here: > > > https://trac.webkit.org/wiki/WebKitGTK/SpeedUpBuild#Usingiceccwithclang The > > > interesting part is on 4.1 > > > > > > But, there is a bug on icecc that will cause that the buildslaves run out of > > > disk space quickly. https://github.com/icecc/icecream/issues/140 > > > > I experienced with GCC only and it doesn't work at all and > > won't in the near future. (until icecc and/or gcc is fixed) > > > > Copying objcopy isn't enough, you have to made icecc to send > > back dwo files. And it is still not enough, because object files > > contains hard coded paths ( eg: /tmp/... ) to the responsible > > dwo files. > > I have just tried it. > > It works fine for me also with GCC. The .dwo objects are copied back just > fine. > > Please try to build your toolchain like this: > > /usr/lib/icecc/icecc-create-env --gcc /usr/bin/gcc /usr/bin/g++ --addfile > /usr/bin/objcopy > > and before running the compilation export the env variable ICECC_VERSION to > the full path of the tar.gz that is created. > > You can do this in one shot with something like: > > export ICECC_VERSION="$(pwd)/$(/usr/lib/icecc/icecc-create-env --gcc > /usr/bin/gcc /usr/bin/g++ --addfile /usr/bin/objcopy|tee /dev/stderr|grep > creating|awk '{print $2}')" > > > > I'm using GCC 4.9.2 and and icecc 1.0.1 I can't believe it. I tried it again, because you seemed too confident, but it doesn't work at all as I said previously. (Ubuntu 15.04. , GCC 4.9.2, icecc 1.0.1, +objcopy included in the tar.gz) Icecc doesn't contain any code path to copy dwo files back to the host where you are building. Icecc doesn't do anything with dwo files, just fill the hard disk of the slaves for nothing. Additionally if you fixed icecc somehow to copy back dwo files, you won't be able to use them for anything, because object files contains wrong paths to the dwo files. (the tmp path of the icecc slaves) Could you possibly explain how did you manage to make icecc+GCC work with debug fission (without ccache)? Maybe ccache uses some tricks to workaround the wrong hard coded dwo path in object files and additionally regenerate dwo files again on that side where you do the build. Could you confirm it please? 1126431 8 ossy 2015-09-16 12:03:05 -0700 (In reply to comment #7) > Maybe ccache uses some tricks to workaround the wrong hard coded > dwo path in object files and additionally regenerate dwo files > again on that side where you do the build. Could you confirm it please? I checked the latest ccache release, it really does the trick somehow. I still don't understand how, but it does. 1126435 9 clopez 2015-09-16 12:35:42 -0700 (In reply to comment #8) > (In reply to comment #7) > > Maybe ccache uses some tricks to workaround the wrong hard coded > > dwo path in object files and additionally regenerate dwo files > > again on that side where you do the build. Could you confirm it please? > > I checked the latest ccache release, it really does the > trick somehow. I still don't understand how, but it does. Yes, I'm using ccache+icecc+clang|gcc. But I have just tested to remove ccache from the mix, and it seems to work ok also. What I suspect may be happening to you is that your buildslaves have cached an old version of your toolchain (the one that didn't includes the objcopy binary). The buildslaves use the filename of the tarball as an index to check if they already have it. If you always use the same filename for the tarball, your buildslaves are not going to update your toolchain when you really want them to update it. If you send as toolchain a file named "my-toolchain.tar.gz", icecc won't upload your toolchain if the directory "/var/cache/icecc/target=x86_64/my-toolchain" already exists on the buildslave. So my recommendation is to use as filename the one that icecc generates by default (that is a hash of the files included on the toolchain). 1126655 10 ossy 2015-09-17 02:20:16 -0700 (In reply to comment #9) > Yes, I'm using ccache+icecc+clang|gcc. But I have just tested to remove > ccache from the mix, and it seems to work ok also. No, it is impossible with stock icecc. Nor icecc 1.0.1, neither trunk icecc supports sending back the dwo files. Additionally trunk icecc forces local build if you pass -gsplit-dwarf command line option. icecc commit: ee5c312675ae099ca9a42b9d6ca180a8e9c85e76 > What I suspect may be happening to you is that your buildslaves have cached > an old version of your toolchain (the one that didn't includes the objcopy > binary). ... I know icecc enough not to make this begginners fault. Of course the good toolchain is sent and dwo files are generated on the slave side. But it isn't sent back, because icecc doesn't know anything about it. I'm pretty sure that ccache does the trick somehow, but I don't know how. 1126666 11 ossy 2015-09-17 03:24:32 -0700 One more thing, ccache+icecc+gcc combo works only if the icecc jobs are sent to the machine which you build on, because icecc does local builds in place instead of creating chroot. So I still say debug fission doesn't work at all with icecc. 1126667 12 ossy 2015-09-17 03:28:54 -0700 (In reply to comment #11) > One more thing, ccache+icecc+gcc combo works only if the icecc jobs are > sent to the machine which you build on, because icecc does local builds > in place instead of creating chroot. > > So I still say debug fission doesn't work at all with icecc. And it won't work with newer icecc which will chrooting to the environment for local builds too since 1e8dc310ff11bb048a2855544040ab7255db7061 . 1126668 13 clopez 2015-09-17 03:50:56 -0700 (In reply to comment #10) > (In reply to comment #9) > > Yes, I'm using ccache+icecc+clang|gcc. But I have just tested to remove > > ccache from the mix, and it seems to work ok also. > > No, it is impossible with stock icecc. Nor icecc 1.0.1, > neither trunk icecc supports sending back the dwo files. > Additionally trunk icecc forces local build if you pass -gsplit-dwarf command > line option. icecc commit: ee5c312675ae099ca9a42b9d6ca180a8e9c85e76 > That change has been done after the release 1.0.1 (which is what I'm using). I'm using icecc=1.0.1-2 directly from Debian (no modifications), and just to be double-sure I just downloaded the sources with "apt-get source icecc", and grepped it for "gsplit-dwarf" strings. Nothing showed. > > What I suspect may be happening to you is that your buildslaves have cached > > an old version of your toolchain (the one that didn't includes the objcopy > > binary). > ... > > I know icecc enough not to make this begginners fault. Of course the good > toolchain is sent and dwo files are generated on the slave side. But > it isn't sent back, because icecc doesn't know anything about it. > > I'm pretty sure that ccache does the trick somehow, but I don't know how. I just tested (again, to be double sure): * Building webkitgtk+ on debug mode (from zero, a clean build) using icecc=1.0.1-2 (with GCC=4.9.2-10) (and without ccache): $ export PATH="/usr/lib/icecc/bin:${PATH}" $ export ICECC_VERSION="$(pwd)/$(/usr/lib/icecc/icecc-create-env --gcc /usr/bin/gcc /usr/bin/g++ --addfile /usr/bin/objcopy|tee /dev/stderr|grep creating|awk '{print $2}')" $ rm -fr WebKitBuild/Debug $ Tools/Scripts/build-webkit --debug --gtk --makeargs=-j48 --cmakeargs=-j48 And: * The .dwo files are generated correctly on the WebKitBuild/Debug directory. $ find WebKitBuild/Debug -type f -name '*.dwo'|wc -l 62 * The jobs are correctly dispatched to all build slaves (double-checked it by looking at the console with the icecream monitor (icemon) ) * The build finishes without trouble. * ccache was not used (double-checked by ensuring it was not on the PATH as also by checking that the output of "ccache -s" before and after the compilation was exactly the same) 1126669 14 clopez 2015-09-17 03:52:16 -0700 (In reply to comment #12) > (In reply to comment #11) > > One more thing, ccache+icecc+gcc combo works only if the icecc jobs are > > sent to the machine which you build on, because icecc does local builds > > in place instead of creating chroot. > > > > So I still say debug fission doesn't work at all with icecc. > > And it won't work with newer icecc which will chrooting to the environment > for local builds too since 1e8dc310ff11bb048a2855544040ab7255db7061 . That is also post 1.0.1 1126671 15 calvaris 2015-09-17 04:07:16 -0700 I am using the same setup with CCache and IceCC that Carlos described. Frankly I haven't devoted any time to check this in depth (as I have Carlos less than 10m away), but I can build with Fission, with and without IceCC. The only thing that I had to do at the beginning was clearing the cache of CCache (ccache -Cz). It it true that when later, when using GDB I am getting a lot of errors that some dwo files can't be found, but I don't know why, symbols are there I don't have any issue with the debugging process. For the record, currently I am using Debian testing with ccache 3.2.3-1 and icecc 1.0.1-2+b1. 1126672 16 ossy 2015-09-17 04:41:14 -0700 (In reply to comment #13) > * The .dwo files are generated correctly on the WebKitBuild/Debug > directory. > > $ find WebKitBuild/Debug -type f -name '*.dwo'|wc -l > 62 > > * The jobs are correctly dispatched to all build slaves (double-checked it > by looking at the console with the icecream monitor (icemon) ) > * The build finishes without trouble. > * ccache was not used (double-checked by ensuring it was not on the PATH > as also by checking that the output of "ccache -s" before and after the > compilation was exactly the same) That's exactly same what I said previously. There are ~4000 object files, but you have only 62 dwo files. You have dwo files for the objects which were built locally, because icecc built them in place instead of chroot. 1126691 17 clopez 2015-09-17 07:43:20 -0700 (In reply to comment #16) > (In reply to comment #13) > > * The .dwo files are generated correctly on the WebKitBuild/Debug > > directory. > > > > $ find WebKitBuild/Debug -type f -name '*.dwo'|wc -l > > 62 > > > > * The jobs are correctly dispatched to all build slaves (double-checked it > > by looking at the console with the icecream monitor (icemon) ) > > * The build finishes without trouble. > > * ccache was not used (double-checked by ensuring it was not on the PATH > > as also by checking that the output of "ccache -s" before and after the > > compilation was exactly the same) > > That's exactly same what I said previously. There are ~4000 object files, > but you have only 62 dwo files. You have dwo files for the objects which > were built locally, because icecc built them in place instead of chroot. And how the build finished correctly? Doesn't it needs the dwo files for linking? The debugger seems to work fine also. :\ ? 1126696 18 clopez 2015-09-17 08:18:21 -0700 I tried re-building now with ccache+icecc+gcc with a clean ccache cache and the behaviour is the same: It builds fine but it only generates a few .dwo files (I guess the ones that were built locally). So I think ccache is not fixing anything. The thing is that it builds fine without the dwo files. Also the debugger (so far) seems to work ok. I'm re-reading the doc: "We can move the .debug_info, .debug_types, and .debug_str sections from the object (.o) files to a separate DWARF object (.dwo) file (or ".dsym", in Apple's nomenclature). Assuming that we could ignore these .dwo files during the link step, this would remove the bulk of the data that would be sent to the linker, and debug builds would be be much closer to non-debug builds in terms of object file size and link speed. [...] In order for the debugger to be able to locate and process the information in raw (unrelocated) .dwo files, some information must still be left behind in the .o files for the linker to combine and relocate. The bulk of the reduction in .o file size will come from moving .debug_info, .debug_types, .debug_loc, and .debug_str sections into the .dwo file (and eliminating most of the associated relocations). The .debug_ranges, .debug_line, .debug_pubnames, .debug_pubtypes, .debug_aranges, and .debug_gdb_scripts sections (and their relocation sections) will remain in the .o file. " https://gcc.gnu.org/wiki/DebugFission So if I understood this right, the dwo files are not needed for linking, and the basic debug information (like in release builds) is still contained on the .o files. That explains why it builds fine and I can run the debugger apparently fine (I guess I'm just missing some extra debug information for not having all the dwo files). 261190 2015-09-15 06:49:27 -0700 2015-09-15 08:01:24 -0700 Patch bug-149161-20150915064915.patch text/plain 1279 ossy U3VidmVyc2lvbiBSZXZpc2lvbjogMTg5ODA1CmRpZmYgLS1naXQgYS9Tb3VyY2UvY21ha2UvT3B0 aW9uc0NvbW1vbi5jbWFrZSBiL1NvdXJjZS9jbWFrZS9PcHRpb25zQ29tbW9uLmNtYWtlCmluZGV4 IDU2ZjcyMGQ5OWUzOWVlNmUzYzNmNTI3NGQ1NDY1ZTAyYmE4MmMyYjkuLmY2Y2FjNjZkN2VlZTk0 Y2FlYjE5NGFlMTQ2MWZjNDRiY2U4MDNmOGUgMTAwNjQ0Ci0tLSBhL1NvdXJjZS9jbWFrZS9PcHRp b25zQ29tbW9uLmNtYWtlCisrKyBiL1NvdXJjZS9jbWFrZS9PcHRpb25zQ29tbW9uLmNtYWtlCkBA IC04NSw3ICs4NSwxMSBAQCBlbmRpZiAoKQogCiBzZXQoRU5BQkxFX0RFQlVHX0ZJU1NJT05fREVG QVVMVCBPRkYpCiBpZiAoVVNFX0xEX0dPTEQgQU5EIENNQUtFX0JVSUxEX1RZUEUgU1RSRVFVQUwg IkRlYnVnIikKLSAgICBzZXQoRU5BQkxFX0RFQlVHX0ZJU1NJT05fREVGQVVMVCBPTikKKyAgICBp bmNsdWRlKFRlc3RDWFhBY2NlcHRzRmxhZykKKyAgICBDSEVDS19DWFhfQUNDRVBUU19GTEFHKC1n c3BsaXQtZHdhcmYgQ1hYX0FDQ0VQVFNfR1NQTElUX0RXQVJGKQorICAgIGlmIChDWFhfQUNDRVBU U19HU1BMSVRfRFdBUkYpCisgICAgICAgIHNldChFTkFCTEVfREVCVUdfRklTU0lPTl9ERUZBVUxU IE9OKQorICAgIGVuZGlmICgpCiBlbmRpZiAoKQogCiBvcHRpb24oREVCVUdfRklTU0lPTiAiVXNl IERlYnVnIEZpc3Npb24gc3VwcG9ydCIgJHtFTkFCTEVfREVCVUdfRklTU0lPTl9ERUZBVUxUfSkK ZGlmZiAtLWdpdCBhL0NoYW5nZUxvZyBiL0NoYW5nZUxvZwppbmRleCA3NmEwNzdiYzgzMDJmYmVh NTczYjJiNDMzYWQ4NmRhNmE5YjA5MmE4Li5hM2YyOWQzNTA2OGFhYWYyNzNhODU5YjQwOTllZDZm ZTg1ODRkNmMwIDEwMDY0NAotLS0gYS9DaGFuZ2VMb2cKKysrIGIvQ2hhbmdlTG9nCkBAIC0xLDMg KzEsMTIgQEAKKzIwMTUtMDktMTUgIENzYWJhIE9zenRyb2dvbsOhYyAgPG9zc3lAd2Via2l0Lm9y Zz4KKworICAgICAgICBbY21ha2VdIEVuYWJsZSBkZWJ1ZyBmaXNzaW9uIG9ubHkgaWYgaXQgaXMg c3VwcG9ydGVkCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9p ZD0xNDkxNjEKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAg ICAqIFNvdXJjZS9jbWFrZS9PcHRpb25zQ29tbW9uLmNtYWtlOgorCiAyMDE1LTA5LTE0ICBBbGV4 IENocmlzdGVuc2VuICA8YWNocmlzdGVuc2VuQHdlYmtpdC5vcmc+CiAKICAgICAgICAgUHJvZ3Jl c3MgdG93YXJkcyBDTWFrZSBvbiBNYWMuCg==