Bug 149161 – [cmake] Enable debug fission only if it is supported

• Component: New Bugs
• Status: RESOLVED FIXED

[Csaba Osztrogonác]

https://trac.webkit.org/changeset/189179 broke the debug build 
with icecc, because debug fission isn't supported with icecc.

We should enable debug fission only if the compiler accepts -gsplit-dwarf option.

[Csaba Osztrogonác]

Created attachment 261190 [details]
Patch

[Carlos Alberto Lopez Perez]

(In reply to comment #0)
> https://trac.webkit.org/changeset/189179 broke the debug build 
> with icecc, because debug fission isn't supported with icecc.
> 
> We should enable debug fission only if the compiler accepts -gsplit-dwarf
> option.

I'm using succesfully icecc with clang and debug fission (also with ccache).

The trick is to send your toolchain to the build slaves. You have to include /usr/bin/objcopy in the toolchain.

Check the documentation here: https://trac.webkit.org/wiki/WebKitGTK/SpeedUpBuild#Usingiceccwithclang The interesting part is on 4.1

But, there is a bug on icecc that will cause that the buildslaves run out of disk space quickly. https://github.com/icecc/icecream/issues/140

[Csaba Osztrogonác]

(In reply to comment #2)
> (In reply to comment #0)
> > https://trac.webkit.org/changeset/189179 broke the debug build 
> > with icecc, because debug fission isn't supported with icecc.
> > 
> > We should enable debug fission only if the compiler accepts -gsplit-dwarf
> > option.
> 
> I'm using succesfully icecc with clang and debug fission (also with ccache).
> 
> The trick is to send your toolchain to the build slaves. You have to include
> /usr/bin/objcopy in the toolchain.
> 
> Check the documentation here:
> https://trac.webkit.org/wiki/WebKitGTK/SpeedUpBuild#Usingiceccwithclang The
> interesting part is on 4.1
> 
> But, there is a bug on icecc that will cause that the buildslaves run out of
> disk space quickly. https://github.com/icecc/icecream/issues/140

I experienced with GCC only and it doesn't work at all and 
won't in the near future. (until icecc and/or gcc is fixed)

Copying objcopy isn't enough, you have to made icecc to send
back dwo files. And it is still not enough, because object files
contains hard coded paths ( eg: /tmp/... ) to the responsible
dwo files.

[Carlos Alberto Lopez Perez]

(In reply to comment #3)
> (In reply to comment #2)
> > (In reply to comment #0)
> > > https://trac.webkit.org/changeset/189179 broke the debug build 
> > > with icecc, because debug fission isn't supported with icecc.
> > > 
> > > We should enable debug fission only if the compiler accepts -gsplit-dwarf
> > > option.
> > 
> > I'm using succesfully icecc with clang and debug fission (also with ccache).
> > 
> > The trick is to send your toolchain to the build slaves. You have to include
> > /usr/bin/objcopy in the toolchain.
> > 
> > Check the documentation here:
> > https://trac.webkit.org/wiki/WebKitGTK/SpeedUpBuild#Usingiceccwithclang The
> > interesting part is on 4.1
> > 
> > But, there is a bug on icecc that will cause that the buildslaves run out of
> > disk space quickly. https://github.com/icecc/icecream/issues/140
> 
> I experienced with GCC only and it doesn't work at all and 
> won't in the near future. (until icecc and/or gcc is fixed)
> 
> Copying objcopy isn't enough, you have to made icecc to send
> back dwo files. And it is still not enough, because object files
> contains hard coded paths ( eg: /tmp/... ) to the responsible
> dwo files.

I have just tried it.

It works fine for me also with GCC. The .dwo objects are copied back just fine.

Please try to build your toolchain like this:

/usr/lib/icecc/icecc-create-env --gcc /usr/bin/gcc /usr/bin/g++ --addfile /usr/bin/objcopy

and before running the compilation export the env variable ICECC_VERSION to the full path of the tar.gz that is created.

You can do this in one shot with something like:

export ICECC_VERSION="$(pwd)/$(/usr/lib/icecc/icecc-create-env --gcc /usr/bin/gcc /usr/bin/g++ --addfile /usr/bin/objcopy|tee /dev/stderr|grep creating|awk '{print $2}')"



I'm using GCC 4.9.2 and and icecc 1.0.1

[WebKit Commit Bot]

Comment on attachment 261190 [details]
Patch

Clearing flags on attachment: 261190

Committed r189807: <http://trac.webkit.org/changeset/189807>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Csaba Osztrogonác]

(In reply to comment #4)
> (In reply to comment #3)
> > (In reply to comment #2)
> > > (In reply to comment #0)
> > > > https://trac.webkit.org/changeset/189179 broke the debug build 
> > > > with icecc, because debug fission isn't supported with icecc.
> > > > 
> > > > We should enable debug fission only if the compiler accepts -gsplit-dwarf
> > > > option.
> > > 
> > > I'm using succesfully icecc with clang and debug fission (also with ccache).
> > > 
> > > The trick is to send your toolchain to the build slaves. You have to include
> > > /usr/bin/objcopy in the toolchain.
> > > 
> > > Check the documentation here:
> > > https://trac.webkit.org/wiki/WebKitGTK/SpeedUpBuild#Usingiceccwithclang The
> > > interesting part is on 4.1
> > > 
> > > But, there is a bug on icecc that will cause that the buildslaves run out of
> > > disk space quickly. https://github.com/icecc/icecream/issues/140
> > 
> > I experienced with GCC only and it doesn't work at all and 
> > won't in the near future. (until icecc and/or gcc is fixed)
> > 
> > Copying objcopy isn't enough, you have to made icecc to send
> > back dwo files. And it is still not enough, because object files
> > contains hard coded paths ( eg: /tmp/... ) to the responsible
> > dwo files.
> 
> I have just tried it.
> 
> It works fine for me also with GCC. The .dwo objects are copied back just
> fine.
> 
> Please try to build your toolchain like this:
> 
> /usr/lib/icecc/icecc-create-env --gcc /usr/bin/gcc /usr/bin/g++ --addfile
> /usr/bin/objcopy
> 
> and before running the compilation export the env variable ICECC_VERSION to
> the full path of the tar.gz that is created.
> 
> You can do this in one shot with something like:
> 
> export ICECC_VERSION="$(pwd)/$(/usr/lib/icecc/icecc-create-env --gcc
> /usr/bin/gcc /usr/bin/g++ --addfile /usr/bin/objcopy|tee /dev/stderr|grep
> creating|awk '{print $2}')"
> 
> 
> 
> I'm using GCC 4.9.2 and and icecc 1.0.1

I can't believe it. I tried it again, because you seemed too confident,
but it doesn't work at all as I said previously. (Ubuntu 15.04. ,
GCC 4.9.2, icecc 1.0.1, +objcopy included in the tar.gz)

Icecc doesn't contain any code path to copy dwo files back to the host
where you are building. Icecc doesn't do anything with dwo files, just
fill the hard disk of the slaves for nothing.

Additionally if you fixed icecc somehow to copy back dwo files, you
won't be able to use them for anything, because object files contains
wrong paths to the dwo files. (the tmp path of the icecc slaves)

Could you possibly explain how did you manage to make 
icecc+GCC work with debug fission (without ccache)?

Maybe ccache uses some tricks to workaround the wrong hard coded
dwo path in object files and additionally regenerate dwo files
again on that side where you do the build. Could you confirm it please?

[Csaba Osztrogonác]

(In reply to comment #7)
> Maybe ccache uses some tricks to workaround the wrong hard coded
> dwo path in object files and additionally regenerate dwo files
> again on that side where you do the build. Could you confirm it please?

I checked the latest ccache release, it really does the 
trick somehow. I still don't understand how, but it does.

[Carlos Alberto Lopez Perez]

(In reply to comment #8)
> (In reply to comment #7)
> > Maybe ccache uses some tricks to workaround the wrong hard coded
> > dwo path in object files and additionally regenerate dwo files
> > again on that side where you do the build. Could you confirm it please?
> 
> I checked the latest ccache release, it really does the 
> trick somehow. I still don't understand how, but it does.

Yes, I'm using ccache+icecc+clang|gcc. But I have just tested to remove ccache from the mix, and it seems to work ok also.

What I suspect may be happening to you is that your buildslaves have cached an old version of your toolchain (the one that didn't includes the objcopy binary).

The buildslaves use the filename of the tarball as an index to check if they already have it. If you always use the same filename for the tarball, your buildslaves are not going to update your toolchain when you really want them to update it.

If you send as toolchain a file named "my-toolchain.tar.gz", icecc won't upload your toolchain if the directory "/var/cache/icecc/target=x86_64/my-toolchain" already exists on the buildslave.

So my recommendation is to use as filename the one that icecc generates by default (that is a hash of the files included on the toolchain).

[Csaba Osztrogonác]

(In reply to comment #9)
> Yes, I'm using ccache+icecc+clang|gcc. But I have just tested to remove
> ccache from the mix, and it seems to work ok also.

No, it is impossible with stock icecc. Nor icecc 1.0.1, 
neither trunk icecc supports sending back the dwo files.
Additionally trunk icecc forces local build if you pass -gsplit-dwarf command
line option. icecc commit: ee5c312675ae099ca9a42b9d6ca180a8e9c85e76
 
> What I suspect may be happening to you is that your buildslaves have cached
> an old version of your toolchain (the one that didn't includes the objcopy
> binary).
...

I know icecc enough not to make this begginners fault. Of course the good
toolchain is sent and dwo files are generated on the slave side. But
it isn't sent back, because icecc doesn't know anything about it.

I'm pretty sure that ccache does the trick somehow, but I don't know how.

[Csaba Osztrogonác]

One more thing, ccache+icecc+gcc combo works only if the icecc jobs are
sent to the machine which you build on, because icecc does local builds
in place instead of creating chroot.

So I still say debug fission doesn't work at all with icecc.

[Csaba Osztrogonác]

(In reply to comment #11)
> One more thing, ccache+icecc+gcc combo works only if the icecc jobs are
> sent to the machine which you build on, because icecc does local builds
> in place instead of creating chroot.
> 
> So I still say debug fission doesn't work at all with icecc.

And it won't work with newer icecc which will chrooting to the environment 
for local builds too since 1e8dc310ff11bb048a2855544040ab7255db7061 .

[Carlos Alberto Lopez Perez]

(In reply to comment #10)
> (In reply to comment #9)
> > Yes, I'm using ccache+icecc+clang|gcc. But I have just tested to remove
> > ccache from the mix, and it seems to work ok also.
> 
> No, it is impossible with stock icecc. Nor icecc 1.0.1, 
> neither trunk icecc supports sending back the dwo files.
> Additionally trunk icecc forces local build if you pass -gsplit-dwarf command
> line option. icecc commit: ee5c312675ae099ca9a42b9d6ca180a8e9c85e76
>  

That change has been done after the release 1.0.1 (which is what I'm using).

I'm using icecc=1.0.1-2 directly from Debian (no modifications), and just to be double-sure I just downloaded the sources with "apt-get source icecc", and grepped it for "gsplit-dwarf" strings. Nothing showed.


> > What I suspect may be happening to you is that your buildslaves have cached
> > an old version of your toolchain (the one that didn't includes the objcopy
> > binary).
> ...
> 
> I know icecc enough not to make this begginners fault. Of course the good
> toolchain is sent and dwo files are generated on the slave side. But
> it isn't sent back, because icecc doesn't know anything about it.
> 
> I'm pretty sure that ccache does the trick somehow, but I don't know how.

I just tested (again, to be double sure):

* Building webkitgtk+ on debug mode (from zero, a clean build) using icecc=1.0.1-2 (with GCC=4.9.2-10) (and without ccache):

$ export PATH="/usr/lib/icecc/bin:${PATH}"
$ export ICECC_VERSION="$(pwd)/$(/usr/lib/icecc/icecc-create-env --gcc /usr/bin/gcc /usr/bin/g++ --addfile /usr/bin/objcopy|tee /dev/stderr|grep creating|awk '{print $2}')"
$ rm -fr WebKitBuild/Debug
$ Tools/Scripts/build-webkit --debug --gtk --makeargs=-j48 --cmakeargs=-j48

And:

  * The .dwo files are generated correctly on the WebKitBuild/Debug directory.

$ find WebKitBuild/Debug -type f -name '*.dwo'|wc -l
62

  * The jobs are correctly dispatched to all build slaves (double-checked it by looking at the console with the icecream monitor (icemon) )
  * The build finishes without trouble.
  * ccache was not used (double-checked by ensuring it was not on the PATH as also by checking that the output of "ccache -s" before and after the compilation was exactly the same)

[Carlos Alberto Lopez Perez]

(In reply to comment #12)
> (In reply to comment #11)
> > One more thing, ccache+icecc+gcc combo works only if the icecc jobs are
> > sent to the machine which you build on, because icecc does local builds
> > in place instead of creating chroot.
> > 
> > So I still say debug fission doesn't work at all with icecc.
> 
> And it won't work with newer icecc which will chrooting to the environment 
> for local builds too since 1e8dc310ff11bb048a2855544040ab7255db7061 .

That is also post 1.0.1

[Xabier Rodríguez Calvar]

I am using the same setup with CCache and IceCC that Carlos described. Frankly I haven't devoted any time to check this in depth (as I have Carlos less than 10m away), but I can build with Fission, with and without IceCC. The only thing that I had to do at the beginning was clearing the cache of CCache (ccache -Cz).

It it true that when later, when using GDB I am getting a lot of errors that some dwo files can't be found, but I don't know why, symbols are there I don't have any issue with the debugging process.

For the record, currently I am using Debian testing with ccache 3.2.3-1 and icecc                                 1.0.1-2+b1.

[Csaba Osztrogonác]

(In reply to comment #13)
>   * The .dwo files are generated correctly on the WebKitBuild/Debug
> directory.
> 
> $ find WebKitBuild/Debug -type f -name '*.dwo'|wc -l
> 62
> 
>   * The jobs are correctly dispatched to all build slaves (double-checked it
> by looking at the console with the icecream monitor (icemon) )
>   * The build finishes without trouble.
>   * ccache was not used (double-checked by ensuring it was not on the PATH
> as also by checking that the output of "ccache -s" before and after the
> compilation was exactly the same)

That's exactly same what I said previously. There are ~4000 object files,
but you have only 62 dwo files. You have dwo files for the objects which
were built locally, because icecc built them in place instead of chroot.

[Carlos Alberto Lopez Perez]

(In reply to comment #16)
> (In reply to comment #13)
> >   * The .dwo files are generated correctly on the WebKitBuild/Debug
> > directory.
> > 
> > $ find WebKitBuild/Debug -type f -name '*.dwo'|wc -l
> > 62
> > 
> >   * The jobs are correctly dispatched to all build slaves (double-checked it
> > by looking at the console with the icecream monitor (icemon) )
> >   * The build finishes without trouble.
> >   * ccache was not used (double-checked by ensuring it was not on the PATH
> > as also by checking that the output of "ccache -s" before and after the
> > compilation was exactly the same)
> 
> That's exactly same what I said previously. There are ~4000 object files,
> but you have only 62 dwo files. You have dwo files for the objects which
> were built locally, because icecc built them in place instead of chroot.

And how the build finished correctly? Doesn't it needs the dwo files for linking?

The debugger seems to work fine also. :\ ?

[Carlos Alberto Lopez Perez]

I tried re-building now with ccache+icecc+gcc with a clean ccache cache and the behaviour is the same:

It builds fine but it only generates a few .dwo files (I guess the ones that were built locally).

So I think ccache is not fixing anything. The thing is that it builds fine without the dwo files. Also the debugger (so far) seems to work ok.

I'm re-reading the doc:

"We can move the .debug_info, .debug_types, and .debug_str sections from the object (.o) files to a separate DWARF object (.dwo) file (or ".dsym", in Apple's nomenclature). Assuming that we could ignore these .dwo files during the link step, this would remove the bulk of the data that would be sent to the linker, and debug builds would be be much closer to non-debug builds in terms of object file size and link speed.
[...]
In order for the debugger to be able to locate and process the information in raw (unrelocated) .dwo files, some information must still be left behind in the .o files for the linker to combine and relocate. The bulk of the reduction in .o file size will come from moving .debug_info, .debug_types, .debug_loc, and .debug_str sections into the .dwo file (and eliminating most of the associated relocations). The .debug_ranges, .debug_line, .debug_pubnames, .debug_pubtypes, .debug_aranges, and .debug_gdb_scripts sections (and their relocation sections) will remain in the .o file.
"  https://gcc.gnu.org/wiki/DebugFission

So if I understood this right, the dwo files are not needed for linking, and the basic debug information (like in release builds) is still contained on the .o files.

That explains why it builds fine and I can run the debugger apparently fine (I guess I'm just missing some extra debug information for not having all the dwo files).