Bug 118732 – Use of ar T option not supported by older binutils

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Landry Breuil]

OpenBSD still uses binutils 2.15, and our ar(1) doesn't support the 'T' flag used in various places :

./Tools/TestWebKitAPI/GNUmakefile.am:   $(AM_V_at)$(foreach archive, $(webcore_layer_archives), $(shell ar t $(archive) | xargs -n50 ar cruT $@))
./Source/WebKit2/GNUmakefile.am:        $(AM_V_at)$(foreach archive, $(webcore_layer_gtk2_archives), $(shell ar t $(archive) | xargs -n50 ar cruT $@))
./Source/autotools/SetupLibtool.m4:    AR_FLAGS="cruT"

https://bugs.webkit.org/show_bug.cgi?id=115918

It would be nice to test for its support before bindly assuming all platforms have it. It was added in binutils 2.19.1.

Maybe i can workaround it in SetupLibtool.m4 by setting AR_FLAGS in the env, but the two other occurences hardcode it.

[Landry Breuil]

So far, i've tried the following workarounds without success :

- Removing T from ar flags, just using cru:

ar t | ar cru fails because it cant find the object files composing all the $(archives) ? maybe an xargs issue, or a link/search path issue ?

- hacking makefile to avoid using a temporary library
-       $(AM_V_at)$(foreach archive, $(webcore_layer_gtk2_archives), $(shell ar t $(archive) | xargs -n50 ar cruT $@))
+       $(AM_V_at)ar cru $@ $(webcore_layer_gtk2_archives)

-       $(AM_V_at)$(foreach archive, $(webcore_layer_archives), $(shell ar t $(archive) | xargs -n50 ar cruT $@))
+       $(AM_V_at)ar cru $@ $(webcore_layer_archives)

this generates a WebCoreLayerGtk2.a without symbols, so WebKitPluginProcess cant link against it.

- using a linker script :

webkitgtk-2.1.3/ $cat WebCoreLayerGtk2.a                                                                                                            
GROUP (libPlatform.a libPlatformGtk2.a libWebCorePlatform.a libWebCoreGtk2.a libWebCore.a libWebCoreModules.a libWebCoreSVG.a libANGLE.a)

=> blows at link time with tons of undefined refs. I suppose fallout from the circular deps between all the libs forming WebCoreLayerGtk2.a ?

- using --start-group $(webcore_layer_gtk2_archives) -end-group => libtool reorders flags, and messes up with linking order

So as of now, building with webkit2 enabled is not possible for me.

[Landry Breuil]

(In reply to comment #1)
> So far, i've tried the following workarounds without success :
> 
> - Removing T from ar flags, just using cru:
> 
> ar t | ar cru fails because it cant find the object files composing all the $(archives) ? maybe an xargs issue, or a link/search path issue ?

More on this :

$ar t .libs/libPlatformGtk2.a | head -2                   
libPlatformGtk2_la-AudioBusGtk.o
libPlatformGtk2_la-GraphicsContextCairo.o
$find . -name libPlatformGtk2_la-AudioBusGtk.o
./Source/WebCore/platform/audio/gtk/.libs/libPlatformGtk2_la-AudioBusGtk.o

how is ar cru supposed to reach that object file when archiving WebCoreLayerGtk2.a ?

[Zan Dobersek]

(In reply to comment #2)
> (In reply to comment #1)
> > So far, i've tried the following workarounds without success :
> > 
> > - Removing T from ar flags, just using cru:
> > 
> > ar t | ar cru fails because it cant find the object files composing all the $(archives) ? maybe an xargs issue, or a link/search path issue ?
> 
> More on this :
> 
> $ar t .libs/libPlatformGtk2.a | head -2                   
> libPlatformGtk2_la-AudioBusGtk.o
> libPlatformGtk2_la-GraphicsContextCairo.o
> $find . -name libPlatformGtk2_la-AudioBusGtk.o
> ./Source/WebCore/platform/audio/gtk/.libs/libPlatformGtk2_la-AudioBusGtk.o
> 
> how is ar cru supposed to reach that object file when archiving WebCoreLayerGtk2.a ?

Don't know exactly what's going on in .libs/libPlatformGtk2.a, but the object file path should be of relative form and pointing to the object file, in AudioBusGtk.o case that's in Source/WebCore/platform/audio/gtk/.libs/.

[Zan Dobersek]

(In reply to comment #0)
> Maybe i can workaround it in SetupLibtool.m4 by setting AR_FLAGS in the env, but the two other occurences hardcode it.

AR_FLAGS should be taken into the account here I think. Would you be willing to try out a patch if/when the archive member file path output issue (comment #2) is resolved?

[Landry Breuil]

(In reply to comment #3)
> (In reply to comment #2)
> > (In reply to comment #1)
> > > So far, i've tried the following workarounds without success :
> > > 
> > > - Removing T from ar flags, just using cru:
> > > 
> > > ar t | ar cru fails because it cant find the object files composing all the $(archives) ? maybe an xargs issue, or a link/search path issue ?
> > 
> > More on this :
> > 
> > $ar t .libs/libPlatformGtk2.a | head -2                   
> > libPlatformGtk2_la-AudioBusGtk.o
> > libPlatformGtk2_la-GraphicsContextCairo.o
> > $find . -name libPlatformGtk2_la-AudioBusGtk.o
> > ./Source/WebCore/platform/audio/gtk/.libs/libPlatformGtk2_la-AudioBusGtk.o
> > 
> > how is ar cru supposed to reach that object file when archiving WebCoreLayerGtk2.a ?
> 
> Don't know exactly what's going on in .libs/libPlatformGtk2.a, but the object file path should be of relative form and pointing to the object file, in AudioBusGtk.o case that's in Source/WebCore/platform/audio/gtk/.libs/.

I dont know what's supposed to be used when generating .libs/libPlatformGtk2.a but my ar(1) manpage states for option P :

       P   Use the full path name when matching names in the archive.  GNU ar
           can not create an archive with a full path name (such archives are
           not POSIX complaint), but other archive creators can.  This option
           will cause GNU ar to match file names using a complete path name,
           which can be convenient when extracting a single file from an
           archive created by another tool.

Is webkit expecting archives to be built with another tool, and not posix compliant ? or a more recent ar/binutils supporting some specific options is required ?

[Landry Breuil]

Fwiw, libtool indeed calls ar cru with the full path to object files :

libtool: link: ar cru .libs/libWebKit2Platform.a Source/WebKit2/Platform/CoreIPC/.libs/libWebKit2Platform_la-ArgumentCoders.o Source/WebKit2/Platform/CoreIPC/.libs/libWebKit2Platform_la-ArgumentDecoder.o Source/WebKit2/Platform/CoreIPC/.libs/libWebKit2Platform_la-ArgumentEncoder.o Source/WebKit2/Platform/CoreIPC/.libs/libWebKit2Platform_la-Attachment.o 

but the resulting .a file doesnt have the paths registered in its file list ?

[Zan Dobersek]

The full paths are specific to thin archives which are used by default.
http://sourceware.org/cgi-bin/cvsweb.cgi/~checkout~/src/binutils/NEWS?rev=1.109&content-type=text/plain&cvsroot=src&only_with_tag=binutils-binutils-2_23_1
('Changes in 2.19' part explains the behavior.)

WebCoreLayer.a and WebCoreLayerGtk2.a generation depends on the given archives being thin.

I think that using ar < 2.19 might still work, but would need a non-thin-archives-specific approach.

[Landry Breuil]

(In reply to comment #7)
> The full paths are specific to thin archives which are used by default.
> http://sourceware.org/cgi-bin/cvsweb.cgi/~checkout~/src/binutils/NEWS?rev=1.109&content-type=text/plain&cvsroot=src&only_with_tag=binutils-binutils-2_23_1
> ('Changes in 2.19' part explains the behavior.)
> 
> WebCoreLayer.a and WebCoreLayerGtk2.a generation depends on the given archives being thin.
> 
> I think that using ar < 2.19 might still work, but would need a non-thin-archives-specific approach.


I've found a _huge_ workaround to replace the need for thin archives.

Instead of using :

ar t $(archive) | xargs -n50 ar cruT $@

i'm using :

ar t $(archive) | while read f ; do find . -name $$f ; done | xargs -n50 ar cru $@

Add to that a hard/softlink between WebCoreLayerGtk2.a and libWebCoreLayerGtk2.a (our libtool implementation doesnt like linking static libraries not named libsomething.a) and 2.1.3 builds, links, and even runs.

Webkit2 is still unusable at runtime as with 2.0.x, but that'll be for another bug report.

[Martin Robinson]

(In reply to comment #8)

> I've found a _huge_ workaround to replace the need for thin archives.
> 
> Instead of using :
> 
> ar t $(archive) | xargs -n50 ar cruT $@
> 
> i'm using :
> 
> ar t $(archive) | while read f ; do find . -name $$f ; done | xargs -n50 ar cru $@
> 
> Add to that a hard/softlink between WebCoreLayerGtk2.a and libWebCoreLayerGtk2.a (our libtool implementation doesnt like linking static libraries not named libsomething.a) and 2.1.3 builds, links, and even runs.
> 
> Webkit2 is still unusable at runtime as with 2.0.x, but that'll be for another bug report.

Instead of adding more hacks to the build system, why don't we just apply the optimizations conditionally on systems that support them?

[Zan Dobersek]

(In reply to comment #9)
> Instead of adding more hacks to the build system, why don't we just apply the optimizations conditionally on systems that support them?

AR_FLAGS overriding can be easily fixed - I'll upload that patch in a separate bug.

As for the WebCoreLayer and WebCoreLayerGtk2 archives generation, as in comment #8, I'd use the `find` approach, but do it once at the start to find every object file and then operate with that.

I'll get something functioning and upload it for testing and reviewing.

[Zan Dobersek]

Created attachment 208199 [details]
Patch

[Zan Dobersek]

Is this still an issue?

[Landry Breuil]

(In reply to comment #12)
> Is this still an issue?

Yes, since nothing changed on our side and we're still using an ar that doesnt support -T, so we have a local patch for this:

http://www.openbsd.org/cgi-bin/cvsweb/ports/www/webkit/patches/patch-GNUmakefile_in?rev=1.25;content-type=text%2Fx-cvsweb-markup

(the two last chunks)

[Gustavo Noronha (kov)]

Comment on attachment 208199 [details]
Patch

I'm concerned about the performance of that many grep runs, it would be great if we could do that only for ar tools that do not support thin archives like we discussed =)

[Zan Dobersek]

(In reply to comment #14)
> (From update of attachment 208199 [details])
> I'm concerned about the performance of that many grep runs, it would be great if we could do that only for ar tools that do not support thin archives like we discussed =)

Grepping doesn't introduce performance penalties, readlink invocations do. Fortunately those are not necessary, so I'll remove them.

[Zan Dobersek]

Created attachment 218851 [details]
Patch

[Zan Dobersek]

Created attachment 218856 [details]
Patch

[Martin Robinson]

Comment on attachment 218856 [details]
Patch

Perhaps we should just consider moving the thin-archives support back to build-webkit instead of having it run for downstream? That might be easier to maintain than lots of automake magic.

[Zan Dobersek]

Works for me.

Thin archives could still be enabled manually by exporting AR_FLAGS (which would be exported in build-webkit for development builds).

Gustavo?

[Zan Dobersek]

Actually, this still wouldn't solve the problem.

WebCoreLayer.a and WebCoreLayerGtk2.a archives are still required, and should work with both thin and thick(?) archives, in development and product builds.

[Gustavo Noronha (kov)]

Comment on attachment 218856 [details]
Patch

I think if this works we should go with it (thus setting r+) and not do bigger changes such as moving the support to build-webkit. Either that or we leave this alone and let the downstreams keep dealing with it while we work to fully replace the build system.

[Zan Dobersek]

Landry, any chance you could test out this patch?

I've successfully compiled WebKitGTK+ when using both thin and non-thin archives, but your feedback here would be best.

[Landry Breuil]

(In reply to comment #22)
> Landry, any chance you could test out this patch?
> 
> I've successfully compiled WebKitGTK+ when using both thin and non-thin archives, but your feedback here would be best.

Will test it by manually backporting it to 2.2.3, but from what i understand, AR_FLAGS still need to be overriden to be 'cru' instead of 'cruT' for us, right ?

[Zan Dobersek]

(In reply to comment #23)
> (In reply to comment #22)
> > Landry, any chance you could test out this patch?
> > 
> > I've successfully compiled WebKitGTK+ when using both thin and non-thin archives, but your feedback here would be best.
> 
> Will test it by manually backporting it to 2.2.3, but from what i understand, AR_FLAGS still need to be overriden to be 'cru' instead of 'cruT' for us, right ?

Correct. For now, AR_FLAGS defaults to 'cruT' if not set in the environment.

[Landry Breuil]

Yes, that seems to do the trick for me - i still need to add 'ln $@ libWebCoreLayer.a' after having it archived, but i dont really remember why now - probably because our own libtool only wants to work with files named libsomething.

[Zan Dobersek]

That makes sense. I'll do that through a separate bug.

[Zan Dobersek]

Comment on attachment 218856 [details]
Patch

Clearing flags on attachment: 218856

Committed r160482: <http://trac.webkit.org/changeset/160482>

[Zan Dobersek]

All reviewed patches have been landed.  Closing bug.