Bug 118732 - Use of ar T option not supported by older binutils
Summary: Use of ar T option not supported by older binutils
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Zan Dobersek
URL:
Keywords:
Depends on: 107400 115918
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-16 07:36 PDT by Landry Breuil
Modified: 2013-12-12 02:07 PST (History)
5 users (show)

See Also:


Attachments
Patch (5.34 KB, patch)
2013-08-06 10:19 PDT, Zan Dobersek
no flags Details | Formatted Diff | Diff
Patch (5.34 KB, patch)
2013-12-10 04:27 PST, Zan Dobersek
no flags Details | Formatted Diff | Diff
Patch (5.34 KB, patch)
2013-12-10 05:45 PST, Zan Dobersek
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Landry Breuil 2013-07-16 07:36:00 PDT
OpenBSD still uses binutils 2.15, and our ar(1) doesn't support the 'T' flag used in various places :

./Tools/TestWebKitAPI/GNUmakefile.am:   $(AM_V_at)$(foreach archive, $(webcore_layer_archives), $(shell ar t $(archive) | xargs -n50 ar cruT $@))
./Source/WebKit2/GNUmakefile.am:        $(AM_V_at)$(foreach archive, $(webcore_layer_gtk2_archives), $(shell ar t $(archive) | xargs -n50 ar cruT $@))
./Source/autotools/SetupLibtool.m4:    AR_FLAGS="cruT"

https://bugs.webkit.org/show_bug.cgi?id=115918

It would be nice to test for its support before bindly assuming all platforms have it. It was added in binutils 2.19.1.

Maybe i can workaround it in SetupLibtool.m4 by setting AR_FLAGS in the env, but the two other occurences hardcode it.
Comment 1 Landry Breuil 2013-07-20 01:33:52 PDT
So far, i've tried the following workarounds without success :

- Removing T from ar flags, just using cru:

ar t | ar cru fails because it cant find the object files composing all the $(archives) ? maybe an xargs issue, or a link/search path issue ?

- hacking makefile to avoid using a temporary library
-       $(AM_V_at)$(foreach archive, $(webcore_layer_gtk2_archives), $(shell ar t $(archive) | xargs -n50 ar cruT $@))
+       $(AM_V_at)ar cru $@ $(webcore_layer_gtk2_archives)

-       $(AM_V_at)$(foreach archive, $(webcore_layer_archives), $(shell ar t $(archive) | xargs -n50 ar cruT $@))
+       $(AM_V_at)ar cru $@ $(webcore_layer_archives)

this generates a WebCoreLayerGtk2.a without symbols, so WebKitPluginProcess cant link against it.

- using a linker script :

webkitgtk-2.1.3/ $cat WebCoreLayerGtk2.a                                                                                                            
GROUP (libPlatform.a libPlatformGtk2.a libWebCorePlatform.a libWebCoreGtk2.a libWebCore.a libWebCoreModules.a libWebCoreSVG.a libANGLE.a)

=> blows at link time with tons of undefined refs. I suppose fallout from the circular deps between all the libs forming WebCoreLayerGtk2.a ?

- using --start-group $(webcore_layer_gtk2_archives) -end-group => libtool reorders flags, and messes up with linking order

So as of now, building with webkit2 enabled is not possible for me.
Comment 2 Landry Breuil 2013-07-20 01:38:17 PDT
(In reply to comment #1)
> So far, i've tried the following workarounds without success :
> 
> - Removing T from ar flags, just using cru:
> 
> ar t | ar cru fails because it cant find the object files composing all the $(archives) ? maybe an xargs issue, or a link/search path issue ?

More on this :

$ar t .libs/libPlatformGtk2.a | head -2                   
libPlatformGtk2_la-AudioBusGtk.o
libPlatformGtk2_la-GraphicsContextCairo.o
$find . -name libPlatformGtk2_la-AudioBusGtk.o
./Source/WebCore/platform/audio/gtk/.libs/libPlatformGtk2_la-AudioBusGtk.o

how is ar cru supposed to reach that object file when archiving WebCoreLayerGtk2.a ?
Comment 3 Zan Dobersek 2013-07-20 04:42:50 PDT
(In reply to comment #2)
> (In reply to comment #1)
> > So far, i've tried the following workarounds without success :
> > 
> > - Removing T from ar flags, just using cru:
> > 
> > ar t | ar cru fails because it cant find the object files composing all the $(archives) ? maybe an xargs issue, or a link/search path issue ?
> 
> More on this :
> 
> $ar t .libs/libPlatformGtk2.a | head -2                   
> libPlatformGtk2_la-AudioBusGtk.o
> libPlatformGtk2_la-GraphicsContextCairo.o
> $find . -name libPlatformGtk2_la-AudioBusGtk.o
> ./Source/WebCore/platform/audio/gtk/.libs/libPlatformGtk2_la-AudioBusGtk.o
> 
> how is ar cru supposed to reach that object file when archiving WebCoreLayerGtk2.a ?

Don't know exactly what's going on in .libs/libPlatformGtk2.a, but the object file path should be of relative form and pointing to the object file, in AudioBusGtk.o case that's in Source/WebCore/platform/audio/gtk/.libs/.
Comment 4 Zan Dobersek 2013-07-20 04:47:16 PDT
(In reply to comment #0)
> Maybe i can workaround it in SetupLibtool.m4 by setting AR_FLAGS in the env, but the two other occurences hardcode it.

AR_FLAGS should be taken into the account here I think. Would you be willing to try out a patch if/when the archive member file path output issue (comment #2) is resolved?
Comment 5 Landry Breuil 2013-07-20 08:56:13 PDT
(In reply to comment #3)
> (In reply to comment #2)
> > (In reply to comment #1)
> > > So far, i've tried the following workarounds without success :
> > > 
> > > - Removing T from ar flags, just using cru:
> > > 
> > > ar t | ar cru fails because it cant find the object files composing all the $(archives) ? maybe an xargs issue, or a link/search path issue ?
> > 
> > More on this :
> > 
> > $ar t .libs/libPlatformGtk2.a | head -2                   
> > libPlatformGtk2_la-AudioBusGtk.o
> > libPlatformGtk2_la-GraphicsContextCairo.o
> > $find . -name libPlatformGtk2_la-AudioBusGtk.o
> > ./Source/WebCore/platform/audio/gtk/.libs/libPlatformGtk2_la-AudioBusGtk.o
> > 
> > how is ar cru supposed to reach that object file when archiving WebCoreLayerGtk2.a ?
> 
> Don't know exactly what's going on in .libs/libPlatformGtk2.a, but the object file path should be of relative form and pointing to the object file, in AudioBusGtk.o case that's in Source/WebCore/platform/audio/gtk/.libs/.

I dont know what's supposed to be used when generating .libs/libPlatformGtk2.a but my ar(1) manpage states for option P :

       P   Use the full path name when matching names in the archive.  GNU ar
           can not create an archive with a full path name (such archives are
           not POSIX complaint), but other archive creators can.  This option
           will cause GNU ar to match file names using a complete path name,
           which can be convenient when extracting a single file from an
           archive created by another tool.

Is webkit expecting archives to be built with another tool, and not posix compliant ? or a more recent ar/binutils supporting some specific options is required ?
Comment 6 Landry Breuil 2013-07-20 09:19:49 PDT
Fwiw, libtool indeed calls ar cru with the full path to object files :

libtool: link: ar cru .libs/libWebKit2Platform.a Source/WebKit2/Platform/CoreIPC/.libs/libWebKit2Platform_la-ArgumentCoders.o Source/WebKit2/Platform/CoreIPC/.libs/libWebKit2Platform_la-ArgumentDecoder.o Source/WebKit2/Platform/CoreIPC/.libs/libWebKit2Platform_la-ArgumentEncoder.o Source/WebKit2/Platform/CoreIPC/.libs/libWebKit2Platform_la-Attachment.o 

but the resulting .a file doesnt have the paths registered in its file list ?
Comment 7 Zan Dobersek 2013-07-20 09:28:37 PDT
The full paths are specific to thin archives which are used by default.
http://sourceware.org/cgi-bin/cvsweb.cgi/~checkout~/src/binutils/NEWS?rev=1.109&content-type=text/plain&cvsroot=src&only_with_tag=binutils-binutils-2_23_1
('Changes in 2.19' part explains the behavior.)

WebCoreLayer.a and WebCoreLayerGtk2.a generation depends on the given archives being thin.

I think that using ar < 2.19 might still work, but would need a non-thin-archives-specific approach.
Comment 8 Landry Breuil 2013-07-22 12:44:39 PDT
(In reply to comment #7)
> The full paths are specific to thin archives which are used by default.
> http://sourceware.org/cgi-bin/cvsweb.cgi/~checkout~/src/binutils/NEWS?rev=1.109&content-type=text/plain&cvsroot=src&only_with_tag=binutils-binutils-2_23_1
> ('Changes in 2.19' part explains the behavior.)
> 
> WebCoreLayer.a and WebCoreLayerGtk2.a generation depends on the given archives being thin.
> 
> I think that using ar < 2.19 might still work, but would need a non-thin-archives-specific approach.


I've found a _huge_ workaround to replace the need for thin archives.

Instead of using :

ar t $(archive) | xargs -n50 ar cruT $@

i'm using :

ar t $(archive) | while read f ; do find . -name $$f ; done | xargs -n50 ar cru $@

Add to that a hard/softlink between WebCoreLayerGtk2.a and libWebCoreLayerGtk2.a (our libtool implementation doesnt like linking static libraries not named libsomething.a) and 2.1.3 builds, links, and even runs.

Webkit2 is still unusable at runtime as with 2.0.x, but that'll be for another bug report.
Comment 9 Martin Robinson 2013-07-22 12:59:14 PDT
(In reply to comment #8)

> I've found a _huge_ workaround to replace the need for thin archives.
> 
> Instead of using :
> 
> ar t $(archive) | xargs -n50 ar cruT $@
> 
> i'm using :
> 
> ar t $(archive) | while read f ; do find . -name $$f ; done | xargs -n50 ar cru $@
> 
> Add to that a hard/softlink between WebCoreLayerGtk2.a and libWebCoreLayerGtk2.a (our libtool implementation doesnt like linking static libraries not named libsomething.a) and 2.1.3 builds, links, and even runs.
> 
> Webkit2 is still unusable at runtime as with 2.0.x, but that'll be for another bug report.

Instead of adding more hacks to the build system, why don't we just apply the optimizations conditionally on systems that support them?
Comment 10 Zan Dobersek 2013-08-06 08:05:19 PDT
(In reply to comment #9)
> Instead of adding more hacks to the build system, why don't we just apply the optimizations conditionally on systems that support them?

AR_FLAGS overriding can be easily fixed - I'll upload that patch in a separate bug.

As for the WebCoreLayer and WebCoreLayerGtk2 archives generation, as in comment #8, I'd use the `find` approach, but do it once at the start to find every object file and then operate with that.

I'll get something functioning and upload it for testing and reviewing.
Comment 11 Zan Dobersek 2013-08-06 10:19:55 PDT
Created attachment 208199 [details]
Patch
Comment 12 Zan Dobersek 2013-12-09 05:03:04 PST
Is this still an issue?
Comment 13 Landry Breuil 2013-12-09 05:36:13 PST
(In reply to comment #12)
> Is this still an issue?

Yes, since nothing changed on our side and we're still using an ar that doesnt support -T, so we have a local patch for this:

http://www.openbsd.org/cgi-bin/cvsweb/ports/www/webkit/patches/patch-GNUmakefile_in?rev=1.25;content-type=text%2Fx-cvsweb-markup

(the two last chunks)
Comment 14 Gustavo Noronha (kov) 2013-12-09 08:00:45 PST
Comment on attachment 208199 [details]
Patch

I'm concerned about the performance of that many grep runs, it would be great if we could do that only for ar tools that do not support thin archives like we discussed =)
Comment 15 Zan Dobersek 2013-12-10 03:55:39 PST
(In reply to comment #14)
> (From update of attachment 208199 [details])
> I'm concerned about the performance of that many grep runs, it would be great if we could do that only for ar tools that do not support thin archives like we discussed =)

Grepping doesn't introduce performance penalties, readlink invocations do. Fortunately those are not necessary, so I'll remove them.
Comment 16 Zan Dobersek 2013-12-10 04:27:40 PST
Created attachment 218851 [details]
Patch
Comment 17 Zan Dobersek 2013-12-10 05:45:11 PST
Created attachment 218856 [details]
Patch
Comment 18 Martin Robinson 2013-12-10 05:50:46 PST
Comment on attachment 218856 [details]
Patch

Perhaps we should just consider moving the thin-archives support back to build-webkit instead of having it run for downstream? That might be easier to maintain than lots of automake magic.
Comment 19 Zan Dobersek 2013-12-10 05:59:09 PST
Works for me.

Thin archives could still be enabled manually by exporting AR_FLAGS (which would be exported in build-webkit for development builds).

Gustavo?
Comment 20 Zan Dobersek 2013-12-10 06:04:53 PST
Actually, this still wouldn't solve the problem.

WebCoreLayer.a and WebCoreLayerGtk2.a archives are still required, and should work with both thin and thick(?) archives, in development and product builds.
Comment 21 Gustavo Noronha (kov) 2013-12-11 00:52:19 PST
Comment on attachment 218856 [details]
Patch

I think if this works we should go with it (thus setting r+) and not do bigger changes such as moving the support to build-webkit. Either that or we leave this alone and let the downstreams keep dealing with it while we work to fully replace the build system.
Comment 22 Zan Dobersek 2013-12-11 04:47:42 PST
Landry, any chance you could test out this patch?

I've successfully compiled WebKitGTK+ when using both thin and non-thin archives, but your feedback here would be best.
Comment 23 Landry Breuil 2013-12-11 04:58:07 PST
(In reply to comment #22)
> Landry, any chance you could test out this patch?
> 
> I've successfully compiled WebKitGTK+ when using both thin and non-thin archives, but your feedback here would be best.

Will test it by manually backporting it to 2.2.3, but from what i understand, AR_FLAGS still need to be overriden to be 'cru' instead of 'cruT' for us, right ?
Comment 24 Zan Dobersek 2013-12-11 04:59:58 PST
(In reply to comment #23)
> (In reply to comment #22)
> > Landry, any chance you could test out this patch?
> > 
> > I've successfully compiled WebKitGTK+ when using both thin and non-thin archives, but your feedback here would be best.
> 
> Will test it by manually backporting it to 2.2.3, but from what i understand, AR_FLAGS still need to be overriden to be 'cru' instead of 'cruT' for us, right ?

Correct. For now, AR_FLAGS defaults to 'cruT' if not set in the environment.
Comment 25 Landry Breuil 2013-12-11 12:07:18 PST
Yes, that seems to do the trick for me - i still need to add 'ln $@ libWebCoreLayer.a' after having it archived, but i dont really remember why now - probably because our own libtool only wants to work with files named libsomething.
Comment 26 Zan Dobersek 2013-12-12 01:14:02 PST
That makes sense. I'll do that through a separate bug.
Comment 27 Zan Dobersek 2013-12-12 02:07:30 PST
Comment on attachment 218856 [details]
Patch

Clearing flags on attachment: 218856

Committed r160482: <http://trac.webkit.org/changeset/160482>
Comment 28 Zan Dobersek 2013-12-12 02:07:45 PST
All reviewed patches have been landed.  Closing bug.