Bug 8528 – Bakefiles (and generated Makefiles) for wx and gdk ports

• Component: WebCore Misc.
• Status: RESOLVED FIXED

[Kevin Ollivier]

This patch introduces a Bakefile-based build system for WebKit ports to the wx and GDK platforms. The documentation introducing this build system, explaining how to get started using it, and noting any remaining issues in its adoption can be found in WebKit/Bakefiles/Readme.txt. Please let me know of any questions or unaddressed issues.

This system was tested on Win XP SP2, Linux (FC4), and OS X (Intel Mac, OS X 10.4.6). By following the instructions in Readme.txt, you should be able to achieve a complete build on all of these platforms with latest CVS as of the filing of this patch (April 21st.)

Since this patch will probably change somewhat before it is submitted for review, I have not yet attached a ChangeLog, but will do so before the patch is submitted for review. If a ChangeLog should be added immediately, though, please let me know.

Many thanks for your consideration and testing of this patch!

[Kevin Ollivier]

Created attachment 7895 [details]
Patch adding Bakefile-based build system for wx and GDK ports

[Kevin Ollivier]

I forgot to note the compiler versions I used. gcc 4.0.1 on OS X, gcc 4.0.2 on FC4, and MSVC8 (Express) on Win XP SP2.

[Michael Emmel]

The bakefiles need to be updated to use WebCore/DerivedSources.make not make-generated-sources.h

[Michael Emmel]

Comments needed to added to the readme to show how to generate the makefiles from the bakefiles also how to get and install bakefile itself

[Michael Emmel]

(In reply to comment #3)
> The bakefiles need to be updated to use WebCore/DerivedSources.make not
> make-generated-sources.h
> 

Okay you through me with the script name do you know how to get bakefile to generate a rule for a shellscript I could not get this to work myself its a big issues since you can't create generic rules for all using bakefile or at least I did not figure out how

I think what needs to happen is there needs to be a way to add anothere dependency to the all rule that allow us to skip having to run the shell script.
If it can't be done then we need to fix this asap in bakefile.

Also the way you almost do it which is what I was trying


   <exe id="dftables">
        <dirname>$(SRCDIR)</dirname>
        <sources>$(SRCDIR)/pcre/dftables.c</sources>
    </exe>

    <action id="DerivedSources">
        <depends>dftables</depends>
        <command>./make-generated-sources.sh</command>
    </action>

Is a bit verbose that should look more like a tradtional makefile and be done under one toplevel tag.

[Kevin Ollivier]

(In reply to comment #4)
> Comments needed to added to the readme to show how to generate the makefiles
> from the bakefiles also how to get and install bakefile itself
> 

Both of those tasks are handled by the WebKitScripts/Tools/regenerate-makefiles script. This is mentioned in the "Regenerating the Makefiles" section of the readme, but I've added a note that it automatically downloads and installs Bakefile for you and will include it in the next patch. 

[Kevin Ollivier]

(In reply to comment #5)
> (In reply to comment #3)
> > The bakefiles need to be updated to use WebCore/DerivedSources.make not
> > make-generated-sources.h
> > 
> 
> Okay you through me with the script name do you know how to get bakefile to
> generate a rule for a shellscript I could not get this to work myself its a big
> issues since you can't create generic rules for all using bakefile or at least
> I did not figure out how
> 
> I think what needs to happen is there needs to be a way to add anothere
> dependency to the all rule that allow us to skip having to run the shell
> script.
> If it can't be done then we need to fix this asap in bakefile.
> 
> Also the way you almost do it which is what I was trying
> 
> 
>    <exe id="dftables">
>         <dirname>$(SRCDIR)</dirname>
>         <sources>$(SRCDIR)/pcre/dftables.c</sources>
>     </exe>
> 
>     <action id="DerivedSources">
>         <depends>dftables</depends>
>         <command>./make-generated-sources.sh</command>
>     </action>
> 
> Is a bit verbose that should look more like a tradtional makefile and be done
> under one toplevel tag.
> 

I spoke with Vaclav about why this doesn't work, and what we need to do is add an <is-phony/> tag to the DerivedSources target, like so:

    <action id="DerivedSources">
        <is-phony/>
        <depends>dftables</depends>
        <command>./make-generated-sources.sh</command>
    </action>

This makes sure the makefile doesn't check the target for existance and thus the step will always run.

It looks a bit verbose, yes, but then again typically we would have written DerivedSources.make in Bakefile and each target would correspond to one file that needs to be built/updated, so doing things this way (having a phony target that runs a set of commands on numerous files) is a bit of an unusual case. Please let me know if this works for you. I'm going to test it out here as well and see if we can't remove most of the manual build steps from the process this way.

[Kevin Ollivier]

(In reply to comment #7)
> (In reply to comment #5)
> > (In reply to comment #3)
> > > The bakefiles need to be updated to use WebCore/DerivedSources.make not
> > > make-generated-sources.h
> > > 
> > 
> > Okay you through me with the script name do you know how to get bakefile to
> > generate a rule for a shellscript I could not get this to work myself its a big
> > issues since you can't create generic rules for all using bakefile or at least
> > I did not figure out how
> > 
> > I think what needs to happen is there needs to be a way to add anothere
> > dependency to the all rule that allow us to skip having to run the shell
> > script.
> > If it can't be done then we need to fix this asap in bakefile.
> > 
> > Also the way you almost do it which is what I was trying
> > 
> > 
> >    <exe id="dftables">
> >         <dirname>$(SRCDIR)</dirname>
> >         <sources>$(SRCDIR)/pcre/dftables.c</sources>
> >     </exe>
> > 
> >     <action id="DerivedSources">
> >         <depends>dftables</depends>
> >         <command>./make-generated-sources.sh</command>
> >     </action>
> > 
> > Is a bit verbose that should look more like a tradtional makefile and be done
> > under one toplevel tag.
> > 
> 
> I spoke with Vaclav about why this doesn't work, and what we need to do is add
> an <is-phony/> tag to the DerivedSources target, like so:
> 
>     <action id="DerivedSources">
>         <is-phony/>
>         <depends>dftables</depends>
>         <command>./make-generated-sources.sh</command>
>     </action>
> 
> This makes sure the makefile doesn't check the target for existance and thus
> the step will always run.
> 
> It looks a bit verbose, yes, but then again typically we would have written
> DerivedSources.make in Bakefile and each target would correspond to one file
> that needs to be built/updated, so doing things this way (having a phony target
> that runs a set of commands on numerous files) is a bit of an unusual case.
> Please let me know if this works for you. I'm going to test it out here as well
> and see if we can't remove most of the manual build steps from the process this
> way.
> 

Actually, also, instead of <depends> we need <objects-depend>DerivedSources</objects-depend> here to make sure the script runs before any sources are built. With this change in place, I'm able to build JSCore now simply by running `make -f GNUMakefile`. I'm going to add this to the wx makefile too, update the docs, and submit a new patch where the only 'setup' step will be running WebKitTools/Scripts/install-unix-extras. 

[Kevin Ollivier]

Created attachment 7918 [details]
Second draft of patch, simplifying build on Unix.

This patch integrates the setup commands (e.g. DerivedSources) into the makefiles on Unix/Mac so that now you just need to run ./install-unix-extras and then run the build commands. On Windows, running the setup commands manually unfortunately is still needed, at least with MSVC6 project files, because Bakefile doesn't currently support their pre-build/post-link commands. I'm looking into getting this matter resolved, but in the meantime if everything else checks out okay I might want to submit this for review. 

It also re-adds move-js-headers.sh, which is needed in order for JavaScriptCore "Framework-style" headers to work. 

Thanks in advance for any testing and comments/suggestions! :-)

[Michael Emmel]

Comment on attachment 7918 [details]
Second draft of patch, simplifying build on Unix.


bakefile_gen is missing from the patch

[Michael Emmel]

The name of the makefile should be changed to GNUmakefile not GNUMakefile so gnu make will pick it automatically

[Kevin Ollivier]

(In reply to comment #10)
> (From update of attachment 7918 [details] [edit])
> 
> bakefile_gen is missing from the patch
> 

bakefile_gen is part of the Bakefile package.

[Kevin Ollivier]

(In reply to comment #11)
> The name of the makefile should be changed to GNUmakefile not GNUMakefile so
> gnu make will pick it automatically
> 

Done, thanks!

[Kevin Ollivier]

Created attachment 7934 [details]
Renamed GNU makefiles to GNUmakefile and integrated GDK port updates. 

Please check Bakefiles/Readme.txt and ChangeLogs to learn more about this patch. Comments and suggestions welcome! Thanks!

[Kevin Ollivier]

(In reply to comment #14)
> Created an attachment (id=7934) [edit]
> Renamed GNU makefiles to GNUmakefile and integrated GDK port updates. 
> 
> Please check Bakefiles/Readme.txt and ChangeLogs to learn more about this
> patch. Comments and suggestions welcome! Thanks!
> 

Sorry, I did forget to make one small change to the Readme.txt docs. You no longer need to specify the filename to make on Unix. Just running 'make' works fine.

[Darin Adler]

Comment on attachment 7934 [details]
Renamed GNU makefiles to GNUmakefile and integrated GDK port updates. 

Why are we checking in generated projects and makefiles? We usually do not check in generated files.

It seems strange to have the bakefiles in a Bakefiles directory instead of inside the individual project directories.

[Kevin Ollivier]

(In reply to comment #16)
> (From update of attachment 7934 [details] [edit])
> Why are we checking in generated projects and makefiles? We usually do not
> check in generated files.

I was just concerned that some people who are used to checkout -> build would not like the process to become checkout -> generate projects -> build. (e.g. if we ever moved the VC2005 projects over to using this system) The other scripts to create generated files are run automatically via the project files, but obviously we can't do that for the generation of the project files themselves, which means we add a manual step which would have to be run after each update. But at the same time, I see the logic for not keeping generated files in SVN, so I'm open to doing it either way.

If you want, I can go ahead and remove those generated files from the patch.

> It seems strange to have the bakefiles in a Bakefiles directory instead of
> inside the individual project directories.

The main issue is that several of those Bakefiles are shared between WebCore ports, and presets.bkl is shared between JavaScriptCore and WebCore, so coming up with 'appropriate' placement of shared items seemed more involved than just having one directory for all the Bakefiles. Of course, that could just be my organization style. :-) Did you want me to move the files into various directories? If so, do you have any suggestions as to where would be a good location for the presets.bkl and readme.txt files? 

Thanks!

[Vaclav Slavik]

> obviously we can't do that for the generation of the project files themselves,
> which means we add a manual step which would have to be run after each update.

You could a check to makefiles, so that they are regenerated when they are older than the source bakefiles -- then this step would be manual only once. Obviously, this wouldn't work for project files, the best thing you can hope for here is a check in prebuild step that causes an error if the project file is older than the bakefiles.

[Darin Adler]

Comment on attachment 7934 [details]
Renamed GNU makefiles to GNUmakefile and integrated GDK port updates. 

(In reply to comment #17)
> I was just concerned that some people who are used to checkout -> build would
> not like the process to become checkout -> generate projects -> build. (e.g. if
> we ever moved the VC2005 projects over to using this system) The other scripts
> to create generated files are run automatically via the project files, but
> obviously we can't do that for the generation of the project files themselves,
> which means we add a manual step which would have to be run after each update.
> But at the same time, I see the logic for not keeping generated files in SVN,
> so I'm open to doing it either way.
> 
> If you want, I can go ahead and remove those generated files from the patch.

Yes, I'd like to see this done without checking in the generated files.

People who want to just be able to check out and build can use a build script of some sort, and we can automate the checking at least as suggested by Vaclav Slavik -- for some build systems it can be entirely automatic, for the others it can at least be an error that is detected. Over time we may find ways to automate even for things like Microsoft Visual Studio projects.

> > It seems strange to have the bakefiles in a Bakefiles directory instead of
> > inside the individual project directories.
> 
> The main issue is that several of those Bakefiles are shared between WebCore
> ports, and presets.bkl is shared between JavaScriptCore and WebCore, so coming
> up with 'appropriate' placement of shared items seemed more involved than just
> having one directory for all the Bakefiles. Of course, that could just be my
> organization style. :-) Did you want me to move the files into various
> directories? If so, do you have any suggestions as to where would be a good
> location for the presets.bkl and readme.txt files? 

The files that list all the things in a project should be inside that project. For example, if there's a list of all the files in WebCore, it's very important that's inside the WebCore directory.

If there are shared things, it's fine for them to be in a shared place, but anything that has to change as a particular project changes should be in that project if possible.

Could you take another pass at this given those comments?

[Kevin Ollivier]

Created attachment 8042 [details]
Moved bkl files, removed gen. files, and added MSVS 2005 support

Please let me know if this works for you, or if you want the Bakefiles folder completely gone. Generated files are gone from the tree. Also, I added MSVS 2005 support as it greatly simplifies the Windows build process and removes the need to up-convert. Thanks!

[Darin Adler]

Comment on attachment 8042 [details]
Moved bkl files, removed gen. files, and added MSVS 2005 support

Even though I don't understand all the details, this looks good. Lets get this in there. r=me

[Michael Emmel]

(In reply to comment #21)
> (From update of attachment 8042 [details] [edit])
> Even though I don't understand all the details, this looks good. Lets get this
> in there. r=me
> 

Cool once its in I'll send my latest patch with all the linux sources for building.
I was trying to fix bugs while this was still out :)

[Kevin Ollivier]

Created attachment 8259 [details]
Needed to update to add XPath support and because WTF happened

[Michael Emmel]

I've been waiting on this patch to submit my linux port. Hopefully it can go in soon.

Mike

[Timothy Hatcher]

Landed in r14359