Bug 76388 – [GTK] Clean build is broken when using make -j

• Component: WebKitGTK
• Status: RESOLVED FIXED

[Carlos Garcia Campos]

If make is called without -j it works. I thibk the problem was introduced in r103920.


Creating hashtable for ../Source/JavaScriptCore/runtime/StringPrototype.cpp
  CXX    Source/WebKit/gtk/WebCoreSupport/libwebkitgtk_3_0_la-AcceleratedCompositingContextClutter.lo
  CXX    Source/WebKit/gtk/WebCoreSupport/libwebkitgtk_3_0_la-AssertMatchingEnums.lo
  CXX    Source/WebKit/gtk/WebCoreSupport/libwebkitgtk_3_0_la-ChromeClientGtk.lo
  CXX    Source/WebKit/gtk/WebCoreSupport/libwebkitgtk_3_0_la-ContextMenuClientGtk.lo
In file included from ../Source/WebCore/bindings/js/ScriptController.h:26:0,
                 from ../Source/WebCore/page/Frame.h:40,
                 from ../Source/WebCore/platform/gtk/PasteboardHelper.h:28,
                 from ../Source/WebKit/gtk/WebCoreSupport/AssertMatchingEnums.cpp:29:
../Source/WebCore/bindings/js/JSDOMWindowShell.h:32:25: fatal error: JSDOMWindow.h: No such file or directory
compilation terminated.
In file included from ../Source/WebKit/gtk/webkit/webkitwebview.h:30:0,
                 from ../Source/WebKit/gtk/WebCoreSupport/AcceleratedCompositingContext.h:26,
                 from ../Source/WebKit/gtk/WebCoreSupport/AcceleratedCompositingContextClutter.cpp:21:
./DerivedSources/webkit/webkitdom.h:23:34: fatal error: webkit/WebKitDOMAttr.h: No such file or directory
compilation terminated.
In file included from ../Source/WebKit/gtk/webkit/webkitwebview.h:30:0,
                 from ../Source/WebKit/gtk/WebCoreSupport/AcceleratedCompositingContext.h:26,
                 from ../Source/WebKit/gtk/webkit/webkitwebviewprivate.h:26,
                 from ../Source/WebKit/gtk/WebCoreSupport/ContextMenuClientGtk.cpp:29:
./DerivedSources/webkit/webkitdom.h:23:34: fatal error: webkit/WebKitDOMAttr.h: No such file or directory
compilation terminated.
In file included from ../Source/WebCore/bindings/js/ScriptController.h:26:0,
                 from ../Source/WebCore/page/Frame.h:40,
                 from ../Source/WebKit/gtk/webkit/webkitwebframeprivate.h:26,
                 from ../Source/WebKit/gtk/WebCoreSupport/ChromeClientGtk.cpp:59:
../Source/WebCore/bindings/js/JSDOMWindowShell.h:32:25: fatal error: JSDOMWindow.h: No such file or directory
compilation terminated.
make[1]: *** [Source/WebKit/gtk/WebCoreSupport/libwebkitgtk_3_0_la-AssertMatchingEnums.lo] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [Source/WebKit/gtk/WebCoreSupport/libwebkitgtk_3_0_la-AcceleratedCompositingContextClutter.lo] Error 1
make[1]: *** [Source/WebKit/gtk/WebCoreSupport/libwebkitgtk_3_0_la-ContextMenuClientGtk.lo] Error 1
make[1]: *** [Source/WebKit/gtk/WebCoreSupport/libwebkitgtk_3_0_la-ChromeClientGtk.lo] Error 1
make[1]: Leaving directory `/home/cgarcia/src/git/gnome/WebKit/WebKitBuild/webkit-1.7.4/_build'
make: *** [distcheck] Error 1

[Martin Robinson]

It seems this is likely caused by supplemental IDL changes. haraken, do you think you can take a look at this?

http://trac.webkit.org/changeset/103920
http://trac.webkit.org/changeset/103729

[Kentaro Hara]

(In reply to comment #1)
> It seems this is likely caused by supplemental IDL changes. haraken, do you think you can take a look at this?
> 
> http://trac.webkit.org/changeset/103920
> http://trac.webkit.org/changeset/103729

I guess that the cause is that no dependency is written between $(webcore_built_sources) and $(webcore_sources).

    nodist_libWebCore_la_SOURCES = $(webcore_built_sources)
    libWebCore_la_SOURCES = $(webcore_sources)

We need to guarantee that $(webcore_sources) are built _after_ $(webcore_built_sources). So I guess we can fix the issue by adding the following line:

    $(webcore_sources) : $(webcore_built_sources)

I am making a patch. (But I guess that potentially this issue would have occurred before the [supplemental] change.)

[Martin Robinson]

(In reply to comment #2)
 
> We need to guarantee that $(webcore_sources) are built _after_ $(webcore_built_sources). So I guess we can fix the issue by adding the following line:

Typically the way to ensure this is to include auto-generated source files in BUILT_SOURCES. That's a hint to the build tools that the files should be generated during the first phase of the build.

[Kentaro Hara]

Created attachment 125730 [details]
Patch

[Kentaro Hara]

Actually I could not reproduce the make -j error in my local GTK environment. If you think the patch is OK, I'd like to commit it. Otherwise, I am happy if you would check that the patch really fixes the make -j error in your GTK environment before committing it.

[Martin Robinson]

Comment on attachment 125730 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=125730&action=review

This fix looks sane, but do you mind making the small change below before landing?

> Source/WebCore/GNUmakefile.am:812
> +	$(webkitgtk_gdom_built_sources)

The GDOM sources are actually part of the WebKit build, not the WebCore one. It might make more sense to add it to BUILT_SOURCES in Source/WebCore/bindings/gobject/GNUmakefile.am.

[Martin Robinson]

Hrm. Actually it seems that webcore_built_sources is already added to BUILT_SOURCES in the toplevel GNUmakefile.am.

[Kentaro Hara]

(In reply to comment #7)
> Hrm. Actually it seems that webcore_built_sources is already added to BUILT_SOURCES in the toplevel GNUmakefile.am.

Then, the patch won't fix the issue...?

[Martin Robinson]

Carlos, this looks like the same issue that made me put $(BUILT_SOURCES) as a dependency for 'make docs' The issue, I think, is that you cannot do 'make distcheck' from a clean build. If you've already built (which works fine by the way), it should work.

[Martin Robinson]

I say this because it's failing for me before it event tries to build the tarball it creates. Here's the full build output:


martin@thurgoodmcburdle:~/WebKit/WebKitBuild/Release$ ../../Tools/gtk/run-with-jhbuild make -j12 distcheck
  GEN    DerivedSources/WebCore/InspectorBackendDispatcher.cpp
{ test ! -d "webkit-1.7.5" || { find "webkit-1.7.5" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -fr "webkit-1.7.5"; }; }
test -d "webkit-1.7.5" || mkdir "webkit-1.7.5"
make  \
	  top_distdir="webkit-1.7.5" distdir="webkit-1.7.5" \
	  dist-hook
make[1]: Entering directory `/home/martin/WebKit/WebKitBuild/Release'
  GEN    DerivedSources/WebCore/supplemental_dependency.tmp
  GEN    DerivedSources/webkit/WebKitDOMObject.h
  GEN    DerivedSources/webkit/WebKitDOMCustom.h
  GEN    DerivedSources/webkit/WebKitDOMEventTarget.h
  GEN    DerivedSources/webkit/webkitdomdefines.h
  GEN    DerivedSources/webkit/webkitdom.h
  GEN    stamp-webkitmarshal.cpp
  CXX    Source/WebCore/html/libWebCore_la-ValidationMessage.lo
  CXX    Source/WebCore/html/libWebCore_la-URLInputType.lo
  CXX    Source/WebCore/html/libWebCore_la-ValidityState.lo
  CXX    Source/WebCore/html/libWebCore_la-WeekInputType.lo
../../Source/WebCore/html/ValidationMessage.cpp:34:30: fatal error: CSSPropertyNames.h: No such file or directory
compilation terminated.
In file included from ../../Source/WebCore/dom/EventTarget.h:36:0,
                 from ../../Source/WebCore/dom/Node.h:29,
                 from ../../Source/WebCore/dom/ContainerNode.h:28,
                 from ../../Source/WebCore/dom/Document.h:34,
                 from ../../Source/WebCore/dom/Element.h:29,
                 from ../../Source/WebCore/dom/StyledElement.h:28,
                 from ../../Source/WebCore/html/HTMLElement.h:26,
                 from ../../Source/WebCore/html/FormAssociatedElement.h:27,
                 from ../../Source/WebCore/html/HTMLFormControlElement.h:27,
                 from ../../Source/WebCore/html/HTMLFormControlElementWithState.h:27,
                 from ../../Source/WebCore/html/HTMLTextFormControlElement.h:28,
                 from ../../Source/WebCore/html/HTMLInputElement.h:27,
                 from ../../Source/WebCore/html/URLInputType.cpp:34:

[Kentaro Hara]

(In reply to comment #9)
> If you've already built (which works fine by the way), it should work.

Yeah, maybe that is just because the generated files are already there. But why isn't the dependency from $(webcore_sources) to $(webcore_built_sources) guaranteed even if we add $(webcore_built_sources) to BUILT_SOURCES?

[Martin Robinson]

(In reply to comment #11)

> Yeah, maybe that is just because the generated files are already there. But why isn't the dependency from $(webcore_sources) to $(webcore_built_sources) guaranteed even if we add $(webcore_built_sources) to BUILT_SOURCES?

I'm still looking into this, but it may be because the $(BUILT_SOURCES) niceties only come when you are building the 'all' target.

[Martin Robinson]

(In reply to comment #12)

> I'm still looking into this, but it may be because the $(BUILT_SOURCES) niceties only come when you are building the 'all' target.

I confirmed locally that 'make -j12 distcheck' worked fine after running 'make -j12' first. Perhaps the problem Carlos is seeing is a different one, but I can't reproduce it right now.

[Priit Laes (IRC: plaes)]

(In reply to comment #13)
> (In reply to comment #12)
> 
> > I'm still looking into this, but it may be because the $(BUILT_SOURCES) niceties only come when you are building the 'all' target.
> 
> I confirmed locally that 'make -j12 distcheck' worked fine after running 'make -j12' first. Perhaps the problem Carlos is seeing is a different one, but I can't reproduce it right now.

One way to get build error is by adding --disable-dependency-tracking to configure (this used to work also before the supplemental IDL change stuff).

[Carlos Garcia Campos]

It still happens to me even with a normal make -j4 (no distcheck) in a clean WebKitBuild dir.


  CXX    Source/WebKit/gtk/WebCoreSupport/libwebkitgtk_3_0_la-ChromeClientGtk.lo
In file included from ../Source/WebCore/bindings/js/ScriptController.h:26:0,
                 from ../Source/WebCore/page/Frame.h:40,
                 from ../Source/WebCore/platform/gtk/PasteboardHelper.h:28,
                 from ../Source/WebKit/gtk/WebCoreSupport/AssertMatchingEnums.cpp:29:
../Source/WebCore/bindings/js/JSDOMWindowShell.h:32:25: error fatal: JSDOMWindow.h: No existe el archivo o el directorio
compilación terminada.
In file included from ../Source/WebKit/gtk/webkit/webkitwebview.h:30:0,
                 from ../Source/WebKit/gtk/WebCoreSupport/AcceleratedCompositingContext.h:27,
                 from ../Source/WebKit/gtk/WebCoreSupport/AcceleratedCompositingContextGL.cpp:20:
./DerivedSources/webkit/webkitdom.h:23:34: error fatal: webkit/WebKitDOMAttr.h: No existe el archivo o el directorio
compilación terminada.
In file included from ../Source/WebKit/gtk/webkit/webkitwebview.h:30:0,
                 from ../Source/WebKit/gtk/WebCoreSupport/AcceleratedCompositingContext.h:27,
                 from ../Source/WebKit/gtk/WebCoreSupport/AcceleratedCompositingContextClutter.cpp:21:
./DerivedSources/webkit/webkitdom.h:23:34: error fatal: webkit/WebKitDOMAttr.h: No existe el archivo o el directorio
compilación terminada.
In file included from ../Source/WebCore/bindings/js/ScriptController.h:26:0,
                 from ../Source/WebCore/page/Frame.h:40,
                 from ../Source/WebKit/gtk/webkit/webkitwebframeprivate.h:26,
                 from ../Source/WebKit/gtk/WebCoreSupport/ChromeClientGtk.cpp:60:
../Source/WebCore/bindings/js/JSDOMWindowShell.h:32:25: error fatal: JSDOMWindow.h: No existe el archivo o el directorio

So I guess it fails always running distcheck, not because distcheck makes anything different, but because it always builds on a clean build dir. 
Just running make -j4 again after the failure makes the build to continue, but that's not possible with distcheck, because every make distcheck starts from scratch

[Kentaro Hara]

(In reply to comment #15)
> It still happens to me even with a normal make -j4 (no distcheck) in a clean WebKitBuild dir.
> So I guess it fails always running distcheck, not because distcheck makes anything different, but because it always builds on a clean build dir. 
> Just running make -j4 again after the failure makes the build to continue, but that's not possible with distcheck, because every make distcheck starts from scratch

I could not reproduce it maybe because my machine has only 2 CPUs. Anyway the cause is that it is not guaranteed that $(webcore_sources) are built _after_ $(webcore_built_sources). I am not sure how to write the dependency correctly in automake, but would you try to add the following line in WebCore/GNUMakefile.am and see what happens?

    $(webcore_sources) : $(webcore_built_sources)

[Carlos Garcia Campos]

(In reply to comment #16)
> I could not reproduce it maybe because my machine has only 2 CPUs. Anyway the cause is that it is not guaranteed that $(webcore_sources) are built _after_ $(webcore_built_sources). I am not sure how to write the dependency correctly in automake, but would you try to add the following line in WebCore/GNUMakefile.am and see what happens?
> 
>     $(webcore_sources) : $(webcore_built_sources)

It doesn't help :-( I'm not sure the problem is that webcore_sources depends on webcore_built_sources. In the Makefile the all: rule depends on BUILT_SOURCES, so they should all be generated before anything else. The problem is that libwebkitgtk is built before libWebCore, since it depends on it at link time, but libwebkitgtk depends on webcore generated sources (at least the headers). 

When the compilation starts, BUILT_SOURCES are indeed created, since it's a dependency of all:
The first file generated is the supplemental file:

  GEN    DerivedSources/WebCore/supplemental_dependency.tmp

Then it generates the libwebkitgtk built sources, webkit2 built sources, JavaScriptCore built sources, and then it generates some of the webcore built sources:

  GEN    DerivedSources/WebCore/CSSGrammar.cpp
  GEN    DerivedSources/WebCore/CSSPropertyNames.h
  GEN    DerivedSources/WebCore/CSSValueKeywords.h
  GEN    DerivedSources/WebCore/ColorData.cpp
  GEN    DerivedSources/WebCore/EventFactory.cpp
  GEN    DerivedSources/WebCore/EventHeaders.h
  GEN    DerivedSources/WebCore/EventInterfaces.h
  GEN    DerivedSources/WebCore/EventTargetHeaders.h
  GEN    DerivedSources/WebCore/EventTargetInterfaces.h
  GEN    DerivedSources/WebCore/ExceptionCodeDescription.cpp
  GEN    DerivedSources/WebCore/ExceptionInterfaces.h
  GEN    DerivedSources/WebCore/ExceptionHeaders.h
  GEN    DerivedSources/WebCore/HTMLNames.h
  GEN    DerivedSources/WebCore/HTMLEntityTable.cpp
  GEN    DerivedSources/WebCore/InjectedScriptSource.h
  GEN    DerivedSources/WebCore/InspectorBackendDispatcher.cpp
  GEN    DerivedSources/WebCore/InspectorProtocolVersion.h
  GEN    DerivedSources/WebCore/SVGElementFactory.cpp
  GEN    DerivedSources/WebCore/MathMLElementFactory.cpp
  GEN    DerivedSources/WebCore/MathMLNames.cpp
  GEN    DerivedSources/WebCore/UserAgentStyleSheets.h
  GEN    DerivedSources/WebCore/WebKitFontFamilyNames.h
  GEN    DerivedSources/WebCore/XLinkNames.cpp
  GEN    DerivedSources/WebCore/XMLNames.cpp
  GEN    DerivedSources/WebCore/XMLNames.h
  GEN    DerivedSources/WebCore/XMLNSNames.cpp
  GEN    DerivedSources/WebCore/XMLNSNames.h
  GEN    DerivedSources/WebCore/XMLViewerCSS.h
  GEN    DerivedSources/WebCore/XMLViewerJS.h
  GEN    DerivedSources/WebCore/XPathGrammar.cpp
  GEN    DerivedSources/ANGLE/glslang.cpp
  GEN    DerivedSources/ANGLE/glslang_tab.cpp

These are the ones that have an explicit rule in the Makefile and don't depend on the supplemental temp file. The rule for webcore built sources starting with JS is not executed, I guess  it's because all dependencies already exist. I have tried several changes in the makefile and I haven't managed to make it build.

[Carlos Garcia Campos]

When running make again after it fails, all DerivedSources/webkit/WebKitDOM* files are generated and then all DerivedSources/WebCore/JS* and the build completes successfully.

[Kentaro Hara]

(In reply to comment #17)
> These are the ones that have an explicit rule in the Makefile and don't depend on the supplemental temp file. The rule for webcore built sources starting with JS is not executed, I guess  it's because all dependencies already exist.

Carlos: Sorry for the trouble. In my understanding, the supplemental change should not have changed dependencies in makefile (but actually it has changed the dependencies).

Just a random thought, but maybe is it related to .SECONDARY? What happens if we remove two ".SECONDARY"s from $(SUPPLEMENTAL_DEPENDENCY_FILE) and DerivedSources/WebCore/JS%.h?

[Carlos Garcia Campos]

(In reply to comment #19)
> (In reply to comment #17)
> > These are the ones that have an explicit rule in the Makefile and don't depend on the supplemental temp file. The rule for webcore built sources starting with JS is not executed, I guess  it's because all dependencies already exist.
> 
> Carlos: Sorry for the trouble. In my understanding, the supplemental change should not have changed dependencies in makefile (but actually it has changed the dependencies).
> 
> Just a random thought, but maybe is it related to .SECONDARY? What happens if we remove two ".SECONDARY"s from $(SUPPLEMENTAL_DEPENDENCY_FILE) and DerivedSources/WebCore/JS%.h?

I think I already tried to remove the .SECONDARY rules and it didn't help. Btw, .SECONDARY rule with no deps marks all targets in the makefile as secondary, so we don't need two .SECONDARY rules there

[Martin Robinson]

I just spent hours debugging this issue, but I think I've managed to produce a fix. As far as I can tell the problem is this:

1. BUILT_SOURCES doesn't do anything except put certain targets at the front of the source list by making all depend on $(BUILT_SOURCES).
2. If those targets cannot be built, because a dependency hasn't been generated yet, GNUmake is smart enough to build them later.
3. Once this happens GNUmake will continue walking down the target list past BUILT_SOURCES into non-autogenerated sources. Ideally $(BUILT_SOURCES) should be built serially before the rest of the 'all' target dependencies, but autotools doesn't do that. It may not even be possible without a new hook in GNUmake.

This is even more of an issue because generating the supplemental IDL information is really slow. It's often the case that GNUmake will try to build a file that depends on a generated-by-IDL file before the supplemental IDL target finishes.

The solution here, as with most problems in GNUmake is to properly specify dependencies.  Patch incoming.

[Kentaro Hara]

(In reply to comment #21)
> I just spent hours debugging this issue, but I think I've managed to produce a fix. 

Ah, really great, thanks! I also just spent hours debugging this issue today but have not yet reached the solution:-(

[Martin Robinson]

Committed r108523: <http://trac.webkit.org/changeset/108523>

[Kentaro Hara]

Really thanks!

[Philippe Normand]

If I disable WebKit2 build, jsc-3 is created but the build fails later on:

  CXXLD  Programs/jsc-3
  CXXLD  libjavascriptcoregtk-3.0.la
  CXXLD  Libraries/libTestRunnerInjectedBundle.la
libtool: link: cannot find the library `libjavascriptcoregtk-3.0.la' or unhandled argument `libjavascriptcoregtk-3.0.la'
make[1]: *** [Programs/minidom] Error 1
make[1]: *** Waiting for unfinished jobs....
libtool: link: cannot find the library `libjavascriptcoregtk-3.0.la' or unhandled argument `libjavascriptcoregtk-3.0.la'
make[1]: *** [Programs/jsc-3] Error 1
make[1]: Leaving directory `/home/phil/gst/jhbuild/build/WebKit/WebKitBuild/tmp/Release'
make: *** [all] Error 2

Could this whole issue be related with r108523 ?

[Philippe Normand]

wrong bug, sorry