Bug 183788 – [GTK][STABLE] Non-deterministic build failure (missing JavaScriptCore/JSContextRef.h)

• Component: WebKitGTK
• Status: RESOLVED FIXED

[Jan Tojnar]

Building WebkitGTK on NixOS fails non-deterministically with the following error:

FAILED: Source/JavaScriptCore/CMakeFiles/LLIntOffsetsExtractor.dir/llint/LLIntOffsetsExtractor.cpp.o
/nix/store/kw0c6x65jx28d5mr6zbb6978w61b8n93-gcc-wrapper-6.4.0/bin/g++  -DBUILDING_GTK__=1 -DBUILDING_WITH_CMAKE=1 -DDATA_DIR=\"share\" -DGETTEXT_PACKAGE=\"WebKit2GTK-4.0\" -DHAVE_CONFIG_H=1 -DLIBDIR=\"/nix/store/fqncjlck2aqdmr8c0r9c8jj4c3819v51-webkitgtk-2.20.0/lib\" -DSTATICALLY_LINKED_WITH_WTF -DWEBKITGTK_API_VERSION_STRING=\"4.0\" -IDerivedSources/ForwardingHeaders -I. -I../Source/JavaScriptCore -I../Source/JavaScriptCore/API -I../Source/JavaScriptCore/assembler -I../Source/JavaScriptCore/b3 -I../Source/JavaScriptCore/b3/air -I../Source/JavaScriptCore/bindings -I../Source/JavaScriptCore/builtins -I../Source/JavaScriptCore/bytecode -I../Source/JavaScriptCore/bytecompiler -I../Source/JavaScriptCore/dfg -I../Source/JavaScriptCore/disassembler -I../Source/JavaScriptCore/disassembler/ARM64 -I../Source/JavaScriptCore/disassembler/ARMv7 -I../Source/JavaScriptCore/disassembler/udis86 -I../Source/JavaScriptCore/domjit -I../Source/JavaScriptCore/ftl -I../Source/JavaScriptCore/heap -I../Source/JavaScriptCore/debugger -I../Source/JavaScriptCore/inspector -I../Source/JavaScriptCore/inspector/agents -I../Source/JavaScriptCore/inspector/augmentable -I../Source/JavaScriptCore/inspector/remote -I../Source/JavaScriptCore/interpreter -I../Source/JavaScriptCore/jit -I../Source/JavaScriptCore/llint -I../Source/JavaScriptCore/parser -I../Source/JavaScriptCore/profiler -I../Source/JavaScriptCore/runtime -I../Source/JavaScriptCore/tools -I../Source/JavaScriptCore/wasm -I../Source/JavaScriptCore/wasm/js -I../Source/JavaScriptCore/yarr -IDerivedSources/JavaScriptCore -IDerivedSources/JavaScriptCore/inspector -IDerivedSources/JavaScriptCore/runtime -IDerivedSources/JavaScriptCore/yarr -I../Source/JavaScriptCore/inspector/remote/glib -isystem /nix/store/slggzv99kinwm4c2lm7wfmifh8y69l7r-glib-2.56.0-dev/include/glib-2.0 -isystem /nix/store/dqlc8y4phlg1hmdbwkhqfwhnxcac88d1-glib-2.56.0/lib/glib-2.0/include -I../Source/bmalloc -IDerivedSources -I../Source/ThirdParty -fdiagnostics-color=always -Wno-attributes -Wno-maybe-uninitialized -Wwrite-strings -Wundef -Wpointer-arith -Wmissing-format-attribute -Wformat-security -Wcast-align -Wextra -Wall  -fno-strict-aliasing -fno-exceptions -std=c++14 -fno-rtti -O3 -DNDEBUG -fPIE -MD -MT Source/JavaScriptCore/CMakeFiles/LLIntOffsetsExtractor.dir/llint/LLIntOffsetsExtractor.cpp.o -MF Source/JavaScriptCore/CMakeFiles/LLIntOffsetsExtractor.dir/llint/LLIntOffsetsExtractor.cpp.o.d -o Source/JavaScriptCore/CMakeFiles/LLIntOffsetsExtractor.dir/llint/LLIntOffsetsExtractor.cpp.o -c ../Source/JavaScriptCore/llint/LLIntOffsetsExtractor.cpp
In file included from ../Source/JavaScriptCore/heap/HeapFinalizerCallback.h:28:0,
                 from ../Source/JavaScriptCore/heap/Heap.h:33,
                 from ../Source/JavaScriptCore/heap/DeferGC.h:29,
                 from ../Source/JavaScriptCore/runtime/ConcurrentJSLock.h:28,
                 from ../Source/JavaScriptCore/bytecode/ArrayProfile.h:28,
                 from ../Source/JavaScriptCore/llint/LLIntOffsetsExtractor.cpp:28:
../Source/JavaScriptCore/API/JSHeapFinalizerPrivate.h:29:41: fatal error: JavaScriptCore/JSContextRef.h: No such file or directory
 #include <JavaScriptCore/JSContextRef.h>
                                         ^

[Adrian Perez]

I've seen this happen in the EWS that I maintain, but back then I
just went for a wipe of the build directory and that fixed things,
so I assumed that the build failure had been caused by some patch
that the EWS had tried and didn't recover from. I will try to
investigate if this happens again in my EWS.

Jan: By any chance did this happen to you in a machine with many
CPUs? I have only seen the issue in the EWS, which runs on a 64-core
box, but I've never seen it in my work laptop. It looks a bit like
it could be some parallel build issue, which would explain why it
triggers more easily with a high core count.

[Jan Tojnar]

Undeclared dependencies causing trouble during parallel builds is what I would suspect but we have a report of failure with parallel builds disabled (https://github.com/NixOS/nixpkgs/pull/36893#issuecomment-373922664).

I can reproduce it on my four core laptop (build flags: -j4 -l4).

[Jan Tojnar]

Lowering the parallelism seemingly increases the chance of encountering this issue https://github.com/NixOS/nixpkgs/issues/37878#comment-377233401

[li]

This bug happens every time for me. I build it with ninja -j1.
I think the problem is in some CMake rules.
The missing header JavaScriptCore/JsContextRef.h should be generated during build, as with many other headers, they're are copied from their respective locations to build/DerivedSources/ForwardingHeaders. There's a rule in build/build.ninja to make this happen: 
build JavascriptCoreForwardingHeaders: phony Source/JavaScriptCore/JavaScriptCoreForwardingHeaders

This rule should be a prerequisite of JavaScriptCore, but it doesn't happen due to some bugs in Cmake rules I suppose.

The lines in Cmake files to generate the ninja rules are at: Source/JavaScriptCore/CMakeLists.txt:1152,
it calls function WEBKIT_MAKE_FORWARDING_HEADERS, which is defined at Source/cmake/WebKitMacros.cmake
:218

Here line 267 might be the problem: add_dependencies(...)
in short, this line should instruct cmake to add JavaScriptCoreForwardingHeaders as a prerequisite for JavaScriptCore in the generated ninja build file, but for some reason it doesn't work.

Due to my limited knowledge to CMake, I couldn't​ figure out the right way to fix this. Well still there's a workaround, that is to call ninja to generate those headers explicitly. Do this after cmake and before you run ninja to build: 
ninja JavaScriptCoreForwardingHeaders WTFForwardingHeaders

[Michael Catanzaro]

*** Bug 187894 has been marked as a duplicate of this bug. ***

[Fujii Hironori]

https://github.com/WebKit/webkit/blob/2071d472faf10105ea6858e7795e5366f6203667/Source/JavaScriptCore/CMakeLists.txt#L258

> add_dependencies(LLIntOffsetsExtractor JavaScriptCoreForwardingHeaders)

LLIntOffsetsExtractor has a dependency to JavaScriptCoreForwardingHeaders.
But, this is a link-time dependency. It should be a compile-time dependency.

Caused By: Bug 182303 and Bug 183251.

[Fujii Hironori]

I'm wrong. 
A following command doesn't fail in my env.

> ./Tools/Scripts/build-webkit --gtk --release --makeargs="-j1 Source/JavaScriptCore/CMakeFiles/LLIntOffsetsExtractor.dir/llint/LLIntOffsetsExtractor.cpp.o"

Looking into build.ninja, LLIntOffsetsExtractor.cpp.o has a dependency to JavaScriptCoreForwardingHeaders.
I'm using CMake 3.10.2.

[Fujii Hironori]

I can't figure out what is happening in their build env.

I uploaded my build log which successes.

https://gist.github.com/fujii/ed836a1be308c5775482c0bef927e902

> [474/483] Generating ../../DerivedSources/ForwardingHeaders/JavaScriptCore/JSContextRef.h
(...)
> [483/483] Building CXX object Source/JavaScriptCore/CMakeFiles/LLIntOffsetsExtractor.dir/llint/LLIntOffsetsExtractor.cpp.o

ForwardingHeaders/JavaScriptCore/JSContextRef.h was generated before compiling LLIntOffsetsExtractor.cpp.o.

[Fujii Hironori]

Finally, I can reproduce the same build error with webkitgtk-2.20.3.tar.xz.
The source code doesn't have a following code.

> add_dependencies(LLIntOffsetsExtractor JavaScriptCoreForwardingHeaders)

This problem had been introduce in r227952, and fixed in r229282.
webkitgtk-2.20 was branch from trunk@228089.

Bug 182303 (r227952)
Bug 183251 (r229282)

[Michael Catanzaro]

Thanks Fujii! I've added r229282 to the backports list for 2.20.4.

[Michael Catanzaro]

(In reply to Michael Catanzaro from comment #10)
> Thanks Fujii! I've added r229282 to the backports list for 2.20.4.

It doesn't apply cleanly, so I'll do this manually.

See also: https://bugs.gentoo.org/662002

[Michael Catanzaro]

Committed r234198: <https://trac.webkit.org/changeset/234198>