Bug 212436 – REGRESSION (r262203): Build errors during JavaScriptCore "Generate Derived Sources" about missing Availability.h header and invalid SDKROOT [-Wmissing-sysroot]

• Status: RESOLVED
• Resolution: FIXED
• Component: JavaScriptCore

[David Kilzer (:ddkilzer)]

Build errors during JavaScriptCore "Generate Derived Sources" about missing Availability.h header and invalid SDKROOT [-Wmissing-sysroot].

[...]
=== BUILD AGGREGATE TARGET Derived Sources OF PROJECT JavaScriptCore WITH CONFIGURATION Debug ===

Check dependencies

PhaseScriptExecution Generate\ Derived\ Sources /var/build/JavaScriptCore.build/Debug/Derived\ Sources.build/Script-65FB3F6509D11E9100F49DEB.sh
    cd /Users/ddkilzer/Data/WebKit.git/Source/JavaScriptCore
    /bin/sh -c \"/var/build/JavaScriptCore.build/Debug/Derived\ Sources.build/Script-65FB3F6509D11E9100F49DEB.sh\"
clang: warning: no such sysroot directory: 'macosx10.14internal' [-Wmissing-sysroot]
In file included from <built-in>:1:
In file included from /var/build/Debug/usr/local/include/wtf/Platform.h:44:
/var/build/Debug/usr/local/include/wtf/PlatformOS.h:36:10: fatal error: 'Availability.h' file not found
#include <Availability.h>
         ^~~~~~~~~~~~~~~~
1 error generated.
clang: warning: no such sysroot directory: 'macosx10.14internal' [-Wmissing-sysroot]
In file included from <built-in>:1:
In file included from /var/build/Debug/usr/local/include/wtf/Platform.h:44:
/var/build/Debug/usr/local/include/wtf/PlatformOS.h:36:10: fatal error: 'Availability.h' file not found
#include <Availability.h>
         ^~~~~~~~~~~~~~~~
1 error generated.
clang: warning: no such sysroot directory: 'macosx10.14internal' [-Wmissing-sysroot]
In file included from <built-in>:1:
In file included from /var/build/Debug/usr/local/include/wtf/Platform.h:44:
/var/build/Debug/usr/local/include/wtf/PlatformOS.h:36:10: fatal error: 'Availability.h' file not found
#include <Availability.h>
         ^~~~~~~~~~~~~~~~
1 error generated.
clang: warning: no such sysroot directory: 'macosx10.14internal' [-Wmissing-sysroot]
In file included from <built-in>:1:
In file included from /var/build/Debug/usr/local/include/wtf/Platform.h:44:
/var/build/Debug/usr/local/include/wtf/PlatformOS.h:36:10: fatal error: 'Availability.h' file not found
#include <Availability.h>
         ^~~~~~~~~~~~~~~~
1 error generated.
python JavaScriptCore/Scripts/UpdateContents.py 'JavaScriptCore/builtins/AsyncFromSyncIteratorPrototype.js JavaScriptCore/builtins/ArrayConstructor.js JavaScriptCore/builtins/ArrayIteratorPrototype.js JavaScriptCore/builtins/ArrayPrototype.js JavaScriptCore/builtins/AsyncIteratorPrototype.js JavaScriptCore/builtins/AsyncFunctionPrototype.js JavaScriptCore/builtins/AsyncGeneratorPrototype.js JavaScriptCore/builtins/DatePrototype.js JavaScriptCore/builtins/FunctionPrototype.js JavaScriptCore/builtins/GeneratorPrototype.js JavaScriptCore/builtins/GlobalObject.js JavaScriptCore/builtins/GlobalOperations.js JavaScriptCore/builtins/InspectorInstrumentationObject.js JavaScriptCore/builtins/InternalPromiseConstructor.js JavaScriptCore/builtins/IteratorHelpers.js JavaScriptCore/builtins/IteratorPrototype.js JavaScriptCore/builtins/MapIteratorPrototype.js JavaScriptCore/builtins/MapPrototype.js JavaScriptCore/builtins/ModuleLoader.js JavaScriptCore/builtins/NumberConstructor.js JavaScriptCore/builtins/ObjectConstructor.js JavaScriptCore/builtins/PromiseConstructor.js JavaScriptCore/builtins/PromiseOperations.js JavaScriptCore/builtins/PromisePrototype.js JavaScriptCore/builtins/ReflectObject.js JavaScriptCore/builtins/RegExpPrototype.js JavaScriptCore/builtins/RegExpStringIteratorPrototype.js JavaScriptCore/builtins/SetIteratorPrototype.js JavaScriptCore/builtins/SetPrototype.js JavaScriptCore/builtins/StringConstructor.js JavaScriptCore/builtins/StringIteratorPrototype.js JavaScriptCore/builtins/StringPrototype.js JavaScriptCore/builtins/TypedArrayConstructor.js JavaScriptCore/builtins/TypedArrayPrototype.js JavaScriptCore/builtins/WebAssembly.js  JavaScriptCore/Scripts/wkbuiltins/__init__.py JavaScriptCore/Scripts/wkbuiltins/wkbuiltins.py JavaScriptCore/Scripts/wkbuiltins/builtins_generator.py JavaScriptCore/Scripts/wkbuiltins/builtins_model.py JavaScriptCore/Scripts/wkbuiltins/builtins_templates.py JavaScriptCore/Scripts/wkbuiltins/builtins_generate_combined_header.py JavaScriptCore/Scripts/wkbuiltins/builtins_generate_combined_implementation.py JavaScriptCore/Scripts/wkbuiltins/builtins_generate_separate_header.py JavaScriptCore/Scripts/wkbuiltins/builtins_generate_separate_implementation.py JavaScriptCore/Scripts/wkbuiltins/builtins_generate_wrapper_header.py JavaScriptCore/Scripts/wkbuiltins/builtins_generate_wrapper_implementation.py JavaScriptCore/Scripts/wkbuiltins/builtins_generate_internals_wrapper_header.py JavaScriptCore/Scripts/wkbuiltins/builtins_generate_internals_wrapper_implementation.py JavaScriptCore/Scripts/generate-js-builtins.py JavaScriptCore/Scripts/lazywriter.py ' JavaScriptCore_BUILTINS_DEPENDENCIES_LIST
python JavaScriptCore/Scripts/UpdateContents.py 'JavaScriptCore/inspector/protocol/Animation.json JavaScriptCore/inspector/protocol/ApplicationCache.json JavaScriptCore/inspector/protocol/Audit.json JavaScriptCore/inspector/protocol/Browser.json JavaScriptCore/inspector/protocol/CPUProfiler.json JavaScriptCore/inspector/protocol/CSS.json JavaScriptCore/inspector/protocol/Canvas.json JavaScriptCore/inspector/protocol/Console.json JavaScriptCore/inspector/protocol/DOM.json JavaScriptCore/inspector/protocol/DOMDebugger.json JavaScriptCore/inspector/protocol/DOMStorage.json JavaScriptCore/inspector/protocol/Database.json JavaScriptCore/inspector/protocol/Debugger.json JavaScriptCore/inspector/protocol/GenericTypes.json JavaScriptCore/inspector/protocol/Heap.json JavaScriptCore/inspector/protocol/IndexedDB.json JavaScriptCore/inspector/protocol/Inspector.json JavaScriptCore/inspector/protocol/LayerTree.json JavaScriptCore/inspector/protocol/Memory.json JavaScriptCore/inspector/protocol/Network.json JavaScriptCore/inspector/protocol/Page.json JavaScriptCore/inspector/protocol/Recording.json JavaScriptCore/inspector/protocol/Runtime.json JavaScriptCore/inspector/protocol/ScriptProfiler.json JavaScriptCore/inspector/protocol/Security.json JavaScriptCore/inspector/protocol/ServiceWorker.json JavaScriptCore/inspector/protocol/Target.json JavaScriptCore/inspector/protocol/Timeline.json JavaScriptCore/inspector/protocol/Worker.json ' EnabledInspectorDomains
Command /bin/sh emitted errors but did not return a nonzero exit code to indicate failure

[...]

[David Kilzer (:ddkilzer)]

This fixes it:

$ git diff --cached
diff --git a/Source/JavaScriptCore/Scripts/generate-derived-sources.sh b/Source/JavaScriptCore/Scripts/generate-derived-sources.sh
index 2414765431d..17939f2c920 100755
--- a/Source/JavaScriptCore/Scripts/generate-derived-sources.sh
+++ b/Source/JavaScriptCore/Scripts/generate-derived-sources.sh
@@ -15,4 +15,4 @@ if [ ! $CC ]; then
     export CC="`xcrun -find clang`"
 fi
 
-make --no-builtin-rules -f "JavaScriptCore/DerivedSources.make" -j `/usr/sbin/sysctl -n hw.ncpu` "${ARGS[@]}"
+make --no-builtin-rules -f "JavaScriptCore/DerivedSources.make" -j `/usr/sbin/sysctl -n hw.ncpu` SDKROOT="${SDKROOT}" "${ARGS[@]}"

Will check in momentarily.  Building with Apple internal SDK on 10.14.6.

[David Kilzer (:ddkilzer)]

I suspect this issue may have always been present, but it didn't matter previously because the <Availability.h> header wasn't being included during this step before.

[David Kilzer (:ddkilzer)]

I was building using these commands (both of which worked until recently):

    make debug SDKROOT=macosx.internal
    make debug SDKROOT=macosx10.14internal

Maybe a regression from Bug 209890 (untested):

    Enable the use of XCBuild by default in Apple builds
    <https://bugs.webkit.org/show_bug.cgi?id=209890>

[David Kilzer (:ddkilzer)]

Created attachment 400386 [details]
Patch v1

[David Kilzer (:ddkilzer)]

Letting EWS chew on this before landing an unreviewed build fix.

[Radar WebKit Bug Importer]

<rdar://problem/63692399>

[Keith Rollin]

Just curious, what changed such that <Availability.h> is now being pulled in?

Also, I would think that SDKROOT is already exported. Why does defining it on the command line work?

[David Kilzer (:ddkilzer)]

Comment on attachment 400386 [details]
Patch v1

All Apple OS targets built, and I reviewed build logs and didn't see any errors on the build phase script that changed.  Marking cq+.

[David Kilzer (:ddkilzer)]

(In reply to Keith Rollin from comment #7)
> Just curious, what changed such that <Availability.h> is now being pulled in?

I don't know the answer to this because I've been prioritizing a fix to be committed first.  Will try to answer if I have time today.

> Also, I would think that SDKROOT is already exported. Why does defining it
> on the command line work?

During debugging I printed out all of `env` in the script (and changed the script to add -x to the #! line to see all the environment variables).  In that case, the SDKROOT environment variable was a fully qualified path to the SDK in /Applications/Xcode.app/Contents/.../.

I honestly don't know where JavaScriptCore/DerivedSources.make was getting the original Makefile argument from as it didn't show up as part of "${ARGS[@]}".  I assumed it was some issue with Xcode that I couldn't reason about, and passing the argument explicitly fixed it, so that's the change I made.

[David Kilzer (:ddkilzer)]

(In reply to David Kilzer (:ddkilzer) from comment #9)
> (In reply to Keith Rollin from comment #7)
> > Just curious, what changed such that <Availability.h> is now being pulled in?
> 
> I don't know the answer to this because I've been prioritizing a fix to be
> committed first.  Will try to answer if I have time today.
> 
> > Also, I would think that SDKROOT is already exported. Why does defining it
> > on the command line work?
> 
> During debugging I printed out all of `env` in the script (and changed the
> script to add -x to the #! line to see all the environment variables).  In
> that case, the SDKROOT environment variable was a fully qualified path to
> the SDK in /Applications/Xcode.app/Contents/.../.
> 
> I honestly don't know where JavaScriptCore/DerivedSources.make was getting
> the original Makefile argument from as it didn't show up as part of
> "${ARGS[@]}".  I assumed it was some issue with Xcode that I couldn't reason
> about, and passing the argument explicitly fixed it, so that's the change I
> made.

Defining it on the command-line works because Source/JavaScriptCore/DerivedSources.make does this:

ifneq ($(SDKROOT),)
    SDK_FLAGS=-isysroot $(SDKROOT)
endif

So the SDKROOT I pass in on the command-line overrides whatever value it was picking up before.

[EWS]

Committed r262222: <https://trac.webkit.org/changeset/262222>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 400386 [details].

[David Kilzer (:ddkilzer)]

Using `git bisect`, I found the regression was the fix for Bug 210014 in r262203:

    Web Inspector: replace `featureGuard` and `availability` with a combined `condition` that accepts any macro
    https://bugs.webkit.org/show_bug.cgi?id=210014
    https://trac.webkit.org/r262203

That added all this code, which wasn't there before:

+ifneq ($(SDKROOT),)
+    SDK_FLAGS=-isysroot $(SDKROOT)
+endif

Which were then used in these new commands (which didn't exist before):

+ifeq ($(shell $(CC) -std=gnu++1z -x c++ -E -P -dM $(SDK_FLAGS) $(TARGET_TRIPLE_FLAGS) $(FRAMEWORK_FLAGS) $(HEADER_FLAGS) -include "wtf/Platform.h" /d
ev/null | grep ' WTF_PLATFORM_MAC ' | cut -d' ' -f3), 1)
+    PREPROCESSOR_DEFINES += WTF_PLATFORM_MAC
+endif
+
+ifeq ($(shell $(CC) -std=gnu++1z -x c++ -E -P -dM $(SDK_FLAGS) $(TARGET_TRIPLE_FLAGS) $(FRAMEWORK_FLAGS) $(HEADER_FLAGS) -include "wtf/Platform.h" /d
ev/null | grep ' WTF_PLATFORM_IOS_FAMILY ' | cut -d' ' -f3), 1)
+    PREPROCESSOR_DEFINES += WTF_PLATFORM_IOS_FAMILY
+endif
+
+ifeq ($(shell $(CC) -std=gnu++1z -x c++ -E -P -dM $(SDK_FLAGS) $(TARGET_TRIPLE_FLAGS) $(FRAMEWORK_FLAGS) $(HEADER_FLAGS) -include "wtf/Platform.h" /d
ev/null | grep ' USE_CF ' | cut -d' ' -f3), 1)
+    PREPROCESSOR_DEFINES += USE_CF
+endif
+
+ifeq ($(shell $(CC) -std=gnu++1z -x c++ -E -P -dM $(SDK_FLAGS) $(TARGET_TRIPLE_FLAGS) $(FRAMEWORK_FLAGS) $(HEADER_FLAGS) -include "wtf/Platform.h" /d
ev/null | grep HAVE_OS_DARK_MODE_SUPPORT | cut -d' ' -f3), 1)
+    PREPROCESSOR_DEFINES += HAVE_OS_DARK_MODE_SUPPORT
+else

I'm still not sure why DerivedSources.make was using the SDKROOT that I passed in on the `make debug SDKROOT=...` command-line, but I think the fix was correct.

[David Kilzer (:ddkilzer)]

Another point of interest:  Source/WebCore/Scripts/generate-derived-sources.sh also includes the SDKROOT="${SDKROOT}" argument when running a command on Source/WebCore/DerivedSources.make.

It has been present since Source/WebCore/Scripts/generate-derived-sources.sh was a build phase script contained within WebCore.xcodeproj/project.pbxproj, and it was added in r142546:

Bug 109324: Make WebCore Derived Sources work with SDK identifiers too
<https://bugs.webkit.org/show_bug.cgi?id=109324>
<https://trac.webkit.org/r142546>