Bug 188754 – Tool to fix folders that should map to directories in Xcode project files

• Component: Tools / Tests
• Status: NEW

[David Kilzer (:ddkilzer)]

In Bug 188700, Andy Estes manually fixed a folder in the WebKit Xcode project to map to an actual directory on disk.

It turns out these changes can be automated, so let's write a script to do it.

Doing this also exposed inconsistencies where files are located in the WebCore project, and found a strange issue where a group defined both a name and a path in the WebCore project.

[David Kilzer (:ddkilzer)]

While writing the script, one thing I noticed was that the `mainGroup` in an Xcode project file sometimes has a name, and sometimes doesn't.  (See also Bug  Turns out that the `mainGroup` name is controlled by the `productName` setting in the group entry for the `mainGroup`.

Even more interesting is that all of these projects share the exact same unique identifier (`0867D691FE84028FC02AAC07`) for their `mainGroup`:

$ find Source -name project.pbxproj -exec grep -H 'mainGroup = ' {} \; | grep 0867D691FE84028FC02AAC07 | cut -d: -f1 | sort | uniq 
Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj
Source/ThirdParty/gtest/xcode/Samples/FrameworkSample/WidgetFramework.xcodeproj/project.pbxproj
Source/ThirdParty/gtest/xcode/gtest.xcodeproj/project.pbxproj
Source/ThirdParty/libwebrtc/Source/third_party/boringssl/src/third_party/googletest/xcode/Samples/FrameworkSample/WidgetFramework.xcodeproj/project.pbxproj
Source/ThirdParty/libwebrtc/Source/third_party/boringssl/src/third_party/googletest/xcode/gtest.xcodeproj/project.pbxproj
Source/ThirdParty/libwebrtc/Source/third_party/googletest/googletest/xcode/Samples/FrameworkSample/WidgetFramework.xcodeproj/project.pbxproj
Source/ThirdParty/libwebrtc/Source/third_party/googletest/googletest/xcode/gtest.xcodeproj/project.pbxproj
Source/WebCore/WebCore.xcodeproj/project.pbxproj
Source/WebKit/WebKit.xcodeproj/project.pbxproj
Source/WebKitLegacy/WebKitLegacy.xcodeproj/project.pbxproj

I'd guess that folks on the WebKit project tend to make a copy of an existing Xcode project file and then modify it to create projects.  :)

Also, the `productName` of the WebCore Xcode project was set to `WebKit` ever since it was committed back in <https://trac.webkit.org/r9857>.  I fixed that in these two commits:

<https://trac.webkit.org/r235015>
<https://trac.webkit.org/r235091>

[David Kilzer (:ddkilzer)]

(In reply to David Kilzer (:ddkilzer) from comment #1)
> While writing the script, one thing I noticed was that the `mainGroup` in an
> Xcode project file sometimes has a name, and sometimes doesn't.  (See also
> Bug  Turns out that the `mainGroup` name is controlled by the `productName`
> setting in the group entry for the `mainGroup`.

Oops, it's the `name` setting in the group, not the `productName` setting, that controls the `mainGroup` value.

> Even more interesting is that all of these projects share the exact same
> unique identifier (`0867D691FE84028FC02AAC07`) for their `mainGroup`:
> 
> $ find Source -name project.pbxproj -exec grep -H 'mainGroup = ' {} \; |
> grep 0867D691FE84028FC02AAC07 | cut -d: -f1 | sort | uniq 
> Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj
> Source/ThirdParty/gtest/xcode/Samples/FrameworkSample/WidgetFramework.
> xcodeproj/project.pbxproj
> Source/ThirdParty/gtest/xcode/gtest.xcodeproj/project.pbxproj
> Source/ThirdParty/libwebrtc/Source/third_party/boringssl/src/third_party/
> googletest/xcode/Samples/FrameworkSample/WidgetFramework.xcodeproj/project.
> pbxproj
> Source/ThirdParty/libwebrtc/Source/third_party/boringssl/src/third_party/
> googletest/xcode/gtest.xcodeproj/project.pbxproj
> Source/ThirdParty/libwebrtc/Source/third_party/googletest/googletest/xcode/
> Samples/FrameworkSample/WidgetFramework.xcodeproj/project.pbxproj
> Source/ThirdParty/libwebrtc/Source/third_party/googletest/googletest/xcode/
> gtest.xcodeproj/project.pbxproj
> Source/WebCore/WebCore.xcodeproj/project.pbxproj
> Source/WebKit/WebKit.xcodeproj/project.pbxproj
> Source/WebKitLegacy/WebKitLegacy.xcodeproj/project.pbxproj
> 
> I'd guess that folks on the WebKit project tend to make a copy of an
> existing Xcode project file and then modify it to create projects.  :)
> 
> Also, the `productName` of the WebCore Xcode project was set to `WebKit`
> ever since it was committed back in <https://trac.webkit.org/r9857>.  I
> fixed that in these two commits:

I meant to type "`mainGroup` name" instead of "`productName`" here.

> <https://trac.webkit.org/r235015>
> <https://trac.webkit.org/r235091>

[David Kilzer (:ddkilzer)]

(In reply to David Kilzer (:ddkilzer) from comment #0)
> Doing this also exposed inconsistencies where files are located in the
> WebCore project, and found a strange issue where a group defined both a name
> and a path in the WebCore project.

Fixed the last item here:

REGRESSION (r223192): Remove invalid `path` for Modules/geolocation/ios folder
<https://trac.webkit.org/r235092>

[Radar WebKit Bug Importer]

<rdar://problem/43546353>

[David Kilzer (:ddkilzer)]

WebCore contained over 80 files (mostly css/js/png resources) that had been deleted from the source repository, but mot from the Xcode project.  These were fixed in:

<https://trac.webkit.org/r235164>

The script will refuse to make any changes if it finds any missing files because it can't reason about why they're missing.  (I'll include a --force switch in case you really know what you're doing, though.)

--

The last clean-up step before running the script for WebCore (and posting a patch) is to move files within the project to reside in the directories they're supposed to be in per the Xcode project.  There are 6 files out-of-place currently:

$ ./Tools/Scripts/fix-directory-paths-in-Xcode-project-file --force Source/WebCore/WebCore.xcodeproj/
MOVE: Source/WebCore/LocalizableStrings.pm Source/WebCore/Scripts/LocalizableStrings.pm
MOVE: Source/WebCore/extract-localizable-strings.pl Source/WebCore/Scripts/extract-localizable-strings.pl
MOVE: Source/WebCore/platform/audio/WebAudioBufferList.h Source/WebCore/platform/audio/cocoa/WebAudioBufferList.h
MOVE: Source/WebCore/platform/audio/WebAudioBufferList.cpp Source/WebCore/platform/audio/cocoa/WebAudioBufferList.cpp
MOVE: Source/WebCore/platform/graphics/avfoundation/objc/../MediaSampleAVFObjC.h Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h
MOVE: Source/WebCore/platform/ios/wak/../../WAKViewInternal.h Source/WebCore/platform/ios/wak/WAKViewInternal.h

[David Kilzer (:ddkilzer)]

BTW, the other advantage to mapping Xcode folders to directories on disk is that it's super-easy to know when then Xcode project is missing files from that directory, such as the Modules/modern-media-controls directory.

[David Kilzer (:ddkilzer)]

One thing I should note: This script currently finds directories that exist on disk that match a folder in the Xcode project, then updates the Xcode project to use the directory on disk.

It does not (currently) try to find folders that exist in the Xcode project, but do not exist on disk, and create them.  We can do that as a future enhancement request.

[Jer Noble]

(In reply to David Kilzer (:ddkilzer) from comment #5)
> The last clean-up step before running the script for WebCore (and posting a
> patch) is to move files within the project to reside in the directories
> they're supposed to be in per the Xcode project.  There are 6 files
> out-of-place currently:

(snip)

> MOVE: Source/WebCore/platform/audio/WebAudioBufferList.h
> Source/WebCore/platform/audio/cocoa/WebAudioBufferList.h
> MOVE: Source/WebCore/platform/audio/WebAudioBufferList.cpp
> Source/WebCore/platform/audio/cocoa/WebAudioBufferList.cpp
> MOVE:
> Source/WebCore/platform/graphics/avfoundation/objc/../MediaSampleAVFObjC.h
> Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h

These moves all look correct.

[David Kilzer (:ddkilzer)]

Thinking about calling this script tidy-Xcode-project or tidy-Xcode-project-file, and folding sort-Xcode-project-file into it (and turning sort-Xcode-project-file into a script that runs the new one).

[David Kilzer (:ddkilzer)]

Created attachment 348331 [details]
Patch v1

[David Kilzer (:ddkilzer)]

Created attachment 348344 [details]
Patch v2

Fix some false-positive MISSING warnings.

[David Kilzer (:ddkilzer)]

Created attachment 348345 [details]
Patch v3

Fix more false-positive MISSING warnings by switching -f tests to -e tests to include folders.

[David Kilzer (:ddkilzer)]

Changes to clean up Xcode project files in preparation for this tool:

<https://trac.webkit.org/r235439>
<https://trac.webkit.org/r235443>
<https://trac.webkit.org/r235455>

[David Kilzer (:ddkilzer)]

More changes to clean up Xcode project files in preparation for this tool:

<https://trac.webkit.org/r235541>
<https://trac.webkit.org/r235542>
<https://trac.webkit.org/r235546>

[David Kilzer (:ddkilzer)]

Adding a -m|--missing command-line switch (to validate my understanding of how paths are built while traversing folders) found some more issues that weren't found by the -g|--convert-groups switch (because the code to check for possible conversions did not see the group or the enclosing group as candidates for conversion, or because of bugs in the conversion code).

<https://trac.webkit.org/r235663>
<https://trac.webkit.org/r235664>
<https://trac.webkit.org/r235665>
<https://trac.webkit.org/r235666>

[David Kilzer (:ddkilzer)]

A few more clean-ups using the -m|--missing command-line switch:

<https://trac.webkit.org/r235672>
<https://trac.webkit.org/r235687>
<https://trac.webkit.org/r235708>
<https://trac.webkit.org/r235709>

[David Kilzer (:ddkilzer)]

Additional fixes since then:

<https://trac.webkit.org/r240196>

[David Kilzer (:ddkilzer)]

This clean-up was done using `tidy-Xcode-project-file --missing --dry-run`:

Clean up libwebrtc Xcode project
https://bugs.webkit.org/show_bug.cgi?id=262527
Committed 268771@main (34ebac9a5f68): <https://commits.webkit.org/268771@main>

[David Kilzer (:ddkilzer)]

Created attachment 468044 [details]
Patch v4

- Fix warnings in later versions of Perl 5 about unescaped left curly brace ({) in a `m##` regular expression.
- Merge patch to commit 268771@main.

[David Kilzer (:ddkilzer)]

(In reply to David Kilzer (:ddkilzer) from comment #18)
> This clean-up was done using `tidy-Xcode-project-file --map-folders --dry-run`:
> 
> Clean up libwebrtc Xcode project
> https://bugs.webkit.org/show_bug.cgi?id=262527
> Committed 268771@main (34ebac9a5f68): <https://commits.webkit.org/268771@main>

Then committed the automated fixes from the `tidy-Xcode-project-file` tool:

Committed 268773@main (36c3a8590f2b): <https://commits.webkit.org/268773@main>