Bug 212500 – REGRESSION (r262245): WebCore's "Check .xcfilelists" build phase is ~100x slower

• Component: Tools / Tests
• Status: RESOLVED DUPLICATE

[Andy Estes]

Prior to <https://trac.webkit.org/r262245>, when building the 'All Source' scheme in WebKit.xcworkspace, WebCore's "Check.xcfilelists" build phase took ~1 second. With r262245, it takes ~95 seconds.

Modifying Source/WebCore/Scripts/check-xcfilelists.sh to invoke generate-xcfilelists with the "--debug" flag, I see that almost all the time is spent in a subshell invoking Source/WebCore/Scripts/generate-derived-sources.sh.

[Radar WebKit Bug Importer]

<rdar://problem/63736777>

[Darin Adler]

The changes in the patch in bug 212420 might fix this. For one thing, it won’t call generate-derived-sources.sh unless something has changed.

[Darin Adler]

I have no idea why this happened, but I do think that the patch from that other bug will make the problem disappear.

[Darin Adler]

Maybe I am being overconfident here; I don’t really know the details of what check xcfilelists build phase does, so I am not sure how I caused this. We could roll out until we figure it out?

Keith, do you understand what this could be?

[Andy Estes]

(In reply to Darin Adler from comment #2)
> The changes in the patch in bug 212420 might fix this. For one thing, it
> won’t call generate-derived-sources.sh unless something has changed.

I applied attachment #400525 [details] to my tree but I still see ~95s runtimes for WebCore's "Check .xcfilelists".

[Keith Rollin]

(In reply to Darin Adler from comment #4)
> 
> Keith, do you understand what this could be?

Not off the top of my head. Andy says that the time is spent in generate-derived-sources.sh. The xcfilelist checking process invokes this script to run `make -f DerivedSources.make -d` so that it can capture the dependency information. There's not much to the script other than invoking `make` so I'd think all the time would be spent in `make`. I wonder if there's something weird about your changes and in invoking `make` with the -d option (which is what dumps the dependency information). Could `make` now be dumping tons of information?

[Darin Adler]

(In reply to Keith Rollin from comment #6)
> The xcfilelist checking process invokes this
> script to run `make -f DerivedSources.make -d` so that it can capture the
> dependency information.

Oh! Had no idea. I thought that DerivedSources.make was only processed once when building. Need to figure out why it’s taking *so* long to process the makefile.

Please roll r262245 out until we can solve this!

[Darin Adler]

It’s possible that make now knows about a lot more dependencies, so the output of "make -d" is much bigger, and either make itself is slow, or the code to process it is slow when there's a lot of data.

It’s possible the invocation of "make -d" is not passing the same environment variables that generate-derived-sources.sh is, leading to something particularly slow.

But neither of those things seems particularly likely. I think we have a mystery.

[Darin Adler]

Maybe the $(shell) commands are evaluated every time you use FEATURE_AND_PLATFORM_DEFINES rather than only once as the file is processed?

Might just need to use an operator other than "=" to fix this.

[Andy Estes]

(In reply to Darin Adler from comment #7)
> (In reply to Keith Rollin from comment #6)
> > The xcfilelist checking process invokes this
> > script to run `make -f DerivedSources.make -d` so that it can capture the
> > dependency information.
> 
> Oh! Had no idea. I thought that DerivedSources.make was only processed once
> when building. Need to figure out why it’s taking *so* long to process the
> makefile.
> 
> Please roll r262245 out until we can solve this!

Reverted in https://trac.webkit.org/r262303.

[Andy Estes]

*** Bug 212531 has been marked as a duplicate of this bug. ***

[Darin Adler]

Yes!

The fix is to write ":=" instead of "=" on these lines using @(shell).

Of course I figure this out right after getting the patch rolled out.

[Andy Estes]

*** This bug has been marked as a duplicate of bug 212451 ***