Bug 212500 - REGRESSION (r262245): WebCore's "Check .xcfilelists" build phase is ~100x slower
Summary: REGRESSION (r262245): WebCore's "Check .xcfilelists" build phase is ~100x slower
Status: RESOLVED DUPLICATE of bug 212451
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Local Build
Hardware: Unspecified macOS 10.15
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
: 212531 (view as bug list)
Depends on: 212451
Blocks:
  Show dependency treegraph
 
Reported: 2020-05-28 16:04 PDT by Andy Estes
Modified: 2020-05-29 12:37 PDT (History)
8 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Andy Estes 2020-05-28 16:04:23 PDT
Prior to <https://trac.webkit.org/r262245>, when building the 'All Source' scheme in WebKit.xcworkspace, WebCore's "Check.xcfilelists" build phase took ~1 second. With r262245, it takes ~95 seconds.

Modifying Source/WebCore/Scripts/check-xcfilelists.sh to invoke generate-xcfilelists with the "--debug" flag, I see that almost all the time is spent in a subshell invoking Source/WebCore/Scripts/generate-derived-sources.sh.
Comment 1 Radar WebKit Bug Importer 2020-05-28 16:08:18 PDT
<rdar://problem/63736777>
Comment 2 Darin Adler 2020-05-28 16:35:25 PDT
The changes in the patch in bug 212420 might fix this. For one thing, it won’t call generate-derived-sources.sh unless something has changed.
Comment 3 Darin Adler 2020-05-28 17:46:47 PDT
I have no idea why this happened, but I do think that the patch from that other bug will make the problem disappear.
Comment 4 Darin Adler 2020-05-29 09:16:29 PDT
Maybe I am being overconfident here; I don’t really know the details of what check xcfilelists build phase does, so I am not sure how I caused this. We could roll out until we figure it out?

Keith, do you understand what this could be?
Comment 5 Andy Estes 2020-05-29 09:47:19 PDT
(In reply to Darin Adler from comment #2)
> The changes in the patch in bug 212420 might fix this. For one thing, it
> won’t call generate-derived-sources.sh unless something has changed.

I applied attachment #400525 [details] to my tree but I still see ~95s runtimes for WebCore's "Check .xcfilelists".
Comment 6 Keith Rollin 2020-05-29 10:22:10 PDT
(In reply to Darin Adler from comment #4)
> 
> Keith, do you understand what this could be?

Not off the top of my head. Andy says that the time is spent in generate-derived-sources.sh. The xcfilelist checking process invokes this script to run `make -f DerivedSources.make -d` so that it can capture the dependency information. There's not much to the script other than invoking `make` so I'd think all the time would be spent in `make`. I wonder if there's something weird about your changes and in invoking `make` with the -d option (which is what dumps the dependency information). Could `make` now be dumping tons of information?
Comment 7 Darin Adler 2020-05-29 10:35:39 PDT
(In reply to Keith Rollin from comment #6)
> The xcfilelist checking process invokes this
> script to run `make -f DerivedSources.make -d` so that it can capture the
> dependency information.

Oh! Had no idea. I thought that DerivedSources.make was only processed once when building. Need to figure out why it’s taking *so* long to process the makefile.

Please roll r262245 out until we can solve this!
Comment 8 Darin Adler 2020-05-29 10:40:25 PDT
It’s possible that make now knows about a lot more dependencies, so the output of "make -d" is much bigger, and either make itself is slow, or the code to process it is slow when there's a lot of data.

It’s possible the invocation of "make -d" is not passing the same environment variables that generate-derived-sources.sh is, leading to something particularly slow.

But neither of those things seems particularly likely. I think we have a mystery.
Comment 9 Darin Adler 2020-05-29 10:42:38 PDT
Maybe the $(shell) commands are evaluated every time you use FEATURE_AND_PLATFORM_DEFINES rather than only once as the file is processed?

Might just need to use an operator other than "=" to fix this.
Comment 10 Andy Estes 2020-05-29 10:43:40 PDT
(In reply to Darin Adler from comment #7)
> (In reply to Keith Rollin from comment #6)
> > The xcfilelist checking process invokes this
> > script to run `make -f DerivedSources.make -d` so that it can capture the
> > dependency information.
> 
> Oh! Had no idea. I thought that DerivedSources.make was only processed once
> when building. Need to figure out why it’s taking *so* long to process the
> makefile.
> 
> Please roll r262245 out until we can solve this!

Reverted in https://trac.webkit.org/r262303.
Comment 11 Andy Estes 2020-05-29 10:43:56 PDT
*** Bug 212531 has been marked as a duplicate of this bug. ***
Comment 12 Darin Adler 2020-05-29 10:45:47 PDT
Yes!

The fix is to write ":=" instead of "=" on these lines using @(shell).

Of course I figure this out right after getting the patch rolled out.
Comment 13 Andy Estes 2020-05-29 12:37:24 PDT

*** This bug has been marked as a duplicate of bug 212451 ***