RESOLVED FIXED 238119
Enable PGO when building for release and production
https://bugs.webkit.org/show_bug.cgi?id=238119
Summary Enable PGO when building for release and production
Wenson Hsieh
Reported 2022-03-19 15:28:27 PDT
.
Attachments
Test attachment (121.96 MB, text/html)
2022-03-19 15:41 PDT, Wenson Hsieh
no flags
Test attachment (121.96 MB, text/html)
2022-03-20 11:49 PDT, Wenson Hsieh
no flags
Test attachment (121.96 MB, text/html)
2022-03-20 16:45 PDT, Wenson Hsieh
no flags
Test attachment (56.40 MB, text/html)
2022-03-21 18:38 PDT, Wenson Hsieh
no flags
v3 (21.40 KB, patch)
2022-03-22 10:17 PDT, Wenson Hsieh
no flags
v4 (24.06 KB, patch)
2022-03-23 12:34 PDT, Wenson Hsieh
no flags
v4 (address feedback) (24.69 KB, patch)
2022-03-23 14:42 PDT, Wenson Hsieh
no flags
Adjust decompression (26.50 KB, patch)
2022-03-24 13:44 PDT, Wenson Hsieh
no flags
Adjust decompression (26.52 KB, patch)
2022-03-24 13:49 PDT, Wenson Hsieh
no flags
Rebase on trunk (26.47 KB, patch)
2022-03-24 14:47 PDT, Wenson Hsieh
no flags
Rebase (again) (25.28 KB, patch)
2022-03-24 15:02 PDT, Wenson Hsieh
no flags
Fix production iOS builds (25.33 KB, patch)
2022-03-24 16:23 PDT, Wenson Hsieh
no flags
Add flag to OTHER_CFLAGS too (26.09 KB, patch)
2022-03-24 18:34 PDT, Wenson Hsieh
no flags
For EWS (26.17 KB, patch)
2022-03-25 10:32 PDT, Wenson Hsieh
no flags
Wenson Hsieh
Comment 1 2022-03-19 15:41:51 PDT Comment hidden (obsolete)
Wenson Hsieh
Comment 2 2022-03-20 11:49:06 PDT Comment hidden (obsolete)
Wenson Hsieh
Comment 3 2022-03-20 16:29:25 PDT
Wenson Hsieh
Comment 4 2022-03-20 16:45:06 PDT Comment hidden (obsolete)
Saam Barati
Comment 5 2022-03-20 20:34:02 PDT
r=me. (Patch review isn't actually letting me set the r+ flag though b/c the patch size)
Wenson Hsieh
Comment 6 2022-03-21 07:53:00 PDT
(In reply to Saam Barati from comment #5) > r=me. (Patch review isn't actually letting me set the r+ flag though b/c the > patch size) Thanks for the review! In lieu of EWS, I ran a (large) subset of the layout tests on macOS locally, and confirmed that it doesn't break anything 🤞🏻. I'll commit this manually, and will also be keeping an eye out for the post-commit bots (and EWS).
Wenson Hsieh
Comment 7 2022-03-21 08:26:29 PDT
Jonathan Bedard
Comment 8 2022-03-21 09:29:16 PDT
Aakash Jain
Comment 9 2022-03-21 11:36:40 PDT
For what it's worth, EWS also has a limit of 100 MB (for total patch size) as set in https://github.com/WebKit/WebKit/blob/main/Tools/CISupport/ews-build/master.cfg#L26 That's why EWS wasn't run on this patch.
Geoffrey Garen
Comment 10 2022-03-21 11:55:55 PDT
r=me on "Don't limit to ARM Macs" Not sure how to set r+, or how to land this patch, with the 100MB file
Saam Barati
Comment 11 2022-03-21 13:44:47 PDT
(In reply to Aakash Jain from comment #9) > For what it's worth, EWS also has a limit of 100 MB (for total patch size) > as set in > https://github.com/WebKit/WebKit/blob/main/Tools/CISupport/ews-build/master. > cfg#L26 > > That's why EWS wasn't run on this patch. Can we remove that limit?
Wenson Hsieh
Comment 12 2022-03-21 18:38:14 PDT Comment hidden (obsolete)
Wenson Hsieh
Comment 13 2022-03-21 18:38:19 PDT Comment hidden (obsolete)
Wenson Hsieh
Comment 14 2022-03-21 18:39:39 PDT
(In reply to Wenson Hsieh from comment #13) > Created attachment 455314 [details] > v2 (This version of the patch includes only a tarballs of the raw profiling data, which brings the in-repo size down from 300 MB to about 44 MB).
Wenson Hsieh
Comment 15 2022-03-22 10:17:57 PDT Comment hidden (obsolete)
Wenson Hsieh
Comment 16 2022-03-22 10:30:32 PDT
(In reply to Wenson Hsieh from comment #15) > Created attachment 455386 [details] > v3 This latest iteration of the patch does not include the JavaScriptCore/WebCore/WebKit profiling data at all, and instead only adds build system support for incorporating compressed .profdata files when building for Release/Production configurations. If the appropriate data can't be found, we simply fall back to a tiny, (mostly) empty profdata file. On the appropriate build configurations, profiling data will be downloaded prior to building the WebKit stack instead of falling back to the empty profiling data.
Wenson Hsieh
Comment 17 2022-03-22 10:58:10 PDT Comment hidden (obsolete)
Geoffrey Garen
Comment 18 2022-03-22 11:03:20 PDT
Comment on attachment 455386 [details] v3 r=me Seems like a fine approach.
Alexey Proskuryakov
Comment 19 2022-03-22 11:48:38 PDT
Comment on attachment 455386 [details] v3 View in context: https://bugs.webkit.org/attachment.cgi?id=455386&action=review I like this approach. > Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj:12132 > + "$(SRCROOT)/../../Tools/Profiling/Empty.profdata", This file won't be available in production builds, which I think results in the behavior we want (failing the build). > Source/WebCore/WebCore.xcodeproj/project.pbxproj:38745 > + shellScript = "if [ \"${ACTION}\" = \"installhdrs\" -o \"${ACTION}\" = \"installapi\" ]; then\n exit 0;\nfi\n\nif [ -z \"${PROFILE_DATA_FLAGS}\" ]; then\n exit 0;\nfi\n\nif [ -f \"${SRCROOT}/WebCore.profdata.tar.gz\" ]; then\n echo \"Extracting ${SRCROOT}/WebCore.profdata.tar.gz\";\n tar -xzvf ${SRCROOT}/WebCore.profdata.tar.gz -C ${BUILT_PRODUCTS_DIR}/DerivedSources/WebCore;\n exit 0;\nfi\n\necho \"${SRCROOT}/WebCore.profdata.tar.gz does not exist - falling back to empty profiling data\";\ncp ${SRCROOT}/../../Tools/Profiling/Empty.profdata ${BUILT_PRODUCTS_DIR}/DerivedSources/WebCore/WebCore.profdata;\n"; I don't quite understand how git-lfs works, but won't ${SRCROOT}/WebCore.profdata.tar.gz actually exist as a stub? Longer term, we probably need multiple profdata files (there's no way iOS and macOS profiles are the same, for example).
Wenson Hsieh
Comment 20 2022-03-23 12:01:48 PDT
Comment on attachment 455386 [details] v3 Thank you for taking a look! After offline discussion with Geoff, Alexey, and (many) others, I think we've all agreed on a way forward here — to: • Upload the profiling data to WebKitAdditions using `git lfs`, and copy into the Apple internal SDK. • Make build scripts look for the profiling data in the build directory (or SDK, if it's not in the build directory). • If the profiling data can't be found for a (non-production) release build, then fall back to Empty.profdata in Tools. • If the profiling data can't be found for a production build, then exit with a nonzero error code (and a descriptive message). I'll upload a v4 of patch that does this, momentarily!
Wenson Hsieh
Comment 21 2022-03-23 12:34:49 PDT Comment hidden (obsolete)
Wenson Hsieh
Comment 22 2022-03-23 12:36:23 PDT
(In reply to Wenson Hsieh from comment #21) > Created attachment 455539 [details] > v4 (The .pbxproj format makes it kind of annoying to read the build phase script, so I put it here as well, for reviewers' convenience) ``` if [ "${ACTION}" = "installhdrs" -o "${ACTION}" = "installapi" ]; then exit 0; fi if [ -z "${PROFILE_DATA_FLAGS}" ]; then exit 0; fi RELATIVE_PROFILE_DATA_PATH="usr/local/include/WebKitAdditions/Profiling/JavaScriptCore.profdata.tar.gz"; ABSOLUTE_PROFILE_DATA_PATH="${BUILT_PRODUCTS_DIR}/${RELATIVE_PROFILE_DATA_PATH}"; if [ ! -f "${ABSOLUTE_PROFILE_DATA_PATH}" ]; then ABSOLUTE_PROFILE_DATA_PATH="${SDK_DIR}/${RELATIVE_PROFILE_DATA_PATH}"; fi if [ ! -f "${ABSOLUTE_PROFILE_DATA_PATH}" ] || [ $(wc -c <${ABSOLUTE_PROFILE_DATA_PATH}) -lt 4096 ]; then if [ "${CONFIGURATION}" = "Production" ]; then echo "Error: production build missing profiling data at ${ABSOLUTE_PROFILE_DATA_PATH}"; exit 1; fi echo "Missing or invalid profiling data at ${ABSOLUTE_PROFILE_DATA_PATH} - falling back to empty file"; cp ${SRCROOT}/../../Tools/Profiling/Empty.profdata ${BUILT_PRODUCTS_DIR}/DerivedSources/JavaScriptCore/JavaScriptCore.profdata; else echo "Copying profiling data at ${ABSOLUTE_PROFILE_DATA_PATH}"; tar -xzvf ${ABSOLUTE_PROFILE_DATA_PATH} -C ${BUILT_PRODUCTS_DIR}/DerivedSources/JavaScriptCore; fi ```
Alexey Proskuryakov
Comment 23 2022-03-23 13:04:34 PDT
> echo "Error: production build missing profiling data at ${ABSOLUTE_PROFILE_DATA_PATH}"; This error seems slightly misleading, should be saying that it's missing at both RELATIVE_PROFILE_DATA_PATH and ABSOLUTE_PROFILE_DATA_PATH. The plan looks good to me overall. Please build various configurations before landing (iOS, watchOS, macCatalyst release, debug and production come to mind). It's not great that names of the profdata files are so generic - there is no affordance to make them CPU architecture specific, or different between AppKit and macCatalyst.
Alexey Proskuryakov
Comment 24 2022-03-23 13:14:25 PDT
Comment on attachment 455539 [details] v4 View in context: https://bugs.webkit.org/attachment.cgi?id=455539&action=review > Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj:12131 > + "$(BUILT_PRODUCTS_DIR)/usr/local/include/WebKitAdditions/Profiling/JavaScriptCore.profdata.tar.gz", This is only one of the two paths the script is looking at. I think that we need both as inputs (same in other projects). > Tools/ChangeLog:9 > + Add an empty profiling data file that can be used as a fallback for production and release builds, when the The ChangeLog is stale, as we discussed. Empty.profdata can't be used as a fallback for production builds.
Wenson Hsieh
Comment 25 2022-03-23 13:33:46 PDT
(In reply to Alexey Proskuryakov from comment #23) > > echo "Error: production build missing profiling data at ${ABSOLUTE_PROFILE_DATA_PATH}"; > > This error seems slightly misleading, should be saying that it's missing at > both RELATIVE_PROFILE_DATA_PATH and ABSOLUTE_PROFILE_DATA_PATH. Sounds good to me! Will tweak the error message to mention that the profiling data is missing at both "${BUILT_PRODUCTS_DIR}/${RELATIVE_PROFILE_DATA_PATH}" and the current value of "${ABSOLUTE_PROFILE_DATA_PATH}". > > The plan looks good to me overall. Please build various configurations > before landing (iOS, watchOS, macCatalyst release, debug and production come > to mind). Will do! (..and will also coordinate with ops folks to make sure this change doesn't break any of our test and perf automation). > > It's not great that names of the profdata files are so generic - there is no > affordance to make them CPU architecture specific, or different between > AppKit and macCatalyst. While generic, our experiments have shown that this profiling data significantly improves performance across all architectures and platforms where this patch enables PGO. I think in the future, we should do further experiments to see if we can get more wins out of crafting PGO files specific to architecture and platform (at which point we could then add the affordances necessary to maintain and deploy the relevant PGO file, given something like the target triple, or perhaps just architecture + platform).
Wenson Hsieh
Comment 26 2022-03-23 14:15:12 PDT
(In reply to Alexey Proskuryakov from comment #24) > Comment on attachment 455539 [details] > v4 > > View in context: > https://bugs.webkit.org/attachment.cgi?id=455539&action=review > > > Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj:12131 > > + "$(BUILT_PRODUCTS_DIR)/usr/local/include/WebKitAdditions/Profiling/JavaScriptCore.profdata.tar.gz", > > This is only one of the two paths the script is looking at. I think that we > need both as inputs (same in other projects). Good catch — will add the SDK WKA paths here as well. > > > Tools/ChangeLog:9 > > + Add an empty profiling data file that can be used as a fallback for production and release builds, when the > > The ChangeLog is stale, as we discussed. Empty.profdata can't be used as a > fallback for production builds. Whoops! Will update the ChangeLog to reflect the actual behavior.
Wenson Hsieh
Comment 27 2022-03-23 14:42:01 PDT
Created attachment 455559 [details] v4 (address feedback)
Alexey Proskuryakov
Comment 28 2022-03-23 16:46:38 PDT
> maintain and deploy the relevant PGO file, given something like the target triple, or perhaps just architecture + platform). We don't need the platform here - WebKitAdditions can and should decide which platform to copy profiles for, because it's building for the same platform as WebKit. We only need to differentiate by file name when there are multiple in the same build (so, architecture and macCatalyst variant are the only ones that come to mind).
Wenson Hsieh
Comment 29 2022-03-23 16:48:46 PDT
(In reply to Alexey Proskuryakov from comment #28) > > maintain and deploy the relevant PGO file, given something like the target triple, or perhaps just architecture + platform). > > We don't need the platform here - WebKitAdditions can and should decide > which platform to copy profiles for, because it's building for the same > platform as WebKit. We only need to differentiate by file name when there > are multiple in the same build (so, architecture and macCatalyst variant are > the only ones that come to mind). I see — yes, that makes sense! (and also lets us avoid making the build script more complicated)
Alexey Proskuryakov
Comment 30 2022-03-23 16:49:29 PDT
Comment on attachment 455559 [details] v4 (address feedback) Looks good to me. Marking cq- for now as a reminder to wait for bot updates, and to perform all of the builds for testing. As this is new ground, others are more than welcome to chime in.
Wenson Hsieh
Comment 31 2022-03-23 16:51:11 PDT
(In reply to Alexey Proskuryakov from comment #30) > Comment on attachment 455559 [details] > v4 (address feedback) > > Looks good to me. Marking cq- for now as a reminder to wait for bot updates, > and to perform all of the builds for testing. > > As this is new ground, others are more than welcome to chime in. Sounds good — thank you for the review! (Adding Ryan and bot watchers as well, for visibility)
Wenson Hsieh
Comment 32 2022-03-24 13:44:18 PDT
Created attachment 455674 [details] Adjust decompression
Wenson Hsieh
Comment 33 2022-03-24 13:45:28 PDT
Comment on attachment 455674 [details] Adjust decompression (Oops, I forgot to update the input file list for the build scripts — need to fix that)
Wenson Hsieh
Comment 34 2022-03-24 13:49:34 PDT Comment hidden (obsolete)
Wenson Hsieh
Comment 35 2022-03-24 14:47:18 PDT Comment hidden (obsolete)
Wenson Hsieh
Comment 36 2022-03-24 15:02:10 PDT
Created attachment 455693 [details] Rebase (again)
Wenson Hsieh
Comment 37 2022-03-24 16:23:13 PDT
Created attachment 455702 [details] Fix production iOS builds
Alexey Proskuryakov
Comment 38 2022-03-24 16:31:46 PDT
Comment on attachment 455702 [details] Fix production iOS builds View in context: https://bugs.webkit.org/attachment.cgi?id=455702&action=review > Source/JavaScriptCore/Configurations/JavaScriptCore.xcconfig:55 > +OTHER_CPLUSPLUSFLAGS = $(inherited) -fno-slp-vectorize --system-header-prefix=unicode/ $(PROFILE_DATA_FLAGS); Just noticed this - why is it sufficient to only add the flag to C++ files?
Wenson Hsieh
Comment 39 2022-03-24 17:03:52 PDT
(In reply to Alexey Proskuryakov from comment #38) > Comment on attachment 455702 [details] > Fix production iOS builds > > View in context: > https://bugs.webkit.org/attachment.cgi?id=455702&action=review > > > Source/JavaScriptCore/Configurations/JavaScriptCore.xcconfig:55 > > +OTHER_CPLUSPLUSFLAGS = $(inherited) -fno-slp-vectorize --system-header-prefix=unicode/ $(PROFILE_DATA_FLAGS); > > Just noticed this - why is it sufficient to only add the flag to C++ files? This was not an intentional omission (though, given that the vast majority of our source files are C++ anyways, I wouldn't expect it to make much difference). I'll add it to `OTHER_CFLAGS` as well, with: ``` OTHER_CFLAGS = $(inherited) … $(PROFILE_DATA_FLAGS); ```
Wenson Hsieh
Comment 40 2022-03-24 17:32:54 PDT
(In reply to Wenson Hsieh from comment #39) > (In reply to Alexey Proskuryakov from comment #38) > > Comment on attachment 455702 [details] > > Fix production iOS builds > > > > View in context: > > https://bugs.webkit.org/attachment.cgi?id=455702&action=review > > > > > Source/JavaScriptCore/Configurations/JavaScriptCore.xcconfig:55 > > > +OTHER_CPLUSPLUSFLAGS = $(inherited) -fno-slp-vectorize --system-header-prefix=unicode/ $(PROFILE_DATA_FLAGS); > > > > Just noticed this - why is it sufficient to only add the flag to C++ files? > > This was not an intentional omission (though, given that the vast majority > of our source files are C++ anyways, I wouldn't expect it to make much > difference). I'll add it to `OTHER_CFLAGS` as well, with: > > ``` > OTHER_CFLAGS = $(inherited) … $(PROFILE_DATA_FLAGS); > ``` Oddly, enabling this in WebKit2 leads to: ``` In file included from /Users/whsieh/Build/Release/DerivedSources/WebKit/unified-sources/UnifiedSource13-mm.mm:1: /Users/whsieh/work/OpenSource/Source/WebKit/Shared/Cocoa/SandboxExtensionCocoa.mm:64:9: error: use of undeclared identifier 'memset_s' memset_s(m_token, length, 0, length); 1 error generated. ``` It even happens if I simply add the following, without the -fprofile-* flags, so I think it's unrelated to profiling (and instead related to how the OTHER_CFLAGS build setting in WebKit2 is set...) ``` OTHER_CFLAGS = $(inherited); ``` Going to leave it out of there for now, and see what the bots think.
Wenson Hsieh
Comment 41 2022-03-24 18:30:05 PDT
(In reply to Wenson Hsieh from comment #40) > > Going to leave it out of there for now, and see what the bots think. Actually, I think I figured it out. So in BaseTarget.xcconfig, OTHER_CPLUSPLUSFLAGS includes OTHER_CFLAGS already: ``` OTHER_CPLUSPLUSFLAGS = $(OTHER_CFLAGS) …; ``` ...which seems to cause problems if WebKit.xcconfig then goes along and tries to change both OTHER_CFLAGS and OTHER_CPLUSPLUSFLAGS. To fix this, I just moved the part that adds $(PROFILE_DATA_FLAGS) to OTHER_CFLAGS to Source/WebKit/Configurations/BaseTarget.xcconfig instead, and then OTHER_CPLUSPLUSFLAGS just gets it by virtue of including all of $(OTHER_CFLAGS).
Wenson Hsieh
Comment 42 2022-03-24 18:34:43 PDT
Created attachment 455717 [details] Add flag to OTHER_CFLAGS too
Alexey Proskuryakov
Comment 43 2022-03-25 09:19:14 PDT
> moved the part that adds $(PROFILE_DATA_FLAGS) to OTHER_CFLAGS to Source/WebKit/Configurations/BaseTarget.xcconfig This means that the profile will be in use when building all of WebKit project targets, not just the framework. Which is probably harmless, but does not feel quite right. It is a little mysterious to me why we need to add System.framework via -isystem. For some context, OTHER_CFLAGS is the default value for OTHER_CPLUSPLUSFLAGS, so if we don't define OTHER_CPLUSPLUSFLAGS as all, it matches. But once we start defining it, we need everything, they are not merged together.
Wenson Hsieh
Comment 44 2022-03-25 09:51:13 PDT
(In reply to Alexey Proskuryakov from comment #43) > > moved the part that adds $(PROFILE_DATA_FLAGS) to OTHER_CFLAGS to Source/WebKit/Configurations/BaseTarget.xcconfig > > This means that the profile will be in use when building all of WebKit > project targets, not just the framework. Which is probably harmless, but > does not feel quite right. I agree it is a bit odd. Is it fine here, or should I move this back to WebKit.xcconfig and just make it apply to `OTHER_CPLUSPLUSFLAGS`? > > It is a little mysterious to me why we need to add System.framework via > -isystem. > > For some context, OTHER_CFLAGS is the default value for > OTHER_CPLUSPLUSFLAGS, so if we don't define OTHER_CPLUSPLUSFLAGS as all, it > matches. But once we start defining it, we need everything, they are not > merged together. So it looks like the -isystem was added in r213492 to fix the build after r213483. It does sound like we should be able to omit this line altogether if we add the profiling flags to OTHER_CFLAGS, since OTHER_CPLUSPLUSFLAGS will simply get both the profiling flags and the System.framework header includes from OTHER_CFLAGS. I was able to still build internal iOS and macOS locally after removing this line — let's see what EWS says.
Wenson Hsieh
Comment 45 2022-03-25 10:32:10 PDT
Alexey Proskuryakov
Comment 46 2022-03-25 12:41:10 PDT
> I agree it is a bit odd. Is it fine here, or should I move this back to WebKit.xcconfig and just make it apply to `OTHER_CPLUSPLUSFLAGS`? It would certainly be nicer and consistent with other projects if moved to WebKit.xcconfig. But I don't want to delay landing this.
Wenson Hsieh
Comment 47 2022-03-25 13:05:11 PDT
(In reply to Alexey Proskuryakov from comment #46) > > I agree it is a bit odd. Is it fine here, or should I move this back to WebKit.xcconfig and just make it apply to `OTHER_CPLUSPLUSFLAGS`? > > It would certainly be nicer and consistent with other projects if moved to > WebKit.xcconfig. But I don't want to delay landing this. Sounds good — I've filed <https://webkit.org/b/238390> to track this. Thanks!
Wenson Hsieh
Comment 48 2022-03-25 14:37:16 PDT
Wenson Hsieh
Comment 49 2022-03-25 14:38:17 PDT
Note You need to log in before you can comment on or make changes to this bug.