Bug 226070 - [MSVC] The "/Zc:lambda" switch is recommended to use the new lambda processor.
Summary: [MSVC] The "/Zc:lambda" switch is recommended to use the new lambda processor.
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: JavaScriptCore (show other bugs)
Version: WebKit Nightly Build
Hardware: PC Windows 10
: P2 Normal
Assignee: Fujii Hironori
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2021-05-21 00:04 PDT by QuellaZhang
Modified: 2021-09-28 22:51 PDT (History)
19 users (show)

See Also:


Attachments
Patch (1.22 KB, patch)
2021-05-25 14:30 PDT, Fujii Hironori
no flags Details | Formatted Diff | Diff
Patch (1.48 KB, patch)
2021-09-23 13:24 PDT, Fujii Hironori
no flags Details | Formatted Diff | Diff
Wrong Patch (5.49 KB, patch)
2021-09-28 22:49 PDT, Fujii Hironori
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description QuellaZhang 2021-05-21 00:04:10 PDT
Environment:
VS 2019 + Windows Server 2016

Issue description:
The MSVC team uses Webkit project as a test case to detect compiler regressions. Recently, two Webkit ports WinCairo and AppleWin triggered an old lambda processor issue on MSVC, which is an irreconcilable old lambda processor issue. Therefore, it is recommended to turn on "/Zc:lambda" switch to use new lambda processor to avoid this problem.

Some opinions from MSVC developers:
Note: the Webkit failure is an issue with the old lambda processor in which one or more lambdas (each of which contains one or more nested lambdas) are generated from a macro expansion. Trying to tell the lambdas apart is an exercise in futility (in the old, token based lambda processor the ‘identity’ for a lambda is the SFA of the initial ‘[‘) and code like this was one of the motivating reasons for starting on the lambda rejuvenation effort.

Repro Steps:
1. git clone https://github.com/WebKit/webkit F:\gitP\WebKit\webkit
2. open a VS 2019 x64 command prompt as admin and browse to F:\gitP\WebKit\webkit
3. download latest WebKitAuxiliaryLibrary.zip from https://developer.apple.com/opensource/internet/WebKitAuxiliaryLibrary.zip and unzip to F:\gitP\WebKit\webkit\WebKitLibraries\win
4. download latest WebKitSupportLibrary.zip from https://developer.apple.com/opensource/internet/webkit_sptlib_agree.html and unzip to F:\gitP\WebKit\webkit\WebKitLibraries\win
5. mkdir F:\gitP\WebKit\webkit\build_amd64_AppleWin
6. cd F:\gitP\WebKit\webkit\build_amd64_AppleWin
7. cmake -G "Visual Studio 16 2019" -A x64 -DCMAKE_SYSTEM_VERSION=10.0.18362.0 -DCMAKE_BUILD_TYPE=Release -DRUBY_LIBRARY=C:\tools\ruby26\lib -DPORT="AppleWin" ..
8. msbuild /p:Platform=x64 /p:Configuration=Release WebKit.sln /t:Rebuild

Error Info:
F:\gitP\WebKit\webkit\Source\JavaScriptCore\heap\Heap.cpp(2827): Assertion failed: iter->second.EncodedIndex() != encodedIndex, file d:\agent\_work\4\s\src\vctools\Compiler\CxxFE\sl\p1\c\lambdas.cpp, line 2464 [when compiling F:\gitP\WebKit\webkit\build_amd64_AppleWin\DerivedSources\JavaScriptCore\unified-sources\UnifiedSource-ee8a7a7a-3.cpp]
    c1xx!assertfe()+0x106
        d:\agent\_work\4\s\src\vctools\Compiler\CxxFE\sl\p1\c\debug.c Line 333
    c1xx!LambdaProcessor::Process()+0x8c3
        d:\agent\_work\4\s\src\vctools\Compiler\CxxFE\sl\p1\c\lambdas.cpp Line 2462
    c1xx!PrimaryParser::Parse()+0xd9ae
        d:\agent\_work\4\s\src\vctools\Compiler\CxxFE\sl\p1\cxx\grammar.y Line 11319
    c1xx!`CallPrimaryParser'::`2'::<lambda_1>::operator()()+0x19
        d:\agent\_work\4\s\src\vctools\Compiler\CxxFE\sl\p1\cxx\grammar.y Line 13808
    c1xx!CallPrimaryParser()+0x14e
        d:\agent\_work\4\s\src\vctools\Compiler\CxxFE\sl\p1\cxx\grammar.y Line 13829
    c1xx!main_compile()+0xfa2
        d:\agent\_work\4\s\src\vctools\Compiler\CxxFE\sl\p1\c\main.c Line 3094
Comment 1 Fujii Hironori 2021-05-21 00:30:15 PDT
/Zc:lambda seems an undocumented switch. Which MSVC version started to support it?
IIRC, AppleWin is still using Visual Studio 2019 version 16.3
Comment 2 Fujii Hironori 2021-05-23 13:11:02 PDT
See also: Bug 209311 – [WinCairo][MSVC] NetworkProcess.cpp(1896): fatal error C1001: Internal compiler error with Visual Studio 16.5.0
Comment 3 QuellaZhang 2021-05-24 01:38:55 PDT
Thanks for providing more information, Fujii.

>>/Zc:lambda seems an undocumented switch. Which MSVC version started to support it?
This does not seem to have official documents at present. I can find that the earliest VS version that contains the modification(Some people apparently don't like the continued use of 'experimental', so the team removed "/experimental:newLambdaProcessor" and added "/Zc:lambda" switch) is VS 16.4 Preview2, and the official version may be VS 16.5 and later.

>>See also: Bug 209311 – [WinCairo][MSVC] NetworkProcess.cpp(1896): fatal error C1001: Internal compiler error with Visual Studio 16.5.0
Check the datails of dup bug https://bugs.webkit.org/show_bug.cgi?id=209358, the internal error occurred on VS 16.5.0, which should be caused by the above change. So now it may be necessary to add the new lambda processor back.


>>IIRC, AppleWin is still using Visual Studio 2019 version 16.3
On my side, AppleWin and WinCairo are currently compiled successfully with "/Zc:lambda". I tested it on VS 16.9.1 and the latest version.
Comment 4 Fujii Hironori 2021-05-25 14:30:27 PDT
Created attachment 429692 [details]
Patch
Comment 5 Don Olmstead 2021-05-25 15:54:52 PDT
Isn't VS 2022 releasing soon? Is the new lambda processor on by default for it?
Comment 6 Fujii Hironori 2021-05-25 18:03:44 PDT
WinCairo EWS is reporting an compilation error for the patch.
https://ews-build.webkit.org/#/builders/12/builds/53128

FAILED: Source/JavaScriptCore/CMakeFiles/JavaScriptCore.dir/__/__/JavaScriptCore/DerivedSources/unified-sources/UnifiedSource-ee8a7a7a-3.cpp.obj 
C:\MSVS\VC\Tools\MSVC\14.28.29333\bin\Hostx64\x64\cl.exe  /nologo /TP -DBUILDING_JavaScriptCore -DBUILDING_WITH_CMAKE=1 -DHAVE_CONFIG_H=1 -DJavaScriptCore_EXPORTS -DNOCRYPT -DNOMINMAX -DUNICODE -DWINVER=0x601 -DWTF_PLATFORM_WIN_CAIRO=1 -D_CRT_SECURE_CPP_OVERLOAD_STANDARD_NAMES=1 -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_UNICODE -D_WIN32_WINNT=0x601 -D_WINDOWS -D_WINSOCKAPI_="" -IJavaScriptCore\Headers -I. -I..\..\Source\JavaScriptCore -I..\..\Source\JavaScriptCore\API -I..\..\Source\JavaScriptCore\assembler -I..\..\Source\JavaScriptCore\b3 -I..\..\Source\JavaScriptCore\b3\air -I..\..\Source\JavaScriptCore\bindings -I..\..\Source\JavaScriptCore\builtins -I..\..\Source\JavaScriptCore\bytecode -I..\..\Source\JavaScriptCore\bytecompiler -I..\..\Source\JavaScriptCore\dfg -I..\..\Source\JavaScriptCore\disassembler -I..\..\Source\JavaScriptCore\disassembler\ARM64 -I..\..\Source\JavaScriptCore\disassembler\udis86 -I..\..\Source\JavaScriptCore\domjit -I..\..\Source\JavaScriptCore\ftl -I..\..\Source\JavaScriptCore\heap -I..\..\Source\JavaScriptCore\debugger -I..\..\Source\JavaScriptCore\inspector -I..\..\Source\JavaScriptCore\inspector\agents -I..\..\Source\JavaScriptCore\inspector\augmentable -I..\..\Source\JavaScriptCore\inspector\remote -I..\..\Source\JavaScriptCore\interpreter -I..\..\Source\JavaScriptCore\jit -I..\..\Source\JavaScriptCore\llint -I..\..\Source\JavaScriptCore\parser -I..\..\Source\JavaScriptCore\profiler -I..\..\Source\JavaScriptCore\runtime -I..\..\Source\JavaScriptCore\tools -I..\..\Source\JavaScriptCore\wasm -I..\..\Source\JavaScriptCore\wasm\js -I..\..\Source\JavaScriptCore\yarr -IJavaScriptCore\DerivedSources -IJavaScriptCore\DerivedSources\inspector -IJavaScriptCore\DerivedSources\runtime -IJavaScriptCore\DerivedSources\yarr -I..\include\private -I..\..\Source\JavaScriptCore\inspector\remote\socket -IWTF\Headers -I..\..\WebKitLibraries\win\include /W4 /DWIN32 /D_WINDOWS  /GR- /EHs- /EHc-  /MD /O2 /Ob2 /DNDEBUG /wd4018 /wd4060 /wd4068 /wd4100 /wd4127 /wd4146 /wd4189 /wd4201 /wd4244 /wd4245 /wd4251 /wd4275 /wd4267 /wd4305 /wd4309 /wd4312 /wd4324 /wd4389 /wd4456 /wd4457 /wd4458 /wd4459 /wd4505 /wd4611 /wd4646 /wd4701 /wd4702 /wd4706 /wd4715 /wd4722 /wd4838 /wd4840 /wd4996 /wd5205 /Zi /GS /EHa- /EHc- /EHs- /fp:except- /analyze- /bigobj /utf-8 /validate-charset /Oy- /Zc:lambda -std:c++17 /showIncludes /FoSource\JavaScriptCore\CMakeFiles\JavaScriptCore.dir\__\__\JavaScriptCore\DerivedSources\unified-sources\UnifiedSource-ee8a7a7a-3.cpp.obj /FdSource\JavaScriptCore\CMakeFiles\JavaScriptCore.dir\ /FS -c JavaScriptCore\DerivedSources\unified-sources\UnifiedSource-ee8a7a7a-3.cpp
C:\Buildbot\WinCairo-EWS\build\Source\JavaScriptCore\heap/Heap.cpp(2874): error C2065: 'Visitor': undeclared identifier
C:\Buildbot\WinCairo-EWS\build\Source\JavaScriptCore\heap/Heap.cpp(2901): note: see reference to function template instantiation 'auto JSC::Heap::addCoreConstraints::<lambda_12>::()::<lambda_2>::operator ()<JSC::IsoCellSet>(_T1 &) const' being compiled
        with
        [
            _T1=JSC::IsoCellSet
        ]
C:\Buildbot\WinCairo-EWS\build\WebKitBuild\Release\WTF\Headers\wtf/Function.h(52): note: see reference to function template instantiation 'auto JSC::Heap::addCoreConstraints::<lambda_12>::operator ()<JSC::SlotVisitor>(_T1 &) const' being compiled
        with
        [
            _T1=JSC::SlotVisitor
        ]
C:\Buildbot\WinCairo-EWS\build\WebKitBuild\Release\WTF\Headers\wtf/Function.h(52): note: while compiling class template member function 'Out WTF::Detail::CallableWrapper<CallableType,Out,JSC::SlotVisitor &>::call(JSC::SlotVisitor &)'
        with
        [
            Out=void,
            CallableType=JSC::Heap::addCoreConstraints::<lambda_12>
        ]
C:\Buildbot\WinCairo-EWS\build\WebKitBuild\Release\WTF\Headers\wtf/StdLibExtras.h(505): note: see reference to class template instantiation 'WTF::Detail::CallableWrapper<CallableType,Out,JSC::SlotVisitor &>' being compiled
        with
        [
            CallableType=JSC::Heap::addCoreConstraints::<lambda_12>,
            Out=void
        ]
C:\Buildbot\WinCairo-EWS\build\WebKitBuild\Release\WTF\Headers\wtf/Function.h(74): note: see reference to function template instantiation 'std::unique_ptr<T,std::default_delete<T>> WTF::makeUnique<WTF::Detail::CallableWrapper<CallableType,Out,JSC::SlotVisitor &>,_Ty>(_Ty &&)' being compiled
        with
        [
            T=WTF::Detail::CallableWrapper<JSC::Heap::addCoreConstraints::<lambda_12>,void,JSC::SlotVisitor &>,
            CallableType=JSC::Heap::addCoreConstraints::<lambda_12>,
            Out=void,
            _Ty=JSC::Heap::addCoreConstraints::<lambda_12>
        ]
C:\Buildbot\WinCairo-EWS\build\Source\JavaScriptCore\heap/Heap.cpp(2901): note: see reference to function template instantiation 'WTF::Function<void (JSC::SlotVisitor &)>::Function<JSC::Heap::addCoreConstraints::<lambda_12>,void>(CallableType &&)' being compiled
        with
        [
            CallableType=JSC::Heap::addCoreConstraints::<lambda_12>
        ]
C:\Buildbot\WinCairo-EWS\build\Source\JavaScriptCore\heap/Heap.cpp(2872): note: see reference to function template instantiation 'WTF::Function<void (JSC::SlotVisitor &)>::Function<JSC::Heap::addCoreConstraints::<lambda_12>,void>(CallableType &&)' being compiled
        with
        [
            CallableType=JSC::Heap::addCoreConstraints::<lambda_12>
        ]
C:\Buildbot\WinCairo-EWS\build\Source\JavaScriptCore\heap/Heap.cpp(2874): error C2672: 'JSC::IsoCellSet::forEachMarkedCellInParallel': no matching overloaded function found
C:\Buildbot\WinCairo-EWS\build\Source\JavaScriptCore\heap/Heap.cpp(2901): error C2974: 'JSC::IsoCellSet::forEachMarkedCellInParallel': invalid template argument for 'Visitor', type expected
C:\Buildbot\WinCairo-EWS\build\Source\JavaScriptCore\heap\IsoCellSet.h(62): note: see declaration of 'JSC::IsoCellSet::forEachMarkedCellInParallel'
C:\Buildbot\WinCairo-EWS\build\Source\JavaScriptCore\heap/Heap.cpp(2874): error C2672: 'JSC::Subspace::forEachMarkedCellInParallel': no matching overloaded function found
C:\Buildbot\WinCairo-EWS\build\Source\JavaScriptCore\heap/Heap.cpp(2901): error C2974: 'JSC::Subspace::forEachMarkedCellInParallel': invalid template argument for 'Visitor', type expected
C:\Buildbot\WinCairo-EWS\build\Source\JavaScriptCore\heap\Subspace.h(90): note: see declaration of 'JSC::Subspace::forEachMarkedCellInParallel'
Comment 7 Fujii Hironori 2021-05-25 18:39:40 PDT
I don't see the error on my PC.
Is this caused by specific MSVC version.
I'm using MSVC 14.29 (Visual Studio 2019 Version 16.9).
WinCairo EWS seems using MSVC 14.28 (Visual Studio 2019 Version 16.8).
Comment 8 QuellaZhang 2021-05-26 02:08:39 PDT
Hmm, for error C2065,I found relevant information, it is indeed a bug of MSVC, currently fixed in 16.9. You may need to upgrade VS.

https://developercommunity.visualstudio.com/t/captured-variable-not-recognized-when-calling-temp/1137293
https://developercommunity.visualstudio.com/t/nested-c20-template-lambdas-two-issues-with-captur/1150322

I don’t know much about VS2022, there is a related blog https://devblogs.microsoft.com/visualstudio/visual-studio-2022/.
Comment 9 Don Olmstead 2021-05-26 09:09:36 PDT
(In reply to QuellaZhang from comment #8)
> Hmm, for error C2065,I found relevant information, it is indeed a bug of
> MSVC, currently fixed in 16.9. You may need to upgrade VS.
> 
> https://developercommunity.visualstudio.com/t/captured-variable-not-
> recognized-when-calling-temp/1137293
> https://developercommunity.visualstudio.com/t/nested-c20-template-lambdas-
> two-issues-with-captur/1150322
> 
> I don’t know much about VS2022, there is a related blog
> https://devblogs.microsoft.com/visualstudio/visual-studio-2022/.

Fujii's patch adds a compiler flag based on versioning I'm just curious if that flag will still be valid with 2022. If not then there'd need to be an upper bound.
Comment 10 QuellaZhang 2021-05-27 18:46:35 PDT
Hey Don,

Confirmed with the developer: 

>>Isn't VS 2022 releasing soon? Is the new lambda processor on by default for it?

Not: at the moment: it is the default if you compile with /std:c++latest or /permissive- -- it may become the default in VS2022 but we haven't made that decision yet.
Comment 11 Radar WebKit Bug Importer 2021-05-28 00:05:20 PDT
<rdar://problem/78607135>
Comment 12 Don Olmstead 2021-05-28 11:37:40 PDT
(In reply to QuellaZhang from comment #10)
> Hey Don,
> 
> Confirmed with the developer: 
> 
> >>Isn't VS 2022 releasing soon? Is the new lambda processor on by default for it?
> 
> Not: at the moment: it is the default if you compile with /std:c++latest or
> /permissive- -- it may become the default in VS2022 but we haven't made that
> decision yet.

Thanks for letting us know!

Fujii I think its probably better to start compiling with /permissive- rather than this. Thoughts?
Comment 13 Fujii Hironori 2021-05-30 13:09:07 PDT
(In reply to Don Olmstead from comment #12)
> (In reply to QuellaZhang from comment #10)
> > Hey Don,
> > 
> > Confirmed with the developer: 
> > 
> > >>Isn't VS 2022 releasing soon? Is the new lambda processor on by default for it?
> > 
> > Not: at the moment: it is the default if you compile with /std:c++latest or
> > /permissive- -- it may become the default in VS2022 but we haven't made that
> > decision yet.
> 
> Thanks for letting us know!
> 
> Fujii I think its probably better to start compiling with /permissive-
> rather than this. Thoughts?

> the first public preview of Visual Studio 2022 will be released this summer.
https://devblogs.microsoft.com/visualstudio/visual-studio-2022/

QuellaZhang, when do you want to solve the original problem (comment#0)?
I think waiting for the official VS2022 release doesn't meat the QuellaZhang's schedule.
Comment 14 Fujii Hironori 2021-05-30 13:12:32 PDT
meet, not 🐷.
Comment 15 QuellaZhang 2021-05-31 03:43:17 PDT
Hi Fujii,

We added "/Zc:lambda" to command line as workaround to unblock us. Please follow your schedule to fix this issue. :)
Comment 16 Fujii Hironori 2021-09-22 14:57:12 PDT
Bug 230599 can be solved by the /Zc:lambda switch.
Comment 17 Fujii Hironori 2021-09-23 13:24:06 PDT
Created attachment 439085 [details]
Patch
Comment 18 Ross Kirsling 2021-09-23 14:26:19 PDT
Comment on attachment 439085 [details]
Patch

Seems fine to me.
Comment 19 Fujii Hironori 2021-09-23 14:35:16 PDT
Comment on attachment 439085 [details]
Patch

Clearing flags on attachment: 439085

Committed r283007 (242072@main): <https://commits.webkit.org/242072@main>
Comment 20 Fujii Hironori 2021-09-23 14:35:22 PDT
All reviewed patches have been landed.  Closing bug.
Comment 21 Fujii Hironori 2021-09-28 22:49:25 PDT
Reopening to attach new patch.
Comment 22 Fujii Hironori 2021-09-28 22:49:34 PDT
Created attachment 439568 [details]
Wrong Patch
Comment 23 EWS Watchlist 2021-09-28 22:50:30 PDT
This patch modifies the JS builtins code generator. Please ensure that you have rebaselined any generator test results (i.e., by running `Tools/Scripts/run-builtins-generator-tests --reset-results`)