Bug 226070 – [MSVC] The "/Zc:lambda" switch is recommended to use the new lambda processor.

• Component: JavaScriptCore
• Status: RESOLVED FIXED

[QuellaZhang]

Environment:
VS 2019 + Windows Server 2016

Issue description:
The MSVC team uses Webkit project as a test case to detect compiler regressions. Recently, two Webkit ports WinCairo and AppleWin triggered an old lambda processor issue on MSVC, which is an irreconcilable old lambda processor issue. Therefore, it is recommended to turn on "/Zc:lambda" switch to use new lambda processor to avoid this problem.

Some opinions from MSVC developers:
Note: the Webkit failure is an issue with the old lambda processor in which one or more lambdas (each of which contains one or more nested lambdas) are generated from a macro expansion. Trying to tell the lambdas apart is an exercise in futility (in the old, token based lambda processor the ‘identity’ for a lambda is the SFA of the initial ‘[‘) and code like this was one of the motivating reasons for starting on the lambda rejuvenation effort.

Repro Steps:
1. git clone https://github.com/WebKit/webkit F:\gitP\WebKit\webkit
2. open a VS 2019 x64 command prompt as admin and browse to F:\gitP\WebKit\webkit
3. download latest WebKitAuxiliaryLibrary.zip from https://developer.apple.com/opensource/internet/WebKitAuxiliaryLibrary.zip and unzip to F:\gitP\WebKit\webkit\WebKitLibraries\win
4. download latest WebKitSupportLibrary.zip from https://developer.apple.com/opensource/internet/webkit_sptlib_agree.html and unzip to F:\gitP\WebKit\webkit\WebKitLibraries\win
5. mkdir F:\gitP\WebKit\webkit\build_amd64_AppleWin
6. cd F:\gitP\WebKit\webkit\build_amd64_AppleWin
7. cmake -G "Visual Studio 16 2019" -A x64 -DCMAKE_SYSTEM_VERSION=10.0.18362.0 -DCMAKE_BUILD_TYPE=Release -DRUBY_LIBRARY=C:\tools\ruby26\lib -DPORT="AppleWin" ..
8. msbuild /p:Platform=x64 /p:Configuration=Release WebKit.sln /t:Rebuild

Error Info:
F:\gitP\WebKit\webkit\Source\JavaScriptCore\heap\Heap.cpp(2827): Assertion failed: iter->second.EncodedIndex() != encodedIndex, file d:\agent\_work\4\s\src\vctools\Compiler\CxxFE\sl\p1\c\lambdas.cpp, line 2464 [when compiling F:\gitP\WebKit\webkit\build_amd64_AppleWin\DerivedSources\JavaScriptCore\unified-sources\UnifiedSource-ee8a7a7a-3.cpp]
    c1xx!assertfe()+0x106
        d:\agent\_work\4\s\src\vctools\Compiler\CxxFE\sl\p1\c\debug.c Line 333
    c1xx!LambdaProcessor::Process()+0x8c3
        d:\agent\_work\4\s\src\vctools\Compiler\CxxFE\sl\p1\c\lambdas.cpp Line 2462
    c1xx!PrimaryParser::Parse()+0xd9ae
        d:\agent\_work\4\s\src\vctools\Compiler\CxxFE\sl\p1\cxx\grammar.y Line 11319
    c1xx!`CallPrimaryParser'::`2'::<lambda_1>::operator()()+0x19
        d:\agent\_work\4\s\src\vctools\Compiler\CxxFE\sl\p1\cxx\grammar.y Line 13808
    c1xx!CallPrimaryParser()+0x14e
        d:\agent\_work\4\s\src\vctools\Compiler\CxxFE\sl\p1\cxx\grammar.y Line 13829
    c1xx!main_compile()+0xfa2
        d:\agent\_work\4\s\src\vctools\Compiler\CxxFE\sl\p1\c\main.c Line 3094

[Fujii Hironori]

/Zc:lambda seems an undocumented switch. Which MSVC version started to support it?
IIRC, AppleWin is still using Visual Studio 2019 version 16.3

[Fujii Hironori]

See also: Bug 209311 – [WinCairo][MSVC] NetworkProcess.cpp(1896): fatal error C1001: Internal compiler error with Visual Studio 16.5.0

[QuellaZhang]

Thanks for providing more information, Fujii.

>>/Zc:lambda seems an undocumented switch. Which MSVC version started to support it?
This does not seem to have official documents at present. I can find that the earliest VS version that contains the modification(Some people apparently don't like the continued use of 'experimental', so the team removed "/experimental:newLambdaProcessor" and added "/Zc:lambda" switch) is VS 16.4 Preview2, and the official version may be VS 16.5 and later.

>>See also: Bug 209311 – [WinCairo][MSVC] NetworkProcess.cpp(1896): fatal error C1001: Internal compiler error with Visual Studio 16.5.0
Check the datails of dup bug https://bugs.webkit.org/show_bug.cgi?id=209358, the internal error occurred on VS 16.5.0, which should be caused by the above change. So now it may be necessary to add the new lambda processor back.


>>IIRC, AppleWin is still using Visual Studio 2019 version 16.3
On my side, AppleWin and WinCairo are currently compiled successfully with "/Zc:lambda". I tested it on VS 16.9.1 and the latest version.

[Fujii Hironori]

Created attachment 429692 [details]
Patch

[Don Olmstead]

Isn't VS 2022 releasing soon? Is the new lambda processor on by default for it?

[Fujii Hironori]

WinCairo EWS is reporting an compilation error for the patch.
https://ews-build.webkit.org/#/builders/12/builds/53128

FAILED: Source/JavaScriptCore/CMakeFiles/JavaScriptCore.dir/__/__/JavaScriptCore/DerivedSources/unified-sources/UnifiedSource-ee8a7a7a-3.cpp.obj 
C:\MSVS\VC\Tools\MSVC\14.28.29333\bin\Hostx64\x64\cl.exe  /nologo /TP -DBUILDING_JavaScriptCore -DBUILDING_WITH_CMAKE=1 -DHAVE_CONFIG_H=1 -DJavaScriptCore_EXPORTS -DNOCRYPT -DNOMINMAX -DUNICODE -DWINVER=0x601 -DWTF_PLATFORM_WIN_CAIRO=1 -D_CRT_SECURE_CPP_OVERLOAD_STANDARD_NAMES=1 -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_UNICODE -D_WIN32_WINNT=0x601 -D_WINDOWS -D_WINSOCKAPI_="" -IJavaScriptCore\Headers -I. -I..\..\Source\JavaScriptCore -I..\..\Source\JavaScriptCore\API -I..\..\Source\JavaScriptCore\assembler -I..\..\Source\JavaScriptCore\b3 -I..\..\Source\JavaScriptCore\b3\air -I..\..\Source\JavaScriptCore\bindings -I..\..\Source\JavaScriptCore\builtins -I..\..\Source\JavaScriptCore\bytecode -I..\..\Source\JavaScriptCore\bytecompiler -I..\..\Source\JavaScriptCore\dfg -I..\..\Source\JavaScriptCore\disassembler -I..\..\Source\JavaScriptCore\disassembler\ARM64 -I..\..\Source\JavaScriptCore\disassembler\udis86 -I..\..\Source\JavaScriptCore\domjit -I..\..\Source\JavaScriptCore\ftl -I..\..\Source\JavaScriptCore\heap -I..\..\Source\JavaScriptCore\debugger -I..\..\Source\JavaScriptCore\inspector -I..\..\Source\JavaScriptCore\inspector\agents -I..\..\Source\JavaScriptCore\inspector\augmentable -I..\..\Source\JavaScriptCore\inspector\remote -I..\..\Source\JavaScriptCore\interpreter -I..\..\Source\JavaScriptCore\jit -I..\..\Source\JavaScriptCore\llint -I..\..\Source\JavaScriptCore\parser -I..\..\Source\JavaScriptCore\profiler -I..\..\Source\JavaScriptCore\runtime -I..\..\Source\JavaScriptCore\tools -I..\..\Source\JavaScriptCore\wasm -I..\..\Source\JavaScriptCore\wasm\js -I..\..\Source\JavaScriptCore\yarr -IJavaScriptCore\DerivedSources -IJavaScriptCore\DerivedSources\inspector -IJavaScriptCore\DerivedSources\runtime -IJavaScriptCore\DerivedSources\yarr -I..\include\private -I..\..\Source\JavaScriptCore\inspector\remote\socket -IWTF\Headers -I..\..\WebKitLibraries\win\include /W4 /DWIN32 /D_WINDOWS  /GR- /EHs- /EHc-  /MD /O2 /Ob2 /DNDEBUG /wd4018 /wd4060 /wd4068 /wd4100 /wd4127 /wd4146 /wd4189 /wd4201 /wd4244 /wd4245 /wd4251 /wd4275 /wd4267 /wd4305 /wd4309 /wd4312 /wd4324 /wd4389 /wd4456 /wd4457 /wd4458 /wd4459 /wd4505 /wd4611 /wd4646 /wd4701 /wd4702 /wd4706 /wd4715 /wd4722 /wd4838 /wd4840 /wd4996 /wd5205 /Zi /GS /EHa- /EHc- /EHs- /fp:except- /analyze- /bigobj /utf-8 /validate-charset /Oy- /Zc:lambda -std:c++17 /showIncludes /FoSource\JavaScriptCore\CMakeFiles\JavaScriptCore.dir\__\__\JavaScriptCore\DerivedSources\unified-sources\UnifiedSource-ee8a7a7a-3.cpp.obj /FdSource\JavaScriptCore\CMakeFiles\JavaScriptCore.dir\ /FS -c JavaScriptCore\DerivedSources\unified-sources\UnifiedSource-ee8a7a7a-3.cpp
C:\Buildbot\WinCairo-EWS\build\Source\JavaScriptCore\heap/Heap.cpp(2874): error C2065: 'Visitor': undeclared identifier
C:\Buildbot\WinCairo-EWS\build\Source\JavaScriptCore\heap/Heap.cpp(2901): note: see reference to function template instantiation 'auto JSC::Heap::addCoreConstraints::<lambda_12>::()::<lambda_2>::operator ()<JSC::IsoCellSet>(_T1 &) const' being compiled
        with
        [
            _T1=JSC::IsoCellSet
        ]
C:\Buildbot\WinCairo-EWS\build\WebKitBuild\Release\WTF\Headers\wtf/Function.h(52): note: see reference to function template instantiation 'auto JSC::Heap::addCoreConstraints::<lambda_12>::operator ()<JSC::SlotVisitor>(_T1 &) const' being compiled
        with
        [
            _T1=JSC::SlotVisitor
        ]
C:\Buildbot\WinCairo-EWS\build\WebKitBuild\Release\WTF\Headers\wtf/Function.h(52): note: while compiling class template member function 'Out WTF::Detail::CallableWrapper<CallableType,Out,JSC::SlotVisitor &>::call(JSC::SlotVisitor &)'
        with
        [
            Out=void,
            CallableType=JSC::Heap::addCoreConstraints::<lambda_12>
        ]
C:\Buildbot\WinCairo-EWS\build\WebKitBuild\Release\WTF\Headers\wtf/StdLibExtras.h(505): note: see reference to class template instantiation 'WTF::Detail::CallableWrapper<CallableType,Out,JSC::SlotVisitor &>' being compiled
        with
        [
            CallableType=JSC::Heap::addCoreConstraints::<lambda_12>,
            Out=void
        ]
C:\Buildbot\WinCairo-EWS\build\WebKitBuild\Release\WTF\Headers\wtf/Function.h(74): note: see reference to function template instantiation 'std::unique_ptr<T,std::default_delete<T>> WTF::makeUnique<WTF::Detail::CallableWrapper<CallableType,Out,JSC::SlotVisitor &>,_Ty>(_Ty &&)' being compiled
        with
        [
            T=WTF::Detail::CallableWrapper<JSC::Heap::addCoreConstraints::<lambda_12>,void,JSC::SlotVisitor &>,
            CallableType=JSC::Heap::addCoreConstraints::<lambda_12>,
            Out=void,
            _Ty=JSC::Heap::addCoreConstraints::<lambda_12>
        ]
C:\Buildbot\WinCairo-EWS\build\Source\JavaScriptCore\heap/Heap.cpp(2901): note: see reference to function template instantiation 'WTF::Function<void (JSC::SlotVisitor &)>::Function<JSC::Heap::addCoreConstraints::<lambda_12>,void>(CallableType &&)' being compiled
        with
        [
            CallableType=JSC::Heap::addCoreConstraints::<lambda_12>
        ]
C:\Buildbot\WinCairo-EWS\build\Source\JavaScriptCore\heap/Heap.cpp(2872): note: see reference to function template instantiation 'WTF::Function<void (JSC::SlotVisitor &)>::Function<JSC::Heap::addCoreConstraints::<lambda_12>,void>(CallableType &&)' being compiled
        with
        [
            CallableType=JSC::Heap::addCoreConstraints::<lambda_12>
        ]
C:\Buildbot\WinCairo-EWS\build\Source\JavaScriptCore\heap/Heap.cpp(2874): error C2672: 'JSC::IsoCellSet::forEachMarkedCellInParallel': no matching overloaded function found
C:\Buildbot\WinCairo-EWS\build\Source\JavaScriptCore\heap/Heap.cpp(2901): error C2974: 'JSC::IsoCellSet::forEachMarkedCellInParallel': invalid template argument for 'Visitor', type expected
C:\Buildbot\WinCairo-EWS\build\Source\JavaScriptCore\heap\IsoCellSet.h(62): note: see declaration of 'JSC::IsoCellSet::forEachMarkedCellInParallel'
C:\Buildbot\WinCairo-EWS\build\Source\JavaScriptCore\heap/Heap.cpp(2874): error C2672: 'JSC::Subspace::forEachMarkedCellInParallel': no matching overloaded function found
C:\Buildbot\WinCairo-EWS\build\Source\JavaScriptCore\heap/Heap.cpp(2901): error C2974: 'JSC::Subspace::forEachMarkedCellInParallel': invalid template argument for 'Visitor', type expected
C:\Buildbot\WinCairo-EWS\build\Source\JavaScriptCore\heap\Subspace.h(90): note: see declaration of 'JSC::Subspace::forEachMarkedCellInParallel'

[Fujii Hironori]

I don't see the error on my PC.
Is this caused by specific MSVC version.
I'm using MSVC 14.29 (Visual Studio 2019 Version 16.9).
WinCairo EWS seems using MSVC 14.28 (Visual Studio 2019 Version 16.8).

[QuellaZhang]

Hmm, for error C2065，I found relevant information, it is indeed a bug of MSVC, currently fixed in 16.9. You may need to upgrade VS.

https://developercommunity.visualstudio.com/t/captured-variable-not-recognized-when-calling-temp/1137293
https://developercommunity.visualstudio.com/t/nested-c20-template-lambdas-two-issues-with-captur/1150322

I don’t know much about VS2022, there is a related blog https://devblogs.microsoft.com/visualstudio/visual-studio-2022/.

[Don Olmstead]

(In reply to QuellaZhang from comment #8)
> Hmm, for error C2065，I found relevant information, it is indeed a bug of
> MSVC, currently fixed in 16.9. You may need to upgrade VS.
> 
> https://developercommunity.visualstudio.com/t/captured-variable-not-
> recognized-when-calling-temp/1137293
> https://developercommunity.visualstudio.com/t/nested-c20-template-lambdas-
> two-issues-with-captur/1150322
> 
> I don’t know much about VS2022, there is a related blog
> https://devblogs.microsoft.com/visualstudio/visual-studio-2022/.

Fujii's patch adds a compiler flag based on versioning I'm just curious if that flag will still be valid with 2022. If not then there'd need to be an upper bound.

[QuellaZhang]

Hey Don,

Confirmed with the developer: 

>>Isn't VS 2022 releasing soon? Is the new lambda processor on by default for it?

Not: at the moment: it is the default if you compile with /std:c++latest or /permissive- -- it may become the default in VS2022 but we haven't made that decision yet.

[Radar WebKit Bug Importer]

<rdar://problem/78607135>

[Don Olmstead]

(In reply to QuellaZhang from comment #10)
> Hey Don,
> 
> Confirmed with the developer: 
> 
> >>Isn't VS 2022 releasing soon? Is the new lambda processor on by default for it?
> 
> Not: at the moment: it is the default if you compile with /std:c++latest or
> /permissive- -- it may become the default in VS2022 but we haven't made that
> decision yet.

Thanks for letting us know!

Fujii I think its probably better to start compiling with /permissive- rather than this. Thoughts?

[Fujii Hironori]

(In reply to Don Olmstead from comment #12)
> (In reply to QuellaZhang from comment #10)
> > Hey Don,
> > 
> > Confirmed with the developer: 
> > 
> > >>Isn't VS 2022 releasing soon? Is the new lambda processor on by default for it?
> > 
> > Not: at the moment: it is the default if you compile with /std:c++latest or
> > /permissive- -- it may become the default in VS2022 but we haven't made that
> > decision yet.
> 
> Thanks for letting us know!
> 
> Fujii I think its probably better to start compiling with /permissive-
> rather than this. Thoughts?

> the first public preview of Visual Studio 2022 will be released this summer.
https://devblogs.microsoft.com/visualstudio/visual-studio-2022/

QuellaZhang, when do you want to solve the original problem (comment#0)?
I think waiting for the official VS2022 release doesn't meat the QuellaZhang's schedule.

[Fujii Hironori]

meet, not 🐷.

[QuellaZhang]

Hi Fujii,

We added "/Zc:lambda" to command line as workaround to unblock us. Please follow your schedule to fix this issue. :)

[Fujii Hironori]

Bug 230599 can be solved by the /Zc:lambda switch.

[Fujii Hironori]

Created attachment 439085 [details]
Patch

[Ross Kirsling]

Comment on attachment 439085 [details]
Patch

Seems fine to me.

[Fujii Hironori]

Comment on attachment 439085 [details]
Patch

Clearing flags on attachment: 439085

Committed r283007 (242072@main): <https://commits.webkit.org/242072@main>

[Fujii Hironori]

All reviewed patches have been landed.  Closing bug.

[Fujii Hironori]

Reopening to attach new patch.

[Fujii Hironori]

Created attachment 439568 [details]
Wrong Patch

[EWS Watchlist]

This patch modifies the JS builtins code generator. Please ensure that you have rebaselined any generator test results (i.e., by running `Tools/Scripts/run-builtins-generator-tests --reset-results`)

[Fujii Hironori]

Hi QuellaZhang,

After this change, we have another problem. Could you take a look?
  Bug 230877 – [Win] WebCore.dir\Debug\WebCore.lib : fatal error LNK1248: image size (1000309F0) exceeds maximum allowable size (FFFFFFFF)

[April]

Yes, the AppleWin port failed with one known bug: https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1490389/

[Fujii Hironori]

(In reply to April from comment #25)
> Yes, the AppleWin port failed with one known bug:
> https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1490389/

I can't see the page.
> 401 - Uh-oh, you do not have access.
If you have a problem, please create a new ticket.