Bug 187725

Summary: [CMake][Ninja][Win] midl.exe fails to create, open and write a temporary file in parallel build in Docker container
Product: WebKit Reporter: Fujii Hironori <Hironori.Fujii>
Component: Tools / TestsAssignee: Fujii Hironori <Hironori.Fujii>
Status: RESOLVED FIXED    
Severity: Normal CC: achristensen, bfulgham, commit-queue, don.olmstead, lforschler, mcatanzaro, pvollan, rniwa, ross.kirsling, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=184761
Attachments:
Description Flags
Patch to run midl.exe sequentially
none
Patch to run midl.exe sequentially
none
Patch to run midl.exe sequentially
none
Patch none

Description Fujii Hironori 2018-07-17 03:13:36 PDT
[WinCairo][EWS] midl : command line error MIDL1016 : cannot create intermediate file C:\Users\ContainerAdministrator\AppData\Local\Temp\MIDEFA7.tmp

https://webkit-queues.webkit.org/patch/345018/wincairo-ews

> midl : command line error MIDL1016 : cannot create intermediate file C:\Users\ContainerAdministrator\AppData\Local\Temp\MIDEFA7.tmp
Comment 1 Fujii Hironori 2018-07-20 00:33:13 PDT
A following error looks similar:

https://webkit-queues.webkit.org/patch/343701/wincairo-ews

> FAILED: DerivedSources/WebKitLegacy/Interfaces/AccessibleStates.h 
> 
> cmd.exe /C "cd /D C:\WebKit-EWS\WebKit\WebKitBuild\Release\Source\WebKitLegacy && midl.exe /I C:/WebKit-EWS/WebKit/Source/WebKitLegacy/win/Interfaces /I C:/WebKit-EWS/WebKit/Source/WebKitLegacy/win/Interfaces/Accessible2 /I C:/WebKit-EWS/WebKit/WebKitBuild/Release/DerivedSources/WebKitLegacy/include /I C:/WebKit-EWS/WebKit/Source/WebKitLegacy/win /WX /char signed /env win32 /tlb C:/WebKit-EWS/WebKit/WebKitBuild/Release/bin64/AccessibleStates.tlb /out C:/WebKit-EWS/WebKit/WebKitBuild/Release/DerivedSources/WebKitLegacy/Interfaces /h C:/WebKit-EWS/WebKit/WebKitBuild/Release/DerivedSources/WebKitLegacy/Interfaces/AccessibleStates.h /iid AccessibleStates_i.c "/D \"__PRODUCTION__=01\"" C:/WebKit-EWS/WebKit/Source/WebKitLegacy/win/Interfaces/Accessible2/AccessibleStates.idl"
> 
> Microsoft (R) 32b/64b MIDL Compiler Version 8.01.0622 
> 
> Copyright (c) Microsoft Corporation. All rights reserved.
> 
> midl : error MIDL2212 : error while writing to file C:\Users\ContainerAdministrator\AppData\Local\Temp\MID9469.tmp
Comment 2 Ross Kirsling 2018-09-02 13:04:26 PDT
Another one here. :(
https://webkit-queues.webkit.org/patch/348745/wincairo-ews
Comment 3 Ross Kirsling 2018-09-04 15:30:36 PDT
Although the one in my previous comment was the same as the first (MIDL1016), https://webkit-queues.webkit.org/results/9093311 has yet a third error code:

> [4501/6255] Generating ../../DerivedSources/WebKitLegacy/Interfaces/AccessibleStates.h
> FAILED: DerivedSources/WebKitLegacy/Interfaces/AccessibleStates.h 
> cmd.exe /C "cd /D C:\WebKit-EWS\WebKit\WebKitBuild\Release\Source\WebKitLegacy && midl.exe /I C:/WebKit-EWS/WebKit/Source/WebKitLegacy/win/Interfaces /I C:/WebKit-EWS/WebKit/Source/WebKitLegacy/win/Interfaces/Accessible2 /I C:/WebKit-EWS/WebKit/WebKitBuild/Release/DerivedSources/WebKitLegacy/include /I C:/WebKit-EWS/WebKit/Source/WebKitLegacy/win /WX /char signed /env win32 /tlb C:/WebKit-EWS/WebKit/WebKitBuild/Release/bin64/AccessibleStates.tlb /out C:/WebKit-EWS/WebKit/WebKitBuild/Release/DerivedSources/WebKitLegacy/Interfaces /h C:/WebKit-EWS/WebKit/WebKitBuild/Release/DerivedSources/WebKitLegacy/Interfaces/AccessibleStates.h /iid AccessibleStates_i.c "/D \"__PRODUCTION__=01\"" C:/WebKit-EWS/WebKit/Source/WebKitLegacy/win/Interfaces/Accessible2/AccessibleStates.idl"
> Microsoft (R) 32b/64b MIDL Compiler Version 8.01.0622 
> 
> Copyright (c) Microsoft Corporation. All rights reserved.
> 
> midl : error MIDL2419 : Unable to open intermediate compiler file : Permission denied  (C:\Users\ContainerAdministrator\AppData\Local\Temp\MID1C3B.tmp)
Comment 4 Fujii Hironori 2018-10-01 02:02:00 PDT
I can reproduce this issue by using Docker container on my PC.

perl Tools\Scripts\build-webkit --wincairo
cd WebKitBuild\Release
del DerivedSources/WebKitLegacy/Interfaces/*
ninja

If I invoke ninja -j1, it seems that it'd not happen.
Comment 5 Fujii Hironori 2018-10-01 02:26:17 PDT
Created attachment 351227 [details]
Patch to run midl.exe sequentially

I created a patch to run midl.exe sequentially, but this doesn't address the issue.
So strange.
Comment 6 Fujii Hironori 2018-10-01 02:28:59 PDT
Created attachment 351228 [details]
Patch to run midl.exe sequentially

Wrong patch attached. This is the patch I wanted to attach.
Comment 7 Fujii Hironori 2018-10-01 02:54:04 PDT
Created attachment 351230 [details]
Patch to run midl.exe sequentially
Comment 8 Fujii Hironori 2018-10-01 23:46:36 PDT
I created a PowerShell script to run midl.exe parallely in Docker.
https://gist.github.com/fujii/5dabb82dbd62dd535be6a44d6815b37b

Then, I got following errors.

> midl : error MIDL2419 : Unable to open intermediate compiler file : Permission denied  (C:\Users\ContainerAdministrator\AppData\Local\Temp\MID9E47.tmp)

> midl : command line error MIDL1016 : cannot create intermediate file C:\Users\ContainerAdministrator\AppData\Local\Temp\MID2B18.tmp

> midl : command line error MIDL1016 : cannot create intermediate file C:\Users\ContainerAdministrator\AppData\Local\Temp\MID3E58.tmp

But, If I run it sequentially, no such errors happened.
Comment 9 Fujii Hironori 2018-10-02 00:03:32 PDT
My conclusion is Docker seems to have a problem of parallel file system access.
WinCairo port should give up either Docker or parallel build.
Comment 10 Fujii Hironori 2018-10-03 21:16:54 PDT
Comment on attachment 351230 [details]
Patch to run midl.exe sequentially

These my patches are completely wrong. JOB_POOL_COMPILE target property doesn't affect to custom commands.
I should try USE_TERMINAL of add_custom_command to run midl.exe sequentially.
Comment 11 Fujii Hironori 2018-10-05 03:52:46 PDT
I've confirmed it still happens even with USES_TERMINAL. :-(
Comment 12 Fujii Hironori 2018-10-24 20:23:05 PDT
(In reply to Fujii Hironori from comment #11)
> I've confirmed it still happens even with USES_TERMINAL. :-(

This is wrong. I tested again. USES_TERMINAL actually solves this issue.
Comment 13 Fujii Hironori 2018-10-24 20:35:51 PDT
Created attachment 353076 [details]
Patch
Comment 14 Don Olmstead 2018-10-24 20:56:54 PDT
Informal r+

This will add the command to the job pool https://cmake.org/cmake/help/v3.13/prop_gbl/JOB_POOLS.html#prop_gbl:JOB_POOLS
Comment 15 WebKit Commit Bot 2018-10-25 09:33:40 PDT
Comment on attachment 353076 [details]
Patch

Clearing flags on attachment: 353076

Committed r237416: <https://trac.webkit.org/changeset/237416>
Comment 16 WebKit Commit Bot 2018-10-25 09:33:42 PDT
All reviewed patches have been landed.  Closing bug.
Comment 17 Radar WebKit Bug Importer 2018-10-25 09:34:41 PDT
<rdar://problem/45555931>