Bug 140845 - [Win] Switch build to MSBuild so we get more useful stdout logging in our buildbot infrastructure
Summary: [Win] Switch build to MSBuild so we get more useful stdout logging in our bui...
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: PC All
: P2 Normal
Assignee: Brent Fulgham
Depends on:
Blocks: 142360
  Show dependency treegraph
Reported: 2015-01-23 16:07 PST by Brent Fulgham
Modified: 2015-03-05 13:17 PST (History)
6 users (show)

See Also:

Patch (8.59 KB, patch)
2015-03-05 10:43 PST, Brent Fulgham
ddkilzer: review+
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Brent Fulgham 2015-01-23 16:07:18 PST
See Bug 140819 for context.

Basically, we encounter false 'timeout' events on Windows builds because the Visual Studio-based build system doesn't output anything to stdout until the build is complete. Since our timeout was previously 20 minutes, a full rebuild (which takes around 40 minutes or so on Windows) would generate a "build failed" message even though there was nothing wrong.

This was worked around in Bug 140819, however a proper fix for this is to use MSBuild to drive the compile. This generates stdout messages which will keep buildbot happy.
Comment 1 Brent Fulgham 2015-03-05 09:59:14 PST
I spent some time playing with this last night, and found a few useful things:

1. MSBuild output appears to go to stdout, so we might be able to remove the Windows-specific build timeout we were using to avoid false build failures.

2. MSBuild accepts an input flag "/verbosity", which can be set to "minimal". When I do so, we get very concise logging, that only shows files that were built and any errors or warnings. All the other cruft is missing.

3. MSBuild is perfectly happy running with out existing Solutions. So there is very little effort to turn this on, and very little harm in reverting to our old way if we encounter problems.

Once I confirm proper build behavior locally (in the next hour or so), I will propose a patch to switch to MSBuild.
Comment 2 Brent Fulgham 2015-03-05 10:43:43 PST
Created attachment 247967 [details]
Comment 3 Brent Fulgham 2015-03-05 10:55:19 PST
This patch may not work on EWS without rebooting those bots. It looks like the existing MSBuild processes triggered by the build loop are locking the log file we attempt to write.
Comment 4 David Kilzer (:ddkilzer) 2015-03-05 11:46:22 PST
Comment on attachment 247967 [details]

Comment 5 Brent Fulgham 2015-03-05 12:05:41 PST
Committed r181092: <http://trac.webkit.org/changeset/181092>
Comment 6 Brent Fulgham 2015-03-05 12:14:13 PST
I'm noticing this in the logs:

  TestWebKitAPI.vcxproj -> C:\cygwin\home\buildbot\slave\win-release\build\WebKitBuild\Release\bin32\TestWebKitAPI.exe
  The process cannot access the file because it is being used by another process.
  The process cannot access the file because it is being used by another process.

I think the AssembleBuildLogs command is trying to grab old log files and add them to the new BuildOutput.html. However, this is locked by MSBuild (since that is the output file it generates).

I may need to clean up the build machines to get rid of this warning message.
Comment 7 Brent Fulgham 2015-03-05 13:07:03 PST
The EWS bots are failing because MSBuild wants the build output directory to exist at launch so that it can immediately start writing logs.

Visual Studio was apparently happy starting up, creating any directories it needed, then starting logging as part of a pre-build step.

I need to revise the Windows build logic to make sure the build target directory exists at the time we trigger MSBuild.