Bug 33655 – Chromium builders fail "gclient" step every so often

• Component: Tools / Tests
• Status: RESOLVED WONTFIX

[Eric Seidel (no email)]

Chromium builders fail "gclient" step every so often

This is the 3rd time I've seen this.  It's not very common, but it's common enough that gclient fails that we should investigate making it retry.  Yaar mentioned over email that chromium.org had some retry magic in their buildbot that we should consider using.

http://build.webkit.org/builders/Chromium%20Mac%20Release/builds/1732/steps/gclient/logs/stdio

This failure is actually probably a bad example as it may be a legit src.chromium.org downtime:

________ running 'svn update /Users/cltbld/Desktop/BuildSlaveData/WebKit-BuildSlave/chromium-mac-release/build/WebKit/chromium/net --revision 31834' in '/Users/cltbld/Desktop/BuildSlaveData/WebKit-BuildSlave/chromium-mac-release/build/WebKit/chromium'
svn: REPORT request failed on '/svn/!svn/vcc/default'
svn: REPORT of '/svn/!svn/vcc/default': 502 Bad Gateway (http://src.chromium.org)
Error: failed to run command: svn update /Users/cltbld/Desktop/BuildSlaveData/WebKit-BuildSlave/chromium-mac-release/build/WebKit/chromium/net --revision 31834
Died at ./WebKitTools/Scripts/update-webkit-chromium line 55.

[Eric Seidel (no email)]

Another gclient failure:
http://build.webkit.org/builders/Chromium%20Win%20Release/builds/1062/steps/gclient/logs/stdio
________ running 'svn update C:\WebKitBuildSlave\chromium-win-release\build\WebKit\chromium\third_party\icu --revision 31724' in 'C:\WebKitBuildSlave\chromium-win-release\build\WebKit\chromium'
svn: Server sent unexpected return value (502 Bad Gateway) in response to PROPFIND request for '/svn/trunk/deps/third_party/icu42'
Error: failed to run command: svn update C:\WebKitBuildSlave\chromium-win-release\build\WebKit\chromium\third_party\icu --revision 31724
No such file or directory at ./WebKitTools/Scripts/update-webkit-chromium line 55.
program finished with exit code 2

[Yaar Schnitman]

Opened a bug on gclient:
http://code.google.com/p/gclient/issues/detail?id=16

[Eric Seidel (no email)]

Failed again:
http://build.webkit.org/builders/Chromium%20Win%20Release/builds/1448/steps/gclient/logs/stdio

________ running 'svn update C:\WebKitBuildSlave\chromium-win-release\build\WebKit\chromium\third_party\npapi --revision 31834' in 'C:\WebKitBuildSlave\chromium-win-release\build\WebKit\chromium'
svn: Server sent unexpected return value (502 Bad Gateway) in response to OPTIONS request for 'http://src.chromium.org/svn/trunk/src/third_party/npapi'
Error: failed to run command: svn update C:\WebKitBuildSlave\chromium-win-release\build\WebKit\chromium\third_party\npapi --revision 31834
No such file or directory at ./WebKitTools/Scripts/update-webkit-chromium line 55.

[Eric Seidel (no email)]

Failed again:
http://build.webkit.org/builders/Chromium%20Linux%20Release/builds/1557/steps/gclient/logs/stdio

[Eric Seidel (no email)]

Again:
http://build.webkit.org/builders/Chromium%20Linux%20Release/builds/1692/steps/gclient/logs/stdio
________ running 'svn update /WebKit-BuildSlave/chromium-linux-release/build/WebKit/chromium/third_party/libxslt --revision 31834' in '/WebKit-BuildSlave/chromium-linux-release/build/WebKit/chromium'
svn: Server sent unexpected return value (502 Bad Gateway) in response to PROPFIND request for '/svn/trunk/src/third_party/libxslt'
Error: failed to run command: svn update /WebKit-BuildSlave/chromium-linux-release/build/WebKit/chromium/third_party/libxslt --revision 31834
Bad file descriptor at ./WebKitTools/Scripts/update-webkit-chromium line 55.

[Eric Seidel (no email)]

This should be helped by:
http://code.google.com/p/gclient/issues/detail?id=16

[Eric Seidel (no email)]

Syncing projects:  46% (12/26)  
________ running 'svn update /WebKit-BuildSlave/chromium-linux-release/build/WebKit/chromium/third_party/icu --revision 31724' in '/WebKit-BuildSlave/chromium-linux-release/build/WebKit/chromium'
svn: Working copy '/WebKit-BuildSlave/chromium-linux-release/build/WebKit/chromium/third_party/icu' locked
svn: run 'svn cleanup' to remove locks (type 'svn help cleanup' for details)
Error: failed to run command: svn update /WebKit-BuildSlave/chromium-linux-release/build/WebKit/chromium/third_party/icu --revision 31724
Bad file descriptor at ./WebKitTools/Scripts/update-webkit-chromium line 55.

The update step for gclient doesnt' know how to rm -rf after failure like the normal svn update step for buildbot does.

[Eric Seidel (no email)]

Failed again:
http://build.webkit.org/builders/Chromium%20Win%20Release/builds/3956/steps/gclient/logs/stdio

[Eric Seidel (no email)]

We may need to consider removing chromium from core builders.  This is the third bot outage for CR bots this week. :(

[Eric Seidel (no email)]

Again:
http://build.webkit.org/builders/Chromium%20Win%20Release/builds/4026/steps/gclient/logs/stdio

[Eric Seidel (no email)]

This still happens!

http://build.webkit.org/builders/Chromium%20Linux%20Release/builds/7681/steps/gclient/logs/stdio

We need to either hack around this, fix the problem, or kick Chromium out of the Core builders.  This is unacceptable.

Qt hacked around their builder-lameness with https://bugs.webkit.org/show_bug.cgi?id=39282 (which I'm not encouraging). But even that would be better than the current situation of commit-queue stalls and ever-so-often false blames.

[Eric Seidel (no email)]

Ok, that one was an actual real DEPS problem.  But there was at least one example earlier today as well:
http://build.webkit.org/builders/Chromium%20Win%20Release/builds/7264/steps/gclient/logs/stdio

That one was a bad-gateway error talking to chromium svn.

[Eric Seidel (no email)]

And again:
http://build.webkit.org/builders/Chromium%20Linux%20Release/builds/7688/steps/gclient/logs/stdio

Just now some sort of XML error.

[Stephen Chenney]

LayoutTest failures for Chromium are being marked WontFix. The Bug is still accessible and referenced from TestExpectations.