This is the latest log from the win-ews bot: Cleaning working directory Building WebKit Failed to run "['WebKitTools/Scripts/build-webkit', '--debug']" exit_code: 1 Unable to perform a build Unable to process work item. Sleeping until 2010-07-01 01:49:52 (2 mins). Fetching: https://bugs.webkit.org/attachment.cgi?id=60152&action=edit Traceback (most recent call last): File "/mnt/svn/webkit-script/WebKitTools/Scripts/webkitpy/tool/bot/queueengin .py", line 97, in run work_item = self._delegate.next_work_item() File "/mnt/svn/webkit-script/WebKitTools/Scripts/webkitpy/tool/commands/queue .py", line 358, in next_work_item return self.tool.bugs.fetch_attachment(patch_id) File "/mnt/svn/webkit-script/WebKitTools/Scripts/webkitpy/common/net/bugzilla py", line 564, in fetch_attachment bug_id = self.bug_id_for_attachment_id(attachment_id) File "/mnt/svn/webkit-script/WebKitTools/Scripts/webkitpy/common/net/bugzilla py", line 554, in bug_id_for_attachment_id page = self.browser.open(attachment_url) File "/home/Administrator/svn/webkit-script/WebKitTools/Scripts/webkitpy/thir party/autoinstalled/mechanize/_mechanize.py", line 209, in open File "/home/Administrator/svn/webkit-script/WebKitTools/Scripts/webkitpy/thir party/autoinstalled/mechanize/_mechanize.py", line 236, in _mech_open File "/home/Administrator/svn/webkit-script/WebKitTools/Scripts/webkitpy/thir party/autoinstalled/mechanize/_opener.py", line 191, in open File "/usr/lib/python2.5/urllib2.py", line 399, in _open '_open', req) File "/usr/lib/python2.5/urllib2.py", line 360, in _call_chain result = func(*args) File "/home/Administrator/svn/webkit-script/WebKitTools/Scripts/webkitpy/thir party/autoinstalled/mechanize/_http.py", line 756, in https_open File "/home/Administrator/svn/webkit-script/WebKitTools/Scripts/webkitpy/thir party/autoinstalled/mechanize/_http.py", line 706, in do_open URLError: <urlopen error (8, 'EOF occurred in violation of protocol')> Exception while preparing queue Sleeping until 2010-07-01 02:05:38 (2 mins). Fetching: https://bugs.webkit.org/attachment.cgi?id=60152&action=edit Fetching: https://bugs.webkit.org/show_bug.cgi?id=41413&ctype=xml Logging in as webkit.review.bot@gmail.com... User terminated queue. Updating hacked copy of webkit-patch Updating OpenSource Still investigating.
The bot was hung. The last log was about trying to log into bug.webkit.org. Not sure what it was hung on.
It could have been an EC2 problem, since all of the affected bots (cr-linux, gtk, and win) are run off of EC2 instances.
Why didn't the autorestart script cure things? Maybe we need a hang detector.
I think we may need a hang detector. Again, not sure where it was hung, but it was definitely hung.
Doesn't sounds like something we'll investigate 4 years later.