| Summary: | webkitpy Bugzilla transactions should retry on URLError | ||||||
|---|---|---|---|---|---|---|---|
| Product: | WebKit | Reporter: | Aakash Jain <aakash_jain> | ||||
| Component: | Tools / Tests | Assignee: | Aakash Jain <aakash_jain> | ||||
| Status: | RESOLVED FIXED | ||||||
| Severity: | Normal | CC: | aakash_jain, ap, commit-queue, dean_johnson, ews-watchlist, glenn, jbedard, lforschler, thorton, webkit-bug-importer | ||||
| Priority: | P2 | Keywords: | InRadar | ||||
| Version: | Other | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| See Also: |
https://bugs.webkit.org/show_bug.cgi?id=182420 https://bugs.webkit.org/show_bug.cgi?id=182987 https://bugs.webkit.org/show_bug.cgi?id=183222 |
||||||
| Attachments: |
|
||||||
|
Description
Aakash Jain
2018-02-26 19:16:04 PST
Few examples of failures while talking to bugzilla: https://webkit-queues.webkit.org/results/6681924 https://webkit-queues.webkit.org/results/6681223 https://webkit-queues.webkit.org/results/6681840 e.g.: 2018-02-25 12:28:08,404 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334583&action=edit 2018-02-25 12:28:08,866 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183109&ctype=xml&excludefield=attachmentdata 2018-02-25 12:28:09,186 - Running: webkit-patch --status-host=webkit-queues.webkit.org --bot-id=webkit-misc-style apply-attachment --no-update --non-interactive 334583 2018-02-25 12:28:26,507 - Applied patch 2018-02-25 12:28:27,929 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334583&action=edit Traceback (most recent call last): File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/bot/queueengine.py", line 103, in run if not self._delegate.process_work_item(work_item): File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/commands/queues.py", line 452, in process_work_item passed = self.review_patch(patch) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/commands/queues.py", line 476, in review_patch style_check_succeeded = task.run() File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/bot/stylequeuetask.py", line 75, in run self._apply_watch_list() File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/bot/stylequeuetask.py", line 66, in _apply_watch_list "Unabled to apply watchlist") File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py", line 97, in _run_command if not self.validate(): File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/bot/stylequeuetask.py", line 39, in validate self._patch = self._delegate.refetch_patch(self._patch) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/commands/queues.py", line 511, in refetch_patch return self._tool.bugs.fetch_attachment(patch.id()) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla.py", line 518, in fetch_attachment bug_id = self.bug_id_for_attachment_id(attachment_id) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla.py", line 508, in bug_id_for_attachment_id page = self.browser.open(attachment_url) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_mechanize.py", line 203, in open return self._mech_open(url, data, timeout=timeout) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_mechanize.py", line 230, in _mech_open response = UserAgentBase.open(self, request, data) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_opener.py", line 193, in open response = urlopen(self, req, data) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_urllib2_fork.py", line 344, in _open '_open', req) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_urllib2_fork.py", line 332, in _call_chain result = func(*args) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_urllib2_fork.py", line 1170, in https_open return self.do_open(conn_factory, req) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_urllib2_fork.py", line 1118, in do_open raise URLError(err) URLError: <urlopen error [Errno 60] Operation timed out> Created attachment 334665 [details]
Proposed patch
iOS EWS for this patch failed with exactly same network issue which would be fixed by this patch. Aakash asked me to look this over this afternoon; I took a look at NetworkTransaction's implementation and the use of lambdas here and everything looks like it should work as described by Aakash. Unofficial r=me. Comment on attachment 334665 [details] Proposed patch Clearing flags on attachment: 334665 Committed r229064: <https://trac.webkit.org/changeset/229064> All reviewed patches have been landed. Closing bug. This seems to help significantly. Noticing the logs, all the patches below would have failed ews with network exceptions, which now worked properly. ews109: 2018-02-28 09:55:14,708 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334748&action=edit 2018-02-28 09:55:22,578 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-28 10:54:30,971 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183191&ctype=xml&excludefield=attachmentdata 2018-02-28 10:55:45,975 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-28 10:55:56,258 - Started processing patch 2018-02-28 11:30:09,177 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183184&ctype=xml&excludefield=attachmentdata 2018-02-28 11:30:17,102 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-28 11:30:27,401 - Started processing patch 2018-02-28 11:30:28,085 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183184&ctype=xml&excludefield=attachmentdata 2018-02-28 11:30:36,017 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-28 11:30:46,312 - Running: webkit-patch --status-host=webkit-queues.webkit.org --bot-id=ews109 clean --port=ios-device --architecture=arm64 ews108: 2018-02-28 06:33:50,141 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334742&action=edit 2018-02-28 06:35:05,177 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-28 06:35:15,187 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334742&action=edit 2018-02-28 06:36:30,202 - Received URLError: [Errno 60] Operation timed out. Retrying in 15.0 seconds... 2018-02-28 06:36:45,211 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334742&action=edit 2018-02-28 08:56:08,825 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334369&action=edit 2018-02-28 08:57:23,847 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-28 08:57:33,857 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334369&action=edit 2018-02-28 11:50:43,646 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183218&ctype=xml&excludefield=attachmentdata 2018-02-28 11:50:51,515 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-28 11:51:01,759 - Running: webkit-patch --status-host=webkit-queues.webkit.org --bot-id=ews108 clean --port=ios-device --architecture=arm64 2018-02-28 12:59:05,387 - Applied patch 2018-02-28 12:59:05,788 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334759&action=edit 2018-02-28 12:59:13,791 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-28 12:59:23,799 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334759&action=edit Note that this fixed Note that this added retry for some of the (frequent) network transactions with Bugzilla. We should add retry for ALL the network transaction with Bugzilla. Tracked in https://bugs.webkit.org/show_bug.cgi?id=183222 |