Bug 183156 – webkitpy Bugzilla transactions should retry on URLError

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Aakash Jain]

We have been seeing frequent network errors recently in https://bugs.webkit.org/show_bug.cgi?id=182420. We are debugging the root-cause to fix the underlying network issue in <rdar://problem/37716391>.

However, our code should be more robust against network failures and it should retry when it encounters network issue like: URLError: <urlopen error [Errno 60] Operation timed out>.

We added URLError handling in NetworkTransaction class, and Network issues while communicating with webkit-queues server reduced drastically (https://bugs.webkit.org/show_bug.cgi?id=182987). Bugzilla class in webkitpy should also use NetworkTransaction class to use the retry functionality.

[Aakash Jain]

Few examples of failures while talking to bugzilla:

https://webkit-queues.webkit.org/results/6681924
https://webkit-queues.webkit.org/results/6681223
https://webkit-queues.webkit.org/results/6681840

[Aakash Jain]

e.g.:

2018-02-25 12:28:08,404 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334583&action=edit
2018-02-25 12:28:08,866 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183109&ctype=xml&excludefield=attachmentdata
2018-02-25 12:28:09,186 - Running: webkit-patch --status-host=webkit-queues.webkit.org --bot-id=webkit-misc-style apply-attachment --no-update --non-interactive 334583
2018-02-25 12:28:26,507 - Applied patch
2018-02-25 12:28:27,929 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334583&action=edit
Traceback (most recent call last):
  File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/bot/queueengine.py", line 103, in run 
    if not self._delegate.process_work_item(work_item):
  File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/commands/queues.py", line 452, in process_work_item
    passed = self.review_patch(patch)
  File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/commands/queues.py", line 476, in review_patch
    style_check_succeeded = task.run()
  File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/bot/stylequeuetask.py", line 75, in run 
    self._apply_watch_list()
  File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/bot/stylequeuetask.py", line 66, in _apply_watch_list
    "Unabled to apply watchlist")
  File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py", line 97, in _run_command
    if not self.validate():
  File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/bot/stylequeuetask.py", line 39, in validate
    self._patch = self._delegate.refetch_patch(self._patch)
  File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/commands/queues.py", line 511, in refetch_patch
    return self._tool.bugs.fetch_attachment(patch.id())
  File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla.py", line 518, in fetch_attachment
    bug_id = self.bug_id_for_attachment_id(attachment_id)
  File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla.py", line 508, in bug_id_for_attachment_id
    page = self.browser.open(attachment_url)
  File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_mechanize.py", line 203, in open
    return self._mech_open(url, data, timeout=timeout)
  File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_mechanize.py", line 230, in _mech_open
    response = UserAgentBase.open(self, request, data)
  File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_opener.py", line 193, in open
    response = urlopen(self, req, data)
  File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_urllib2_fork.py", line 344, in _open
    '_open', req)
  File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_urllib2_fork.py", line 332, in _call_chain
    result = func(*args)
  File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_urllib2_fork.py", line 1170, in https_open
    return self.do_open(conn_factory, req)
  File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_urllib2_fork.py", line 1118, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 60] Operation timed out>

[Aakash Jain]

Created attachment 334665 [details]
Proposed patch

[Aakash Jain]

iOS EWS for this patch failed with exactly same network issue which would be fixed by this patch.

[Dean Johnson]

Aakash asked me to look this over this afternoon; I took a look at NetworkTransaction's implementation and the use of lambdas here and everything looks like it should work as described by Aakash.

Unofficial r=me.

[WebKit Commit Bot]

Comment on attachment 334665 [details]
Proposed patch

Clearing flags on attachment: 334665

Committed r229064: <https://trac.webkit.org/changeset/229064>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/37945984>

[Aakash Jain]

This seems to help significantly. Noticing the logs, all the patches below would have failed ews with network exceptions, which now worked properly.


ews109:
2018-02-28 09:55:14,708 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334748&action=edit
2018-02-28 09:55:22,578 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds...


2018-02-28 10:54:30,971 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183191&ctype=xml&excludefield=attachmentdata
2018-02-28 10:55:45,975 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds...
2018-02-28 10:55:56,258 - Started processing patch

2018-02-28 11:30:09,177 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183184&ctype=xml&excludefield=attachmentdata
2018-02-28 11:30:17,102 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds...
2018-02-28 11:30:27,401 - Started processing patch

2018-02-28 11:30:28,085 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183184&ctype=xml&excludefield=attachmentdata
2018-02-28 11:30:36,017 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds...
2018-02-28 11:30:46,312 - Running: webkit-patch --status-host=webkit-queues.webkit.org --bot-id=ews109 clean --port=ios-device --architecture=arm64


ews108:
2018-02-28 06:33:50,141 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334742&action=edit
2018-02-28 06:35:05,177 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds...
2018-02-28 06:35:15,187 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334742&action=edit
2018-02-28 06:36:30,202 - Received URLError: [Errno 60] Operation timed out. Retrying in 15.0 seconds...
2018-02-28 06:36:45,211 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334742&action=edit

2018-02-28 08:56:08,825 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334369&action=edit
2018-02-28 08:57:23,847 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds...
2018-02-28 08:57:33,857 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334369&action=edit


2018-02-28 11:50:43,646 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183218&ctype=xml&excludefield=attachmentdata
2018-02-28 11:50:51,515 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds...
2018-02-28 11:51:01,759 - Running: webkit-patch --status-host=webkit-queues.webkit.org --bot-id=ews108 clean --port=ios-device --architecture=arm64


2018-02-28 12:59:05,387 - Applied patch
2018-02-28 12:59:05,788 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334759&action=edit
2018-02-28 12:59:13,791 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds...
2018-02-28 12:59:23,799 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334759&action=edit

[Aakash Jain]

Note that this fixed

[Aakash Jain]

Note that this added retry for some of the (frequent) network transactions with Bugzilla. We should add retry for ALL the network transaction with Bugzilla. Tracked in https://bugs.webkit.org/show_bug.cgi?id=183222