183156 2018-02-26 19:16:04 -0800 webkitpy Bugzilla transactions should retry on URLError 2018-02-28 13:35:44 -0800 1 1 1 Unclassified WebKit Tools / Tests Other Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=182420 https://bugs.webkit.org/show_bug.cgi?id=182987 https://bugs.webkit.org/show_bug.cgi?id=183222 InRadar P2 Normal --- 1 aakash_jain aakash_jain aakash_jain ap commit-queue dean_johnson ews-watchlist glenn jbedard lforschler thorton webkit-bug-importer oldest_to_newest 1402246 0 aakash_jain 2018-02-26 19:16:04 -0800 We have been seeing frequent network errors recently in https://bugs.webkit.org/show_bug.cgi?id=182420. We are debugging the root-cause to fix the underlying network issue in <rdar://problem/37716391>. However, our code should be more robust against network failures and it should retry when it encounters network issue like: URLError: <urlopen error [Errno 60] Operation timed out>. We added URLError handling in NetworkTransaction class, and Network issues while communicating with webkit-queues server reduced drastically (https://bugs.webkit.org/show_bug.cgi?id=182987). Bugzilla class in webkitpy should also use NetworkTransaction class to use the retry functionality. 1402247 1 aakash_jain 2018-02-26 19:16:28 -0800 Few examples of failures while talking to bugzilla: https://webkit-queues.webkit.org/results/6681924 https://webkit-queues.webkit.org/results/6681223 https://webkit-queues.webkit.org/results/6681840 1402248 2 aakash_jain 2018-02-26 19:18:20 -0800 e.g.: 2018-02-25 12:28:08,404 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334583&action=edit 2018-02-25 12:28:08,866 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183109&ctype=xml&excludefield=attachmentdata 2018-02-25 12:28:09,186 - Running: webkit-patch --status-host=webkit-queues.webkit.org --bot-id=webkit-misc-style apply-attachment --no-update --non-interactive 334583 2018-02-25 12:28:26,507 - Applied patch 2018-02-25 12:28:27,929 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334583&action=edit Traceback (most recent call last): File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/bot/queueengine.py", line 103, in run if not self._delegate.process_work_item(work_item): File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/commands/queues.py", line 452, in process_work_item passed = self.review_patch(patch) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/commands/queues.py", line 476, in review_patch style_check_succeeded = task.run() File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/bot/stylequeuetask.py", line 75, in run self._apply_watch_list() File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/bot/stylequeuetask.py", line 66, in _apply_watch_list "Unabled to apply watchlist") File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py", line 97, in _run_command if not self.validate(): File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/bot/stylequeuetask.py", line 39, in validate self._patch = self._delegate.refetch_patch(self._patch) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/commands/queues.py", line 511, in refetch_patch return self._tool.bugs.fetch_attachment(patch.id()) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla.py", line 518, in fetch_attachment bug_id = self.bug_id_for_attachment_id(attachment_id) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla.py", line 508, in bug_id_for_attachment_id page = self.browser.open(attachment_url) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_mechanize.py", line 203, in open return self._mech_open(url, data, timeout=timeout) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_mechanize.py", line 230, in _mech_open response = UserAgentBase.open(self, request, data) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_opener.py", line 193, in open response = urlopen(self, req, data) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_urllib2_fork.py", line 344, in _open '_open', req) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_urllib2_fork.py", line 332, in _call_chain result = func(*args) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_urllib2_fork.py", line 1170, in https_open return self.do_open(conn_factory, req) File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_urllib2_fork.py", line 1118, in do_open raise URLError(err) URLError: <urlopen error [Errno 60] Operation timed out> 1402249 3 334665 aakash_jain 2018-02-26 19:20:03 -0800 Created attachment 334665 Proposed patch 1402275 4 aakash_jain 2018-02-26 21:25:36 -0800 iOS EWS for this patch failed with exactly same network issue which would be fixed by this patch. 1402292 5 dean_johnson 2018-02-26 23:37:13 -0800 Aakash asked me to look this over this afternoon; I took a look at NetworkTransaction's implementation and the use of lambdas here and everything looks like it should work as described by Aakash. Unofficial r=me. 1402351 6 334665 commit-queue 2018-02-27 09:54:18 -0800 Comment on attachment 334665 Proposed patch Clearing flags on attachment: 334665 Committed r229064: <https://trac.webkit.org/changeset/229064> 1402352 7 commit-queue 2018-02-27 09:54:20 -0800 All reviewed patches have been landed. Closing bug. 1402353 8 webkit-bug-importer 2018-02-27 09:55:29 -0800 <rdar://problem/37945984> 1402702 9 aakash_jain 2018-02-28 13:26:13 -0800 This seems to help significantly. Noticing the logs, all the patches below would have failed ews with network exceptions, which now worked properly. ews109: 2018-02-28 09:55:14,708 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334748&action=edit 2018-02-28 09:55:22,578 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-28 10:54:30,971 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183191&ctype=xml&excludefield=attachmentdata 2018-02-28 10:55:45,975 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-28 10:55:56,258 - Started processing patch 2018-02-28 11:30:09,177 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183184&ctype=xml&excludefield=attachmentdata 2018-02-28 11:30:17,102 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-28 11:30:27,401 - Started processing patch 2018-02-28 11:30:28,085 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183184&ctype=xml&excludefield=attachmentdata 2018-02-28 11:30:36,017 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-28 11:30:46,312 - Running: webkit-patch --status-host=webkit-queues.webkit.org --bot-id=ews109 clean --port=ios-device --architecture=arm64 ews108: 2018-02-28 06:33:50,141 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334742&action=edit 2018-02-28 06:35:05,177 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-28 06:35:15,187 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334742&action=edit 2018-02-28 06:36:30,202 - Received URLError: [Errno 60] Operation timed out. Retrying in 15.0 seconds... 2018-02-28 06:36:45,211 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334742&action=edit 2018-02-28 08:56:08,825 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334369&action=edit 2018-02-28 08:57:23,847 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-28 08:57:33,857 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334369&action=edit 2018-02-28 11:50:43,646 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183218&ctype=xml&excludefield=attachmentdata 2018-02-28 11:50:51,515 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-28 11:51:01,759 - Running: webkit-patch --status-host=webkit-queues.webkit.org --bot-id=ews108 clean --port=ios-device --architecture=arm64 2018-02-28 12:59:05,387 - Applied patch 2018-02-28 12:59:05,788 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334759&action=edit 2018-02-28 12:59:13,791 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-28 12:59:23,799 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334759&action=edit 1402705 10 aakash_jain 2018-02-28 13:28:08 -0800 Note that this fixed 1402706 11 aakash_jain 2018-02-28 13:35:44 -0800 Note that this added retry for some of the (frequent) network transactions with Bugzilla. We should add retry for ALL the network transaction with Bugzilla. Tracked in https://bugs.webkit.org/show_bug.cgi?id=183222 334665 2018-02-26 19:20:03 -0800 2018-02-27 09:54:18 -0800 Proposed patch bugzilla_retry.patch text/plain 2265 aakash_jain SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDIyOTA1MikKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDE2IEBACisyMDE4LTAyLTI2ICBBYWthc2ggSmFpbiAgPGFha2FzaF9qYWluQGFwcGxlLmNv bT4KKworICAgICAgICB3ZWJraXRweSBCdWd6aWxsYSB0cmFuc2FjdGlvbnMgc2hvdWxkIHJldHJ5 IG9uIFVSTEVycm9yCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNn aT9pZD0xODMxNTYKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAg ICAgICAqIFNjcmlwdHMvd2Via2l0cHkvY29tbW9uL25ldC9idWd6aWxsYS9idWd6aWxsYS5weToK KyAgICAgICAgKEJ1Z3ppbGxhKTogVXNlIE5ldHdvcmtUcmFuc2FjdGlvbi4KKyAgICAgICAgKEJ1 Z3ppbGxhLl9mZXRjaF9idWdfcGFnZV9ieV91cmwpOgorICAgICAgICAoQnVnemlsbGEuYnVnX2lk X2Zvcl9hdHRhY2htZW50X2lkKToKKyAgICAgICAgKEJ1Z3ppbGxhLmdldF9idWdfaWRfZm9yX2F0 dGFjaG1lbnRfaWQpOgorCiAyMDE4LTAyLTI2ICBEb24gT2xtc3RlYWQgIDxkb24ub2xtc3RlYWRA c29ueS5jb20+CiAKICAgICAgICAgRml4IFdlYkNvcmUgaW5jbHVkZXMgd2l0aGluIFdlYktpdExl Z2FjeQpJbmRleDogVG9vbHMvU2NyaXB0cy93ZWJraXRweS9jb21tb24vbmV0L2J1Z3ppbGxhL2J1 Z3ppbGxhLnB5Cj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT0KLS0tIFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvY29tbW9uL25l dC9idWd6aWxsYS9idWd6aWxsYS5weQkocmV2aXNpb24gMjI5MDUyKQorKysgVG9vbHMvU2NyaXB0 cy93ZWJraXRweS9jb21tb24vbmV0L2J1Z3ppbGxhL2J1Z3ppbGxhLnB5CSh3b3JraW5nIGNvcHkp CkBAIC00Niw2ICs0Niw3IEBAIGZyb20gLmJ1ZyBpbXBvcnQgQnVnCiBmcm9tIHdlYmtpdHB5LmNv bW1vbi5jb25maWcgaW1wb3J0IGNvbW1pdHRlcnMKIGltcG9ydCB3ZWJraXRweS5jb21tb24uY29u ZmlnLnVybHMgYXMgY29uZmlnX3VybHMKIGZyb20gd2Via2l0cHkuY29tbW9uLm5ldC5jcmVkZW50 aWFscyBpbXBvcnQgQ3JlZGVudGlhbHMKK2Zyb20gd2Via2l0cHkuY29tbW9uLm5ldC5uZXR3b3Jr dHJhbnNhY3Rpb24gaW1wb3J0IE5ldHdvcmtUcmFuc2FjdGlvbgogZnJvbSB3ZWJraXRweS5jb21t b24uc3lzdGVtLnVzZXIgaW1wb3J0IFVzZXIKIGZyb20gd2Via2l0cHkudGhpcmRwYXJ0eS5CZWF1 dGlmdWxTb3VwIGltcG9ydCBCZWF1dGlmdWxTb3VwLCBCZWF1dGlmdWxTdG9uZVNvdXAsIFNvdXBT dHJhaW5lcgogCkBAIC00NjgsNiArNDY5LDkgQEAgY2xhc3MgQnVnemlsbGEob2JqZWN0KToKICAg ICBkZWYgX2ZldGNoX2J1Z19wYWdlKHNlbGYsIGJ1Z19pZCk6CiAgICAgICAgIGJ1Z191cmwgPSBz ZWxmLmJ1Z191cmxfZm9yX2J1Z19pZChidWdfaWQsIHhtbD1UcnVlKQogICAgICAgICBfbG9nLmlu Zm8oIkZldGNoaW5nOiAlcyIgJSBidWdfdXJsKQorICAgICAgICByZXR1cm4gTmV0d29ya1RyYW5z YWN0aW9uKCkucnVuKGxhbWJkYTogc2VsZi5fZmV0Y2hfYnVnX3BhZ2VfYnlfdXJsKGJ1Z191cmwp KQorCisgICAgZGVmIF9mZXRjaF9idWdfcGFnZV9ieV91cmwoc2VsZiwgYnVnX3VybCk6CiAgICAg ICAgIHJldHVybiBzZWxmLmJyb3dzZXIub3BlbihidWdfdXJsKQogCiAgICAgZGVmIGZldGNoX2J1 Z19kaWN0aW9uYXJ5KHNlbGYsIGJ1Z19pZCk6CkBAIC01MDEsNiArNTA1LDkgQEAgY2xhc3MgQnVn emlsbGEob2JqZWN0KToKICAgICAgICAgcmV0dXJuIGludChtYXRjaC5ncm91cCgnYnVnX2lkJykp CiAKICAgICBkZWYgYnVnX2lkX2Zvcl9hdHRhY2htZW50X2lkKHNlbGYsIGF0dGFjaG1lbnRfaWQp OgorICAgICAgICByZXR1cm4gTmV0d29ya1RyYW5zYWN0aW9uKCkucnVuKGxhbWJkYTogc2VsZi5n ZXRfYnVnX2lkX2Zvcl9hdHRhY2htZW50X2lkKGF0dGFjaG1lbnRfaWQpKQorCisgICAgZGVmIGdl dF9idWdfaWRfZm9yX2F0dGFjaG1lbnRfaWQoc2VsZiwgYXR0YWNobWVudF9pZCk6CiAgICAgICAg IHNlbGYuYXV0aGVudGljYXRlKCkKIAogICAgICAgICBhdHRhY2htZW50X3VybCA9IHNlbGYuYXR0 YWNobWVudF91cmxfZm9yX2lkKGF0dGFjaG1lbnRfaWQsICdlZGl0JykK