182987 2018-02-20 15:50:10 -0800 webkitpy NetworkTransaction should retry on URLError 2018-02-26 19:20:45 -0800 1 1 1 Unclassified WebKit Tools / Tests Other Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=182420 https://bugs.webkit.org/show_bug.cgi?id=183156 InRadar P1 Normal --- 1 aakash_jain aakash_jain aakash_jain ap commit-queue ews-watchlist glenn jbedard lforschler webkit-bug-importer oldest_to_newest 1400930 0 aakash_jain 2018-02-20 15:50:10 -0800 We have been seeing frequent network errors recently in https://bugs.webkit.org/show_bug.cgi?id=182420. We are debugging the root-cause to fix the underlying network issue in <rdar://problem/37716391>. However, our code should be more robust against network failures and it should retry when it encounters network issue like: URLError: <urlopen error [Errno 60] Operation timed out>. 1400933 1 334310 aakash_jain 2018-02-20 15:53:55 -0800 Created attachment 334310 Proposed patch 1400942 2 aakash_jain 2018-02-20 16:06:53 -0800 This will add retry for webkit-queues network transactions, since they use NetworkTransaction class. This wouldn't fix network issues with Bugzilla as Bugzilla code use Mechanize directly. 1401113 3 jbedard 2018-02-21 08:29:00 -0800 Unofficial R+. 1401171 4 334310 ap 2018-02-21 11:03:15 -0800 Comment on attachment 334310 Proposed patch View in context: https://bugs.webkit.org/attachment.cgi?id=334310&action=review Seems fine given that e.filename doesn't appear to work. But it would be good to fix printing the URL both in this case and in the existing HTTPError case. > Tools/ChangeLog:3 > + webkitpy NetworkTransaction should retry on URLError Is there any possibility of deadlocking here? 1401179 5 334310 commit-queue 2018-02-21 11:27:56 -0800 Comment on attachment 334310 Proposed patch Clearing flags on attachment: 334310 Committed r228885: <https://trac.webkit.org/changeset/228885> 1401180 6 commit-queue 2018-02-21 11:27:58 -0800 All reviewed patches have been landed. Closing bug. 1401181 7 webkit-bug-importer 2018-02-21 11:28:20 -0800 <rdar://problem/37754241> 1401224 8 aakash_jain 2018-02-21 12:38:29 -0800 > Is there any possibility of deadlocking here? This class has a maximum Timeout of 10 minutes (timeout_seconds=(10 * 60)), so it wouldn't be stuck indefinitely retrying on network errors. 1401792 9 aakash_jain 2018-02-23 12:57:59 -0800 This patch has definitely helped with network issues. Noticing the logs, all the patches below would have failed ews with network exceptions, which now worked properly. e.g.: ews109: 2018-02-21 12:55:17,769 - Started processing patch 2018-02-21 12:56:32,778 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-21 12:56:43,353 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334405&action=edit 2018-02-21 12:56:43,628 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183013&ctype=xml&excludefield=attachmentdata 2018-02-21 12:56:43,891 - Running: webkit-patch --status-host=webkit-queues.webkit.org --bot-id=ews109 clean --port=ios-device --architecture=arm64 2018-02-21 16:22:46,539 - Applied patch 2018-02-21 16:24:01,662 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-21 16:24:12,036 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334424&action=edit 2018-02-21 16:24:12,322 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=182891&ctype=xml&excludefield=attachmentdata 2018-02-21 12:55:17,769 - Started processing patch 2018-02-21 12:56:32,778 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-21 12:56:43,353 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334405&action=edit 2018-02-21 12:56:43,628 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183013&ctype=xml&excludefield=attachmentdata ews108: 2018-02-23 06:57:01,397 - Built patch 2018-02-23 06:58:18,323 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-23 06:59:43,470 - Received URLError: [Errno 60] Operation timed out. Retrying in 15.0 seconds... 2018-02-23 06:59:58,908 - Pass 2018-02-23 07:40:12,103 - Cleaned working directory 2018-02-23 07:41:27,849 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds... 2018-02-23 07:41:39,461 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334533&action=edit 2018-02-23 07:41:39,752 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183027&ctype=xml&excludefield=attachmentdata 334310 2018-02-20 15:53:55 -0800 2018-02-21 11:27:56 -0800 Proposed patch webkitpy_retry_on_timeout.patch text/plain 1517 aakash_jain SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDIyODg0OSkKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDEzIEBACisyMDE4LTAyLTIwICBBYWthc2ggSmFpbiAgPGFha2FzaF9qYWluQGFwcGxlLmNv bT4KKworICAgICAgICB3ZWJraXRweSBOZXR3b3JrVHJhbnNhY3Rpb24gc2hvdWxkIHJldHJ5IG9u IFVSTEVycm9yCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9p ZD0xODI5ODcKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAg ICAqIFNjcmlwdHMvd2Via2l0cHkvY29tbW9uL25ldC9uZXR3b3JrdHJhbnNhY3Rpb24ucHk6Cisg ICAgICAgIChOZXR3b3JrVHJhbnNhY3Rpb24ucnVuKTogUmV0cnkgb24gdXJsbGliMi5VUkxFcnJv ci4KKwogMjAxOC0wMi0yMCAgTmFuIFdhbmcgIDxuX3dhbmdAYXBwbGUuY29tPgogCiAgICAgICAg IEFYOiBBT006IERpc3BhdGNoIGFjY2Vzc2libGVjbGljayBldmVudApJbmRleDogVG9vbHMvU2Ny aXB0cy93ZWJraXRweS9jb21tb24vbmV0L25ldHdvcmt0cmFuc2FjdGlvbi5weQo9PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 Ci0tLSBUb29scy9TY3JpcHRzL3dlYmtpdHB5L2NvbW1vbi9uZXQvbmV0d29ya3RyYW5zYWN0aW9u LnB5CShyZXZpc2lvbiAyMjg4NDkpCisrKyBUb29scy9TY3JpcHRzL3dlYmtpdHB5L2NvbW1vbi9u ZXQvbmV0d29ya3RyYW5zYWN0aW9uLnB5CSh3b3JraW5nIGNvcHkpCkBAIC01Nyw2ICs1NywxMCBA QCBjbGFzcyBOZXR3b3JrVHJhbnNhY3Rpb24ob2JqZWN0KToKICAgICAgICAgICAgICAgICBzZWxm Ll9jaGVja19mb3JfdGltZW91dCgpCiAgICAgICAgICAgICAgICAgX2xvZy53YXJuKCJSZWNlaXZl ZCBIVFRQIHN0YXR1cyAlcyBsb2FkaW5nIFwiJXNcIi4gIFJldHJ5aW5nIGluICVzIHNlY29uZHMu Li4iICUgKGUuY29kZSwgZS5maWxlbmFtZSwgc2VsZi5fYmFja29mZl9zZWNvbmRzKSkKICAgICAg ICAgICAgICAgICBzZWxmLl9zbGVlcCgpCisgICAgICAgICAgICBleGNlcHQgdXJsbGliMi5VUkxF cnJvciBhcyBlOgorICAgICAgICAgICAgICAgIHNlbGYuX2NoZWNrX2Zvcl90aW1lb3V0KCkKKyAg ICAgICAgICAgICAgICBfbG9nLndhcm4oIlJlY2VpdmVkIFVSTEVycm9yOiB7fS4gUmV0cnlpbmcg aW4ge30gc2Vjb25kcy4uLiIuZm9ybWF0KGUucmVhc29uLCBzZWxmLl9iYWNrb2ZmX3NlY29uZHMp KQorICAgICAgICAgICAgICAgIHNlbGYuX3NsZWVwKCkKIAogICAgIGRlZiBfY2hlY2tfZm9yX3Rp bWVvdXQoc2VsZik6CiAgICAgICAgIGlmIHNlbGYuX3RvdGFsX3NsZWVwICsgc2VsZi5fYmFja29m Zl9zZWNvbmRzID4gc2VsZi5fdGltZW91dF9zZWNvbmRzOgo=