Bug 182987 – webkitpy NetworkTransaction should retry on URLError

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Aakash Jain]

We have been seeing frequent network errors recently in https://bugs.webkit.org/show_bug.cgi?id=182420. We are debugging the root-cause to fix the underlying network issue in <rdar://problem/37716391>.

However, our code should be more robust against network failures and it should retry when it encounters network issue like: URLError: <urlopen error [Errno 60] Operation timed out>.

[Aakash Jain]

Created attachment 334310 [details]
Proposed patch

[Aakash Jain]

This will add retry for webkit-queues network transactions, since they use NetworkTransaction class. This wouldn't fix network issues with Bugzilla as Bugzilla code use Mechanize directly.

[Jonathan Bedard]

Unofficial R+.

[Alexey Proskuryakov]

Comment on attachment 334310 [details]
Proposed patch

View in context: https://bugs.webkit.org/attachment.cgi?id=334310&action=review

Seems fine given that e.filename doesn't appear to work. But it would be good to fix printing the URL both in this case and in the existing HTTPError case.

> Tools/ChangeLog:3
> +        webkitpy NetworkTransaction should retry on URLError

Is there any possibility of deadlocking here?

[WebKit Commit Bot]

Comment on attachment 334310 [details]
Proposed patch

Clearing flags on attachment: 334310

Committed r228885: <https://trac.webkit.org/changeset/228885>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/37754241>

[Aakash Jain]

> Is there any possibility of deadlocking here?
This class has a maximum Timeout of 10 minutes (timeout_seconds=(10 * 60)), so it wouldn't be stuck indefinitely retrying on network errors.

[Aakash Jain]

This patch has definitely helped with network issues. Noticing the logs, all the patches below would have failed ews with network exceptions, which now worked properly.

e.g.:
ews109:
2018-02-21 12:55:17,769 - Started processing patch
2018-02-21 12:56:32,778 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds...
2018-02-21 12:56:43,353 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334405&action=edit
2018-02-21 12:56:43,628 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183013&ctype=xml&excludefield=attachmentdata
2018-02-21 12:56:43,891 - Running: webkit-patch --status-host=webkit-queues.webkit.org --bot-id=ews109 clean --port=ios-device --architecture=arm64


2018-02-21 16:22:46,539 - Applied patch
2018-02-21 16:24:01,662 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds...
2018-02-21 16:24:12,036 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334424&action=edit
2018-02-21 16:24:12,322 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=182891&ctype=xml&excludefield=attachmentdata

2018-02-21 12:55:17,769 - Started processing patch
2018-02-21 12:56:32,778 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds...
2018-02-21 12:56:43,353 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334405&action=edit
2018-02-21 12:56:43,628 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183013&ctype=xml&excludefield=attachmentdata

ews108:

2018-02-23 06:57:01,397 - Built patch
2018-02-23 06:58:18,323 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds...
2018-02-23 06:59:43,470 - Received URLError: [Errno 60] Operation timed out. Retrying in 15.0 seconds...
2018-02-23 06:59:58,908 - Pass


2018-02-23 07:40:12,103 - Cleaned working directory
2018-02-23 07:41:27,849 - Received URLError: [Errno 60] Operation timed out. Retrying in 10 seconds...
2018-02-23 07:41:39,461 - Fetching: https://bugs.webkit.org/attachment.cgi?id=334533&action=edit
2018-02-23 07:41:39,752 - Fetching: https://bugs.webkit.org/show_bug.cgi?id=183027&ctype=xml&excludefield=attachmentdata