Bug 203246

Summary:

EWS bots failing with to reach amazon aws

Product:

WebKit

Reporter:

Aakash Jain <aakash_jain>

Component:

Tools / Tests

Assignee:

Aakash Jain <aakash_jain>

Status:

RESOLVED FIXED

Severity:

Normal

CC:

aakash_jain, ap, jbedard, lingcherd_ho, webkit-bot-watchers-bugzilla

Priority:

Version:

Other

Hardware:

Unspecified

OS:

Unspecified

See Also:

https://bugs.webkit.org/show_bug.cgi?id=203263

Attachments:

Description	Flags
Screenshot - Amazon DNS Resolution Error	none
Screenshot - [RESOLVED] Intermittent DNS Resolution Errors	none

Aakash Jain

Reported 2019-10-22 08:02:35 PDT

Many EWS bots are failing with to reach amazon aws, with the error: curl: (6) Could not resolve host: s3-us-west-2.amazonaws.com https://ews-build.webkit.org/#/builders/9/builds/10556/steps/6/logs/stdio https://ews-build.webkit.org/#/builders/3/builds/10644 https://ews-build.webkit.org/#/builders/14/builds/5202 https://ews-build.webkit.org/#/builders/15/builds/5699 https://ews-build.webkit.org/#/builders/17/builds/4471

Attachments
Screenshot - Amazon DNS Resolution Error (96.86 KB, image/png) 2019-10-22 14:28 PDT, Aakash Jain	no flags	Details
Screenshot - [RESOLVED] Intermittent DNS Resolution Errors (113.14 KB, image/png) 2019-10-22 21:46 PDT, Aakash Jain	no flags	Details
View All Add attachment proposed patch, testcase, etc.

Aakash Jain

Comment 1 2019-10-22 08:06:12 PDT

This issue seems to have started this morning (~3 hours ago). First noticed failure in: https://ews-build.webkit.org/#/builders/24/builds/2224

Ling Ho

Comment 2 2019-10-22 09:31:45 PDT

DNS service Looks ok. ews116:~ buildbot$ host s3-us-west-2.amazonaws.com s3-us-west-2.amazonaws.com has address 52.218.229.192

Alexey Proskuryakov

Comment 3 2019-10-22 12:44:04 PDT

Amazon status dashboard is now updated with DNS resolution flakiness: https://status.aws.amazon.com Aakash is looking into whether switching to a different address will help.

Aakash Jain

Comment 4 2019-10-22 14:28:06 PDT

Created attachment 381607 [details] Screenshot - Amazon DNS Resolution Error

Aakash Jain

Comment 5 2019-10-22 14:30:57 PDT

The issues are intermittent, and due to DNS Resolution Errors on Amazon side. I have added a fallback option (to download build archive from build master) in https://bugs.webkit.org/show_bug.cgi?id=203263

Aakash Jain

Comment 6 2019-10-22 21:38:16 PDT

As per https://status.aws.amazon.com: [RESOLVED] Intermittent DNS Resolution Errors Between 10:30 AM and 6:30 PM PDT, we experienced intermittent errors with resolution of some AWS DNS names. Beginning at 5:16 PM, a very small number of specific DNS names experienced a higher error rate. These issues have been resolved.

Aakash Jain

Comment 7 2019-10-22 21:46:29 PDT

Created attachment 381652 [details] Screenshot - [RESOLVED] Intermittent DNS Resolution Errors

Note You need to log in before you can comment on or make changes to this bug.