Bug 203246

Summary:

EWS bots failing with to reach amazon aws

Product:

WebKit

Reporter:

Aakash Jain <aakash_jain>

Component:

Tools / Tests

Assignee:

Aakash Jain <aakash_jain>

Status:

RESOLVED FIXED

Severity:

Normal

CC:

aakash_jain, ap, jbedard, lingcherd_ho, webkit-bot-watchers-bugzilla

Priority:

Version:

Other

Hardware:

Unspecified

OS:

Unspecified

See Also:

https://bugs.webkit.org/show_bug.cgi?id=203263

Attachments:

Description	Flags
Screenshot - Amazon DNS Resolution Error	none
Screenshot - [RESOLVED] Intermittent DNS Resolution Errors	none

Description Aakash Jain 2019-10-22 08:02:35 PDT

Many EWS bots are failing with to reach amazon aws, with the error: curl: (6) Could not resolve host: s3-us-west-2.amazonaws.com

https://ews-build.webkit.org/#/builders/9/builds/10556/steps/6/logs/stdio
https://ews-build.webkit.org/#/builders/3/builds/10644
https://ews-build.webkit.org/#/builders/14/builds/5202
https://ews-build.webkit.org/#/builders/15/builds/5699
https://ews-build.webkit.org/#/builders/17/builds/4471

Comment 1 Aakash Jain 2019-10-22 08:06:12 PDT

This issue seems to have started this morning (~3 hours ago).

First noticed failure in: https://ews-build.webkit.org/#/builders/24/builds/2224

Comment 2 Ling Ho 2019-10-22 09:31:45 PDT

DNS service Looks ok.

ews116:~ buildbot$ host s3-us-west-2.amazonaws.com
s3-us-west-2.amazonaws.com has address 52.218.229.192

Comment 3 Alexey Proskuryakov 2019-10-22 12:44:04 PDT

Amazon status dashboard is now updated with DNS resolution flakiness: https://status.aws.amazon.com

Aakash is looking into whether switching to a different address will help.

Comment 4 Aakash Jain 2019-10-22 14:28:06 PDT

Created attachment 381607 [details]
Screenshot - Amazon DNS Resolution Error

Comment 5 Aakash Jain 2019-10-22 14:30:57 PDT

The issues are intermittent, and due to DNS Resolution Errors on Amazon side.

I have added a fallback option (to download build archive from build master) in https://bugs.webkit.org/show_bug.cgi?id=203263

Comment 6 Aakash Jain 2019-10-22 21:38:16 PDT

As per https://status.aws.amazon.com:

[RESOLVED] Intermittent DNS Resolution Errors
Between 10:30 AM and 6:30 PM PDT, we experienced intermittent errors with resolution of some AWS DNS names. Beginning at 5:16 PM, a very small number of specific DNS names experienced a higher error rate. These issues have been resolved.

Comment 7 Aakash Jain 2019-10-22 21:46:29 PDT

Created attachment 381652 [details]
Screenshot - [RESOLVED] Intermittent DNS Resolution Errors