Bug 207475 – [ews] Notify bot watchers about pre-existing and flaky test failures found on ews bots

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Ryan Haddad]

[ews-build] Create a mechanism to notify interested parties about pre-existing test failures found on bots

[Radar WebKit Bug Importer]

<rdar://problem/59314769>

[Aakash Jain]

Created attachment 405040 [details]
Patch

[Aakash Jain]

Created attachment 405067 [details]
Patch

[Jonathan Bedard]

Comment on attachment 405067 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=405067&action=review

> Tools/BuildSlaveSupport/ews-build/steps.py:1871
> +                self.send_email_for_flaky_failure(flaky_failure)

Do we have any understanding of how much email this is going to generate?

Seems like it would be a ton. Are the recipients prepared for that?

> Tools/BuildSlaveSupport/ews-build/steps.py:2006
> +    def send_email_for_pre_existing_failure(self, test_name):

This seems way more useful then the flaky one at the moment, because there will be far less of them

> Tools/BuildSlaveSupport/ews-build/steps_unittest.py:1745
> +    def test_flaky_failures_in_first_run(self):

What's the point of this test if we aren't testing the email sending in some way?

[Aakash Jain]

Comment on attachment 405067 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=405067&action=review

>> Tools/BuildSlaveSupport/ews-build/steps.py:1871
>> +                self.send_email_for_flaky_failure(flaky_failure)
> 
> Do we have any understanding of how much email this is going to generate?
> 
> Seems like it would be a ton. Are the recipients prepared for that?

We created a new email address to receive these emails see: https://trac.webkit.org/changeset/264779/webkit

Number of emails would vary based on how many flaky tests or pre-existing failures are there on a given day on trunk. We started these emails today, and we got 35 emails (26 for flaky tests and 9 for pre-existing test failures) so far.

>> Tools/BuildSlaveSupport/ews-build/steps_unittest.py:1745
>> +    def test_flaky_failures_in_first_run(self):
> 
> What's the point of this test if we aren't testing the email sending in some way?

This test is independent of this patch. I could have added it in a separate patch.

[Jonathan Bedard]

Comment on attachment 405067 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=405067&action=review

>>> Tools/BuildSlaveSupport/ews-build/steps.py:1871
>>> +                self.send_email_for_flaky_failure(flaky_failure)
>> 
>> Do we have any understanding of how much email this is going to generate?
>> 
>> Seems like it would be a ton. Are the recipients prepared for that?
> 
> We created a new email address to receive these emails see: https://trac.webkit.org/changeset/264779/webkit
> 
> Number of emails would vary based on how many flaky tests or pre-existing failures are there on a given day on trunk. We started these emails today, and we got 35 emails (26 for flaky tests and 9 for pre-existing test failures) so far.

Does that mean we already have a version of this deployed?

[Aakash Jain]

(In reply to Jonathan Bedard from comment #6)
> Does that mean we already have a version of this deployed?
Yeah

[EWS]

Committed r264795: <https://trac.webkit.org/changeset/264795>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 405067 [details].