| Summary: | [ews] Notify bot watchers about pre-existing and flaky test failures found on ews bots | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | WebKit | Reporter: | Ryan Haddad <ryanhaddad> | ||||||
| Component: | Tools / Tests | Assignee: | Aakash Jain <aakash_jain> | ||||||
| Status: | RESOLVED FIXED | ||||||||
| Severity: | Normal | CC: | aakash_jain, ap, jbedard, ryanhaddad, webkit-bot-watchers-bugzilla, webkit-bug-importer | ||||||
| Priority: | P2 | Keywords: | InRadar | ||||||
| Version: | WebKit Nightly Build | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| See Also: |
https://bugs.webkit.org/show_bug.cgi?id=214727 https://bugs.webkit.org/show_bug.cgi?id=214693 https://bugs.webkit.org/show_bug.cgi?id=214841 https://bugs.webkit.org/show_bug.cgi?id=215274 https://bugs.webkit.org/show_bug.cgi?id=215948 https://bugs.webkit.org/show_bug.cgi?id=215949 |
||||||||
| Bug Depends on: | 214670 | ||||||||
| Bug Blocks: | |||||||||
| Attachments: |
|
||||||||
|
Description
Ryan Haddad
2020-02-10 10:07:15 PST
Created attachment 405040 [details]
Patch
Created attachment 405067 [details]
Patch
Comment on attachment 405067 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=405067&action=review > Tools/BuildSlaveSupport/ews-build/steps.py:1871 > + self.send_email_for_flaky_failure(flaky_failure) Do we have any understanding of how much email this is going to generate? Seems like it would be a ton. Are the recipients prepared for that? > Tools/BuildSlaveSupport/ews-build/steps.py:2006 > + def send_email_for_pre_existing_failure(self, test_name): This seems way more useful then the flaky one at the moment, because there will be far less of them > Tools/BuildSlaveSupport/ews-build/steps_unittest.py:1745 > + def test_flaky_failures_in_first_run(self): What's the point of this test if we aren't testing the email sending in some way? Comment on attachment 405067 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=405067&action=review >> Tools/BuildSlaveSupport/ews-build/steps.py:1871 >> + self.send_email_for_flaky_failure(flaky_failure) > > Do we have any understanding of how much email this is going to generate? > > Seems like it would be a ton. Are the recipients prepared for that? We created a new email address to receive these emails see: https://trac.webkit.org/changeset/264779/webkit Number of emails would vary based on how many flaky tests or pre-existing failures are there on a given day on trunk. We started these emails today, and we got 35 emails (26 for flaky tests and 9 for pre-existing test failures) so far. >> Tools/BuildSlaveSupport/ews-build/steps_unittest.py:1745 >> + def test_flaky_failures_in_first_run(self): > > What's the point of this test if we aren't testing the email sending in some way? This test is independent of this patch. I could have added it in a separate patch. Comment on attachment 405067 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=405067&action=review >>> Tools/BuildSlaveSupport/ews-build/steps.py:1871 >>> + self.send_email_for_flaky_failure(flaky_failure) >> >> Do we have any understanding of how much email this is going to generate? >> >> Seems like it would be a ton. Are the recipients prepared for that? > > We created a new email address to receive these emails see: https://trac.webkit.org/changeset/264779/webkit > > Number of emails would vary based on how many flaky tests or pre-existing failures are there on a given day on trunk. We started these emails today, and we got 35 emails (26 for flaky tests and 9 for pre-existing test failures) so far. Does that mean we already have a version of this deployed? (In reply to Jonathan Bedard from comment #6) > Does that mean we already have a version of this deployed? Yeah Committed r264795: <https://trac.webkit.org/changeset/264795> All reviewed patches have been landed. Closing bug and clearing flags on attachment 405067 [details]. |