[ews-build] Create a mechanism to notify interested parties about pre-existing test failures found on bots
<rdar://problem/59314769>
Created attachment 405040 [details] Patch
Created attachment 405067 [details] Patch
Comment on attachment 405067 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=405067&action=review > Tools/BuildSlaveSupport/ews-build/steps.py:1871 > + self.send_email_for_flaky_failure(flaky_failure) Do we have any understanding of how much email this is going to generate? Seems like it would be a ton. Are the recipients prepared for that? > Tools/BuildSlaveSupport/ews-build/steps.py:2006 > + def send_email_for_pre_existing_failure(self, test_name): This seems way more useful then the flaky one at the moment, because there will be far less of them > Tools/BuildSlaveSupport/ews-build/steps_unittest.py:1745 > + def test_flaky_failures_in_first_run(self): What's the point of this test if we aren't testing the email sending in some way?
Comment on attachment 405067 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=405067&action=review >> Tools/BuildSlaveSupport/ews-build/steps.py:1871 >> + self.send_email_for_flaky_failure(flaky_failure) > > Do we have any understanding of how much email this is going to generate? > > Seems like it would be a ton. Are the recipients prepared for that? We created a new email address to receive these emails see: https://trac.webkit.org/changeset/264779/webkit Number of emails would vary based on how many flaky tests or pre-existing failures are there on a given day on trunk. We started these emails today, and we got 35 emails (26 for flaky tests and 9 for pre-existing test failures) so far. >> Tools/BuildSlaveSupport/ews-build/steps_unittest.py:1745 >> + def test_flaky_failures_in_first_run(self): > > What's the point of this test if we aren't testing the email sending in some way? This test is independent of this patch. I could have added it in a separate patch.
Comment on attachment 405067 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=405067&action=review >>> Tools/BuildSlaveSupport/ews-build/steps.py:1871 >>> + self.send_email_for_flaky_failure(flaky_failure) >> >> Do we have any understanding of how much email this is going to generate? >> >> Seems like it would be a ton. Are the recipients prepared for that? > > We created a new email address to receive these emails see: https://trac.webkit.org/changeset/264779/webkit > > Number of emails would vary based on how many flaky tests or pre-existing failures are there on a given day on trunk. We started these emails today, and we got 35 emails (26 for flaky tests and 9 for pre-existing test failures) so far. Does that mean we already have a version of this deployed?
(In reply to Jonathan Bedard from comment #6) > Does that mean we already have a version of this deployed? Yeah
Committed r264795: <https://trac.webkit.org/changeset/264795> All reviewed patches have been landed. Closing bug and clearing flags on attachment 405067 [details].