When WebKitTest (layout-tests) fail, we re-run them. If the re-run pass, we mark the build as successful. The build summary is set to: 'Passed layout tests'. It would be better to indicate the flaky test name in the build summary in this case, so that bot-watchers can easily notice flaky tests (and the pattern).
Created attachment 389364 [details] Patch
This change will help in easily identifying flaky tests from builds like these (in which layout-tests have some test failures but re-run-layout-tests passes): https://ews-build.webkit.org/#/builders/24/builds/9685 https://ews-build.webkit.org/#/builders/24/builds/9686 https://ews-build.webkit.org/#/builders/24/builds/9661 https://ews-build.webkit.org/#/builders/24/builds/9600 https://ews-build.webkit.org/#/builders/24/builds/9588 https://ews-build.webkit.org/#/builders/24/builds/9592
Sample run: https://ews-build.webkit-uat.org/#/builders/16/builds/33
Comment on attachment 389364 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=389364&action=review > Tools/BuildSlaveSupport/ews-build/steps.py:1360 > + flaky_failures_string = ', '.join(flaky_failures) This looks good for now. But, once we have this information, we could cross reference these tests by name against the results database and see if they've failed in the last week. If they haven't, that's a decent indication of a newly flakey test (perhaps caused by the patch) > Tools/BuildSlaveSupport/ews-build/steps.py:-1355 > - message = 'Passed layout tests' Why did we move this out of the if statement.
Comment on attachment 389364 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=389364&action=review >> Tools/BuildSlaveSupport/ews-build/steps.py:1360 >> + flaky_failures_string = ', '.join(flaky_failures) > > This looks good for now. > > But, once we have this information, we could cross reference these tests by name against the results database and see if they've failed in the last week. If they haven't, that's a decent indication of a newly flakey test (perhaps caused by the patch) Agree. Another thing we need to do is to automatically file bugs for these flaky tests. >> Tools/BuildSlaveSupport/ews-build/steps.py:-1355 >> - message = 'Passed layout tests' > > Why did we move this out of the if statement. No strong reason. Moving it back.
Committed r255513: <https://trac.webkit.org/changeset/255513>
<rdar://problem/59069094>
Deployed on production server.