When WebKitTest (layout-tests) fail, we re-run them. If some tests fail in re-run as well, we run the tests on clean tree. If there are pre-existing test failures, the build summary is set to something like: 'Found 1 pre-existing test failure: test1.' It would be better to include the flaky test name as well in the build summary, if there were any flaky failures. e.g.: 'Found 1 pre-existing test failure: test1, Found flaky tests: test2, test3'. This would help bot-watchers easily notice flaky tests (and the pattern).
Created attachment 390289 [details] Patch
Sample runs: https://ews-build.webkit-uat.org/#/builders/16/builds/85 https://ews-build.webkit-uat.org/#/builders/16/builds/84
Committed r256212: <https://trac.webkit.org/changeset/256212>
<rdar://problem/59324970>