Add build step to analyze JavaScriptCore tests results, including re-run and clean-tree-run. Determine if the failures were introduced by the patch, were pre-existing.
Created attachment 383502 [details] Patch
Sample runs: JSC stress test failure: https://ews-build.webkit-uat.org/#/builders/21/builds/430 binary and stress test failure: https://ews-build.webkit-uat.org/#/builders/21/builds/442 Flaky tests: https://ews-build.webkit-uat.org/#/builders/21/builds/435 Pre-existing failure: https://ews-build.webkit-uat.org/#/builders/21/builds/457
Comment on attachment 383502 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=383502&action=review > Tools/BuildSlaveSupport/ews-build/steps.py:1081 > + flaky_binary_failures = first_run_binary_failures.union(second_run_binary_failures) - first_run_binary_failures.intersection(second_run_binary_failures) I'm not convinced that we should give a green bubble with flaky binary failures....From talking to Zhifei, I get the sense that the binaries don't have the same sort of flake problems that layout tests or jsc stress tests have.
wait, if we have already submitted results to results database, why we need to rerun it without patch ?
(In reply to Zhifei Fang from comment #4) > wait, if we have already submitted results to results database, why we need > to rerun it without patch ? We don't have EWS plugged in to the results database yet. We totally could now, though. Ling recently sorted out the network access.
Comment on attachment 383502 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=383502&action=review >> Tools/BuildSlaveSupport/ews-build/steps.py:1081 >> + flaky_binary_failures = first_run_binary_failures.union(second_run_binary_failures) - first_run_binary_failures.intersection(second_run_binary_failures) > > I'm not convinced that we should give a green bubble with flaky binary failures....From talking to Zhifei, I get the sense that the binaries don't have the same sort of flake problems that layout tests or jsc stress tests have. Aakash, Zhifei and I talked about this a bit offline. Binary tests probably aren't flakey, but this is what old EWS did and we don't actually have concrete data on if test binaries are flakey without https://bugs.webkit.org/show_bug.cgi?id=204091. We might come back and revisit this, but for the sake of converting JSC EWS to new EWS, this behavior is the same as what we were doing before.
Comment on attachment 383502 [details] Patch Clearing flags on attachment: 383502 Committed r252446: <https://trac.webkit.org/changeset/252446>
All reviewed patches have been landed. Closing bug.
<rdar://problem/57177809>