Bug 210855 – [ews] Display flaky test names in build summary when ReRunJavaScriptCoreTests passes

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Aakash Jain]

When JSC tests (RunJavaScriptCoreTests) fail, we re-run them. If the re-run pass, we mark the build as successful. The build summary is set to: 'Passed JSC tests'. It would be better to indicate the flaky test name in the build summary in this case, so that bot-watchers can easily notice flaky tests (and the pattern).

[Aakash Jain]

stress/array-buffer-view-watchpoint-can-be-fired-in-really-add-in-dfg.js.default was flaky in https://ews-build.webkit.org/#/builders/26/builds/15609 and https://ews-build.webkit.org/#/builders/26/builds/15541

but it's hard to find it since this flakiness information is not shown in build summary and so it doesn't show up on corresponding builder page (https://ews-build.webkit.org/#/builders/26). Showing this info (as build summary for the build) on builder page would make it easy for bot-watchers or someone to easily notice flakiness info and the pattern.

[Aakash Jain]

Created attachment 397951 [details]
Patch

[Jonathan Bedard]

Comment on attachment 397951 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=397951&action=review

> Tools/BuildSlaveSupport/ews-build/steps.py:1514
> +            pluralSuffix = 's' if len(flaky_failures) > 1 else ''

It seems possible that there are no flakey failures, right? Wouldn't we want to keep the 'Passed JSC tests' in that case?

[Aakash Jain]

(In reply to Jonathan Bedard from comment #3)
> It seems possible that there are no flakey failures, right?
This step (ReRunJavaScriptCoreTests) would be run only if there were some failures in first run. Since first run had some failures, and re-run passed, that means the failures were flaky.

[EWS]

Committed r260897: <https://trac.webkit.org/changeset/260897>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 397951 [details].

[Radar WebKit Bug Importer]

<rdar://problem/62601528>

[Aakash Jain]

Restarted EWS server to pick up this change.