Bug 210855

Summary: [ews] Display flaky test names in build summary when ReRunJavaScriptCoreTests passes
Product: WebKit Reporter: Aakash Jain <aakash_jain>
Component: Tools / TestsAssignee: Aakash Jain <aakash_jain>
Status: RESOLVED FIXED    
Severity: Normal CC: ap, jbedard, webkit-bot-watchers-bugzilla, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=207504
Attachments:
Description Flags
Patch none

Description Aakash Jain 2020-04-22 05:27:54 PDT
When JSC tests (RunJavaScriptCoreTests) fail, we re-run them. If the re-run pass, we mark the build as successful. The build summary is set to: 'Passed JSC tests'. It would be better to indicate the flaky test name in the build summary in this case, so that bot-watchers can easily notice flaky tests (and the pattern).
Comment 1 Aakash Jain 2020-04-29 03:12:14 PDT
stress/array-buffer-view-watchpoint-can-be-fired-in-really-add-in-dfg.js.default was flaky in https://ews-build.webkit.org/#/builders/26/builds/15609 and https://ews-build.webkit.org/#/builders/26/builds/15541

but it's hard to find it since this flakiness information is not shown in build summary and so it doesn't show up on corresponding builder page (https://ews-build.webkit.org/#/builders/26). Showing this info (as build summary for the build) on builder page would make it easy for bot-watchers or someone to easily notice flakiness info and the pattern.
Comment 2 Aakash Jain 2020-04-29 03:18:12 PDT
Created attachment 397951 [details]
Patch
Comment 3 Jonathan Bedard 2020-04-29 07:00:26 PDT
Comment on attachment 397951 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=397951&action=review

> Tools/BuildSlaveSupport/ews-build/steps.py:1514
> +            pluralSuffix = 's' if len(flaky_failures) > 1 else ''

It seems possible that there are no flakey failures, right? Wouldn't we want to keep the 'Passed JSC tests' in that case?
Comment 4 Aakash Jain 2020-04-29 07:11:57 PDT
(In reply to Jonathan Bedard from comment #3)
> It seems possible that there are no flakey failures, right?
This step (ReRunJavaScriptCoreTests) would be run only if there were some failures in first run. Since first run had some failures, and re-run passed, that means the failures were flaky.
Comment 5 EWS 2020-04-29 08:25:12 PDT
Committed r260897: <https://trac.webkit.org/changeset/260897>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 397951 [details].
Comment 6 Radar WebKit Bug Importer 2020-04-29 08:26:13 PDT
<rdar://problem/62601528>
Comment 7 Aakash Jain 2020-04-29 09:30:42 PDT
Restarted EWS server to pick up this change.