Bug 210855 - [ews] Display flaky test names in build summary when ReRunJavaScriptCoreTests passes
Summary: [ews] Display flaky test names in build summary when ReRunJavaScriptCoreTests...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Aakash Jain
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2020-04-22 05:27 PDT by Aakash Jain
Modified: 2020-04-29 09:30 PDT (History)
4 users (show)

See Also:


Attachments
Patch (4.59 KB, patch)
2020-04-29 03:18 PDT, Aakash Jain
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Aakash Jain 2020-04-22 05:27:54 PDT
When JSC tests (RunJavaScriptCoreTests) fail, we re-run them. If the re-run pass, we mark the build as successful. The build summary is set to: 'Passed JSC tests'. It would be better to indicate the flaky test name in the build summary in this case, so that bot-watchers can easily notice flaky tests (and the pattern).
Comment 1 Aakash Jain 2020-04-29 03:12:14 PDT
stress/array-buffer-view-watchpoint-can-be-fired-in-really-add-in-dfg.js.default was flaky in https://ews-build.webkit.org/#/builders/26/builds/15609 and https://ews-build.webkit.org/#/builders/26/builds/15541

but it's hard to find it since this flakiness information is not shown in build summary and so it doesn't show up on corresponding builder page (https://ews-build.webkit.org/#/builders/26). Showing this info (as build summary for the build) on builder page would make it easy for bot-watchers or someone to easily notice flakiness info and the pattern.
Comment 2 Aakash Jain 2020-04-29 03:18:12 PDT
Created attachment 397951 [details]
Patch
Comment 3 Jonathan Bedard 2020-04-29 07:00:26 PDT
Comment on attachment 397951 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=397951&action=review

> Tools/BuildSlaveSupport/ews-build/steps.py:1514
> +            pluralSuffix = 's' if len(flaky_failures) > 1 else ''

It seems possible that there are no flakey failures, right? Wouldn't we want to keep the 'Passed JSC tests' in that case?
Comment 4 Aakash Jain 2020-04-29 07:11:57 PDT
(In reply to Jonathan Bedard from comment #3)
> It seems possible that there are no flakey failures, right?
This step (ReRunJavaScriptCoreTests) would be run only if there were some failures in first run. Since first run had some failures, and re-run passed, that means the failures were flaky.
Comment 5 EWS 2020-04-29 08:25:12 PDT
Committed r260897: <https://trac.webkit.org/changeset/260897>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 397951 [details].
Comment 6 Radar WebKit Bug Importer 2020-04-29 08:26:13 PDT
<rdar://problem/62601528>
Comment 7 Aakash Jain 2020-04-29 09:30:42 PDT
Restarted EWS server to pick up this change.