Bug 199525 - [ews-build] Display pre-existing API test names in the build summary
Summary: [ews-build] Display pre-existing API test names in the build summary
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: Other
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Aakash Jain
Keywords: InRadar
Depends on:
Reported: 2019-07-05 12:15 PDT by Aakash Jain
Modified: 2019-10-31 12:57 PDT (History)
7 users (show)

See Also:

Patch (2.14 KB, patch)
2019-07-05 12:17 PDT, Aakash Jain
no flags Details | Formatted Diff | Diff
Patch (2.38 KB, patch)
2019-10-31 11:39 PDT, Aakash Jain
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Aakash Jain 2019-07-05 12:15:06 PDT
Currently when there is a failure on clean tree (ToT), API test build summary shows: "Found x pre-existing API test failure". This is fine, but not very informative. It would be better to include the names of the pre-existing failures in the build summary string. Those failing test names would show up on the builder page, and it would be very convenient to notice the test failure pattern over multiple builds.
Comment 1 Aakash Jain 2019-07-05 12:17:04 PDT
Created attachment 373522 [details]
Comment 3 Jonathan Bedard 2019-07-08 09:02:00 PDT
Comment on attachment 373522 [details]

View in context: https://bugs.webkit.org/attachment.cgi?id=373522&action=review

> Tools/BuildSlaveSupport/ews-build/steps.py:1088
> +        clean_tree_failures_string = ', '.join([failure_name.replace('TestWebKitAPI.', '') for failure_name in clean_tree_failures])

What we call 'API' tests is actually 2 different binaries on Mac, 3 for Windows (at the moment, at least). That first key in the test name is the binary. It could be TestWebKitAPI, TestWTF, TestWTF.exe, TestWebCore.exe or TestWebKitLegacy.exe. I'm not sure it makes sense to strip the binary name at all, run-webkit-tests is smart enough to understand it. If we do want to string the binary name, though, we need to do it with split('.').
Comment 4 Aakash Jain 2019-10-31 11:39:33 PDT
Created attachment 382483 [details]
Comment 5 Aakash Jain 2019-10-31 11:42:31 PDT
Not removing 'TestWebKitAPI.' string anymore.

Also updated the patch to also limit the number of failures to display to 10 (although it should be rare to have more than 10 failures on trunk).

This change would not affect the status-bubble tooltip.

Sample runs:
Comment 6 WebKit Commit Bot 2019-10-31 12:56:40 PDT
Comment on attachment 382483 [details]

Clearing flags on attachment: 382483

Committed r251869: <https://trac.webkit.org/changeset/251869>
Comment 7 WebKit Commit Bot 2019-10-31 12:56:42 PDT
All reviewed patches have been landed.  Closing bug.
Comment 8 Radar WebKit Bug Importer 2019-10-31 12:57:35 PDT