Bug 199525 – [ews-build] Display pre-existing API test names in the build summary

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Aakash Jain]

Currently when there is a failure on clean tree (ToT), API test build summary shows: "Found x pre-existing API test failure". This is fine, but not very informative. It would be better to include the names of the pre-existing failures in the build summary string. Those failing test names would show up on the builder page, and it would be very convenient to notice the test failure pattern over multiple builds.

[Aakash Jain]

Created attachment 373522 [details]
Patch

[Aakash Jain]

Sample runs:
https://ews-build.webkit.org/#/builders/9/builds/4064
https://ews-build.webkit.org/#/builders/3/builds/4179

Also notice the build statuses on builder page: https://ews-build.webkit.org/#/builders/3

[Jonathan Bedard]

Comment on attachment 373522 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=373522&action=review

> Tools/BuildSlaveSupport/ews-build/steps.py:1088
> +        clean_tree_failures_string = ', '.join([failure_name.replace('TestWebKitAPI.', '') for failure_name in clean_tree_failures])

What we call 'API' tests is actually 2 different binaries on Mac, 3 for Windows (at the moment, at least). That first key in the test name is the binary. It could be TestWebKitAPI, TestWTF, TestWTF.exe, TestWebCore.exe or TestWebKitLegacy.exe. I'm not sure it makes sense to strip the binary name at all, run-webkit-tests is smart enough to understand it. If we do want to string the binary name, though, we need to do it with split('.').

[Aakash Jain]

Created attachment 382483 [details]
Patch

[Aakash Jain]

Not removing 'TestWebKitAPI.' string anymore.

Also updated the patch to also limit the number of failures to display to 10 (although it should be rare to have more than 10 failures on trunk).

This change would not affect the status-bubble tooltip.

Sample runs:
https://ews-build.webkit-uat.org/#/builders/4/builds/67
https://ews-build.webkit-uat.org/#/builders/4/builds/66

[WebKit Commit Bot]

Comment on attachment 382483 [details]
Patch

Clearing flags on attachment: 382483

Committed r251869: <https://trac.webkit.org/changeset/251869>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/56791844>