Bug 207050 – [ews] Display flaky test names in build summary when ReRunWebKitTests passes

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Aakash Jain]

When WebKitTest (layout-tests) fail, we re-run them. If the re-run pass, we mark the build as successful. The build summary is set to: 'Passed layout tests'. It would be better to indicate the flaky test name in the build summary in this case, so that bot-watchers can easily notice flaky tests (and the pattern).

[Aakash Jain]

Created attachment 389364 [details]
Patch

[Aakash Jain]

This change will help in easily identifying flaky tests from builds like these (in which layout-tests have some test failures but re-run-layout-tests passes):

https://ews-build.webkit.org/#/builders/24/builds/9685
https://ews-build.webkit.org/#/builders/24/builds/9686
https://ews-build.webkit.org/#/builders/24/builds/9661
https://ews-build.webkit.org/#/builders/24/builds/9600
https://ews-build.webkit.org/#/builders/24/builds/9588
https://ews-build.webkit.org/#/builders/24/builds/9592

[Aakash Jain]

Sample run: https://ews-build.webkit-uat.org/#/builders/16/builds/33

[Jonathan Bedard]

Comment on attachment 389364 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=389364&action=review

> Tools/BuildSlaveSupport/ews-build/steps.py:1360
> +        flaky_failures_string = ', '.join(flaky_failures)

This looks good for now.

But, once we have this information, we could cross reference these tests by name against the results database and see if they've failed in the last week. If they haven't, that's a decent indication of a newly flakey test (perhaps caused by the patch)

> Tools/BuildSlaveSupport/ews-build/steps.py:-1355
> -            message = 'Passed layout tests'

Why did we move this out of the if statement.

[Aakash Jain]

Comment on attachment 389364 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=389364&action=review

>> Tools/BuildSlaveSupport/ews-build/steps.py:1360
>> +        flaky_failures_string = ', '.join(flaky_failures)
> 
> This looks good for now.
> 
> But, once we have this information, we could cross reference these tests by name against the results database and see if they've failed in the last week. If they haven't, that's a decent indication of a newly flakey test (perhaps caused by the patch)

Agree. Another thing we need to do is to automatically file bugs for these flaky tests.

>> Tools/BuildSlaveSupport/ews-build/steps.py:-1355
>> -            message = 'Passed layout tests'
> 
> Why did we move this out of the if statement.

No strong reason. Moving it back.

[Aakash Jain]

Committed r255513: <https://trac.webkit.org/changeset/255513>

[Radar WebKit Bug Importer]

<rdar://problem/59069094>

[Aakash Jain]

Deployed on production server.