Bug 207050

Summary: [ews] Display flaky test names in build summary when ReRunWebKitTests passes
Product: WebKit Reporter: Aakash Jain <aakash_jain>
Component: Tools / TestsAssignee: Aakash Jain <aakash_jain>
Status: RESOLVED FIXED    
Severity: Normal CC: aakash_jain, ap, jbedard, webkit-bot-watchers-bugzilla, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: Other   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=207504
Attachments:
Description Flags
Patch none

Description Aakash Jain 2020-01-31 09:20:29 PST
When WebKitTest (layout-tests) fail, we re-run them. If the re-run pass, we mark the build as successful. The build summary is set to: 'Passed layout tests'. It would be better to indicate the flaky test name in the build summary in this case, so that bot-watchers can easily notice flaky tests (and the pattern).
Comment 1 Aakash Jain 2020-01-31 09:36:24 PST
Created attachment 389364 [details]
Patch
Comment 3 Aakash Jain 2020-01-31 09:41:29 PST
Sample run: https://ews-build.webkit-uat.org/#/builders/16/builds/33
Comment 4 Jonathan Bedard 2020-01-31 10:03:19 PST
Comment on attachment 389364 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=389364&action=review

> Tools/BuildSlaveSupport/ews-build/steps.py:1360
> +        flaky_failures_string = ', '.join(flaky_failures)

This looks good for now.

But, once we have this information, we could cross reference these tests by name against the results database and see if they've failed in the last week. If they haven't, that's a decent indication of a newly flakey test (perhaps caused by the patch)

> Tools/BuildSlaveSupport/ews-build/steps.py:-1355
> -            message = 'Passed layout tests'

Why did we move this out of the if statement.
Comment 5 Aakash Jain 2020-01-31 11:34:09 PST
Comment on attachment 389364 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=389364&action=review

>> Tools/BuildSlaveSupport/ews-build/steps.py:1360
>> +        flaky_failures_string = ', '.join(flaky_failures)
> 
> This looks good for now.
> 
> But, once we have this information, we could cross reference these tests by name against the results database and see if they've failed in the last week. If they haven't, that's a decent indication of a newly flakey test (perhaps caused by the patch)

Agree. Another thing we need to do is to automatically file bugs for these flaky tests.

>> Tools/BuildSlaveSupport/ews-build/steps.py:-1355
>> -            message = 'Passed layout tests'
> 
> Why did we move this out of the if statement.

No strong reason. Moving it back.
Comment 6 Aakash Jain 2020-01-31 11:38:22 PST
Committed r255513: <https://trac.webkit.org/changeset/255513>
Comment 7 Radar WebKit Bug Importer 2020-01-31 11:39:18 PST
<rdar://problem/59069094>
Comment 8 Aakash Jain 2020-02-01 04:36:01 PST
Deployed on production server.