Bug 204155 - Make webkitperl EWS robust against flakiness in webkitperl tests
Summary: Make webkitperl EWS robust against flakiness in webkitperl tests
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: Other
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Aakash Jain
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2019-11-13 07:02 PST by Aakash Jain
Modified: 2019-11-19 12:28 PST (History)
6 users (show)

See Also:


Attachments
Patch (4.80 KB, patch)
2019-11-13 07:06 PST, Aakash Jain
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Aakash Jain 2019-11-13 07:02:55 PST
We noticed in Bug 203964 that test262 in webkitperl are flaky. Ideally the flakiness should be fixed.

However, we should also try to make EWS robust against such flakiness. When the webkitperl tests fail, we should run the one more time to confirm that the tests are actually failing, and it's not just a one time flakiness.  

Most of the time these tests pass on re-running. e.g.: In https://ews-build.webkit.org/#/builders/19/builds/13090, webkitperl tests failed, however on retry it passed (in https://ews-build.webkit.org/#/builders/19/builds/13091).
Comment 1 Aakash Jain 2019-11-13 07:06:55 PST
Created attachment 383454 [details]
Patch
Comment 2 Aakash Jain 2019-11-13 07:08:59 PST
Sample runs:

Both first and second run failed: https://ews-build.webkit-uat.org/#/builders/17/builds/289
First run failed and second run passed: https://ews-build.webkit-uat.org/#/builders/17/builds/290
First run passed: https://ews-build.webkit-uat.org/#/builders/17/builds/291
Comment 3 Jonathan Bedard 2019-11-13 07:37:11 PST
Comment on attachment 383454 [details]
Patch

Makes me kinda sad that Perl tests need this, they don't really have a good excuse for being flakey.
Comment 4 WebKit Commit Bot 2019-11-13 08:22:02 PST
Comment on attachment 383454 [details]
Patch

Clearing flags on attachment: 383454

Committed r252411: <https://trac.webkit.org/changeset/252411>
Comment 5 WebKit Commit Bot 2019-11-13 08:22:03 PST
All reviewed patches have been landed.  Closing bug.
Comment 6 Radar WebKit Bug Importer 2019-11-13 08:23:16 PST
<rdar://problem/57154721>
Comment 7 Aakash Jain 2019-11-19 12:27:54 PST
Example where this change was helpful in preventing false results (due to flakiness):
https://ews-build.webkit.org/#/builders/19/builds/13668