Bug 204155 – Make webkitperl EWS robust against flakiness in webkitperl tests

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Aakash Jain]

We noticed in Bug 203964 that test262 in webkitperl are flaky. Ideally the flakiness should be fixed.

However, we should also try to make EWS robust against such flakiness. When the webkitperl tests fail, we should run the one more time to confirm that the tests are actually failing, and it's not just a one time flakiness.  

Most of the time these tests pass on re-running. e.g.: In https://ews-build.webkit.org/#/builders/19/builds/13090, webkitperl tests failed, however on retry it passed (in https://ews-build.webkit.org/#/builders/19/builds/13091).

[Aakash Jain]

Created attachment 383454 [details]
Patch

[Aakash Jain]

Sample runs:

Both first and second run failed: https://ews-build.webkit-uat.org/#/builders/17/builds/289
First run failed and second run passed: https://ews-build.webkit-uat.org/#/builders/17/builds/290
First run passed: https://ews-build.webkit-uat.org/#/builders/17/builds/291

[Jonathan Bedard]

Comment on attachment 383454 [details]
Patch

Makes me kinda sad that Perl tests need this, they don't really have a good excuse for being flakey.

[WebKit Commit Bot]

Comment on attachment 383454 [details]
Patch

Clearing flags on attachment: 383454

Committed r252411: <https://trac.webkit.org/changeset/252411>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/57154721>

[Aakash Jain]

Example where this change was helpful in preventing false results (due to flakiness):
https://ews-build.webkit.org/#/builders/19/builds/13668