204155 2019-11-13 07:02:55 -0800 Make webkitperl EWS robust against flakiness in webkitperl tests 2019-11-19 12:28:13 -0800 1 1 1 Unclassified WebKit Tools / Tests Other Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=203964 InRadar P2 Normal --- 1 aakash_jain aakash_jain aakash_jain ap commit-queue jbedard webkit-bot-watchers-bugzilla webkit-bug-importer oldest_to_newest 1589848 0 aakash_jain 2019-11-13 07:02:55 -0800 We noticed in Bug 203964 that test262 in webkitperl are flaky. Ideally the flakiness should be fixed. However, we should also try to make EWS robust against such flakiness. When the webkitperl tests fail, we should run the one more time to confirm that the tests are actually failing, and it's not just a one time flakiness. Most of the time these tests pass on re-running. e.g.: In https://ews-build.webkit.org/#/builders/19/builds/13090, webkitperl tests failed, however on retry it passed (in https://ews-build.webkit.org/#/builders/19/builds/13091). 1589849 1 383454 aakash_jain 2019-11-13 07:06:55 -0800 Created attachment 383454 Patch 1589850 2 aakash_jain 2019-11-13 07:08:59 -0800 Sample runs: Both first and second run failed: https://ews-build.webkit-uat.org/#/builders/17/builds/289 First run failed and second run passed: https://ews-build.webkit-uat.org/#/builders/17/builds/290 First run passed: https://ews-build.webkit-uat.org/#/builders/17/builds/291 1589865 3 383454 jbedard 2019-11-13 07:37:11 -0800 Comment on attachment 383454 Patch Makes me kinda sad that Perl tests need this, they don't really have a good excuse for being flakey. 1589879 4 383454 commit-queue 2019-11-13 08:22:02 -0800 Comment on attachment 383454 Patch Clearing flags on attachment: 383454 Committed r252411: <https://trac.webkit.org/changeset/252411> 1589880 5 commit-queue 2019-11-13 08:22:03 -0800 All reviewed patches have been landed. Closing bug. 1589881 6 webkit-bug-importer 2019-11-13 08:23:16 -0800 <rdar://problem/57154721> 1591645 7 aakash_jain 2019-11-19 12:27:54 -0800 Example where this change was helpful in preventing false results (due to flakiness): https://ews-build.webkit.org/#/builders/19/builds/13668 383454 2019-11-13 07:06:55 -0800 2019-11-13 08:22:02 -0800 Patch bug-204155-20191113100654.patch text/plain 4920 aakash_jain SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDI1MjQwOCkKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDE4IEBACisyMDE5LTExLTEzICBBYWthc2ggSmFpbiAgPGFha2FzaF9qYWluQGFwcGxlLmNv bT4KKworICAgICAgICBNYWtlIHdlYmtpdHBlcmwgRVdTIHJvYnVzdCBhZ2FpbnN0IGZsYWtpbmVz cyBpbiB3ZWJraXRwZXJsIHRlc3RzCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3No b3dfYnVnLmNnaT9pZD0yMDQxNTUKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMh KS4KKworICAgICAgICAqIEJ1aWxkU2xhdmVTdXBwb3J0L2V3cy1idWlsZC9zdGVwcy5weToKKyAg ICAgICAgKFJ1bldlYktpdFBlcmxUZXN0cyk6IFNldCBoYWx0T25GYWlsdXJlIGFuZCBmbHVua09u RmFpbHVyZSB0byBmYWxzZS4KKyAgICAgICAgKFJ1bldlYktpdFBlcmxUZXN0cy5ldmFsdWF0ZUNv bW1hbmQpOiBBZGQgcmUtcnVuIHN0ZXAgaW4gY2FzZSBvZiBmYWlsdXJlLgorICAgICAgICAoUmVS dW5XZWJLaXRQZXJsVGVzdHMpOgorICAgICAgICAoUmVSdW5XZWJLaXRQZXJsVGVzdHMuZXZhbHVh dGVDb21tYW5kKTogT3ZlcnJpZGVkIHRoaXMgdG8gYXZvaWQgcnVubmluZyB0aGlzIHN0ZXAgYWdh aW4uCisgICAgICAgICogQnVpbGRTbGF2ZVN1cHBvcnQvZXdzLWJ1aWxkL3N0ZXBzX3VuaXR0ZXN0 LnB5OgorICAgICAgICAoVGVzdFJlUnVuSmF2YVNjcmlwdENvcmVUZXN0cyk6IEFkZGVkIHVuaXQt dGVzdHMuCisKIDIwMTktMTEtMTMgIEFha2FzaCBKYWluICA8YWFrYXNoX2phaW5AYXBwbGUuY29t PgogCiAgICAgICAgIEZsYWt5IEFQSSBUZXN0IFRlc3RXZWJLaXRBUEkuU2VydmljZVdvcmtlcnMu VGhyb3R0bGVDcmFzaApJbmRleDogVG9vbHMvQnVpbGRTbGF2ZVN1cHBvcnQvZXdzLWJ1aWxkL3N0 ZXBzLnB5Cj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT0KLS0tIFRvb2xzL0J1aWxkU2xhdmVTdXBwb3J0L2V3cy1idWlsZC9z dGVwcy5weQkocmV2aXNpb24gMjUyNDA4KQorKysgVG9vbHMvQnVpbGRTbGF2ZVN1cHBvcnQvZXdz LWJ1aWxkL3N0ZXBzLnB5CSh3b3JraW5nIGNvcHkpCkBAIC02MTgsNyArNjE4LDggQEAgY2xhc3Mg UnVuV2ViS2l0UGVybFRlc3RzKHNoZWxsLlNoZWxsQ29tbQogICAgIG5hbWUgPSAnd2Via2l0cGVy bC10ZXN0cycKICAgICBkZXNjcmlwdGlvbiA9IFsnd2Via2l0cGVybC10ZXN0cyBydW5uaW5nJ10K ICAgICBkZXNjcmlwdGlvbkRvbmUgPSBbJ3dlYmtpdHBlcmwtdGVzdHMnXQotICAgIGZsdW5rT25G YWlsdXJlID0gVHJ1ZQorICAgIGZsdW5rT25GYWlsdXJlID0gRmFsc2UKKyAgICBoYWx0T25GYWls dXJlID0gRmFsc2UKICAgICBjb21tYW5kID0gWydwZXJsJywgJ1Rvb2xzL1NjcmlwdHMvdGVzdC13 ZWJraXRwZXJsJ10KIAogICAgIGRlZiBfX2luaXRfXyhzZWxmLCAqKmt3YXJncyk6CkBAIC02MzEs NiArNjMyLDIxIEBAIGNsYXNzIFJ1bldlYktpdFBlcmxUZXN0cyhzaGVsbC5TaGVsbENvbW0KICAg ICAgICAgICAgIHJldHVybiB7dSdzdGVwJzogdW5pY29kZShtZXNzYWdlKX0KICAgICAgICAgcmV0 dXJuIHt1J3N0ZXAnOiB1J0ZhaWxlZCB3ZWJraXRwZXJsIHRlc3RzJ30KIAorICAgIGRlZiBldmFs dWF0ZUNvbW1hbmQoc2VsZiwgY21kKToKKyAgICAgICAgcmMgPSBzaGVsbC5TaGVsbENvbW1hbmQu ZXZhbHVhdGVDb21tYW5kKHNlbGYsIGNtZCkKKyAgICAgICAgaWYgcmMgPT0gRkFJTFVSRToKKyAg ICAgICAgICAgIHNlbGYuYnVpbGQuYWRkU3RlcHNBZnRlckN1cnJlbnRTdGVwKFtSZVJ1bldlYktp dFBlcmxUZXN0cygpXSkKKyAgICAgICAgcmV0dXJuIHJjCisKKworY2xhc3MgUmVSdW5XZWJLaXRQ ZXJsVGVzdHMoUnVuV2ViS2l0UGVybFRlc3RzKToKKyAgICBuYW1lID0gJ3JlLXJ1bi13ZWJraXRw ZXJsLXRlc3RzJworICAgIGZsdW5rT25GYWlsdXJlID0gVHJ1ZQorICAgIGhhbHRPbkZhaWx1cmUg PSBUcnVlCisKKyAgICBkZWYgZXZhbHVhdGVDb21tYW5kKHNlbGYsIGNtZCk6CisgICAgICAgIHJl dHVybiBzaGVsbC5TaGVsbENvbW1hbmQuZXZhbHVhdGVDb21tYW5kKHNlbGYsIGNtZCkKKwogCiBj bGFzcyBSdW5CdWlsZFdlYktpdE9yZ1VuaXRUZXN0cyhzaGVsbC5TaGVsbENvbW1hbmQpOgogICAg IG5hbWUgPSAnYnVpbGQtd2Via2l0LW9yZy11bml0LXRlc3RzJwpJbmRleDogVG9vbHMvQnVpbGRT bGF2ZVN1cHBvcnQvZXdzLWJ1aWxkL3N0ZXBzX3VuaXR0ZXN0LnB5Cj09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIFRv b2xzL0J1aWxkU2xhdmVTdXBwb3J0L2V3cy1idWlsZC9zdGVwc191bml0dGVzdC5weQkocmV2aXNp b24gMjUyNDA4KQorKysgVG9vbHMvQnVpbGRTbGF2ZVN1cHBvcnQvZXdzLWJ1aWxkL3N0ZXBzX3Vu aXR0ZXN0LnB5CSh3b3JraW5nIGNvcHkpCkBAIC0zOCw3ICszOCw3IEBAIGZyb20gc3RlcHMgaW1w b3J0IChBbmFseXplQVBJVGVzdHNSZXN1bHQKICAgICAgICAgICAgICAgICAgICBDaGVja091dFNv dXJjZSwgQ2hlY2tPdXRTcGVjaWZpY1JldmlzaW9uLCBDaGVja1BhdGNoUmVsZXZhbmNlLCBDaGVj a1N0eWxlLCBDbGVhbkJ1aWxkLCBDbGVhblVwR2l0SW5kZXhMb2NrLCBDbGVhbldvcmtpbmdEaXJl Y3RvcnksCiAgICAgICAgICAgICAgICAgICAgQ29tcGlsZUpTQywgQ29tcGlsZUpTQ1RvVCwgQ29t cGlsZVdlYktpdCwgQ29tcGlsZVdlYktpdFRvVCwgQ29uZmlndXJlQnVpbGQsCiAgICAgICAgICAg ICAgICAgICAgRG93bmxvYWRCdWlsdFByb2R1Y3QsIERvd25sb2FkQnVpbHRQcm9kdWN0RnJvbU1h c3RlciwgRXh0cmFjdEJ1aWx0UHJvZHVjdCwgRXh0cmFjdFRlc3RSZXN1bHRzLCBJbnN0YWxsR3Rr RGVwZW5kZW5jaWVzLCBJbnN0YWxsV3BlRGVwZW5kZW5jaWVzLCBLaWxsT2xkUHJvY2Vzc2VzLAot ICAgICAgICAgICAgICAgICAgIFByaW50Q29uZmlndXJhdGlvbiwgUmVSdW5BUElUZXN0cywgUmVS dW5KYXZhU2NyaXB0Q29yZVRlc3RzLCBSZVJ1bldlYktpdFRlc3RzLCBSdW5BUElUZXN0cywgUnVu QVBJVGVzdHNXaXRob3V0UGF0Y2gsCisgICAgICAgICAgICAgICAgICAgUHJpbnRDb25maWd1cmF0 aW9uLCBSZVJ1bkFQSVRlc3RzLCBSZVJ1bkphdmFTY3JpcHRDb3JlVGVzdHMsIFJlUnVuV2ViS2l0 UGVybFRlc3RzLCBSZVJ1bldlYktpdFRlc3RzLCBSdW5BUElUZXN0cywgUnVuQVBJVGVzdHNXaXRo b3V0UGF0Y2gsCiAgICAgICAgICAgICAgICAgICAgUnVuQmluZGluZ3NUZXN0cywgUnVuQnVpbGRX ZWJLaXRPcmdVbml0VGVzdHMsIFJ1bkVXU0J1aWxkYm90Q2hlY2tDb25maWcsIFJ1bkVXU1VuaXRU ZXN0cywgUnVuSmF2YVNjcmlwdENvcmVUZXN0cywgUnVuSlNDVGVzdHNXaXRob3V0UGF0Y2gsIFJ1 bldlYktpdDFUZXN0cywKICAgICAgICAgICAgICAgICAgICBSdW5XZWJLaXRQZXJsVGVzdHMsIFJ1 bldlYktpdFB5VGVzdHMsIFJ1bldlYktpdFRlc3RzLCBSdW5XZWJLaXRUZXN0c1dpdGhvdXRQYXRj aCwgVGVzdFdpdGhGYWlsdXJlQ291bnQsIFRyaWdnZXIsIFRyYW5zZmVyVG9TMywgVW5BcHBseVBh dGNoSWZSZXF1aXJlZCwKICAgICAgICAgICAgICAgICAgICBVcGRhdGVXb3JraW5nRGlyZWN0b3J5 LCBVcGxvYWRCdWlsdFByb2R1Y3QsIFVwbG9hZFRlc3RSZXN1bHRzLCBWYWxpZGF0ZVBhdGNoKQpA QCAtMzgwLDggKzM4MCwxMSBAQCBjbGFzcyBUZXN0UnVuV2ViS2l0UGVybFRlc3RzKEJ1aWxkU3Rl cE1pCiAgICAgZGVmIHRlYXJEb3duKHNlbGYpOgogICAgICAgICByZXR1cm4gc2VsZi50ZWFyRG93 bkJ1aWxkU3RlcCgpCiAKLSAgICBkZWYgdGVzdF9zdWNjZXNzKHNlbGYpOgorICAgIGRlZiBjb25m aWd1cmVTdGVwKHNlbGYpOgogICAgICAgICBzZWxmLnNldHVwU3RlcChSdW5XZWJLaXRQZXJsVGVz dHMoKSkKKworICAgIGRlZiB0ZXN0X3N1Y2Nlc3Moc2VsZik6CisgICAgICAgIHNlbGYuY29uZmln dXJlU3RlcCgpCiAgICAgICAgIHNlbGYuZXhwZWN0UmVtb3RlQ29tbWFuZHMoCiAgICAgICAgICAg ICBFeHBlY3RTaGVsbCh3b3JrZGlyPSd3a2RpcicsCiAgICAgICAgICAgICAgICAgICAgICAgICBs b2dFbnZpcm9uPUZhbHNlLApAQCAtMzk0LDcgKzM5Nyw3IEBAIGNsYXNzIFRlc3RSdW5XZWJLaXRQ ZXJsVGVzdHMoQnVpbGRTdGVwTWkKICAgICAgICAgcmV0dXJuIHNlbGYucnVuU3RlcCgpCiAKICAg ICBkZWYgdGVzdF9mYWlsdXJlKHNlbGYpOgotICAgICAgICBzZWxmLnNldHVwU3RlcChSdW5XZWJL aXRQZXJsVGVzdHMoKSkKKyAgICAgICAgc2VsZi5jb25maWd1cmVTdGVwKCkKICAgICAgICAgc2Vs Zi5leHBlY3RSZW1vdGVDb21tYW5kcygKICAgICAgICAgICAgIEV4cGVjdFNoZWxsKHdvcmtkaXI9 J3drZGlyJywKICAgICAgICAgICAgICAgICAgICAgICAgIGxvZ0Vudmlyb249RmFsc2UsCkBAIC00 MTEsNiArNDE0LDExIEBAIEZhaWxlZCAxLzQwIHRlc3QgcHJvZ3JhbXMuIDEwLzYzMCBzdWJ0ZXMK ICAgICAgICAgcmV0dXJuIHNlbGYucnVuU3RlcCgpCiAKIAorY2xhc3MgVGVzdFJlUnVuSmF2YVNj cmlwdENvcmVUZXN0cyhUZXN0UnVuV2ViS2l0UGVybFRlc3RzKToKKyAgICBkZWYgY29uZmlndXJl U3RlcChzZWxmKToKKyAgICAgICAgc2VsZi5zZXR1cFN0ZXAoUmVSdW5XZWJLaXRQZXJsVGVzdHMo KSkKKworCiBjbGFzcyBUZXN0V2ViS2l0UHlUZXN0cyhCdWlsZFN0ZXBNaXhpbkFkZGl0aW9ucywg dW5pdHRlc3QuVGVzdENhc2UpOgogICAgIGRlZiBzZXRVcChzZWxmKToKICAgICAgICAgc2VsZi5s b25nTWVzc2FnZSA9IFRydWUK