207050 2020-01-31 09:20:29 -0800 [ews] Display flaky test names in build summary when ReRunWebKitTests passes 2020-02-10 13:47:28 -0800 1 1 1 Unclassified WebKit Tools / Tests Other Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=207504 InRadar P2 Normal --- 1 aakash_jain aakash_jain aakash_jain ap jbedard webkit-bot-watchers-bugzilla webkit-bug-importer oldest_to_newest 1613400 0 aakash_jain 2020-01-31 09:20:29 -0800 When WebKitTest (layout-tests) fail, we re-run them. If the re-run pass, we mark the build as successful. The build summary is set to: 'Passed layout tests'. It would be better to indicate the flaky test name in the build summary in this case, so that bot-watchers can easily notice flaky tests (and the pattern). 1613421 1 389364 aakash_jain 2020-01-31 09:36:24 -0800 Created attachment 389364 Patch 1613426 2 aakash_jain 2020-01-31 09:41:07 -0800 This change will help in easily identifying flaky tests from builds like these (in which layout-tests have some test failures but re-run-layout-tests passes): https://ews-build.webkit.org/#/builders/24/builds/9685 https://ews-build.webkit.org/#/builders/24/builds/9686 https://ews-build.webkit.org/#/builders/24/builds/9661 https://ews-build.webkit.org/#/builders/24/builds/9600 https://ews-build.webkit.org/#/builders/24/builds/9588 https://ews-build.webkit.org/#/builders/24/builds/9592 1613427 3 aakash_jain 2020-01-31 09:41:29 -0800 Sample run: https://ews-build.webkit-uat.org/#/builders/16/builds/33 1613454 4 389364 jbedard 2020-01-31 10:03:19 -0800 Comment on attachment 389364 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=389364&action=review > Tools/BuildSlaveSupport/ews-build/steps.py:1360 > + flaky_failures_string = ', '.join(flaky_failures) This looks good for now. But, once we have this information, we could cross reference these tests by name against the results database and see if they've failed in the last week. If they haven't, that's a decent indication of a newly flakey test (perhaps caused by the patch) > Tools/BuildSlaveSupport/ews-build/steps.py:-1355 > - message = 'Passed layout tests' Why did we move this out of the if statement. 1613505 5 389364 aakash_jain 2020-01-31 11:34:09 -0800 Comment on attachment 389364 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=389364&action=review >> Tools/BuildSlaveSupport/ews-build/steps.py:1360 >> + flaky_failures_string = ', '.join(flaky_failures) > > This looks good for now. > > But, once we have this information, we could cross reference these tests by name against the results database and see if they've failed in the last week. If they haven't, that's a decent indication of a newly flakey test (perhaps caused by the patch) Agree. Another thing we need to do is to automatically file bugs for these flaky tests. >> Tools/BuildSlaveSupport/ews-build/steps.py:-1355 >> - message = 'Passed layout tests' > > Why did we move this out of the if statement. No strong reason. Moving it back. 1613509 6 aakash_jain 2020-01-31 11:38:22 -0800 Committed r255513: <https://trac.webkit.org/changeset/255513> 1613510 7 webkit-bug-importer 2020-01-31 11:39:18 -0800 <rdar://problem/59069094> 1613727 8 aakash_jain 2020-02-01 04:36:01 -0800 Deployed on production server. 389364 2020-01-31 09:36:24 -0800 2020-01-31 12:06:46 -0800 Patch bug-207050-20200131123623.patch text/plain 2238 aakash_jain SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDI1NTUwMSkKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDEzIEBACisyMDIwLTAxLTMxICBBYWthc2ggSmFpbiAgPGFha2FzaF9qYWluQGFwcGxlLmNv bT4KKworICAgICAgICBbZXdzXSBEaXNwbGF5IGZsYWt5IHRlc3QgbmFtZXMgaW4gYnVpbGQgc3Vt bWFyeSB3aGVuIFJlUnVuV2ViS2l0VGVzdHMgcGFzc2VzCisgICAgICAgIGh0dHBzOi8vYnVncy53 ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0yMDcwNTAKKworICAgICAgICBSZXZpZXdlZCBieSBO T0JPRFkgKE9PUFMhKS4KKworICAgICAgICAqIEJ1aWxkU2xhdmVTdXBwb3J0L2V3cy1idWlsZC9z dGVwcy5weToKKyAgICAgICAgKFJlUnVuV2ViS2l0VGVzdHMuZXZhbHVhdGVDb21tYW5kKToKKwog MjAyMC0wMS0zMSAgSm9uYXRoYW4gQmVkYXJkICA8amJlZGFyZEBhcHBsZS5jb20+CiAKICAgICAg ICAgcmVzdWx0cy53ZWJraXQub3JnOiBIYW5kbGUgbW9kaWZpZWQgZmlyZXdhbGwgcnVsZXMKSW5k ZXg6IFRvb2xzL0J1aWxkU2xhdmVTdXBwb3J0L2V3cy1idWlsZC9zdGVwcy5weQo9PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 Ci0tLSBUb29scy9CdWlsZFNsYXZlU3VwcG9ydC9ld3MtYnVpbGQvc3RlcHMucHkJKHJldmlzaW9u IDI1NTUwMCkKKysrIFRvb2xzL0J1aWxkU2xhdmVTdXBwb3J0L2V3cy1idWlsZC9zdGVwcy5weQko d29ya2luZyBjb3B5KQpAQCAtMTM1MSwxMCArMTM1MSwyMCBAQCBjbGFzcyBSZVJ1bldlYktpdFRl c3RzKFJ1bldlYktpdFRlc3RzKToKIAogICAgIGRlZiBldmFsdWF0ZUNvbW1hbmQoc2VsZiwgY21k KToKICAgICAgICAgcmMgPSBzZWxmLmV2YWx1YXRlUmVzdWx0KGNtZCkKKyAgICAgICAgZmlyc3Rf cmVzdWx0c19kaWRfZXhjZWVkX3Rlc3RfZmFpbHVyZV9saW1pdCA9IHNlbGYuZ2V0UHJvcGVydHko J2ZpcnN0X3Jlc3VsdHNfZXhjZWVkX2ZhaWx1cmVfbGltaXQnKQorICAgICAgICBmaXJzdF9yZXN1 bHRzX2ZhaWxpbmdfdGVzdHMgPSBzZXQoc2VsZi5nZXRQcm9wZXJ0eSgnZmlyc3RfcnVuX2ZhaWx1 cmVzJywgW10pKQorICAgICAgICBzZWNvbmRfcmVzdWx0c19kaWRfZXhjZWVkX3Rlc3RfZmFpbHVy ZV9saW1pdCA9IHNlbGYuZ2V0UHJvcGVydHkoJ3NlY29uZF9yZXN1bHRzX2V4Y2VlZF9mYWlsdXJl X2xpbWl0JykKKyAgICAgICAgc2Vjb25kX3Jlc3VsdHNfZmFpbGluZ190ZXN0cyA9IHNldChzZWxm LmdldFByb3BlcnR5KCdzZWNvbmRfcnVuX2ZhaWx1cmVzJywgW10pKQorICAgICAgICB0ZXN0c190 aGF0X2NvbnNpc3RlbnRseV9mYWlsZWQgPSBmaXJzdF9yZXN1bHRzX2ZhaWxpbmdfdGVzdHMuaW50 ZXJzZWN0aW9uKHNlY29uZF9yZXN1bHRzX2ZhaWxpbmdfdGVzdHMpCisgICAgICAgIGZsYWt5X2Zh aWx1cmVzID0gZmlyc3RfcmVzdWx0c19mYWlsaW5nX3Rlc3RzLnVuaW9uKHNlY29uZF9yZXN1bHRz X2ZhaWxpbmdfdGVzdHMpIC0gZmlyc3RfcmVzdWx0c19mYWlsaW5nX3Rlc3RzLmludGVyc2VjdGlv bihzZWNvbmRfcmVzdWx0c19mYWlsaW5nX3Rlc3RzKQorICAgICAgICBmbGFreV9mYWlsdXJlc19z dHJpbmcgPSAnLCAnLmpvaW4oZmxha3lfZmFpbHVyZXMpCisgICAgICAgIG1lc3NhZ2UgPSAnUGFz c2VkIGxheW91dCB0ZXN0cycKKwogICAgICAgICBpZiByYyA9PSBTVUNDRVNTIG9yIHJjID09IFdB Uk5JTkdTOgotICAgICAgICAgICAgbWVzc2FnZSA9ICdQYXNzZWQgbGF5b3V0IHRlc3RzJwogICAg ICAgICAgICAgc2VsZi5kZXNjcmlwdGlvbkRvbmUgPSBtZXNzYWdlCiAgICAgICAgICAgICBzZWxm LmJ1aWxkLnJlc3VsdHMgPSBTVUNDRVNTCisgICAgICAgICAgICBpZiBub3QgZmlyc3RfcmVzdWx0 c19kaWRfZXhjZWVkX3Rlc3RfZmFpbHVyZV9saW1pdDoKKyAgICAgICAgICAgICAgICBtZXNzYWdl ID0gJ0ZvdW5kIGZsYWt5IHRlc3RzOiB7fScuZm9ybWF0KGZsYWt5X2ZhaWx1cmVzX3N0cmluZykK ICAgICAgICAgICAgIHNlbGYuYnVpbGQuYnVpbGRGaW5pc2hlZChbbWVzc2FnZV0sIFNVQ0NFU1Mp CiAgICAgICAgIGVsc2U6CiAgICAgICAgICAgICBzZWxmLnNldFByb3BlcnR5KCdwYXRjaEZhaWxl ZFRlc3RzJywgVHJ1ZSkK