204769 2019-12-02 14:39:36 -0800 Do not retry the EWS build due to flaky failures in layout-test 2019-12-03 13:08:33 -0800 1 1 1 Unclassified WebKit Tools / Tests Other Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=204704 InRadar P2 Normal --- 1 aakash_jain aakash_jain aakash_jain ap commit-queue dbates dean_johnson jbedard kocsen_chung ryanhaddad webkit-bot-watchers-bugzilla webkit-bug-importer oldest_to_newest 1594799 0 aakash_jain 2019-12-02 14:39:36 -0800 In EWS, when layout-tests run fails few tests, we re-run the tests. Then we run these tests on clean-tree, and compare the results of first run, second run, and clean tree run. In case there is flakiness (different tests fail in first and second runs), and no consistent failure is introduced by the patch being tested, we retry the entire build. We have this logic for years in old EWS and we copied it in new EWS. See: https://trac.webkit.org/browser/webkit/trunk/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py#L352 (Defer means retrying the build). ''' 350 # At this point we know that at least one test flaked, but no consistent failures 351 # were introduced. This is a bit of a grey-zone. 352 return False # Defer patch ''' Retrying the entire build, just because of few flaky tests seems excessive and inefficient. This means that if someone commits a very flaky test (which is not rare), then EWS will keep retrying the patch (almost in a infinite loop), and would be almost stuck until the flakiness is fixed. Also, we know that we already have various tests in trunk which are flaky. In other test-suites like API-tests and JSC-tests, we do not retry the build on flaky test failures, we ignore the flaky tests, and only focus on tests which failed consistently with the patch, and passed without the patch. We should do the similar thing for layout-tests as well. 1594806 1 aakash_jain 2019-12-02 14:46:50 -0800 For e.g.: In https://ews-build.webkit.org/#/builders/17/builds/7537 first_run_failures: "loader/stateobjects/pushstate-size.html" second_run_failures: "loader/stateobjects/pushstate-size.html", "webaudio/audioparam-setTargetAtTime.html" clean_tree_run_failures: "loader/stateobjects/pushstate-size.html" There was a pre-existing test-failure on trunk, and just because of one flaky test (webaudio/audioparam-setTargetAtTime.html), entire build was retried. This build should have been marked as SUCCESS. 1594829 2 384663 aakash_jain 2019-12-02 15:36:30 -0800 Created attachment 384663 Patch 1595015 3 384663 jbedard 2019-12-03 08:05:13 -0800 Comment on attachment 384663 Patch Seems like this is what we want more often than not, but I still dislike our flakiness story.... 1595045 4 384663 commit-queue 2019-12-03 09:56:22 -0800 Comment on attachment 384663 Patch Clearing flags on attachment: 384663 Committed r253049: <https://trac.webkit.org/changeset/253049> 1595046 5 commit-queue 2019-12-03 09:56:23 -0800 All reviewed patches have been landed. Closing bug. 1595047 6 webkit-bug-importer 2019-12-03 09:57:19 -0800 <rdar://problem/57593402> 1595138 7 aakash_jain 2019-12-03 13:08:33 -0800 Deployed on production server. 384663 2019-12-02 15:36:30 -0800 2019-12-03 09:56:22 -0800 Patch bug-204769-20191202183628.patch text/plain 4773 aakash_jain SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDI1MzAxNCkKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDE4IEBACisyMDE5LTEyLTAyICBBYWthc2ggSmFpbiAgPGFha2FzaF9qYWluQGFwcGxlLmNv bT4KKworICAgICAgICBEbyBub3QgcmV0cnkgdGhlIEVXUyBidWlsZCBkdWUgdG8gZmxha3kgZmFp bHVyZXMgaW4gbGF5b3V0LXRlc3QKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hv d19idWcuY2dpP2lkPTIwNDc2OQorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEp LgorCisgICAgICAgICogQnVpbGRTbGF2ZVN1cHBvcnQvZXdzLWJ1aWxkL3N0ZXBzLnB5OgorICAg ICAgICAoQW5hbHl6ZUxheW91dFRlc3RzUmVzdWx0cy5zdGFydCk6IERvIG5vdCByZXRyeSB0aGUg YnVpbGQgZHVlIHRvIGZsYWt5IHRlc3RzIGZhaWx1cmVzLgorICAgICAgICAqIEJ1aWxkU2xhdmVT dXBwb3J0L2V3cy1idWlsZC9zdGVwc191bml0dGVzdC5weToKKyAgICAgICAgKHRlc3RfZmxha3lf YW5kX2luY29uc2lzdGVudF9mYWlsdXJlc193aXRob3V0X2NsZWFuX3RyZWVfZmFpbHVyZXMpOiBl eHBlY3RlZCBvdXRjb21lIGNoYW5nZWQgZnJvbSBSRVRSWSB0byBTVUNDRVNTLgorICAgICAgICAo dGVzdF9mbGFreV9hbmRfaW5jb25zaXN0ZW50X2ZhaWx1cmVzX3dpdGhfY2xlYW5fdHJlZV9mYWls dXJlcyk6IERpdHRvLgorICAgICAgICAodGVzdF9mbGFreV9hbmRfY29uc2lzdGVudF9mYWlsdXJl c193aXRoX2NsZWFuX3RyZWVfZmFpbHVyZXMpOiBEaXR0by4KKyAgICAgICAgKHRlc3RfbWlsZGx5 X2ZsYWt5X3BhdGNoX3dpdGhfc29tZV90cmVlX3JlZG5lc3NfYW5kX2ZsYWtpbmVzcyk6IERpdHRv LgorCiAyMDE5LTEyLTAyICBBYWthc2ggSmFpbiAgPGFha2FzaF9qYWluQGFwcGxlLmNvbT4KIAog ICAgICAgICBSZXZlcnQgLSBbRVdTXSBEbyBub3QgcmV0cnkgbGF5b3V0LXRlc3RzIGJ1aWxkIGlm IHRoZSBmbGFreSB0ZXN0IGZhaWx1cmVzIGFyZSBhbHNvIHByZXNlbnQgaW4gY2xlYW4gdHJlZSBy dW4KSW5kZXg6IFRvb2xzL0J1aWxkU2xhdmVTdXBwb3J0L2V3cy1idWlsZC9zdGVwcy5weQo9PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09Ci0tLSBUb29scy9CdWlsZFNsYXZlU3VwcG9ydC9ld3MtYnVpbGQvc3RlcHMucHkJKHJl dmlzaW9uIDI1MzAxNCkKKysrIFRvb2xzL0J1aWxkU2xhdmVTdXBwb3J0L2V3cy1idWlsZC9zdGVw cy5weQkod29ya2luZyBjb3B5KQpAQCAtMTQ1Myw4ICsxNDUzLDkgQEAgY2xhc3MgQW5hbHl6ZUxh eW91dFRlc3RzUmVzdWx0cyhidWlsZHN0ZQogICAgICAgICAgICAgICAgICAgICByZXR1cm4gc2Vs Zi5yZXBvcnRfZmFpbHVyZShmYWlsdXJlc19pbnRyb2R1Y2VkX2J5X3BhdGNoKQogCiAgICAgICAg ICAgICAjIEF0IHRoaXMgcG9pbnQgd2Uga25vdyB0aGF0IGF0IGxlYXN0IG9uZSB0ZXN0IGZsYWtl ZCwgYnV0IG5vIGNvbnNpc3RlbnQgZmFpbHVyZXMKLSAgICAgICAgICAgICMgd2VyZSBpbnRyb2R1 Y2VkLiBUaGlzIGlzIGEgYml0IG9mIGEgZ3JleS16b25lLgotICAgICAgICAgICAgcmV0dXJuIHNl bGYucmV0cnlfYnVpbGQoKQorICAgICAgICAgICAgIyB3ZXJlIGludHJvZHVjZWQuIFRoaXMgaXMg YSBiaXQgb2YgYSBncmV5LXpvbmUuIEl0J3MgcG9zc2libGUgdGhhdCB0aGUgcGF0Y2ggaW50cm9k dWNlZCBzb21lIGZsYWtpbmVzcy4KKyAgICAgICAgICAgICMgV2Ugc3RpbGwgbWFyayB0aGUgYnVp bGQgYXMgU1VDQ0VTUy4KKyAgICAgICAgICAgIHJldHVybiBzZWxmLnJlcG9ydF9wcmVfZXhpc3Rp bmdfZmFpbHVyZXMoY2xlYW5fdHJlZV9yZXN1bHRzX2ZhaWxpbmdfdGVzdHMpCiAKICAgICAgICAg aWYgY2xlYW5fdHJlZV9yZXN1bHRzX2RpZF9leGNlZWRfdGVzdF9mYWlsdXJlX2xpbWl0OgogICAg ICAgICAgICAgcmV0dXJuIHNlbGYucmV0cnlfYnVpbGQoKQpJbmRleDogVG9vbHMvQnVpbGRTbGF2 ZVN1cHBvcnQvZXdzLWJ1aWxkL3N0ZXBzX3VuaXR0ZXN0LnB5Cj09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIFRvb2xz L0J1aWxkU2xhdmVTdXBwb3J0L2V3cy1idWlsZC9zdGVwc191bml0dGVzdC5weQkocmV2aXNpb24g MjUzMDE0KQorKysgVG9vbHMvQnVpbGRTbGF2ZVN1cHBvcnQvZXdzLWJ1aWxkL3N0ZXBzX3VuaXR0 ZXN0LnB5CSh3b3JraW5nIGNvcHkpCkBAIC0xNjM5LDcgKzE2MzksNyBAQCBjbGFzcyBUZXN0QW5h bHl6ZUxheW91dFRlc3RzUmVzdWx0cyhCdWlsCiAgICAgICAgIHNlbGYuY29uZmlndXJlU3RlcCgp CiAgICAgICAgIHNlbGYuc2V0UHJvcGVydHkoJ2ZpcnN0X3J1bl9mYWlsdXJlcycsIFsndGVzdDEn LCAndGVzdDInXSkKICAgICAgICAgc2VsZi5zZXRQcm9wZXJ0eSgnc2Vjb25kX3J1bl9mYWlsdXJl cycsIFsndGVzdDMnXSkKLSAgICAgICAgc2VsZi5leHBlY3RPdXRjb21lKHJlc3VsdD1SRVRSWSwg c3RhdGVfc3RyaW5nPSdVbmFibGUgdG8gY29uZmlybSBpZiB0ZXN0IGZhaWx1cmVzIGFyZSBpbnRy b2R1Y2VkIGJ5IHBhdGNoLCByZXRyeWluZyBidWlsZCAocmV0cnkpJykKKyAgICAgICAgc2VsZi5l eHBlY3RPdXRjb21lKHJlc3VsdD1TVUNDRVNTLCBzdGF0ZV9zdHJpbmc9J1Bhc3NlZCBsYXlvdXQg dGVzdHMnKQogICAgICAgICByZXR1cm4gc2VsZi5ydW5TdGVwKCkKIAogICAgIGRlZiB0ZXN0X2Zs YWt5X2FuZF9pbmNvbnNpc3RlbnRfZmFpbHVyZXNfd2l0aF9jbGVhbl90cmVlX2ZhaWx1cmVzKHNl bGYpOgpAQCAtMTY0Nyw3ICsxNjQ3LDcgQEAgY2xhc3MgVGVzdEFuYWx5emVMYXlvdXRUZXN0c1Jl c3VsdHMoQnVpbAogICAgICAgICBzZWxmLnNldFByb3BlcnR5KCdmaXJzdF9ydW5fZmFpbHVyZXMn LCBbJ3Rlc3QxJywgJ3Rlc3QyJ10pCiAgICAgICAgIHNlbGYuc2V0UHJvcGVydHkoJ3NlY29uZF9y dW5fZmFpbHVyZXMnLCBbJ3Rlc3QzJ10pCiAgICAgICAgIHNlbGYuc2V0UHJvcGVydHkoJ2NsZWFu X3RyZWVfcnVuX2ZhaWx1cmVzJywgWyd0ZXN0MScsICd0ZXN0MicsICd0ZXN0MyddKQotICAgICAg ICBzZWxmLmV4cGVjdE91dGNvbWUocmVzdWx0PVJFVFJZLCBzdGF0ZV9zdHJpbmc9J1VuYWJsZSB0 byBjb25maXJtIGlmIHRlc3QgZmFpbHVyZXMgYXJlIGludHJvZHVjZWQgYnkgcGF0Y2gsIHJldHJ5 aW5nIGJ1aWxkIChyZXRyeSknKQorICAgICAgICBzZWxmLmV4cGVjdE91dGNvbWUocmVzdWx0PVNV Q0NFU1MsIHN0YXRlX3N0cmluZz0nUGFzc2VkIGxheW91dCB0ZXN0cycpCiAgICAgICAgIHJldHVy biBzZWxmLnJ1blN0ZXAoKQogCiAgICAgZGVmIHRlc3RfZmxha3lfYW5kX2NvbnNpc3RlbnRfZmFp bHVyZXNfd2l0aF9jbGVhbl90cmVlX2ZhaWx1cmVzKHNlbGYpOgpAQCAtMTY1NSw3ICsxNjU1LDcg QEAgY2xhc3MgVGVzdEFuYWx5emVMYXlvdXRUZXN0c1Jlc3VsdHMoQnVpbAogICAgICAgICBzZWxm LnNldFByb3BlcnR5KCdmaXJzdF9ydW5fZmFpbHVyZXMnLCBbJ3Rlc3QxJywgJ3Rlc3QyJ10pCiAg ICAgICAgIHNlbGYuc2V0UHJvcGVydHkoJ3NlY29uZF9ydW5fZmFpbHVyZXMnLCBbJ3Rlc3QxJ10p CiAgICAgICAgIHNlbGYuc2V0UHJvcGVydHkoJ2NsZWFuX3RyZWVfcnVuX2ZhaWx1cmVzJywgWyd0 ZXN0MScsICd0ZXN0MiddKQotICAgICAgICBzZWxmLmV4cGVjdE91dGNvbWUocmVzdWx0PVJFVFJZ LCBzdGF0ZV9zdHJpbmc9J1VuYWJsZSB0byBjb25maXJtIGlmIHRlc3QgZmFpbHVyZXMgYXJlIGlu dHJvZHVjZWQgYnkgcGF0Y2gsIHJldHJ5aW5nIGJ1aWxkIChyZXRyeSknKQorICAgICAgICBzZWxm LmV4cGVjdE91dGNvbWUocmVzdWx0PVNVQ0NFU1MsIHN0YXRlX3N0cmluZz0nUGFzc2VkIGxheW91 dCB0ZXN0cycpCiAgICAgICAgIHJldHVybiBzZWxmLnJ1blN0ZXAoKQogCiAgICAgZGVmIHRlc3Rf bWlsZGx5X2ZsYWt5X3BhdGNoX3dpdGhfc29tZV90cmVlX3JlZG5lc3NfYW5kX2ZsYWtpbmVzcyhz ZWxmKToKQEAgLTE2NjMsNyArMTY2Myw3IEBAIGNsYXNzIFRlc3RBbmFseXplTGF5b3V0VGVzdHNS ZXN1bHRzKEJ1aWwKICAgICAgICAgc2VsZi5zZXRQcm9wZXJ0eSgnZmlyc3RfcnVuX2ZhaWx1cmVz JywgWydQcmVFeGlzdGluZ0ZhaWwxJywgJ1ByZUV4aXN0aW5nRmFpbDInLCAnRmFpbDEnXSkKICAg ICAgICAgc2VsZi5zZXRQcm9wZXJ0eSgnc2Vjb25kX3J1bl9mYWlsdXJlcycsIFsnUHJlRXhpc3Rp bmdGYWlsMScsICdQcmVFeGlzdGluZ0ZhaWwyJ10pCiAgICAgICAgIHNlbGYuc2V0UHJvcGVydHko J2NsZWFuX3RyZWVfcnVuX2ZhaWx1cmVzJywgWydQcmVFeGlzdGluZ0ZhaWwxJywgJ1ByZUV4aXN0 aW5nRmFpbDInLCAnRmFpbDInXSkKLSAgICAgICAgc2VsZi5leHBlY3RPdXRjb21lKHJlc3VsdD1S RVRSWSwgc3RhdGVfc3RyaW5nPSdVbmFibGUgdG8gY29uZmlybSBpZiB0ZXN0IGZhaWx1cmVzIGFy ZSBpbnRyb2R1Y2VkIGJ5IHBhdGNoLCByZXRyeWluZyBidWlsZCAocmV0cnkpJykKKyAgICAgICAg c2VsZi5leHBlY3RPdXRjb21lKHJlc3VsdD1TVUNDRVNTLCBzdGF0ZV9zdHJpbmc9J1Bhc3NlZCBs YXlvdXQgdGVzdHMnKQogICAgICAgICByZXR1cm4gc2VsZi5ydW5TdGVwKCkKIAogICAgIGRlZiB0 ZXN0X2ZpcnN0X3J1bl9leGNlZWRfZmFpbHVyZV9saW1pdChzZWxmKToK