204707 2019-11-29 10:06:01 -0800 Add a unit-test to EWS for scenario when there is flakiness with patch and also on clean tree 2019-11-29 22:06:53 -0800 1 1 1 Unclassified WebKit Tools / Tests Other Unspecified Unspecified RESOLVED FIXED InRadar P2 Normal --- 1 aakash_jain aakash_jain aakash_jain ap ews-watchlist glenn jbedard webkit-bug-importer oldest_to_newest 1594220 0 aakash_jain 2019-11-29 10:06:01 -0800 Add a unit-test to old EWS for scenario when there is flakiness with patch and also on clean tree. 1594221 1 384518 aakash_jain 2019-11-29 10:08:16 -0800 Created attachment 384518 Unit-test for old EWS 1594222 2 aakash_jain 2019-11-29 10:09:51 -0800 Committed r252943: <https://trac.webkit.org/changeset/252943> 1594223 3 webkit-bug-importer 2019-11-29 10:10:18 -0800 <rdar://problem/57533998> 1594228 4 384519 aakash_jain 2019-11-29 10:16:59 -0800 Created attachment 384519 Similar unit-test for new EWS 1594229 5 aakash_jain 2019-11-29 10:18:13 -0800 . 1594230 6 aakash_jain 2019-11-29 10:20:48 -0800 Committed r252946: <https://trac.webkit.org/changeset/252946> 1594287 7 ap 2019-11-29 22:06:53 -0800 I think that I know why I didn't believe that this was the old EWS behavior. Looking at the "if self._results_failed_different_tests" branch in PatchAnalysisTask._retry_layout_tests, I see that it calls self._should_defer_patch_or_throw, which says that the result should be a PASS. But then we ignore the result, and make it a DEFER anyway! if tests_that_consistently_failed: if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id): return False # Defer patch # At this point we know that at least one test flaked, but no consistent failures # were introduced. This is a bit of a grey-zone. return False # Defer patch What's the point of the first "return False" here if we are going to defer either way? 384518 2019-11-29 10:08:16 -0800 2019-11-29 10:17:32 -0800 Unit-test for old EWS bug-204707-20191129130815.patch text/plain 1836 aakash_jain SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDI1Mjk0MikKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDEyIEBACisyMDE5LTExLTI5ICBBYWthc2ggSmFpbiAgPGFha2FzaF9qYWluQGFwcGxlLmNv bT4KKworICAgICAgICBBZGQgYSB1bml0LXRlc3QgdG8gb2xkIEVXUyBmb3Igc2NlbmFyaW8gd2hl biB0aGVyZSBpcyBmbGFraW5lc3Mgd2l0aCBwYXRjaCBhbmQgYWxzbyBvbiBjbGVhbiB0cmVlCisg ICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0yMDQ3MDcKKwor ICAgICAgICBVbnJldmlld2VkIHVuaXQtdGVzdC4KKworICAgICAgICAqIFNjcmlwdHMvd2Via2l0 cHkvdG9vbC9ib3QvY29tbWl0cXVldWV0YXNrX3VuaXR0ZXN0LnB5OgorCiAyMDE5LTExLTI5ICBQ YXVsbyBNYXRvcyAgPHBtYXRvc0BpZ2FsaWEuY29tPgogCiAgICAgICAgIE1vdmUganNjLWkzODYg ZnJvbSBvbGQgdG8gbmV3IEVXUwpJbmRleDogVG9vbHMvU2NyaXB0cy93ZWJraXRweS90b29sL2Jv dC9jb21taXRxdWV1ZXRhc2tfdW5pdHRlc3QucHkKPT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gVG9vbHMvU2NyaXB0 cy93ZWJraXRweS90b29sL2JvdC9jb21taXRxdWV1ZXRhc2tfdW5pdHRlc3QucHkJKHJldmlzaW9u IDI1Mjk0MikKKysrIFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvdG9vbC9ib3QvY29tbWl0cXVldWV0 YXNrX3VuaXR0ZXN0LnB5CSh3b3JraW5nIGNvcHkpCkBAIC01NDUsNiArNTQ1LDE0IEBAIGNvbW1h bmRfZmFpbGVkOiBmYWlsdXJlX21lc3NhZ2U9J1VuYWJsZSAKIAogICAgICAgICBzZWxmLl9ydW5f YW5kX2V4cGVjdF9wYXRjaF9hbmFseXNpc19yZXN1bHQoY29tbWl0X3F1ZXVlLCBQYXRjaEFuYWx5 c2lzUmVzdWx0LkRFRkVSLCBleHBlY3RfY2xlYW5fdGVzdHNfdG9fcnVuPVRydWUsIGV4cGVjdGVk X3JlcG9ydGVkX2ZsYWt5X3Rlc3RzPVsiRmFpbDEiLCAiRmFpbDIiXSkKIAorICAgIGRlZiB0ZXN0 X21pbGRseV9mbGFreV9wYXRjaF93aXRoX3NvbWVfdHJlZV9yZWRuZXNzX2FuZF9mbGFraW5lc3Mo c2VsZik6CisgICAgICAgIGNvbW1pdF9xdWV1ZSA9IE1vY2tTaW1wbGVUZXN0UGxhbkNvbW1pdFF1 ZXVlKAorICAgICAgICAgICAgZmlyc3RfdGVzdF9mYWlsdXJlcz1bIlByZUV4aXN0aW5nRmFpbDEi LCAiUHJlRXhpc3RpbmdGYWlsMiIsICJGYWlsMSJdLAorICAgICAgICAgICAgc2Vjb25kX3Rlc3Rf ZmFpbHVyZXM9WyJQcmVFeGlzdGluZ0ZhaWwxIiwgIlByZUV4aXN0aW5nRmFpbDIiXSwKKyAgICAg ICAgICAgIGNsZWFuX3Rlc3RfZmFpbHVyZXM9WyJQcmVFeGlzdGluZ0ZhaWwxIiwgIlByZUV4aXN0 aW5nRmFpbDIiLCAiRmFpbDIiXSkKKworICAgICAgICBzZWxmLl9ydW5fYW5kX2V4cGVjdF9wYXRj aF9hbmFseXNpc19yZXN1bHQoY29tbWl0X3F1ZXVlLCBQYXRjaEFuYWx5c2lzUmVzdWx0LkRFRkVS LCBleHBlY3RfY2xlYW5fdGVzdHNfdG9fcnVuPVRydWUsIGV4cGVjdGVkX3JlcG9ydGVkX2ZsYWt5 X3Rlc3RzPVsiRmFpbDEiXSkKKwogICAgIGRlZiB0ZXN0X3RyZWVfbW9yZV9yZWRfdGhhbl9wYXRj aChzZWxmKToKICAgICAgICAgY29tbWl0X3F1ZXVlID0gTW9ja1NpbXBsZVRlc3RQbGFuQ29tbWl0 UXVldWUoCiAgICAgICAgICAgICBmaXJzdF90ZXN0X2ZhaWx1cmVzPVsiRmFpbDEiLCAiRmFpbDIi LCAiRmFpbDMiXSwK 384519 2019-11-29 10:16:59 -0800 2019-11-29 10:16:59 -0800 Similar unit-test for new EWS unit-test-flakiness.patch text/plain 2025 aakash_jain SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDI1Mjk0NCkKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDE0IEBACisyMDE5LTExLTI5ICBBYWthc2ggSmFpbiAgPGFha2FzaF9qYWluQGFwcGxlLmNv bT4KKworICAgICAgICBBZGQgYSB1bml0LXRlc3QgdG8gbmV3IEVXUyBmb3Igc2NlbmFyaW8gd2hl biB0aGVyZSBpcyBmbGFraW5lc3Mgd2l0aCBwYXRjaCBhbmQgYWxzbyBvbiBjbGVhbiB0cmVlCisg ICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0yMDQ3MDcKKyAg ICAgICAgPHJkYXI6Ly9wcm9ibGVtLzU3NTMzOTk4PgorCisgICAgICAgIFVucmV2aWV3ZWQgdW5p dC10ZXN0LgorCisgICAgICAgICogQnVpbGRTbGF2ZVN1cHBvcnQvZXdzLWJ1aWxkL3N0ZXBzX3Vu aXR0ZXN0LnB5OgorICAgICAgICAoVGVzdEFuYWx5emVMYXlvdXRUZXN0c1Jlc3VsdHMudGVzdF9t aWxkbHlfZmxha3lfcGF0Y2hfd2l0aF9zb21lX3RyZWVfcmVkbmVzc19hbmRfZmxha2luZXNzKToK KwogMjAxOS0xMS0yOSAgQWFrYXNoIEphaW4gIDxhYWthc2hfamFpbkBhcHBsZS5jb20+CiAKICAg ICAgICAgQWRkIGEgdW5pdC10ZXN0IHRvIG9sZCBFV1MgZm9yIHNjZW5hcmlvIHdoZW4gdGhlcmUg aXMgZmxha2luZXNzIHdpdGggcGF0Y2ggYW5kIGFsc28gb24gY2xlYW4gdHJlZQpJbmRleDogVG9v bHMvQnVpbGRTbGF2ZVN1cHBvcnQvZXdzLWJ1aWxkL3N0ZXBzX3VuaXR0ZXN0LnB5Cj09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT0KLS0tIFRvb2xzL0J1aWxkU2xhdmVTdXBwb3J0L2V3cy1idWlsZC9zdGVwc191bml0dGVzdC5w eQkocmV2aXNpb24gMjUyOTQ0KQorKysgVG9vbHMvQnVpbGRTbGF2ZVN1cHBvcnQvZXdzLWJ1aWxk L3N0ZXBzX3VuaXR0ZXN0LnB5CSh3b3JraW5nIGNvcHkpCkBAIC0xNjU4LDYgKzE2NTgsMTQgQEAg Y2xhc3MgVGVzdEFuYWx5emVMYXlvdXRUZXN0c1Jlc3VsdHMoQnVpbAogICAgICAgICBzZWxmLmV4 cGVjdE91dGNvbWUocmVzdWx0PVJFVFJZLCBzdGF0ZV9zdHJpbmc9J1VuYWJsZSB0byBjb25maXJt IGlmIHRlc3QgZmFpbHVyZXMgYXJlIGludHJvZHVjZWQgYnkgcGF0Y2gsIHJldHJ5aW5nIGJ1aWxk IChyZXRyeSknKQogICAgICAgICByZXR1cm4gc2VsZi5ydW5TdGVwKCkKIAorICAgIGRlZiB0ZXN0 X21pbGRseV9mbGFreV9wYXRjaF93aXRoX3NvbWVfdHJlZV9yZWRuZXNzX2FuZF9mbGFraW5lc3Mo c2VsZik6CisgICAgICAgIHNlbGYuY29uZmlndXJlU3RlcCgpCisgICAgICAgIHNlbGYuc2V0UHJv cGVydHkoJ2ZpcnN0X3J1bl9mYWlsdXJlcycsIFsnUHJlRXhpc3RpbmdGYWlsMScsICdQcmVFeGlz dGluZ0ZhaWwyJywgJ0ZhaWwxJ10pCisgICAgICAgIHNlbGYuc2V0UHJvcGVydHkoJ3NlY29uZF9y dW5fZmFpbHVyZXMnLCBbJ1ByZUV4aXN0aW5nRmFpbDEnLCAnUHJlRXhpc3RpbmdGYWlsMiddKQor ICAgICAgICBzZWxmLnNldFByb3BlcnR5KCdjbGVhbl90cmVlX3J1bl9mYWlsdXJlcycsIFsnUHJl RXhpc3RpbmdGYWlsMScsICdQcmVFeGlzdGluZ0ZhaWwyJywgJ0ZhaWwyJ10pCisgICAgICAgIHNl bGYuZXhwZWN0T3V0Y29tZShyZXN1bHQ9UkVUUlksIHN0YXRlX3N0cmluZz0nVW5hYmxlIHRvIGNv bmZpcm0gaWYgdGVzdCBmYWlsdXJlcyBhcmUgaW50cm9kdWNlZCBieSBwYXRjaCwgcmV0cnlpbmcg YnVpbGQgKHJldHJ5KScpCisgICAgICAgIHJldHVybiBzZWxmLnJ1blN0ZXAoKQorCiAgICAgZGVm IHRlc3RfZmlyc3RfcnVuX2V4Y2VlZF9mYWlsdXJlX2xpbWl0KHNlbGYpOgogICAgICAgICBzZWxm LmNvbmZpZ3VyZVN0ZXAoKQogICAgICAgICBzZWxmLnNldFByb3BlcnR5KCdmaXJzdF9yZXN1bHRz X2V4Y2VlZF9mYWlsdXJlX2xpbWl0JywgVHJ1ZSkK