207504 2020-02-10 13:40:10 -0800 [ews] Display flaky layout test names in build summary 2020-04-29 03:12:18 -0700 1 1 1 Unclassified WebKit Tools / Tests Other Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=207050 https://bugs.webkit.org/show_bug.cgi?id=210855 InRadar P2 Normal --- 1 aakash_jain aakash_jain aakash_jain ap jbedard webkit-bug-importer oldest_to_newest 1617077 0 aakash_jain 2020-02-10 13:40:10 -0800 When WebKitTest (layout-tests) fail, we re-run them. If some tests fail in re-run as well, we run the tests on clean tree. If there are pre-existing test failures, the build summary is set to something like: 'Found 1 pre-existing test failure: test1.' It would be better to include the flaky test name as well in the build summary, if there were any flaky failures. e.g.: 'Found 1 pre-existing test failure: test1, Found flaky tests: test2, test3'. This would help bot-watchers easily notice flaky tests (and the pattern). 1617083 1 390289 aakash_jain 2020-02-10 13:47:05 -0800 Created attachment 390289 Patch 1617084 2 aakash_jain 2020-02-10 13:48:20 -0800 Sample runs: https://ews-build.webkit-uat.org/#/builders/16/builds/85 https://ews-build.webkit-uat.org/#/builders/16/builds/84 1617103 3 aakash_jain 2020-02-10 14:09:01 -0800 Committed r256212: <https://trac.webkit.org/changeset/256212> 1617104 4 webkit-bug-importer 2020-02-10 14:09:15 -0800 <rdar://problem/59324970> 390289 2020-02-10 13:47:05 -0800 2020-02-10 14:06:26 -0800 Patch bug-207504-20200210164704.patch text/plain 4977 aakash_jain SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDI1NjIxMCkKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDE1IEBACisyMDIwLTAyLTEwICBBYWthc2ggSmFpbiAgPGFha2FzaF9qYWluQGFwcGxlLmNv bT4KKworICAgICAgICBbZXdzXSBEaXNwbGF5IGZsYWt5IGxheW91dCB0ZXN0IG5hbWVzIGluIGJ1 aWxkIHN1bW1hcnkKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dp P2lkPTIwNzUwNAorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisgICAg ICAgICogQnVpbGRTbGF2ZVN1cHBvcnQvZXdzLWJ1aWxkL3N0ZXBzLnB5OgorICAgICAgICAoUmVS dW5XZWJLaXRUZXN0cy5ldmFsdWF0ZUNvbW1hbmQpOiBEcml2ZS1ieSBmaXggdG8gY29ycmVjdCB0 aGUgcGx1cmFsaXphdGlvbi4KKyAgICAgICAgKEFuYWx5emVMYXlvdXRUZXN0c1Jlc3VsdHMucmVw b3J0X3ByZV9leGlzdGluZ19mYWlsdXJlcyk6IEFwcGVuZCB0aGUgZmxha3kgZmFpbHVyZSBpbmZv cm1hdGlvbi4KKyAgICAgICAgKEFuYWx5emVMYXlvdXRUZXN0c1Jlc3VsdHMuc3RhcnQpOgorCiAy MDIwLTAyLTEwICBBYWthc2ggSmFpbiAgPGFha2FzaF9qYWluQGFwcGxlLmNvbT4KIAogICAgICAg ICBbZXdzXSBVc2UgU2V0QnVpbGRTdW1tYXJ5IGluc3RlYWQgb2YgYnVpbGRGaW5pc2hlZCBpbiBM YXlvdXQgdGVzdHMKSW5kZXg6IFRvb2xzL0J1aWxkU2xhdmVTdXBwb3J0L2V3cy1idWlsZC9zdGVw cy5weQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09Ci0tLSBUb29scy9CdWlsZFNsYXZlU3VwcG9ydC9ld3MtYnVpbGQvc3Rl cHMucHkJKHJldmlzaW9uIDI1NjIxMCkKKysrIFRvb2xzL0J1aWxkU2xhdmVTdXBwb3J0L2V3cy1i dWlsZC9zdGVwcy5weQkod29ya2luZyBjb3B5KQpAQCAtMTQ3NCw3ICsxNDc0LDggQEAgY2xhc3Mg UmVSdW5XZWJLaXRUZXN0cyhSdW5XZWJLaXRUZXN0cyk6CiAgICAgICAgICAgICBzZWxmLmRlc2Ny aXB0aW9uRG9uZSA9IG1lc3NhZ2UKICAgICAgICAgICAgIHNlbGYuYnVpbGQucmVzdWx0cyA9IFNV Q0NFU1MKICAgICAgICAgICAgIGlmIG5vdCBmaXJzdF9yZXN1bHRzX2RpZF9leGNlZWRfdGVzdF9m YWlsdXJlX2xpbWl0OgotICAgICAgICAgICAgICAgIG1lc3NhZ2UgPSAnRm91bmQgZmxha3kgdGVz dHM6IHt9Jy5mb3JtYXQoZmxha3lfZmFpbHVyZXNfc3RyaW5nKQorICAgICAgICAgICAgICAgIHBs dXJhbFN1ZmZpeCA9ICdzJyBpZiBsZW4oZmxha3lfZmFpbHVyZXMpID4gMSBlbHNlICcnCisgICAg ICAgICAgICAgICAgbWVzc2FnZSA9ICdGb3VuZCBmbGFreSB0ZXN0e306IHt9Jy5mb3JtYXQocGx1 cmFsU3VmZml4LCBmbGFreV9mYWlsdXJlc19zdHJpbmcpCiAgICAgICAgICAgICBzZWxmLnNldFBy b3BlcnR5KCdidWlsZF9zdW1tYXJ5JywgbWVzc2FnZSkKICAgICAgICAgZWxzZToKICAgICAgICAg ICAgIHNlbGYuc2V0UHJvcGVydHkoJ3BhdGNoRmFpbGVkVGVzdHMnLCBUcnVlKQpAQCAtMTUzOCwx MyArMTUzOSwxOSBAQCBjbGFzcyBBbmFseXplTGF5b3V0VGVzdHNSZXN1bHRzKGJ1aWxkc3RlCiAg ICAgICAgIHNlbGYuYnVpbGQuYnVpbGRGaW5pc2hlZChbbWVzc2FnZV0sIEZBSUxVUkUpCiAgICAg ICAgIHJldHVybiBkZWZlci5zdWNjZWVkKE5vbmUpCiAKLSAgICBkZWYgcmVwb3J0X3ByZV9leGlz dGluZ19mYWlsdXJlcyhzZWxmLCBjbGVhbl90cmVlX2ZhaWx1cmVzKToKKyAgICBkZWYgcmVwb3J0 X3ByZV9leGlzdGluZ19mYWlsdXJlcyhzZWxmLCBjbGVhbl90cmVlX2ZhaWx1cmVzLCBmbGFreV9m YWlsdXJlcyk6CiAgICAgICAgIHNlbGYuZmluaXNoZWQoU1VDQ0VTUykKICAgICAgICAgc2VsZi5i dWlsZC5yZXN1bHRzID0gU1VDQ0VTUwogICAgICAgICBzZWxmLmRlc2NyaXB0aW9uRG9uZSA9ICdQ YXNzZWQgbGF5b3V0IHRlc3RzJwotICAgICAgICBwbHVyYWxTdWZmaXggPSAncycgaWYgbGVuKGNs ZWFuX3RyZWVfZmFpbHVyZXMpID4gMSBlbHNlICcnCi0gICAgICAgIGNsZWFuX3RyZWVfZmFpbHVy ZXNfc3RyaW5nID0gJywgJy5qb2luKFtmYWlsdXJlX25hbWUgZm9yIGZhaWx1cmVfbmFtZSBpbiBj bGVhbl90cmVlX2ZhaWx1cmVzXSkKLSAgICAgICAgbWVzc2FnZSA9ICdGb3VuZCB7fSBwcmUtZXhp c3RpbmcgdGVzdCBmYWlsdXJle306IHt9Jy5mb3JtYXQobGVuKGNsZWFuX3RyZWVfZmFpbHVyZXMp LCBwbHVyYWxTdWZmaXgsIGNsZWFuX3RyZWVfZmFpbHVyZXNfc3RyaW5nKQorICAgICAgICBtZXNz YWdlID0gJycKKyAgICAgICAgaWYgY2xlYW5fdHJlZV9mYWlsdXJlczoKKyAgICAgICAgICAgIGNs ZWFuX3RyZWVfZmFpbHVyZXNfc3RyaW5nID0gJywgJy5qb2luKFtmYWlsdXJlX25hbWUgZm9yIGZh aWx1cmVfbmFtZSBpbiBjbGVhbl90cmVlX2ZhaWx1cmVzXSkKKyAgICAgICAgICAgIHBsdXJhbFN1 ZmZpeCA9ICdzJyBpZiBsZW4oY2xlYW5fdHJlZV9mYWlsdXJlcykgPiAxIGVsc2UgJycKKyAgICAg ICAgICAgIG1lc3NhZ2UgPSAnRm91bmQge30gcHJlLWV4aXN0aW5nIHRlc3QgZmFpbHVyZXt9OiB7 fScuZm9ybWF0KGxlbihjbGVhbl90cmVlX2ZhaWx1cmVzKSwgcGx1cmFsU3VmZml4LCBjbGVhbl90 cmVlX2ZhaWx1cmVzX3N0cmluZykKKyAgICAgICAgaWYgZmxha3lfZmFpbHVyZXM6CisgICAgICAg ICAgICBmbGFreV9mYWlsdXJlc19zdHJpbmcgPSAnLCAnLmpvaW4oZmxha3lfZmFpbHVyZXMpCisg ICAgICAgICAgICBwbHVyYWxTdWZmaXggPSAncycgaWYgbGVuKGZsYWt5X2ZhaWx1cmVzKSA+IDEg ZWxzZSAnJworICAgICAgICAgICAgbWVzc2FnZSArPSAnIEZvdW5kIGZsYWt5IHRlc3R7fToge30n LmZvcm1hdChwbHVyYWxTdWZmaXgsIGZsYWt5X2ZhaWx1cmVzX3N0cmluZykKICAgICAgICAgc2Vs Zi5zZXRQcm9wZXJ0eSgnYnVpbGRfc3VtbWFyeScsIG1lc3NhZ2UpCiAgICAgICAgIHJldHVybiBk ZWZlci5zdWNjZWVkKE5vbmUpCiAKQEAgLTE1NjksNiArMTU3Niw3IEBAIGNsYXNzIEFuYWx5emVM YXlvdXRUZXN0c1Jlc3VsdHMoYnVpbGRzdGUKICAgICAgICAgc2Vjb25kX3Jlc3VsdHNfZmFpbGlu Z190ZXN0cyA9IHNldChzZWxmLmdldFByb3BlcnR5KCdzZWNvbmRfcnVuX2ZhaWx1cmVzJywgW10p KQogICAgICAgICBjbGVhbl90cmVlX3Jlc3VsdHNfZGlkX2V4Y2VlZF90ZXN0X2ZhaWx1cmVfbGlt aXQgPSBzZWxmLmdldFByb3BlcnR5KCdjbGVhbl90cmVlX3Jlc3VsdHNfZXhjZWVkX2ZhaWx1cmVf bGltaXQnKQogICAgICAgICBjbGVhbl90cmVlX3Jlc3VsdHNfZmFpbGluZ190ZXN0cyA9IHNldChz ZWxmLmdldFByb3BlcnR5KCdjbGVhbl90cmVlX3J1bl9mYWlsdXJlcycsIFtdKSkKKyAgICAgICAg Zmxha3lfZmFpbHVyZXMgPSBmaXJzdF9yZXN1bHRzX2ZhaWxpbmdfdGVzdHMudW5pb24oc2Vjb25k X3Jlc3VsdHNfZmFpbGluZ190ZXN0cykgLSBmaXJzdF9yZXN1bHRzX2ZhaWxpbmdfdGVzdHMuaW50 ZXJzZWN0aW9uKHNlY29uZF9yZXN1bHRzX2ZhaWxpbmdfdGVzdHMpCiAKICAgICAgICAgaWYgZmly c3RfcmVzdWx0c19kaWRfZXhjZWVkX3Rlc3RfZmFpbHVyZV9saW1pdCBhbmQgc2Vjb25kX3Jlc3Vs dHNfZGlkX2V4Y2VlZF90ZXN0X2ZhaWx1cmVfbGltaXQ6CiAgICAgICAgICAgICBpZiAobGVuKGZp cnN0X3Jlc3VsdHNfZmFpbGluZ190ZXN0cykgLSBsZW4oY2xlYW5fdHJlZV9yZXN1bHRzX2ZhaWxp bmdfdGVzdHMpKSA8PSA1OgpAQCAtMTYxMiw3ICsxNjIwLDcgQEAgY2xhc3MgQW5hbHl6ZUxheW91 dFRlc3RzUmVzdWx0cyhidWlsZHN0ZQogICAgICAgICAgICAgIyBBdCB0aGlzIHBvaW50IHdlIGtu b3cgdGhhdCBhdCBsZWFzdCBvbmUgdGVzdCBmbGFrZWQsIGJ1dCBubyBjb25zaXN0ZW50IGZhaWx1 cmVzCiAgICAgICAgICAgICAjIHdlcmUgaW50cm9kdWNlZC4gVGhpcyBpcyBhIGJpdCBvZiBhIGdy ZXktem9uZS4gSXQncyBwb3NzaWJsZSB0aGF0IHRoZSBwYXRjaCBpbnRyb2R1Y2VkIHNvbWUgZmxh a2luZXNzLgogICAgICAgICAgICAgIyBXZSBzdGlsbCBtYXJrIHRoZSBidWlsZCBhcyBTVUNDRVNT LgotICAgICAgICAgICAgcmV0dXJuIHNlbGYucmVwb3J0X3ByZV9leGlzdGluZ19mYWlsdXJlcyhj bGVhbl90cmVlX3Jlc3VsdHNfZmFpbGluZ190ZXN0cykKKyAgICAgICAgICAgIHJldHVybiBzZWxm LnJlcG9ydF9wcmVfZXhpc3RpbmdfZmFpbHVyZXMoY2xlYW5fdHJlZV9yZXN1bHRzX2ZhaWxpbmdf dGVzdHMsIGZsYWt5X2ZhaWx1cmVzKQogCiAgICAgICAgIGlmIGNsZWFuX3RyZWVfcmVzdWx0c19k aWRfZXhjZWVkX3Rlc3RfZmFpbHVyZV9saW1pdDoKICAgICAgICAgICAgIHJldHVybiBzZWxmLnJl dHJ5X2J1aWxkKCkKQEAgLTE2MjMsNyArMTYzMSw3IEBAIGNsYXNzIEFuYWx5emVMYXlvdXRUZXN0 c1Jlc3VsdHMoYnVpbGRzdGUKICAgICAgICAgIyBBdCB0aGlzIHBvaW50LCB3ZSBrbm93IHRoYXQg dGhlIGZpcnN0IGFuZCBzZWNvbmQgcnVucyBoYWQgdGhlIGV4YWN0IHNhbWUgZmFpbHVyZXMsCiAg ICAgICAgICMgYW5kIHRoYXQgdGhvc2UgZmFpbHVyZXMgYXJlIGFsbCBwcmVzZW50IG9uIHRoZSBj bGVhbiB0cmVlLCBzbyB3ZSBjYW4gc2F5IHdpdGggY2VydGFpbnR5CiAgICAgICAgICMgdGhhdCB0 aGUgcGF0Y2ggaXMgZ29vZC4KLSAgICAgICAgcmV0dXJuIHNlbGYucmVwb3J0X3ByZV9leGlzdGlu Z19mYWlsdXJlcyhjbGVhbl90cmVlX3Jlc3VsdHNfZmFpbGluZ190ZXN0cykKKyAgICAgICAgcmV0 dXJuIHNlbGYucmVwb3J0X3ByZV9leGlzdGluZ19mYWlsdXJlcyhjbGVhbl90cmVlX3Jlc3VsdHNf ZmFpbGluZ190ZXN0cywgZmxha3lfZmFpbHVyZXMpCiAKIAogY2xhc3MgUnVuV2ViS2l0MVRlc3Rz KFJ1bldlYktpdFRlc3RzKToK