231790 2021-10-14 18:22:04 -0700 [webkitpy] webkit-test-runner doesn't report all results when a test is run several times 2021-10-25 08:17:56 -0700 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=229788 https://bugs.webkit.org/show_bug.cgi?id=231241 https://bugs.webkit.org/show_bug.cgi?id=231999 InRadar P2 Normal --- 1 clopez clopez aakash_jain ap clopez dewei_zhu ews-watchlist glenn jbedard ryanhaddad webkit-bug-importer oldest_to_newest 1804634 0 clopez 2021-10-14 18:22:04 -0700 webkit-test-runner is not correctly reporting flaky tests when a test is run more than once (for example, when passing the flag --repeat-each=X or when specifiyng the test twice or more on the command line). This is due to how webkit-test-runner accounts the test result values This is what is happening when passing the flag --repeat-each=X is passed: 1. Run the test X times and if it fails on any of those times, annotate one of the failure types (for example: timeout) 2. Repeat the test one run more (on the retry step) and if it gives any different result than the failure previously annotated then mark the test as " Valuefrom_1 | Valuefrom_retry" And this has several issues - The first issue is that when on the step 1 a test gives different results like: "Pass, Timeout, Fail" webkit-test-runner only picks one of the failures. For example, for a test that gives - Pass - Pass - Timeout - Failure - Pass - Failure - Timeout - Pass It will pick "Timeout" as the value from the first run. Then on the retry step, suppose the test gives "Pass" So webkit-test-runner will mark the test as flaky "[ Timeout Pass ]", which is wrong as it has ignored completely the "Failure" value. - And the second issue happens when you pass the flag "--no-retry-failures". Since there is no retry step then it just picks the last failure value. So on the previous example it will just mark the test as "[ Timeout ]" and it will not report the test as flaky. You can verify the current bug with the following patch that adds a test named fast/random/fails-timeout-pass.html that will "Pass|Fail|Timeout" with equal probability (1/3 of the times) So to test this apply the patch: http://sprunge.us/88Ks7Y and run webkit-test-runner as follows: 1. $ Tools/Scripts/run-webkit-tests --debug-rwt-logging --release --repeat-each=10 --time-out-ms=1000 fast/random/fails-timeout-pass.html You should see the above behaviour (it picks only one failure value from the first try and then the value of the retry step (if different)) 2. Now try to run it passing "--no-retry-failures" $ Tools/Scripts/run-webkit-tests --debug-rwt-logging --release --repeat-each=10 --time-out-ms=1000 --no-retry-failures fast/random/fails-timeout-pass.html And you will see that it just picks only one failure value, ignoring any pass and any other different failure value Note: if you get a python KeyError exception then apply the fix from bug 229788 (if still didn't landed) See for example this test run that just happened here http://sprunge.us/CRNNCn : - On the first 10 retries it gave "pass, pass, fail, pass, fail, timeout, timeout, pass, fail, timeout" and on the retry step it gave "timeout" and WTR marked this as [ Timeout ] (no flaky) 1804640 1 441316 clopez 2021-10-14 18:43:16 -0700 Created attachment 441316 Patch 1807488 2 webkit-bug-importer 2021-10-21 18:23:12 -0700 <rdar://problem/84531445> 1808203 3 clopez 2021-10-24 05:32:56 -0700 ping reviewers? 1808407 4 ews-feeder 2021-10-25 08:17:53 -0700 Committed r284784 (243493@main): <https://commits.webkit.org/243493@main> All reviewed patches have been landed. Closing bug and clearing flags on attachment 441316. 441316 2021-10-14 18:43:16 -0700 2021-10-25 08:17:54 -0700 Patch bug-231790-20211015024230.patch text/plain 5013 clopez U3VidmVyc2lvbiBSZXZpc2lvbjogMjg0MjIyCmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggYjVjNTgwZGFiNjNhNjg4YWVmYzQ2NzkxN2EwN2VhZjE0 YzkzYmNjNy4uNmU4MmU3Y2QzZmU4Nzk5ZDExMDk5ZGRiNGVmMDExN2MzY2JmNDU0ZSAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDI1 IEBACisyMDIxLTEwLTE0ICBDYXJsb3MgQWxiZXJ0byBMb3BleiBQZXJleiAgPGNsb3BlekBpZ2Fs aWEuY29tPgorCisgICAgICAgIFt3ZWJraXRweV0gd2Via2l0LXRlc3QtcnVubmVyIGRvZXNuJ3Qg cmVwb3J0IGFsbCByZXN1bHRzIHdoZW4gYSB0ZXN0IGlzIHJ1biBzZXZlcmFsIHRpbWVzCisgICAg ICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0yMzE3OTAKKworICAg ICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICBXVFIgd2FzIG9ubHkg cGlja2luZyBvbmUgb2YgdGhlIGZhaWx1cmUgcmVzdWx0cyB3aGVuIGEgdGVzdCB3YXMgcnVuIG1v cmUgdGhhbiBvbmNlCisgICAgICAgIChmb3IgZXhhbXBsZSB3aXRoIHRoZSBmbGFnIC0tcmVwZWF0 LWVhY2g9WCksIHNvIGl0IHdhcyBub3QgcmVwb3J0aW5nIGFsbCB0aGUgdmFsdWVzCisgICAgICAg IHRoYXQgdGhlIHRlc3QgZ2VuZXJhdGVkLiBUaGlzIGlzIGEgbWFqb3IgaXNzdWUgd2hlbiBzZWFy Y2hpbmcgZm9yIGZsYWt5IHRlc3RzLgorCisgICAgICAgIFRoaXMgcGF0Y2ggYWRkcyBhIGRpY3Rp b25hcnkgdG8gc3RvcmUgYWxsIHRoZSByZXN1bHRzIG9mIGEgZ2l2ZW4gdGVzdCBvbiB0aGUgcmVw ZWF0ZWQKKyAgICAgICAgcmVwZWF0cyBhbmQgdGhlbiBpdCBlbnN1cmVzIHRob3NlIHZhbHVlcyBh cmUgdGFrZW4gaW50byBhY2NvdW50IHdoZW4gcmVwb3J0aW5nIHRoZQorICAgICAgICBmaW5hbCBy ZXN1bHRzIGZvciB0aGUgdGVzdC4gSXQgbWFya3MgdGhlIHRlc3QgYXMgZmxha3kgaWYgbW9yZSB0 aGFuIG9uZSBkaWZmZXJlbnQgdmFsdWUKKyAgICAgICAgd2FzIGdlbmVyYXRlZC4KKworICAgICAg ICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL21vZGVscy90ZXN0X3J1bl9yZXN1bHRz LnB5OgorICAgICAgICAoVGVzdFJ1blJlc3VsdHMuX19pbml0X18pOgorICAgICAgICAoVGVzdFJ1 blJlc3VsdHMuYWRkKToKKyAgICAgICAgKFRlc3RSdW5SZXN1bHRzLm1lcmdlKToKKyAgICAgICAg KHN1bW1hcml6ZV9yZXN1bHRzKToKKwogMjAyMS0xMC0xNCAgQWxleCBDaHJpc3RlbnNlbiAgPGFj aHJpc3RlbnNlbkB3ZWJraXQub3JnPgogCiAgICAgICAgIEZpeCBzb21lIGlPUyBidWlsZHMgYWZ0 ZXIgcjI4NDIwMwpkaWZmIC0tZ2l0IGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVz dHMvbW9kZWxzL3Rlc3RfcnVuX3Jlc3VsdHMucHkgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xh eW91dF90ZXN0cy9tb2RlbHMvdGVzdF9ydW5fcmVzdWx0cy5weQppbmRleCAyODgxYzc2MGE4ZTU5 Y2E3NzVkODYxNGQ5YTNiYzNmZDFkOGZmMTBkLi5lYWIzYmQ4ZjgyY2RlNmY5YmNiOWZjMTE1YTg2 ZjczNWVlNGRhMzZjIDEwMDY0NAotLS0gYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90 ZXN0cy9tb2RlbHMvdGVzdF9ydW5fcmVzdWx0cy5weQorKysgYi9Ub29scy9TY3JpcHRzL3dlYmtp dHB5L2xheW91dF90ZXN0cy9tb2RlbHMvdGVzdF9ydW5fcmVzdWx0cy5weQpAQCAtNTgsNiArNTgs NyBAQCBjbGFzcyBUZXN0UnVuUmVzdWx0cyhvYmplY3QpOgogICAgICAgICBzZWxmLmV4cGVjdGVk X3Jlc3VsdHNfYnlfbmFtZSA9IHt9CiAgICAgICAgIHNlbGYudW5leHBlY3RlZF9yZXN1bHRzX2J5 X25hbWUgPSB7fQogICAgICAgICBzZWxmLmZhaWx1cmVzX2J5X25hbWUgPSB7fQorICAgICAgICBz ZWxmLnJlcGVhdGVkX3Jlc3VsdHNfYnlfbmFtZSA9IHt9ICAjIE1hcCBvZiB0ZXN0IG5hbWUgdG8g YSBsaXN0IG9mIHJlc3VsdHMsIHdoZW4gYSB0ZXN0cyBpcyBydW4gbW9yZSB0aGFuIG9uY2UgKGxp a2Ugd2hlbiBwYXNzaW5nIC0tcmVwZWF0LWVhY2gpCiAgICAgICAgIHNlbGYudG90YWxfZmFpbHVy ZXMgPSAwCiAgICAgICAgIHNlbGYuZXhwZWN0ZWRfc2tpcHMgPSAwCiAgICAgICAgIGZvciBleHBl Y3RhdGlvbiBpbiB0ZXN0X2V4cGVjdGF0aW9ucy5UZXN0RXhwZWN0YXRpb25zLkVYUEVDVEFUSU9O Uy52YWx1ZXMoKToKQEAgLTcwLDYgKzcxLDkgQEAgY2xhc3MgVGVzdFJ1blJlc3VsdHMob2JqZWN0 KToKIAogICAgIGRlZiBhZGQoc2VsZiwgdGVzdF9yZXN1bHQsIGV4cGVjdGVkKToKICAgICAgICAg c2VsZi50ZXN0c19ieV9leHBlY3RhdGlvblt0ZXN0X3Jlc3VsdC50eXBlXS5hZGQodGVzdF9yZXN1 bHQudGVzdF9uYW1lKQorICAgICAgICBpZiB0ZXN0X3Jlc3VsdC50ZXN0X25hbWUgbm90IGluIHNl bGYucmVwZWF0ZWRfcmVzdWx0c19ieV9uYW1lOgorICAgICAgICAgICAgc2VsZi5yZXBlYXRlZF9y ZXN1bHRzX2J5X25hbWVbdGVzdF9yZXN1bHQudGVzdF9uYW1lXSA9IHNldCgpCisgICAgICAgIHNl bGYucmVwZWF0ZWRfcmVzdWx0c19ieV9uYW1lW3Rlc3RfcmVzdWx0LnRlc3RfbmFtZV0uYWRkKHRl c3RfcmVzdWx0LnR5cGUpCiAgICAgICAgIHNlbGYucmVzdWx0c19ieV9uYW1lW3Rlc3RfcmVzdWx0 LnRlc3RfbmFtZV0gPSBzZWxmLnJlc3VsdHNfYnlfbmFtZS5nZXQodGVzdF9yZXN1bHQudGVzdF9u YW1lLCB0ZXN0X3Jlc3VsdCkKICAgICAgICAgaWYgdGVzdF9yZXN1bHQuaXNfb3RoZXJfY3Jhc2g6 CiAgICAgICAgICAgICByZXR1cm4KQEAgLTE1OCw2ICsxNjIsNyBAQCBjbGFzcyBUZXN0UnVuUmVz dWx0cyhvYmplY3QpOgogICAgICAgICBzZWxmLnVuZXhwZWN0ZWRfdGltZW91dHMgKz0gdGVzdF9y dW5fcmVzdWx0cy51bmV4cGVjdGVkX3RpbWVvdXRzCiAgICAgICAgIHNlbGYudGVzdHNfYnlfZXhw ZWN0YXRpb24gPSBtZXJnZV9kaWN0X3NldHMoc2VsZi50ZXN0c19ieV9leHBlY3RhdGlvbiwgdGVz dF9ydW5fcmVzdWx0cy50ZXN0c19ieV9leHBlY3RhdGlvbikKICAgICAgICAgc2VsZi50ZXN0c19i eV90aW1lbGluZSA9IG1lcmdlX2RpY3Rfc2V0cyhzZWxmLnRlc3RzX2J5X3RpbWVsaW5lLCB0ZXN0 X3J1bl9yZXN1bHRzLnRlc3RzX2J5X3RpbWVsaW5lKQorICAgICAgICBzZWxmLnJlcGVhdGVkX3Jl c3VsdHNfYnlfbmFtZSA9IG1lcmdlX2RpY3Rfc2V0cyhzZWxmLnJlcGVhdGVkX3Jlc3VsdHNfYnlf bmFtZSwgdGVzdF9ydW5fcmVzdWx0cy5yZXBlYXRlZF9yZXN1bHRzX2J5X25hbWUpCiAgICAgICAg IHNlbGYucmVzdWx0c19ieV9uYW1lLnVwZGF0ZSh0ZXN0X3J1bl9yZXN1bHRzLnJlc3VsdHNfYnlf bmFtZSkKICAgICAgICAgc2VsZi5hbGxfcmVzdWx0cyArPSB0ZXN0X3J1bl9yZXN1bHRzLmFsbF9y ZXN1bHRzCiAgICAgICAgIHNlbGYuZXhwZWN0ZWRfcmVzdWx0c19ieV9uYW1lLnVwZGF0ZSh0ZXN0 X3J1bl9yZXN1bHRzLmV4cGVjdGVkX3Jlc3VsdHNfYnlfbmFtZSkKQEAgLTMxNSwxMyArMzIwLDI0 IEBAIGRlZiBzdW1tYXJpemVfcmVzdWx0cyhwb3J0X29iaiwgZXhwZWN0YXRpb25zX2J5X3R5cGUs IGluaXRpYWxfcmVzdWx0cywgcmV0cnlfcmVzCiAgICAgICAgICAgICAgICAgICAgIG51bV9yZWdy ZXNzaW9ucyArPSAxCiAgICAgICAgICAgICAgICAgICAgIHRlc3RfZGljdFsncmVwb3J0J10gPSAn UkVHUkVTU0lPTicKICAgICAgICAgICAgIGVsaWYgcmV0cnlfcmVzdWx0cyBhbmQgdGVzdF9uYW1l IGluIHJldHJ5X3Jlc3VsdHMuZXhwZWN0ZWRfcmVzdWx0c19ieV9uYW1lOgotICAgICAgICAgICAg ICAgIGFjdHVhbC5hcHBlbmQoa2V5d29yZHNbcmV0cnlfcmVzdWx0cy5leHBlY3RlZF9yZXN1bHRz X2J5X25hbWVbdGVzdF9uYW1lXS50eXBlXSkKKyAgICAgICAgICAgICAgICByZXRyeV9yZXN1bHRf bmFtZSA9IGtleXdvcmRzW3JldHJ5X3Jlc3VsdHMuZXhwZWN0ZWRfcmVzdWx0c19ieV9uYW1lW3Rl c3RfbmFtZV0udHlwZV0KKyAgICAgICAgICAgICAgICBpZiByZXRyeV9yZXN1bHRfbmFtZSBub3Qg aW4gYWN0dWFsOgorICAgICAgICAgICAgICAgICAgICBhY3R1YWwuYXBwZW5kKHJldHJ5X3Jlc3Vs dF9uYW1lKQogICAgICAgICAgICAgICAgIG51bV9mbGFreSArPSAxCiAgICAgICAgICAgICAgICAg dGVzdF9kaWN0WydyZXBvcnQnXSA9ICdGTEFLWScKICAgICAgICAgICAgIGVsc2U6CiAgICAgICAg ICAgICAgICAgbnVtX3JlZ3Jlc3Npb25zICs9IDEKICAgICAgICAgICAgICAgICB0ZXN0X2RpY3Rb J3JlcG9ydCddID0gJ1JFR1JFU1NJT04nCiAKKyAgICAgICAgIyBJZiBhIHRlc3Qgd2FzIHJ1biBt b3JlIHRoYW4gb25jZSBvbiB0aGUgaW5pdGlhbF9yZXN1bHRzIChmb3IgZXhhbXBsZSB3aXRoIC0t cmVwZWF0LWVhY2gpLAorICAgICAgICAjIGNoZWNrIGZvciBwb3NzaWJsZSBmbGFraW5lc3MgdGhl cmUgYW5kIGFsc28gYWNjb3VudCBpbiAiYWN0dWFsIiBmb3IgdGhlIGV4dHJhIHJlc3VsdHMuCisg ICAgICAgIGZvciByZXBlYXRlZF9yZXN1bHQgaW4gaW5pdGlhbF9yZXN1bHRzLnJlcGVhdGVkX3Jl c3VsdHNfYnlfbmFtZVt0ZXN0X25hbWVdOgorICAgICAgICAgICAgcmVwZWF0ZWRfcmVzdWx0X25h bWUgPSBrZXl3b3Jkc1tyZXBlYXRlZF9yZXN1bHRdCisgICAgICAgICAgICBpZiByZXBlYXRlZF9y ZXN1bHRfbmFtZSBub3QgaW4gYWN0dWFsOgorICAgICAgICAgICAgICAgIGFjdHVhbC5hcHBlbmQo cmVwZWF0ZWRfcmVzdWx0X25hbWUpCisgICAgICAgICAgICAgICAgaWYgdGVzdF9uYW1lIGluIGlu aXRpYWxfcmVzdWx0cy51bmV4cGVjdGVkX3Jlc3VsdHNfYnlfbmFtZToKKyAgICAgICAgICAgICAg ICAgICAgdGVzdF9kaWN0WydyZXBvcnQnXSA9ICdGTEFLWScKKwogICAgICAgICB0ZXN0X2RpY3Rb J2V4cGVjdGVkJ10gPSBleHBlY3RlZAogICAgICAgICB0ZXN0X2RpY3RbJ2FjdHVhbCddID0gIiAi LmpvaW4oYWN0dWFsKQogICAgICAgICBpZiBpbmNsdWRlX3RpbWVfYW5kX21vZGlmaWVyczoK