231241 2021-10-05 11:12:45 -0700 [webkitpy] The actual results reported for a flaky tests shouldn't include the expectation 2021-10-14 18:42:38 -0700 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=231790 InRadar P2 Normal --- 1 clopez clopez aakash_jain ap dewei_zhu ews-watchlist glenn jbedard ryanhaddad webkit-bug-importer oldest_to_newest 1800651 0 clopez 2021-10-05 11:12:45 -0700 This is about the following corner-case -> When a test is marked as flaky and fails the run (doesn't match the expectation) but passes the run on the retry. In this case, at the end of the run we currently report as actual results of the tests the union between the first run and the expectations. So it happens that, for example, when we have a test marked as "[ Crash Timeout Pass ]" and the fail gives the results "Fail" and "Pass" we currently report the actual results of the tests to be "[ Fail Crash Timeout Pass ]" when "Crash" and "Timeout" didn't happened. So this makes difficult to know when a tests stops giving a specific result. Once it is marked as flaky with a set of results there is no way to know if some of the results in the set not longer happen. Here is a test-case to test this: 1. Apply this test patch <http://sprunge.us/MqyGkc> to add a test that fails the 50% of times and passes the other 50% and such test is marked with an expectation of fast/random/fails-half.html [ Crash Timeout Pass ] 2. Run the test like: $ Tools/Scripts/run-webkit-tests --no-build --no-show-results --release --results-directory layout-test-results --debug-rwt-logging fast/random/fails-half.html Repeat the run above until you get the test to fail on the first run and pass on the second. When that happens you will see this <http://sprunge.us/vwM9B7>: Unexpected flakiness: text-only failures (1) fast/random/fails-half.html [ Crash Timeout Pass Failure ] And on the json file at layout-test-results/full_results.json you will see: ADD_RESULTS({"tests":{"fast":{"random":{"fails-half.html":{"report":"FLAKY","expected":"PASS TIMEOUT CRASH","actual":"TEXT PASS TIMEOUT CRASH"}}}} So I think this is misleading, because the _actual_ result of this test was "FAIL/TEXT" and "PASS" but it didn't give TIMEOUT or CRASH 1800682 1 440241 clopez 2021-10-05 11:42:31 -0700 Created attachment 440241 Patch Patch fixing the issue. This is the output the above example gives after this patch: http://sprunge.us/ta54tE 1803242 2 webkit-bug-importer 2021-10-12 11:13:17 -0700 <rdar://problem/84156894> 1803277 3 440241 jbedard 2021-10-12 11:49:57 -0700 Comment on attachment 440241 Patch Sorry for the delay looking at this. I was a bit worried about the --repeat case where a test has multiple results. Looks like this performs as expected in that case, though. 1804502 4 ews-feeder 2021-10-14 14:18:52 -0700 Committed r284198 (243012@main): <https://commits.webkit.org/243012@main> All reviewed patches have been landed. Closing bug and clearing flags on attachment 440241. 440241 2021-10-05 11:42:31 -0700 2021-10-14 14:18:54 -0700 Patch bug-231241-20211005194148.patch text/plain 5969 clopez U3VidmVyc2lvbiBSZXZpc2lvbjogMjgzNDkyCmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggZDUzZTdhZTFhNjJjMzc0MjVlNmRhYWY4NjViNWUwNjZl NTdiYTc0Mi4uNmQ0ZWZmZmFkNTJhNTBmYWE1NTVjZjU2M2NiMWU4ZTYxZjNlZTA3YiAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDI2 IEBACisyMDIxLTEwLTA1ICBDYXJsb3MgQWxiZXJ0byBMb3BleiBQZXJleiAgPGNsb3BlekBpZ2Fs aWEuY29tPgorCisgICAgICAgIFt3ZWJraXRweV0gVGhlIGFjdHVhbCByZXN1bHRzIHJlcG9ydGVk IGZvciBhIGZsYWt5IHRlc3RzIHNob3VsZG4ndCBpbmNsdWRlIHRoZSBleHBlY3RhdGlvbgorICAg ICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9MjMxMjQxCisKKyAg ICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISkuCisKKyAgICAgICAgV2hlbiBhIHRlc3Qg aXMgbWFya2VkIGFzIGZsYWt5IGFuZCBmYWlscyB0aGUgZXhwZWN0YXRpb25zIG9uIHRoZQorICAg ICAgICBmaXJzdCBydW4gYnV0IHBhc3NlcyBvbiB0aGUgc2Vjb25kIHJ1biAodGhlIHJldHJ5KSB0 aGUgY3VycmVudCBjb2RlCisgICAgICAgIHdhcyBhZGRpbmcgdGhlIGV4cGVjdGF0aW9ucyB0byB0 aGUgYWN0dWFsIHJlc3VsdHMuCisgICAgICAgIFRoaXMgaXMgbWlzc2xlYWRpbmcgYW5kIG1ha2Vz IHJlYWxseSBoYXJkIHRvIGRldGVjdCB3aGVuIGEgdGVzdCBzdG9wcworICAgICAgICBnaXZpbmcg YSBzcGVjaWZpYyBleHBlY3RhdGlvbi4KKworICAgICAgICBJbnN0ZWFkIG9mIGRvaW5nIHRoYXQg cmVwb3J0IHRoZSBhY3R1YWwgcmVzdWx0cyBvZiB0aGUgdGVzdCBvbiBib3RoIHJ1bnMuCisKKyAg ICAgICAgKiBTY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9tb2RlbHMvdGVzdF9ydW5fcmVz dWx0cy5weToKKyAgICAgICAgKFRlc3RSdW5SZXN1bHRzLl9faW5pdF9fKToKKyAgICAgICAgKFRl c3RSdW5SZXN1bHRzLmFkZCk6CisgICAgICAgIChUZXN0UnVuUmVzdWx0cy5tZXJnZSk6CisgICAg ICAgIChzdW1tYXJpemVfcmVzdWx0cyk6CisgICAgICAgICogU2NyaXB0cy93ZWJraXRweS9sYXlv dXRfdGVzdHMvdmlld3MvYnVpbGRib3RfcmVzdWx0cy5weToKKyAgICAgICAgKEJ1aWxkQm90UHJp bnRlci5wcmludF91bmV4cGVjdGVkX3Jlc3VsdHMpOgorCiAyMDIxLTEwLTAzICBEYXZpZCBLaWx6 ZXIgIDxkZGtpbHplckBhcHBsZS5jb20+CiAKICAgICAgICAgV1RGOjpSZXRhaW5QdHI8PiBhbGxv d3MgYXNzaWdubWVudCBvZiB0d28gcG9pbnRlciB0eXBlcyB0aGF0IGFyZSBub3QgYXNzaWduYWJs ZQpkaWZmIC0tZ2l0IGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvbW9kZWxz L3Rlc3RfcnVuX3Jlc3VsdHMucHkgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0 cy9tb2RlbHMvdGVzdF9ydW5fcmVzdWx0cy5weQppbmRleCBiODUwMzVjMDFmNDAwYWNjMGRiZmY5 NmY5YmNkOWZkYzVhOTI1OTUyLi4yODgxYzc2MGE4ZTU5Y2E3NzVkODYxNGQ5YTNiYzNmZDFkOGZm MTBkIDEwMDY0NAotLS0gYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9tb2Rl bHMvdGVzdF9ydW5fcmVzdWx0cy5weQorKysgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91 dF90ZXN0cy9tb2RlbHMvdGVzdF9ydW5fcmVzdWx0cy5weQpAQCAtNTUsNiArNTUsNyBAQCBjbGFz cyBUZXN0UnVuUmVzdWx0cyhvYmplY3QpOgogICAgICAgICBzZWxmLnRlc3RzX2J5X3RpbWVsaW5l ID0ge30KICAgICAgICAgc2VsZi5yZXN1bHRzX2J5X25hbWUgPSB7fSAgIyBNYXAgb2YgdGVzdCBu YW1lIHRvIHRoZSBsYXN0IHJlc3VsdCBmb3IgdGhlIHRlc3QuCiAgICAgICAgIHNlbGYuYWxsX3Jl c3VsdHMgPSBbXSAgIyBBbGwgcmVzdWx0cyBmcm9tIGEgcnVuLCBpbmNsdWRpbmcgZXZlcnkgaXRl cmF0aW9uIG9mIGV2ZXJ5IHRlc3QuCisgICAgICAgIHNlbGYuZXhwZWN0ZWRfcmVzdWx0c19ieV9u YW1lID0ge30KICAgICAgICAgc2VsZi51bmV4cGVjdGVkX3Jlc3VsdHNfYnlfbmFtZSA9IHt9CiAg ICAgICAgIHNlbGYuZmFpbHVyZXNfYnlfbmFtZSA9IHt9CiAgICAgICAgIHNlbGYudG90YWxfZmFp bHVyZXMgPSAwCkBAIC03OSw2ICs4MCw3IEBAIGNsYXNzIFRlc3RSdW5SZXN1bHRzKG9iamVjdCk6 CiAgICAgICAgICAgICBzZWxmLnRvdGFsX2ZhaWx1cmVzICs9IDEKICAgICAgICAgICAgIHNlbGYu ZmFpbHVyZXNfYnlfbmFtZVt0ZXN0X3Jlc3VsdC50ZXN0X25hbWVdID0gdGVzdF9yZXN1bHQuZmFp bHVyZXMKICAgICAgICAgaWYgZXhwZWN0ZWQ6CisgICAgICAgICAgICBzZWxmLmV4cGVjdGVkX3Jl c3VsdHNfYnlfbmFtZVt0ZXN0X3Jlc3VsdC50ZXN0X25hbWVdID0gdGVzdF9yZXN1bHQKICAgICAg ICAgICAgIHNlbGYuZXhwZWN0ZWQgKz0gMQogICAgICAgICAgICAgaWYgdGVzdF9yZXN1bHQudHlw ZSA9PSB0ZXN0X2V4cGVjdGF0aW9ucy5TS0lQOgogICAgICAgICAgICAgICAgIHNlbGYuZXhwZWN0 ZWRfc2tpcHMgKz0gMQpAQCAtMTU4LDYgKzE2MCw3IEBAIGNsYXNzIFRlc3RSdW5SZXN1bHRzKG9i amVjdCk6CiAgICAgICAgIHNlbGYudGVzdHNfYnlfdGltZWxpbmUgPSBtZXJnZV9kaWN0X3NldHMo c2VsZi50ZXN0c19ieV90aW1lbGluZSwgdGVzdF9ydW5fcmVzdWx0cy50ZXN0c19ieV90aW1lbGlu ZSkKICAgICAgICAgc2VsZi5yZXN1bHRzX2J5X25hbWUudXBkYXRlKHRlc3RfcnVuX3Jlc3VsdHMu cmVzdWx0c19ieV9uYW1lKQogICAgICAgICBzZWxmLmFsbF9yZXN1bHRzICs9IHRlc3RfcnVuX3Jl c3VsdHMuYWxsX3Jlc3VsdHMKKyAgICAgICAgc2VsZi5leHBlY3RlZF9yZXN1bHRzX2J5X25hbWUu dXBkYXRlKHRlc3RfcnVuX3Jlc3VsdHMuZXhwZWN0ZWRfcmVzdWx0c19ieV9uYW1lKQogICAgICAg ICBzZWxmLnVuZXhwZWN0ZWRfcmVzdWx0c19ieV9uYW1lLnVwZGF0ZSh0ZXN0X3J1bl9yZXN1bHRz LnVuZXhwZWN0ZWRfcmVzdWx0c19ieV9uYW1lKQogICAgICAgICBzZWxmLmZhaWx1cmVzX2J5X25h bWUudXBkYXRlKHRlc3RfcnVuX3Jlc3VsdHMuZmFpbHVyZXNfYnlfbmFtZSkKICAgICAgICAgc2Vs Zi50b3RhbF9mYWlsdXJlcyArPSB0ZXN0X3J1bl9yZXN1bHRzLnRvdGFsX2ZhaWx1cmVzCkBAIC0y OTYsMTEgKzI5OSw3IEBAIGRlZiBzdW1tYXJpemVfcmVzdWx0cyhwb3J0X29iaiwgZXhwZWN0YXRp b25zX2J5X3R5cGUsIGluaXRpYWxfcmVzdWx0cywgcmV0cnlfcmVzCiAgICAgICAgICAgICAgICAg bnVtX21pc3NpbmcgKz0gMQogICAgICAgICAgICAgICAgIHRlc3RfZGljdFsncmVwb3J0J10gPSAn TUlTU0lORycKICAgICAgICAgZWxpZiB0ZXN0X25hbWUgaW4gaW5pdGlhbF9yZXN1bHRzLnVuZXhw ZWN0ZWRfcmVzdWx0c19ieV9uYW1lOgotICAgICAgICAgICAgaWYgcmV0cnlfcmVzdWx0cyBhbmQg dGVzdF9uYW1lIG5vdCBpbiByZXRyeV9yZXN1bHRzLnVuZXhwZWN0ZWRfcmVzdWx0c19ieV9uYW1l OgotICAgICAgICAgICAgICAgIGFjdHVhbC5leHRlbmQoZXhwZWN0YXRpb25zLm1vZGVsKCkuZ2V0 X2V4cGVjdGF0aW9uc19zdHJpbmcodGVzdF9uYW1lKS5zcGxpdCgiICIpKQotICAgICAgICAgICAg ICAgIG51bV9mbGFreSArPSAxCi0gICAgICAgICAgICAgICAgdGVzdF9kaWN0WydyZXBvcnQnXSA9 ICdGTEFLWScKLSAgICAgICAgICAgIGVsaWYgcmV0cnlfcmVzdWx0czoKKyAgICAgICAgICAgIGlm IHJldHJ5X3Jlc3VsdHMgYW5kIHRlc3RfbmFtZSBpbiByZXRyeV9yZXN1bHRzLnVuZXhwZWN0ZWRf cmVzdWx0c19ieV9uYW1lOgogICAgICAgICAgICAgICAgIHJldHJ5X3Jlc3VsdF90eXBlID0gcmV0 cnlfcmVzdWx0cy51bmV4cGVjdGVkX3Jlc3VsdHNfYnlfbmFtZVt0ZXN0X25hbWVdLnR5cGUKICAg ICAgICAgICAgICAgICBpZiByZXN1bHRfdHlwZSAhPSByZXRyeV9yZXN1bHRfdHlwZToKICAgICAg ICAgICAgICAgICAgICAgaWYgZW5hYmxlZF9waXhlbF90ZXN0c19pbl9yZXRyeSBhbmQgcmVzdWx0 X3R5cGUgPT0gdGVzdF9leHBlY3RhdGlvbnMuVEVYVCBhbmQgKHJldHJ5X3Jlc3VsdF90eXBlID09 IHRlc3RfZXhwZWN0YXRpb25zLklNQUdFX1BMVVNfVEVYVCBvciByZXRyeV9yZXN1bHRfdHlwZSA9 PSB0ZXN0X2V4cGVjdGF0aW9ucy5NSVNTSU5HKToKQEAgLTMxNSw2ICszMTQsMTAgQEAgZGVmIHN1 bW1hcml6ZV9yZXN1bHRzKHBvcnRfb2JqLCBleHBlY3RhdGlvbnNfYnlfdHlwZSwgaW5pdGlhbF9y ZXN1bHRzLCByZXRyeV9yZXMKICAgICAgICAgICAgICAgICBlbHNlOgogICAgICAgICAgICAgICAg ICAgICBudW1fcmVncmVzc2lvbnMgKz0gMQogICAgICAgICAgICAgICAgICAgICB0ZXN0X2RpY3Rb J3JlcG9ydCddID0gJ1JFR1JFU1NJT04nCisgICAgICAgICAgICBlbGlmIHJldHJ5X3Jlc3VsdHMg YW5kIHRlc3RfbmFtZSBpbiByZXRyeV9yZXN1bHRzLmV4cGVjdGVkX3Jlc3VsdHNfYnlfbmFtZToK KyAgICAgICAgICAgICAgICBhY3R1YWwuYXBwZW5kKGtleXdvcmRzW3JldHJ5X3Jlc3VsdHMuZXhw ZWN0ZWRfcmVzdWx0c19ieV9uYW1lW3Rlc3RfbmFtZV0udHlwZV0pCisgICAgICAgICAgICAgICAg bnVtX2ZsYWt5ICs9IDEKKyAgICAgICAgICAgICAgICB0ZXN0X2RpY3RbJ3JlcG9ydCddID0gJ0ZM QUtZJwogICAgICAgICAgICAgZWxzZToKICAgICAgICAgICAgICAgICBudW1fcmVncmVzc2lvbnMg Kz0gMQogICAgICAgICAgICAgICAgIHRlc3RfZGljdFsncmVwb3J0J10gPSAnUkVHUkVTU0lPTicK ZGlmZiAtLWdpdCBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3ZpZXdzL2J1 aWxkYm90X3Jlc3VsdHMucHkgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy92 aWV3cy9idWlsZGJvdF9yZXN1bHRzLnB5CmluZGV4IDBjNzJmNDQ2ZTg0ZDY3ZmRmMmY1ZDY0YmNi NTM2YWU4Njc1YjdmMjIuLmMzN2Q2OGQ1MmJjNDY3MDY3OTdhZjIzYzg5Y2U2YjJkZTAzMzlhMzQg MTAwNjQ0Ci0tLSBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3ZpZXdzL2J1 aWxkYm90X3Jlc3VsdHMucHkKKysrIGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVz dHMvdmlld3MvYnVpbGRib3RfcmVzdWx0cy5weQpAQCAtMTQ2LDkgKzE0Niw3IEBAIGNsYXNzIEJ1 aWxkQm90UHJpbnRlcihvYmplY3QpOgogICAgICAgICAgICAgICAgIGZvciB0ZXN0IGluIHRlc3Rz OgogICAgICAgICAgICAgICAgICAgICByZXN1bHQgPSByZXN1bHRzanNvbnBhcnNlci5yZXN1bHRf Zm9yX3Rlc3Qoc3VtbWFyaXplZF9yZXN1bHRzWyd0ZXN0cyddLCB0ZXN0KQogICAgICAgICAgICAg ICAgICAgICBhY3R1YWwgPSByZXN1bHRbJ2FjdHVhbCddLnNwbGl0KCIgIikKLSAgICAgICAgICAg ICAgICAgICAgZXhwZWN0ZWQgPSByZXN1bHRbJ2V4cGVjdGVkJ10uc3BsaXQoIiAiKQotICAgICAg ICAgICAgICAgICAgICAjIEZJWE1FOiBjbGVhbiB0aGlzIHVwIG9uY2UgdGhlIG9sZCBzeW50YXgg aXMgZ29uZQotICAgICAgICAgICAgICAgICAgICBuZXdfZXhwZWN0YXRpb25zX2xpc3QgPSBbVGVz dEV4cGVjdGF0aW9uUGFyc2VyLl9pbnZlcnRlZF9leHBlY3RhdGlvbl90b2tlbnNbZXhwXSBmb3Ig ZXhwIGluIGxpc3Qoc2V0KGFjdHVhbCkgfCBzZXQoZXhwZWN0ZWQpKV0KKyAgICAgICAgICAgICAg ICAgICAgbmV3X2V4cGVjdGF0aW9uc19saXN0ID0gW1Rlc3RFeHBlY3RhdGlvblBhcnNlci5faW52 ZXJ0ZWRfZXhwZWN0YXRpb25fdG9rZW5zW2V4cF0gZm9yIGV4cCBpbiBsaXN0KHNldChhY3R1YWwp KV0KICAgICAgICAgICAgICAgICAgICAgc2VsZi5fcHJpbnQoIiAgJXMgWyAlcyBdIiAlICh0ZXN0 LCAiICIuam9pbihuZXdfZXhwZWN0YXRpb25zX2xpc3QpKSkKICAgICAgICAgICAgICAgICBzZWxm Ll9wcmludCgiIikKICAgICAgICAgICAgIHNlbGYuX3ByaW50KCIiKQo=