90892 2012-07-10 10:00:46 -0700 results.html should handle flaky tests differently 2012-07-17 11:44:44 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified RESOLVED FIXED P2 Normal --- 1 ojan ojan abarth dpranke kkristof rniwa simon.fraser oldest_to_newest 665831 0 ojan 2012-07-10 10:00:46 -0700 We should have two flaky lists. 1. Tests that fail the first run and pass the second. 2. Tests that fail both runs but in different ways. List 1 should come after tests that timed out and tests with stderr output (before "expected to fail but passed"). List 2 should be where the flaky tests currently are. List 1 is consistently just noise that makes the page harder to make sense of. Also, I frequently want to flag the list of the reliable failures to rerun. It's annoying to have to tab through all the flaky passes to get the the timeouts. 666944 1 dpranke 2012-07-11 12:23:01 -0700 I'm sure you realize this already but we don't currently have a way to compare the output from the two runs to see if they are different. I don't know that it would be particularly hard to add that. Also, I'm a bit concerned that implementing this just makes it even easier to ignore the tests in list 1, that seem to be should either be marked as expected flaky or actually be fixed. 667104 2 ojan 2012-07-11 15:05:24 -0700 (In reply to comment #1) > I'm sure you realize this already but we don't currently have a way to compare the output from the two runs to see if they are different. I don't know that it would be particularly hard to add that. full_results.json, which is what results.html uses, has this information and shows it in the UI already. We don't technically know which run was first and which was second and we don't have the -actual.* files for the first run, but we have the type of failure for each run. > Also, I'm a bit concerned that implementing this just makes it even easier to ignore the tests in list 1, that seem to be should either be marked as expected flaky or actually be fixed. That's true. In practice, I think that this is already ignored. So the cost is that other non-flaky failures get missed. In fact, upon further thought, I think we should hide list 1 by default. There's just too much noise right now in the results.html output. 667112 3 dpranke 2012-07-11 15:11:13 -0700 (In reply to comment #2) > (In reply to comment #1) > > I'm sure you realize this already but we don't currently have a way to compare the output from the two runs to see if they are different. I don't know that it would be particularly hard to add that. > > full_results.json, which is what results.html uses, has this information and shows it in the UI already. We don't technically know which run was first and which was second and we don't have the -actual.* files for the first run, but we have the type of failure for each run. > True. Shouldn't we have the -actuals for the first run as well? > > Also, I'm a bit concerned that implementing this just makes it even easier to ignore the tests in list 1, that seem to be should either be marked as expected flaky or actually be fixed. > > That's true. In practice, I think that this is already ignored. So the cost is that other non-flaky failures get missed. > > In fact, upon further thought, I think we should hide list 1 by default. There's just too much noise right now in the results.html output. That would be hiding unexpected behavior, which seems kinda bad. If others thought this was a good idea, I'd be willing to give it a shot, though. 667121 4 ojan 2012-07-11 15:18:28 -0700 (In reply to comment #3) > (In reply to comment #2) > > (In reply to comment #1) > > > I'm sure you realize this already but we don't currently have a way to compare the output from the two runs to see if they are different. I don't know that it would be particularly hard to add that. > > > > full_results.json, which is what results.html uses, has this information and shows it in the UI already. We don't technically know which run was first and which was second and we don't have the -actual.* files for the first run, but we have the type of failure for each run. > > > > True. Shouldn't we have the -actuals for the first run as well? Do we? Where do we store them? I thought the second run overwrote the first one. > > > Also, I'm a bit concerned that implementing this just makes it even easier to ignore the tests in list 1, that seem to be should either be marked as expected flaky or actually be fixed. > > > > That's true. In practice, I think that this is already ignored. So the cost is that other non-flaky failures get missed. > > > > In fact, upon further thought, I think we should hide list 1 by default. There's just too much noise right now in the results.html output. > > That would be hiding unexpected behavior, which seems kinda bad. If others thought this was a good idea, I'd be willing to give it a shot, though. I suppose. I see a slew (~12) of flaky passes everytime I run the tests. Maybe I just encounter this more because I usually run with -f and I'm just getting what I ask for. 667169 5 dpranke 2012-07-11 15:55:16 -0700 the output for the retry is in layout-test-results/retries/... 671398 6 152788 ojan 2012-07-17 10:59:46 -0700 Created attachment 152788 Patch 671451 7 ojan 2012-07-17 11:44:44 -0700 Committed r122864: <http://trac.webkit.org/changeset/122864> 152788 2012-07-17 10:59:46 -0700 2012-07-17 11:41:00 -0700 Patch bug-90892-20120717105945.patch text/plain 6025 ojan U3VidmVyc2lvbiBSZXZpc2lvbjogMTIyODQ4CmRpZmYgLS1naXQgYS9MYXlvdXRUZXN0cy9DaGFu Z2VMb2cgYi9MYXlvdXRUZXN0cy9DaGFuZ2VMb2cKaW5kZXggN2I5YjgwNjg4MGZlOWVlNzNlZDc0 ZTE4NDY0NTQwMTBmYmJlOWE4ZC4uYWVmOTViYTRkZDNkNGM5Y2MyYWRhZDE0ODQxMzQ2N2RiNzI3 MGI1YyAxMDA2NDQKLS0tIGEvTGF5b3V0VGVzdHMvQ2hhbmdlTG9nCisrKyBiL0xheW91dFRlc3Rz L0NoYW5nZUxvZwpAQCAtMSwzICsxLDE3IEBACisyMDEyLTA3LTE3ICBPamFuIFZhZmFpICA8b2ph bkBjaHJvbWl1bS5vcmc+CisKKyAgICAgICAgcmVzdWx0cy5odG1sIHNob3VsZCBoYW5kbGUgZmxh a3kgdGVzdHMgZGlmZmVyZW50bHkKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hv d19idWcuY2dpP2lkPTkwODkyCisKKyAgICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISku CisKKyAgICAgICAgLVB1dCB0ZXN0cyB0aGF0IGZhaWxlZCBib3RoIHJ1bnMgaW50byB0aGUgbWFp biBmYWlsdXJlcyB0YWJsZS4KKyAgICAgICAgLVB1dCBmbGFreSBwYXNzZXMgYmVsb3cgdGhlIGxp c3Qgb2YgdGltZW91dCB0ZXN0cy4KKworICAgICAgICAqIGZhc3QvaGFybmVzcy9yZXNvdXJjZXMv cmVzdWx0cy10ZXN0LmpzOgorICAgICAgICAqIGZhc3QvaGFybmVzcy9yZXN1bHRzLWV4cGVjdGVk LnR4dDoKKyAgICAgICAgKiBmYXN0L2hhcm5lc3MvcmVzdWx0cy5odG1sOgorCiAyMDEyLTA3LTE2 ICBQYXZlbCBGZWxkbWFuICA8cGZlbGRtYW5AY2hyb21pdW0ub3JnPgogCiAgICAgICAgIFdlYiBJ bnNwZWN0b3I6IGltcGxlbWVudCBzZWFyY2ggLyByZXBsYWNlIGluIHNvdXJjZSBmaWxlcyAoYmVo aW5kIGV4cGVyaW1lbnQgZmxhZykKZGlmZiAtLWdpdCBhL0xheW91dFRlc3RzL2Zhc3QvaGFybmVz cy9yZXNvdXJjZXMvcmVzdWx0cy10ZXN0LmpzIGIvTGF5b3V0VGVzdHMvZmFzdC9oYXJuZXNzL3Jl c291cmNlcy9yZXN1bHRzLXRlc3QuanMKaW5kZXggNTNhZmRmYjQ5OGMzMzRkM2YxMDllMGVhMTM3 ZDhiNzNiMzI0Y2VhNy4uZWQzZTAwMTc3ZGZmNWI1MGJkOWJhZjUyODdlNThlMjMzYmNmOTg0OCAx MDA2NDQKLS0tIGEvTGF5b3V0VGVzdHMvZmFzdC9oYXJuZXNzL3Jlc291cmNlcy9yZXN1bHRzLXRl c3QuanMKKysrIGIvTGF5b3V0VGVzdHMvZmFzdC9oYXJuZXNzL3Jlc291cmNlcy9yZXN1bHRzLXRl c3QuanMKQEAgLTY4MSwxMiArNjgxLDI3IEBAIGZ1bmN0aW9uIHJ1blRlc3RzKCkKICAgICBydW5U ZXN0KHJlc3VsdHMsIGZ1bmN0aW9uKCkgewogICAgICAgICB2YXIgdGl0bGVzID0gZG9jdW1lbnQu Z2V0RWxlbWVudHNCeVRhZ05hbWUoJ2gxJyk7CiAgICAgICAgIGFzc2VydFRydWUodGl0bGVzWzBd LnRleHRDb250ZW50ID09ICdUZXN0cyB0aGF0IGNyYXNoZWQgKDEpOicpOwotICAgICAgICBhc3Nl cnRUcnVlKHRpdGxlc1sxXS50ZXh0Q29udGVudCA9PSAnVGVzdHMgd2hlcmUgcmVzdWx0cyBkaWQg bm90IG1hdGNoIGV4cGVjdGVkIHJlc3VsdHMgKDMpOicpOworICAgICAgICBhc3NlcnRUcnVlKHRp dGxlc1sxXS50ZXh0Q29udGVudCA9PSAnVGVzdHMgdGhhdCBmYWlsZWQgdGV4dC9waXhlbC9hdWRp byBkaWZmICgzKTonKTsKICAgICAgICAgYXNzZXJ0VHJ1ZSh0aXRsZXNbMl0udGV4dENvbnRlbnQg PT0nVGVzdHMgdGhhdCBoYWQgbm8gZXhwZWN0ZWQgcmVzdWx0cyAocHJvYmFibHkgbmV3KSAoMSk6 Jyk7CiAgICAgICAgIGFzc2VydFRydWUodGl0bGVzWzNdLnRleHRDb250ZW50ID09J1Rlc3RzIGV4 cGVjdGVkIHRvIGZhaWwgYnV0IHBhc3NlZCAoMSk6Jyk7CiAgICAgICAgIGxvZ1Bhc3MoJ1BBU1Mn KTsKICAgICB9KTsKIAorICAgIHJlc3VsdHMgPSBtb2NrUmVzdWx0cygpOworICAgIHZhciBzdWJ0 cmVlID0gcmVzdWx0cy50ZXN0c1snZm9vJ10gPSB7fQorICAgIHN1YnRyZWVbJ2Jhci5odG1sJ10g PSBtb2NrRXhwZWN0YXRpb24oJycsICdQQVNTIFRFWFQnKTsKKyAgICBzdWJ0cmVlWydiYXItMS5o dG1sJ10gPSBtb2NrRXhwZWN0YXRpb24oJycsICdURVhUIElNQUdFJyk7CisgICAgc3VidHJlZVsn YmFyLTIuaHRtbCddID0gbW9ja0V4cGVjdGF0aW9uKCdJTUFHRSBURVhUJywgJ1RFWFQgSU1BR0Un KTsKKyAgICBzdWJ0cmVlWydiYXItMy5odG1sJ10gPSBtb2NrRXhwZWN0YXRpb24oJ1BBU1MgVEVY VCcsICdURVhUIFBBU1MnKTsKKyAgICBydW5UZXN0KHJlc3VsdHMsIGZ1bmN0aW9uKCkgeworICAg ICAgICB2YXIgdGl0bGVzID0gZG9jdW1lbnQuZ2V0RWxlbWVudHNCeVRhZ05hbWUoJ2gxJyk7Cisg ICAgICAgIGFzc2VydFRydWUodGl0bGVzWzBdLnRleHRDb250ZW50ID09ICdUZXN0cyB0aGF0IGZh aWxlZCB0ZXh0L3BpeGVsL2F1ZGlvIGRpZmYgKDEpOicpOworICAgICAgICBhc3NlcnRUcnVlKHRp dGxlc1sxXS50ZXh0Q29udGVudCA9PSdGbGFreSB0ZXN0cyAoZmFpbGVkIHRoZSBmaXJzdCBydW4g YW5kIHBhc3NlZCBvbiByZXRyeSkgKDEpOicpOworCisgICAgICAgIGFzc2VydFRydWUoZG9jdW1l bnQucXVlcnlTZWxlY3RvckFsbCgnI3Jlc3VsdHMtdGFibGUgdGJvZHknKS5sZW5ndGggPT0gMik7 CisgICAgICAgIGFzc2VydFRydWUoZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCgnI2ZsYWt5LXRl c3RzLXRhYmxlIHRib2R5JykubGVuZ3RoID09IDIpOworICAgIH0pOworCiAgICAgZG9jdW1lbnQu Ym9keS5pbm5lckhUTUwgPSAnPHByZT4nICsgZ19sb2cuam9pbignXG4nKSArICc8L3ByZT4nOwog fQogCmRpZmYgLS1naXQgYS9MYXlvdXRUZXN0cy9mYXN0L2hhcm5lc3MvcmVzdWx0cy1leHBlY3Rl ZC50eHQgYi9MYXlvdXRUZXN0cy9mYXN0L2hhcm5lc3MvcmVzdWx0cy1leHBlY3RlZC50eHQKaW5k ZXggNTkxNmVkY2VkNGRiZDE2OGUwY2M0Yzk4MjIwNTcxNmMzMTUyNzY5My4uNWFmODgzZDQ1ZTg3 ZWM3M2U4Y2YzYzZjYWQ0MWNjZjM1N2EzMzY0ZSAxMDA2NDQKLS0tIGEvTGF5b3V0VGVzdHMvZmFz dC9oYXJuZXNzL3Jlc3VsdHMtZXhwZWN0ZWQudHh0CisrKyBiL0xheW91dFRlc3RzL2Zhc3QvaGFy bmVzcy9yZXN1bHRzLWV4cGVjdGVkLnR4dApAQCAtMjEwLDMgKzIxMCw3IEBAIFRFU1QtMzc6IFBB U1MKIFRFU1QtMzc6IFBBU1MKIFRFU1QtMzc6IFBBU1MKIFRFU1QtMzc6IFBBU1MKK1RFU1QtMzg6 IFBBU1MKK1RFU1QtMzg6IFBBU1MKK1RFU1QtMzg6IFBBU1MKK1RFU1QtMzg6IFBBU1MKZGlmZiAt LWdpdCBhL0xheW91dFRlc3RzL2Zhc3QvaGFybmVzcy9yZXN1bHRzLmh0bWwgYi9MYXlvdXRUZXN0 cy9mYXN0L2hhcm5lc3MvcmVzdWx0cy5odG1sCmluZGV4IGNiNTVlM2FiM2Y5NDNiNzQ0YWY5ZTU3 Y2ZiMTlmYjU5NTk2YjhkNzUuLjY5MjU0MGQ0ODYwZjgwNTE0NmNhNTYwODQ1NWY3NTI3ZDM1NjQ2 ZmIgMTAwNjQ0Ci0tLSBhL0xheW91dFRlc3RzL2Zhc3QvaGFybmVzcy9yZXN1bHRzLmh0bWwKKysr IGIvTGF5b3V0VGVzdHMvZmFzdC9oYXJuZXNzL3Jlc3VsdHMuaHRtbApAQCAtMjA0LDE0ICsyMDQs MTQgQEAgZnVuY3Rpb24gZ2xvYmFsU3RhdGUoKQogICAgIGlmICghZ19zdGF0ZSkgewogICAgICAg ICBnX3N0YXRlID0gewogICAgICAgICAgICAgY3Jhc2hUZXN0czogW10sCi0gICAgICAgICAgICBm bGFreVRlc3RzOiBbXSwKKyAgICAgICAgICAgIGZsYWt5UGFzc1Rlc3RzOiBbXSwKICAgICAgICAg ICAgIGhhc0h0dHBUZXN0czogZmFsc2UsCiAgICAgICAgICAgICBoYXNJbWFnZUZhaWx1cmVzOiBm YWxzZSwKICAgICAgICAgICAgIGhhc1RleHRGYWlsdXJlczogZmFsc2UsCiAgICAgICAgICAgICBt aXNzaW5nUmVzdWx0czogW10sCiAgICAgICAgICAgICByZXN1bHRzOiB7fSwKICAgICAgICAgICAg IHNob3VsZFRvZ2dsZUltYWdlczogdHJ1ZSwKLSAgICAgICAgICAgIG5vbkZsYWt5RmFpbGluZ1Rl c3RzOiBbXSwKKyAgICAgICAgICAgIGZhaWxpbmdUZXN0czogW10sCiAgICAgICAgICAgICB0ZXN0 c1dpdGhTdGRlcnI6IFtdLAogICAgICAgICAgICAgdGltZW91dFRlc3RzOiBbXSwKICAgICAgICAg ICAgIHVuZXhwZWN0ZWRQYXNzVGVzdHM6IFtdCkBAIC00NzUsOCArNDc1LDggQEAgZnVuY3Rpb24g cHJvY2Vzc0dsb2JhbFN0YXRlRm9yKHRlc3RPYmplY3QpCiAgICAgICAgIHJldHVybjsKICAgICB9 CiAKLSAgICBpZiAoYWN0dWFsLmluZGV4T2YoJyAnKSAhPSAtMSkgewotICAgICAgICBnbG9iYWxT dGF0ZSgpLmZsYWt5VGVzdHMucHVzaCh0ZXN0T2JqZWN0KTsKKyAgICBpZiAoYWN0dWFsLmluZGV4 T2YoJyAnKSAhPSAtMSAmJiBhY3R1YWwuaW5kZXhPZignUEFTUycpICE9IC0xKSB7CisgICAgICAg IGdsb2JhbFN0YXRlKCkuZmxha3lQYXNzVGVzdHMucHVzaCh0ZXN0T2JqZWN0KTsKICAgICAgICAg cmV0dXJuOwogICAgIH0KIApAQCAtNDk3LDcgKzQ5Nyw3IEBAIGZ1bmN0aW9uIHByb2Nlc3NHbG9i YWxTdGF0ZUZvcih0ZXN0T2JqZWN0KQogICAgICAgICByZXR1cm47CiAgICAgfQogICAgIAotICAg IGdsb2JhbFN0YXRlKCkubm9uRmxha3lGYWlsaW5nVGVzdHMucHVzaCh0ZXN0T2JqZWN0KTsKKyAg ICBnbG9iYWxTdGF0ZSgpLmZhaWxpbmdUZXN0cy5wdXNoKHRlc3RPYmplY3QpOwogfQogCiBmdW5j dGlvbiB0b2dnbGVJbWFnZXMoKQpAQCAtMTI1NywyMSArMTI1NywyMSBAQCBmdW5jdGlvbiBnZW5l cmF0ZVBhZ2UoKQogICAgIGlmIChnbG9iYWxTdGF0ZSgpLmNyYXNoVGVzdHMubGVuZ3RoKQogICAg ICAgICBodG1sICs9IHRlc3RMaXN0KGdsb2JhbFN0YXRlKCkuY3Jhc2hUZXN0cywgJ1Rlc3RzIHRo YXQgY3Jhc2hlZCcsICdjcmFzaC10ZXN0cy10YWJsZScpOwogCi0gICAgaHRtbCArPSBmYWlsaW5n VGVzdHNUYWJsZShnbG9iYWxTdGF0ZSgpLm5vbkZsYWt5RmFpbGluZ1Rlc3RzLAotICAgICAgICAn VGVzdHMgd2hlcmUgcmVzdWx0cyBkaWQgbm90IG1hdGNoIGV4cGVjdGVkIHJlc3VsdHMnLCAncmVz dWx0cy10YWJsZScpOworICAgIGh0bWwgKz0gZmFpbGluZ1Rlc3RzVGFibGUoZ2xvYmFsU3RhdGUo KS5mYWlsaW5nVGVzdHMsCisgICAgICAgICdUZXN0cyB0aGF0IGZhaWxlZCB0ZXh0L3BpeGVsL2F1 ZGlvIGRpZmYnLCAncmVzdWx0cy10YWJsZScpOwogCiAgICAgaHRtbCArPSBmYWlsaW5nVGVzdHNU YWJsZShnbG9iYWxTdGF0ZSgpLm1pc3NpbmdSZXN1bHRzLAogICAgICAgICAnVGVzdHMgdGhhdCBo YWQgbm8gZXhwZWN0ZWQgcmVzdWx0cyAocHJvYmFibHkgbmV3KScsICdtaXNzaW5nLXRhYmxlJyk7 CiAKLSAgICBodG1sICs9IGZhaWxpbmdUZXN0c1RhYmxlKGdsb2JhbFN0YXRlKCkuZmxha3lUZXN0 cywKLSAgICAgICAgJ0ZsYWt5IHRlc3RzIChmYWlsZWQgdGhlIGZpcnN0IHJ1biBhbmQgZ290IGEg ZGlmZmVyZW50IHJlc3VsdCBvbiByZXRyeSknLCAnZmxha3ktdGVzdHMtdGFibGUnKTsKLQogICAg IGlmIChnbG9iYWxTdGF0ZSgpLnRpbWVvdXRUZXN0cy5sZW5ndGgpCiAgICAgICAgIGh0bWwgKz0g dGVzdExpc3QoZ2xvYmFsU3RhdGUoKS50aW1lb3V0VGVzdHMsICdUZXN0cyB0aGF0IHRpbWVkIG91 dCcsICd0aW1lb3V0LXRlc3RzLXRhYmxlJyk7CiAKICAgICBpZiAoZ2xvYmFsU3RhdGUoKS50ZXN0 c1dpdGhTdGRlcnIubGVuZ3RoKQogICAgICAgICBodG1sICs9IHRlc3RMaXN0KGdsb2JhbFN0YXRl KCkudGVzdHNXaXRoU3RkZXJyLCAnVGVzdHMgdGhhdCBoYWQgc3RkZXJyIG91dHB1dCcsICdzdGRl cnItdGFibGUnKTsKIAorICAgIGh0bWwgKz0gZmFpbGluZ1Rlc3RzVGFibGUoZ2xvYmFsU3RhdGUo KS5mbGFreVBhc3NUZXN0cywKKyAgICAgICAgJ0ZsYWt5IHRlc3RzIChmYWlsZWQgdGhlIGZpcnN0 IHJ1biBhbmQgcGFzc2VkIG9uIHJldHJ5KScsICdmbGFreS10ZXN0cy10YWJsZScpOworCiAgICAg aWYgKGdsb2JhbFN0YXRlKCkucmVzdWx0cy51c2VzX2V4cGVjdGF0aW9uc19maWxlICYmIGdsb2Jh bFN0YXRlKCkudW5leHBlY3RlZFBhc3NUZXN0cy5sZW5ndGgpCiAgICAgICAgIGh0bWwgKz0gdGVz dExpc3QoZ2xvYmFsU3RhdGUoKS51bmV4cGVjdGVkUGFzc1Rlc3RzLCAnVGVzdHMgZXhwZWN0ZWQg dG8gZmFpbCBidXQgcGFzc2VkJywgJ3Bhc3Nlcy10YWJsZScpOwogCg==