123385 2013-10-25 23:29:20 -0700 New flakiness dashboard shouldn't treat tests with right expectations as failing 2013-10-27 19:56:29 -0700 1 1 1 Unclassified WebKit WebKit Website 528+ (Nightly build) Unspecified Unspecified RESOLVED FIXED P2 Normal --- 1 rniwa rniwa ap commit-queue lforschler simon.fraser thorton zan oldest_to_newest 943857 0 rniwa 2013-10-25 23:29:20 -0700 Right now, if you select "failing" tests on the builder pane, the new flakiness dashboard lists all failing tests including ones that have the right test expectation. It should instead only list tests that are failing and don't have the right expectation that are making bots red. 943858 1 215240 rniwa 2013-10-25 23:33:19 -0700 Created attachment 215240 Changes the behavior 943931 2 215240 ap 2013-10-26 16:52:45 -0700 Comment on attachment 215240 Changes the behavior I've never used this feature on the old dashboard, so it's not clear to me if either behavior is useful. What are the use cases? If this is a replacement for regular dashboard, then we should consider just removing the duplicate functionality. r=me 943934 3 rniwa 2013-10-26 17:21:01 -0700 (In reply to comment #2) > (From update of attachment 215240 [details]) > I've never used this feature on the old dashboard, so it's not clear to me if either behavior is useful. What are the use cases? If this is a replacement for regular dashboard, then we should consider just removing the duplicate functionality. This shows the list of failing tests on the bots. 943937 4 215240 commit-queue 2013-10-26 17:45:38 -0700 Comment on attachment 215240 Changes the behavior Clearing flags on attachment: 215240 Committed r158093: <http://trac.webkit.org/changeset/158093> 943938 5 commit-queue 2013-10-26 17:45:40 -0700 All reviewed patches have been landed. Closing bug. 943966 6 ap 2013-10-27 10:06:03 -0700 > This shows the list of failing tests on the bots. I don't think that this answers my question about use cases. Listing tests that are currently failing is not a job for the dashboard, which is for historic analysis of results. 943972 7 rniwa 2013-10-27 11:04:55 -0700 (In reply to comment #6) > > This shows the list of failing tests on the bots. > > I don't think that this answers my question about use cases. Listing tests that are currently failing is not a job for the dashboard, which is for historic analysis of results. If you're talking about http://build.webkit.org/dashboard/, I find it impossible to use because it doesn't have links to builder's page and it has -webkit-user-select: none along with dozens of other problems. 943974 8 ap 2013-10-27 11:41:48 -0700 Can you please file bugs for those? That is the tool intended to be used for looking at immediate state of the bots, and adding duplicate functionality to other tools is not the best path forward. We'll just end up with a set of tools that no one but their creators understand or use. build.webkit.org/dashboard is also meant to be the primary entry point into the regression test bot system for most people, because checking historic flakiness is an activity that is secondary to checking immediate state. Buildbot waterfall and console certainly have their use, but mostly for people who administer the system, not for WebKit developers in my opinion. There is a bunch of bugs and enhancement requests filed already, you can find these by searching for "build.webkit.org/dashboard" in Bugzilla titles. I encourage you to file bugs in terms of use cases that aren't addressed well (i.e. not simply "please remove user-select:none", but "I often need to do XXX when bot watching, and it's difficult to do now"). 943977 9 rniwa 2013-10-27 12:03:58 -0700 (In reply to comment #8) > build.webkit.org/dashboard is also meant to be the primary entry point into the regression test bot system for most people, because checking historic flakiness is an activity that is secondary to checking immediate state. Buildbot waterfall and console certainly have their use, but mostly for people who administer the system, not for WebKit developers in my opinion. I don't see a point in doing that given I'm satisfied with what build.webkit.org/waterfall and build.webkit.org/console provides. Those two pages provides exactly the kind of information I need. 944009 10 ap 2013-10-27 19:19:43 -0700 > I'm satisfied with what build.webkit.org/waterfall and build.webkit.org/console provides In this case, can we just get rid of the "failing" display in the new flakiness dashboard? 944010 11 rniwa 2013-10-27 19:24:24 -0700 (In reply to comment #10) > > I'm satisfied with what build.webkit.org/waterfall and build.webkit.org/console provides > > In this case, can we just get rid of the "failing" display in the new flakiness dashboard? Why? The historical results of currently failing tests is exactly what bot watchers need to see to determine which patch caused the failure and whether tests have been flaky or not. 944011 12 rniwa 2013-10-27 19:56:29 -0700 I think I'm disagreeing with the statement that "checking historic flakiness is an activity that is secondary to checking immediate state". In my experience, viewing the historical results of a test has been essential in determining the culprit and the correct test expectation to add. Knowing how many tests are failing on a builder doesn't get me anywhere as a bot watcher because my primary job as a bot watcher (contacting the patch author, etc…) cannot be carried out until the culprit is determined. I don't know what revision number http://build.webkit.org/dashboard/ is showing but automatically determining the culprit has already been tried by TestFailures and garden-o-magic. They have both miserably failed to carry out the promise. The task of this sort is best done by humans. 215240 2013-10-25 23:33:19 -0700 2013-10-26 17:45:38 -0700 Changes the behavior fix123385 text/plain 2293 rniwa SW5kZXg6IFdlYnNpdGVzL3Rlc3QtcmVzdWx0cy9DaGFuZ2VMb2cKPT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gV2Vi c2l0ZXMvdGVzdC1yZXN1bHRzL0NoYW5nZUxvZwkocmV2aXNpb24gMTU4MDgwKQorKysgV2Vic2l0 ZXMvdGVzdC1yZXN1bHRzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSw1ICsxLDE5IEBA CiAyMDEzLTEwLTI1ICBSeW9zdWtlIE5pd2EgIDxybml3YUB3ZWJraXQub3JnPgogCisgICAgICAg IE5ldyBmbGFraW5lc3MgZGFzaGJvYXJkIHNob3VsZG4ndCB0cmVhdCB0ZXN0cyB3aXRoIHJpZ2h0 IGV4cGVjdGF0aW9ucyBhcyBmYWlsaW5nCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3Jn L3Nob3dfYnVnLmNnaT9pZD0xMjMzODUKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9P UFMhKS4KKworICAgICAgICBXZSBkZWZpbmUgZmFpbGluZyB0ZXN0cyB0byBiZSB0ZXN0cyB3aXRo IHdyb25nIGV4cGVjdGF0aW9ucyB3aG9zZSBhY3R1YWwgcmVzdWx0cyBhcmUgbm90IFBBU1MKKyAg ICAgICAgc2luY2UgdGVzdHMgd2l0aCBURVhULCBJTUFHRSwgZXRjLi4uIGZhaWx1cmVzIGRvIG5v dCB0dXJuIHRoZSBib3RzIHJlZCBhcyBsb25nIGFzIHRoZSBleHBlY3RhdGlvbgorICAgICAgICBv ZiB0aGUgc2FtZSB0eXBlIGlzIHNwZWNpZmllZCBpbiBUZXN0RXhwZWN0YXRpb24gZmlsZXMuCisK KyAgICAgICAgKiBwdWJsaWMvaW5jbHVkZS90ZXN0LXJlc3VsdHMucGhwOgorICAgICAgICAoRmFp bGluZ1Jlc3VsdHNKU09OV3JpdGVyKTogSW5oZXJpdCBmcm9tIFdyb25nRXhwZWN0YXRpb25zUmVz dWx0c0pTT05Xcml0ZXIuCisKKzIwMTMtMTAtMjUgIFJ5b3N1a2UgTml3YSAgPHJuaXdhQHdlYmtp dC5vcmc+CisKICAgICAgICAgQnVpbGQgZml4LiBUaGUgcXVlcnkgcmVzdWx0cyB3ZXJlbid0IHNv cnRlZCBieSB0aGUgbGF0ZXN0IGNvbW1pdCB0aW1lLAogICAgICAgICB5aWVsZGluZyB3cm9uZyBz ZXQgb2YgdGVzdHMgdG8gYmUgbGlzdGVkIGluIHRoZSBidWlsZGVyIHBhbmUuCiAKSW5kZXg6IFdl YnNpdGVzL3Rlc3QtcmVzdWx0cy9wdWJsaWMvaW5jbHVkZS90ZXN0LXJlc3VsdHMucGhwCj09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT0KLS0tIFdlYnNpdGVzL3Rlc3QtcmVzdWx0cy9wdWJsaWMvaW5jbHVkZS90ZXN0LXJlc3Vs dHMucGhwCShyZXZpc2lvbiAxNTgwODApCisrKyBXZWJzaXRlcy90ZXN0LXJlc3VsdHMvcHVibGlj L2luY2x1ZGUvdGVzdC1yZXN1bHRzLnBocAkod29ya2luZyBjb3B5KQpAQCAtMTM3LDEzICsxMzcs NiBAQAogICAgIGFic3RyYWN0IHByb3RlY3RlZCBmdW5jdGlvbiBwYXNzX2Zvcl9mYWlsdXJlX3R5 cGUoJiRyZXN1bHRzKTsKIH0KIAotY2xhc3MgRmFpbGluZ1Jlc3VsdHNKU09OV3JpdGVyIGV4dGVu ZHMgUmVzdWx0c0pTT05Xcml0ZXIgewotICAgIHB1YmxpYyBmdW5jdGlvbiBfX2NvbnN0cnVjdCgk ZnApIHsgcGFyZW50OjpfX2NvbnN0cnVjdCgkZnApOyB9Ci0gICAgcHJvdGVjdGVkIGZ1bmN0aW9u IHBhc3NfZm9yX2ZhaWx1cmVfdHlwZSgmJHJlc3VsdHMpIHsKLSAgICAgICAgcmV0dXJuICRyZXN1 bHRzWzBdWydhY3R1YWwnXSA9PSAnUEFTUyc7Ci0gICAgfQotfQotCiBjbGFzcyBGbGFreVJlc3Vs dHNKU09OV3JpdGVyIGV4dGVuZHMgUmVzdWx0c0pTT05Xcml0ZXIgewogICAgIHB1YmxpYyBmdW5j dGlvbiBfX2NvbnN0cnVjdCgkZnApIHsgcGFyZW50OjpfX2NvbnN0cnVjdCgkZnApOyB9CiAgICAg cHJvdGVjdGVkIGZ1bmN0aW9uIHBhc3NfZm9yX2ZhaWx1cmVfdHlwZSgmJHJlc3VsdHMpIHsKQEAg LTE3NSw2ICsxNjgsMTMgQEAKICAgICB9CiB9CiAKK2NsYXNzIEZhaWxpbmdSZXN1bHRzSlNPTldy aXRlciBleHRlbmRzIFdyb25nRXhwZWN0YXRpb25zUmVzdWx0c0pTT05Xcml0ZXIgeworICAgIHB1 YmxpYyBmdW5jdGlvbiBfX2NvbnN0cnVjdCgkZnApIHsgcGFyZW50OjpfX2NvbnN0cnVjdCgkZnAp OyB9CisgICAgcHJvdGVjdGVkIGZ1bmN0aW9uIHBhc3NfZm9yX2ZhaWx1cmVfdHlwZSgmJHJlc3Vs dHMpIHsKKyAgICAgICAgcmV0dXJuICRyZXN1bHRzWzBdWydhY3R1YWwnXSA9PSAnUEFTUycgfHwg cGFyZW50OjpwYXNzX2Zvcl9mYWlsdXJlX3R5cGUoJHJlc3VsdHMpOworICAgIH0KK30KKwogY2xh c3MgUmVzdWx0c0pTT05HZW5lcmF0b3IgewogICAgIHByaXZhdGUgJGRiOwogICAgIHByaXZhdGUg JGJ1aWxkZXJfaWQ7Cg==