58588 2011-04-14 15:46:08 -0700 only show expected failure type for ports that use an expectations file 2011-04-14 16:31:08 -0700 1 1 1 Unclassified WebKit New Bugs 528+ (Nightly build) Other OS X 10.5 RESOLVED FIXED P2 Normal --- 1 ojan ojan dpranke eric tony oldest_to_newest 386122 0 ojan 2011-04-14 15:46:08 -0700 only show expected failure type for ports that use an expectations file 386126 1 89673 ojan 2011-04-14 15:49:21 -0700 Created attachment 89673 Patch 386132 2 89673 eric 2011-04-14 15:51:43 -0700 Comment on attachment 89673 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=89673&action=review > Tools/Scripts/webkitpy/layout_tests/layout_package/test_runner.py:159 > + # FIXME: If non-chromium ports start using an expectations file, > + # we should make this check more robust. > + results['uses_expectations_file'] = port_obj.name().find('chromium') != -1 If... They're about to. Sounds like dirk is working on the NRWT switch... and abarth and I are about to bring up a NRWT-using commit-queue. :) 386138 3 ojan 2011-04-14 15:57:49 -0700 (In reply to comment #2) > (From update of attachment 89673 [details]) > View in context: https://bugs.webkit.org/attachment.cgi?id=89673&action=review > > > Tools/Scripts/webkitpy/layout_tests/layout_package/test_runner.py:159 > > + # FIXME: If non-chromium ports start using an expectations file, > > + # we should make this check more robust. > > + results['uses_expectations_file'] = port_obj.name().find('chromium') != -1 > > If... They're about to. Sounds like dirk is working on the NRWT switch... and abarth and I are about to bring up a NRWT-using commit-queue. :) Using new-run-webkit-tests does not necessarily imply using expectations files. For now, new-run-webkit-tests uses Skipped files just fine. The discussion of whether to use expectations files is (and should be) totally orthogonal. In either case, if that changes, we can easily revert this patch. 386144 4 89673 eric 2011-04-14 16:01:53 -0700 Comment on attachment 89673 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=89673&action=review >>> Tools/Scripts/webkitpy/layout_tests/layout_package/test_runner.py:159 >>> + results['uses_expectations_file'] = port_obj.name().find('chromium') != -1 >> >> If... They're about to. Sounds like dirk is working on the NRWT switch... and abarth and I are about to bring up a NRWT-using commit-queue. :) > > Using new-run-webkit-tests does not necessarily imply using expectations files. For now, new-run-webkit-tests uses Skipped files just fine. The discussion of whether to use expectations files is (and should be) totally orthogonal. In either case, if that changes, we can easily revert this patch. I see. OK, I misunderstood what that was doing/and-or just didn't read carefullly. :p 386150 5 dpranke 2011-04-14 16:09:28 -0700 I think the logic might be better/more robust if instead of guess as to whether this is a chromium/non-chromium port, or a port that does use an expectations file, you look at the actual data and see if any of the rows contain an "expected" field (or one with a value != 'PASS')? At any rate, the apple mac NRWT port uses Skipped files plus an expectations file currently. The patch looks fine to me otherwise. 386156 6 ojan 2011-04-14 16:21:13 -0700 (In reply to comment #5) > I think the logic might be better/more robust if instead of guess as to whether this is a chromium/non-chromium port, or a port that does use an expectations file, you look at the actual data and see if any of the rows contain an "expected" field (or one with a value != 'PASS')? > > At any rate, the apple mac NRWT port uses Skipped files plus an expectations file currently. > > The patch looks fine to me otherwise. Whoops. Already committed r83908: <http://trac.webkit.org/changeset/83908>. Will fix in a separate patch. 386162 7 ojan 2011-04-14 16:29:16 -0700 > At any rate, the apple mac NRWT port uses Skipped files plus an expectations file currently. Actually, looking at this more, I'm not convinced it makes sense. While technically the Apple mac port uses expectations files, I'm not convinced that they will want expectations files once they actually transition to using NRWT. I have certainly heard opposition to the idea of expectations files. I'll leave this FIXME in for now. 386165 8 dpranke 2011-04-14 16:31:08 -0700 (In reply to comment #7) > > At any rate, the apple mac NRWT port uses Skipped files plus an expectations file currently. > > Actually, looking at this more, I'm not convinced it makes sense. While technically the Apple mac port uses expectations files, I'm not convinced that they will want expectations files once they actually transition to using NRWT. I have certainly heard opposition to the idea of expectations files. > > I'll leave this FIXME in for now. Well, I still think it might be more robust to decide what to display based on what is in the actual data rather than what you expect to be in the data (based on the port or bot name). 89673 2011-04-14 15:49:21 -0700 2011-04-14 16:02:04 -0700 Patch bug-58588-20110414154920.patch text/plain 3363 ojan U3VidmVyc2lvbiBSZXZpc2lvbjogODM5MDIKZGlmZiAtLWdpdCBhL1Rvb2xzL0NoYW5nZUxvZyBi L1Rvb2xzL0NoYW5nZUxvZwppbmRleCA5ODFiODZjZDA2MDg2MGNkZDYwNjNiMGIyOTk0NjgxYTI0 YzM4NThhLi43YTEzZTZhZTU3YjI1OGQ2MTFiY2E2MTM3MWE0ZjhlNTMyZTlmMDY5IDEwMDY0NAot LS0gYS9Ub29scy9DaGFuZ2VMb2cKKysrIGIvVG9vbHMvQ2hhbmdlTG9nCkBAIC0xLDUgKzEsMTUg QEAKIDIwMTEtMDQtMTQgIE9qYW4gVmFmYWkgIDxvamFuQGNocm9taXVtLm9yZz4KIAorICAgICAg ICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICBvbmx5IHNob3cgZXhwZWN0 ZWQgZmFpbHVyZSB0eXBlIGZvciBwb3J0cyB0aGF0IHVzZSBhbiBleHBlY3RhdGlvbnMgZmlsZQor ICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9NTg1ODgKKwor ICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL2pz b25fcmVzdWx0cy5odG1sOgorICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3Rz L2xheW91dF9wYWNrYWdlL3Rlc3RfcnVubmVyLnB5OgorCisyMDExLTA0LTE0ICBPamFuIFZhZmFp ICA8b2phbkBjaHJvbWl1bS5vcmc+CisKICAgICAgICAgUmV2aWV3ZWQgYnkgVG9ueSBDaGFuZy4K IAogICAgICAgICBzb3J0IGNvbHVtbnMgaW4gdGhlIG5ldyBqc29uX3Jlc3VsdHMuaHRtbCBmaWxl CmRpZmYgLS1naXQgYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9sYXlvdXRf cGFja2FnZS9qc29uX3Jlc3VsdHMuaHRtbCBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0 X3Rlc3RzL2xheW91dF9wYWNrYWdlL2pzb25fcmVzdWx0cy5odG1sCmluZGV4IGIxNTkyNThlYzM2 YzlhNGI4NjYwZWVhNWMyOTY1NmVhYWNlZmY3ODUuLjIzYmUyNGRhNDVjYjdiZmJjYzA5YTMxY2Ey ZWE3NjNlOGE4ZjNkNGUgMTAwNjQ0Ci0tLSBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0 X3Rlc3RzL2xheW91dF9wYWNrYWdlL2pzb25fcmVzdWx0cy5odG1sCisrKyBiL1Rvb2xzL1Njcmlw dHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL2pzb25fcmVzdWx0cy5odG1s CkBAIC01MCw5ICs1MCwxMiBAQCB2YXIgaHRtbCA9ICdUZXN0cyB3aGVyZSByZXN1bHRzIGRpZCBu b3QgbWF0Y2ggZXhwZWN0ZWQgcmVzdWx0czo8dGFibGU+JyArCiAgICAgICAgICc8dGg+dGVzdDwv dGg+JyArCiAgICAgICAgICc8dGggaWQ9InRleHQtcmVzdWx0cy1oZWFkZXIiPnRleHQgcmVzdWx0 czwvdGg+JyArCiAgICAgICAgICc8dGggaWQ9ImltYWdlLXJlc3VsdHMtaGVhZGVyIj5pbWFnZSBy ZXN1bHRzPC90aD4nICsKLSAgICAgICAgJzx0aD5mYWlsdXJlIHR5cGU8L3RoPicgKwotICAgICAg ICAnPHRoPmV4cGVjdGVkIGZhaWx1cmUgdHlwZTwvdGg+JyArCi0gICAgJzwvdHI+PC90aGVhZD4n OworICAgICAgICAnPHRoPmZhaWx1cmUgdHlwZTwvdGg+JzsKKyAgICAgICAgCitpZiAocmVzdWx0 cy51c2VzX2V4cGVjdGF0aW9uc19maWxlKQorICAgIGh0bWwgKz0gJzx0aD5leHBlY3RlZCBmYWls dXJlIHR5cGU8L3RoPic7CisKK2h0bWwgKz0gJzwvdHI+PC90aGVhZD4nOwogCiAvLyBGSVhNRTog U2hvdWxkIHRoaXMgcG9pbnQgdG8gdGhlIGxvY2FsIGZpbGUgaW5zdGVhZD8gU2hvdWxkIGl0IGR5 bmFtaWNhbGx5IGZpZ3VyZSBvdXQgd2hlcmUgdG8gcG9pbnQ/CiB2YXIgdGVzdF9iYXNlX3BhdGgg PSAnaHR0cDovL3RyYWMud2Via2l0Lm9yZy9icm93c2VyL3RydW5rL0xheW91dFRlc3RzLyc7CkBA IC02NSw3ICs2OCw2IEBAIGZ1bmN0aW9uIHJlc3VsdExpbmsodGVzdF9wcmVmaXgsIHN1ZmZpeCwg Y29udGVudHMpCiAvLyBGSVhNRTogc2hvdyBleHBlY3RlZC9hY3R1YWwvZGlmZiBjb250ZW50cyBp bmxpbmUgaW4gaWZyYW1lcwogLy8gRklYTUU6IGFsbG93IHpvb21pbmcgaW4gb24gcGl4ZWwgZGlm ZnMKIC8vIEZJWE1FOiBzdG9yZSBzdGRlcnIgaW5mb3JtYXRpb24gaW4gdGhlIGpzb24KLS8vIEZJ WE1FOiBkb24ndCBzaG93IGV4cGVjdGVkIGZhaWx1cmUgdHlwZSBmb3Igbm9uLWNocm9taXVtIHBv cnRzCiAKIHZhciBoYXNUZXh0RmFpbHVyZXMgPSBmYWxzZTsKIHZhciBoYXNJbWFnZUZhaWx1cmVz ID0gZmFsc2U7CkBAIC0xMDIsNyArMTA0LDEwIEBAIGZvciAodmFyIHRlc3QgaW4gcmVzdWx0cy50 ZXN0cykgewogICByb3cgKz0gJzwvdGQ+JzsKICAgLy8gRklYTUU6IEhhbmRsZSBzdGRlcnIgb3V0 cHV0LgogICByb3cgKz0gJzx0ZD4nICsgYWN0dWFsICsgJzwvdGQ+JzsKLSAgcm93ICs9ICc8dGQ+ JyArIHJlc3VsdHMudGVzdHNbdGVzdF0uZXhwZWN0ZWQgKyAnPC90ZD4nOworICAKKyAgaWYgKHJl c3VsdHMudXNlc19leHBlY3RhdGlvbnNfZmlsZSkKKyAgICByb3cgKz0gJzx0ZD4nICsgcmVzdWx0 cy50ZXN0c1t0ZXN0XS5leHBlY3RlZCArICc8L3RkPic7CisKICAgaHRtbCArPSAnPHRyPicgKyBy b3cgKyAnPC90cj4nOwogfQogCmRpZmYgLS1naXQgYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xh eW91dF90ZXN0cy9sYXlvdXRfcGFja2FnZS90ZXN0X3J1bm5lci5weSBiL1Rvb2xzL1NjcmlwdHMv d2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL3Rlc3RfcnVubmVyLnB5CmluZGV4 IGQ4MWI5Yzg4N2E0Mjg5NDA0Nzk1YTFjNmZjNmMwNzA0NDI1MjE0MTEuLmRhMTFlZDA3NGFiYzQx N2RhNjE2MGUwNGY0ZmUzNWE3ZDQ2YTM2ZTEgMTAwNjQ0Ci0tLSBhL1Rvb2xzL1NjcmlwdHMvd2Vi a2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL3Rlc3RfcnVubmVyLnB5CisrKyBiL1Rv b2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL3Rlc3RfcnVu bmVyLnB5CkBAIC0xNTQsNiArMTU0LDkgQEAgZGVmIHN1bW1hcml6ZV9yZXN1bHRzKHBvcnRfb2Jq LCBleHBlY3RhdGlvbnMsIHJlc3VsdF9zdW1tYXJ5LCByZXRyeV9zdW1tYXJ5LCB0ZXMKICAgICBy ZXN1bHRzWydudW1fcGFzc2VzJ10gPSBudW1fcGFzc2VzCiAgICAgcmVzdWx0c1snbnVtX2ZsYWt5 J10gPSBudW1fZmxha3kKICAgICByZXN1bHRzWydudW1fcmVncmVzc2lvbnMnXSA9IG51bV9yZWdy ZXNzaW9ucworICAgICMgRklYTUU6IElmIG5vbi1jaHJvbWl1bSBwb3J0cyBzdGFydCB1c2luZyBh biBleHBlY3RhdGlvbnMgZmlsZSwKKyAgICAjIHdlIHNob3VsZCBtYWtlIHRoaXMgY2hlY2sgbW9y ZSByb2J1c3QuCisgICAgcmVzdWx0c1sndXNlc19leHBlY3RhdGlvbnNfZmlsZSddID0gcG9ydF9v YmoubmFtZSgpLmZpbmQoJ2Nocm9taXVtJykgIT0gLTEKIAogICAgIHJldHVybiByZXN1bHRzCiAK