60295 2011-05-05 13:53:00 -0700 Add an option to NRWT to skip failing tests 2011-05-06 00:44:10 -0700 1 1 1 Unclassified WebKit New Bugs 528+ (Nightly build) Unspecified Unspecified RESOLVED FIXED P2 Normal --- 1 abarth abarth dpranke eric mihaip ojan oldest_to_newest 398992 0 abarth 2011-05-05 13:53:00 -0700 Add an option to NRWT to skip failing tests 398996 1 92461 abarth 2011-05-05 13:55:43 -0700 Created attachment 92461 Patch 399016 2 92461 eric 2011-05-05 14:12:37 -0700 Comment on attachment 92461 Patch Seems reasonable to me. 399028 3 dpranke 2011-05-05 14:16:58 -0700 View in context: https://bugs.webkit.org/attachment.cgi?id=92461&action=review > Tools/ChangeLog:12 > + I wanted to write a test for this change, but I couldn't figure out how. try harder, or ask? :) > Tools/Scripts/webkitpy/layout_tests/layout_package/manager.py:342 > + # FIXME: This method is complicated and needs unit testing! There are tests for this method in run_webkit_tests_unittest.py. See, for example, test_run_part() or test_run_single_file(). You will want to call the get_tests_run() helper function with your command line flag added, and then asser that the tests that ran are the tests you expected to run. > Tools/Scripts/webkitpy/layout_tests/layout_package/manager.py:365 > + self._test_files -= failing I think this will still run flaky tests, which may or may not be what you want. I also had to double-check the code to see if FAIL would do what you want, i.e., if it actually included IMAGE, IMAGE+TEXT, CRASH, etc. as well as tests marked as FAIL. It looks like it does, but this is why you need tests. FAIL is a deprecated option that should go away at some point. You should also log something indicating that you're doing this, since this could easily confuse people otherwise. Something like self._printer.print_expected('Skipping failing tests') (see the example for --force on line 464). 399045 4 eric 2011-05-05 14:32:09 -0700 Thank you for looking Dirk! 399058 5 92461 ojan 2011-05-05 14:37:15 -0700 Comment on attachment 92461 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=92461&action=review > Tools/Scripts/webkitpy/layout_tests/layout_package/manager.py:364 > + failing = self._expectations.get_tests_with_result_type( > + test_expectations.FAIL) This will only exclude tests marked "FAIL" in test_expectaitons. It won't exclude TEXT, IMAGE, IMAGE+TEXT, etc. As the code is now, you'll need to exclude each one. You can iterate through TestExpectations.EXPECTATIONS for all the values. 399059 6 92461 ojan 2011-05-05 14:38:17 -0700 Comment on attachment 92461 Patch Whoops. NM. This code is...confusing. 399062 7 dpranke 2011-05-05 14:40:21 -0700 (In reply to comment #6) > (From update of attachment 92461 [details]) > Whoops. NM. This code is...confusing. Hence my comments, and the need for tests to verify that it's doing what we actually want it to do. 399386 8 abarth 2011-05-06 00:33:51 -0700 Committed r85927: <http://trac.webkit.org/changeset/85927> 399394 9 abarth 2011-05-06 00:44:10 -0700 I added a test before landing. In the course of adding the tests I discovered that these tests are massively slow: https://bugs.webkit.org/show_bug.cgi?id=60350 92461 2011-05-05 13:55:43 -0700 2011-05-05 14:38:17 -0700 Patch bug-60295-20110505135542.patch text/plain 3151 abarth SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDg1ODc0KQorKysgVG9vbHMvQ2hhbmdlTG9nCSh3b3JraW5nIGNvcHkpCkBAIC0xLDMg KzEsMTkgQEAKKzIwMTEtMDUtMDUgIEFkYW0gQmFydGggIDxhYmFydGhAd2Via2l0Lm9yZz4KKwor ICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICBBZGQgYW4gb3B0 aW9uIHRvIE5SV1QgdG8gc2tpcCBmYWlsaW5nIHRlc3RzCisgICAgICAgIGh0dHBzOi8vYnVncy53 ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD02MDI5NQorCisgICAgICAgIFRoaXMgb3B0aW9uIGlz IHNpbWlsYXIgdG8gdGhlIHdheSBvbGQtcnVuLXdlYmtpdC10ZXN0cyB3b3Jrcy4gIFdlJ3JlCisg ICAgICAgIHBsYW5uaW5nIHRvIHVzZSB0aGlzIG9wdGlvbiBvbiB0aGUgdGVzdGluZyBFV1MgYm90 cyBiZWNhdXNlIHRoZXkgZG9uJ3QKKyAgICAgICAgdXNlIGluZm9ybWF0aW9uIGZyb20gZmFpbGlu ZyB0ZXN0cyB5ZXQuCisKKyAgICAgICAgSSB3YW50ZWQgdG8gd3JpdGUgYSB0ZXN0IGZvciB0aGlz IGNoYW5nZSwgYnV0IEkgY291bGRuJ3QgZmlndXJlIG91dCBob3cuCisKKyAgICAgICAgKiBTY3Jp cHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9sYXlvdXRfcGFja2FnZS9tYW5hZ2VyLnB5OgorICAg ICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRfdGVzdHMucHk6 CisKIDIwMTEtMDUtMDUgIEVyaWMgU2VpZGVsICA8ZXJpY0B3ZWJraXQub3JnPgogCiAgICAgICAg IFJldmlld2VkIGJ5IEFkYW0gQmFydGguCkluZGV4OiBUb29scy9TY3JpcHRzL3dlYmtpdHB5L2xh eW91dF90ZXN0cy9ydW5fd2Via2l0X3Rlc3RzLnB5Cj09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIFRvb2xzL1Njcmlw dHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRfdGVzdHMucHkJKHJldmlzaW9uIDg1 ODcyKQorKysgVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcnVuX3dlYmtpdF90 ZXN0cy5weQkod29ya2luZyBjb3B5KQpAQCAtMjk5LDYgKzI5OSwxMCBAQCBkZWYgcGFyc2VfYXJn cyhhcmdzPU5vbmUpOgogICAgICAgICBvcHRwYXJzZS5tYWtlX29wdGlvbigiLS1yZXNldC1yZXN1 bHRzIiwgYWN0aW9uPSJzdG9yZV90cnVlIiwKICAgICAgICAgICAgIGRlZmF1bHQ9RmFsc2UsIGhl bHA9IlJlc2V0IGFueSBleGlzdGluZyBiYXNlbGluZXMgdG8gdGhlICIKICAgICAgICAgICAgICAg ICAgImdlbmVyYXRlZCByZXN1bHRzIiksCisgICAgICAgIG9wdHBhcnNlLm1ha2Vfb3B0aW9uKCIt LXNraXAtZmFpbGluZy10ZXN0cyIsIGFjdGlvbj0ic3RvcmVfdHJ1ZSIsCisgICAgICAgICAgICBk ZWZhdWx0PUZhbHNlLCBoZWxwPSJTa2lwIHRlc3RzIHRoYXQgYXJlIGV4cGVjdGVkIHRvIGZhaWwu ICIKKyAgICAgICAgICAgICAgICAgIk5vdGU6IFdoZW4gdXNpbmcgdGhpcyBvcHRpb24sIHlvdSBt aWdodCBtaXNzIG5ldyBjcmFzaGVzICIKKyAgICAgICAgICAgICAgICAgImluIHRoZXNlIHRlc3Rz LiIpLAogICAgICAgICBvcHRwYXJzZS5tYWtlX29wdGlvbigiLS1hZGRpdGlvbmFsLWRydC1mbGFn IiwgYWN0aW9uPSJhcHBlbmQiLAogICAgICAgICAgICAgZGVmYXVsdD1bXSwgaGVscD0iQWRkaXRp b25hbCBjb21tYW5kIGxpbmUgZmxhZyB0byBwYXNzIHRvIER1bXBSZW5kZXJUcmVlICIKICAgICAg ICAgICAgICAgICAgIlNwZWNpZnkgbXVsdGlwbGUgdGltZXMgdG8gYWRkIG11bHRpcGxlIGZsYWdz LiIpLApJbmRleDogVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvbGF5b3V0X3Bh Y2thZ2UvbWFuYWdlci5weQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9TY3JpcHRzL3dlYmtpdHB5L2xh eW91dF90ZXN0cy9sYXlvdXRfcGFja2FnZS9tYW5hZ2VyLnB5CShyZXZpc2lvbiA4NTg3MikKKysr IFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL21hbmFn ZXIucHkJKHdvcmtpbmcgY29weSkKQEAgLTMzOSw2ICszMzksNyBAQCBjbGFzcyBNYW5hZ2VyOgog ICAgICAgICAgICAgcG9ydC50ZXN0X2V4cGVjdGF0aW9uc19vdmVycmlkZXMoKSkKIAogICAgICMg RklYTUU6IFRoaXMgbWV0aG9kIGlzIHdheSB0b28gbG9uZyBhbmQgbmVlZHMgdG8gYmUgYnJva2Vu IGludG8gcGllY2VzLgorICAgICMgRklYTUU6IFRoaXMgbWV0aG9kIGlzIGNvbXBsaWNhdGVkIGFu ZCBuZWVkcyB1bml0IHRlc3RpbmchCiAgICAgZGVmIHByZXBhcmVfbGlzdHNfYW5kX3ByaW50X291 dHB1dChzZWxmKToKICAgICAgICAgIiIiQ3JlYXRlIGFwcHJvcHJpYXRlIHN1YnNldHMgb2YgdGVz dCBsaXN0cyBhbmQgcmV0dXJucyBhCiAgICAgICAgIFJlc3VsdFN1bW1hcnkgb2JqZWN0LiBBbHNv IHByaW50cyBleHBlY3RlZCB0ZXN0IGNvdW50cy4KQEAgLTM1OCw2ICszNTksMTAgQEAgY2xhc3Mg TWFuYWdlcjoKICAgICAgICAgICAgIHNraXBwZWQgPSBzZWxmLl9leHBlY3RhdGlvbnMuZ2V0X3Rl c3RzX3dpdGhfcmVzdWx0X3R5cGUoCiAgICAgICAgICAgICAgICAgICAgICAgICAgICB0ZXN0X2V4 cGVjdGF0aW9ucy5TS0lQKQogICAgICAgICAgICAgc2VsZi5fdGVzdF9maWxlcyAtPSBza2lwcGVk CisgICAgICAgICAgICBpZiBzZWxmLl9vcHRpb25zLnNraXBfZmFpbGluZ190ZXN0czoKKyAgICAg ICAgICAgICAgICBmYWlsaW5nID0gc2VsZi5fZXhwZWN0YXRpb25zLmdldF90ZXN0c193aXRoX3Jl c3VsdF90eXBlKAorICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRlc3RfZXhwZWN0YXRp b25zLkZBSUwpCisgICAgICAgICAgICAgICAgc2VsZi5fdGVzdF9maWxlcyAtPSBmYWlsaW5nCiAK ICAgICAgICAgIyBDcmVhdGUgYSBzb3J0ZWQgbGlzdCBvZiB0ZXN0IGZpbGVzIHNvIHRoZSBzdWJz ZXQgY2h1bmssCiAgICAgICAgICMgaWYgdXNlZCwgY29udGFpbnMgYWxwaGFiZXRpY2FsbHkgY29u c2VjdXRpdmUgdGVzdHMuCg==