98260 2012-10-03 06:29:53 -0700 [NRWT] --skipped option is ignored when --test-list is used 2012-10-04 03:18:48 -0700 1 1 1 Unclassified WebKit New Bugs 528+ (Nightly build) All All RESOLVED FIXED NRWT P2 Major --- 1 ossy dpranke abarth dpranke eric kadam kbalazs kkristof ojan ossy tony webkit.review.bot oldest_to_newest 733633 0 ossy 2012-10-03 06:29:53 -0700 Now NRWT runs all tests listed in test-list file, --skipped option is absolutely ignored. In my opinion it should respect --skipped option too. This bug is revealed with a real life usecase: - After a fix I run all tests on WK1: run-webkit-tests - There are 600+ failing tests. I checked one by one, only rebase needed. (list-of-rebased-tests.txt) - I did the rebase to platform/qt - double check for WK2 (maybe some of them need qt-5.0-wk2 specific result): run-webkit-tests --test-list=./list-of-rebased-tests.txt --qt -2 --> But unfortunately NRWT runs skipped tests (wk2, qt-5.0-wk2) too, but it shouldn't Any idea how to fix this annoying bug? 733828 1 ap 2012-10-03 10:37:09 -0700 FWIW, it's intentional that tests listed on command line are run regardless of being skipped. Test-list file is similar conceptually, although use cases are clearly different. 733896 2 dpranke 2012-10-03 11:54:24 -0700 Alexey is basically right on the implementation and original intent, but that doesn't mean we shouldn't make your case work. To make sure that I understand what you're seeing / asking for, you have a list of tests in a file, and some of those tests are listed as [ Skip ] in TestExpectations, but when you run NRWT, they get executed, right? So, you'd actually like the expectations to be honored and the files skipped if they do appear in the TestExpectations file (as skips)? If so, I actually ran into this exact same problem a couple days ago, and the patch I made (but haven't posted yet) is a change to add --skipped=expectations which means "skip the tests listed in the expectations file even if they were listed on in a text file or on the command line". It's a ~3 line change :). Would that work for you? 733899 3 ossy 2012-10-03 12:06:07 -0700 Ah, I can understand the intention of the implemetation too. My use case is that I don't want to run all tests, but a small subset of tests. But it would be great if somehow TestExpectations would honored. Now the following 2 commands are same: (--test-list implicate --skipped=ignore) - run-webkit-tests --test-list=./list.txt - run-webkit-tests --test-list=./list.txt --skipped=ignore What if we can override it with explicit --skipped=default option? - run-webkit-tests --test-list=./list.txt --skipped=default Additionally it would we great if we add a comment to the help of --test-list option to make it clear it implicates --skipped=ignore. How does it sound? 733929 4 dpranke 2012-10-03 12:28:28 -0700 (In reply to comment #3) > Ah, I can understand the intention of the implemetation too. My use case is > that I don't want to run all tests, but a small subset of tests. But it would > be great if somehow TestExpectations would honored. > > Now the following 2 commands are same: (--test-list implicate --skipped=ignore) > - run-webkit-tests --test-list=./list.txt > - run-webkit-tests --test-list=./list.txt --skipped=ignore > > What if we can override it with explicit --skipped=default option? > - run-webkit-tests --test-list=./list.txt --skipped=default > > Additionally it would we great if we add a comment to the help of > --test-list option to make it clear it implicates --skipped=ignore. > > How does it sound? That's almost exactly what my patch does. I'll post it shortly. -- Dirk 734325 5 167026 dpranke 2012-10-03 20:03:02 -0700 Created attachment 167026 Patch 734479 6 kbalazs 2012-10-04 01:09:22 -0700 It's nice that you fixed it, I was also struggling with that. 734545 7 167026 ossy 2012-10-04 02:51:44 -0700 Comment on attachment 167026 Patch Thanks for the quick fix. 734558 8 167026 webkit.review.bot 2012-10-04 03:18:45 -0700 Comment on attachment 167026 Patch Clearing flags on attachment: 167026 Committed r130381: <http://trac.webkit.org/changeset/130381> 734559 9 webkit.review.bot 2012-10-04 03:18:48 -0700 All reviewed patches have been landed. Closing bug. 167026 2012-10-03 20:03:02 -0700 2012-10-04 03:18:45 -0700 Patch bug-98260-20121003200213.patch text/plain 4831 dpranke U3VidmVyc2lvbiBSZXZpc2lvbjogMTMwMzU2CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggNWVlMzliYzliOGYyNDc5YzQ0NzAyOGE0YzExNjA5ZWUw YzZlZGU2Yi4uOWU1ZDExNTJjNTVhMjYwZTRhYzM0YzI4MWY2Yzc2YzlhZTQ2YmRkMCAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSw1ICsxLDI3 IEBACiAyMDEyLTEwLTAzICBEaXJrIFByYW5rZSAgPGRwcmFua2VAY2hyb21pdW0ub3JnPgogCisg ICAgICAgIFtOUldUXSAtLXNraXBwZWQgb3B0aW9uIGlzIGlnbm9yZWQgd2hlbiAtLXRlc3QtbGlz dCBpcyB1c2VkCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9p ZD05ODI2MAorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisgICAgICAg IEFkZHMgYSAtLXNraXBwZWQ9YWx3YXlzIGZsYWcgdGhhdCB3aWxsIHNraXAgYW55IHRlc3RzIGxp c3RlZCBpbgorICAgICAgICBUZXN0RXhwZWN0YXRpb25zIGV2ZW4gaWYgdGhleSdyZSBsaXN0ZWQg ZXhwbGljaXRseSBvbiB0aGUKKyAgICAgICAgY29tbWFuZCBsaW5lLgorCisgICAgICAgIFRoaXMg aXMgbW9zdCB1c2VmdWwgaWYgeW91IGFyZSB1c2luZyAtLXRlc3QtbGlzdCB0byBzcGVjaWZ5IGEK KyAgICAgICAgbG9uZyBsaXN0IG9mIGZpbGVzIGJ1dCB5b3Ugc3RpbGwgd2FudCB0byBza2lwIHNv bWUgb2YgdGhlbQorICAgICAgICBkZXBlbmRpbmcgb24gd2hhdCdzIGluIFRlc3RFeHBlY3RhdGlv bnMuCisKKyAgICAgICAgKiBTY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9jb250cm9sbGVy cy9sYXlvdXRfdGVzdF9maW5kZXIucHk6CisgICAgICAgIChMYXlvdXRUZXN0RmluZGVyLnNraXBf dGVzdHMpOgorICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJr aXRfdGVzdHMucHk6CisgICAgICAgIChwYXJzZV9hcmdzKToKKyAgICAgICAgKiBTY3JpcHRzL3dl YmtpdHB5L2xheW91dF90ZXN0cy9ydW5fd2Via2l0X3Rlc3RzX2ludGVncmF0aW9udGVzdC5weToK KyAgICAgICAgKE1haW5UZXN0LnRlc3Rfc2tpcHBlZF9mbGFnKToKKworMjAxMi0xMC0wMyAgRGly ayBQcmFua2UgIDxkcHJhbmtlQGNocm9taXVtLm9yZz4KKwogICAgICAgICBydW4td2Via2l0LXRl c3RzIHRlc3RzIG9uIHdpbjMyIGFmdGVyIHIxMjczMDIKICAgICAgICAgaHR0cHM6Ly9idWdzLndl YmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTk4MzIzCiAKZGlmZiAtLWdpdCBhL1Rvb2xzL1Njcmlw dHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2NvbnRyb2xsZXJzL2xheW91dF90ZXN0X2ZpbmRlci5w eSBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2NvbnRyb2xsZXJzL2xheW91 dF90ZXN0X2ZpbmRlci5weQppbmRleCBhN2I0OTgzMWMxYmE5MGMwZDc3YmExN2M2YTAzZWM4Mjdk YTcyNTcxLi42NDQ3YzhmYjQ4OWExNGViZmZhNjU4OWNjNDc3YmVjZGQ5ZjRhZTlmIDEwMDY0NAot LS0gYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9jb250cm9sbGVycy9sYXlv dXRfdGVzdF9maW5kZXIucHkKKysrIGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVz dHMvY29udHJvbGxlcnMvbGF5b3V0X3Rlc3RfZmluZGVyLnB5CkBAIC0xMDYsMTEgKzEwNiw5IEBA IGNsYXNzIExheW91dFRlc3RGaW5kZXIob2JqZWN0KToKICAgICAgICAgICAgIHRlc3RzX3RvX3Nr aXAgPSBhbGxfdGVzdHMgLSB0ZXN0c190b19za2lwCiAgICAgICAgIGVsaWYgc2VsZi5fb3B0aW9u cy5za2lwcGVkID09ICdpZ25vcmUnOgogICAgICAgICAgICAgdGVzdHNfdG9fc2tpcCA9IHNldCgp Ci0gICAgICAgIGVsaWYgc2VsZi5fb3B0aW9ucy5za2lwcGVkID09ICdkZWZhdWx0JzoKLSAgICAg ICAgICAgIHBhc3MgICMgbGlzdGVkIGZvciBjb21wbGV0ZW5lc3MKLQotICAgICAgICAjIG1ha2Ug c3VyZSB3ZSdyZSBleHBsaWNpdGx5IHJ1bm5pbmcgYW55IHRlc3RzIHBhc3NlZCBvbiB0aGUgY29t bWFuZCBsaW5lLgotICAgICAgICB0ZXN0c190b19za2lwIC09IHBhdGhzCisgICAgICAgIGVsaWYg c2VsZi5fb3B0aW9ucy5za2lwcGVkICE9ICdhbHdheXMnOgorICAgICAgICAgICAgIyBtYWtlIHN1 cmUgd2UncmUgZXhwbGljaXRseSBydW5uaW5nIGFueSB0ZXN0cyBwYXNzZWQgb24gdGhlIGNvbW1h bmQgbGluZTsgZXF1aXZhbGVudCB0byAnZGVmYXVsdCcuCisgICAgICAgICAgICB0ZXN0c190b19z a2lwIC09IHBhdGhzCiAKICAgICAgICAgIyB1bmxlc3Mgb2YgY291cnNlIHdlIGRvbid0IHdhbnQg dG8gcnVuIHRoZSBIVFRQIHRlc3RzIDopCiAgICAgICAgIGlmIG5vdCBzZWxmLl9vcHRpb25zLmh0 dHA6CmRpZmYgLS1naXQgYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9ydW5f d2Via2l0X3Rlc3RzLnB5IGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcnVu X3dlYmtpdF90ZXN0cy5weQppbmRleCBlNzg0Y2I2MWRlNWIzNTJjM2U3NWE3NGI4MWIwZTk4NjYz Y2QyODIyLi5hMjIwMzI5ZDVlOTRmNWU2MWJmOTNjOTFmMWZjNDlhOGYwNWU0YjhiIDEwMDc1NQot LS0gYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9ydW5fd2Via2l0X3Rlc3Rz LnB5CisrKyBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRf dGVzdHMucHkKQEAgLTM3NSw3ICszNzUsMTEgQEAgZGVmIHBhcnNlX2FyZ3MoYXJncz1Ob25lKToK ICAgICAgICAgb3B0cGFyc2UubWFrZV9vcHRpb24oIi0tdGVzdC1saXN0IiwgYWN0aW9uPSJhcHBl bmQiLAogICAgICAgICAgICAgaGVscD0icmVhZCBsaXN0IG9mIHRlc3RzIHRvIHJ1biBmcm9tIGZp bGUiLCBtZXRhdmFyPSJGSUxFIiksCiAgICAgICAgIG9wdHBhcnNlLm1ha2Vfb3B0aW9uKCItLXNr aXBwZWQiLCBhY3Rpb249InN0b3JlIiwgZGVmYXVsdD0iZGVmYXVsdCIsCi0gICAgICAgICAgICBo ZWxwPSJjb250cm9sIGhvdyB0ZXN0cyBtYXJrZWQgU0tJUCBhcmUgcnVuLiAnZGVmYXVsdCcgPT0g U2tpcCwgJ2lnbm9yZScgPT0gUnVuIHRoZW0gYW55d2F5LCAnb25seScgPT0gb25seSBydW4gdGhl IFNLSVAgdGVzdHMuIiksCisgICAgICAgICAgICBoZWxwPSgiY29udHJvbCBob3cgdGVzdHMgbWFy a2VkIFNLSVAgYXJlIHJ1bi4gIgorICAgICAgICAgICAgICAgICAiJ2RlZmF1bHQnID09IFNraXAg dGVzdHMgdW5sZXNzIGV4cGxpY2l0bHkgbGlzdGVkIG9uIHRoZSBjb21tYW5kIGxpbmUsICIKKyAg ICAgICAgICAgICAgICAgIidpZ25vcmUnID09IFJ1biB0aGVtIGFueXdheSwgIgorICAgICAgICAg ICAgICAgICAiJ29ubHknID09IG9ubHkgcnVuIHRoZSBTS0lQIHRlc3RzLCAiCisgICAgICAgICAg ICAgICAgICInYWx3YXlzJyA9PSBhbHdheXMgc2tpcCwgZXZlbiBpZiBsaXN0ZWQgb24gdGhlIGNv bW1hbmQgbGluZS4iKSksCiAgICAgICAgIG9wdHBhcnNlLm1ha2Vfb3B0aW9uKCItLWZvcmNlIiwg ZGVzdD0ic2tpcHBlZCIsIGFjdGlvbj0ic3RvcmVfY29uc3QiLCBjb25zdD0naWdub3JlJywKICAg ICAgICAgICAgIGhlbHA9IlJ1biBhbGwgdGVzdHMsIGV2ZW4gdGhvc2UgbWFya2VkIFNLSVAgaW4g dGhlIHRlc3QgbGlzdCAoc2FtZSBhcyAtLXNraXBwZWQ9aWdub3JlKSIpLAogICAgICAgICBvcHRw YXJzZS5tYWtlX29wdGlvbigiLS10aW1lLW91dC1tcyIsCmRpZmYgLS1naXQgYS9Ub29scy9TY3Jp cHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9ydW5fd2Via2l0X3Rlc3RzX2ludGVncmF0aW9udGVz dC5weSBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRfdGVz dHNfaW50ZWdyYXRpb250ZXN0LnB5CmluZGV4IDNiNDA1MTlmMzhhMzdmMTYzZmYyNjU0YTg5NWI5 OWM4OGIzY2ZjYzUuLjI0YmExNWE0N2EwNGRlYjY2YTlmYWI2Mjg2Nzg3MTZkN2VlZDZmMTkgMTAw NzU1Ci0tLSBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRf dGVzdHNfaW50ZWdyYXRpb250ZXN0LnB5CisrKyBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5 b3V0X3Rlc3RzL3J1bl93ZWJraXRfdGVzdHNfaW50ZWdyYXRpb250ZXN0LnB5CkBAIC00MTgsNiAr NDE4LDEwIEBAIGNsYXNzIE1haW5UZXN0KHVuaXR0ZXN0LlRlc3RDYXNlLCBTdHJlYW1UZXN0aW5n TWl4aW4pOgogICAgICAgICBzZWxmLmFzc2VydEVxdWFscyhnZXRfdGVzdHNfcnVuKFsnLS1za2lw cGVkPW9ubHknLCAncGFzc2VzJ10sIHRlc3RzX2luY2x1ZGVkPVRydWUsIGZsYXR0ZW5fYmF0Y2hl cz1UcnVlKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgWydwYXNzZXMvc2tpcHBlZC9za2lw Lmh0bWwnXSkKIAorICAgICAgICAjIE5vdyBjaGVjayB0aGF0IHdlIGRvbid0IHJ1biBhbnl0aGlu Zy4KKyAgICAgICAgc2VsZi5hc3NlcnRFcXVhbHMoZ2V0X3Rlc3RzX3J1bihbJy0tc2tpcHBlZD1h bHdheXMnLCAncGFzc2VzL3NraXBwZWQvc2tpcC5odG1sJ10sIHRlc3RzX2luY2x1ZGVkPVRydWUs IGZsYXR0ZW5fYmF0Y2hlcz1UcnVlKSwKKyAgICAgICAgICAgICAgICAgICAgICAgICAgW10pCisK ICAgICBkZWYgdGVzdF9pdGVyYXRpb25zKHNlbGYpOgogICAgICAgICB0ZXN0c190b19ydW4gPSBb J3Bhc3Nlcy9pbWFnZS5odG1sJywgJ3Bhc3Nlcy90ZXh0Lmh0bWwnXQogICAgICAgICB0ZXN0c19y dW4gPSBnZXRfdGVzdHNfcnVuKFsnLS1pdGVyYXRpb25zJywgJzInXSArIHRlc3RzX3RvX3J1biwg dGVzdHNfaW5jbHVkZWQ9VHJ1ZSwgZmxhdHRlbl9iYXRjaGVzPVRydWUpCg==