37783 2010-04-18 16:46:41 -0700 new-run-webkit-tests: add a way to print out or retry the last set of tests that failed 2010-04-19 14:57:08 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) PC OS X 10.5 RESOLVED FIXED P2 Normal --- 1 dpranke dpranke abarth cjerdonek eric ojan tony oldest_to_newest 213862 0 dpranke 2010-04-18 16:46:41 -0700 I frequently want to re-run new-run-webkit-tests and just re-test the set of tests that failed. It would be nice if there was a way to print the last tests out easily or read them into new-run-webkit-tests. The data is in the unexpected_results.json file, so we just have to process it. 213867 1 53646 dpranke 2010-04-18 17:17:08 -0700 Created attachment 53646 Patch 213927 2 53646 eric 2010-04-19 00:08:45 -0700 Comment on attachment 53646 Patch Looks useful. 214036 3 53646 ojan 2010-04-19 08:14:06 -0700 Comment on attachment 53646 Patch > + optparse.make_option("--print-last-results", action="store_true", > + default=False, help="print the tests in the last run that " > + "had unexpected results."), Do we really need an option for this? Lets just always do this. If python had a way of doing "hidden" commands the way the webkit-patch does for commands, I'd be OK with adding all these options. But having them all visible to users of the script by default makes the script harder to use. > + optparse.make_option("--retry-last-results", action="store_true", > + default=False, help="re-try the tests in the last run that " > + "had unexpected results."), Nit: last-results isn't clear without reading the help text or the code. How about --retry-unexpected-failures? 214099 4 dpranke 2010-04-19 10:41:31 -0700 (In reply to comment #3) > (From update of attachment 53646 [details]) > > + optparse.make_option("--print-last-results", action="store_true", > > + default=False, help="print the tests in the last run that " > > + "had unexpected results."), > > Do we really need an option for this? Lets just always do this. If python had a > way of doing "hidden" commands the way the webkit-patch does for commands, I'd > be OK with adding all these options. But having them all visible to users of > the script by default makes the script harder to use. What do you mean by "always do this"? The patch prints the list and exits if that flag is passed in. > > + optparse.make_option("--retry-last-results", action="store_true", > > + default=False, help="re-try the tests in the last run that " > > + "had unexpected results."), > > Nit: last-results isn't clear without reading the help text or the code. How > about --retry-unexpected-failures? Of course, we retry unexpected passes as well. --retry-unexpected-results? I could also live with -failures, as that is the main intent. -- Dirk 214262 5 53723 dpranke 2010-04-19 14:56:05 -0700 Created attachment 53723 Patch 214264 6 dpranke 2010-04-19 14:57:08 -0700 Committed r57843: <http://trac.webkit.org/changeset/57843> 53646 2010-04-18 17:17:08 -0700 2010-04-19 14:56:00 -0700 Patch bug-37783-20100418171707.patch text/plain 2726 dpranke ZGlmZiAtLWdpdCBhL1dlYktpdFRvb2xzL0NoYW5nZUxvZyBiL1dlYktpdFRvb2xzL0NoYW5nZUxv ZwppbmRleCA5OThhNmQzLi5lNTc0MWExIDEwMDY0NAotLS0gYS9XZWJLaXRUb29scy9DaGFuZ2VM b2cKKysrIGIvV2ViS2l0VG9vbHMvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsMTUgQEAKKzIwMTAtMDQt MTggIERpcmsgUHJhbmtlICA8ZHByYW5rZUBjaHJvbWl1bS5vcmc+CisKKyAgICAgICAgUmV2aWV3 ZWQgYnkgTk9CT0RZIChPT1BTISkuCisKKyAgICAgICAgbmV3LXJ1bi13ZWJraXQtdGVzdHM6IGFk ZCBhIHdheSAoLS1wcmludC1sYXN0LXJlc3VsdHMpIHRvIChyZS0pcHJpbnQKKyAgICAgICAgb3V0 IHRoZSB0ZXN0cyB0aGF0IHByb2R1Y2VkIHVuZXhwZWN0ZWQgcmVzdWx0cyBpbiB0aGUgbGFzdCBy dW4uIEFsc28KKyAgICAgICAgYWRkIGEgd2F5ICgtLXJldHJ5LWxhc3QtcmVzdWx0cykgdG8gYXV0 b21hdGljYWxseSByZS10ZXN0IHRoZW0uCisKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5v cmcvc2hvd19idWcuY2dpP2lkPTM3NzgzCisKKyAgICAgICAgKiBTY3JpcHRzL3dlYmtpdHB5L2xh eW91dF90ZXN0cy9ydW5fd2Via2l0X3Rlc3RzLnB5OgorCiAyMDEwLTA0LTE1ICBTYW0gV2Vpbmln ICA8c2FtQHdlYmtpdC5vcmc+CiAKICAgICAgICAgUmV2aWV3ZWQgYnkgQW5kZXJzIENhcmxzc29u LgpkaWZmIC0tZ2l0IGEvV2ViS2l0VG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMv cnVuX3dlYmtpdF90ZXN0cy5weSBiL1dlYktpdFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0 X3Rlc3RzL3J1bl93ZWJraXRfdGVzdHMucHkKaW5kZXggMDU5ZmQwOS4uMjc3MTY0OSAxMDA3NTUK LS0tIGEvV2ViS2l0VG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcnVuX3dlYmtp dF90ZXN0cy5weQorKysgYi9XZWJLaXRUb29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0 cy9ydW5fd2Via2l0X3Rlc3RzLnB5CkBAIC0xNDE5LDYgKzE0MTksMTcgQEAgZGVmIG1haW4ob3B0 aW9ucywgYXJncywgcHJpbnRfcmVzdWx0cz1UcnVlKToKICAgICAgICAgIyBEZWJ1ZyBvciBSZWxl YXNlLgogICAgICAgICBvcHRpb25zLnJlc3VsdHNfZGlyZWN0b3J5ID0gcG9ydF9vYmoucmVzdWx0 c19kaXJlY3RvcnkoKQogCisgICAgaWYgb3B0aW9ucy5wcmludF9sYXN0X3Jlc3VsdHMgb3Igb3B0 aW9ucy5yZXRyeV9sYXN0X3Jlc3VsdHM6CisgICAgICAgIHVuZXhwZWN0ZWRfcmVzdWx0c19maWxl bmFtZSA9IG9zLnBhdGguam9pbigKKyAgICAgICAgICAgb3B0aW9ucy5yZXN1bHRzX2RpcmVjdG9y eSwgInVuZXhwZWN0ZWRfcmVzdWx0cy5qc29uIikKKyAgICAgICAgZiA9IGZpbGUodW5leHBlY3Rl ZF9yZXN1bHRzX2ZpbGVuYW1lKQorICAgICAgICByZXN1bHRzID0gc2ltcGxlanNvbi5sb2FkKGYp CisgICAgICAgIGYuY2xvc2UoKQorICAgICAgICBsYXN0X3Jlc3VsdHMgPSByZXN1bHRzWyd0ZXN0 cyddLmtleXMoKQorICAgICAgICBpZiBvcHRpb25zLnByaW50X2xhc3RfcmVzdWx0czoKKyAgICAg ICAgICAgIHByaW50ICJcbiIuam9pbihsYXN0X3Jlc3VsdHMpICsgIlxuIgorICAgICAgICAgICAg cmV0dXJuIDAKKwogICAgIGlmIG9wdGlvbnMuY2xvYmJlcl9vbGRfcmVzdWx0czoKICAgICAgICAg IyBKdXN0IGNsb2JiZXIgdGhlIGFjdHVhbCB0ZXN0IHJlc3VsdHMgZGlyZWN0b3JpZXMgc2luY2Ug dGhlIG90aGVyCiAgICAgICAgICMgZmlsZXMgaW4gdGhlIHJlc3VsdHMgZGlyZWN0b3J5IGFyZSBl eHBsaWNpdGx5IHVzZWQgZm9yIGNyb3NzLXJ1bgpAQCAtMTQ2Miw2ICsxNDczLDggQEAgZGVmIG1h aW4ob3B0aW9ucywgYXJncywgcHJpbnRfcmVzdWx0cz1UcnVlKToKICAgICBwYXRocyA9IG5ld19h cmdzCiAgICAgaWYgbm90IHBhdGhzOgogICAgICAgICBwYXRocyA9IFtdCisgICAgaWYgb3B0aW9u cy5yZXRyeV9sYXN0X3Jlc3VsdHM6CisgICAgICAgIHBhdGhzICs9IGxhc3RfcmVzdWx0cwogICAg IGlmIG9wdGlvbnMudGVzdF9saXN0OgogICAgICAgICBwYXRocyArPSByZWFkX3Rlc3RfZmlsZXMo b3B0aW9ucy50ZXN0X2xpc3QpCiAKQEAgLTE2ODQsNiArMTY5NywxMiBAQCBkZWYgcGFyc2VfYXJn cyhhcmdzPU5vbmUpOgogICAgICAgICAjICAgICAgRXhpdCBhZnRlciB0aGUgZmlyc3QgTiBmYWls dXJlcyBpbnN0ZWFkIG9mIHJ1bm5pbmcgYWxsIHRlc3RzCiAgICAgICAgICMgRklYTUU6IGNvbnNp ZGVyOiAtLWl0ZXJhdGlvbnMgbgogICAgICAgICAjICAgICAgTnVtYmVyIG9mIHRpbWVzIHRvIHJ1 biB0aGUgc2V0IG9mIHRlc3RzIChlLmcuIEFCQ0FCQ0FCQykKKyAgICAgICAgb3B0cGFyc2UubWFr ZV9vcHRpb24oIi0tcHJpbnQtbGFzdC1yZXN1bHRzIiwgYWN0aW9uPSJzdG9yZV90cnVlIiwKKyAg ICAgICAgICAgIGRlZmF1bHQ9RmFsc2UsIGhlbHA9InByaW50IHRoZSB0ZXN0cyBpbiB0aGUgbGFz dCBydW4gdGhhdCAiCisgICAgICAgICAgICAiaGFkIHVuZXhwZWN0ZWQgcmVzdWx0cy4iKSwKKyAg ICAgICAgb3B0cGFyc2UubWFrZV9vcHRpb24oIi0tcmV0cnktbGFzdC1yZXN1bHRzIiwgYWN0aW9u PSJzdG9yZV90cnVlIiwKKyAgICAgICAgICAgIGRlZmF1bHQ9RmFsc2UsIGhlbHA9InJlLXRyeSB0 aGUgdGVzdHMgaW4gdGhlIGxhc3QgcnVuIHRoYXQgIgorICAgICAgICAgICAgImhhZCB1bmV4cGVj dGVkIHJlc3VsdHMuIiksCiAgICAgXQogCiAgICAgbWlzY19vcHRpb25zID0gWwo= 53723 2010-04-19 14:56:05 -0700 2010-04-19 14:56:26 -0700 Patch bug-37783-20100419145604.patch text/plain 2934 dpranke ZGlmZiAtLWdpdCBhL1dlYktpdFRvb2xzL0NoYW5nZUxvZyBiL1dlYktpdFRvb2xzL0NoYW5nZUxv ZwppbmRleCBmNTQ3ZmU1Li44MzJkZGY2IDEwMDY0NAotLS0gYS9XZWJLaXRUb29scy9DaGFuZ2VM b2cKKysrIGIvV2ViS2l0VG9vbHMvQ2hhbmdlTG9nCkBAIC0yLDYgKzIsMTkgQEAKIAogICAgICAg ICBSZXZpZXdlZCBieSBFcmljIFNlaWRlbC4KIAorICAgICAgICBuZXctcnVuLXdlYmtpdC10ZXN0 czogYWRkIGEgd2F5ICgtLXByaW50LXVuZXhwZWN0ZWQtcmVzdWx0cykgdG8KKyAgICAgICAgKHJl LSlwcmludCBvdXQgdGhlIHRlc3RzIHRoYXQgcHJvZHVjZWQgdW5leHBlY3RlZCByZXN1bHRzIGlu IHRoZQorICAgICAgICBsYXN0IHJ1bi4gQWxzbyBhZGQgYSB3YXkgKC0tcmV0cnktdW5leHBlY3Rl ZC1yZXN1bHRzKSB0bworICAgICAgICBhdXRvbWF0aWNhbGx5IHJlLXRlc3QgdGhlbS4KKworICAg ICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9Mzc3ODMKKworICAg ICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRfdGVzdHMucHk6 CisKKzIwMTAtMDQtMTkgIERpcmsgUHJhbmtlICA8ZHByYW5rZUBjaHJvbWl1bS5vcmc+CisKKyAg ICAgICAgUmV2aWV3ZWQgYnkgRXJpYyBTZWlkZWwuCisKICAgICAgICAgUmVzdHJ1Y3R1cmUgdGhl IGxvZ2dpbmcgaW4gbmV3LXJ1bi13ZWJraXQtdGVzdHMgc28gdGhhdCBtYW55IG9mIGxvZwogICAg ICAgICBtZXNzYWdlcyB0aGF0IHdlcmUgbG9nZ2VkIHRvIHRoZSBNZXRlcmVkU3RyZWFtIGFsc28g Z2V0IGxvZ2dlZCBpbgogICAgICAgICAtLXZlcmJvc2UgbW9kZS4KZGlmZiAtLWdpdCBhL1dlYktp dFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRfdGVzdHMucHkg Yi9XZWJLaXRUb29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9ydW5fd2Via2l0X3Rl c3RzLnB5CmluZGV4IGM3NGE3OTkuLjk3NzAwODYgMTAwNzU1Ci0tLSBhL1dlYktpdFRvb2xzL1Nj cmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRfdGVzdHMucHkKKysrIGIvV2Vi S2l0VG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcnVuX3dlYmtpdF90ZXN0cy5w eQpAQCAtMTQzOSw2ICsxNDM5LDE4IEBAIGRlZiBtYWluKG9wdGlvbnMsIGFyZ3MsIHByaW50X3Jl c3VsdHM9VHJ1ZSk6CiAgICAgICAgICMgRGVidWcgb3IgUmVsZWFzZS4KICAgICAgICAgb3B0aW9u cy5yZXN1bHRzX2RpcmVjdG9yeSA9IHBvcnRfb2JqLnJlc3VsdHNfZGlyZWN0b3J5KCkKIAorICAg IGxhc3RfdW5leHBlY3RlZF9yZXN1bHRzID0gW10KKyAgICBpZiBvcHRpb25zLnByaW50X3VuZXhw ZWN0ZWRfcmVzdWx0cyBvciBvcHRpb25zLnJldHJ5X3VuZXhwZWN0ZWRfcmVzdWx0czoKKyAgICAg ICAgdW5leHBlY3RlZF9yZXN1bHRzX2ZpbGVuYW1lID0gb3MucGF0aC5qb2luKAorICAgICAgICAg ICBvcHRpb25zLnJlc3VsdHNfZGlyZWN0b3J5LCAidW5leHBlY3RlZF9yZXN1bHRzLmpzb24iKQor ICAgICAgICBmID0gZmlsZSh1bmV4cGVjdGVkX3Jlc3VsdHNfZmlsZW5hbWUpCisgICAgICAgIHJl c3VsdHMgPSBzaW1wbGVqc29uLmxvYWQoZikKKyAgICAgICAgZi5jbG9zZSgpCisgICAgICAgIGxh c3RfdW5leHBlY3RlZF9yZXN1bHRzID0gcmVzdWx0c1sndGVzdHMnXS5rZXlzKCkKKyAgICAgICAg aWYgb3B0aW9ucy5wcmludF91bmV4cGVjdGVkX3Jlc3VsdHM6CisgICAgICAgICAgICBwcmludCAi XG4iLmpvaW4obGFzdF91bmV4cGVjdGVkX3Jlc3VsdHMpICsgIlxuIgorICAgICAgICAgICAgcmV0 dXJuIDAKKwogICAgIGlmIG9wdGlvbnMuY2xvYmJlcl9vbGRfcmVzdWx0czoKICAgICAgICAgIyBK dXN0IGNsb2JiZXIgdGhlIGFjdHVhbCB0ZXN0IHJlc3VsdHMgZGlyZWN0b3JpZXMgc2luY2UgdGhl IG90aGVyCiAgICAgICAgICMgZmlsZXMgaW4gdGhlIHJlc3VsdHMgZGlyZWN0b3J5IGFyZSBleHBs aWNpdGx5IHVzZWQgZm9yIGNyb3NzLXJ1bgpAQCAtMTQ4Miw2ICsxNDk0LDcgQEAgZGVmIG1haW4o b3B0aW9ucywgYXJncywgcHJpbnRfcmVzdWx0cz1UcnVlKToKICAgICBwYXRocyA9IG5ld19hcmdz CiAgICAgaWYgbm90IHBhdGhzOgogICAgICAgICBwYXRocyA9IFtdCisgICAgcGF0aHMgKz0gbGFz dF91bmV4cGVjdGVkX3Jlc3VsdHMKICAgICBpZiBvcHRpb25zLnRlc3RfbGlzdDoKICAgICAgICAg cGF0aHMgKz0gcmVhZF90ZXN0X2ZpbGVzKG9wdGlvbnMudGVzdF9saXN0KQogCkBAIC0xNzMxLDYg KzE3NDQsMTIgQEAgZGVmIHBhcnNlX2FyZ3MoYXJncz1Ob25lKToKICAgICAgICAgIyAgICAgIEV4 aXQgYWZ0ZXIgdGhlIGZpcnN0IE4gZmFpbHVyZXMgaW5zdGVhZCBvZiBydW5uaW5nIGFsbCB0ZXN0 cwogICAgICAgICAjIEZJWE1FOiBjb25zaWRlcjogLS1pdGVyYXRpb25zIG4KICAgICAgICAgIyAg ICAgIE51bWJlciBvZiB0aW1lcyB0byBydW4gdGhlIHNldCBvZiB0ZXN0cyAoZS5nLiBBQkNBQkNB QkMpCisgICAgICAgIG9wdHBhcnNlLm1ha2Vfb3B0aW9uKCItLXByaW50LXVuZXhwZWN0ZWQtcmVz dWx0cyIsIGFjdGlvbj0ic3RvcmVfdHJ1ZSIsCisgICAgICAgICAgICBkZWZhdWx0PUZhbHNlLCBo ZWxwPSJwcmludCB0aGUgdGVzdHMgaW4gdGhlIGxhc3QgcnVuIHRoYXQgIgorICAgICAgICAgICAg ImhhZCB1bmV4cGVjdGVkIHJlc3VsdHMuIiksCisgICAgICAgIG9wdHBhcnNlLm1ha2Vfb3B0aW9u KCItLXJldHJ5LXVuZXhwZWN0ZWQtcmVzdWx0cyIsIGFjdGlvbj0ic3RvcmVfdHJ1ZSIsCisgICAg ICAgICAgICBkZWZhdWx0PUZhbHNlLCBoZWxwPSJyZS10cnkgdGhlIHRlc3RzIGluIHRoZSBsYXN0 IHJ1biB0aGF0ICIKKyAgICAgICAgICAgICJoYWQgdW5leHBlY3RlZCByZXN1bHRzLiIpLAogICAg IF0KIAogICAgIG1pc2Nfb3B0aW9ucyA9IFsK