31829 2009-11-24 06:02:12 -0800 In run-webkit-tests $failureCount shouldn't include new tests 2010-05-12 13:18:28 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) PC All RESOLVED FIXED P2 Normal --- 1 ossy webkit-unassigned abarth commit-queue eric hausmann kenneth mrobinson mrowe oldest_to_newest 166047 0 ossy 2009-11-24 06:02:12 -0800 If you execute run-webkit-tests with --exitAfterNFailures NUM, the script will exit after NUM failed tests. I think, the $failureCount shouldn't include new tests. 166048 1 43765 ossy 2009-11-24 06:04:56 -0800 Created attachment 43765 proposed patch 166105 2 eric 2009-11-24 08:32:12 -0800 Can you explain the use case? The bots want this behavior since new tests will still cause run-webkit-tests to exit(1) in the end, even if you change it to not exit early here. 166106 3 ossy 2009-11-24 08:48:24 -0800 (In reply to comment #2) > Can you explain the use case? The bots want this behavior since new tests will > still cause run-webkit-tests to exit(1) in the end, even if you change it to > not exit early here. If there aren't failing and crashing tests but some new, the buildbot will be henceforward green.(run-webkit-tests case - orange) In my opinion, only new tests don't explain stop run-webkit-tests on buildbot ahead of time. Because stopping buildbot earlier can hide latter failing or crashing tests. 166110 4 eric 2009-11-24 09:00:07 -0800 I think this is papering over the symptom instead of solving the problem. The Qt bot shouldn't have missing test results in the first place. :( In the case of the commit-queue we would prefer to exit after the first test requiring new results or the first failure. If that has to be refactored into separate options to work with the Qt buildbot that's OK, but as this is this would cause the commit-queue to fail slower (not a huge issue, but certainly one worth mentioning). 166118 5 ossy 2009-11-24 09:16:17 -0800 (In reply to comment #4) > I think this is papering over the symptom instead of solving the problem. The > Qt bot shouldn't have missing test results in the first place. :( Yes, you're right, our port shouldn't have so many missing test results. Unfortunately we can't keep step with new tests. We regularly put these tests into skiplist when we haven't any time to make expected files. > In the case of the commit-queue we would prefer to exit after the first test > requiring new results or the first failure. If that has to be refactored into > separate options to work with the Qt buildbot that's OK, but as this is this > would cause the commit-queue to fail slower (not a huge issue, but certainly > one worth mentioning). Refactoring into separate options is a good idea, I will try to do it. 166119 6 eric 2009-11-24 09:23:21 -0800 Can't we just fix the Qt bot having so many missing tests? We don't generally have 20 tests added per day. Chromium has a rebaselining tool, which knows how to suck results off of a buildbot output, which is used in chromium land to suck results off of the try bot results, and thus provide patch-writters with complete results for all platforms. We could build something similar in WebKit land, if only for Qt. :) If folks aren't looking at the bots enough now so as to keep them from going red, I'm not sure why preventing them from displaying less data after 20 failures, vs. 40 failures matters. And I'm not sure why new tests being "false failures" matters either. I think we're still trying to solve the wrong problems with this bug (and the 20 -> 40 change you filed). :) 166534 7 43864 ossy 2009-11-25 12:54:33 -0800 Created attachment 43864 updated patch 166542 8 ossy 2009-11-25 13:15:14 -0800 (In reply to comment #7) > Created an attachment (id=43864) [details] > updated patch sorry, wrong bug :) 224820 9 43765 eric 2010-05-12 12:52:59 -0700 Comment on attachment 43765 proposed patch OK. 224829 10 ossy 2010-05-12 13:00:37 -0700 *** Bug 39012 has been marked as a duplicate of this bug. *** 224837 11 ossy 2010-05-12 13:18:28 -0700 (In reply to comment #9) > (From update of attachment 43765 [details]) > OK. Landed: http://trac.webkit.org/changeset/59254 and my fault fixed here: http://trac.webkit.org/changeset/59255 43765 2009-11-24 06:04:56 -0800 2010-05-12 12:52:59 -0700 proposed patch exitAfterN.patch text/plain 1513 ossy ZGlmZiAtLWdpdCBhL1dlYktpdFRvb2xzL0NoYW5nZUxvZyBiL1dlYktpdFRvb2xzL0NoYW5nZUxv ZwppbmRleCAzOTI5NWY5Li44OWQ4N2YyIDEwMDY0NAotLS0gYS9XZWJLaXRUb29scy9DaGFuZ2VM b2cKKysrIGIvV2ViS2l0VG9vbHMvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsMTIgQEAKKzIwMDktMTEt MjQgIENzYWJhIE9zenRyb2dvbsOhYyAgPG9zc3lAd2Via2l0Lm9yZz4KKworICAgICAgICBSZXZp ZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICBJbiBydW4td2Via2l0LXRlc3RzICRm YWlsdXJlQ291bnQgc2hvdWxkbid0IGluY2x1ZGUgbmV3IHRlc3RzCisgICAgICAgIGh0dHBzOi8v YnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0zMTgyOQorCisgICAgICAgICogU2NyaXB0 cy9ydW4td2Via2l0LXRlc3RzOgorCiAyMDA5LTExLTIzICBMYXN6bG8gR29tYm9zICA8bGFzemxv LjEuZ29tYm9zQG5va2lhLmNvbT4KIAogICAgICAgICBSZXZpZXdlZCBieSBLZW5uZXRoIFJvaGRl IENocmlzdGlhbnNlbi4KZGlmZiAtLWdpdCBhL1dlYktpdFRvb2xzL1NjcmlwdHMvcnVuLXdlYmtp dC10ZXN0cyBiL1dlYktpdFRvb2xzL1NjcmlwdHMvcnVuLXdlYmtpdC10ZXN0cwppbmRleCAxZTM3 Mjg2Li4wZWZmY2JjIDEwMDc1NQotLS0gYS9XZWJLaXRUb29scy9TY3JpcHRzL3J1bi13ZWJraXQt dGVzdHMKKysrIGIvV2ViS2l0VG9vbHMvU2NyaXB0cy9ydW4td2Via2l0LXRlc3RzCkBAIC05MTcs NyArOTE3LDggQEAgZm9yIG15ICR0ZXN0IChAdGVzdHMpIHsKICAgICAjIC0tcmVzZXQtcmVzdWx0 cyBkb2VzIG5vdCBjaGVjayBwYXNzIHZzLiBmYWlsLCBzbyBleGl0QWZ0ZXJORmFpbHVyZXMgbWFr ZXMgbm8gc2Vuc2Ugd2l0aCAtLXJlc2V0LXJlc3VsdHMuCiAgICAgaWYgKCRleGl0QWZ0ZXJORmFp bHVyZXMgJiYgISRyZXNldFJlc3VsdHMpIHsKICAgICAgICAgbXkgJHBhc3NDb3VudCA9ICRjb3Vu dHN7bWF0Y2h9IHx8IDA7ICMgJGNvdW50c3ttYXRjaH0gd2lsbCBiZSB1bmRlZmluZWQgaWYgd2Un dmUgbm90IHlldCBwYXNzZWQgYSB0ZXN0IChlLmcuIHRoZSBmaXJzdCB0ZXN0IGZhaWxzKS4KLSAg ICAgICAgbXkgJGZhaWx1cmVDb3VudCA9ICRjb3VudCAtICRwYXNzQ291bnQ7ICMgIkZhaWx1cmUi IGhlcmUgaW5jbHVkZXMgbmV3IHRlc3RzLCB0aW1lb3V0cywgY3Jhc2hlcywgZXRjLgorICAgICAg ICBteSAkbmV3Q291bnQgPSAkY291bnRze25ld30gfHwgMDsKKyAgICAgICAgbXkgJGZhaWx1cmVD b3VudCA9ICRjb3VudCAtICRwYXNzQ291bnQgLSAkbmV3Q291bnQ7ICMgIkZhaWx1cmUiIGhlcmUg aW5jbHVkZXMgdGltZW91dHMsIGNyYXNoZXMsIGV0Yy4KICAgICAgICAgaWYgKCRmYWlsdXJlQ291 bnQgPj0gJGV4aXRBZnRlck5GYWlsdXJlcykgewogICAgICAgICAgICAgcHJpbnQgIlxuRXhpdGlu ZyBlYXJseSBhZnRlciAkZmFpbHVyZUNvdW50IGZhaWx1cmVzLiAkY291bnQgdGVzdHMgcnVuLiI7 CiAgICAgICAgICAgICBjbG9zZUR1bXBUb29sKCk7Cg== 43864 2009-11-25 12:54:33 -0800 2010-05-12 12:49:28 -0700 updated patch skippedonly_proposed2.patch text/plain 3742 ossy ZGlmZiAtLWdpdCBhL1dlYktpdFRvb2xzL0NoYW5nZUxvZyBiL1dlYktpdFRvb2xzL0NoYW5nZUxv ZwppbmRleCBlMTQwZjIwLi40MzJmY2VhIDEwMDY0NAotLS0gYS9XZWJLaXRUb29scy9DaGFuZ2VM b2cKKysrIGIvV2ViS2l0VG9vbHMvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsMTIgQEAKKzIwMDktMTEt MjUgIENzYWJhIE9zenRyb2dvbsOhYyAgPG9zc3lAd2Via2l0Lm9yZz4KKworICAgICAgICBSZXZp ZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICBydW4td2Via2l0LXRlc3RzIGRvZXNu J3QgYWNjZXB0IGRpcmVjdG9yaWVzL2ZpbGVzIHdpdGggLS1za2lwcGVkPW9ubHkgcGFyYW1ldGVy CisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0zMTc5OQor CisgICAgICAgICogU2NyaXB0cy9ydW4td2Via2l0LXRlc3RzOiBGaXhlZC4KKwogMjAwOS0xMS0y NSAgRXJpYyBTZWlkZWwgIDxlcmljQHdlYmtpdC5vcmc+CiAKICAgICAgICAgUmV2aWV3ZWQgYnkg QWRhbSBCYXJ0aC4KZGlmZiAtLWdpdCBhL1dlYktpdFRvb2xzL1NjcmlwdHMvcnVuLXdlYmtpdC10 ZXN0cyBiL1dlYktpdFRvb2xzL1NjcmlwdHMvcnVuLXdlYmtpdC10ZXN0cwppbmRleCAxZTM3Mjg2 Li42YTkxZTgwIDEwMDc1NQotLS0gYS9XZWJLaXRUb29scy9TY3JpcHRzL3J1bi13ZWJraXQtdGVz dHMKKysrIGIvV2ViS2l0VG9vbHMvU2NyaXB0cy9ydW4td2Via2l0LXRlc3RzCkBAIC05Nyw2ICs5 Nyw3IEBAIHN1YiBwYXRoY21wKCQkKTsKIHN1YiBwcmludEZhaWx1cmVNZXNzYWdlRm9yVGVzdCgk JCk7CiBzdWIgcHJvY2Vzc0lnbm9yZVRlc3RzKCQkKTsKIHN1YiByZWFkRnJvbUR1bXBUb29sV2l0 aFRpbWVyKCoqKTsKK3N1YiByZWFkU2tpcHBlZEZpbGVzKCQpOwogc3ViIHJlY29yZEFjdHVhbFJl c3VsdHNBbmREaWZmKCQkKTsKIHN1YiBzYW1wbGVEdW1wVG9vbCgpOwogc3ViIHNldEZpbGVIYW5k bGVOb25CbG9ja2luZygqJCk7CkBAIC0yOTcsOCArMjk4LDYgQEAgaWYgKCEkZ2V0T3B0aW9uc1Jl c3VsdCB8fCAkc2hvd0hlbHApIHsKIG15ICRpZ25vcmVTa2lwcGVkID0gJHRyZWF0U2tpcHBlZCBl cSAiaWdub3JlIjsKIG15ICRza2lwcGVkT25seSA9ICR0cmVhdFNraXBwZWQgZXEgIm9ubHkiOwog Ci0hJHNraXBwZWRPbmx5IHx8IEBBUkdWID09IDAgb3IgZGllICItLXNraXBwZWQ9b25seSBjYW5u b3QgYmUgdXNlZCB3aGVuIHRlc3RzIGFyZSBzcGVjaWZpZWQgb24gdGhlIGNvbW1hbmQgbGluZS4i OwotCiBteSAkY29uZmlndXJhdGlvbiA9IGNvbmZpZ3VyYXRpb24oKTsKIAogJHZlcmJvc2UgPSAx IGlmICR0ZXN0c1BlckR1bXBUb29sID09IDE7CkBAIC00NTUsNyArNDU0LDE3IEBAIGlmICghY2hl Y2tXZWJDb3JlV0NTU1N1cHBvcnQoMCkpIHsKIH0KIAogcHJvY2Vzc0lnbm9yZVRlc3RzKCRpZ25v cmVUZXN0cywgImlnbm9yZS10ZXN0cyIpIGlmICRpZ25vcmVUZXN0czsKLXJlYWRTa2lwcGVkRmls ZXMoKSB1bmxlc3MgJGlnbm9yZVNraXBwZWQ7CitpZiAoISRpZ25vcmVTa2lwcGVkKSB7CisgICAg aWYgKCEkc2tpcHBlZE9ubHkgfHwgQEFSR1YgPT0gMCkgeworICAgICAgICByZWFkU2tpcHBlZEZp bGVzKCIiKTsKKyAgICB9IGVsc2UgeworICAgICAgICAjIFNpbmNlIHJlYWRTa2lwcGVkRmlsZXMo KSBhcHBlbmRzIHRvIEBBUkdWLCB3ZSBtdXN0IHVzZSBhIGZvcmVhY2gKKyAgICAgICAgIyBsb29w IHNvIHRoYXQgd2Ugb25seSBpdGVyYXRlIG92ZXIgdGhlIG9yaWdpbmFsIGFyZ3VtZW50IGxpc3Qu CisgICAgICAgIGZvcmVhY2ggbXkgJGFyZ251bSAoMCAuLiAkI0FSR1YpIHsKKyAgICAgICAgICAg IHJlYWRTa2lwcGVkRmlsZXMoc2hpZnQgQEFSR1YpOworICAgICAgICB9CisgICAgfQorfQogCiBt eSBAdGVzdHMgPSBmaW5kVGVzdHNUb1J1bigpOwogCkBAIC0yMDg5LDggKzIwOTgsMTAgQEAgc3Vi IGZpbGVTaG91bGRCZUlnbm9yZWQKICAgICByZXR1cm4gMDsKIH0KIAotc3ViIHJlYWRTa2lwcGVk RmlsZXMKK3N1YiByZWFkU2tpcHBlZEZpbGVzKCQpCiB7CisgICAgbXkgKCRjb25zdHJhaW50UGF0 aCkgPSBAXzsKKwogICAgIGZvcmVhY2ggbXkgJGxldmVsIChAcGxhdGZvcm1UZXN0SGllcmFyY2h5 KSB7CiAgICAgICAgIGlmIChvcGVuIFNLSVBQRUQsICI8IiwgIiRsZXZlbC9Ta2lwcGVkIikgewog ICAgICAgICAgICAgaWYgKCR2ZXJib3NlKSB7CkBAIC0yMTA2LDcgKzIxMTcsMTggQEAgc3ViIHJl YWRTa2lwcGVkRmlsZXMKICAgICAgICAgICAgICAgICBpZiAoJHNraXBwZWQgJiYgJHNraXBwZWQg IX4gL14jLykgewogICAgICAgICAgICAgICAgICAgICBpZiAoJHNraXBwZWRPbmx5KSB7CiAgICAg ICAgICAgICAgICAgICAgICAgICBpZiAoISZmaWxlU2hvdWxkQmVJZ25vcmVkKCRza2lwcGVkKSkg ewotICAgICAgICAgICAgICAgICAgICAgICAgICAgIHB1c2goQEFSR1YsICRza2lwcGVkKTsKKyAg ICAgICAgICAgICAgICAgICAgICAgICAgICBpZiAoISRjb25zdHJhaW50UGF0aCkgeworICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAjIEFsd2F5cyBhZGQgJHNraXBwZWQgc2luY2Ugbm8g Y29uc3RyYWludCBwYXRoIHdhcyBzcGVjaWZpZWQgb24gdGhlIGNvbW1hbmQgbGluZS4KKyAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgcHVzaChAQVJHViwgJHNraXBwZWQpOworICAgICAg ICAgICAgICAgICAgICAgICAgICAgIH0gZWxzaWYgKCRza2lwcGVkID1+IC9eKCRjb25zdHJhaW50 UGF0aCkvKSB7CisgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICMgQWRkICRza2lwcGVk IG9ubHkgaWYgaXQgbWF0Y2hlcyB0aGUgY3VycmVudCBwYXRoIGNvbnN0cmFpbnQsIGUuZy4sCisg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICMgIi0tc2tpcHBlZD1vbmx5IGRpcjEiIHdp dGggImRpcjEvZmlsZTEuaHRtbCIgb24gdGhlIHNraXBwZWQgbGlzdC4KKyAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgcHVzaChAQVJHViwgJHNraXBwZWQpOworICAgICAgICAgICAgICAg ICAgICAgICAgICAgIH0gZWxzaWYgKCRjb25zdHJhaW50UGF0aCA9fiAvXigkc2tpcHBlZCkvKSB7 CisgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICMgQWRkIGN1cnJlbnQgcGF0aCBjb25z dHJhaW50IGlmIGl0IGlzIG1vcmUgc3BlY2lmaWMgdGhhbiB0aGUgc2tpcCBsaXN0IGVudHJ5LAor ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAjIGUuZy4sICItLXNraXBwZWQ9b25seSBk aXIxL2RpcjIvZGlyMyIgd2l0aCAiZGlyMSIgb24gdGhlIHNraXBwZWQgbGlzdC4KKyAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgcHVzaChAQVJHViwgJGNvbnN0cmFpbnRQYXRoKTsKKyAg ICAgICAgICAgICAgICAgICAgICAgICAgICB9CiAgICAgICAgICAgICAgICAgICAgICAgICB9IGVs c2lmICgkdmVyYm9zZSkgewogICAgICAgICAgICAgICAgICAgICAgICAgICAgIHByaW50ICIgICAg JHNraXBwZWRcbiI7CiAgICAgICAgICAgICAgICAgICAgICAgICB9CkBAIC0yMTc3LDYgKzIxOTks OSBAQCBzdWIgZmluZFRlc3RzVG9SdW4KICAgICAgICAgfQogICAgIH0KIAorICAgICMgUmVtb3Zl IGR1cGxpY2F0ZSB0ZXN0cworICAgIEB0ZXN0c1RvUnVuID0ga2V5cyAle3sgbWFwIHsgJF8gPT4g MSB9IEB0ZXN0c1RvUnVuIH19OworCiAgICAgQHRlc3RzVG9SdW4gPSBzb3J0IHBhdGhjbXAgQHRl c3RzVG9SdW47CiAKICAgICAjIFJldmVyc2UgdGhlIHRlc3RzCg==