93434 2012-08-07 22:17:27 -0700 [NRWT] REGRESSION(r124967): New tests without expected results handled as failures 2012-08-07 23:27:16 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) All All RESOLVED FIXED NRWT P2 Normal --- 88702 1 ossy ossy abarth dpranke ojan ossy rniwa webkit.review.bot oldest_to_newest 689879 0 ossy 2012-08-07 22:17:27 -0700 Before r124967 new tests without expected files were handled as new tests, not as failures. Which means only orange buildbot, not red buildbot. But after r124967, new tests make the bot red, which is not the expected behaviour. Let's see the NRWT's output: before r124967: ---------------- Unexpected flakiness: text diff mismatch (4) fast/forms/range/slider-mouse-events.html = TEXT PASS fast/forms/range/slider-onchange-event.html = TEXT PASS http/tests/security/contentSecurityPolicy/object-src-url-allowed.html = TEXT PASS http/tests/security/cross-frame-access-put.html = TEXT PASS Regressions: Unexpected no expected results found : (2) fast/table/table-row-outline-paint.html = MISSING svg/as-image/animated-svg-repaints-completely-in-hidpi.html = MISSING and the buildbot reports: 4 flakes 2 missing results after r124967: --------------- Unexpected flakiness: text failures (4) fast/forms/range/slider-mouse-events.html = TEXT PASS fast/forms/range/slider-onchange-event.html = TEXT PASS http/tests/security/contentSecurityPolicy/object-src-url-allowed.html = TEXT PASS http/tests/security/cross-frame-access-put.html = TEXT PASS Regressions: Unexpected missing results : (2) fast/table/table-row-outline-paint.html = MISSING svg/as-image/animated-svg-repaints-completely-in-hidpi.html = MISSING and the buildbot reports: 2 failures 4 flakes It seems it is a similar bug as https://bugs.webkit.org/show_bug.cgi?id=93346 689881 1 ossy 2012-08-07 22:20:11 -0700 Yes, it is so similar bug ... The new "Regressions: Unexpected missing results" match for the regexp of failures .. Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg ----------------------------------------------------------- ... def _parseNewRunWebKitTestsOutput(self, logText): incorrectLayoutLines = [] expressions = [ ('flakes', re.compile(r'Unexpected flakiness.+:?\s*\((\d+)\)')), ('new passes', re.compile(r'Expected to .+, but passed:\s+\((\d+)\)')), ('missing results', re.compile(r'no expected results found\s*:\s+\((\d+)\)')), ('failures', re.compile(r'Regressions: Unexpected.+:?\s*\((\d+)\)')), ] testFailures = {} ... 689893 2 157115 ossy 2012-08-07 22:29:05 -0700 Created attachment 157115 Patch 689897 3 ossy 2012-08-07 22:31:46 -0700 If we really want to change the output of NRWT, please make it sync with master.cfg changes. And please contact with Lucas and/or Bill to restart the master after the change to ensure correct behaviour. And please ping me, because I have to make similar changes in master.cfg of build.webkit.sed.hu. (Maybe you have to make similar changes in master.cfg of build.chromium.org too) 689903 4 157115 ossy 2012-08-07 22:35:00 -0700 Comment on attachment 157115 Patch Clearing flags on attachment: 157115 Committed r124988: <http://trac.webkit.org/changeset/124988> 689904 5 ossy 2012-08-07 22:35:07 -0700 All reviewed patches have been landed. Closing bug. 689912 6 157115 dpranke 2012-08-07 22:40:49 -0700 Comment on attachment 157115 Patch Sorry! (again). 689914 7 dpranke 2012-08-07 22:42:20 -0700 (In reply to comment #6) > (From update of attachment 157115 [details]) > Sorry! (again). To be clear(er), it was not actually my intention to change the output that the bot would be seeing :(. I did not mean to break the parser (or require changes to the parser). 689962 8 ossy 2012-08-07 23:27:16 -0700 not problem ;) I didn't notice that the order of two messages changed (printed for each test, printed at the end), so my fix wasn't correct. I fixed it - https://trac.webkit.org/changeset/124994 157115 2012-08-07 22:29:05 -0700 2012-08-07 22:40:49 -0700 Patch bug-93434-20120808072839.patch text/plain 1830 ossy U3VidmVyc2lvbiBSZXZpc2lvbjogMTI0OTg3CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggM2YzOGJkYTM0YzcwNGQyZDI3MDg3NjYxYzE2ZTgyODRh NDFiZWRiOC4uNzA2NmI3NmY4NWJkOTQ4NGE4MzZlNTY1M2Q4MGRlNWQwNDg2NDRjYyAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSw1ICsxLDE3 IEBACiAyMDEyLTA4LTA3ICBDc2FiYSBPc3p0cm9nb27DoWMgIDxvc3N5QHdlYmtpdC5vcmc+CiAK KyAgICAgICAgW05SV1RdIFJFR1JFU1NJT04ocjEyNDk2Nyk6IE5ldyB0ZXN0cyB3aXRob3V0IGV4 cGVjdGVkIHJlc3VsdHMgaGFuZGxlZCBhcyBmYWlsdXJlcworICAgICAgICBodHRwczovL2J1Z3Mu d2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9OTM0MzQKKworICAgICAgICBSZXZpZXdlZCBieSBO T0JPRFkgKE9PUFMhKS4KKworICAgICAgICBDaGFuZ2UgYmFjayAibWlzc2luZyByZXN1bHRzIiB0 byAibm8gZXhwZWN0ZWQgcmVzdWx0IGZvdW5kIiwgYmVjYXVzZSBtYXN0ZXIuY2ZnJ3MgcmVzdWx0 cyBwYXJzZXIgZXhwZWN0cyBpdC4KKworICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0 X3Rlc3RzL21vZGVscy90ZXN0X2V4cGVjdGF0aW9ucy5weToKKyAgICAgICAgKFRlc3RFeHBlY3Rh dGlvbnMpOgorCisyMDEyLTA4LTA3ICBDc2FiYSBPc3p0cm9nb27DoWMgIDxvc3N5QHdlYmtpdC5v cmc+CisKICAgICAgICAgYnVpbGRib3Q6IFJlbW92ZSB0aGUgY29uZmlndXJhdGlvbiBvZiBoZnJl eXRoZXIqCiAgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD05 MTg0OQogCmRpZmYgLS1naXQgYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9t b2RlbHMvdGVzdF9leHBlY3RhdGlvbnMucHkgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91 dF90ZXN0cy9tb2RlbHMvdGVzdF9leHBlY3RhdGlvbnMucHkKaW5kZXggOTY1MTUxODYzYjdjOTFl ZjAxNzc5NTY1M2RkMWNmNThjNGI5MWUyNC4uODA3NGUxYTE0YTJiNTVjYjQ3YjU3MzRiY2Y5NjQ5 MGYxMjBhNGVjOSAxMDA2NDQKLS0tIGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVz dHMvbW9kZWxzL3Rlc3RfZXhwZWN0YXRpb25zLnB5CisrKyBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0 cHkvbGF5b3V0X3Rlc3RzL21vZGVscy90ZXN0X2V4cGVjdGF0aW9ucy5weQpAQCAtNjk5LDcgKzY5 OSw3IEBAIGNsYXNzIFRlc3RFeHBlY3RhdGlvbnMob2JqZWN0KToKICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgQVVESU86ICgnYXVkaW8gZmFpbHVyZXMnLCAnZmFpbGVkJywgJyAoYXVk aW8gZGlmZiknKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgQ1JBU0g6ICgnY3Jh c2hlcycsICdjcmFzaGVkJywgJycpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBU SU1FT1VUOiAoJ3RpbWVvdXRzJywgJ3RpbWVkIG91dCcsICcnKSwKLSAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgTUlTU0lORzogKCdtaXNzaW5nIHJlc3VsdHMnLCAnaXMgbWlzc2luZyBh biBleHBlY3RlZCByZXN1bHQnLCAnJyl9CisgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IE1JU1NJTkc6ICgnbm8gZXhwZWN0ZWQgcmVzdWx0IGZvdW5kJywgJ25vIGV4cGVjdGVkIHJlc3Vs dHMgZm91bmQnLCAnJyl9CiAKICAgICBFWFBFQ1RBVElPTl9PUkRFUiA9IChQQVNTLCBDUkFTSCwg VElNRU9VVCwgTUlTU0lORywgSU1BR0VfUExVU19URVhULCBURVhULCBJTUFHRSwgQVVESU8sIFNL SVApCiAK