101469 2012-11-07 07:00:05 -0800 REGRESSION(r133380): new tests without expected file reported as failing tests on the bots 2012-12-13 09:22:15 -0800 1 1 1 Unclassified WebKit New Bugs 420+ Unspecified Unspecified RESOLVED FIXED NRWT P2 Normal --- 100915 1 ossy dpranke dpranke eric lforschler ojan ossy rniwa tmpsantos oldest_to_newest 760660 0 ossy 2012-11-07 07:00:05 -0800 http://trac.webkit.org/changeset/133380 changed the output from "no expected results found" to "missing results", but _parseNewRunWebKitTestsOutput() in master.cfg still uses the old format: http://trac.webkit.org/browser/trunk/Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg?rev=133299#L365 It is similar breakage to https://bugs.webkit.org/show_bug.cgi?id=93537 I think the best way to fix it is moving the output parsing to NRWT and dumping only one, easily parsable, never changing line for master.cfg. 761012 1 dpranke 2012-11-07 13:25:20 -0800 Argh, I checked the parsing code for the chromium bots but forgot that the webkit.org bots might be using a different parsing algorithm :(. I don't think it's reasonable to expect the output to be "never changing", but it makes sense to adopt something easier to parse, that's for sure. I've often thought the thing to do is to echo a subset of results.json (or even all of it, maybe) to stdout and just parse that. We should also add actual tests that check the format matches what we expect somewhere, and add comments about what needs to be checked if those tests start to fail. I have at least one fixme to do this in the printing_unittest.py, I think. 785675 2 ossy 2012-12-06 23:32:46 -0800 Are you going to fix this annoying regression? 785907 3 dpranke 2012-12-07 08:56:16 -0800 Oh, I'm sorry, I thought this had been immediately fixed. It must've fallen off my radar somehow. Yes, I'll look into fixing this today. As to the longer-term fix, I've just landed a long series of patches that get us much closer to the point of producing easily parsable output. I'll probably look into doing that next week. 787895 4 dpranke 2012-12-10 17:47:29 -0800 Okay, I thought this had been fixed ... were the changes in bug 93537 and bug 93789 not sufficient? What's still broken? 788282 5 ossy 2012-12-11 03:44:15 -0800 (In reply to comment #4) > Okay, I thought this had been fixed ... were the changes in bug 93537 and bug 93789 not sufficient? What's still broken? No, it isn't fixed. It is a new regression caused by r133380. Previously a new test was reported as: "Regressions: Unexpected no expected results found : (1)", and "Regressions: Unexpected missing results (1)" after r133380. Previously a new test without expected result wasn't calculated as failure and wasn't make the bot red if there was only new tests and no other regressions. The bug is somewhere in _parseNewRunWebKitTestsOutput and/or evaluateCommand. 788286 6 ossy 2012-12-11 03:46:29 -0800 Here is an output where you can test these parser functions - http://build.webkit.sed.hu/builders/x86-32%20Linux%20Qt%20Release%20NRWT/builds/25127/steps/layout-test/logs/stdio (It is an artificial example, because I removed the expected file on the bot manually to produce the buggy output for you to help debugging.) 788782 7 178871 dpranke 2012-12-11 14:13:27 -0800 Created attachment 178871 Patch 788785 8 178871 rniwa 2012-12-11 14:15:48 -0800 Comment on attachment 178871 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=178871&action=review > Tools/BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:82 > +Regressions: Unexpected missing results (1) > + svg/custom/zero-path-square-cap-rendering2.svg [ Missing ] We probably shouldn't report missing results as "Regressions" if tests just had been added. 788786 9 178871 eric 2012-12-11 14:16:05 -0800 Comment on attachment 178871 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=178871&action=review > Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg:371 > - ('flakes', re.compile(r'[Uu]nexpected flakiness.+:?\s*\((\d+)\)')), > + ('flakes', re.compile(r'Unexpected flakiness.+\((\d+)\)')), > ('new passes', re.compile(r'Expected to .+, but passed:\s+\((\d+)\)')), > - ('missing results', re.compile(r'(?:no expected results found|missing results)\s*:\s+\((\d+)\)')), > - ('failures', re.compile(r'Regressions: [Uu]nexpected.+:?\s*\((\d+)\)')), > + ('missing results', re.compile(r'Regressions: Unexpected missing results\s+\((\d+)\)')), > + ('failures', re.compile(r'Regressions: Unexpected.+\((\d+)\)')), Is this removing ORWT support? 788789 10 178871 rniwa 2012-12-11 14:17:30 -0800 Comment on attachment 178871 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=178871&action=review >> Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg:371 >> + ('failures', re.compile(r'Regressions: Unexpected.+\((\d+)\)')), > > Is this removing ORWT support? No, this function is specific to NRWT. 788791 11 dpranke 2012-12-11 14:19:19 -0800 (In reply to comment #8) > (From update of attachment 178871 [details]) > View in context: https://bugs.webkit.org/attachment.cgi?id=178871&action=review > > > Tools/BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:82 > > +Regressions: Unexpected missing results (1) > > + svg/custom/zero-path-square-cap-rendering2.svg [ Missing ] > > We probably shouldn't report missing results as "Regressions" if tests just had been added. Unfortunately, there's no way to tell if the tests had just been added, but this change is actually at least breaking out the missing category from the other failures. 789090 12 dpranke 2012-12-11 18:23:23 -0800 Committed r137404: <http://trac.webkit.org/changeset/137404> 789692 13 dpranke 2012-12-12 10:42:00 -0800 The bots are running w/ this change and seem happy so far, but I haven't seen any "missing" results yet, so I'm not 100% positive it worked. Ossy, can you help me keep on the lookout for this? 790519 14 tmpsantos 2012-12-13 05:09:39 -0800 Hi Dirk, I stepped over this failure here and I'm suspected that it might be caused by this patch. Can you confirm? $ ./mastercfg_unittest.py /usr/lib/python2.7/dist-packages/twisted/spread/jelly.py:102: DeprecationWarning: the sets module is deprecated import sets as _sets ./mastercfg_unittest.py:819: DeprecationWarning: Passing a BuildStep subclass to factory.addStep is deprecated. Please pass a BuildStep instance instead. Support will be dropped in v0.8.7. ...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................F............. ====================================================================== FAIL: test_nrwt_leaks_parsing (__main__.MasterCfgTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "./mastercfg_unittest.py", line 64, in test_nrwt_leaks_parsing self.fail() AssertionError: None ---------------------------------------------------------------------- Ran 529 tests in 0.031s FAILED (failures=1) 790671 15 dpranke 2012-12-13 09:22:15 -0800 (In reply to comment #14) > Hi Dirk, > > I stepped over this failure here and I'm suspected that it might be caused by this patch. Can you confirm? > > $ ./mastercfg_unittest.py > /usr/lib/python2.7/dist-packages/twisted/spread/jelly.py:102: DeprecationWarning: the sets module is deprecated > import sets as _sets > ./mastercfg_unittest.py:819: DeprecationWarning: Passing a BuildStep subclass to factory.addStep is deprecated. Please pass a BuildStep instance instead. Support will be dropped in v0.8.7. > ...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................F............. > ====================================================================== > FAIL: test_nrwt_leaks_parsing (__main__.MasterCfgTest) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "./mastercfg_unittest.py", line 64, in test_nrwt_leaks_parsing > self.fail() > AssertionError: None > > ---------------------------------------------------------------------- > Ran 529 tests in 0.031s > > FAILED (failures=1) Whoops :). Yeah, my mistake. Will fix shortly. 178871 2012-12-11 14:13:27 -0800 2012-12-11 14:17:30 -0800 Patch bug-101469-20121211141048.patch text/plain 4581 dpranke U3VidmVyc2lvbiBSZXZpc2lvbjogMTM3MzU2CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggOTY5NDA4ZjVkYzgzN2M0YmM2N2Y3MmRlZjhlMzE4ZjRh ZThmMTBkNS4uYWQ0ODNjYTU0MGE0ZmUwODkyOTBmZjlkMWRlMzIzZDBhYzQ3YjZhYyAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSw1ICsxLDIx IEBACiAyMDEyLTEyLTExICBEaXJrIFByYW5rZSAgPGRwcmFua2VAY2hyb21pdW0ub3JnPgogCisg ICAgICAgIFJFR1JFU1NJT04ocjEzMzM4MCk6IG5ldyB0ZXN0cyB3aXRob3V0IGV4cGVjdGVkIGZp bGUgcmVwb3J0ZWQgYXMgZmFpbGluZyB0ZXN0cyBvbiB0aGUgYm90cworICAgICAgICBodHRwczov L2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9MTAxNDY5CisKKyAgICAgICAgUmV2aWV3 ZWQgYnkgTk9CT0RZIChPT1BTISkuCisKKyAgICAgICAgRml4IHBhcnNpbmcgb2YgTlJXVCdzIG91 dHB1dCB0byBpZGVudGlmeSBtaXNzaW5nIHJlc3VsdHMgc2VwYXJhdGVseQorICAgICAgICBmcm9t IG90aGVyIHJlZ3Jlc3Npb25zLgorCisgICAgICAgICogQnVpbGRTbGF2ZVN1cHBvcnQvYnVpbGQu d2Via2l0Lm9yZy1jb25maWcvbWFzdGVyLmNmZzoKKyAgICAgICAgKFJ1bldlYktpdFRlc3RzLl9w YXJzZU5ld1J1bldlYktpdFRlc3RzT3V0cHV0KToKKyAgICAgICAgKiBCdWlsZFNsYXZlU3VwcG9y dC9idWlsZC53ZWJraXQub3JnLWNvbmZpZy9tYXN0ZXJjZmdfdW5pdHRlc3QucHk6CisgICAgICAg IChNYXN0ZXJDZmdUZXN0LnRlc3RfbnJ3dF9sZWFrc19wYXJzaW5nKToKKyAgICAgICAgKHRlc3Rf bnJ3dF9taXNzaW5nX3Jlc3VsdHMpOgorCisyMDEyLTEyLTExICBEaXJrIFByYW5rZSAgPGRwcmFu a2VAY2hyb21pdW0ub3JnPgorCiAgICAgICAgIG5yd3Q6IHJlY29yZCB0aGUgb3JkZXIgYW5kIHJ1 biB0aW1lcyBvZiBhbGwgb2YgdGhlIHRlc3RzIGluIGEgc2luZ2xlIGZpbGUKICAgICAgICAgaHR0 cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTEwNDE4NAogCmRpZmYgLS1naXQg YS9Ub29scy9CdWlsZFNsYXZlU3VwcG9ydC9idWlsZC53ZWJraXQub3JnLWNvbmZpZy9tYXN0ZXIu Y2ZnIGIvVG9vbHMvQnVpbGRTbGF2ZVN1cHBvcnQvYnVpbGQud2Via2l0Lm9yZy1jb25maWcvbWFz dGVyLmNmZwppbmRleCA2ODdjMTk3ZGIxYWQ3YjVmNjQ1YWYyMzc2MWRlNWZhNTM5OGNkNTQ5Li44 NDBkZWMxMDk0OGVmMzVkM2QyYWRlNTVjNjgwYjdlYjE0Mjg4NmFkIDEwMDY0NAotLS0gYS9Ub29s cy9CdWlsZFNsYXZlU3VwcG9ydC9idWlsZC53ZWJraXQub3JnLWNvbmZpZy9tYXN0ZXIuY2ZnCisr KyBiL1Rvb2xzL0J1aWxkU2xhdmVTdXBwb3J0L2J1aWxkLndlYmtpdC5vcmctY29uZmlnL21hc3Rl ci5jZmcKQEAgLTM2NSwxMCArMzY1LDEwIEBAIGNsYXNzIFJ1bldlYktpdFRlc3RzKHNoZWxsLlRl c3QpOgogICAgIGRlZiBfcGFyc2VOZXdSdW5XZWJLaXRUZXN0c091dHB1dChzZWxmLCBsb2dUZXh0 KToKICAgICAgICAgaW5jb3JyZWN0TGF5b3V0TGluZXMgPSBbXQogICAgICAgICBleHByZXNzaW9u cyA9IFsKLSAgICAgICAgICAgICgnZmxha2VzJywgcmUuY29tcGlsZShyJ1tVdV1uZXhwZWN0ZWQg Zmxha2luZXNzLis6P1xzKlwoKFxkKylcKScpKSwKKyAgICAgICAgICAgICgnZmxha2VzJywgcmUu Y29tcGlsZShyJ1VuZXhwZWN0ZWQgZmxha2luZXNzLitcKChcZCspXCknKSksCiAgICAgICAgICAg ICAoJ25ldyBwYXNzZXMnLCByZS5jb21waWxlKHInRXhwZWN0ZWQgdG8gLissIGJ1dCBwYXNzZWQ6 XHMrXCgoXGQrKVwpJykpLAotICAgICAgICAgICAgKCdtaXNzaW5nIHJlc3VsdHMnLCByZS5jb21w aWxlKHInKD86bm8gZXhwZWN0ZWQgcmVzdWx0cyBmb3VuZHxtaXNzaW5nIHJlc3VsdHMpXHMqOlxz K1woKFxkKylcKScpKSwKLSAgICAgICAgICAgICgnZmFpbHVyZXMnLCByZS5jb21waWxlKHInUmVn cmVzc2lvbnM6IFtVdV1uZXhwZWN0ZWQuKzo/XHMqXCgoXGQrKVwpJykpLAorICAgICAgICAgICAg KCdtaXNzaW5nIHJlc3VsdHMnLCByZS5jb21waWxlKHInUmVncmVzc2lvbnM6IFVuZXhwZWN0ZWQg bWlzc2luZyByZXN1bHRzXHMrXCgoXGQrKVwpJykpLAorICAgICAgICAgICAgKCdmYWlsdXJlcycs IHJlLmNvbXBpbGUocidSZWdyZXNzaW9uczogVW5leHBlY3RlZC4rXCgoXGQrKVwpJykpLAogICAg ICAgICBdCiAgICAgICAgIHRlc3RGYWlsdXJlcyA9IHt9CiAKZGlmZiAtLWdpdCBhL1Rvb2xzL0J1 aWxkU2xhdmVTdXBwb3J0L2J1aWxkLndlYmtpdC5vcmctY29uZmlnL21hc3RlcmNmZ191bml0dGVz dC5weSBiL1Rvb2xzL0J1aWxkU2xhdmVTdXBwb3J0L2J1aWxkLndlYmtpdC5vcmctY29uZmlnL21h c3RlcmNmZ191bml0dGVzdC5weQppbmRleCBhZTIxZTIzN2VhMDgxNWQzZjUyNWNlOGVkZDliZmE2 Yjg5NWNkYzI3Li4yNDhjODExNjUxMzE0NDQ1YWU2ZDAwYmE0ZjE3MGY2MjM4YTg4MDEyIDEwMDc1 NQotLS0gYS9Ub29scy9CdWlsZFNsYXZlU3VwcG9ydC9idWlsZC53ZWJraXQub3JnLWNvbmZpZy9t YXN0ZXJjZmdfdW5pdHRlc3QucHkKKysrIGIvVG9vbHMvQnVpbGRTbGF2ZVN1cHBvcnQvYnVpbGQu d2Via2l0Lm9yZy1jb25maWcvbWFzdGVyY2ZnX3VuaXR0ZXN0LnB5CkBAIC01MSw3ICs1MSw3IEBA IGNsYXNzIEJ1aWxkQm90Q29uZmlnTG9hZGVyKG9iamVjdCk6CiAKIGNsYXNzIE1hc3RlckNmZ1Rl c3QodW5pdHRlc3QuVGVzdENhc2UpOgogICAgIGRlZiB0ZXN0X25yd3RfbGVha3NfcGFyc2luZyhz ZWxmKToKLSAgICAgICAgcnVuX3dlYmtpdF90ZXN0cyA9IFJ1bldlYktpdFRlc3RzKCkKKyAgICAg ICAgcnVuX3dlYmtpdF90ZXN0cyA9IFJ1bldlYktpdFRlc3RzKCkgICMgcHlsaW50IGlzIGNvbmZ1 c2VkIGJ5IHRoZSB3YXkgd2UgaW1wb3J0IHRoZSBtb2R1bGUgLi4uIHB5bGludDogZGlzYWJsZS1t c2c9RTA2MDIKICAgICAgICAgbG9nX3RleHQgPSAiIiIKIDIwMTEtMDgtMDkgMTA6MDU6MTgsNTgw IDI5NDg2IG1hYy5weToyNzUgSU5GTyBsZWFrcyBmb3VuZCBmb3IgYSB0b3RhbCBvZiAxOTcsOTM2 IGJ5dGVzIQogMjAxMS0wOC0wOSAxMDowNToxOCw1ODAgMjk0ODYgbWFjLnB5OjI3NiBJTkZPIDEg dW5pcXVlIGxlYWtzIGZvdW5kIQpAQCAtNjEsOCArNjEsMzIgQEAgY2xhc3MgTWFzdGVyQ2ZnVGVz dCh1bml0dGVzdC5UZXN0Q2FzZSk6CiAgICAgICAgICAgICAnMSB1bmlxdWUgbGVha3MgZm91bmQh JywKICAgICAgICAgXQogICAgICAgICBydW5fd2Via2l0X3Rlc3RzLl9wYXJzZU5ld1J1bldlYktp dFRlc3RzT3V0cHV0KGxvZ190ZXh0KQorICAgICAgICBzZWxmLmZhaWwoKQogICAgICAgICBzZWxm LmFzc2VydEVxdWFsKHJ1bl93ZWJraXRfdGVzdHMuaW5jb3JyZWN0TGF5b3V0TGluZXMsIGV4cGVj dGVkX2luY29ycmVjdF9saW5lcykKIAorICAgIGRlZiB0ZXN0X25yd3RfbWlzc2luZ19yZXN1bHRz KHNlbGYpOgorICAgICAgICBydW5fd2Via2l0X3Rlc3RzID0gUnVuV2ViS2l0VGVzdHMoKSAgIyBw eWxpbnQgaXMgY29uZnVzZWQgYnkgdGhlIHdheSB3ZSBpbXBvcnQgdGhlIG1vZHVsZSAuLi4gcHls aW50OiBkaXNhYmxlLW1zZz1FMDYwMgorICAgICAgICBsb2dfdGV4dCA9ICIiIgorRXhwZWN0ZWQg dG8gZmFpbCwgYnV0IHBhc3NlZDogKDIpCisgIGFuaW1hdGlvbnMvYWRkaXRpdmUtdHJhbnNmb3Jt LWFuaW1hdGlvbnMuaHRtbAorICBhbmltYXRpb25zL2Nyb3NzLWZhZGUtd2Via2l0LW1hc2stYm94 LWltYWdlLmh0bWwKKworVW5leHBlY3RlZCBmbGFraW5lc3M6IHRleHQtb25seSBmYWlsdXJlcyAo MikKKyAgZmFzdC9ldmVudHMvdG91Y2gvdG91Y2gtaW5zaWRlLWlmcmFtZS5odG1sIFsgRmFpbHVy ZSBQYXNzIF0KKyAgaHR0cC90ZXN0cy9pbnNwZWN0b3ItZW5hYmxlZC9jb25zb2xlLWNsZWFyLWFy Z3VtZW50cy1vbi1mcmFtZS1uYXZpZ2F0aW9uLmh0bWwgWyBGYWlsdXJlIFBhc3MgXQorCitVbmV4 cGVjdGVkIGZsYWtpbmVzczogdGltZW91dHMgKDEpCisgIHN2Zy90ZXh0L2ZvcmVpZ25PYmplY3Qt cmVwYWludC54bWwgWyBUaW1lb3V0IFBhc3MgXQorCitSZWdyZXNzaW9uczogVW5leHBlY3RlZCBt aXNzaW5nIHJlc3VsdHMgKDEpCisgIHN2Zy9jdXN0b20vemVyby1wYXRoLXNxdWFyZS1jYXAtcmVu ZGVyaW5nMi5zdmcgWyBNaXNzaW5nIF0KKworUmVncmVzc2lvbnM6IFVuZXhwZWN0ZWQgdGV4dC1v bmx5IGZhaWx1cmVzICgxKQorICBzdmcvY3VzdG9tL3plcm8tcGF0aC1zcXVhcmUtY2FwLXJlbmRl cmluZzIuc3ZnIFsgRmFpbHVyZSBdCisiIiIKKyAgICAgICAgcnVuX3dlYmtpdF90ZXN0cy5fcGFy c2VOZXdSdW5XZWJLaXRUZXN0c091dHB1dChsb2dfdGV4dCkKKyAgICAgICAgc2VsZi5hc3NlcnRF cXVhbChzZXQocnVuX3dlYmtpdF90ZXN0cy5pbmNvcnJlY3RMYXlvdXRMaW5lcyksCisgICAgICAg ICAgICBzZXQoWycyIG5ldyBwYXNzZXMnLCAnMyBmbGFrZXMnLCAnMSBtaXNzaW5nIHJlc3VsdHMn LCAnMSBmYWlsdXJlcyddKSkKIAogY2xhc3MgU3R1YlN0ZGlvKG9iamVjdCk6CiAgICAgZGVmIF9f aW5pdF9fKHNlbGYsIHN0ZGlvKToK