107905 2013-01-24 20:34:53 -0800 Quick fix for Chromium EWS bots running out of space due to a tmp file leak 2013-01-25 10:26:05 -0800 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified RESOLVED FIXED P2 Normal --- 1 alancutter alancutter abarth eric lforschler ossy timloh tony webkit.review.bot oldest_to_newest 816131 0 alancutter 2013-01-24 20:34:53 -0800 The gce-cr-linux-## bots have been hitting disk capacity recently due to directories being left behind by testing. The directories were of the form "/tmp/.org.chromium.Chromium.XXXXXX" with XXXXXX being random characters. This patch is to apply a quick fix to remove all temporary files after each bot cycle so queues don't get blocked again while the root problem is investigated. webkit-commit-queue.appspot.com/results/16118062 816142 1 184650 alancutter 2013-01-24 20:46:44 -0800 Created attachment 184650 Patch 816144 2 184650 eric 2013-01-24 20:51:10 -0800 Comment on attachment 184650 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=184650&action=review > Tools/EWSTools/start-queue.sh:47 > + # This clears any temporary file leaks after running tests. > + # Not the nicest solution but it will keep the queues running instead of > + # filling up all remaining disk space. > + find /tmp/* -delete This seems quite dangerous... I could imagine that other long-running process might want their temp files to not die every 2 hours... :) But we can also try this and iterate. 816145 3 184650 eric 2013-01-24 20:56:38 -0800 Comment on attachment 184650 Patch Is there and easy way for you to try this on the GCE queues w/o shipping this to everyone? It seems like a more dangerous change than I would like. 816146 4 eric 2013-01-24 20:57:41 -0800 Otherwise stated: I worry that non-GCE clients may be using our scripts and not expect them to eat their entire /tmp folder. mabye we should just clear /tmp/.org.chromium.Chromium* to start with, as that fixes this bug and is much safer. 816148 5 184650 eric 2013-01-24 20:58:12 -0800 Comment on attachment 184650 Patch Yeah, I've changed my mind. I think this is too dangerous. :) Lets start with something more surgical. 816208 6 184650 lforschler 2013-01-24 21:57:33 -0800 Comment on attachment 184650 Patch Mac EWS prunes logs older than 14 days, and rotates out the main log file every 10 iterations. See the start-queue-mac for what we used. I'm open to suggestions or alternatives. 816311 7 184681 alancutter 2013-01-24 23:44:55 -0800 Created attachment 184681 Patch 816316 8 alancutter 2013-01-24 23:57:43 -0800 (In reply to comment #7) > Created an attachment (id=184681) [details] > Patch Updated patch to only clear out the ".org.chromium.Chromium.*" files/directories found in /tmp. I agree the first version was too violent a solution. 816319 9 alancutter 2013-01-25 00:01:41 -0800 (In reply to comment #3) > (From update of attachment 184650 [details]) > Is there and easy way for you to try this on the GCE queues w/o shipping this to everyone? It seems like a more dangerous change than I would like. It should be noted that this change won't take effect on most if not all bots until it's manually implemented. The bot bash scripts are normally located outside of the repository which calls the python scripts inside the repo. 816321 10 184681 eric 2013-01-25 00:08:52 -0800 Comment on attachment 184681 Patch LGTM. 816323 11 eric 2013-01-25 00:09:37 -0800 I believe this makes this dangerous to run this script on your personal machine, btw. It will try and eat your chrome temp files. :) 816324 12 184681 webkit.review.bot 2013-01-25 00:10:06 -0800 Comment on attachment 184681 Patch Rejecting attachment 184681 from commit-queue. Failed to run "['/mnt/git/webkit-commit-queue/Tools/Scripts/webkit-patch', '--status-host=queues.webkit.org', '--bot-id=gce-cq-04', 'apply-attachment', '--no-update', '--non-interactive', 184681, '--port=chromium-xvfb']" exit_code: 2 cwd: /mnt/git/webkit-commit-queue Last 500 characters of output: Failed to run "[u'/mnt/git/webkit-commit-queue/Tools/Scripts/svn-apply', '--force', '--reviewer', 'Eric Seidel']" exit_code: 2 cwd: /mnt/git/webkit-commit-queue Parsed 2 diffs from patch file(s). patch: **** Can't create file /tmp/ppThWn7v : No space left on device patch: **** Can't create file /tmp/ppvL0Bkw : No space left on device Failed to run "[u'/mnt/git/webkit-commit-queue/Tools/Scripts/svn-apply', '--force', '--reviewer', 'Eric Seidel']" exit_code: 2 cwd: /mnt/git/webkit-commit-queue Full output: http://queues.webkit.org/results/16116203 816353 13 184692 alancutter 2013-01-25 00:54:29 -0800 Created attachment 184692 Patch 816354 14 alancutter 2013-01-25 00:56:00 -0800 (In reply to comment #12) > (From update of attachment 184681 [details]) > Rejecting attachment 184681 [details] from commit-queue. > > Failed to run "['/mnt/git/webkit-commit-queue/Tools/Scripts/webkit-patch', '--status-host=queues.webkit.org', '--bot-id=gce-cq-04', 'apply-attachment', '--no-update', '--non-interactive', 184681, '--port=chromium-xvfb']" exit_code: 2 cwd: /mnt/git/webkit-commit-queue > > Last 500 characters of output: > > Failed to run "[u'/mnt/git/webkit-commit-queue/Tools/Scripts/svn-apply', '--force', '--reviewer', 'Eric Seidel']" exit_code: 2 cwd: /mnt/git/webkit-commit-queue > > Parsed 2 diffs from patch file(s). > patch: **** Can't create file /tmp/ppThWn7v : No space left on device > patch: **** Can't create file /tmp/ppvL0Bkw : No space left on device > > Failed to run "[u'/mnt/git/webkit-commit-queue/Tools/Scripts/svn-apply', '--force', '--reviewer', 'Eric Seidel']" exit_code: 2 cwd: /mnt/git/webkit-commit-queue > > Full output: http://queues.webkit.org/results/16116203 (In reply to comment #13) > Created an attachment (id=184692) [details] > Patch Trying again. (: 816389 15 184692 webkit.review.bot 2013-01-25 01:57:49 -0800 Comment on attachment 184692 Patch Clearing flags on attachment: 184692 Committed r140801: <http://trac.webkit.org/changeset/140801> 816391 16 webkit.review.bot 2013-01-25 01:57:54 -0800 All reviewed patches have been landed. Closing bug. 816690 17 tony 2013-01-25 09:57:27 -0800 A few comments: - Deleting .org.chromium.Chromium.* files should be safe if you're running Chrome, because Chrome would use .com.google.Chrome.*. It might stomp on temp files used by a developer build of Chromium. - This is what the waterfall bots already do. They run Tools/BuildSlaveSupport/chromium/remove-crash-logs. There might be a code sharing opportunity here. - An additional option would be for NRWT to set the environment variable TMPDIR for DRT and blow away that whole directory after the tests are done. I've filed bug 107959 for this (it'll help keep developer machines clean too). 816721 18 eric 2013-01-25 10:26:05 -0800 (In reply to comment #17) > A few comments: > - An additional option would be for NRWT to set the environment variable TMPDIR for DRT and blow away that whole directory after the tests are done. I've filed bug 107959 for this (it'll help keep developer machines clean too). Yeah, I briefly considered this, but wasn't sure how well the TMPDIR environment variable would be support by all the code we care about. :) I'm sure some parts hard-code /tmp. If that works though, it sounds like the best solution. Then not only can we clean up after leaks like this, we can also monitor them! 184650 2013-01-24 20:46:44 -0800 2013-01-24 23:44:51 -0800 Patch bug-107905-20130125154334.patch text/plain 1441 alancutter U3VidmVyc2lvbiBSZXZpc2lvbjogMTQwNzczCmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggZWQwYWRjOGY2MWQwNTk5M2Q5ZjA4NWVkZTdlNjdiNmNi YzRmYTI5NC4uNWRhMjhlODA5NTlhNGI5MWNiNTI0M2IyMWM0YWU4NzlmNzgwM2ZjYiAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDE0 IEBACisyMDEzLTAxLTI0ICBBbGFuIEN1dHRlciAgPGFsYW5jdXR0ZXJAY2hyb21pdW0ub3JnPgor CisgICAgICAgIFF1aWNrIGZpeCBmb3IgQ2hyb21pdW0gRVdTIGJvdHMgcnVubmluZyBvdXQgb2Yg c3BhY2UgZHVlIHRvIGEgdG1wIGZpbGUgbGVhaworICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0 Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9MTA3OTA1CisKKyAgICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZ IChPT1BTISkuCisKKyAgICAgICAgQWRkZWQgbGluZSB0byBFV1MgbG9vcCB0byBjbGVhciAvdG1w LgorCisgICAgICAgICogRVdTVG9vbHMvc3RhcnQtcXVldWUuc2g6CisKIDIwMTMtMDEtMjQgIEpv c2h1YSBCZWxsICA8anNiZWxsQGNocm9taXVtLm9yZz4KIAogICAgICAgICBBZGQgd2F0Y2hsaXN0 IGZvciBJbmRleGVkREIgZGV2ZWxvcG1lbnQsIGFuZCBhZGQgbXlzZWxmIGFuZCBvdGhlcnMuCmRp ZmYgLS1naXQgYS9Ub29scy9FV1NUb29scy9zdGFydC1xdWV1ZS5zaCBiL1Rvb2xzL0VXU1Rvb2xz L3N0YXJ0LXF1ZXVlLnNoCmluZGV4IGVjZDkxNjJkOTBkYjUyMzlkYjBlYTg0NjA2ZWU0NGI4ZjI0 MzY5MDUuLmNlOGUzYzc2MDk0MTI4MmY2YTM1NDRlODY5OWRlOTM3OWE2NGZkNGQgMTAwNzU1Ci0t LSBhL1Rvb2xzL0VXU1Rvb2xzL3N0YXJ0LXF1ZXVlLnNoCisrKyBiL1Rvb2xzL0VXU1Rvb2xzL3N0 YXJ0LXF1ZXVlLnNoCkBAIC00MSw2ICs0MSwxMSBAQCBRVUVVRV9QQVJBTVM9IiRAIgogY2QgL21u dC9naXQvd2Via2l0LSRRVUVVRV9OQU1FCiB3aGlsZSA6CiBkbworICAgICMgVGhpcyBjbGVhcnMg YW55IHRlbXBvcmFyeSBmaWxlIGxlYWtzIGFmdGVyIHJ1bm5pbmcgdGVzdHMuCisgICAgIyBOb3Qg dGhlIG5pY2VzdCBzb2x1dGlvbiBidXQgaXQgd2lsbCBrZWVwIHRoZSBxdWV1ZXMgcnVubmluZyBp bnN0ZWFkIG9mCisgICAgIyBmaWxsaW5nIHVwIGFsbCByZW1haW5pbmcgZGlzayBzcGFjZS4KKyAg ICBmaW5kIC90bXAvKiAtZGVsZXRlCisKICAgICAjIFRoaXMgc29tZXdoYXQgcXVpcmt5IHNlcXVl bmNlIG9mIHN0ZXBzIHNlZW1zIHRvIGNsZWFyIHVwIGFsbCB0aGUgYnJva2VuCiAgICAgIyBnaXQg c2l0dWF0aW9ucyB3ZSd2ZSBnb3R0ZW4gb3Vyc2VsZiBpbnRvIGluIHRoZSBwYXN0LgogICAgIGdp dCBjbGVhbiAtZiAjIFJlbW92ZSBhbnkgbGVmdC1vdmVyIGxheW91dCB0ZXN0IHJlc3VsdHMsIGFk ZGVkIGZpbGVzLCBldGMuCg== 184681 2013-01-24 23:44:55 -0800 2013-01-25 00:54:25 -0800 Patch bug-107905-20130125184145.patch text/plain 1590 alancutter U3VidmVyc2lvbiBSZXZpc2lvbjogMTQwNzczCmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggZWQwYWRjOGY2MWQwNTk5M2Q5ZjA4NWVkZTdlNjdiNmNi YzRmYTI5NC4uYzE2MDk5NGE3NzRkYjE0Mzk5ODg4NTYxZTFkZGU5YjNiNDU0NTUzOSAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDE1 IEBACisyMDEzLTAxLTI0ICBBbGFuIEN1dHRlciAgPGFsYW5jdXR0ZXJAY2hyb21pdW0ub3JnPgor CisgICAgICAgIFF1aWNrIGZpeCBmb3IgQ2hyb21pdW0gRVdTIGJvdHMgcnVubmluZyBvdXQgb2Yg c3BhY2UgZHVlIHRvIGEgdG1wIGZpbGUgbGVhaworICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0 Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9MTA3OTA1CisKKyAgICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZ IChPT1BTISkuCisKKyAgICAgICAgQWRkZWQgbGluZSB0byBFV1MgbG9vcCB0byBjbGVhciAvdG1w IG9mIGtub3duIENocm9taXVtIHRlc3QgbGVhay4KKyAgICAgICAgRGlyZWN0b3JpZXMgbmFtZWQg Ii5vcmcuY2hyb21pdW0uQ2hyb21pdW0uWFhYWFhYIiBub3QgZ2V0dGluZyBjbGVhbmVkIHVwLgor CisgICAgICAgICogRVdTVG9vbHMvc3RhcnQtcXVldWUuc2g6CisKIDIwMTMtMDEtMjQgIEpvc2h1 YSBCZWxsICA8anNiZWxsQGNocm9taXVtLm9yZz4KIAogICAgICAgICBBZGQgd2F0Y2hsaXN0IGZv ciBJbmRleGVkREIgZGV2ZWxvcG1lbnQsIGFuZCBhZGQgbXlzZWxmIGFuZCBvdGhlcnMuCmRpZmYg LS1naXQgYS9Ub29scy9FV1NUb29scy9zdGFydC1xdWV1ZS5zaCBiL1Rvb2xzL0VXU1Rvb2xzL3N0 YXJ0LXF1ZXVlLnNoCmluZGV4IGVjZDkxNjJkOTBkYjUyMzlkYjBlYTg0NjA2ZWU0NGI4ZjI0MzY5 MDUuLjg4NWE0NjJkOWZkMzViZDg3YWY5YWUxZjdiZDQyNTNmZGQ5N2U5ZDEgMTAwNzU1Ci0tLSBh L1Rvb2xzL0VXU1Rvb2xzL3N0YXJ0LXF1ZXVlLnNoCisrKyBiL1Rvb2xzL0VXU1Rvb2xzL3N0YXJ0 LXF1ZXVlLnNoCkBAIC00MSw2ICs0MSwxMSBAQCBRVUVVRV9QQVJBTVM9IiRAIgogY2QgL21udC9n aXQvd2Via2l0LSRRVUVVRV9OQU1FCiB3aGlsZSA6CiBkbworICAgICMgVGhpcyBjbGVhcnMgYW55 IHRlbXBvcmFyeSBmaWxlIGxlYWtzIGFmdGVyIHJ1bm5pbmcgdGVzdHMuCisgICAgIyBOb3QgdGhl IG5pY2VzdCBzb2x1dGlvbiBidXQgaXQgd2lsbCBrZWVwIHRoZSBxdWV1ZXMgcnVubmluZyBpbnN0 ZWFkIG9mCisgICAgIyBmaWxsaW5nIHVwIGFsbCByZW1haW5pbmcgZGlzayBzcGFjZS4KKyAgICBm aW5kIC90bXAgLW5hbWUgIi5vcmcuY2hyb21pdW0uQ2hyb21pdW0uKiIgfCB4YXJncyBybSAtcmYK KwogICAgICMgVGhpcyBzb21ld2hhdCBxdWlya3kgc2VxdWVuY2Ugb2Ygc3RlcHMgc2VlbXMgdG8g Y2xlYXIgdXAgYWxsIHRoZSBicm9rZW4KICAgICAjIGdpdCBzaXR1YXRpb25zIHdlJ3ZlIGdvdHRl biBvdXJzZWxmIGludG8gaW4gdGhlIHBhc3QuCiAgICAgZ2l0IGNsZWFuIC1mICMgUmVtb3ZlIGFu eSBsZWZ0LW92ZXIgbGF5b3V0IHRlc3QgcmVzdWx0cywgYWRkZWQgZmlsZXMsIGV0Yy4K 184692 2013-01-25 00:54:29 -0800 2013-01-25 01:57:49 -0800 Patch bug-107905-20130125195119.patch text/plain 1739 alancutter U3VidmVyc2lvbiBSZXZpc2lvbjogMTQwNzczCmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggZWQwYWRjOGY2MWQwNTk5M2Q5ZjA4NWVkZTdlNjdiNmNi YzRmYTI5NC4uYzNjMzZkMmY5N2U0NmI0NDRkNWU3ZjNjOGRhMjFlNGY0YWZkYTEzNSAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDE1 IEBACisyMDEzLTAxLTI1ICBBbGFuIEN1dHRlciAgPGFsYW5jdXR0ZXJAY2hyb21pdW0ub3JnPgor CisgICAgICAgIFF1aWNrIGZpeCBmb3IgQ2hyb21pdW0gRVdTIGJvdHMgcnVubmluZyBvdXQgb2Yg c3BhY2UgZHVlIHRvIGEgdG1wIGZpbGUgbGVhaworICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0 Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9MTA3OTA1CisKKyAgICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZ IChPT1BTISkuCisKKyAgICAgICAgQWRkZWQgbGluZSB0byBFV1MgbG9vcCB0byBjbGVhciAvdG1w IG9mIGtub3duIENocm9taXVtIHRlc3QgbGVhay4KKyAgICAgICAgRGlyZWN0b3JpZXMgbmFtZWQg Ii5vcmcuY2hyb21pdW0uQ2hyb21pdW0uWFhYWFhYIiBub3QgZ2V0dGluZyBjbGVhbmVkIHVwLgor CisgICAgICAgICogRVdTVG9vbHMvc3RhcnQtcXVldWUuc2g6CisKIDIwMTMtMDEtMjQgIEpvc2h1 YSBCZWxsICA8anNiZWxsQGNocm9taXVtLm9yZz4KIAogICAgICAgICBBZGQgd2F0Y2hsaXN0IGZv ciBJbmRleGVkREIgZGV2ZWxvcG1lbnQsIGFuZCBhZGQgbXlzZWxmIGFuZCBvdGhlcnMuCmRpZmYg LS1naXQgYS9Ub29scy9FV1NUb29scy9zdGFydC1xdWV1ZS5zaCBiL1Rvb2xzL0VXU1Rvb2xzL3N0 YXJ0LXF1ZXVlLnNoCmluZGV4IGVjZDkxNjJkOTBkYjUyMzlkYjBlYTg0NjA2ZWU0NGI4ZjI0MzY5 MDUuLjQ4YWIzMmU1YWJiNDI2OTZkMjNkM2I2MDUzN2VlZGIxM2IxNGZkNzAgMTAwNzU1Ci0tLSBh L1Rvb2xzL0VXU1Rvb2xzL3N0YXJ0LXF1ZXVlLnNoCisrKyBiL1Rvb2xzL0VXU1Rvb2xzL3N0YXJ0 LXF1ZXVlLnNoCkBAIC00MSw2ICs0MSwxMyBAQCBRVUVVRV9QQVJBTVM9IiRAIgogY2QgL21udC9n aXQvd2Via2l0LSRRVUVVRV9OQU1FCiB3aGlsZSA6CiBkbworICAgICMgVGhpcyBjbGVhcnMgYW55 IHRlbXBvcmFyeSBmaWxlIGxlYWtzIGFmdGVyIHJ1bm5pbmcgdGVzdHMuCisgICAgIyBOb3QgdGhl IG5pY2VzdCBzb2x1dGlvbiBidXQgaXQgd2lsbCBrZWVwIHRoZSBxdWV1ZXMgcnVubmluZyBpbnN0 ZWFkIG9mCisgICAgIyBmaWxsaW5nIHVwIGFsbCByZW1haW5pbmcgZGlzayBzcGFjZS4KKyAgICAj IE5PVEU6IFRoaXMgd2lsbCBjbGVhciBhbnkgQ2hyb21pdW0gYnJvd3NlciB0ZW1wb3JhcnkgZmls ZXMgcnVubmluZyBvbiB0aGUKKyAgICAjIG1hY2hpbmUgYXMgd2VsbC4gTm90IGFkdmlzZWQgdG8g cnVuIHRoaXMgd2hpbGUgdXNpbmcgQ2hyb21lLgorICAgIGZpbmQgL3RtcCAtbmFtZSAiLm9yZy5j aHJvbWl1bS5DaHJvbWl1bS4qIiB8IHhhcmdzIHJtIC1yZgorCiAgICAgIyBUaGlzIHNvbWV3aGF0 IHF1aXJreSBzZXF1ZW5jZSBvZiBzdGVwcyBzZWVtcyB0byBjbGVhciB1cCBhbGwgdGhlIGJyb2tl bgogICAgICMgZ2l0IHNpdHVhdGlvbnMgd2UndmUgZ290dGVuIG91cnNlbGYgaW50byBpbiB0aGUg cGFzdC4KICAgICBnaXQgY2xlYW4gLWYgIyBSZW1vdmUgYW55IGxlZnQtb3ZlciBsYXlvdXQgdGVz dCByZXN1bHRzLCBhZGRlZCBmaWxlcywgZXRjLgo=