Bug 112978

Summary: [chromium] Linux ASAN WebKit canary is running out of memory
Product: WebKit Reporter: Dirk Pranke <dpranke>
Component: Tools / TestsAssignee: Dirk Pranke <dpranke>
Status: RESOLVED FIXED    
Severity: Normal CC: abarth, antonm, eric, inferno, phajdan.jr, thakis, thestig, tony, webkit.review.bot
Priority: P2 Keywords: NRWT
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Patch none

Description Dirk Pranke 2013-03-21 16:32:46 PDT
turns out that running the ASAN processing on the stderr takes an enormous amount of memory, and if you run multiple of these at a time, you can run out of memory on the bot (looks like we can manage two on the bot now, but three probably causes us to run out of memory).

See, e.g.: http://build.chromium.org/p/chromium.webkit/builders/WebKit%20Linux%20ASAN/builds/5269/steps/webkit_tests/logs/stdio

It'd not obvious that there's a great answer here, but for the moment I'm going to try serializing the asan_filter script so we only run one at a time ...
Comment 1 Dirk Pranke 2013-03-21 16:44:03 PDT
Created attachment 194383 [details]
Patch
Comment 2 Tony Chang 2013-03-21 16:46:46 PDT
Are you trying to lock across python processes?  Why don't we just prepend flock to the commands you want to serialize?
Comment 3 Dirk Pranke 2013-03-21 16:49:13 PDT
(In reply to comment #2)
> Are you trying to lock across python processes?  Why don't we just prepend flock to the commands you want to serialize?

Well, this is portable to windows (not that we run asan on windows) and happened to reuse the well-tested stuff we have lying around :). I can change it to just use flock if you'd prefer.
Comment 4 Dirk Pranke 2013-03-21 16:49:52 PDT
but, yes, we're trying to lock across processes.
Comment 5 Dirk Pranke 2013-03-21 16:50:48 PDT
oh, one minor additional difference is that http_lock code ensures access is granted in order (FIFO). AFAIK, regular flock doesn't do that?
Comment 6 Dirk Pranke 2013-03-21 16:51:03 PDT
that may also not really matter in this case.
Comment 7 Tony Chang 2013-03-21 17:02:38 PDT
Comment on attachment 194383 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=194383&action=review

I think it's OK to use HttpLock, but it's a bit confusing (due to naming) and I have more confidence in flock.

> Tools/Scripts/webkitpy/layout_tests/port/chromium.py:383
> +                    stderr = self._executive.run_command(['c++filt'], input=output, decode_output=False)

Can we run c++filt outside the lock?  That's not expensive, is it?

Should there be a unit test for this?
Comment 8 Tony Chang 2013-03-21 17:03:22 PDT
Comment on attachment 194383 [details]
Patch

Feel free to try this, I won't be around later tonight for further reviews.
Comment 9 Dirk Pranke 2013-03-21 17:07:42 PDT
Comment on attachment 194383 [details]
Patch

thanks!
Comment 10 WebKit Review Bot 2013-03-21 17:27:36 PDT
Comment on attachment 194383 [details]
Patch

Clearing flags on attachment: 194383

Committed r146546: <http://trac.webkit.org/changeset/146546>
Comment 11 WebKit Review Bot 2013-03-21 17:27:40 PDT
All reviewed patches have been landed.  Closing bug.