turns out that running the ASAN processing on the stderr takes an enormous amount of memory, and if you run multiple of these at a time, you can run out of memory on the bot (looks like we can manage two on the bot now, but three probably causes us to run out of memory). See, e.g.: http://build.chromium.org/p/chromium.webkit/builders/WebKit%20Linux%20ASAN/builds/5269/steps/webkit_tests/logs/stdio It'd not obvious that there's a great answer here, but for the moment I'm going to try serializing the asan_filter script so we only run one at a time ...
Created attachment 194383 [details] Patch
Are you trying to lock across python processes? Why don't we just prepend flock to the commands you want to serialize?
(In reply to comment #2) > Are you trying to lock across python processes? Why don't we just prepend flock to the commands you want to serialize? Well, this is portable to windows (not that we run asan on windows) and happened to reuse the well-tested stuff we have lying around :). I can change it to just use flock if you'd prefer.
but, yes, we're trying to lock across processes.
oh, one minor additional difference is that http_lock code ensures access is granted in order (FIFO). AFAIK, regular flock doesn't do that?
that may also not really matter in this case.
Comment on attachment 194383 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=194383&action=review I think it's OK to use HttpLock, but it's a bit confusing (due to naming) and I have more confidence in flock. > Tools/Scripts/webkitpy/layout_tests/port/chromium.py:383 > + stderr = self._executive.run_command(['c++filt'], input=output, decode_output=False) Can we run c++filt outside the lock? That's not expensive, is it? Should there be a unit test for this?
Comment on attachment 194383 [details] Patch Feel free to try this, I won't be around later tonight for further reviews.
Comment on attachment 194383 [details] Patch thanks!
Comment on attachment 194383 [details] Patch Clearing flags on attachment: 194383 Committed r146546: <http://trac.webkit.org/changeset/146546>
All reviewed patches have been landed. Closing bug.