Bug 174502 – [GTK][WPE] stress/dont-reserve-huge-capacity-lexer.js JSC test failing

• Status: NEW
• Component: New Bugs

[Zan Dobersek]

The stress/dont-reserve-huge-capacity-lexer.js JSC test, and the various variations of it, is failing on the bots.

The problem isn't in the test itself.  The failures occur when multiple instances of this test (in different variations) are run in parallel.  Because the test on its own consumes a lot of memory, multiple instances consume much more in total, faulting the latter-spawned tests.

The result is a variable amount of failures on specific JSC test runs:
https://build.webkit.org/builders/WPE%20Linux%2064-bit%20Release%20%28Tests%29/builds/1464/steps/jscore-test/logs/stdio
https://build.webkit.org/builders/WPE%20Linux%2064-bit%20Release%20%28Tests%29/builds/1465/steps/jscore-test/logs/stdio

The test itself is already designated as memory-wasteful, and is not run when passing the --memory-limited flag.  But that's not the case on the bots, and these systems obviously don't have enough RAM available to accommodate all the variations of this test that might be run in parallel.

[Zan Dobersek]

One option is to run the tests on the GTK+ and WPE bots with --memory-limited by default.

[Yusuke Suzuki]

(In reply to Zan Dobersek from comment #1)
> One option is to run the tests on the GTK+ and WPE bots with
> --memory-limited by default.

Until we implement the functionality to execute some tests in a non-parallel manner, passing --memory-limited in GTK/WPE bot is good workaround.

[Carlos Alberto Lopez Perez]

(In reply to Yusuke Suzuki from comment #2)
> (In reply to Zan Dobersek from comment #1)
> > One option is to run the tests on the GTK+ and WPE bots with
> > --memory-limited by default.
> 
> Until we implement the functionality to execute some tests in a non-parallel
> manner, passing --memory-limited in GTK/WPE bot is good workaround.

I prefer to skip the specific tests that use insane amounts of RAM ( see bug 171630 ) on WPE and GTK than just skip all tests that can use more RAM than the available on an embedded system (I understand that this is what memory limited means, correct me if I'm wrong).

Typical GTK/WPE bots have at least 16GB of RAM available and run 8 tests in parallel. If a test uses [2-4]GB of RAM is fine. If it uses more can be a problem.

[Zan Dobersek]

(In reply to Carlos Alberto Lopez Perez from comment #3)
> (In reply to Yusuke Suzuki from comment #2)
> > (In reply to Zan Dobersek from comment #1)
> > > One option is to run the tests on the GTK+ and WPE bots with
> > > --memory-limited by default.
> > 
> > Until we implement the functionality to execute some tests in a non-parallel
> > manner, passing --memory-limited in GTK/WPE bot is good workaround.
> 
> I prefer to skip the specific tests that use insane amounts of RAM ( see bug
> 171630 ) on WPE and GTK than just skip all tests that can use more RAM than
> the available on an embedded system (I understand that this is what memory
> limited means, correct me if I'm wrong).
> 

The test in 171630 is a layout test and is easily skippable. This is a file under JSTests/ and can't be as easily skipped.

In total there are 11 tests that are skipped on x86_64 in case the script is passed --memory-limited, so there's not a lot of test coverage that's lost if --memory-limited is used by default on the bots.

> Typical GTK/WPE bots have at least 16GB of RAM available and run 8 tests in
> parallel. If a test uses [2-4]GB of RAM is fine. If it uses more can be a
> problem.

Well obviously 16GB is not enough when a bunch of dont-reserve-huge-capacity-lexer.js variations are run together, so as it stands this might as well be considered a memory-limited configuration.

[Carlos Alberto Lopez Perez]

(In reply to Zan Dobersek from comment #4)
> > Typical GTK/WPE bots have at least 16GB of RAM available and run 8 tests in
> > parallel. If a test uses [2-4]GB of RAM is fine. If it uses more can be a
> > problem.
> 
> Well obviously 16GB is not enough when a bunch of
> dont-reserve-huge-capacity-lexer.js variations are run together, so as it
> stands this might as well be considered a memory-limited configuration.


How much is enough then? 

Mac bots also have 16GB of RAM and they run on Intel quad-core CPUs with hyper-treading (therefore 4*2 = 8 HW threads => 8 parallel process). And they are not running into issues with the memory usage of the JSC tests.

From https://build.webkit.org/buildslaves/bot157 I get that the Mac JSC bot has 16GB of RAM and is a Macmini6,2 and from https://en.wikipedia.org/wiki/Mac_Mini#Specifications_2 I translate that model name to an i7-3615QM CPU and http://ark.intel.com/products/64900/Intel-Core-i7-3615QM-Processor-6M-Cache-up-to-3_30-GHz confirms it has 8 threads.


To me this seems to indicate there is a problem somewhere on how JSC handles the memory on Linux ports. I think passing --memory-limit to our bots will only hide this problem instead of fixing it.

[Carlos Alberto Lopez Perez]

I tried running this test alone both on Mac and Linux/GTK and in both cases it yields similar memory usage numbers: it peaks around 6GB of RSS.

So I have currently no explanation about why this issue only is triggering on the Linux bots when the number of cores/threads and amount of RAM is identical than the Mac ones.


Just trying to guess... maybe the memory pressure handler have some effect on the JSC tests? Any other idea?

[Carlos Alberto Lopez Perez]

(In reply to Carlos Alberto Lopez Perez from comment #6)
> I tried running this test alone both on Mac and Linux/GTK and in both cases
> it yields similar memory usage numbers: it peaks around 6GB of RSS.
> 
> So I have currently no explanation about why this issue only is triggering
> on the Linux bots when the number of cores/threads and amount of RAM is
> identical than the Mac ones.
> 
> 
> Just trying to guess... maybe the memory pressure handler have some effect
> on the JSC tests? Any other idea?

I give up.(In reply to Carlos Alberto Lopez Perez from comment #6)
> I tried running this test alone both on Mac and Linux/GTK and in both cases
> it yields similar memory usage numbers: it peaks around 6GB of RSS.
> 
> So I have currently no explanation about why this issue only is triggering
> on the Linux bots when the number of cores/threads and amount of RAM is
> identical than the Mac ones.
> 
> 
> Just trying to guess... maybe the memory pressure handler have some effect
> on the JSC tests? Any other idea?

I give up after spending several hours trying to understand and/or fix this with different workarounds on the bots by tuning different ulimit/sysctl/cgroup values.

So, let's disable memorylimited on the GTK+ and WPE bots for now as workaround.

Doing that on bug 175140