Bug 42756 - JIT requires VM overcommit (particularly on x86-64), Linux does not by default support this without swap?
Summary: JIT requires VM overcommit (particularly on x86-64), Linux does not by defaul...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: JavaScriptCore (show other bugs)
Version: 528+ (Nightly build)
Hardware: PC Linux
: P2 Normal
Assignee: Xan Lopez
URL:
Keywords:
: 51511 53684 (view as bug list)
Depends on:
Blocks:
 
Reported: 2010-07-21 08:14 PDT by donaldcallen
Modified: 2013-07-16 09:48 PDT (History)
16 users (show)

See Also:


Attachments
gdb backtrace (5.31 KB, text/plain)
2010-07-21 08:14 PDT, donaldcallen
no flags Details
Valgrind output (3.74 KB, text/plain)
2010-07-21 08:17 PDT, donaldcallen
no flags Details
Segfault when running jsc without any argument (debug build) (1.99 KB, text/plain)
2010-08-20 07:46 PDT, Adrien Nader
no flags Details
Segfault when running jsc without any argument (debug build) (1.28 KB, text/plain)
2010-11-14 05:05 PST, ojab
no flags Details
overcommit.diff (6.36 KB, patch)
2011-01-12 14:26 PST, Xan Lopez
no flags Details | Formatted Diff | Diff
overcommit.diff (6.50 KB, patch)
2011-01-12 17:04 PST, Xan Lopez
no flags Details | Formatted Diff | Diff
Patch (2.00 KB, patch)
2011-05-20 08:12 PDT, Xan Lopez
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description donaldcallen 2010-07-21 08:14:36 PDT
Created attachment 62181 [details]
gdb backtrace

I have been experiencing segfaults in memcpy using recent nightly builds. This problem occurs on only one of my 4 computers running Slackware 13.1. The machines are a Lenovo S10 workstation (4-core Q6600 processor, 4 Gb), a mini itx machine with Intel Atom D510 motherboard and 2 Gb, a Thinkpad X61 (Core 2 Duo, 2 Gb) and a Toshiba netbook (single core Atom N450, 2 Gb). The segfault I am reporting occurs only on the mini itx machine. I download the nightly build, plus the webkit slackbuild from slackbuild.org, edit the webkit.Slackbuild script to reflect the proper version number of the nightly build, and build the Slackbuild on the fast Lenovo workstation. If I install the resulting package on the workstation (or the netbook or the Thinkpad) and build uzbl or surf against it, either one will run. Do the same thing on the mini itx machine, and either one will segfault on startup. I tried building webkit on the mini itx machine (which takes a couple of hours on a machine that slow), installed it and built uzbl against that, and it segfaulted. I then copied *that* slackbuild to the workstation, installed it, built and installed uzbl against that version, and it worked on the workstation. The same is true of the version built on the workstation (interestingly, the two slackware packages are not identical, despite the two machines having identical package-sets) -- uzbl built against that works everywhere but the mini itx machine.

If you are suspecting the hardware, all I can tell you is that everything else works correctly on the mini itx machine, including google chrome. There are absolutely no signs of sick hardware, and that machine is my primary machine -- I use it every day.

I built a version of webkit (on the workstation) with CFLAGS changed from -O2 to -g and the slackbuild script modified not to strip the symbols from the executable. I then modified the uzbl-browser script to run uzbl-core under gdb. I am attaching a backtrace of the crash.

I also modified the uzbl-browser script to run uzbl-core under valgrind. The output of that session is also attached.
Comment 1 donaldcallen 2010-07-21 08:17:14 PDT
Created attachment 62182 [details]
Valgrind output
Comment 2 Alexey Proskuryakov 2010-07-21 16:50:47 PDT
#0  0x00007ffff37f2e51 in memcpy () from /lib64/libc.so.6
#1  0x00007ffff7649dba in JSC::JIT::privateCompileCTIMachineTrampolines(WTF::RefPtr<JSC::ExecutablePool>*, JSC::JSGlobalData*, JSC::TrampolineStructure*) () from /usr/lib64/libwebkitgtk-1.0.so.0
#2  0x00007ffff7666400 in JSC::JIT::compileCTIMachineTrampolines(JSC::JSGlobalData*, WTF::RefPtr<JSC::ExecutablePool>*, JSC::TrampolineStructure*) () from /usr/lib64/libwebkitgtk-1.0.so.0
#3  0x00007ffff76ccdb3 in JSC::JSGlobalData::JSGlobalData(JSC::JSGlobalData::GlobalDataType, JSC::ThreadStackType) () from /usr/lib64/libwebkitgtk-1.0.so.0
#4  0x00007ffff76ccf13 in JSC::JSGlobalData::create(JSC::ThreadStackType) () from /usr/lib64/libwebkitgtk-1.0.so.0
#5  0x00007ffff76cd832 in JSC::JSGlobalData::createLeaked(JSC::ThreadStackType) () from /usr/lib64/libwebkitgtk-1.0.so.0
#6  0x00007ffff6b8e5f2 in WebCore::JSDOMWindowBase::commonJSGlobalData() () from /usr/lib64/libwebkitgtk-1.0.so.0
Comment 3 Adrien Nader 2010-08-20 03:21:42 PDT
I am experiencing the same problem on my computer with webkit-gtk 1.3.3.

GtkLauncher fails and simply creating a web_view and using load_uri (no matter if the target exists or not, local or remote) makes it crash.

  g_thread_init(NULL); g_type_init(); gtk_init(&argc, &argv);
  WebKitWebView* wv = WEBKIT_WEB_VIEW(webkit_web_view_new ());
  webkit_web_view_load_uri(wv, "http://google.com");

If I don't load any page, it doesn't crash.


I'm also using slackware 13.1, custom-compiled kernel (2.6.35+, meaning git). gcc 4.4.4, glib2 2.22.5, glibc 2.11.1, gtk+ 2.18.9, libsoup 2.30.2, icu4c 4.4.1, webkit-gtk 1.3.3. Webkit was compiled without any non-default option (cflags=cxxflags="-O2 -fPIC").

I'm compiling on a quad-core (phenom ii x4 955) which has the same configuration (except the kernel) and running an a laptop (core2duo t5450). The quad-core is currently headless.

I'm going to try several builds with different options (first, --disable-jit) and building on my laptop (it's only 5 times slower...) and will report back.
Comment 4 Adrien Nader 2010-08-20 05:49:50 PDT
Here are the results after a few builds.

First, building on the laptop machine (the one running it) did *NOT* solve the problem.
Then, of course, --disable-jit solved it.
Finally, --disable-optimizations didn't solve it.

Also, the 'jsc-1' binary, ran without any argument, crashes immediately. As I said, the build machine is headless so I can't really test webkit-gtk on it. However, trying jsc-1 on the build machine works without problem.
(actually, I tried to create a web_view and load_uri a page, without creating a windows nor displaying anything and I got the expected "cannot open display" warning but no segfault)

The backtrace for jsc-1 on the laptop is:
#0  0x00007ffff50a4e51 in memcpy () from /lib64/libc.so.6
#1  0x00000000004d988f in JSC::JIT::privateCompileCTIMachineTrampolines(WTF::RefPtr<JSC::ExecutablePool>*, JSC::JSGlobalData*, JSC::TrampolineStructure*) ()
#2  0x00000000004f6570 in JSC::JIT::compileCTIMachineTrampolines(JSC::JSGlobalData*, WTF::RefPtr<JSC::ExecutablePool>*, JSC::TrampolineStructure*) ()
#3  0x000000000043f29c in JSC::JSGlobalData::JSGlobalData(JSC::JSGlobalData::GlobalDataType, JSC::ThreadStackType) ()
#4  0x000000000043f663 in JSC::JSGlobalData::create(JSC::ThreadStackType) ()
#5  0x0000000000408a77 in main ()

I'm also going to see if it can be reproduced in qemu, and if not, I can probably give a shell access if someone wants to try to debug and can't reproduce (unfortunately, not before a few days).
Comment 5 Zoltan Herczeg 2010-08-20 06:55:38 PDT
Is it happens in debug as well? Would be a good thing to see some line numbers. I expect missing frames on the stack since many funcions are inlined. (privateCompileCTIMachineTrampolines is the first function which generate machine code, perhaps it cannot allocate executable space)

Address 0x27c is not stack'd, malloc'd or (recently) free'd
This address is way too low...
Comment 6 Adrien Nader 2010-08-20 07:46:30 PDT
Created attachment 64955 [details]
Segfault when running jsc without any argument (debug build)
Comment 7 Zoltan Herczeg 2010-08-20 09:07:11 PDT
(In reply to comment #6)
> Created an attachment (id=64955) [details]
> Segfault when running jsc without any argument (debug build)

Hehe, as I suspected. Something with the allocator. Could you check this call successful on your machine (some printf is enough, mem.base() and mem.size() is important):

311	inline PassRefPtr<ExecutablePool> ExecutablePool::create(size_t n)
312	    {
313	        Allocation mem = systemAlloc(roundUpAllocationSize(n, JIT_ALLOCATOR_PAGE_SIZE));
314	        if (!mem)
315	            return 0;
316	        return adoptRef(new ExecutablePool(mem));
317	    }

CC'ing Gavin, he is the allocator expert.
Comment 8 Adrien Nader 2010-08-20 13:16:10 PDT
In order to try Zoltan's suggestion, I updated to HEAD (from webkit-gtk 1.3.3, which is not a very old release) and the problem evolved: backtrace changed, both happens with and without JIT, but it still only happens on the laptop and not the quad-core.
Another interesting point: I added 5GB swap on the laptop (from the 2GB of RAM) and jsc is now working properly. :-) 

Program received signal SIGSEGV, Segmentation fault.
0x00000000004e7010 in FixedVMPoolAllocator (this=0x9409c0, commonSize=16384, totalHeapSize=2147483648)
    at JavaScriptCore/jit/ExecutableAllocatorFixedVMPool.cpp:303

(gdb) bt
#0  0x00000000004e7010 in FixedVMPoolAllocator (this=0x9409c0, commonSize=16384, totalHeapSize=2147483648)
    at JavaScriptCore/jit/ExecutableAllocatorFixedVMPool.cpp:303
#1  0x00000000004e5de6 in JSC::ExecutableAllocator::isValid (this=0x93fa18) at JavaScriptCore/jit/ExecutableAllocatorFixedVMPool.cpp:442
#2  0x0000000000472684 in JSC::ExecutableAllocator::ExecutableAllocator() ()
#3  0x00000000004714dd in JSGlobalData (this=0x93e240, globalDataType=JSC::JSGlobalData::Default, threadStackType=JSC::ThreadStackTypeLarge)
    at JavaScriptCore/runtime/JSGlobalData.cpp:150
#4  0x000000000047205c in JSC::JSGlobalData::create (type=JSC::ThreadStackTypeLarge) at JavaScriptCore/runtime/JSGlobalData.cpp:231
#5  0x000000000040d47d in main (argc=1, argv=0x7fffffffdde8) at JavaScriptCore/jsc.cpp:346
Comment 9 Maciej Piechotka 2010-08-26 06:56:10 PDT
I reproduced it without swap on Core 2 Duo (64 bit system) [4 GiB of memory + 0 GiB of swap].

==4456== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==4456==  Access not within mapped region at address 0x27C
==4456==    at 0x4C2876D: memcpy (mc_replace_strmem.c:482)
==4456==    by 0x6C60525: JSC::JIT::privateCompileCTIMachineTrampolines(WTF::RefPtr<JSC::ExecutablePool>*, JSC::JSGlobalData*, JSC::TrampolineStructure*) (AssemblerBuffer.h:138)
==4456==    by 0x6C8650F: JSC::JIT::compileCTIMachineTrampolines(JSC::JSGlobalData*, WTF::RefPtr<JSC::ExecutablePool>*, JSC::TrampolineStructure*) (JIT.h:223)
==4456==    by 0x6D008E4: JSC::JSGlobalData::JSGlobalData(JSC::JSGlobalData::GlobalDataType, JSC::ThreadStackType) (JSGlobalData.cpp:152)
==4456==    by 0x6D00EC2: JSC::JSGlobalData::create(JSC::ThreadStackType) (JSGlobalData.cpp:225)
==4456==    by 0x6D00F01: JSC::JSGlobalData::createLeaked(JSC::ThreadStackType) (JSGlobalData.cpp:231)
==4456==    by 0x6464641: WebCore::JSDOMWindowBase::commonJSGlobalData() (JSDOMWindowBase.cpp:160)
==4456==    by 0x64A906B: WebCore::ScriptController::getAllWorlds(WTF::Vector<WebCore::DOMWrapperWorld*, 0ul>&) (ScriptController.cpp:187)
==4456==    by 0x678F271: WebCore::FrameLoader::dispatchDidClearWindowObjectsInAllWorlds() (FrameLoader.cpp:3382)
==4456==    by 0x678F306: WebCore::FrameLoader::receivedFirstData() (FrameLoader.cpp:668)
==4456==    by 0x678459F: WebCore::DocumentWriter::setEncoding(WebCore::String const&, bool) (DocumentWriter.cpp:236)
==4456==    by 0x6B7DCA0: WebKit::FrameLoaderClient::committedLoad(WebCore::DocumentLoader*, char const*, int) (FrameLoaderClientGtk.cpp:152)

With swap on everything is OK [4 GiB of memory + 4 GiB of swap]. Swap is not uses and in total 2.2 GiB is used.

Linux localhost 2.6.34-zen1-static #1 ZEN SMP PREEMPT Tue Aug 24 20:09:30 CEST 2010 x86_64 Intel(R) Core(TM)2 Duo CPU T9600 @ 2.80GHz GenuineIntel GNU/Linux
Comment 10 ojab 2010-11-14 05:02:34 PST
Looks like the same crash also happens for me on Athlon X2 II (Linux x86_64, glibc-2.11.2, no swap), only with JIT.
Comment 11 ojab 2010-11-14 05:05:41 PST
Created attachment 73848 [details]
Segfault when running jsc without any argument (debug build)

WebKit-trunk, git-svn-id: http://svn.webkit.org/repository/webkit/trunk@71981 268f45cc-cd09-0410-ab3c-d52691b4dbfc
Comment 12 ojab 2010-11-19 11:54:04 PST
    git-svn-id: http://svn.webkit.org/repository/webkit/trunk@72416 268f45cc-cd09-0410-ab3c-d52691b4dbfc
Still crashes.
Comment 13 Oliver Hunt 2010-11-19 12:13:51 PST
Is anyone able to repro this crash with their own build?

If so can you see if VM_POOL_ASLR is disabled?

If it is disabled try reducing the size of VM_POOL_SIZE in ExecutableAllocatorFixedVMPool.cpp to 1gig, and then 512Mb and see if that fixes things?

I'm trying to determine exactly what part of this call is failing.

Thanks.
Comment 14 ojab 2010-11-20 13:32:45 PST
OS isn't DARWIN, so I assume that VM_POOL_ASLR is disabled (don't know how to check).
VM_POOL_SIZE = 1Gb/512Mb fixes the issue, jit build doesn't crashes.
Comment 15 Oliver Hunt 2010-11-20 20:32:33 PST
(In reply to comment #14)
> OS isn't DARWIN, so I assume that VM_POOL_ASLR is disabled (don't know how to check).
> VM_POOL_SIZE = 1Gb/512Mb fixes the issue, jit build doesn't crashes.

Could you do me a favour, and on your platform see whether an allocation (via mmap) of 2GB succeeds in a trivial test program?  Maybe there's an upper limit to the amount of memory your platform allows in a single alloc?
Comment 16 ojab 2010-11-21 12:57:49 PST
Unfortunately I don't know C and cannot write testcase to allocate 2Gb via mmap .__. Testcase with malloc() works without crashes.
I have 2Gb ram total and if memory will be used (not only allocated) — program will be killed by OOM killer. WebKit with jit crashes without OOM killer participation.
Comment 17 ojab 2010-11-25 23:58:27 PST
Any additional info needed?
Comment 18 Zoltan Herczeg 2010-11-26 00:12:34 PST
(In reply to comment #16)
> Unfortunately I don't know C and cannot write testcase to allocate 2Gb via mmap .__. Testcase with malloc() works without crashes.

mmap_test.c:

#include <stdio.h>
#include <sys/mman.h>

int main() {
    printf("%p\n", mmap(0, 2ul * 1024ul * 1024ul * 1024ul,
        PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0));
    return 0;
}

Compiling: gcc mmap_test.c -o mmap_test
Comment 19 ojab 2010-11-26 01:15:39 PST
$ ./mmap 
0xffffffffffffffff

works fine.
Comment 20 Zoltan Herczeg 2010-11-26 01:31:35 PST
(In reply to comment #19)
> $ ./mmap 
> 0xffffffffffffffff
> 
> works fine.

0xffffffffffffffff (is -1) is MAP_FAILED (you can check it by a printf MAP_FAILED)

Thus Oliver were right, and you are not allowed to alloc 2G. Could you try it with smaller sizes? (1G, 512M, etc.) until you get something back other than MAP_FAILED ?
Comment 21 ojab 2010-11-26 01:58:06 PST
ook..

(1024ul * 1024ul * 1024ul + 1024ul * 1024ul * 256ul ) in the `while true; do ./mmap && sleep 1; done` loop gives me output like:
0x7f7f21422000
0x7fe0ffac7000
0xffffffffffffffff
0x7f2f98c9a000
0x7f76b0a50000
0xffffffffffffffff
0x7f6444a30000
0x7f2da6a14000
0xffffffffffffffff

With 1G I have normal addressess, but if I'll run memory-hungry task  — I'll see the same sometimes-0xffffffffffffffff output.

Looks like my system can only allocate amount of memory that is free at the moment.
Comment 22 ojab 2011-01-05 20:09:31 PST
    git-svn-id: http://svn.webkit.org/repository/webkit/trunk@75066 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Still the same crash.
Comment 23 Gavin Barraclough 2011-01-06 01:10:36 PST
JSC really needs the OS to provide a mechanism to be able to reserve virtual memory address ranges (in wtf/PageReservation.h).  As well as being used to allocate JIT code buffers, this is also used in the collector.  To support WebKit on Linux without swap, we really need a way for this to work.

PageReservation presently works on HAVE_MMAP platforms (including Linux) by simply mmapping a (potentially very large) region of memory, on the expectation that physical pages will be allocated on demand to the VM space.  If the OS does not allow memory to be overcommitted this isn't going to work.

I imagine the problem for someone to solve here is probably to find a way for the Linux kernel to reserve VM address ranges in a threadsafe fashion.
Comment 24 Gavin Barraclough 2011-01-12 11:56:54 PST
Interesting comment in redhat's bug tracker: https://bugzilla.redhat.com/show_bug.cgi?id=648319

    Pete Zaitcev 2010-12-22 20:03:51 EST
        BTW, an easy workaround if you don't want to use my RPM from comment #31 is:
            echo 1 > /proc/sys/vm/overcommit_memory
        Well, or just buy a couple gigs of RAM.


A more graceful failure for WebKit might be to read '/proc/sys/vm/overcommit_memory' on startup, and if this is not enabled then exit with an error message indicating that this is required.
Comment 25 Gavin Barraclough 2011-01-12 11:57:38 PST
*** Bug 51511 has been marked as a duplicate of this bug. ***
Comment 26 Geoffrey Garen 2011-01-12 13:32:47 PST
Wow -- so, by default, Linux will not commit VM on demand?
Comment 27 Adrien Nader 2011-01-12 14:05:36 PST
I haven't been able to provide a lot of feedback, mostly because of exams but what really leaves me wondering is that the JIT tries to get 4GB of memory. 4GB.

It might not use it but 4GB is simply too much.

What's actually the point? Do we really want that to happen? And why ask for that much if it's not going to be used?

IIRC that also happens without even using javascript. Why does the JIT allocate a huge amount of memory without even requiring any memory actually?
Comment 28 Oliver Hunt 2011-01-12 14:09:31 PST
(In reply to comment #27)
> I haven't been able to provide a lot of feedback, mostly because of exams but what really leaves me wondering is that the JIT tries to get 4GB of memory. 4GB.
> 
> It might not use it but 4GB is simply too much.
> 
> What's actually the point? Do we really want that to happen? And why ask for that much if it's not going to be used?
> 
> IIRC that also happens without even using javascript. Why does the JIT allocate a huge amount of memory without even requiring any memory actually?

It should only be requesting 2gig, and it's expecting the OS to merely reserve address space -- it's not expecting physical memory to be committed.
Comment 29 Adrien Nader 2011-01-12 14:16:36 PST
> It should only be requesting 2gig, and it's expecting the OS to merely reserve address space -- it's not expecting physical memory to be committed.

2GB is still a lot.

And if it counts on the OS not commiting it, then it counts on the memory not being used. Why ask for it then?


Quoting the linux kernel documentation for memory overcommit,
> When this flag is 1, the kernel pretends there is always enough
> memory until it actually runs out.

I doubt that will ever become the default.
It also leaves out a number of OSes that don't use overcommit (iirc, Solaris doesn't for instance).

Why isn't it possible to request 64MB instead? What's the technical reason for asking that much?
Comment 30 Oliver Hunt 2011-01-12 14:25:08 PST
(In reply to comment #29)
> > It should only be requesting 2gig, and it's expecting the OS to merely reserve address space -- it's not expecting physical memory to be committed.
> 
> 2GB is still a lot.
> 
> And if it counts on the OS not commiting it, then it counts on the memory not being used. Why ask for it then?
> 
> 
> Quoting the linux kernel documentation for memory overcommit,
> > When this flag is 1, the kernel pretends there is always enough
> > memory until it actually runs out.
> 
> I doubt that will ever become the default.
> It also leaves out a number of OSes that don't use overcommit (iirc, Solaris doesn't for instance).
> 
> Why isn't it possible to request 64MB instead? What's the technical reason for asking that much?

We allocate 2gig up front and do all allocations of executable memory from that region as we need to be able to guarantee that all executable allocations are within 2gig of each other - it would be a large amount of work (and a significant increase in code size) to support the arbitrary jumps to anywhere in the address space that would otherwise be necessary.

Surely Linux must have some mechanism for reserving an arbitrarily large amount of address space, and then commit that lazily?
Comment 31 Xan Lopez 2011-01-12 14:26:26 PST
Created attachment 78732 [details]
overcommit.diff

Patch for this issue.
Comment 32 Oliver Hunt 2011-01-12 14:52:27 PST
Comment on attachment 78732 [details]
overcommit.diff

i think 32mb will be far too little to be useful -- a simple test would be for you to just try using midori with a jsc build with vmPoolSizeGeneric == 32mb

Also i think the logic to determine what size the allocation should be would be better in a separate function.
Comment 33 Xan Lopez 2011-01-12 15:02:11 PST
(In reply to comment #32)
> (From update of attachment 78732 [details])
> i think 32mb will be far too little to be useful -- a simple test would be for you to just try using midori with a jsc build with vmPoolSizeGeneric == 32mb

I did try running a browser with those values for some minutes and things didn't explode right away, but you are right that they are fairly conservative. Do you want to make them bigger in general or do you want a triplet like generic/conservative desktop/conservative embedded. If the answer is the latter,  how will we know which one to fallback in linux when overcommit is not on?

> 
> Also i think the logic to determine what size the allocation should be would be better in a separate function.
Comment 34 Gavin Barraclough 2011-01-12 15:03:41 PST
Comment on attachment 78732 [details]
overcommit.diff

Looks great xan!
Comment 35 Gavin Barraclough 2011-01-12 15:07:20 PST
(In reply to comment #29)
> > It should only be requesting 2gig, and it's expecting the OS to merely reserve address space -- it's not expecting physical memory to be committed.
> 
> 2GB is still a lot.

For 64-bit OSes that allow more flexible control over the VM space , 2GB is less than 1% of 1% of available VM.  This is not a lot. ;-)

For OSes that insist on early commit it clearly doesn't seem sensible to make such a large request.

(In reply to comment #30)
> Surely Linux must have some mechanism for reserving an arbitrarily large amount of address space, and then commit that lazily?

This would be an improvement over the current fix, if we can find a way to do so.
Comment 36 Gavin Barraclough 2011-01-12 15:08:58 PST
Ooops, my r+ crossed with Oliver's comments,

(In reply to comment #32)
> (From update of attachment 78732 [details])
> i think 32mb will be far too little to be useful -- a simple test would be for you to just try using midori with a jsc build with vmPoolSizeGeneric == 32mb

Not that r74454 should help prevent this from causing problems.

> Also i think the logic to determine what size the allocation should be would be better in a separate function.

^ olliej's right, you should probably do this.
Comment 37 Xan Lopez 2011-01-12 17:04:17 PST
Created attachment 78761 [details]
overcommit.diff

New patch moving the code that messes with the values to a function. No change in the values because:

- We are still discussing that.
- It probably even makes sense to separate these two changes.
- Even if only this goes in, it's already an improvement in general for many people in 64bit Linux.
Comment 38 Gavin Barraclough 2011-01-12 18:44:21 PST
Comment on attachment 78761 [details]
overcommit.diff

Looks great.
Comment 39 Xan Lopez 2011-01-13 04:03:21 PST
Comment on attachment 78761 [details]
overcommit.diff

Landed in http://trac.webkit.org/changeset/75709

I'll open a new bug in case we want to improve the VM values for !overcommit in Linux.
Comment 40 Xan Lopez 2011-01-13 04:03:41 PST
Closing.
Comment 41 Gavin Barraclough 2011-01-28 17:09:20 PST
See changes in https://bugs.webkit.org/show_bug.cgi?id=53352
Comment 42 Gavin Barraclough 2011-01-31 11:29:52 PST
Reopening due to bug #53352.

The upshot of this bug is, the fix landed here wouldn't work anyway (without overcommit you'll allocate the smaller sized buffer, then crash after a short period of browsing once the JIT buffer became nicely fragmented).

Please see comment in bug #53352, options are to not support this configuration, always use the reduced size (and break some stress tests), or reimplement a dynamic/static switch based on whether overcommit is available.

If we need a switching solution, one option might be to always use 1GB tables, to just change the amount we allocate, and then to add a method to the AllocateTableDirectory to mark ranges of the table as unavailable.
Comment 43 Eric Seidel (no email) 2011-01-31 16:07:42 PST
Comment on attachment 78732 [details]
overcommit.diff

Cleared Gavin Barraclough's review+ from obsolete attachment 78732 [details] so that this bug does not appear in http://webkit.org/pending-commit.
Comment 44 Martin Robinson 2011-02-03 12:27:21 PST
*** Bug 53684 has been marked as a duplicate of this bug. ***
Comment 45 Xan Lopez 2011-05-20 07:53:24 PDT
Hey, good news:

turns out Linux has a flag for mmap that is perfect for this, MAP_NORESERVE. From man proc(5):


       /proc/sys/vm/overcommit_memory
              This file contains the kernel virtual memory accounting mode.  Values are:
              
                     0: heuristic overcommit (this is the default)
                     1: always overcommit, never check
                     2: always check, never overcommit
                  
              In  mode  0, calls of mmap(2) with MAP_NORESERVE are not checked, and the default check is very weak, leading to the risk
              of getting a process "OOM-killed".  Under Linux 2.4 any nonzero value implies mode 1.  In mode 2 (available  since  Linux
              2.6),  the  total  virtual address space on the system is limited to (SS + RAM*(r/100)), where SS is the size of the swap
              space, and RAM is the size of the physical memory, and r is the contents of the file /proc/sys/vm/overcommit_ratio.

--

0 is the default on Linux, so I think if we just add a one-line patch in the code adding this flag by default things should work OK. I'll test this and report back.
Comment 46 Xan Lopez 2011-05-20 08:12:55 PDT
Created attachment 94221 [details]
Patch
Comment 47 Oliver Hunt 2011-05-20 08:50:40 PDT
Comment on attachment 94221 [details]
Patch

r=me
Comment 48 Xan Lopez 2011-05-20 08:55:27 PDT
Comment on attachment 94221 [details]
Patch

Clearing flags on attachment: 94221

Committed r86957: <http://trac.webkit.org/changeset/86957>
Comment 49 Xan Lopez 2011-05-20 08:55:39 PDT
All reviewed patches have been landed.  Closing bug.
Comment 50 Ademar Reis 2011-05-20 10:10:07 PDT
Revision r86957 cherry-picked into qtwebkit-2.2 with commit ac3c389 <http://gitorious.org/webkit/qtwebkit/commit/ac3c389>
Comment 51 Gavin Barraclough 2011-05-23 14:24:29 PDT
Nice!
Comment 52 Alexey Proskuryakov 2011-08-05 12:07:02 PDT
See also: bug 65766.
Comment 53 Török Edwin 2013-07-16 07:40:31 PDT
This is not fixed, libqt4-script allocates 2GB of VIRT on Linux for the Javascript JIT.
Comment 54 Török Edwin 2013-07-16 07:45:25 PDT
Opened new bug here, because I cannot change the status on this bug: https://bugs.webkit.org/show_bug.cgi?id=118733
Comment 55 Alexey Proskuryakov 2013-07-16 09:48:49 PDT
Filing a new bug is much better than re-opening a two year old one with a landed patch. Thank you!