Bug 256703 – [iOS] Wasm based WebApp readalong.google.com reproducibly jetsams

• Status: NEW
• Component: WebAssembly

[Pulkit]

Created attachment 466334 [details]
JetsamEvent

Hi, our web-app crashes with message "A problem repeatedly occurred..." on iOS (on devices like iPad mini 6, iPhone 13 pro) soon after the wasm is loaded. It works fine on safari on MacOS for the devices we've tried. The link to open web-app on iOS: https://readalong.google.com/?ios=1

The process is being jettisoned by iOS with reason "per_process_limit" (attached the jetsam event logs).

The memory consumption is very high (>1.4GB as can be seen from the jetsam event) - we suspect that the compiled size of wasm is very high in webkit (it is much lower on Chrome v8). The unzipped size of wasm is ~46MB.

How can we reduce the memory consumption?

[Radar WebKit Bug Importer]

<rdar://problem/109325498>

[Pulkit]

Hi, gentle ping! Did someone get a chance to take a look at this?

[Mark Lam]

Pulkit, you claim that the memory usage is much lower in Chrome v8.  Can you provide numbers from your measurements and details on how you measured it?  Chrome instrumentation doesn’t completely attribute web content memory usage to the web content e.g. they exclude memory used by the GPU process on behalf of the web content.  So knowing the details and methodology of your measurement is important for diagnosing this issue.  Thanks.

[tasnim]

Created attachment 466403 [details]
chrome_memory

[tasnim]

Created attachment 466404 [details]
safari_memory

[tasnim]

We got help from v8 engineer who reported compiled wasm size of ~198MB for the wasm (https://readalong.google.com/wasm/wasm/simd/481866636_7/wasm_wrapper_release_simd.wasm) on arm architecture. Is there a way to do the same for webkit?

On my mac, if I start the web app on chrome, chrome's taskmanager (can be opened up via settings -> more tools -> taskmanager): reports ~500MB (refer attached chrome_memory). Safari on Mac via activity monitor for the same reports to be ~2GB (refer attached safari_memory).

Sorry, I wasn't able to clearly infer chrome tab's memory via activity manager. Any pointers on what more we can do to investigate memory usage on safari?

[Justin Michaud]

(In reply to tasnim from comment #6)
> We got help from v8 engineer who reported compiled wasm size of ~198MB for
> the wasm
> (https://readalong.google.com/wasm/wasm/simd/481866636_7/
> wasm_wrapper_release_simd.wasm) on arm architecture. Is there a way to do
> the same for webkit?
> 
> On my mac, if I start the web app on chrome, chrome's taskmanager (can be
> opened up via settings -> more tools -> taskmanager): reports ~500MB (refer
> attached chrome_memory). Safari on Mac via activity monitor for the same
> reports to be ~2GB (refer attached safari_memory).
> 
> Sorry, I wasn't able to clearly infer chrome tab's memory via activity
> manager. Any pointers on what more we can do to investigate memory usage on
> safari?

I see ~2GB in Chrome too? WebKit assigns the cost GPU process memory for a tab to the tab itself, so it can be confusing to compare.

I think opening Safari and Chrome with only one tab, and measuring the total footprint is probably the best way to measure this. If you want to be extra careful about fixed overhead from the browser app itself, you can open multiple instances of the site and measure the growth rate.

[Justin Michaud]

These data so far seem to show that the memory usage isn't that different between Safari and Chrome, and it is too high in general.

> We got help from v8 engineer who reported compiled wasm size of ~198MB for the wasm (https://readalong.google.com/wasm/wasm/simd/481866636_7/wasm_wrapper_release_simd.wasm) on arm architecture. Is there a way to do the same for webkit?

I am not sure, let me see. Please note though that that is huge. We have a cap at 512MB for performance reasons, see ExecutableAllocator.cpp

In this case though, it seems like GPU memory is the precious resource. I wonder if something is getting leaked, either in your app, the wasm bindings, or inside WebKit.

I can play around with WebInspector later on, but maybe you might be able to get a better idea than me of what is using so much GPU memory on your end.

[tasnim]

(In reply to Justin Michaud from comment #7)
> (In reply to tasnim from comment #6)
> > We got help from v8 engineer who reported compiled wasm size of ~198MB for
> > the wasm
> > (https://readalong.google.com/wasm/wasm/simd/481866636_7/
> > wasm_wrapper_release_simd.wasm) on arm architecture. Is there a way to do
> > the same for webkit?
> > 
> > On my mac, if I start the web app on chrome, chrome's taskmanager (can be
> > opened up via settings -> more tools -> taskmanager): reports ~500MB (refer
> > attached chrome_memory). Safari on Mac via activity monitor for the same
> > reports to be ~2GB (refer attached safari_memory).
> > 
> > Sorry, I wasn't able to clearly infer chrome tab's memory via activity
> > manager. Any pointers on what more we can do to investigate memory usage on
> > safari?
> 
> I see ~2GB in Chrome too? WebKit assigns the cost GPU process memory for a
> tab to the tab itself, so it can be confusing to compare.
Interesting: I have done a number of retries by opening via single tab and don't see that big a number on chrome. Maybe that's because I am looking at the chrome's taskmanager and you might be looking at activitymonitor?
> 
> I think opening Safari and Chrome with only one tab, and measuring the total
> footprint is probably the best way to measure this. If you want to be extra
> careful about fixed overhead from the browser app itself, you can open
> multiple instances of the site and measure the growth rate.
I gave a shot on this to measure the growth rate:

Initially: total memory used via activity monitor: 14.40 GB
when site opened on first tab on chrome: 15.20 GB - (~800MB growth)
when same site opened on second tab on chrome: 15.90 GB - (~700 MB growth)
when site opened on third tab on chrome: 16.6 GB - (700 MB growth)

Same for safari when initial reported was 14.1 GB:
site opened on first tab: 16.70 GB (~2.6GB growth)
site opened on second tab: 19.3 GB (~2.6 GB growth)
safari reported: 22 (~2.7 GB growth) 

So, on my 3 attempts chrome is reporting a growth of 700-800 MB vs safari reporting ~2.6-2.7 GB.

[tasnim]

(In reply to Justin Michaud from comment #8)
> These data so far seem to show that the memory usage isn't that different
> between Safari and Chrome, and it is too high in general.
> 
> > We got help from v8 engineer who reported compiled wasm size of ~198MB for the wasm (https://readalong.google.com/wasm/wasm/simd/481866636_7/wasm_wrapper_release_simd.wasm) on arm architecture. Is there a way to do the same for webkit?
> 
> I am not sure, let me see. Please note though that that is huge. We have a
> cap at 512MB for performance reasons, see ExecutableAllocator.cpp
Thanks! Just to double check what exactly is the cap for. Is it for the compiled wasm?
> 
> In this case though, it seems like GPU memory is the precious resource. I
> wonder if something is getting leaked, either in your app, the wasm
> bindings, or inside WebKit.
> 
> I can play around with WebInspector later on, but maybe you might be able to
> get a better idea than me of what is using so much GPU memory on your end.

[Justin Michaud]

> Interesting: I have done a number of retries by opening via single tab and
> don't see that big a number on chrome. Maybe that's because I am looking at
> the chrome's taskmanager and you might be looking at activitymonitor?

I was looking at your screenshot

> Initially: total memory used via activity monitor: 14.40 GB
> when site opened on first tab on chrome: 15.20 GB - (~800MB growth)
> when same site opened on second tab on chrome: 15.90 GB - (~700 MB growth)
> when site opened on third tab on chrome: 16.6 GB - (700 MB growth)

I misspoke, I meant the total sum of Safari and Chrome processes. Your approach gave some interesting data too though!

> So, on my 3 attempts chrome is reporting a growth of 700-800 MB vs safari
> reporting ~2.6-2.7 GB.

That is pretty interesting, and assuming that there isn't some silly bug in our GPU memory attribution, seems to be pretty indicative of extra memory usage in Safari.

Let me see where that memory is coming from.

> Is it for the compiled wasm?

Yes, and JS

[tasnim]

(In reply to Justin Michaud from comment #11)
> > Interesting: I have done a number of retries by opening via single tab and
> > don't see that big a number on chrome. Maybe that's because I am looking at
> > the chrome's taskmanager and you might be looking at activitymonitor?
> 
> I was looking at your screenshot
> 
> > Initially: total memory used via activity monitor: 14.40 GB
> > when site opened on first tab on chrome: 15.20 GB - (~800MB growth)
> > when same site opened on second tab on chrome: 15.90 GB - (~700 MB growth)
> > when site opened on third tab on chrome: 16.6 GB - (700 MB growth)
> 
> I misspoke, I meant the total sum of Safari and Chrome processes. Your
> approach gave some interesting data too though!
> 
> > So, on my 3 attempts chrome is reporting a growth of 700-800 MB vs safari
> > reporting ~2.6-2.7 GB.
> 
> That is pretty interesting, and assuming that there isn't some silly bug in
> our GPU memory attribution, seems to be pretty indicative of extra memory
> usage in Safari.
> 
> Let me see where that memory is coming from.
Just wanted to check on any updates here and let us know if any inputs are needed from our side to help debug this. Thanks!
> 
> > Is it for the compiled wasm?
> 
> Yes, and JS

[Justin Michaud]

Hey! Thanks again for reporting this. I am currently in the process of working on some general memory improvements, and I will investigate this after they land.

Looking at Inspector, I have a few questions. I believe that this could be caused by a few reasons, and it would be super helpful if you could rule them out.

1) I see that you have a bunch of arrays of canvases. Do you ever clear those? Can you confirm that they aren't being leaked?

The hypothesis here would be that they are always reachable, and therefore never collected. This could also explain why so much "page" memory is being used.

2) Can you confirm that you are serving the same binary to Safari as to Chrome, and that both engines are going down the same code path? For example, one way that this behaviour could be explained is if readalong is using some fallback path for tts, but using a Chrome API that is way more efficient on Chrome.

I am seeing that on Chrome, you are allocating a bunch of SharedArrayBuffers, but there are none in the heap on Safari.

I am also seeing that the JS heap according to inspector is 900mb on Safari, but only 300mb on Chrome.

Another example that I've seen before with emscripten is that it could be taking a slow path because it doesn't detect the browser properly. For example, I could imagine a scenario where on Chrome, emscripten uses a weak map for something but falls back to the default case on Safari.

3) Perhaps we have a memory leak. I can do some debugging on my end.

4) Of course, WebKit could just be using more memory in general. I am working on fixing that now, but the most actionable thing for you to do to get results quickly would be to lower your overall memory usage by clearing caches, breaking reference to large data, and making sure that you don't hold on to references to big native objects like canvas.


I also have noticed that you are using 20 workers. That number seems to be a bit excessive. Note that Safari won't give you an accurate core count for fingerprinting reasons, so perhaps reducing this might be a quick way to work around this issue for now.

[Justin Michaud]

Created attachment 466452 [details]
Inspector 1

[Justin Michaud]

Created attachment 466453 [details]
Inspector 2

[tasnim]

(In reply to Justin Michaud from comment #13)
> Hey! Thanks again for reporting this. I am currently in the process of
> working on some general memory improvements, and I will investigate this
> after they land.
Thanks! can you link bugs / PR here as you work through just so that we are aware of such improvements and can experiment against them.
> 
> Looking at Inspector, I have a few questions. I believe that this could be
> caused by a few reasons, and it would be super helpful if you could rule
> them out.
> 
> 1) I see that you have a bunch of arrays of canvases. Do you ever clear
> those? Can you confirm that they aren't being leaked?
> 
> The hypothesis here would be that they are always reachable, and therefore
> never collected. This could also explain why so much "page" memory is being
> used.
These are actually managed by flutter web framework. I will get back if I find anything but I am expecting these to be not leaked. Also, we have experimented with just the flutter web app (without wasm). It works perfectly fine on iOS. Maybe we might be just close to the memory limit but the browser never reloads as pulkit mentioned initially in the thread. 
> 
> 2) Can you confirm that you are serving the same binary to Safari as to
> Chrome, and that both engines are going down the same code path? For
> example, one way that this behaviour could be explained is if readalong is
> using some fallback path for tts, but using a Chrome API that is way more
> efficient on Chrome.
Earlier, we were doing non_simd variant of wasm for safari but safari now supports simd (https://developer.apple.com/documentation/safari-release-notes/safari-16_4-release-notes) and so, both are getting the same binary.
> 
> I am seeing that on Chrome, you are allocating a bunch of
> SharedArrayBuffers, but there are none in the heap on Safari.
> 
> I am also seeing that the JS heap according to inspector is 900mb on Safari,
> but only 300mb on Chrome.
> 
Let me get back on this. How are you verifying on chrome? I was using performance.measureUserAgentSpecificMemory but it doesn't seem to correctly report back the memory. I think that over counts the SharedArrayBuffer (wasm memory) that's shared across the web workers.

> Another example that I've seen before with emscripten is that it could be
> taking a slow path because it doesn't detect the browser properly. For
> example, I could imagine a scenario where on Chrome, emscripten uses a weak
> map for something but falls back to the default case on Safari.
> 
> 3) Perhaps we have a memory leak. I can do some debugging on my end.
> 
> 4) Of course, WebKit could just be using more memory in general. I am
> working on fixing that now, but the most actionable thing for you to do to
> get results quickly would be to lower your overall memory usage by clearing
> caches, breaking reference to large data, and making sure that you don't
> hold on to references to big native objects like canvas.
Thanks, yeah we are looking into this including lowering of the compiled wasm size.
> 
> 
> I also have noticed that you are using 20 workers. That number seems to be a
> bit excessive. Note that Safari won't give you an accurate core count for
> fingerprinting reasons, so perhaps reducing this might be a quick way to
> work around this issue for now.
Just curious: we need to work on this but how much of this will actually help. I know this will help in lowering the memory usage but not sure by how much? Based on our experiments, we tried against starting with few web workers and it was still easily crashing on iOS safari.

[Justin Michaud]

> > 
> > I am seeing that on Chrome, you are allocating a bunch of
> > SharedArrayBuffers, but there are none in the heap on Safari.
> > 
> > I am also seeing that the JS heap according to inspector is 900mb on Safari,
> > but only 300mb on Chrome.
> > 
> Let me get back on this. How are you verifying on chrome? I was using
> performance.measureUserAgentSpecificMemory but it doesn't seem to correctly
> report back the memory. I think that over counts the SharedArrayBuffer (wasm
> memory) that's shared across the web workers.

I was looking at Inspector in the memory tab. It looks like Chrome vs Safari might be taking different paths here.

[tasnim]

Created attachment 466590 [details]
safari_wasm_table

[tasnim]

Created attachment 466591 [details]
chrome_wasm_table

[tasnim]

While we are working on reducing memory consumption and especially prioritizing on reducing threads, I wanted to clarify the on-device size of wasmTable. On safari main thread (see screenshot safari_wasm_table), it seems to be taking ~19.56MB whereas on chrome, it takes ~2.1MB (see chrome_wasm_table).

wasmTable seems to be copied over to all web workers. Could this explain why creating threads in safari might be expensive and any optimisation that could be done here?

[Mark Lam]

Tasnim, please get us the following info:
1.Do a grep of your code and see if you use SharedArrayBuffers.
2. If so, can you look at that code and check if your usage of it defers on Safari vs Chrome?  If so, how?
3. Does your code do any feature detection / user agent discrimination on browsers at all?

[tasnim]

(In reply to Mark Lam from comment #21)
> Tasnim, please get us the following info:
> 1.Do a grep of your code and see if you use SharedArrayBuffers.
Yes:
1) WasmMemory backed by SAB which is of 256MB (we are looking to lowering this).
2) Separate SAB of 1 byte (i.e. new SharedArrayBuffer(1)).
> 2. If so, can you look at that code and check if your usage of it defers on
> Safari vs Chrome?  If so, how?
No: no browser specific logics there.
> 3. Does your code do any feature detection / user agent discrimination on
> browsers at all?
As mentioned in https://bugs.webkit.org/show_bug.cgi?id=256703#c16, we do have "simd" feature detection where earlier we used to deliver "non-simd" wasm to safari but now it seems both: chrome & safari are getting the "simd" variant.

[Pulkit]

Hi, an update from our end - we've reduced the workers from 20 to 7 for webkit (yet to be released) and another improvement which reduced the wasm unzipped size to ~35M, which reduced the memory consumption and the app works fine on initial load, but jetsam does happen after some activity still. 

Justin, you mentioned about some general memory improvements, any updates or ETA on that?

[Justin Michaud]

Hi folks.

I checked again, and it seems that now the JS heap snapshots look very similar between Chrome and Safari (384MB vs 322MB, and composed of the same kinds of objects).

To me, it is important to rule out different js code paths, so I’m happy that this is now fixed.
 
> Justin, you mentioned about some general memory improvements, any updates or
> ETA on that?

Apple does not comment on future releases. We are always working on improving memory and performance, but I can't really predict when that work will land or ship. In any case, since the JS heaps are the same size now, these improvements will not affect your use case.

> These are actually managed by flutter web framework. I will get back if I find anything but I am expecting these to be not leaked. Also, we have experimented with just the flutter web app (without wasm). It works perfectly fine on iOS. Maybe we might be just close to the memory limit but the browser never reloads as pulkit mentioned initially in the thread. 

Can you please verify that there is no substantial difference in memory usage in the flutter web app without wasm between Chrome and Safari?

Overall: Safari is still using substantially more memory than chrome, and it is not coming from the JS heap.

I disabled gigacage, enabled Malloc Stack Logging, and enabled malloc heap breakdown in my local release build. This should give us the maximum insight into allocations.
```
1005 MallocStackLogging=lite __PC_MallocStackLogging=lite DYLD_FAMEWORK_PATH=/Volumes/WebKit/ReleaseVersion/OpenSource/WebKitBuild/Release __XPC_DYLD_FAMEWORK_PATH=/Volumes/WebKit/ReleaseVersion/OpenSource/WebKitBuild/Release /Volumes/WebKit/ReleaseVersion/OpenSource/WebKitBuild/Release/Safari.app/Contents/MacOS/Safari 
 1006  /usr/bin/notifyutil -p org.WebKit.lowMemory
 1007  /usr/local/bin/mempurge
 1008  leaks --physFootprint --outputGraph foo.memgraph 3648
 1009  heap foo.memgraph -s
```

In this configuration, I see 1.9gb total, which is way more than Chrome. Here are the biggest objects:

500mb JS heap
157mb NativeJITCode
116mb JSC::Wasm::Instance::initElementSegment
88mb JSC::Wasm::Table::tryCreate
78mb JSC::WeakBlock::create
44mb malloc in WTF::HashTable<JSC::Weak<JSC::NativeExecutable>, JSC::Weak<JSC::NativeExecutable>, WTF::IdentityExtractor, JSC::JITThunks::WeakNativeExecutableHash, WTF::HashTraits<JSC::Weak<JSC::NativeExecutable>>, WTF::HashTraits<JSC::Weak<JSC::NativeExecutable>>>::rehash(unsigned int, JSC::Weak<JSC::NativeExecutable>*) 
60mb LLInt bytecode
28mb realloc in JSC::CodeBlock::CodeBlock

I am playing around with my local build. I am definitely failing to symbolicate some allocations. I an explain only a small portion of the memory difference so far. I will keep you updated.

[tasnim]

Created attachment 466776 [details]
safari_timeline_page

[tasnim]

(In reply to Justin Michaud from comment #24)
> Hi folks.
> 
> I checked again, and it seems that now the JS heap snapshots look very
> similar between Chrome and Safari (384MB vs 322MB, and composed of the same
> kinds of objects).
> 
> To me, it is important to rule out different js code paths, so I’m happy
> that this is now fixed.
>  
> > Justin, you mentioned about some general memory improvements, any updates or
> > ETA on that?
> 
> Apple does not comment on future releases. We are always working on
> improving memory and performance, but I can't really predict when that work
> will land or ship. In any case, since the JS heaps are the same size now,
> these improvements will not affect your use case.
> 
> > These are actually managed by flutter web framework. I will get back if I find anything but I am expecting these to be not leaked. Also, we have experimented with just the flutter web app (without wasm). It works perfectly fine on iOS. Maybe we might be just close to the memory limit but the browser never reloads as pulkit mentioned initially in the thread. 
> 
> Can you please verify that there is no substantial difference in memory
> usage in the flutter web app without wasm between Chrome and Safari?
Yeah, should be same.
> 
> Overall: Safari is still using substantially more memory than chrome, and it
> is not coming from the JS heap.
> 
> I disabled gigacage, enabled Malloc Stack Logging, and enabled malloc heap
> breakdown in my local release build. This should give us the maximum insight
> into allocations.
> ```
> 1005 MallocStackLogging=lite __PC_MallocStackLogging=lite
> DYLD_FAMEWORK_PATH=/Volumes/WebKit/ReleaseVersion/OpenSource/WebKitBuild/
> Release
> __XPC_DYLD_FAMEWORK_PATH=/Volumes/WebKit/ReleaseVersion/OpenSource/
> WebKitBuild/Release
> /Volumes/WebKit/ReleaseVersion/OpenSource/WebKitBuild/Release/Safari.app/
> Contents/MacOS/Safari 
>  1006  /usr/bin/notifyutil -p org.WebKit.lowMemory
>  1007  /usr/local/bin/mempurge
>  1008  leaks --physFootprint --outputGraph foo.memgraph 3648
>  1009  heap foo.memgraph -s
> ```
> 
> In this configuration, I see 1.9gb total, which is way more than Chrome.
> Here are the biggest objects:
> 
> 500mb JS heap
> 157mb NativeJITCode
> 116mb JSC::Wasm::Instance::initElementSegment
> 88mb JSC::Wasm::Table::tryCreate
> 78mb JSC::WeakBlock::create
> 44mb malloc in WTF::HashTable<JSC::Weak<JSC::NativeExecutable>,
> JSC::Weak<JSC::NativeExecutable>, WTF::IdentityExtractor,
> JSC::JITThunks::WeakNativeExecutableHash,
> WTF::HashTraits<JSC::Weak<JSC::NativeExecutable>>,
> WTF::HashTraits<JSC::Weak<JSC::NativeExecutable>>>::rehash(unsigned int,
> JSC::Weak<JSC::NativeExecutable>*) 
> 60mb LLInt bytecode
> 28mb realloc in JSC::CodeBlock::CodeBlock
> 
> I am playing around with my local build. I am definitely failing to
> symbolicate some allocations. I an explain only a small portion of the
> memory difference so far. I will keep you updated.
Thanks let us know. Additionally, wanted to ask if we could get more insights on large "page" memory and something that we could investigate at our end to fix. I have attached timeline screenshot of our web app (without wasm, see safari_timeline_page). On https://bugs.webkit.org/show_bug.cgi?id=256703#c13, you mentioned it maybe due to too many "canvas" but just asking if there are others that maybe contributing here.

[tasnim]

(In reply to tasnim from comment #26)
> (In reply to Justin Michaud from comment #24)
> > Hi folks.
> > 
> > I checked again, and it seems that now the JS heap snapshots look very
> > similar between Chrome and Safari (384MB vs 322MB, and composed of the same
> > kinds of objects).
> > 
> > To me, it is important to rule out different js code paths, so I’m happy
> > that this is now fixed.
> >  
> > > Justin, you mentioned about some general memory improvements, any updates or
> > > ETA on that?
> > 
> > Apple does not comment on future releases. We are always working on
> > improving memory and performance, but I can't really predict when that work
> > will land or ship. In any case, since the JS heaps are the same size now,
> > these improvements will not affect your use case.
> > 
> > > These are actually managed by flutter web framework. I will get back if I find anything but I am expecting these to be not leaked. Also, we have experimented with just the flutter web app (without wasm). It works perfectly fine on iOS. Maybe we might be just close to the memory limit but the browser never reloads as pulkit mentioned initially in the thread. 
> > 
> > Can you please verify that there is no substantial difference in memory
> > usage in the flutter web app without wasm between Chrome and Safari?
> Yeah, should be same.
Sorry, I misread your question. Actually, there's a significant diff in terms of total memory footprint.
I opened the same web app (without wasm) in safari & chrome. In safari, total memory is: 460MB (as seen in activity monitor) whereas chrome: 230MB (as seen in chrome's task manager). So, a diff of > 200MB.

But, when I compare the JS heap snapshots, I see minor diff. (chrome: ~80MB vs safari: ~73MB)

> > 
> > Overall: Safari is still using substantially more memory than chrome, and it
> > is not coming from the JS heap.
> > 
> > I disabled gigacage, enabled Malloc Stack Logging, and enabled malloc heap
> > breakdown in my local release build. This should give us the maximum insight
> > into allocations.
> > ```
> > 1005 MallocStackLogging=lite __PC_MallocStackLogging=lite
> > DYLD_FAMEWORK_PATH=/Volumes/WebKit/ReleaseVersion/OpenSource/WebKitBuild/
> > Release
> > __XPC_DYLD_FAMEWORK_PATH=/Volumes/WebKit/ReleaseVersion/OpenSource/
> > WebKitBuild/Release
> > /Volumes/WebKit/ReleaseVersion/OpenSource/WebKitBuild/Release/Safari.app/
> > Contents/MacOS/Safari 
> >  1006  /usr/bin/notifyutil -p org.WebKit.lowMemory
> >  1007  /usr/local/bin/mempurge
> >  1008  leaks --physFootprint --outputGraph foo.memgraph 3648
> >  1009  heap foo.memgraph -s
> > ```
> > 
> > In this configuration, I see 1.9gb total, which is way more than Chrome.
> > Here are the biggest objects:
> > 
> > 500mb JS heap
> > 157mb NativeJITCode
> > 116mb JSC::Wasm::Instance::initElementSegment
> > 88mb JSC::Wasm::Table::tryCreate
> > 78mb JSC::WeakBlock::create
> > 44mb malloc in WTF::HashTable<JSC::Weak<JSC::NativeExecutable>,
> > JSC::Weak<JSC::NativeExecutable>, WTF::IdentityExtractor,
> > JSC::JITThunks::WeakNativeExecutableHash,
> > WTF::HashTraits<JSC::Weak<JSC::NativeExecutable>>,
> > WTF::HashTraits<JSC::Weak<JSC::NativeExecutable>>>::rehash(unsigned int,
> > JSC::Weak<JSC::NativeExecutable>*) 
> > 60mb LLInt bytecode
> > 28mb realloc in JSC::CodeBlock::CodeBlock
> > 
> > I am playing around with my local build. I am definitely failing to
> > symbolicate some allocations. I an explain only a small portion of the
> > memory difference so far. I will keep you updated.
> Thanks let us know. Additionally, wanted to ask if we could get more
> insights on large "page" memory and something that we could investigate at
> our end to fix. I have attached timeline screenshot of our web app (without
> wasm, see safari_timeline_page). On
> https://bugs.webkit.org/show_bug.cgi?id=256703#c13, you mentioned it maybe
> due to too many "canvas" but just asking if there are others that maybe
> contributing here.

[Justin Michaud]

I am leaving some notes from my investigation here:

The current status is 1.6G dirty memory in Safari, 0.9G in Chrome. We need to find 700mb.

I have collected the data with malloc heap breakdown enabled:

```
MallocStackLogging=1 __XPC_MallocStackLogging=1 __XPC_Malloc=1 Malloc=1 DYLD_FAMEWORK_PATH=/Volumes/WebKit/ReleaseVersion/OpenSource/WebKitBuild/Release __XPC_DYLD_FAMEWORK_PATH=/Volumes/WebKit/ReleaseVersion/OpenSource/WebKitBuild/Release /Volumes/WebKit/ReleaseVersion/OpenSource/WebKitBuild/Release/Safari.app/Contents/MacOS/Safari -ExtensionsEnabled NO
```

with the attached patch. I then opened ReadAlong with Instruments’ Allocation tool and looked at the call stacks.

[Justin Michaud]

Created attachment 466785 [details]
Heap Breakdown Patch

Patch to get malloc breakdowns

[Justin Michaud]

Created attachment 466786 [details]
Safari VMMap

[Justin Michaud]

Created attachment 466787 [details]
Chrome VMMap

[Justin Michaud]

Thanks again for your patience.

> Sorry, I misread your question. Actually, there's a significant diff in
> terms of total memory footprint.
> I opened the same web app (without wasm) in safari & chrome. In safari,
> total memory is: 460MB (as seen in activity monitor) whereas chrome: 230MB
> (as seen in chrome's task manager). So, a diff of > 200MB.
> 
> But, when I compare the JS heap snapshots, I see minor diff. (chrome: ~80MB
> vs safari: ~73MB)

Thank you for this data, this is super helpful!

Is there a flag that I can use to get the non-wasm version? This seems like us just generally using more memory, not any specific bug.

[Justin Michaud]

It looks like it falls back gracefully with wasm disabled, and Safari uses 513.7M dirty memory in that case. Sweet.

[tasnim]

(In reply to Justin Michaud from comment #33)
> It looks like it falls back gracefully with wasm disabled, and Safari uses
> 513.7M dirty memory in that case. Sweet.

Just to clarify: how were you able to test against non-wasm? That flag is internel-only and not yet exposed outside as we don't yet have a a valid use-case for users. If you don't have access, will it help to get more insights against non-wasm?

[Justin Michaud]

> how were you able to test against non-wasm?

I disabled wasm in the browser.

It looks like Chrome without wasm used about 0.9gb of dirty memory still without wasm.

[Justin Michaud]

I have looked further into this.

150mb - wasm tables, element sections, as you previously identified. 
- We need to optimize this so that we don't need to generate so many wasm to js wrappers, and so that we can collect them. There is no ETA on that, but we know we need to do it.

WORKAROUND: don't put functions in tables if they aren't used, and don't put data in element sections if it isn't used.

60mb - Interpreter bytecode
120mb - JIT memory

We should be able to collect these before you jetsam, and we have a separate bug about that. We know we need to do that.

WORKAROUND: try building with Os if you aren't already, and measuring if that meets your performance needs.

Thank you for filing this bug. This is a super helpful test case.

I was not able to find a single smoking gun that explains the memory usage difference, but rather many small wasm memory inefficiencies. I will try to work on these in the future, but unfortunately that leaves nothing actionable for you right now. If you can reduce the size of your wasm Table and Memory, then perhaps you can get under the jetsam limit. 

The non-wasm version seemed pretty usable and used substantially less memory, so if shipping that is an option for now, that might be the fastest fix on your end while we work to improve our wasm memory usage in future releases.

I will come back to this bug from time to time to update you on any improvements that land.

[tasnim]

(In reply to Justin Michaud from comment #36)
> I have looked further into this.
> 
> 150mb - wasm tables, element sections, as you previously identified. 
> - We need to optimize this so that we don't need to generate so many wasm to
> js wrappers, and so that we can collect them. There is no ETA on that, but
> we know we need to do it.
> 
> WORKAROUND: don't put functions in tables if they aren't used, and don't put
> data in element sections if it isn't used.
> 
> 60mb - Interpreter bytecode
> 120mb - JIT memory
> 
> We should be able to collect these before you jetsam, and we have a separate
> bug about that. We know we need to do that.
> 
> WORKAROUND: try building with Os if you aren't already, and measuring if
> that meets your performance needs.
We are already build with Oz. Seems this is the best we could do in terms of reducing code size: https://emscripten.org/docs/tools_reference/emcc.html#emcc-oz
> 
> Thank you for filing this bug. This is a super helpful test case.
> 
> I was not able to find a single smoking gun that explains the memory usage
> difference, but rather many small wasm memory inefficiencies. I will try to
> work on these in the future, but unfortunately that leaves nothing
> actionable for you right now. If you can reduce the size of your wasm Table
> and Memory, then perhaps you can get under the jetsam limit. 
> 
> The non-wasm version seemed pretty usable and used substantially less
> memory, so if shipping that is an option for now, that might be the fastest
> fix on your end while we work to improve our wasm memory usage in future
> releases.
> 
> I will come back to this bug from time to time to update you on any
> improvements that land.
Awesome, Thanks!

[tasnim]

Hi Team,
Just wanted to double check if there has been any improvements ?