Bug 178204 – IndexedDB causes massive storage usage on iOS, never recycles usage

• Status: RESOLVED
• Resolution: DUPLICATE
• Component: WebKit Misc.

[Sondre Bjellås]

It appears to be an issue with writing updates a lot of times to IndexedDB on iOS devices. Even the simplest structures in IndexedDB (my test used 2 KB of data) will eventually (easily use more than 1GB a day) result in usage reported by the phone to be in gigabytes of size for the "Documents & Data" section for the app under "iPhone Storage".

I made a simple test app in Cordova, that uses LokiJS (have verified the issue also exists when using IndexedDB API directly) and ran the same operation over and over again writing (updating) 5 entities. The reported storage usage on iOS is continuously increasing and never recycles.

This happened on iOS 10.*, 11.0.2 and 11.0.3 on an iPhone 6, and have been reported to us from other devices as well.

After Chrome 61, it is now possible to see a chart in the Application tab on storage usage. IndexedDB size increases while the test are running, but it will recycle and drop once in a while. This same behavior is experienced for a Cordova app on Android, where the "Data" reported under "Storage" for the app increases a bit, then falls again.

Have verified that storage is recycled properly on Chrome on Windows 10 and OS X.

We have also experienced that Chrome on the desktop sometimes have a similar issue and use a lot of storage, that is never recycled, but don't know if this is related or a different bug.

The amount of data stored in the IndexedDB database, is verified to only be 2 KB in these tests, there are no real size increase (only a date field on the entities are updated) in the data actually stored in the IndexedDB.

One mitigation for this bug is to avoid persisting to IndexedDB on every update, but eventually the app will use up all available storage.

Additional bug details is reported here: https://github.com/techfort/LokiJS/issues/620

[Sondre Bjellås]

Has anyone else been able to reproduce similar problems?

Here is another developer that have similar issue where IndexedDB on iOS uses gigabytes of storage for what is essentially 2MB of data:

https://github.com/localForage/localForage/issues/683

[Sondre Bjellås]

I performed some testing using Safari on the latest iOS 11.2.6 (iPhone 8) using the sample created by cmumford in this bug report:

https://bugs.chromium.org/p/chromium/issues/detail?id=488851

Code:

https://bugs.chromium.org/p/chromium/issues/attachmentText?aid=100949

It appears as long as the db connection is open, the storage reported by the phone for Safari increases every time the same content is written to IndexedDB. When the db connection is closed, either by code or by closing the browser tab, then Safari is able to flush the storage back to what actual written size of data.

Since our app is a long-running app on the device and likely never closed the db connection, it just kept increasing used storage. We are writing a lot of data continuously, but we'll attempt changing our logic. We will attempt to open and close the database on intervals to ensure that it flushes the usage.

[Mohammad Shraim]

Face the same problem on iOS 11.2 and 11.4
Sent apple bug reporter details 2 days ago..

The problem exist on Safari, Chrome and Any application using WKWebView with IndexDB.

to Reproduce the problem on any iOS device i made simple page https://ssbyte.com/dbtest/

To see the big increase in Application Data on any iOS device just open the link on Safari Or Chrome.
* keep repeat the steps of open Safari Or Chrome And closing the application..

Generally every time you will see increase from 0.1 to 2.5 MB in the application data..

I Traced the problem on WKWebView Sample application and Found the FOLOWWING:

To detect the increment we download the device application container through xCode; at application container we found the source of increment on the following file; IndexedDB.sqlite3-wal
and this was the location of file at container:
AppData/Library/WebKit/WebsiteData/IndexedDB/file__0/dbName/IndexedDB.sqlite3-wal

according to https://www.sqlite.org/tempfiles.html

2.2. Write-Ahead Log (WAL) Files
A write-ahead log or WAL file is used in place of a rollback journal when SQLite is operating in WAL mode. As with the rollback journal, the purpose of the WAL file is to implement atomic commit and rollback. The WAL file is always located in the same directory as the database file and has the same name as the database file except with the 4 characters "-wal" appended. The WAL file is created when the first connection to the database is opened and is normally removed when the last connection to the database closes. However, if the last connection does not shutdown cleanly, the WAL file will remain in the filesystem and will be automatically cleaned up the next time the database is opened.

[Radar WebKit Bug Importer]

<rdar://problem/41404717>

[Mohammad Shraim]

i think we should update the  importance of this bug to critical, or even blocker.

[Nishanth]

We are running in to the same issues on ios apps with indexed db for offline storage where customers have complained about app occupying humongous storage space. One device showed the app has used up 18.4G of storage but the actual document size is under 30 Mb. Any help on this will be greatly appreciated.

[Chris Dumez]

(In reply to Nishanth from comment #6)
> We are running in to the same issues on ios apps with indexed db for offline
> storage where customers have complained about app occupying humongous
> storage space. One device showed the app has used up 18.4G of storage but
> the actual document size is under 30 Mb. Any help on this will be greatly
> appreciated.

Does not seem to reproduce for me on iOS 12 beta (or macOS) with Safari.
1. Went to https://ssbyte.com/dbtest/ in Safari
.../Library/WebKit/WebsiteData/IndexedDB/https_ssbyte.com_0/__dbnames/
    68K Jul 20 16:30 IndexedDB.sqlite3
    32K Jul 20 16:30 IndexedDB.sqlite3-shm
   8.1K Jul 20 16:30 IndexedDB.sqlite3-wal

2. reloaded the page several times:
.../Library/WebKit/WebsiteData/IndexedDB/https_ssbyte.com_0/__dbnames/
    68K Jul 20 16:30 IndexedDB.sqlite3
    32K Jul 20 16:30 IndexedDB.sqlite3-shm
   8.1K Jul 20 16:30 IndexedDB.sqlite3-wal

[Chris Dumez]

(In reply to Chris Dumez from comment #7)
> (In reply to Nishanth from comment #6)
> > We are running in to the same issues on ios apps with indexed db for offline
> > storage where customers have complained about app occupying humongous
> > storage space. One device showed the app has used up 18.4G of storage but
> > the actual document size is under 30 Mb. Any help on this will be greatly
> > appreciated.
> 
> Does not seem to reproduce for me on iOS 12 beta (or macOS) with Safari.
> 1. Went to https://ssbyte.com/dbtest/ in Safari
> .../Library/WebKit/WebsiteData/IndexedDB/https_ssbyte.com_0/__dbnames/
>     68K Jul 20 16:30 IndexedDB.sqlite3
>     32K Jul 20 16:30 IndexedDB.sqlite3-shm
>    8.1K Jul 20 16:30 IndexedDB.sqlite3-wal
> 
> 2. reloaded the page several times:
> .../Library/WebKit/WebsiteData/IndexedDB/https_ssbyte.com_0/__dbnames/
>     68K Jul 20 16:30 IndexedDB.sqlite3
>     32K Jul 20 16:30 IndexedDB.sqlite3-shm
>    8.1K Jul 20 16:30 IndexedDB.sqlite3-wal

If I relaunch Safari and reopen the page, it temporarily increases a bit:
    68K Jul 20 16:42 IndexedDB.sqlite3
    32K Jul 20 16:42 IndexedDB.sqlite3-shm
    16K Jul 20 16:42 IndexedDB.sqlite3-wal

But then if I reload, the size goes down again.

[Mohammad Shraim]

Dear Chris Dumez,

Before i report this bug here, i open another on bugs.chromium because i though it is chrome bug, then found its exist on all iOS applications using IndexDB

the following link show you a Short video of case

https://bugs.chromium.org/p/chromium/issues/detail?id=855647#c5

Please note; 
1- I didnt test the case on iOS 12 beta.
2- This problem should happen exactly when u close Safari or Chrome on iOS while a running transaction is not completed; So to have it on your device keep open and close the browser quickly as you can see at video..
3- Personally iam using  JS Library from Firebase (Firestore - with enabled prdisitance) and this using IndexDB but they dont close transactions after each operation, and keep listening for DB changes... so when i open the real website application on iOS i can see the big data increase each time.. sometimes reach 10MB.


(In reply to Chris Dumez from comment #8)
> (In reply to Chris Dumez from comment #7)
> > (In reply to Nishanth from comment #6)
> > > We are running in to the same issues on ios apps with indexed db for offline
> > > storage where customers have complained about app occupying humongous
> > > storage space. One device showed the app has used up 18.4G of storage but
> > > the actual document size is under 30 Mb. Any help on this will be greatly
> > > appreciated.
> > 
> > Does not seem to reproduce for me on iOS 12 beta (or macOS) with Safari.
> > 1. Went to https://ssbyte.com/dbtest/ in Safari
> > .../Library/WebKit/WebsiteData/IndexedDB/https_ssbyte.com_0/__dbnames/
> >     68K Jul 20 16:30 IndexedDB.sqlite3
> >     32K Jul 20 16:30 IndexedDB.sqlite3-shm
> >    8.1K Jul 20 16:30 IndexedDB.sqlite3-wal
> > 
> > 2. reloaded the page several times:
> > .../Library/WebKit/WebsiteData/IndexedDB/https_ssbyte.com_0/__dbnames/
> >     68K Jul 20 16:30 IndexedDB.sqlite3
> >     32K Jul 20 16:30 IndexedDB.sqlite3-shm
> >    8.1K Jul 20 16:30 IndexedDB.sqlite3-wal
> 
> If I relaunch Safari and reopen the page, it temporarily increases a bit:
>     68K Jul 20 16:42 IndexedDB.sqlite3
>     32K Jul 20 16:42 IndexedDB.sqlite3-shm
>     16K Jul 20 16:42 IndexedDB.sqlite3-wal
> 
> But then if I reload, the size goes down again.

[Mohammad Shraim]

Hello Chris Dumez,

I have test the same scenario on iOS 12.6 Beta
Problem still exist..


Thanks

[Matthew Riley MacPherson [:tofumatt]]

We're seeing this issue with localForage on WordPress.com in quite a drastic way: https://github.com/Automattic/wp-calypso/issues/27086

The user in question has a 48.3MB database but a 203GB WAL that (apparently) continually returns and is causing serious issues with their machine.

[Chris Dumez]

*** This bug has been marked as a duplicate of bug 191294 ***

[Lee Robertson]

(In reply to Sondre Bjellås from comment #1)
> Has anyone else been able to reproduce similar problems?
> 
> Here is another developer that have similar issue where IndexedDB on iOS
> uses gigabytes of storage for what is essentially 2MB of data:
> 
> https://github.com/localForage/localForage/issues/683

Yes, I have been having the same problem, my pwa does a high velocity of writes to indexed db, causing the indexeddb.sqlite-wal to grow to 1.2GB and higher. 

As of right now, my only solution is to actually delete the DB, before each write which will remove the old log file.

[Lincoln Baxter, III]

Also experiencing this bug in Safari on Desktop MacOS, and iOS Safari (in XCode Simulators) 13.4.1.

Refreshing the page seems to sometimes create a new IndexedDB instance when one already exists. The instance has exactly the same name, and seems to have the same data as the original DB.

Continuing to refresh will continue to create copies. Eventually the OS, iOS or MacOS, will prompt to allow a storage increase. This continues to MASSIVE sizes, see attachment.