Summary: | IndexedDB causes massive storage usage on iOS, never recycles usage | ||
---|---|---|---|
Product: | WebKit | Reporter: | Sondre Bjellås <pixelworld> |
Component: | WebKit Misc. | Assignee: | Nobody <webkit-unassigned> |
Status: | RESOLVED DUPLICATE | ||
Severity: | Normal | CC: | ap, beidson, cdumez, ggaren, joeyparrish, leechanr21, lincolnbaxter, matt, moh.shraim, nishanth.fairfield, sihui_liu, webkit-bug-importer, youennf |
Priority: | P2 | Keywords: | InRadar |
Version: | Safari 11 | ||
Hardware: | iPhone / iPad | ||
OS: | iOS 11 | ||
URL: | https://www.sqlite.org/wal.html#avoiding_excessively_large_wal_files |
Description
Sondre Bjellås
2017-10-12 02:21:22 PDT
Has anyone else been able to reproduce similar problems? Here is another developer that have similar issue where IndexedDB on iOS uses gigabytes of storage for what is essentially 2MB of data: https://github.com/localForage/localForage/issues/683 I performed some testing using Safari on the latest iOS 11.2.6 (iPhone 8) using the sample created by cmumford in this bug report: https://bugs.chromium.org/p/chromium/issues/detail?id=488851 Code: https://bugs.chromium.org/p/chromium/issues/attachmentText?aid=100949 It appears as long as the db connection is open, the storage reported by the phone for Safari increases every time the same content is written to IndexedDB. When the db connection is closed, either by code or by closing the browser tab, then Safari is able to flush the storage back to what actual written size of data. Since our app is a long-running app on the device and likely never closed the db connection, it just kept increasing used storage. We are writing a lot of data continuously, but we'll attempt changing our logic. We will attempt to open and close the database on intervals to ensure that it flushes the usage. Face the same problem on iOS 11.2 and 11.4 Sent apple bug reporter details 2 days ago.. The problem exist on Safari, Chrome and Any application using WKWebView with IndexDB. to Reproduce the problem on any iOS device i made simple page https://ssbyte.com/dbtest/ To see the big increase in Application Data on any iOS device just open the link on Safari Or Chrome. * keep repeat the steps of open Safari Or Chrome And closing the application.. Generally every time you will see increase from 0.1 to 2.5 MB in the application data.. I Traced the problem on WKWebView Sample application and Found the FOLOWWING: To detect the increment we download the device application container through xCode; at application container we found the source of increment on the following file; IndexedDB.sqlite3-wal and this was the location of file at container: AppData/Library/WebKit/WebsiteData/IndexedDB/file__0/dbName/IndexedDB.sqlite3-wal according to https://www.sqlite.org/tempfiles.html 2.2. Write-Ahead Log (WAL) Files A write-ahead log or WAL file is used in place of a rollback journal when SQLite is operating in WAL mode. As with the rollback journal, the purpose of the WAL file is to implement atomic commit and rollback. The WAL file is always located in the same directory as the database file and has the same name as the database file except with the 4 characters "-wal" appended. The WAL file is created when the first connection to the database is opened and is normally removed when the last connection to the database closes. However, if the last connection does not shutdown cleanly, the WAL file will remain in the filesystem and will be automatically cleaned up the next time the database is opened. i think we should update the importance of this bug to critical, or even blocker. We are running in to the same issues on ios apps with indexed db for offline storage where customers have complained about app occupying humongous storage space. One device showed the app has used up 18.4G of storage but the actual document size is under 30 Mb. Any help on this will be greatly appreciated. (In reply to Nishanth from comment #6) > We are running in to the same issues on ios apps with indexed db for offline > storage where customers have complained about app occupying humongous > storage space. One device showed the app has used up 18.4G of storage but > the actual document size is under 30 Mb. Any help on this will be greatly > appreciated. Does not seem to reproduce for me on iOS 12 beta (or macOS) with Safari. 1. Went to https://ssbyte.com/dbtest/ in Safari .../Library/WebKit/WebsiteData/IndexedDB/https_ssbyte.com_0/__dbnames/ 68K Jul 20 16:30 IndexedDB.sqlite3 32K Jul 20 16:30 IndexedDB.sqlite3-shm 8.1K Jul 20 16:30 IndexedDB.sqlite3-wal 2. reloaded the page several times: .../Library/WebKit/WebsiteData/IndexedDB/https_ssbyte.com_0/__dbnames/ 68K Jul 20 16:30 IndexedDB.sqlite3 32K Jul 20 16:30 IndexedDB.sqlite3-shm 8.1K Jul 20 16:30 IndexedDB.sqlite3-wal (In reply to Chris Dumez from comment #7) > (In reply to Nishanth from comment #6) > > We are running in to the same issues on ios apps with indexed db for offline > > storage where customers have complained about app occupying humongous > > storage space. One device showed the app has used up 18.4G of storage but > > the actual document size is under 30 Mb. Any help on this will be greatly > > appreciated. > > Does not seem to reproduce for me on iOS 12 beta (or macOS) with Safari. > 1. Went to https://ssbyte.com/dbtest/ in Safari > .../Library/WebKit/WebsiteData/IndexedDB/https_ssbyte.com_0/__dbnames/ > 68K Jul 20 16:30 IndexedDB.sqlite3 > 32K Jul 20 16:30 IndexedDB.sqlite3-shm > 8.1K Jul 20 16:30 IndexedDB.sqlite3-wal > > 2. reloaded the page several times: > .../Library/WebKit/WebsiteData/IndexedDB/https_ssbyte.com_0/__dbnames/ > 68K Jul 20 16:30 IndexedDB.sqlite3 > 32K Jul 20 16:30 IndexedDB.sqlite3-shm > 8.1K Jul 20 16:30 IndexedDB.sqlite3-wal If I relaunch Safari and reopen the page, it temporarily increases a bit: 68K Jul 20 16:42 IndexedDB.sqlite3 32K Jul 20 16:42 IndexedDB.sqlite3-shm 16K Jul 20 16:42 IndexedDB.sqlite3-wal But then if I reload, the size goes down again. Dear Chris Dumez, Before i report this bug here, i open another on bugs.chromium because i though it is chrome bug, then found its exist on all iOS applications using IndexDB the following link show you a Short video of case https://bugs.chromium.org/p/chromium/issues/detail?id=855647#c5 Please note; 1- I didnt test the case on iOS 12 beta. 2- This problem should happen exactly when u close Safari or Chrome on iOS while a running transaction is not completed; So to have it on your device keep open and close the browser quickly as you can see at video.. 3- Personally iam using JS Library from Firebase (Firestore - with enabled prdisitance) and this using IndexDB but they dont close transactions after each operation, and keep listening for DB changes... so when i open the real website application on iOS i can see the big data increase each time.. sometimes reach 10MB. (In reply to Chris Dumez from comment #8) > (In reply to Chris Dumez from comment #7) > > (In reply to Nishanth from comment #6) > > > We are running in to the same issues on ios apps with indexed db for offline > > > storage where customers have complained about app occupying humongous > > > storage space. One device showed the app has used up 18.4G of storage but > > > the actual document size is under 30 Mb. Any help on this will be greatly > > > appreciated. > > > > Does not seem to reproduce for me on iOS 12 beta (or macOS) with Safari. > > 1. Went to https://ssbyte.com/dbtest/ in Safari > > .../Library/WebKit/WebsiteData/IndexedDB/https_ssbyte.com_0/__dbnames/ > > 68K Jul 20 16:30 IndexedDB.sqlite3 > > 32K Jul 20 16:30 IndexedDB.sqlite3-shm > > 8.1K Jul 20 16:30 IndexedDB.sqlite3-wal > > > > 2. reloaded the page several times: > > .../Library/WebKit/WebsiteData/IndexedDB/https_ssbyte.com_0/__dbnames/ > > 68K Jul 20 16:30 IndexedDB.sqlite3 > > 32K Jul 20 16:30 IndexedDB.sqlite3-shm > > 8.1K Jul 20 16:30 IndexedDB.sqlite3-wal > > If I relaunch Safari and reopen the page, it temporarily increases a bit: > 68K Jul 20 16:42 IndexedDB.sqlite3 > 32K Jul 20 16:42 IndexedDB.sqlite3-shm > 16K Jul 20 16:42 IndexedDB.sqlite3-wal > > But then if I reload, the size goes down again. Hello Chris Dumez, I have test the same scenario on iOS 12.6 Beta Problem still exist.. Thanks We're seeing this issue with localForage on WordPress.com in quite a drastic way: https://github.com/Automattic/wp-calypso/issues/27086 The user in question has a 48.3MB database but a 203GB WAL that (apparently) continually returns and is causing serious issues with their machine. *** This bug has been marked as a duplicate of bug 191294 *** (In reply to Sondre Bjellås from comment #1) > Has anyone else been able to reproduce similar problems? > > Here is another developer that have similar issue where IndexedDB on iOS > uses gigabytes of storage for what is essentially 2MB of data: > > https://github.com/localForage/localForage/issues/683 Yes, I have been having the same problem, my pwa does a high velocity of writes to indexed db, causing the indexeddb.sqlite-wal to grow to 1.2GB and higher. As of right now, my only solution is to actually delete the DB, before each write which will remove the old log file. Also experiencing this bug in Safari on Desktop MacOS, and iOS Safari (in XCode Simulators) 13.4.1. Refreshing the page seems to sometimes create a new IndexedDB instance when one already exists. The instance has exactly the same name, and seems to have the same data as the original DB. Continuing to refresh will continue to create copies. Eventually the OS, iOS or MacOS, will prompt to allow a storage increase. This continues to MASSIVE sizes, see attachment. |