Bug 191294

Summary: IndexedDB: WAL file keeps growing
Product: WebKit Reporter: Sihui Liu <sihui_liu>
Component: New BugsAssignee: Sihui Liu <sihui_liu>
Status: RESOLVED FIXED    
Severity: Normal CC: beidson, cdumez, commit-queue, ews-watchlist, hi, joeyparrish, juliencdev, leechanr21, lincolnbaxter, maximilian.doepp1, moh.shraim, pfifield, pixelworld
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Patch
none
Patch none

Sihui Liu
Reported 2018-11-05 18:59:42 PST
https://github.com/firebase/firebase-js-sdk/issues/947 Size of WAL files of iOS apps could keep growing every time we launch the app.
Attachments
Patch (17.24 KB, patch)
2018-11-05 19:22 PST, Sihui Liu 		no flags
DetailsFormatted DiffDiff
Patch (17.08 KB, patch)
2018-11-06 10:32 PST, Sihui Liu 		no flags
DetailsFormatted DiffDiff
Show Obsolete (1) View All Add attachment proposed patch, testcase, etc.
Sihui Liu
Comment 1 2018-11-05 19:00:51 PST
<rdar://problem/41333493>
Sihui Liu
Comment 2 2018-11-05 19:22:22 PST
Created attachment 353942 [details] Patch
EWS Watchlist
Comment 3 2018-11-05 19:25:39 PST
Attachment 353942 [details] did not pass style-queue: ERROR: Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:3595: developmentRegion is not en. [xcodeproj/settings] [5] Total errors found: 1 in 7 files If any of these errors are false positives, please file a bug against check-webkit-style.
Chris Dumez
Comment 4 2018-11-05 20:34:59 PST
*** Bug 178204 has been marked as a duplicate of this bug. ***
Pete Fifield
Comment 5 2018-11-06 08:15:16 PST
We have an app that uses IndexedDB and am experiencing a similar problem. Since the bug has been recognized, do you have a workaround while we wait for the iOS phones to update?
Sihui Liu
Comment 6 2018-11-06 09:05:32 PST
(In reply to Pete Fifield from comment #5) > We have an app that uses IndexedDB and am experiencing a similar problem. > Since the bug has been recognized, do you have a workaround while we wait > for the iOS phones to update? There (In reply to Pete Fifield from comment #5) > We have an app that uses IndexedDB and am experiencing a similar problem. > Since the bug has been recognized, do you have a workaround while we wait > for the iOS phones to update? There are two things I can think of: (1) Regularly delete your IndexedDB files, via API like WKWebsiteDataStore removeDataOfTypes: this would clear both .sqlite3 and .sqlite3-wal files. (2) Open a database and close it (e.g. by navigating to another page that doesn't use that database) before the app gets killed. This would make sure sqlite3_open() and sqlite3_close() are called() (https://www.sqlite.org/wal.html, item4), which may truncate or delete the WAL file. If you can reload the content of IDB, I would recommend (1), which has more definitive result.
Chris Dumez
Comment 7 2018-11-06 09:21:00 PST
Comment on attachment 353942 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=353942&action=review > Source/WebCore/platform/sql/SQLiteDatabase.cpp:125 > + SQLiteStatement cpStatement(*this, "PRAGMA wal_checkpoint(TRUNCATE)"); "PRAGMA wal_checkpoint(TRUNCATE)"_s Also, please find a better name for the variable. > Source/WebCore/platform/sql/SQLiteDatabase.cpp:126 > + auto res = cpStatement.prepareAndStep(); Please avoid abbreviation in variable names. Also, int is actually shorter than auto. > Source/WebCore/platform/sql/SQLiteDatabase.cpp:129 > + LOG(SQLDatabase, "SQLite database checkpoint is blocked"); Shouldn't this be a LOG_ERROR? > Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-1.html:1 > +<script> <!DOCTYPE html> > Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-1.html:14 > + for (let i = 0; i < 100; i ++) { no space between i and ++. > Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-2.html:1 > +<script> <!DOCTYPE html> > Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:62 > + RetainPtr<IndexedDBFileSizeMessageHandler> handler = adoptNS([[IndexedDBFileSizeMessageHandler alloc] init]); auto > Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:63 > + RetainPtr<WKWebViewConfiguration> configuration = adoptNS([[WKWebViewConfiguration alloc] init]); auto > Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:69 > + RetainPtr<_WKWebsiteDataStoreConfiguration> websiteDataStoreConfiguration = adoptNS([[_WKWebsiteDataStoreConfiguration alloc] init]); auto > Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:73 > + RetainPtr<NSSet> types = adoptNS([[NSSet alloc] initWithObjects:WKWebsiteDataTypeIndexedDBDatabases, nil]); auto > Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:82 > + RetainPtr<WKWebView> webView = adoptNS([[WKWebView alloc] initWithFrame:NSMakeRect(0, 0, 800, 600) configuration:configuration.get()]); auto > Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:94 > + // Terminate network process to keep WAL on disk Missing period at the end.
Sihui Liu
Comment 8 2018-11-06 10:30:20 PST
Comment on attachment 353942 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=353942&action=review >> Source/WebCore/platform/sql/SQLiteDatabase.cpp:125 >> + SQLiteStatement cpStatement(*this, "PRAGMA wal_checkpoint(TRUNCATE)"); > > "PRAGMA wal_checkpoint(TRUNCATE)"_s > > Also, please find a better name for the variable. Okay. >> Source/WebCore/platform/sql/SQLiteDatabase.cpp:126 >> + auto res = cpStatement.prepareAndStep(); > > Please avoid abbreviation in variable names. > > Also, int is actually shorter than auto. This variable is used for debugging, removed. >> Source/WebCore/platform/sql/SQLiteDatabase.cpp:129 >> + LOG(SQLDatabase, "SQLite database checkpoint is blocked"); > > Shouldn't this be a LOG_ERROR? The statement sets column 0 as 1 for SQLITE_BUSY case, which means it's blocked by other active database connections now and will be executed later (https://www.sqlite.org/c3ref/wal_checkpoint_v2.html). Since it's scheduled, I think it's different from an error. >> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-1.html:1 >> +<script> > > <!DOCTYPE html> Okay. >> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-1.html:14 >> + for (let i = 0; i < 100; i ++) { > > no space between i and ++. Okay. >> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-2.html:1 >> +<script> > > <!DOCTYPE html> Okay. >> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:62 >> + RetainPtr<IndexedDBFileSizeMessageHandler> handler = adoptNS([[IndexedDBFileSizeMessageHandler alloc] init]); > > auto Okay. >> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:63 >> + RetainPtr<WKWebViewConfiguration> configuration = adoptNS([[WKWebViewConfiguration alloc] init]); > > auto Okay. >> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:69 >> + RetainPtr<_WKWebsiteDataStoreConfiguration> websiteDataStoreConfiguration = adoptNS([[_WKWebsiteDataStoreConfiguration alloc] init]); > > auto Okay. >> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:73 >> + RetainPtr<NSSet> types = adoptNS([[NSSet alloc] initWithObjects:WKWebsiteDataTypeIndexedDBDatabases, nil]); > > auto Okay. >> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:82 >> + RetainPtr<WKWebView> webView = adoptNS([[WKWebView alloc] initWithFrame:NSMakeRect(0, 0, 800, 600) configuration:configuration.get()]); > > auto Okay. >> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:94 >> + // Terminate network process to keep WAL on disk > > Missing period at the end. Added.
Sihui Liu
Comment 9 2018-11-06 10:32:15 PST
Created attachment 353974 [details] Patch
EWS Watchlist
Comment 10 2018-11-06 10:35:31 PST
Attachment 353974 [details] did not pass style-queue: ERROR: Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:3595: developmentRegion is not en. [xcodeproj/settings] [5] Total errors found: 1 in 7 files If any of these errors are false positives, please file a bug against check-webkit-style.
Chris Dumez
Comment 11 2018-11-06 10:51:50 PST
Comment on attachment 353974 [details] Patch r=me
WebKit Commit Bot
Comment 12 2018-11-06 12:57:03 PST
Comment on attachment 353974 [details] Patch Clearing flags on attachment: 353974 Committed r237882: <https://trac.webkit.org/changeset/237882>
WebKit Commit Bot
Comment 13 2018-11-06 12:57:04 PST
All reviewed patches have been landed. Closing bug.
Mohammad Shraim
Comment 14 2018-11-07 07:58:56 PST
Is this Fix will be available next iOS release?? Thanks Mohd Shraim
juliencdev
Comment 15 2019-09-01 22:10:30 PDT
Hi there, Which version of WebKit was this patch applied to? We are using Ionic for an iOS app and we experience the same issue. We are building with Xcode 10.3 which, I assume, uses the latest WebKit framework. Regards, Julien.
Sihui Liu
Comment 16 2019-09-03 11:03:22 PDT
(In reply to juliencdev from comment #15) > Hi there, > Which version of WebKit was this patch applied to? We are using Ionic for an > iOS app and we experience the same issue. We are building with Xcode 10.3 > which, I assume, uses the latest WebKit framework. > Regards, > Julien. Hi, this fix is shipped in iOS 12.2+ and macOS 10.14.4+. If you still sees this issue, please file a new bug with reproducible steps for us to look into.
juliencdev
Comment 17 2019-09-04 15:44:54 PDT
Thanks for your reply. Julien.
Lee Robertson
Comment 18 2020-04-17 18:14:39 PDT
This is still an issue in iOS 13.4.1 and Safari 13
Lincoln Baxter, III
Comment 19 2020-05-11 09:14:19 PDT
Also experiencing this bug in Safari on Desktop MacOS 10.15.4, and iOS Safari (in XCode Simulators) 13.4.1. Refreshing the page seems to sometimes create a new IndexedDB instance when one already exists. The instance has exactly the same name, and seems to have the same data as the original DB. Continuing to refresh will continue to create copies. Eventually the OS, iOS or MacOS, will prompt to allow a storage increase. This continues to MASSIVE sizes, see attachment.
Maximilian D.
Comment 20 2020-09-09 00:08:36 PDT
We have the problem for our PWA sales app also. We have ~40MB data that we need, but we have users on iOS that get a warning that more than 1,1 GB will be used. On our Chrome desktop apps we can see that in the folder: https_xxx.xxx.de_0.indexeddb.blob / 1 a lot of subfolders are shown that have the data from different points in time. If we delete the old ones the app will work as before and the storage information in the dev tools are updated correctly. Two questions: 1.) is it possible to reopen the ticket? 2.) does someone know a workaround or a way to delete the old data out of JS?
Sihui Liu
Comment 21 2022-02-01 23:28:54 PST
(In reply to Maximilian D. from comment #20) > We have the problem for our PWA sales app also. > We have ~40MB data that we need, but we have users on iOS that get a warning > that more than 1,1 GB will be used. > > On our Chrome desktop apps we can see that in the folder: > https_xxx.xxx.de_0.indexeddb.blob / 1 a lot of subfolders are shown that > have the data from different points in time. > If we delete the old ones the app will work as before and the storage > information in the dev tools are updated correctly. > > Two questions: > 1.) is it possible to reopen the ticket? > 2.) does someone know a workaround or a way to delete the old data out of JS? Hi Maximilian, If you still see the issue on a recent iOS or macOS build, please file a new bug with reproducible steps (also test files if possible), and add me to CC list. It can be a different issue from this bug and we need to investigate again (e.g. we've tried another fix at https://bugs.webkit.org/show_bug.cgi?id=202137.).
Note You need to log in before you can comment on or make changes to this bug.