191294 – IndexedDB: WAL file keeps growing

RESOLVED FIXED 191294

IndexedDB: WAL file keeps growing

https://bugs.webkit.org/show_bug.cgi?id=191294

Summary IndexedDB: WAL file keeps growing

Sihui Liu

Reported 2018-11-05 18:59:42 PST

https://github.com/firebase/firebase-js-sdk/issues/947 Size of WAL files of iOS apps could keep growing every time we launch the app.

Attachments
Patch (17.24 KB, patch) 2018-11-05 19:22 PST, Sihui Liu	no flags	Details Formatted Diff Diff
Patch (17.08 KB, patch) 2018-11-06 10:32 PST, Sihui Liu	no flags	Details Formatted Diff Diff
Show Obsolete (1) View All Add attachment proposed patch, testcase, etc.

Sihui Liu

Comment 1 2018-11-05 19:00:51 PST

<rdar://problem/41333493>

Sihui Liu

Comment 2 2018-11-05 19:22:22 PST

Created attachment 353942 [details] Patch

EWS Watchlist

Comment 3 2018-11-05 19:25:39 PST

Attachment 353942 [details] did not pass style-queue: ERROR: Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:3595: developmentRegion is not en. [xcodeproj/settings] [5] Total errors found: 1 in 7 files If any of these errors are false positives, please file a bug against check-webkit-style.

Chris Dumez

Comment 4 2018-11-05 20:34:59 PST

*** Bug 178204 has been marked as a duplicate of this bug. ***

Pete Fifield

Comment 5 2018-11-06 08:15:16 PST

We have an app that uses IndexedDB and am experiencing a similar problem. Since the bug has been recognized, do you have a workaround while we wait for the iOS phones to update?

Sihui Liu

Comment 6 2018-11-06 09:05:32 PST

(In reply to Pete Fifield from comment #5) > We have an app that uses IndexedDB and am experiencing a similar problem. > Since the bug has been recognized, do you have a workaround while we wait > for the iOS phones to update? There (In reply to Pete Fifield from comment #5) > We have an app that uses IndexedDB and am experiencing a similar problem. > Since the bug has been recognized, do you have a workaround while we wait > for the iOS phones to update? There are two things I can think of: (1) Regularly delete your IndexedDB files, via API like WKWebsiteDataStore removeDataOfTypes: this would clear both .sqlite3 and .sqlite3-wal files. (2) Open a database and close it (e.g. by navigating to another page that doesn't use that database) before the app gets killed. This would make sure sqlite3_open() and sqlite3_close() are called() (https://www.sqlite.org/wal.html, item4), which may truncate or delete the WAL file. If you can reload the content of IDB, I would recommend (1), which has more definitive result.

Chris Dumez

Comment 7 2018-11-06 09:21:00 PST

Comment on attachment 353942 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=353942&action=review > Source/WebCore/platform/sql/SQLiteDatabase.cpp:125 > + SQLiteStatement cpStatement(*this, "PRAGMA wal_checkpoint(TRUNCATE)"); "PRAGMA wal_checkpoint(TRUNCATE)"_s Also, please find a better name for the variable. > Source/WebCore/platform/sql/SQLiteDatabase.cpp:126 > + auto res = cpStatement.prepareAndStep(); Please avoid abbreviation in variable names. Also, int is actually shorter than auto. > Source/WebCore/platform/sql/SQLiteDatabase.cpp:129 > + LOG(SQLDatabase, "SQLite database checkpoint is blocked"); Shouldn't this be a LOG_ERROR? > Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-1.html:1 > +<script> <!DOCTYPE html> > Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-1.html:14 > + for (let i = 0; i < 100; i ++) { no space between i and ++. > Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-2.html:1 > +<script> <!DOCTYPE html> > Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:62 > + RetainPtr<IndexedDBFileSizeMessageHandler> handler = adoptNS([[IndexedDBFileSizeMessageHandler alloc] init]); auto > Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:63 > + RetainPtr<WKWebViewConfiguration> configuration = adoptNS([[WKWebViewConfiguration alloc] init]); auto > Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:69 > + RetainPtr<_WKWebsiteDataStoreConfiguration> websiteDataStoreConfiguration = adoptNS([[_WKWebsiteDataStoreConfiguration alloc] init]); auto > Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:73 > + RetainPtr<NSSet> types = adoptNS([[NSSet alloc] initWithObjects:WKWebsiteDataTypeIndexedDBDatabases, nil]); auto > Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:82 > + RetainPtr<WKWebView> webView = adoptNS([[WKWebView alloc] initWithFrame:NSMakeRect(0, 0, 800, 600) configuration:configuration.get()]); auto > Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:94 > + // Terminate network process to keep WAL on disk Missing period at the end.

Sihui Liu

Comment 8 2018-11-06 10:30:20 PST

Comment on attachment 353942 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=353942&action=review >> Source/WebCore/platform/sql/SQLiteDatabase.cpp:125 >> + SQLiteStatement cpStatement(*this, "PRAGMA wal_checkpoint(TRUNCATE)"); > > "PRAGMA wal_checkpoint(TRUNCATE)"_s > > Also, please find a better name for the variable. Okay. >> Source/WebCore/platform/sql/SQLiteDatabase.cpp:126 >> + auto res = cpStatement.prepareAndStep(); > > Please avoid abbreviation in variable names. > > Also, int is actually shorter than auto. This variable is used for debugging, removed. >> Source/WebCore/platform/sql/SQLiteDatabase.cpp:129 >> + LOG(SQLDatabase, "SQLite database checkpoint is blocked"); > > Shouldn't this be a LOG_ERROR? The statement sets column 0 as 1 for SQLITE_BUSY case, which means it's blocked by other active database connections now and will be executed later (https://www.sqlite.org/c3ref/wal_checkpoint_v2.html). Since it's scheduled, I think it's different from an error. >> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-1.html:1 >> +<script> > > <!DOCTYPE html> Okay. >> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-1.html:14 >> + for (let i = 0; i < 100; i ++) { > > no space between i and ++. Okay. >> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-2.html:1 >> +<script> > > <!DOCTYPE html> Okay. >> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:62 >> + RetainPtr<IndexedDBFileSizeMessageHandler> handler = adoptNS([[IndexedDBFileSizeMessageHandler alloc] init]); > > auto Okay. >> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:63 >> + RetainPtr<WKWebViewConfiguration> configuration = adoptNS([[WKWebViewConfiguration alloc] init]); > > auto Okay. >> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:69 >> + RetainPtr<_WKWebsiteDataStoreConfiguration> websiteDataStoreConfiguration = adoptNS([[_WKWebsiteDataStoreConfiguration alloc] init]); > > auto Okay. >> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:73 >> + RetainPtr<NSSet> types = adoptNS([[NSSet alloc] initWithObjects:WKWebsiteDataTypeIndexedDBDatabases, nil]); > > auto Okay. >> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:82 >> + RetainPtr<WKWebView> webView = adoptNS([[WKWebView alloc] initWithFrame:NSMakeRect(0, 0, 800, 600) configuration:configuration.get()]); > > auto Okay. >> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:94 >> + // Terminate network process to keep WAL on disk > > Missing period at the end. Added.

Sihui Liu

Comment 9 2018-11-06 10:32:15 PST

Created attachment 353974 [details] Patch

EWS Watchlist

Comment 10 2018-11-06 10:35:31 PST

Attachment 353974 [details] did not pass style-queue: ERROR: Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:3595: developmentRegion is not en. [xcodeproj/settings] [5] Total errors found: 1 in 7 files If any of these errors are false positives, please file a bug against check-webkit-style.

Chris Dumez

Comment 11 2018-11-06 10:51:50 PST

Comment on attachment 353974 [details] Patch r=me

WebKit Commit Bot

Comment 12 2018-11-06 12:57:03 PST

Comment on attachment 353974 [details] Patch Clearing flags on attachment: 353974 Committed r237882: <https://trac.webkit.org/changeset/237882>

WebKit Commit Bot

Comment 13 2018-11-06 12:57:04 PST

All reviewed patches have been landed. Closing bug.

Mohammad Shraim

Comment 14 2018-11-07 07:58:56 PST

Is this Fix will be available next iOS release?? Thanks Mohd Shraim

juliencdev

Comment 15 2019-09-01 22:10:30 PDT

Hi there, Which version of WebKit was this patch applied to? We are using Ionic for an iOS app and we experience the same issue. We are building with Xcode 10.3 which, I assume, uses the latest WebKit framework. Regards, Julien.

Sihui Liu

Comment 16 2019-09-03 11:03:22 PDT

(In reply to juliencdev from comment #15) > Hi there, > Which version of WebKit was this patch applied to? We are using Ionic for an > iOS app and we experience the same issue. We are building with Xcode 10.3 > which, I assume, uses the latest WebKit framework. > Regards, > Julien. Hi, this fix is shipped in iOS 12.2+ and macOS 10.14.4+. If you still sees this issue, please file a new bug with reproducible steps for us to look into.

juliencdev

Comment 17 2019-09-04 15:44:54 PDT

Thanks for your reply. Julien.

Lee Robertson

Comment 18 2020-04-17 18:14:39 PDT

This is still an issue in iOS 13.4.1 and Safari 13

Lincoln Baxter, III

Comment 19 2020-05-11 09:14:19 PDT

Also experiencing this bug in Safari on Desktop MacOS 10.15.4, and iOS Safari (in XCode Simulators) 13.4.1. Refreshing the page seems to sometimes create a new IndexedDB instance when one already exists. The instance has exactly the same name, and seems to have the same data as the original DB. Continuing to refresh will continue to create copies. Eventually the OS, iOS or MacOS, will prompt to allow a storage increase. This continues to MASSIVE sizes, see attachment.

Maximilian D.

Comment 20 2020-09-09 00:08:36 PDT

We have the problem for our PWA sales app also. We have ~40MB data that we need, but we have users on iOS that get a warning that more than 1,1 GB will be used. On our Chrome desktop apps we can see that in the folder: https_xxx.xxx.de_0.indexeddb.blob / 1 a lot of subfolders are shown that have the data from different points in time. If we delete the old ones the app will work as before and the storage information in the dev tools are updated correctly. Two questions: 1.) is it possible to reopen the ticket? 2.) does someone know a workaround or a way to delete the old data out of JS?

Sihui Liu

Comment 21 2022-02-01 23:28:54 PST

(In reply to Maximilian D. from comment #20) > We have the problem for our PWA sales app also. > We have ~40MB data that we need, but we have users on iOS that get a warning > that more than 1,1 GB will be used. > > On our Chrome desktop apps we can see that in the folder: > https_xxx.xxx.de_0.indexeddb.blob / 1 a lot of subfolders are shown that > have the data from different points in time. > If we delete the old ones the app will work as before and the storage > information in the dev tools are updated correctly. > > Two questions: > 1.) is it possible to reopen the ticket? > 2.) does someone know a workaround or a way to delete the old data out of JS? Hi Maximilian, If you still see the issue on a recent iOS or macOS build, please file a new bug with reproducible steps (also test files if possible), and add me to CC list. It can be a different issue from this bug and we need to investigate again (e.g. we've tried another fix at https://bugs.webkit.org/show_bug.cgi?id=202137.).

Note You need to log in before you can comment on or make changes to this bug.

Status RESOLVED

Resolution FIXED

Priority P2

Severity Normal

Classification Unclassified

Version WebKit Nightly Build

Hardware Unspecified

OS Unspecified

Product WebKit

Component New Bugs

Assignee

Sihui Liu

Reported

2018-11-05 18:59 PST

Modified

2022-02-01 23:28 PST History

CC List

13 users Show

URL

Keywords InRadar

Duplicates (1)

178204 View as bug list

Depends on

Blocks