Bug 191294 - IndexedDB: WAL file keeps growing
Summary: IndexedDB: WAL file keeps growing
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: New Bugs (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Sihui Liu
URL:
Keywords: InRadar
: 178204 (view as bug list)
Depends on:
Blocks:
 
Reported: 2018-11-05 18:59 PST by Sihui Liu
Modified: 2020-09-09 00:08 PDT (History)
13 users (show)

See Also:


Attachments
Patch (17.24 KB, patch)
2018-11-05 19:22 PST, Sihui Liu
no flags Details | Formatted Diff | Diff
Patch (17.08 KB, patch)
2018-11-06 10:32 PST, Sihui Liu
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Sihui Liu 2018-11-05 18:59:42 PST
https://github.com/firebase/firebase-js-sdk/issues/947

Size of WAL files of iOS apps could keep growing every time we launch the app.
Comment 1 Sihui Liu 2018-11-05 19:00:51 PST
<rdar://problem/41333493>
Comment 2 Sihui Liu 2018-11-05 19:22:22 PST
Created attachment 353942 [details]
Patch
Comment 3 EWS Watchlist 2018-11-05 19:25:39 PST
Attachment 353942 [details] did not pass style-queue:


ERROR: Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:3595:  developmentRegion is not en.  [xcodeproj/settings] [5]
Total errors found: 1 in 7 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 4 Chris Dumez 2018-11-05 20:34:59 PST
*** Bug 178204 has been marked as a duplicate of this bug. ***
Comment 5 Pete Fifield 2018-11-06 08:15:16 PST
We have an app that uses IndexedDB and am experiencing a similar problem.  Since the bug has been recognized, do you have a workaround while we wait for the iOS phones to update?
Comment 6 Sihui Liu 2018-11-06 09:05:32 PST
(In reply to Pete Fifield from comment #5)
> We have an app that uses IndexedDB and am experiencing a similar problem. 
> Since the bug has been recognized, do you have a workaround while we wait
> for the iOS phones to update?

There (In reply to Pete Fifield from comment #5)
> We have an app that uses IndexedDB and am experiencing a similar problem. 
> Since the bug has been recognized, do you have a workaround while we wait
> for the iOS phones to update?

There are two things I can think of:
(1) Regularly delete your IndexedDB files, via API like WKWebsiteDataStore removeDataOfTypes: this would clear both .sqlite3 and .sqlite3-wal files.
(2) Open a database and close it (e.g. by navigating to another page that doesn't use that database) before the app gets killed. This would make sure sqlite3_open() and sqlite3_close() are called() (https://www.sqlite.org/wal.html, item4), which may truncate or delete the WAL file.

If you can reload the content of IDB, I would recommend (1), which has more definitive result.
Comment 7 Chris Dumez 2018-11-06 09:21:00 PST
Comment on attachment 353942 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=353942&action=review

> Source/WebCore/platform/sql/SQLiteDatabase.cpp:125
> +    SQLiteStatement cpStatement(*this, "PRAGMA wal_checkpoint(TRUNCATE)");

"PRAGMA wal_checkpoint(TRUNCATE)"_s

Also, please find a better name for the variable.

> Source/WebCore/platform/sql/SQLiteDatabase.cpp:126
> +    auto res = cpStatement.prepareAndStep();

Please avoid abbreviation in variable names.

Also, int is actually shorter than auto.

> Source/WebCore/platform/sql/SQLiteDatabase.cpp:129
> +            LOG(SQLDatabase, "SQLite database checkpoint is blocked");

Shouldn't this be a LOG_ERROR?

> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-1.html:1
> +<script>

<!DOCTYPE html>

> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-1.html:14
> +    for (let i = 0; i < 100; i ++) {

no space between i and ++.

> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-2.html:1
> +<script>

<!DOCTYPE html>

> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:62
> +    RetainPtr<IndexedDBFileSizeMessageHandler> handler = adoptNS([[IndexedDBFileSizeMessageHandler alloc] init]);

auto

> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:63
> +    RetainPtr<WKWebViewConfiguration> configuration = adoptNS([[WKWebViewConfiguration alloc] init]);

auto

> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:69
> +    RetainPtr<_WKWebsiteDataStoreConfiguration> websiteDataStoreConfiguration = adoptNS([[_WKWebsiteDataStoreConfiguration alloc] init]);

auto

> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:73
> +    RetainPtr<NSSet> types = adoptNS([[NSSet alloc] initWithObjects:WKWebsiteDataTypeIndexedDBDatabases, nil]);

auto

> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:82
> +    RetainPtr<WKWebView> webView = adoptNS([[WKWebView alloc] initWithFrame:NSMakeRect(0, 0, 800, 600) configuration:configuration.get()]);

auto

> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:94
> +    // Terminate network process to keep WAL on disk

Missing period at the end.
Comment 8 Sihui Liu 2018-11-06 10:30:20 PST
Comment on attachment 353942 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=353942&action=review

>> Source/WebCore/platform/sql/SQLiteDatabase.cpp:125
>> +    SQLiteStatement cpStatement(*this, "PRAGMA wal_checkpoint(TRUNCATE)");
> 
> "PRAGMA wal_checkpoint(TRUNCATE)"_s
> 
> Also, please find a better name for the variable.

Okay.

>> Source/WebCore/platform/sql/SQLiteDatabase.cpp:126
>> +    auto res = cpStatement.prepareAndStep();
> 
> Please avoid abbreviation in variable names.
> 
> Also, int is actually shorter than auto.

This variable is used for debugging, removed.

>> Source/WebCore/platform/sql/SQLiteDatabase.cpp:129
>> +            LOG(SQLDatabase, "SQLite database checkpoint is blocked");
> 
> Shouldn't this be a LOG_ERROR?

The statement sets column 0 as 1 for SQLITE_BUSY case, which means it's blocked by other active database connections now and will be executed later (https://www.sqlite.org/c3ref/wal_checkpoint_v2.html).
Since it's scheduled, I think it's different from an error.

>> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-1.html:1
>> +<script>
> 
> <!DOCTYPE html>

Okay.

>> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-1.html:14
>> +    for (let i = 0; i < 100; i ++) {
> 
> no space between i and ++.

Okay.

>> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-2.html:1
>> +<script>
> 
> <!DOCTYPE html>

Okay.

>> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:62
>> +    RetainPtr<IndexedDBFileSizeMessageHandler> handler = adoptNS([[IndexedDBFileSizeMessageHandler alloc] init]);
> 
> auto

Okay.

>> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:63
>> +    RetainPtr<WKWebViewConfiguration> configuration = adoptNS([[WKWebViewConfiguration alloc] init]);
> 
> auto

Okay.

>> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:69
>> +    RetainPtr<_WKWebsiteDataStoreConfiguration> websiteDataStoreConfiguration = adoptNS([[_WKWebsiteDataStoreConfiguration alloc] init]);
> 
> auto

Okay.

>> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:73
>> +    RetainPtr<NSSet> types = adoptNS([[NSSet alloc] initWithObjects:WKWebsiteDataTypeIndexedDBDatabases, nil]);
> 
> auto

Okay.

>> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:82
>> +    RetainPtr<WKWebView> webView = adoptNS([[WKWebView alloc] initWithFrame:NSMakeRect(0, 0, 800, 600) configuration:configuration.get()]);
> 
> auto

Okay.

>> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:94
>> +    // Terminate network process to keep WAL on disk
> 
> Missing period at the end.

Added.
Comment 9 Sihui Liu 2018-11-06 10:32:15 PST
Created attachment 353974 [details]
Patch
Comment 10 EWS Watchlist 2018-11-06 10:35:31 PST
Attachment 353974 [details] did not pass style-queue:


ERROR: Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:3595:  developmentRegion is not en.  [xcodeproj/settings] [5]
Total errors found: 1 in 7 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 11 Chris Dumez 2018-11-06 10:51:50 PST
Comment on attachment 353974 [details]
Patch

r=me
Comment 12 WebKit Commit Bot 2018-11-06 12:57:03 PST
Comment on attachment 353974 [details]
Patch

Clearing flags on attachment: 353974

Committed r237882: <https://trac.webkit.org/changeset/237882>
Comment 13 WebKit Commit Bot 2018-11-06 12:57:04 PST
All reviewed patches have been landed.  Closing bug.
Comment 14 Mohammad Shraim 2018-11-07 07:58:56 PST
Is this Fix will be available next iOS release??

Thanks
Mohd Shraim
Comment 15 juliencdev 2019-09-01 22:10:30 PDT
Hi there,
Which version of WebKit was this patch applied to? We are using Ionic for an iOS app and we experience the same issue. We are building with Xcode 10.3 which, I assume, uses the latest WebKit framework.
Regards,
Julien.
Comment 16 Sihui Liu 2019-09-03 11:03:22 PDT
(In reply to juliencdev from comment #15)
> Hi there,
> Which version of WebKit was this patch applied to? We are using Ionic for an
> iOS app and we experience the same issue. We are building with Xcode 10.3
> which, I assume, uses the latest WebKit framework.
> Regards,
> Julien.

Hi,

this fix is shipped in iOS 12.2+ and macOS 10.14.4+.

If you still sees this issue, please file a new bug with reproducible steps for us to look into.
Comment 17 juliencdev 2019-09-04 15:44:54 PDT
Thanks for your reply.

Julien.
Comment 18 Lee Robertson 2020-04-17 18:14:39 PDT
This is still an issue in iOS 13.4.1 and Safari 13
Comment 19 Lincoln Baxter, III 2020-05-11 09:14:19 PDT
Also experiencing this bug in Safari on Desktop MacOS 10.15.4, and iOS Safari (in XCode Simulators) 13.4.1.

Refreshing the page seems to sometimes create a new IndexedDB instance when one already exists. The instance has exactly the same name, and seems to have the same data as the original DB.

Continuing to refresh will continue to create copies. Eventually the OS, iOS or MacOS, will prompt to allow a storage increase. This continues to MASSIVE sizes, see attachment.
Comment 20 Maximilian D. 2020-09-09 00:08:36 PDT
We have the problem for our PWA sales app also. 
We have ~40MB data that we need, but we have users on iOS that get a warning that more than 1,1 GB will be used. 

On our Chrome desktop apps we can see that in the folder: https_xxx.xxx.de_0.indexeddb.blob / 1 a lot of subfolders are shown that have the data from different points in time. 
If we delete the old ones the app will work as before and the storage information in the dev tools are updated correctly. 

Two questions:
1.) is it possible to reopen the ticket?
2.) does someone know a workaround or a way to delete the old data out of JS?