Bug 191294

Summary:

IndexedDB: WAL file keeps growing

Product:

WebKit

Reporter:

Sihui Liu <sihui_liu>

Component:

New Bugs

Assignee:

Sihui Liu <sihui_liu>

Status:

RESOLVED FIXED

Severity:

Normal

CC:

beidson, cdumez, commit-queue, ews-watchlist, joeyparrish, juliencdev, leechanr21, lincolnbaxter, matt, maximilian.doepp1, moh.shraim, pfifield, pixelworld

Priority:

Keywords:

InRadar

Version:

WebKit Nightly Build

Hardware:

Unspecified

OS:

Unspecified

Attachments:

Description	Flags
Patch	none
Patch	none

Description Sihui Liu 2018-11-05 18:59:42 PST

https://github.com/firebase/firebase-js-sdk/issues/947

Size of WAL files of iOS apps could keep growing every time we launch the app.

Comment 1 Sihui Liu 2018-11-05 19:00:51 PST

<rdar://problem/41333493>

Comment 2 Sihui Liu 2018-11-05 19:22:22 PST

Created attachment 353942 [details]
Patch

Comment 3 EWS Watchlist 2018-11-05 19:25:39 PST

Attachment 353942 [details] did not pass style-queue:


ERROR: Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:3595:  developmentRegion is not en.  [xcodeproj/settings] [5]
Total errors found: 1 in 7 files


If any of these errors are false positives, please file a bug against check-webkit-style.

Comment 4 Chris Dumez 2018-11-05 20:34:59 PST

*** Bug 178204 has been marked as a duplicate of this bug. ***

Comment 5 Pete Fifield 2018-11-06 08:15:16 PST

We have an app that uses IndexedDB and am experiencing a similar problem.  Since the bug has been recognized, do you have a workaround while we wait for the iOS phones to update?

Comment 6 Sihui Liu 2018-11-06 09:05:32 PST

(In reply to Pete Fifield from comment #5)
> We have an app that uses IndexedDB and am experiencing a similar problem. 
> Since the bug has been recognized, do you have a workaround while we wait
> for the iOS phones to update?

There (In reply to Pete Fifield from comment #5)
> We have an app that uses IndexedDB and am experiencing a similar problem. 
> Since the bug has been recognized, do you have a workaround while we wait
> for the iOS phones to update?

There are two things I can think of:
(1) Regularly delete your IndexedDB files, via API like WKWebsiteDataStore removeDataOfTypes: this would clear both .sqlite3 and .sqlite3-wal files.
(2) Open a database and close it (e.g. by navigating to another page that doesn't use that database) before the app gets killed. This would make sure sqlite3_open() and sqlite3_close() are called() (https://www.sqlite.org/wal.html, item4), which may truncate or delete the WAL file.

If you can reload the content of IDB, I would recommend (1), which has more definitive result.

Comment 7 Chris Dumez 2018-11-06 09:21:00 PST

Comment on attachment 353942 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=353942&action=review

> Source/WebCore/platform/sql/SQLiteDatabase.cpp:125
> +    SQLiteStatement cpStatement(*this, "PRAGMA wal_checkpoint(TRUNCATE)");

"PRAGMA wal_checkpoint(TRUNCATE)"_s

Also, please find a better name for the variable.

> Source/WebCore/platform/sql/SQLiteDatabase.cpp:126
> +    auto res = cpStatement.prepareAndStep();

Please avoid abbreviation in variable names.

Also, int is actually shorter than auto.

> Source/WebCore/platform/sql/SQLiteDatabase.cpp:129
> +            LOG(SQLDatabase, "SQLite database checkpoint is blocked");

Shouldn't this be a LOG_ERROR?

> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-1.html:1
> +<script>

<!DOCTYPE html>

> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-1.html:14
> +    for (let i = 0; i < 100; i ++) {

no space between i and ++.

> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-2.html:1
> +<script>

<!DOCTYPE html>

> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:62
> +    RetainPtr<IndexedDBFileSizeMessageHandler> handler = adoptNS([[IndexedDBFileSizeMessageHandler alloc] init]);

auto

> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:63
> +    RetainPtr<WKWebViewConfiguration> configuration = adoptNS([[WKWebViewConfiguration alloc] init]);

auto

> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:69
> +    RetainPtr<_WKWebsiteDataStoreConfiguration> websiteDataStoreConfiguration = adoptNS([[_WKWebsiteDataStoreConfiguration alloc] init]);

auto

> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:73
> +    RetainPtr<NSSet> types = adoptNS([[NSSet alloc] initWithObjects:WKWebsiteDataTypeIndexedDBDatabases, nil]);

auto

> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:82
> +    RetainPtr<WKWebView> webView = adoptNS([[WKWebView alloc] initWithFrame:NSMakeRect(0, 0, 800, 600) configuration:configuration.get()]);

auto

> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:94
> +    // Terminate network process to keep WAL on disk

Missing period at the end.

Comment 8 Sihui Liu 2018-11-06 10:30:20 PST

Comment on attachment 353942 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=353942&action=review

>> Source/WebCore/platform/sql/SQLiteDatabase.cpp:125
>> +    SQLiteStatement cpStatement(*this, "PRAGMA wal_checkpoint(TRUNCATE)");
> 
> "PRAGMA wal_checkpoint(TRUNCATE)"_s
> 
> Also, please find a better name for the variable.

Okay.

>> Source/WebCore/platform/sql/SQLiteDatabase.cpp:126
>> +    auto res = cpStatement.prepareAndStep();
> 
> Please avoid abbreviation in variable names.
> 
> Also, int is actually shorter than auto.

This variable is used for debugging, removed.

>> Source/WebCore/platform/sql/SQLiteDatabase.cpp:129
>> +            LOG(SQLDatabase, "SQLite database checkpoint is blocked");
> 
> Shouldn't this be a LOG_ERROR?

The statement sets column 0 as 1 for SQLITE_BUSY case, which means it's blocked by other active database connections now and will be executed later (https://www.sqlite.org/c3ref/wal_checkpoint_v2.html).
Since it's scheduled, I think it's different from an error.

>> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-1.html:1
>> +<script>
> 
> <!DOCTYPE html>

Okay.

>> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-1.html:14
>> +    for (let i = 0; i < 100; i ++) {
> 
> no space between i and ++.

Okay.

>> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize-2.html:1
>> +<script>
> 
> <!DOCTYPE html>

Okay.

>> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:62
>> +    RetainPtr<IndexedDBFileSizeMessageHandler> handler = adoptNS([[IndexedDBFileSizeMessageHandler alloc] init]);
> 
> auto

Okay.

>> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:63
>> +    RetainPtr<WKWebViewConfiguration> configuration = adoptNS([[WKWebViewConfiguration alloc] init]);
> 
> auto

Okay.

>> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:69
>> +    RetainPtr<_WKWebsiteDataStoreConfiguration> websiteDataStoreConfiguration = adoptNS([[_WKWebsiteDataStoreConfiguration alloc] init]);
> 
> auto

Okay.

>> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:73
>> +    RetainPtr<NSSet> types = adoptNS([[NSSet alloc] initWithObjects:WKWebsiteDataTypeIndexedDBDatabases, nil]);
> 
> auto

Okay.

>> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:82
>> +    RetainPtr<WKWebView> webView = adoptNS([[WKWebView alloc] initWithFrame:NSMakeRect(0, 0, 800, 600) configuration:configuration.get()]);
> 
> auto

Okay.

>> Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBTempFileSize.mm:94
>> +    // Terminate network process to keep WAL on disk
> 
> Missing period at the end.

Added.

Comment 9 Sihui Liu 2018-11-06 10:32:15 PST

Created attachment 353974 [details]
Patch

Comment 10 EWS Watchlist 2018-11-06 10:35:31 PST

Attachment 353974 [details] did not pass style-queue:


ERROR: Tools/TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:3595:  developmentRegion is not en.  [xcodeproj/settings] [5]
Total errors found: 1 in 7 files


If any of these errors are false positives, please file a bug against check-webkit-style.

Comment 11 Chris Dumez 2018-11-06 10:51:50 PST

Comment on attachment 353974 [details]
Patch

r=me

Comment 12 WebKit Commit Bot 2018-11-06 12:57:03 PST

Comment on attachment 353974 [details]
Patch

Clearing flags on attachment: 353974

Committed r237882: <https://trac.webkit.org/changeset/237882>

Comment 13 WebKit Commit Bot 2018-11-06 12:57:04 PST

All reviewed patches have been landed.  Closing bug.

Comment 14 Mohammad Shraim 2018-11-07 07:58:56 PST

Is this Fix will be available next iOS release??

Thanks
Mohd Shraim

Comment 15 juliencdev 2019-09-01 22:10:30 PDT

Hi there,
Which version of WebKit was this patch applied to? We are using Ionic for an iOS app and we experience the same issue. We are building with Xcode 10.3 which, I assume, uses the latest WebKit framework.
Regards,
Julien.

Comment 16 Sihui Liu 2019-09-03 11:03:22 PDT

(In reply to juliencdev from comment #15)
> Hi there,
> Which version of WebKit was this patch applied to? We are using Ionic for an
> iOS app and we experience the same issue. We are building with Xcode 10.3
> which, I assume, uses the latest WebKit framework.
> Regards,
> Julien.

Hi,

this fix is shipped in iOS 12.2+ and macOS 10.14.4+.

If you still sees this issue, please file a new bug with reproducible steps for us to look into.

Comment 17 juliencdev 2019-09-04 15:44:54 PDT

Thanks for your reply.

Julien.

Comment 18 Lee Robertson 2020-04-17 18:14:39 PDT

This is still an issue in iOS 13.4.1 and Safari 13

Comment 19 Lincoln Baxter, III 2020-05-11 09:14:19 PDT

Also experiencing this bug in Safari on Desktop MacOS 10.15.4, and iOS Safari (in XCode Simulators) 13.4.1.

Refreshing the page seems to sometimes create a new IndexedDB instance when one already exists. The instance has exactly the same name, and seems to have the same data as the original DB.

Continuing to refresh will continue to create copies. Eventually the OS, iOS or MacOS, will prompt to allow a storage increase. This continues to MASSIVE sizes, see attachment.

Comment 20 Maximilian D. 2020-09-09 00:08:36 PDT

We have the problem for our PWA sales app also. 
We have ~40MB data that we need, but we have users on iOS that get a warning that more than 1,1 GB will be used. 

On our Chrome desktop apps we can see that in the folder: https_xxx.xxx.de_0.indexeddb.blob / 1 a lot of subfolders are shown that have the data from different points in time. 
If we delete the old ones the app will work as before and the storage information in the dev tools are updated correctly. 

Two questions:
1.) is it possible to reopen the ticket?
2.) does someone know a workaround or a way to delete the old data out of JS?

Comment 21 Sihui Liu 2022-02-01 23:28:54 PST

(In reply to Maximilian D. from comment #20)
> We have the problem for our PWA sales app also. 
> We have ~40MB data that we need, but we have users on iOS that get a warning
> that more than 1,1 GB will be used. 
> 
> On our Chrome desktop apps we can see that in the folder:
> https_xxx.xxx.de_0.indexeddb.blob / 1 a lot of subfolders are shown that
> have the data from different points in time. 
> If we delete the old ones the app will work as before and the storage
> information in the dev tools are updated correctly. 
> 
> Two questions:
> 1.) is it possible to reopen the ticket?
> 2.) does someone know a workaround or a way to delete the old data out of JS?

Hi Maximilian,

If you still see the issue on a recent iOS or macOS build, please file a new bug with reproducible steps (also test files if possible), and add me to CC list. It can be a different issue from this bug and we need to investigate again (e.g. we've tried another fix at https://bugs.webkit.org/show_bug.cgi?id=202137.).