Bug 266559 – Safari periodically erasing LocalStorage and IndexedDB for all websites

• Component: Website Storage
• Status: RESOLVED FIXED

[Jeff Johnson]

Safari 17.2 (18617.1.17.11.11, 18617) on macOS 13.6.3 (22G436)

For the past several days at least, Safari has been periodically, seemingly randomly erasing LocalStorage and IndexedDB for all websites simultaneously. Some visible consequences of this is that https://bsky.app is logged out, and the display settings on https://twitter.com are lost. This has been happening around once a day, sometimes twice a day. I have no idea what's causing it.

Via Time Machine backups, I discovered that it happened this morning between 9:23am and 10:47am. I actually did a full diff of my ~/Library folder backups (FileMerge with Full Disk Access is nice), but there weren't a lot of differences, and the only things that seemed interesting and relevant were in the ~/Library/Containers/com.apple.Safari/Data/Library/WebKit/WebsiteData/Default/ folder. The LocalStorage and IndexedDB folders were completely gone, though it looks like the ServiceWorkers folders survived.

Is there any good way for me to disagnose this issue?

[Jeff Johnson]

I think it may have something to do with the web inspector.

I've reproduced the issue many times today, but I still can't figure out precise steps.

[Jeff Johnson]

Created attachment 469107 [details]
Xcode project of Safari content blocker for third-party cookie

[Jeff Johnson]

I found 100% reliable steps to reproduce, but they're complicated. It turns out that the issue seems to involve both the web inspector and a Safari content blocker.

Some background: I disabled "Prevent cross-site tracking" because of data loss due to the 7-day cap on script-writeable storage. However, I still wanted third-party cookie blocking, so I wrote a little Safari content blocker with one rule:

        "action": {
            "type": "block-cookies"
        },
        "trigger": {
            "load-type": ["third-party"],
            "url-filter": ".*"
        }

Ironically, this workaround is also causing data loss. I find it very frustrating that I can't use Safari without it automatically erasing data in some way.

Steps to reproduce:
1) Download and unzip the attached sample Xcode project StopTheCookies
2) Add the file DEVELOPMENT_TEAM.xcconfig to the folder with the contents DEVELOPMENT_TEAM = [YourTeamID]
3) Build and run the Xcode project
4) Open Safari Settings
4) Enable StopTheCookies in Extensions Settings, and make sure to Allow in Private Browsing
5) Disable Prevent cross-site tracking in Privacy Settings
6) Set Safari opens with: A new private window in General Settings
7) Open http://www.sharonminsuk.com/code/storage-test.html in a non-private window
8) Enter "testing" and press the Store it button
9) Quit Safari
10) Open Safari
11) Open https://www.cnn.com in the private window
12) Open a new non-private window
13) Open http://www.sharonminsuk.com/code/storage-test.html in the non-private window
14) Close the non-private window
15) Open the web inspector in the CNN tab
16) Close the web inspector
17) Close the CNN private window
18) Open a new non-private window
19) Open http://www.sharonminsuk.com/code/storage-test.html in the non-private window

Expected results: Local storage: "testing"

Actual results: Local storage: empty

[Jeff Johnson]

Update:

After my last comment, I disabled my Safari content blocker and re-enabled "Prevent cross-site tracking". But now the issue has occurred again, so I don't think these two mentioned factors are essential to the bug.

However, I continue to think that the web inspector is involved, because I did in fact use the web inspector afterward.

[Alexey Proskuryakov]

This seems like an issue that could benefit form having a sysdiagnose collected right after reproducing, so that we could correlate logging with a timestamp (not necessary if we can reproduce, but sounds like it's not a given that we will).

I wouldn't upload a sysdiagnose here, but filing a duplicate via Feedback Assistant would be one way to make it available.

[Radar WebKit Bug Importer]

<rdar://problem/119818267>

[Jeff Johnson]

(In reply to Alexey Proskuryakov from comment #5)
> This seems like an issue that could benefit form having a sysdiagnose
> collected right after reproducing, so that we could correlate logging with a
> timestamp (not necessary if we can reproduce, but sounds like it's not a
> given that we will).
> 
> I wouldn't upload a sysdiagnose here, but filing a duplicate via Feedback
> Assistant would be one way to make it available.

I don't really have a timestamp. Except for with my steps to reproduce above, it's unclear exactly when the issue occurs, and I only discover it sometime after the fact.

[Jeff Johnson]

I've discovered semi-reliable steps to reproduce. The bug doesn't happen every time, but it happens a lot of the time with these steps. Here's a screen recording:

https://youtu.be/VMQUZwJB4x8

I'm using http://www.sharonminsuk.com/code/storage-test.html and https://javascript.info/localstorage for testing local storage.

No content blockers or extensions are enabled. Prevent cross-site tracking is enabled.

Safari is set to open with a new private window, but on further testing, this doesn't appear to be essential.

I now believe this is a web inspector bug. I've never been able to reproduce without using the web inspector. Indeed, I believe this is why the bug has affected me so much, because I'm a very frequent user of the web inspector.

[John Wilander]

(In reply to Jeff Johnson from comment #8)
> I've discovered semi-reliable steps to reproduce. The bug doesn't happen
> every time, but it happens a lot of the time with these steps. Here's a
> screen recording:
> 
> https://youtu.be/VMQUZwJB4x8
> 
> I'm using http://www.sharonminsuk.com/code/storage-test.html and
> https://javascript.info/localstorage for testing local storage.
> 
> No content blockers or extensions are enabled. Prevent cross-site tracking
> is enabled.
> 
> Safari is set to open with a new private window, but on further testing,
> this doesn't appear to be essential.
> 
> I now believe this is a web inspector bug. I've never been able to reproduce
> without using the web inspector. Indeed, I believe this is why the bug has
> affected me so much, because I'm a very frequent user of the web inspector.

Thank you for narrowing this down. We have not heard other reports of this issue which would typically be the case if it were affecting all users.

Private Browsing Mode is ephemeral so storage will only exist in-memory and go away at browser restart or if the network process were to crash. Are you using Private Browsing Mode (I see you comment on new private window)? Does it reproduce in regular browsing mode?

[Jeff Johnson]

(In reply to John Wilander from comment #9)
> (In reply to Jeff Johnson from comment #8)
> > I've discovered semi-reliable steps to reproduce. The bug doesn't happen
> > every time, but it happens a lot of the time with these steps. Here's a
> > screen recording:
> Private Browsing Mode is ephemeral so storage will only exist in-memory and
> go away at browser restart or if the network process were to crash. Are you
> using Private Browsing Mode (I see you comment on new private window)? Does
> it reproduce in regular browsing mode?

It's all in the screen recording. ;-)

The bug reproduces in regular browsing mode. I have Safari set to open with a new private window, but as I said, that's nonessential. The steps to reproduce, in text:

1) Open http://www.sharonminsuk.com/code/storage-test.html in a non-private window
2) Store "testing" in Local storage
3) Open https://javascript.info/localstorage in a non-private window
4) Press the localStorage.setItem('test', 1) button
5) Quit Safari
6) Launch Safari
7) Open https://www.cnn.com (in a private or non-private window, doesn't matter)
8) Open a new non-private window
9) Open https://javascript.info/localstorage in the new window
10) Run the localStorage.getItem('test') and confirm that the value is 1
11) Close the https://javascript.info/localstorage window
12) Open the web inspector console in the cnn.com window
13) Scroll the page slowly down to the bottom and back up to the top again
14) Close the window
15) Open a new non-private window
16) Open http://www.sharonminsuk.com/code/storage-test.html

Expected results: "testing"

Actual results: nothing currently stored. Also, https://javascript.info/localstorage now returns null.

[Charlie Wolfe]

(In reply to Jeff Johnson from comment #10)
> (In reply to John Wilander from comment #9)
> > (In reply to Jeff Johnson from comment #8)
> > > I've discovered semi-reliable steps to reproduce. The bug doesn't happen
> > > every time, but it happens a lot of the time with these steps. Here's a
> > > screen recording:
> > Private Browsing Mode is ephemeral so storage will only exist in-memory and
> > go away at browser restart or if the network process were to crash. Are you
> > using Private Browsing Mode (I see you comment on new private window)? Does
> > it reproduce in regular browsing mode?
> 
> It's all in the screen recording. ;-)
> 
> The bug reproduces in regular browsing mode. I have Safari set to open with
> a new private window, but as I said, that's nonessential. The steps to
> reproduce, in text:
> 
> 1) Open http://www.sharonminsuk.com/code/storage-test.html in a non-private
> window
> 2) Store "testing" in Local storage
> 3) Open https://javascript.info/localstorage in a non-private window
> 4) Press the localStorage.setItem('test', 1) button
> 5) Quit Safari
> 6) Launch Safari
> 7) Open https://www.cnn.com (in a private or non-private window, doesn't
> matter)
> 8) Open a new non-private window
> 9) Open https://javascript.info/localstorage in the new window
> 10) Run the localStorage.getItem('test') and confirm that the value is 1
> 11) Close the https://javascript.info/localstorage window
> 12) Open the web inspector console in the cnn.com window
> 13) Scroll the page slowly down to the bottom and back up to the top again
> 14) Close the window
> 15) Open a new non-private window
> 16) Open http://www.sharonminsuk.com/code/storage-test.html
> 
> Expected results: "testing"
> 
> Actual results: nothing currently stored. Also,
> https://javascript.info/localstorage now returns null.

I have not been able to reproduce this issue after trying these steps several times. It would be helpful if you provided a sysdiagnose taken after you have reproduced the issue, even if you cannot give an exact timestamp.

[Jeff Johnson]

Created attachment 469287 [details]
Safari preferences

I just reproduced on another machine, a Mac mini running macOS 14.2.1 and Safari 17.2.1.

I literally never upload sysdiagnose for anything, because I feel that they are too intrusive, but if it helps, I've attached the Safari preferences from the Mac mini.

[Alexey Proskuryakov]

Would you be comfortable with providing `log show` output that covers the period of time when this happens? That's the part of sysdiagnose that's most likely to be helpful here.

(probably via feedback assistant or direct email, not a Bugzilla attachment)

[Jeff Johnson]

I suspect it may depend on what gets displayed on the page at the time, perhaps the ads, because I go through periods of being able to reproduce the issue and periods of not being able to reproduce.

[Jeff Johnson]

I reproduced the bug while watching a subfolder of ~/Library/Containers/com.apple.Safari/Data/Library/WebKit/WebsiteData/Default/ in an open Finder window to see when the LocalStorage folder disappears, and it actually happens immediately when I open the web inspector in step 12, so steps 13-16 are unnecessary.

Watching the Console log at the same time, I see these messages:

GeneralStorageDirectory for session 1 is already in use by session 2

NetworkStorageManager::NetworkStorageManager path for session 1 is already in use by session 2

The message is apparently from https://bugs.webkit.org/show_bug.cgi?id=236844 Add assertion that no two sessions share the same general storage directory.

[Ahmad Saleem]

NOTE - I am not sure this is exact issue or not but on 'https://www.reddit.com/r/Safari/', I have seen similar articles / posts, where people said that they were logged out of websites and local storage not working etc.

e.g., https://www.reddit.com/r/Safari/comments/18f24f3/safari_frequently_dumping_all_cookies/

and 

https://www.reddit.com/r/Safari/comments/185cra6/safari_nuking_cookies_and_local_storage_randomly/

In above, I can see Jeff's response with UnderpassAppCompany account as well.

So it might be bit wide spread but never got reported on 'bugs.webkit.org'.

[John Wilander]

Jeff, since we have not been able to reproduce, I'm wondering if you have some setup that might matter.
- SIP disabled?
- Something that has been granted access to Safari's files?
- Some kind of anti malware software that might be cleaning things?
- Some web extensions granted access to all websites?
- Running local builds of WebKit?

[Jeff Johnson]

(In reply to John Wilander from comment #17)
> Jeff, since we have not been able to reproduce, I'm wondering if you have
> some setup that might matter.
> - SIP disabled?

No.

> - Something that has been granted access to Safari's files?

No.

> - Some kind of anti malware software that might be cleaning things?

No.

> - Some web extensions granted access to all websites?

It happens even with extensions disabled.

> - Running local builds of WebKit?

No.

Moreover, I reproduced it on two different Macs that are independent of each other. The Mac mini is a mostly stock machine that I use exclusively for testing.

[Jeff Johnson]

It may help to enable "Preserve Log" in the web inspector console. This seems to increase the frequency of the issue.

Please continue testing, because I go through periods of not being able to reproduce, and then periods of being able to reproduce. Like I said before, I can see the files disappear in Finder literally simultaneously with opening the web inspector, so it's not another unrelated app deleting the files coincidentally. It's definitely Safari.

[Jeff Johnson]

I've also been able to reproduce the bug several times with the MiniBrowser app in the latest WebKit Build Archive: https://webkit.org/build-archives/#mac-sonoma-x86_64%20arm64

There are many failures to reproduce, but it does happen after a number of tries.

For testing, I used about:blank to Set Default URL to Current URL in MiniBrowser Settings.

[Jeff Johnson]

Created attachment 469351 [details]
com.apple.WebKit.Networking Console log

I think I found the bug in the Console log from MiniBrowser. It says, "BUG IN CLIENT OF libsqlite3.dylib: database integrity compromised by API violation: vnode unlinked while in use". That's when the files get deleted.

Attached are the Console log messages from the com.apple.WebKit.Networking process.

[Charlie Wolfe]

(In reply to Jeff Johnson from comment #21)
> Created attachment 469351 [details]
> com.apple.WebKit.Networking Console log
> 
> I think I found the bug in the Console log from MiniBrowser. It says, "BUG
> IN CLIENT OF libsqlite3.dylib: database integrity compromised by API
> violation: vnode unlinked while in use". That's when the files get deleted.
> 
> Attached are the Console log messages from the com.apple.WebKit.Networking
> process.

Thank you! Those logs certainly appear to be relevant.

I believe I was also just able to reproduce this issue, though it is very flaky for me. It seems that when web inspector is opened, there's a possibility that the network process will delete storage from several origins, which is clearly incorrect.

When the bug reproduces, I see that `OriginStorageManager::deleteData()` is being invoked on origins where it shouldn't be. I will continue to investigate this.

[Charlie Wolfe]

Pull request: https://github.com/WebKit/WebKit/pull/22635

[Alexey Proskuryakov]

> It says, "BUG IN CLIENT OF libsqlite3.dylib: database integrity compromised by API violation: vnode unlinked while in use".

We should really look into this too. Having any code paths where we unlink open database files is risky indeed.

[EWS]

Committed 272951@main (cc8a261932c7): <https://commits.webkit.org/272951@main>

Reviewed commits have been landed. Closing PR #22635 and removing active labels.

[Xiao]

The issue is not resolved after PR #22635 . I haven't find an exact way to reproduce but it's still happening on my mac everyday with Safari 17.5, and there also multiple reports on reddit as well (https://www.reddit.com/r/MacOS/comments/17m0i4p/comment/ku77kun).
A few suspicious actions:
- Open & Close Page Inspector
- Enable multiple user profiles
- Pin more than 5 tabs

[Jeff Johnson]

(In reply to Xiao from comment #26)
> The issue is not resolved after PR #22635 . I haven't find an exact way to
> reproduce but it's still happening on my mac everyday with Safari 17.5, and
> there also multiple reports on reddit as well
> (https://www.reddit.com/r/MacOS/comments/17m0i4p/comment/ku77kun).
> A few suspicious actions:
> - Open & Close Page Inspector
> - Enable multiple user profiles
> - Pin more than 5 tabs

FWIW the issue has been fixed for me, and I haven't experienced it since Safari 17.4 was released.

There may be a different issue, though, which I haven't experienced.