Bug 31791 – localStorage quota exceeded

• Component: New Bugs
• Status: RESOLVED WORKSFORME

[William J. Edney]

Folks (esp. Jeremy ;-) ) -

I'm having trouble with having localStorage exceed its quota storage in the latest versions of Webkit.

I am storing a total of two key/value pairs into localStorage, let's call them 'foo' and 'bar' (I can give you the actual key names, if you want).

'foo' has a data block with a total of 2585125 characters, so slightly more than 2.5M bytes.
'bar' has a data block with a total of 90482 characters, so slightly more than 90K bytes.

So let's call the total around 2.6MB, well below the default storage quota of 5MB.

I'm noticing this on Webkit nightly build 51280 / Mac.

I'm also noticing this on Chrome/Mac and Chrome/Windows, both at version 4.0.249.4 (I'm on the 'developer channel').

I'm *not* noticing this bug on Safari/Mac 4.0.4 and Safari/Windows 4.0.4, which leads me to think that it's a regression, maybe as a result of fixing:

https://bugs.webkit.org/show_bug.cgi?id=31451

???

Not sure. In any case, 2.6MB is nowhere close to the default quota of 5MB, so it leads me to think that something strange is going on here. I noticed by poking around inside of the SQLite file that Webkit generates an index on this data. Could this be a problem where you don't have many keys (in this case a total of 2), but the data being stored is large? I'm not a DB guy, so I don't know.

Thanks in advance!

Cheers,

- Bill

[Jeremy Orlow]

Strings in WebKit are UTF16 so your string is likely seeing near a 2x blowup factor due to that.

That said, I just realized that I never actually investigated to see what encoding SQLite is using.  Can you see what size the LocalStorage database is when you're hitting that?

(I'm on vacation this week, so I may be less responsive than normal, btw.)

[William J. Edney]

Jeremy -

Thanks for the quick reply (and for responding on vacation!).

The database size on the Mac version of my testing is 5369856 bytes before throwing the error.

It is throwing the error after loading 'foo' and trying to load 'bar'. If your suspicion about UTF-16 is correct, the limit of the number of characters would be 2,560,000 for 5MB of storage. Given my numbers below, that would be correct. 'foo' could completely load, but 'bar' would exceed the quota.

So, maybe a more accurate statement would be that the default quota will be '5 million characters' :-).

That would imply storing up to 10MB, if the encoding was to stay at UTF-16.

Cheers,

- Bill

[Jeremy Orlow]

Saying we allow 5mb seems accurate to me...it's just that it might not be obvious that 5mb == 2500000 chars.

Are you suggesting that we raise the default limit?  If so, that might be a good thing to bring up on webkit dev.

[William J. Edney]

Jeremy -

I guess its open to interpretation.

I assumed, as probably would most developers who are doing stuff at the JS level, that '5 MB' would mean '5 million characters', because they would assume a UTF-8 encoding. That very well be a poor assumption, but there it is.

I guess it would be a question for the spec committee. I would like to see one of two solutions:

1. The spec is altered to specify its units as 'characters' rather than bytes.
2. The spec is altered so that I, from a JS interface, can specify the encoding. In this case, I could specify UTF-8 and my data would fit into the default quota.

#2 is preferable, IMO, because I know folks are going to be wanting to eventually store binary data, etc. in the local store, so it wouldn't be wise to put 'characters' into the spec.

Thoughts?

Thanks for the feedback :-)

Cheers,

- Bill

[Jeremy Orlow]

The spec's mentioning of 5mb is really just an example.  There's nothing too special about 5mb.  I guess it could be a bit confusing if that means 5 million characters in other browsers and not WebKit ones.  Can you confirm that other browsers that are implementing quotas typically allow 5 million?

But my point is that the spec probably won't change.  And over time, it's going to only become less transparent to app developers where their space is going.  And I think that's unavoidable, so we should just embrace it now...but I don't have to deal with this stuff on a day to day basis either.

As for encoding: I'm pretty sure Ian would say this is an implementational detail and should not be spec'ed.  But you could always bring it up on whatwg.

I think the next step (if any) is for you to make a case on WebKit-dev that we should either: use UTF-8 encoding in the DB (and use that in the accounting) o raise the WebKit default quota.  Both would be easier to sell if other browsers were allowing for 5 million chars before hitting the limits.

[William J. Edney]

Jeremy -

I can confirm positively that I'm running the same tests in FF 3.5/3.6, IE8 and Safari 4 Win & Mac and those browsers are succeeding in my tests. I haven't altered the quota in any of them, although I do know for a fact that IE8 allows a 10MB / domain quota.

This leads me to believe that at least FF uses UTF-8 encoding. Not sure about IE8, again because of their 10MB quota, they could be storing things in something other than UTF-8 and my tests would still pass.

It's curious that Safari 4 passes the test, being a Webkit-based browser. As I noted in the initial report, Webkit/Mac 51280 does not, which led me to think it was a regression of some sort.

Given all of this, I would guess that maybe specifying UTF-8 in the spec would be the best route. In the meantime, I'll pop up on webkit-dev and make my case there for UTF-8 encoding, referencing this bug.

Thanks again!

Cheers,

- Bill

[Jeremy Orlow]

Quota support was added in only a few months ago, which is why Safari 4 didn't have this.  The next version of safari (probably) will.

I'm fine with raising the default.  Or maybe Chrome should override it.  I'll look forward to the webkit-dev thread.

[William J. Edney]

Jeremy -

Ah, so that's why Safari 4.X is not failing.

I'm totally cool with you raising the quota (of course ;-)). Given that IE8 allows 10MB / domain, I don't think it's a problem. In fact, back in October 2008, I filed a Mozilla bug to ask them to raise their quota to 10MB to match IE8:

https://bugzilla.mozilla.org/show_bug.cgi?id=461684

I guess I've now gotten with the right person on the Webkit team to ask the same :-).

Cheers,

- Bill

[Jeremy Orlow]

Please ask on webkit-dev.  If they agree, we can raise it there.  If not, I'll see if Chromium wants to raise their limit.

I'm generally in favor of it, but this is the type of thing we should discuss as a community first.

[Anne van Kesteren]

The status quo here seems okay these days.