Bug 54384 – Use UTF-8 instead of UTF-16LE for Database Storage

• Status: UNCONFIRMED
• Component: WebCore Misc.

[Christian]

Currently all Strings seem to be stored as UTF-16le in the database. Is it possible to change that in the near future?

I am currently evaluating the development of mobile apps. Because of the large amount of operating systems on mobile devices, a web-based application seems to be an interesting alternative to reach a larger audience. However, since the data is stored in UTF-16le, the space required for storing data is about 3 times higher, then it would require to store data using the native approach. Example:

parsed XML: 64KB
Android database (native): 124KB
WebView database (web-based): 358KB 

On a PC the amount of required space wouldn't be a factor. However, on mobile devices the storage space is limited. Therefore it would be a nice, if it is possible to store the data using UTF-8. Are there any plans concerning that problem?

Thanks for the answer.

Greetings.

Christian

[Patrick R. Gansterer]

I don't see a big problem when implementing this. UTF16 is the native WebKit string representation, so no en/decoding is required, but making it a compile time option like USE(UTF8_DATABASE_STORAGE) might be an option. Are you willing to create a patch?

> the space required for storing data is about 3 times higher
What is your native format? The difference between UTF16 and UTF8 or Latin1 is only factor 2!

[Christian]

> > the space required for storing data is about 3 times higher
> What is your native format? The difference between UTF16 and UTF8 or Latin1 is only factor 2!

The additional space should come from the indexes, which the Web SQL database seems to add for every primary key.

Example:
A table DATATABLE with primary key "primary" creates the following indexes
- on Android database (native - UTF8):
    ==> sqlite_autoindex_DATATABLE_1

- on WebView database (web-based - UTF-16le):
    ==> sqlite_autoindex_DATATABLE_1
    ==> idx_primary

An additional index should mean additional space, so having several tables means several primary keys and therefore several indexes. 

As shown before in our current project, the size of the WebView database is almost 3 times the size of the Android database.