Bug 38003 – We should add FTS3 support to WebKit

• Component: New Bugs
• Status: RESOLVED FIXED

[Dumitru Daniliuc]

Gears supported FTS2, so I think we should add FTS3 support to WebKit, to make it easier for web developers who used Gears port their apps to WebSQLDatabases.

[Darin Fisher (:fishd, Google)]

Can you comment on why FTS3 should be added instead of FTS2?  Are there compatibility concerns?

[Dumitru Daniliuc]

(In reply to comment #1)
> Can you comment on why FTS3 should be added instead of FTS2?  Are there
> compatibility concerns?

FTS2 has a design problem: calling VACUUM on FTS2 databases could corrupt them. According to Scott, FTS3 is pretty much FTS2 with this problem fixed.

It is unclear if FTS2 databases are compatible with FTS3. We believe they should be, but I haven't tested this. However, WebKit doesn't have FTS2 support, and Chromium's FTS2 support is incomplete at the moment because we don't allow the MATCH operator (bug 37049). So I think it should be OK to treat Chromium's FTS2 support as experimental, and say that full-text search will be officially supported starting with FTS3.

[Michael Nordman]

(In reply to comment #1)
> Can you comment on why FTS3 should be added instead of FTS2?  Are there
> compatibility concerns?

FTS3 is being actively supported by SQLite, FTS2 is not.

FTS2 and has a known bug (around vacuum) that is fixed in FTS3. As I understand it, FTS3 was the fix for that known bug.

There is a data format change, FTS2 code can't read FTS3 data and vice versa.

(Just passing along what I've learned from shess)

[Darin Fisher (:fishd, Google)]

OK, I agree that it would be valuable for FTS3 to be enabled across all ports of WebKit.  /cc @mjs for his opinion.

[Brady Eidson]

I've been discussing this in emails with some of our SQLite gurus at Apple.  I think they think this is fine, as long as we still restrict some things with the authorizer.  fts3_tokenizer has come up by name, there may be others.

[Brady Eidson]

(In reply to comment #5)
> I've been discussing this in emails with some of our SQLite gurus at Apple.  I
> think they think this is fine, as long as we still restrict some things with
> the authorizer.  fts3_tokenizer has come up by name, there may be others.

Sorry - also meant to mention that I'll have all the info they mean to give me by Monday.

[Dumitru Daniliuc]

(In reply to comment #6)
> (In reply to comment #5)
> > I've been discussing this in emails with some of our SQLite gurus at Apple.  I
> > think they think this is fine, as long as we still restrict some things with
> > the authorizer.  fts3_tokenizer has come up by name, there may be others.
> 
> Sorry - also meant to mention that I'll have all the info they mean to give me
> by Monday.

Great! fts2_tokenizer() is currently disabled in DatabaseAuthorizer, because our security experts didn't like it too, and I'm pretty sure they agree that fts3_tokenizer() should be disabled too.

[Dumitru Daniliuc]

Brady, any updates?

[Dumitru Daniliuc]

Created attachment 56118 [details]
enable FTS3

[Dumitru Daniliuc]

Ping.

[Brady Eidson]

Comment on attachment 56118 [details]
enable FTS3

It really sucks that we're fragmenting capabilities like this, but I guess we already did it with fts2.  

That said, no one here sees an inherent problem with fts3.

[Dumitru Daniliuc]

Landed as r60180.

[WebKit Review Bot]

http://trac.webkit.org/changeset/60188 might have broken GTK Linux 32-bit Release