Bug 21239 – [CURL] Add on-disk file cache

• Component: WebCore Misc.
• Status: NEW

[Marco Barisione]

At the moment the GTK port doesn't support on-disk cache.
I'm implementing it in a generic way that can be easily used by both the CURL and soup backend.

[Marco Barisione]

Created attachment 26675 [details]
Implement on-disk cache

This is an old patch that I didn't have time to finish and propose for review.

In the patch I paid attention to make it possible to have more processes accessing the cache at the same time without race, corruptions, etc.

From a quick look the obvious problems are:
- Too many .utf8().data() calls, they were only meant as a temporary workaround as the patch to do a proper conversion was not merged yet.
- Tons of g_print for debugging.
- The cache is saved in the current directory and not in a proper location for testing reasons
- The cache should be nester, the cache entry with hash aabbccdd.. should be saved in aa/bbccdd...
- Not completely portable to windows, I have some ideas on how to do that.

ATM there's no way to know the total size of the cache or to remove files but I had some ideas on how to implement it, so ping me on IRC if you want to discuss it.

[Alexander Butenko]

ok. here is an update of Marco patch. There is some issues which i was to discuss here. 

- Too many .utf8().data() calls, they were only meant as a temporary workaround
as the patch to do a proper conversion was not merged yet.
 
i switched most of the String vars into char*. 

- Tons of g_print for debugging.

Thanks, debug is very nice. Patch still have them. What should we do with it? Add ifdef(DEBUG)? 
 
- The cache is saved in the current directory and not in a proper location for
testing reasons

fixed. Current location is $HOME/.cache/webkit/. Do we need any api to redefine this location from applications? From one point, unified cache directory should be a nice way to handle it. Once user have few webkit based applications, he will be able to use all cache from all his applications. From other point its a lack of configuration for advanced. users. 

- The cache should be nester, the cache entry with hash aabbccdd.. should be
saved in aa/bbccdd...

fixed. 

- Not completely portable to windows, I have some ideas on how to do that.

Sorry, no idea here. I have no idea about development for windows plus i dont have any for test. 


- CURL support is broken in mine patch. What should we do here. Marco and I is using GIO way of sending data to client. Is this way will be portable to all OS that webkit supports? What can be done, is that i can move GIO stuff out from soup file into separate file and reuse it for curl. 

[Alexander Butenko]

- Cookies have an 'Expire' field. should we honor it and send non expired files from cache without confirmation from the server that it not out outdated?  

[Alexander Butenko]

Created attachment 26786 [details]
Updated patch v0.1

patch not for a commit yet, but will be nice if somebody will check it.
There is some things is missing, i will update it soon.

[Christian Dywan]

(In reply to comment #4)
> Created an attachment (id=26786) [review]
> Updated patch v0.1
> 
> patch not for a commit yet, but will be nice if somebody will check it.
> There is some things is missing, i will update it soon.

It was mentioned in discussions on IRC that cache as a libSoup session feature makes much more sense than a WebKit internal implementation. We are only going to support libSoup in the future, and being part of the network interface makes it usable outside of WebKit.

[Gustavo Noronha (kov)]

Leaving only Curl as keyword, as this may still be useful for Curl; see Christian Dywan's comments regarding GTK+/Soup.

[Gustavo Noronha (kov)]

Moving away from WebKitGTK

[Denis Cheremisov]

So, what the situation with this issue now? I'm using latest webkit, and it seems both Epiphany-Webkit and Midori use disk too intensive (compared to Opera, Firefox, Chromium, etc)

[Christian Dywan]

(In reply to comment #8)
> So, what the situation with this issue now? I'm using latest webkit, and it
> seems both Epiphany-Webkit and Midori use disk too intensive (compared to
> Opera, Firefox, Chromium, etc)

Xan is working on a SoupSessionFeature for this, ie. a cache that is implemented in libSoup and can be used by all WebKit applications.

[Benjamin Meyer]

The location of the cache should be configurable and should not point to $HOME/.cache/webkit/ as this is not webkit http cache, but curl/soup cache and a specific implementation at that.  Not to mention that it is really gtkwebkit and not webkit.  As it is the cache patch is not shared with chromium or arora (both on linux and both using webkit, not to say we couldn't improve our cache's to all work together)  If in a year someone comes along and comes up with a better way to store the curl/soup cache they might not want to have both cache's in the same directory. 

On Linux $HOME/.cache should not be used, but the location should be determined from the environment variable $XDG_CACHE_HOME falling back to $HOME/.cache when it is empty.  See http://standards.freedesktop.org/basedir-spec/basedir-spec-0.6.html  One OS X and Windows of course this is also different and not $HOME/.cache.

Lastly although I do not see it in the xdg spec looking at my .cache I see that the way that the .cache directory is being used is $XDG_CACHE_HOME/$COMPANY_NAME/$APPLICATION_NAME/ for example $HOME/.cache/midori/ or $HOME/.cache/Trolltech/demobrowser  Different applications have different needs for cache.  One application might hit the same few website(s) for years such as an rss reader.  For that application it is very valuable that the cache not be deleted, but a web browser cache usage is different and could completely change every hour.

[Christian Dywan]

To make this clear: WebKitGTK+ is not using CURL anymore and hence this feature request is obsolete as far as WebKitGTK+ is concerned.

[Sergio Villar Senin]

I guess it's better to close this now that we have https://bugs.webkit.org/show_bug.cgi?id=44261

[Benjamin Meyer]

Shouldn't you leave the old bug with lots of useful information open and close the new duplicate bug that contains one link and move that link here?

[Sergio Villar Senin]

(In reply to comment #13)
> Shouldn't you leave the old bug with lots of useful information open and close the new duplicate bug that contains one link and move that link here?

I don't think so. Mainly because although they both target the same feature they are based on very different implementations. The new one is just for discussing that other implementation.

As kov said curl cache won't be integrated in webkitgtk+ I thought it could be useful for other people to know about new plans. Anyway this can be left open, not strong opinion about it.