104665 2012-12-11 05:53:37 -0800 [EFL] API unit tests are running extremely slow on the bots 2013-01-21 11:33:30 -0800 1 1 1 Unclassified WebKit WebKit EFL 528+ (Nightly build) Unspecified Unspecified RESOLVED FIXED P2 Normal --- 104666 104782 90451 1 tmpsantos tmpsantos cdumez danw d-r gyuyoung.kim jinwoo7.song joone jussi.kukkonen kenneth laszlo.gombos lucas.de.marchi naginenis rakuco ryuan.choi svillar webkit.review.bot oldest_to_newest 788363 0 tmpsantos 2012-12-11 05:53:37 -0800 They take ~60 seconds a full run on my desktop machine. For some reason on the bots we have a lot of dangling WebProcess sucking 100% of the CPU. 788489 1 tmpsantos 2012-12-11 09:02:21 -0800 Problem found! The cache is creating too many garbage at: /home/buildslave-1/.cache/WebKitEfl/ 788498 2 tmpsantos 2012-12-11 09:17:59 -0800 Guys, this is a serious issue. The conclusion here is: soup_cache creates a lot of files over the time at .cache/WebKitEfl. When I checked the bots, it had close to 66k files. Performance will deteriorate over time, since soup_cache_load() at WebProcessMainEfl.cpp does a complete readdir(.cache/WebKitEfl), so it will be O(n) and it takes a lot of CPU for that. *sigh* We have to limit somehow the number of entries at this cache. 788500 3 kenneth 2012-12-11 09:21:07 -0800 void soup_cache_set_max_size(SoupCache *cache, guint max_size); http://developer.gnome.org/libsoup/unstable/SoupCache.html 788502 4 kenneth 2012-12-11 09:27:14 -0800 Why not use a different cache dir when running the tests and clear it before running the tests. Also I think something like 50MB should be sufficient and that should keep the file amount down as well 788504 5 kenneth 2012-12-11 09:30:38 -0800 https://bugzilla.mozilla.org/show_bug.cgi?id=559942 https://code.google.com/p/chromium/issues/detail?id=40079 Info on chosen cache sizes in other browsers 789312 6 svillar 2012-12-12 01:09:54 -0800 (In reply to comment #2) > Guys, this is a serious issue. > > The conclusion here is: soup_cache creates a lot of files over the time at .cache/WebKitEfl. When I checked the bots, it had close to 66k files. > > Performance will deteriorate over time, since soup_cache_load() at WebProcessMainEfl.cpp does a complete readdir(.cache/WebKitEfl), so it will be O(n) and it takes a lot of CPU for that. *sigh* That is not correct. The load of the cache reads the contents of a single file which stores the cache indexes. It's true that we need to improve that (there are several bugs in the gnome bugzilla for that) but it isn't not a complete readdir. > We have to limit somehow the number of entries at this cache. There are a couple of options: * setup a temporal dir for the cache during testing * disable the cache during testing * limit the size of the cache during testing 789353 7 joone 2012-12-12 02:04:15 -0800 I think we need an API to clear the cache files before running the test cases. So, I've filed a bug: https://bugs.webkit.org/show_bug.cgi?id=104782 789573 8 tmpsantos 2012-12-12 08:15:56 -0800 (In reply to comment #6) > (In reply to comment #2) > > Guys, this is a serious issue. > > > > The conclusion here is: soup_cache creates a lot of files over the time at .cache/WebKitEfl. When I checked the bots, it had close to 66k files. > > > > Performance will deteriorate over time, since soup_cache_load() at WebProcessMainEfl.cpp does a complete readdir(.cache/WebKitEfl), so it will be O(n) and it takes a lot of CPU for that. *sigh* > > That is not correct. The load of the cache reads the contents of a single file which stores the cache indexes. It's true that we need to improve that (there are several bugs in the gnome bugzilla for that) but it isn't not a complete readdir. Indeed, but looks like the entry count grows at the same rate as the file count on the folder. The process spends a lot of time inserting these entries on a list. > > > We have to limit somehow the number of entries at this cache. > > There are a couple of options: > > * setup a temporal dir for the cache during testing > * disable the cache during testing > * limit the size of the cache during testing My main concern here is from the browser user point of view. I really don't expect the users to clear the cache when they fell like the browser is not performing. At the time when I deleted the .cache/WebKitEfl folder on the bots, it had 65k files and the size of ~200MB. After removing the cache, the unit test run time dropped from 17 min to 1 min, which tells something about the impact of loading the cache when starting a WebProcess. I did that yesterday and today it is already 1k files and ~3MB. I gonna fix this on the bots by creating some sort of temp dir, but it won't solve the underlining problem. Would be good to work together with the libsoup developers on this issue. 789574 9 d-r 2012-12-12 08:17:48 -0800 CC'ed Dan Winship - maybe he can comment. 789583 10 svillar 2012-12-12 08:26:43 -0800 (In reply to comment #8) > (In reply to comment #6) > > (In reply to comment #2) > > > Guys, this is a serious issue. > > > > > > The conclusion here is: soup_cache creates a lot of files over the time at .cache/WebKitEfl. When I checked the bots, it had close to 66k files. > > > > > > Performance will deteriorate over time, since soup_cache_load() at WebProcessMainEfl.cpp does a complete readdir(.cache/WebKitEfl), so it will be O(n) and it takes a lot of CPU for that. *sigh* > > > > That is not correct. The load of the cache reads the contents of a single file which stores the cache indexes. It's true that we need to improve that (there are several bugs in the gnome bugzilla for that) but it isn't not a complete readdir. > > Indeed, but looks like the entry count grows at the same rate as the file count on the folder. The process spends a lot of time inserting these entries on a list. > > > > > > We have to limit somehow the number of entries at this cache. > > > > There are a couple of options: > > > > * setup a temporal dir for the cache during testing > > * disable the cache during testing > > * limit the size of the cache during testing > > My main concern here is from the browser user point of view. I really don't expect the users to clear the cache when they fell like the browser is not performing. > > At the time when I deleted the .cache/WebKitEfl folder on the bots, it had 65k files and the size of ~200MB. After removing the cache, the unit test run time dropped from 17 min to 1 min, which tells something about the impact of loading the cache when starting a WebProcess. > > I did that yesterday and today it is already 1k files and ~3MB. I gonna fix this on the bots by creating some sort of temp dir, but it won't solve the underlining problem. > > Would be good to work together with the libsoup developers on this issue. So as I said there are a couple of reasons why it takes so long. The main one is likely that the HTTP headers are currently stored in the index. Loading the index means allocating space for all of them and also deserialize them from the index file. I opened https://bugzilla.gnome.org/show_bug.cgi?id=665884 some time ago to fix that. The patches were accepted but I didn't commit them yet because I am not totally sure that they're correct (not having the headers in memory added some complexities to the SoupCache). 789940 11 tmpsantos 2012-12-12 14:47:39 -0800 (In reply to comment #10) > (In reply to comment #8) > > (In reply to comment #6) > > > (In reply to comment #2) > > > > Guys, this is a serious issue. > > > > > > > > The conclusion here is: soup_cache creates a lot of files over the time at .cache/WebKitEfl. When I checked the bots, it had close to 66k files. > > > > > > > > Performance will deteriorate over time, since soup_cache_load() at WebProcessMainEfl.cpp does a complete readdir(.cache/WebKitEfl), so it will be O(n) and it takes a lot of CPU for that. *sigh* > > > > > > That is not correct. The load of the cache reads the contents of a single file which stores the cache indexes. It's true that we need to improve that (there are several bugs in the gnome bugzilla for that) but it isn't not a complete readdir. > > > > Indeed, but looks like the entry count grows at the same rate as the file count on the folder. The process spends a lot of time inserting these entries on a list. > > > > > > > > > We have to limit somehow the number of entries at this cache. > > > > > > There are a couple of options: > > > > > > * setup a temporal dir for the cache during testing > > > * disable the cache during testing > > > * limit the size of the cache during testing > > > > My main concern here is from the browser user point of view. I really don't expect the users to clear the cache when they fell like the browser is not performing. > > > > At the time when I deleted the .cache/WebKitEfl folder on the bots, it had 65k files and the size of ~200MB. After removing the cache, the unit test run time dropped from 17 min to 1 min, which tells something about the impact of loading the cache when starting a WebProcess. > > > > I did that yesterday and today it is already 1k files and ~3MB. I gonna fix this on the bots by creating some sort of temp dir, but it won't solve the underlining problem. > > > > Would be good to work together with the libsoup developers on this issue. > > So as I said there are a couple of reasons why it takes so long. The main one is likely that the HTTP headers are currently stored in the index. Loading the index means allocating space for all of them and also deserialize them from the index file. > > I opened https://bugzilla.gnome.org/show_bug.cgi?id=665884 some time ago to fix that. The patches were accepted but I didn't commit them yet because I am not totally sure that they're correct (not having the headers in memory added some complexities to the SoupCache). I made the mistake of just deleting the cache on the bots. I should have saved it to help on the profiling the hotspots for optimization. Fortunately (if I can say that) the EFL bot maintained by Samsung still has the issue. Gyuyoung, is it possible to share somehow the contents of .cache/WebKitEfl from the efl-buildslave-1 before you remove it? It would be very helpful on debugging this issue. 790231 12 gyuyoung.kim 2012-12-12 20:32:03 -0800 (In reply to comment #11) > Gyuyoung, is it possible to share somehow the contents of .cache/WebKitEfl from the efl-buildslave-1 before you remove it? It would be very helpful on debugging this issue. Sure, I will share it via email or ftp server. 796884 13 180645 joone 2012-12-23 22:46:25 -0800 Created attachment 180645 Proposed Patch I've implemented ewk_context_resource_cache_clear API, which allows to clear all resource caches in memory and local storage. https://bugs.webkit.org/show_bug.cgi?id=104782 We can clear all cache files by calling this API before running the API unit tests. 796921 14 180645 kenneth 2012-12-24 01:48:39 -0800 Comment on attachment 180645 Proposed Patch Is this up for review? I still think the name is not so good, what is a resource cache? Does it include more than http? 797553 15 180645 tmpsantos 2012-12-27 07:38:19 -0800 Comment on attachment 180645 Proposed Patch I'm fine with this patch as long as we create a patch to fix the performance degradation caused by the cache. 798190 16 180645 gyuyoung.kim 2012-12-29 21:03:14 -0800 Comment on attachment 180645 Proposed Patch LGTM 798191 17 180645 gyuyoung.kim 2012-12-29 21:05:31 -0800 Comment on attachment 180645 Proposed Patch It looks someone else might wanna have final a look. 798192 18 joone 2012-12-29 21:07:05 -0800 (In reply to comment #15) > (From update of attachment 180645 [details]) > I'm fine with this patch as long as we create a patch to fix the performance degradation caused by the cache. The ewk_context_resource_cache_clear API landed, so this patch could fix the performance degradation. http://trac.webkit.org/changeset/138554 801511 19 tmpsantos 2013-01-07 11:57:09 -0800 What about landing this patch? Joone? 801681 20 joone 2013-01-07 14:53:25 -0800 (In reply to comment #19) > What about landing this patch? Joone? I think that we need Kenneth's final confirmation. 801772 21 laszlo.gombos 2013-01-07 16:02:20 -0800 (In reply to comment #6) > (In reply to comment #2) > > Guys, this is a serious issue. > > > > The conclusion here is: soup_cache creates a lot of files over the time at .cache/WebKitEfl. When I checked the bots, it had close to 66k files. > > > > Performance will deteriorate over time, since soup_cache_load() at WebProcessMainEfl.cpp does a complete readdir(.cache/WebKitEfl), so it will be O(n) and it takes a lot of CPU for that. *sigh* > > That is not correct. The load of the cache reads the contents of a single file which stores the cache indexes. It's true that we need to improve that (there are several bugs in the gnome bugzilla for that) but it isn't not a complete readdir. Sergio, can you share with us which soup bugs (bug ids or urls) are relevant for this issue so that we can track them ? 801903 22 danw 2013-01-07 18:13:01 -0800 https://bugzilla.gnome.org/show_bug.cgi?id=665884 moves the HTTP headers from the cache index file into the individual cache files, thereby making the index much smaller. https://bugzilla.gnome.org/show_bug.cgi?id=666143 discusses changing the way cache files are stored, but Sergio keeps forgetting to attach his old patches there (hint hint :). 812288 23 180645 webkit.review.bot 2013-01-21 11:33:24 -0800 Comment on attachment 180645 Proposed Patch Clearing flags on attachment: 180645 Committed r140348: <http://trac.webkit.org/changeset/140348> 812289 24 webkit.review.bot 2013-01-21 11:33:30 -0800 All reviewed patches have been landed. Closing bug. 180645 2012-12-23 22:46:25 -0800 2013-01-21 11:33:24 -0800 Proposed Patch clear_cache.patch text/plain 1629 joone ZGlmZiAtLWdpdCBhL1NvdXJjZS9XZWJLaXQyL0NoYW5nZUxvZyBiL1NvdXJjZS9XZWJLaXQyL0No YW5nZUxvZwppbmRleCBjN2RhMTliLi5mYzc2MmEzIDEwMDY0NAotLS0gYS9Tb3VyY2UvV2ViS2l0 Mi9DaGFuZ2VMb2cKKysrIGIvU291cmNlL1dlYktpdDIvQ2hhbmdlTG9nCkBAIC0xLDUgKzEsMTgg QEAKIDIwMTItMTItMjMgIEpvb25lIEh1ciAgPGpvb25lLmh1ckBpbnRlbC5jb20+CiAKKyAgICAg ICAgW0VGTF0gQVBJIHVuaXQgdGVzdHMgYXJlIHJ1bm5pbmcgZXh0cmVtZWx5IHNsb3cgb24gdGhl IGJvdHMKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTEw NDY2NQorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisgICAgICAgIENs ZWFyIEhUVFAgY2FjaGUgZmlsZXMgYmVmb3JlIHJ1bm5pbmcgdGhlIHVuaXQgdGVzdHMsIHdoaWNo IHByZXZlbnRzIAorICAgICAgICBwZXJmb3JtYW5jZSBkZWdyYWRhdGlvbiBkdWUgdG8gc28gbWFu eSBjYWNoZSBmaWxlcy4KKworICAgICAgICAqIFVJUHJvY2Vzcy9BUEkvZWZsL3Rlc3RzL1VuaXRU ZXN0VXRpbHMvRVdLMlVuaXRUZXN0QmFzZS5jcHA6CisgICAgICAgIChFV0syVW5pdFRlc3Q6OkVX SzJVbml0VGVzdEJhc2U6OlNldFVwKToKKworMjAxMi0xMi0yMyAgSm9vbmUgSHVyICA8am9vbmUu aHVyQGludGVsLmNvbT4KKwogICAgICAgICBbRUZMXVtXSzJdIEltcGxlbWVudCBld2tfY29udGV4 dF9yZXNvdXJjZV9jYWNoZV9jbGVhcgogICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9z aG93X2J1Zy5jZ2k/aWQ9MTA0NzgyCiAKZGlmZiAtLWdpdCBhL1NvdXJjZS9XZWJLaXQyL1VJUHJv Y2Vzcy9BUEkvZWZsL3Rlc3RzL1VuaXRUZXN0VXRpbHMvRVdLMlVuaXRUZXN0QmFzZS5jcHAgYi9T b3VyY2UvV2ViS2l0Mi9VSVByb2Nlc3MvQVBJL2VmbC90ZXN0cy9Vbml0VGVzdFV0aWxzL0VXSzJV bml0VGVzdEJhc2UuY3BwCmluZGV4IGIzZmJjYmYuLmU2NGY0Y2MgMTAwNjQ0Ci0tLSBhL1NvdXJj ZS9XZWJLaXQyL1VJUHJvY2Vzcy9BUEkvZWZsL3Rlc3RzL1VuaXRUZXN0VXRpbHMvRVdLMlVuaXRU ZXN0QmFzZS5jcHAKKysrIGIvU291cmNlL1dlYktpdDIvVUlQcm9jZXNzL0FQSS9lZmwvdGVzdHMv VW5pdFRlc3RVdGlscy9FV0syVW5pdFRlc3RCYXNlLmNwcApAQCAtNjIsNiArNjIsOSBAQCB2b2lk IEVXSzJVbml0VGVzdEJhc2U6OlNldFVwKCkKIAogICAgIEV3a19Db250ZXh0KiBuZXdDb250ZXh0 ID0gZXdrX2NvbnRleHRfbmV3KCk7CiAgICAgbV93ZWJWaWV3ID0gZXdrX3ZpZXdfc21hcnRfYWRk KGV2YXMsIHNtYXJ0LCBuZXdDb250ZXh0KTsKKyAgICAvLyBDbGVhciBIVFRQIGNhY2hlIGZpbGVz IGJlZm9yZSBydW5uaW5nIHRoZSB1bml0IHRlc3RzLCB3aGljaCBwcmV2ZW50cyAKKyAgICAvLyBw ZXJmb3JtYW5jZSBkZWdyYWRhdGlvbiBkdWUgdG8gc28gbWFueSBjYWNoZSBmaWxlcy4KKyAgICBl d2tfY29udGV4dF9yZXNvdXJjZV9jYWNoZV9jbGVhcihuZXdDb250ZXh0KTsKICAgICBld2tfb2Jq ZWN0X3VucmVmKG5ld0NvbnRleHQpOwogCiAgICAgZXdrX3ZpZXdfdGhlbWVfc2V0KG1fd2ViVmll dywgZW52aXJvbm1lbnQtPmRlZmF1bHRUaGVtZSgpKTsK