RESOLVED FIXED Bug 155120
Reduce startup and shutdown cost of resource load statistics
https://bugs.webkit.org/show_bug.cgi?id=155120
Summary Reduce startup and shutdown cost of resource load statistics
Brent Fulgham
Reported 2016-03-07 10:20:29 PST
Avoid blocking the main thread when starting up (or shutting down) by loading any existing resource load statistics off of the main thread. Likewise, be keep the on-disk copy of the data up-to-date by writing an update when new data has been added to the set of statistics.
Attachments
Patch (8.37 KB, patch)
2016-03-07 10:41 PST, Brent Fulgham
no flags
Patch (9.47 KB, patch)
2016-03-07 13:20 PST, Brent Fulgham
no flags
Patch (24.13 KB, patch)
2016-03-07 16:48 PST, Brent Fulgham
no flags
Patch (24.10 KB, patch)
2016-03-07 16:55 PST, Brent Fulgham
no flags
Archive of layout-test-results from ews100 for mac-yosemite (909.80 KB, application/zip)
2016-03-07 17:32 PST, Build Bot
no flags
Patch (22.83 KB, patch)
2016-03-07 17:57 PST, Brent Fulgham
no flags
Patch (22.85 KB, patch)
2016-03-07 18:02 PST, Brent Fulgham
aestes: review+
Radar WebKit Bug Importer
Comment 1 2016-03-07 10:33:45 PST
Radar WebKit Bug Importer
Comment 2 2016-03-07 10:33:52 PST
Brent Fulgham
Comment 3 2016-03-07 10:41:54 PST
Brent Fulgham
Comment 4 2016-03-07 11:07:31 PST
Comment on attachment 273194 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=273194&action=review > Source/WebKit2/UIProcess/WebResourceLoadStatisticsStore.cpp:46 > + , m_loadDataQueue(WorkQueue::create("WebResourceLoadStatisticsStore Load Data Queue")) WK2 experts: Do we need two queues here? I was thinking we might want to continue receiving data from the WebProcess while the "old" statistics were read off disk. > Source/WebKit2/UIProcess/WebResourceLoadStatisticsStore.cpp:57 > + Vector<WebCore::ResourceLoadStatistics> statistics(origins); WK2 and C++ exports: This copy may be dumb, I'm not sure I need it. Shouldn't the 'statistics' capture in the lambda (below) cause this copy already?
Andy Estes
Comment 5 2016-03-07 12:49:00 PST
Comment on attachment 273194 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=273194&action=review Brent and I discussed this in person. We don't think we need two additional queues. Rather, we should create one queue that is also the message receiver queue for ResourceLoadStatisticsUpdated. Then we don't have to worry about having a lock to coordinate the file I/O, or about making isolated copies of statistics vectors. >> Source/WebKit2/UIProcess/WebResourceLoadStatisticsStore.cpp:57 >> + Vector<WebCore::ResourceLoadStatistics> statistics(origins); > > WK2 and C++ exports: This copy may be dumb, I'm not sure I need it. Shouldn't the 'statistics' capture in the lambda (below) cause this copy already? If you end up using a separate queue, then you actually need to copy this in a way that makes isolated copies of all the Strings in the Vector.
Brent Fulgham
Comment 6 2016-03-07 13:20:36 PST
Andy Estes
Comment 7 2016-03-07 13:57:20 PST
Comment on attachment 273203 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=273203&action=review > Source/WebKit2/UIProcess/WebResourceLoadStatisticsStore.cpp:78 > + if (!m_resourceLoadStatisticsEnabled || m_dataWasLoadedFromDisk) It's not safe to read this if m_statisticsQueue can write to it concurrently. But if m_resourceLoadStatisticsEnabled == true, wouldn't readDataFromDiskIfNeeded() have already been called? > Source/WebKit2/UIProcess/WebResourceLoadStatisticsStore.cpp:-94 > - if (!m_resourceLoadStatisticsEnabled) > - return; > - > - // FIXME(154642): TEMPORARY CODE: This should not be done in one long operation when exiting. > - writeToDisk(); What guarantees data will be saved before shutdown? Do you not want to make that guarantee anymore? > Source/WebKit2/UIProcess/WebsiteData/WebsiteDataStore.cpp:984 > - connection.addWorkQueueMessageReceiver(Messages::WebResourceLoadStatisticsStore::messageReceiverName(), &m_queue.get(), m_resourceLoadStatistics.get()); > + connection.addWorkQueueMessageReceiver(Messages::WebResourceLoadStatisticsStore::messageReceiverName(), &m_resourceLoadStatistics->queue(), m_resourceLoadStatistics.get()); Instead of adding a public getter for WebResourceLoadStatisticsStore's WorkQueue, I'd follow the pattern set by StorageManager and add processWillOpenConnection() and processDidCloseConnection() functions to WebResourceLoadStatisticsStore that add/remove the work queue message receiver.
Brent Fulgham
Comment 8 2016-03-07 14:23:27 PST
Comment on attachment 273203 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=273203&action=review >> Source/WebKit2/UIProcess/WebResourceLoadStatisticsStore.cpp:78 >> + if (!m_resourceLoadStatisticsEnabled || m_dataWasLoadedFromDisk) > > It's not safe to read this if m_statisticsQueue can write to it concurrently. But if m_resourceLoadStatisticsEnabled == true, wouldn't readDataFromDiskIfNeeded() have already been called? I'm worried about the case where someone enables the feature (activates resource load statistics), then deactivates the feature, then turns the feature back on. This could cause us to double-load the disk copy of the statistics, which would then be merged back into the in-memory data. We could purge the in-memory data when the feature is turned off to avoid the problem, but it seems like using this flag would be less costly.
Brent Fulgham
Comment 9 2016-03-07 16:48:46 PST
Brent Fulgham
Comment 10 2016-03-07 16:55:52 PST
Build Bot
Comment 11 2016-03-07 17:32:26 PST
Comment on attachment 273241 [details] Patch Attachment 273241 [details] did not pass mac-ews (mac): Output: http://webkit-queues.webkit.org/results/938792 New failing tests: http/tests/navigation/statistics.html
Build Bot
Comment 12 2016-03-07 17:32:30 PST
Created attachment 273245 [details] Archive of layout-test-results from ews100 for mac-yosemite The attached test failures were seen while running run-webkit-tests on the mac-ews. Bot: ews100 Port: mac-yosemite Platform: Mac OS X 10.10.5
Brent Fulgham
Comment 13 2016-03-07 17:33:38 PST
Looks like I introduced a crash when migrating code out of WebCore. I'm looking at the cause right now.
Andy Estes
Comment 14 2016-03-07 17:38:52 PST
Comment on attachment 273241 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=273241&action=review > Source/WebCore/loader/ResourceLoadStatisticsStore.cpp:138 > +void ResourceLoadStatisticsStore::mergeStatistics(const ResourceLoadStatisticsStore& statistics) > +{ > + for (auto& statistic : statistics.m_resourceStatisticsMap) { > + auto result = m_resourceStatisticsMap.ensure(statistic.value.highLevelDomain, [&statistic] { > + return ResourceLoadStatistics(statistic.value.highLevelDomain); > + }); > + > + result.iterator->value.merge(statistic.value); > + } > +} I don't think we actually need this. See below. > Source/WebKit2/UIProcess/WebResourceLoadStatisticsStore.cpp:76 > + RefPtr<WebResourceLoadStatisticsStore> self(this); > + m_statisticsQueue->dispatch([self] { > + self->coreStore().clear(); > + }); You only want to do this when enabled == true. Otherwise you'll potentially delete the store while web processes are still sending back data to merge. Let's just make the clearing be part of readDataFromDiskIfNeeded(). It's safe for readDataFromDiskIfNeeded() to always clear the store, because it's guaranteed that (1) any previously received statistics have already been saved to disk, and (2) no statistics will be received concurrently with reading from disk, since both operations occur on the same work queue. > Source/WebKit2/UIProcess/WebResourceLoadStatisticsStore.cpp:100 > + self->coreStore().mergeStatistics(tempStatisticsStore); If we always clear the store when calling readDataFromDiskIfNeeded(), then there's nothing to merge. That means you don't need to add a new mergeStatistics().
Brent Fulgham
Comment 15 2016-03-07 17:43:00 PST
Comment on attachment 273241 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=273241&action=review >> Source/WebCore/loader/ResourceLoadStatisticsStore.cpp:138 >> +} > > I don't think we actually need this. See below. Oh, that's a good point. I'll scrap it. >> Source/WebKit2/UIProcess/WebResourceLoadStatisticsStore.cpp:76 >> + }); > > You only want to do this when enabled == true. Otherwise you'll potentially delete the store while web processes are still sending back data to merge. > > Let's just make the clearing be part of readDataFromDiskIfNeeded(). It's safe for readDataFromDiskIfNeeded() to always clear the store, because it's guaranteed that (1) any previously received statistics have already been saved to disk, and (2) no statistics will be received concurrently with reading from disk, since both operations occur on the same work queue. Agreed. I'll change the code to reflect this. >> Source/WebKit2/UIProcess/WebResourceLoadStatisticsStore.cpp:100 >> + self->coreStore().mergeStatistics(tempStatisticsStore); > > If we always clear the store when calling readDataFromDiskIfNeeded(), then there's nothing to merge. That means you don't need to add a new mergeStatistics(). OK!
Brent Fulgham
Comment 16 2016-03-07 17:57:40 PST
Brent Fulgham
Comment 17 2016-03-07 18:02:39 PST
Brent Fulgham
Comment 18 2016-03-07 18:02:57 PST
Comment on attachment 273250 [details] Patch Attempt to satisfy EFL build.
Andy Estes
Comment 19 2016-03-07 18:49:06 PST
Comment on attachment 273250 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=273250&action=review > Source/WebCore/ChangeLog:3 > + Reduce startup and shutdown cost of resource load statistics Not sure this'll do much for shutdown time, since we synchronize with the statistics queue on shutdown. > Source/WebCore/ChangeLog:10 > + Move all file-related code out of WebCore. Add a new overload so that we can > + merge an entire ResourceLoadStatisticsStore object with another. There's no overload anymore. > Source/WebKit2/ChangeLog:13 > + that it does not delay startup or shutdown. If load statistics are observed > + before the disk read is complete, we simply merge the on-disk data > + with whatever we've seen in the meantime. Load statistics can't be observed during the disk read.
Brent Fulgham
Comment 20 2016-03-07 18:56:11 PST
Brent Fulgham
Comment 21 2016-03-08 09:40:38 PST
Small follow-up bug fix: Committed r197771: <http://trac.webkit.org/changeset/197771>
Note You need to log in before you can comment on or make changes to this bug.