ChangeLog

2021-07-23  Myles C. Maxfield  <mmaxfield@apple.com>

        [GPU Process] Add resource use counter infrastructure to RemoteResourceCache

        https://bugs.webkit.org/show_bug.cgi?id=228222

        Reviewed by NOBODY (OOPS!).

        This is part 2 of https://bugs.webkit.org/show_bug.cgi?id=228216.

        No new tests because there is no behavior change yet. This patch is just adding infrastructure. For now, all these

        use counts sent by the web process are 0 - I'll implement that part in 3 follow-up patches.

        * platform/graphics/displaylists/DisplayListReplayer.h:

        (WebCore::DisplayList::Replayer::Delegate::recordResourceUse):

2021-07-23  Andres Gonzalez  <andresg_22@apple.com>

        Add a method to WebAccessibilityObjectWrapper so that clients can retrieve the text of each line and their corresponding bounding rectangles.

Source/WebKit/ChangeLog

2021-07-23  Myles C. Maxfield  <mmaxfield@apple.com>

        [GPU Process] Add resource use counter infrastructure to RemoteResourceCache

        https://bugs.webkit.org/show_bug.cgi?id=228222

        Reviewed by NOBODY (OOPS!).

        This patch adds the infrastructure for associating a counter with each resource in the RemoteResourceCache.

        As the web process uses a resource during display list recording, it increments a counter, and as the GPU

        process uses a resource, it increments a counter in the RemoteResourceCache. When the web process is done

        with a resource, it sends a release message to the GPU process which crucially includes the web process's

        counter. The GPU process has to then wait to actually delete the resource until its own use counter matches

        the one it received from the web process. This patch implements this deferred deletion infrastructure

        in the GPU process.

        * GPUProcess/graphics/RemoteRenderingBackend.cpp:

        (WebKit::RemoteRenderingBackend::ReplayerDelegate::recordResourceUse):

        (WebKit::RemoteRenderingBackend::releaseRemoteResource): Pass the use count to the RemoteResourceCache.

        * GPUProcess/graphics/RemoteRenderingBackend.h: Add the use count field.

        * GPUProcess/graphics/RemoteRenderingBackend.messages.in: Ditto.

        * GPUProcess/graphics/RemoteResourceCache.cpp:

        (WebKit::RemoteResourceCache::cacheImageBuffer): It's actually okay if there's already an resource cached.

        That just means that the web process started using a resource again before the GPU process consumed all

        commands that used it the first time. All we need to do is increment the open count.

        (WebKit::RemoteResourceCache::cacheNativeImage): Ditto.

        (WebKit::RemoteResourceCache::cacheFont): Ditto.

        (WebKit::RemoteResourceCache::ensureResourceUseCounter):

        (WebKit::RemoteResourceCache::maybeRemoveResource): This is called whenever it's possible for us to be in

        situation where we should be removing a resource (aka whenever the use count increments). It checks to see

        if we can remove the resource, and if it can, does so.

        (WebKit::RemoteResourceCache::recordResourceUse): The callback that runs for each resource use during

        replaying. It can't actually delete any resources, so instead it just records which resources had their

        use counts incremented, and waits for prune() to actually call maybeRemoveResource().

        (WebKit::RemoteResourceCache::prune): After the replay is complete, call maybeRemoveResource() on all the

        resources which had their use counters touched.

        (WebKit::RemoteResourceCache::releaseRemoteResource): Put the relevant resource into the state where we

        are waiting for the use counters to match before we can delete the resource. Also, call maybeRemoveResource()

        so it can be deleted immediately if the counters already match.

        * GPUProcess/graphics/RemoteResourceCache.h: Add a new data structure to hold the use counters. There's

        a big comment in here that describes what these fields mean.

        * WebProcess/GPU/graphics/RemoteImageBufferProxy.h:

        (WebKit::RemoteImageBufferProxy::~RemoteImageBufferProxy): Send a dummy use count for now. This will be

        implemented in a follow-up patch.

        * WebProcess/GPU/graphics/RemoteRenderingBackendProxy.cpp:

        (WebKit::RemoteRenderingBackendProxy::releaseRemoteResource): Add the useCount parameter.

        * WebProcess/GPU/graphics/RemoteRenderingBackendProxy.h: Ditto.

        * WebProcess/GPU/graphics/RemoteResourceCacheProxy.cpp:

        (WebKit::RemoteResourceCacheProxy::releaseNativeImage): Send a dummy use count for now. This will be

        implemented in a follow-up patch.

        (WebKit::RemoteResourceCacheProxy::clearFontMap): Ditto.

        (WebKit::RemoteResourceCacheProxy::finalizeRenderingUpdateForFonts): Ditto.

2021-07-23  Alex Christensen  <achristensen@webkit.org>

        Make WKContentRuleListStore respond to same selectors as _WKUserContentExtensionStore

Source/WebCore/platform/graphics/displaylists/DisplayListReplayer.h

@@public:

        virtual bool apply(ItemHandle, GraphicsContext&) { return false; }

        virtual void didCreateMaskImageBuffer(WebCore::ImageBuffer&) { }

        virtual void didResetMaskImageBuffer() { }

71 virtual void recordResourceUse(RenderingResourceIdentifier) { }

    };

private:

Source/WebKit/GPUProcess/graphics/RemoteRenderingBackend.cpp

@@void RemoteRenderingBackend::createImageBuffer(const FloatSize& logicalSize, Ren

        wakeUpAndApplyDisplayList(std::exchange(m_pendingWakeupInfo, std::nullopt)->arguments);

}

187 RemoteRenderingBackend::ReplayerDelegate::ReplayerDelegate(WebCore::ImageBuffer& destination, RemoteRenderingBackend& remoteRenderingBackend~~, GPUConnectionToWebProcess& gpuConnectionToWebProcess~~)

187RemoteRenderingBackend::ReplayerDelegate::ReplayerDelegate(WebCore::ImageBuffer& destination, RemoteRenderingBackend& remoteRenderingBackend)

188188 : m_destination(destination)

189189 , m_remoteRenderingBackend(remoteRenderingBackend)

190 , m_gpuConnectionToWebProcess(gpuConnectionToWebProcess)

{

}

bool RemoteRenderingBackend::ReplayerDelegate::apply(WebCore::DisplayList::ItemHandle item, WebCore::GraphicsContext& graphicsContext)

{

196 // FIXME: Inspect the item and record resource use here.

197

    auto apply = [&](auto&& destination) {

        return destination.apply(item, graphicsContext);

    };

@@void RemoteRenderingBackend::ReplayerDelegate::didResetMaskImageBuffer()

    m_remoteRenderingBackend.didResetMaskImageBuffer();

}

void RemoteRenderingBackend::ReplayerDelegate::recordResourceUse(RenderingResourceIdentifier renderingResourceIdentifier)

{

    m_remoteRenderingBackend.remoteResourceCache().recordResourceUse(renderingResourceIdentifier);

}

DisplayList::ReplayResult RemoteRenderingBackend::submit(const DisplayList::DisplayList& displayList, ImageBuffer& destination)

{

    if (displayList.isEmpty())

        return { };

222 ReplayerDelegate replayerDelegate(destination, *this~~, m_gpuConnectionToWebProcess~~);

224 ReplayerDelegate replayerDelegate(destination, *this);

    return WebCore::DisplayList::Replayer {

        destination.context(),

@@void RemoteRenderingBackend::deleteAllFonts()

    m_remoteResourceCache.deleteAllFonts();

}

512 void RemoteRenderingBackend::releaseRemoteResource(RenderingResourceIdentifier renderingResourceIdentifier)

514void RemoteRenderingBackend::releaseRemoteResource(RenderingResourceIdentifier renderingResourceIdentifier, uint64_t useCount)

513515{

514516 ASSERT(!RunLoop::isMain());

515 m_remoteResourceCache.releaseRemoteResource(renderingResourceIdentifier);

517 auto success = m_remoteResourceCache.releaseRemoteResource(renderingResourceIdentifier, useCount);

518 MESSAGE_CHECK(success, "Resource is being released before being cached.");

    updateRenderingResourceRequest();

}

Source/WebKit/GPUProcess/graphics/RemoteRenderingBackend.h

@@private:

    void cacheNativeImage(const ShareableBitmap::Handle&, WebCore::RenderingResourceIdentifier);

    void cacheFont(Ref<WebCore::Font>&&);

    void deleteAllFonts();

147 void releaseRemoteResource(WebCore::RenderingResourceIdentifier);

147 void releaseRemoteResource(WebCore::RenderingResourceIdentifier, uint64_t useCount);

    void didCreateSharedDisplayListHandle(WebCore::DisplayList::ItemBufferIdentifier, const SharedMemory::IPCHandle&, WebCore::RenderingResourceIdentifier destinationBufferIdentifier);

    class ReplayerDelegate : public WebCore::DisplayList::Replayer::Delegate {

    public:

152 ReplayerDelegate(WebCore::ImageBuffer&, RemoteRenderingBackend&~~, GPUConnectionToWebProcess&~~);

152 ReplayerDelegate(WebCore::ImageBuffer&, RemoteRenderingBackend&);

    private:

        bool apply(WebCore::DisplayList::ItemHandle, WebCore::GraphicsContext&) final;

        void didCreateMaskImageBuffer(WebCore::ImageBuffer&) final;

        void didResetMaskImageBuffer() final;

158 void recordResourceUse(WebCore::RenderingResourceIdentifier) final;

        WebCore::ImageBuffer& m_destination;

        RemoteRenderingBackend& m_remoteRenderingBackend;

161 Ref<GPUConnectionToWebProcess> m_gpuConnectionToWebProcess;

    };

    struct PendingWakeupInformation {

Source/WebKit/GPUProcess/graphics/RemoteRenderingBackend.messages.in

@@messages -> RemoteRenderingBackend {

    CacheFont(IPC::FontReference font)

    DeleteAllFonts()

    DidCreateSharedDisplayListHandle(WebCore::DisplayList::ItemBufferIdentifier identifier, WebKit::SharedMemory::IPCHandle handle, WebCore::RenderingResourceIdentifier destinationBufferIdentifier)

39 ReleaseRemoteResource(WebCore::RenderingResourceIdentifier renderingResourceIdentifier)

39 ReleaseRemoteResource(WebCore::RenderingResourceIdentifier renderingResourceIdentifier, uint64_t useCount)

}

#endif // ENABLE(GPU_PROCESS)

Source/WebKit/GPUProcess/graphics/RemoteResourceCache.cpp

@@using namespace WebCore;

void RemoteResourceCache::cacheImageBuffer(Ref<ImageBuffer>&& imageBuffer)

{

36 auto addResult = m_imageBuffers.add(imageBuffer->renderingResourceIdentifier(), WTFMove(imageBuffer));

37 ASSERT_UNUSED(addResult, addResult.isNewEntry);

    auto renderingResourceIdentifier = imageBuffer->renderingResourceIdentifier();

    m_imageBuffers.add(renderingResourceIdentifier, WTFMove(imageBuffer));

    ensureResourceUseCounter(renderingResourceIdentifier);

}

ImageBuffer* RemoteResourceCache::cachedImageBuffer(RenderingResourceIdentifier renderingResourceIdentifier)

@@ImageBuffer* RemoteResourceCache::cachedImageBuffer(RenderingResourceIdentifier

void RemoteResourceCache::cacheNativeImage(Ref<NativeImage>&& image)

{

47 auto addResult = m_nativeImages.add(image->renderingResourceIdentifier(), WTFMove(image));

48 ASSERT_UNUSED(addResult, addResult.isNewEntry);

    auto renderingResourceIdentifier = image->renderingResourceIdentifier();

    m_nativeImages.add(renderingResourceIdentifier, WTFMove(image));

    ensureResourceUseCounter(renderingResourceIdentifier);

}

void RemoteResourceCache::cacheFont(Ref<Font>&& font)

{

53 auto addResult = m_fonts.add(font->renderingResourceIdentifier(), WTFMove(font));

54 ASSERT_UNUSED(addResult, addResult.isNewEntry);

    auto renderingResourceIdentifier = font->renderingResourceIdentifier();

    m_fonts.add(renderingResourceIdentifier, WTFMove(font));

    ensureResourceUseCounter(renderingResourceIdentifier);

}

void RemoteResourceCache::ensureResourceUseCounter(RenderingResourceIdentifier renderingResourceIdentifier)

{

    auto result = m_resourceUseCounters.add(renderingResourceIdentifier, ResourceUseCounter { });

    if (!result.isNewEntry) {

        auto& state = result.iterator->value.state;

        RELEASE_ASSERT(state == ResourceState::ToBeDeleted);

        state = ResourceState::Alive;

    }

}

void RemoteResourceCache::deleteAllFonts()

@@void RemoteResourceCache::deleteAllFonts()

    m_fonts.clear();

}

62 v~~oid~~ RemoteResourceCache::~~rele~~aseRemoteResource(RenderingResourceIdentifier renderingResourceIdentifier)

78bool RemoteResourceCache::maybeRemoveResource(RenderingResourceIdentifier renderingResourceIdentifier, ResourceUseCountersMap::iterator& iterator)

6379{

    // It's actually okay if useCount > 0.

    // It just means that there's a cache message pending, and we've eagerly used a previously-cached resource.

    // Which is fine.

    // FIXME: Maybe we shouldn't actually remove the resource in this situation?

    // We may actually be able to get a head start on some display list items if they're ready for us and we have the resources they depend on.

    auto& value = iterator->value;

    if (value.state == ResourceState::Alive || value.useOrPendingCount < 0)

        return true;

    if (!value.useOrPendingCount)

        m_resourceUseCounters.remove(iterator);

6493 if (m_imageBuffers.remove(renderingResourceIdentifier))

65 return;

94 return true;

6695 if (m_nativeImages.remove(renderingResourceIdentifier))

67 return;

96 return true;

6897 if (m_fonts.remove(renderingResourceIdentifier))

69 return;

98 return true;

99

70100 // Caching the remote resource should have happened before releasing it.

71 ASSERT_NOT_REACHED();

    return false;

}

void RemoteResourceCache::recordResourceUse(RenderingResourceIdentifier renderingResourceIdentifier)

{

    auto iterator = m_resourceUseCounters.find(renderingResourceIdentifier);

    ASSERT(iterator != m_resourceUseCounters.end());

    ResourceUseCounter& useCounter = iterator->value;

    ++useCounter.useOrPendingCount;

    maybeRemoveResource(renderingResourceIdentifier, iterator);

}

bool RemoteResourceCache::releaseRemoteResource(RenderingResourceIdentifier renderingResourceIdentifier, uint64_t useCount)

{

    auto iterator = m_resourceUseCounters.find(renderingResourceIdentifier);

    if (iterator == m_resourceUseCounters.end())

        return false;

    ResourceUseCounter& useCounter = iterator->value;

    useCounter.state = ResourceState::ToBeDeleted;

    useCounter.useOrPendingCount -= useCount;

    return maybeRemoveResource(renderingResourceIdentifier, iterator);

}

} // namespace WebKit

Source/WebKit/GPUProcess/graphics/RemoteResourceCache.h

@@public:

    void cacheNativeImage(Ref<WebCore::NativeImage>&&);

    void cacheFont(Ref<WebCore::Font>&&);

    void deleteAllFonts();

49 void releaseRemoteResource(WebCore::RenderingResourceIdentifier);

49 bool releaseRemoteResource(WebCore::RenderingResourceIdentifier, uint64_t useCount);

50 void recordResourceUse(WebCore::RenderingResourceIdentifier);

    const WebCore::ImageBufferHashMap& imageBuffers() const { return m_imageBuffers; }

    const WebCore::NativeImageHashMap& nativeImages() const { return m_nativeImages; }

    const WebCore::FontRenderingResourceMap& fonts() const { return m_fonts; }

private:

    // Because the cache/release messages are sent asynchronously from the display list items which

    // reference the resources, it's totally possible that we see a release message before we've

    // executed all the display list items which reference the resource. The web process tells us

    // how many display list items will reference this resource, and we defer deletion of the resource

    // until we execute that many display list items. It's actually a bit worse than this, though,

    // because we may actually see a *new* cache message during the time when deletion is deferred.

    //

    // We can only safely delete a resource when:

    // 1. All the cache messages have an accompanying release message, and

    // 2. We've processed as many display list items that reference a particular resource as the web

    // process has encoded.

    enum class ResourceState {

        Alive,

        ToBeDeleted

    };

    struct ResourceUseCounter {

        ResourceState state { ResourceState::Alive };

        int64_t useOrPendingCount { 0 };

    };

    using ResourceUseCountersMap = HashMap<WebCore::RenderingResourceIdentifier, ResourceUseCounter>;

    bool maybeRemoveResource(WebCore::RenderingResourceIdentifier, ResourceUseCountersMap::iterator&);

    void ensureResourceUseCounter(WebCore::RenderingResourceIdentifier);

    WebCore::ImageBufferHashMap m_imageBuffers;

    WebCore::NativeImageHashMap m_nativeImages;

    WebCore::FontRenderingResourceMap m_fonts;

84

85 ResourceUseCountersMap m_resourceUseCounters;

};

} // namespace WebKit

Source/WebKit/WebProcess/GPU/graphics/RemoteImageBufferProxy.h

@@public:

        flushDrawingContext();

        m_remoteRenderingBackendProxy->remoteResourceCacheProxy().releaseImageBuffer(m_renderingResourceIdentifier);

75 m_remoteRenderingBackendProxy->releaseRemoteResource(m_renderingResourceIdentifier);

75 m_remoteRenderingBackendProxy->releaseRemoteResource(m_renderingResourceIdentifier, 0); // FIXME: Pass the real use count here.

    }

    ImageBufferBackendHandle createImageBufferBackendHandle()

Source/WebKit/WebProcess/GPU/graphics/RemoteRenderingBackendProxy.cpp

@@void RemoteRenderingBackendProxy::deleteAllFonts()

    send(Messages::RemoteRenderingBackend::DeleteAllFonts(), renderingBackendIdentifier(), IPC::SendOption::DispatchMessageEvenWhenWaitingForSyncReply);

}

252 void RemoteRenderingBackendProxy::releaseRemoteResource(RenderingResourceIdentifier renderingResourceIdentifier)

252void RemoteRenderingBackendProxy::releaseRemoteResource(RenderingResourceIdentifier renderingResourceIdentifier, uint64_t useCount)

{

    if (renderingResourceIdentifier == m_currentDestinationImageBufferIdentifier)

        m_currentDestinationImageBufferIdentifier = std::nullopt;

257 send(Messages::RemoteRenderingBackend::ReleaseRemoteResource(renderingResourceIdentifier), renderingBackendIdentifier(), IPC::SendOption::DispatchMessageEvenWhenWaitingForSyncReply);

257 send(Messages::RemoteRenderingBackend::ReleaseRemoteResource(renderingResourceIdentifier, useCount), renderingBackendIdentifier(), IPC::SendOption::DispatchMessageEvenWhenWaitingForSyncReply);

}

void RemoteRenderingBackendProxy::didCreateImageBufferBackend(ImageBufferBackendHandle handle, RenderingResourceIdentifier renderingResourceIdentifier)

Source/WebKit/WebProcess/GPU/graphics/RemoteRenderingBackendProxy.h

@@public:

    void cacheNativeImage(const ShareableBitmap::Handle&, WebCore::RenderingResourceIdentifier);

    void cacheFont(Ref<WebCore::Font>&&);

    void deleteAllFonts();

107 void releaseRemoteResource(WebCore::RenderingResourceIdentifier);

107 void releaseRemoteResource(WebCore::RenderingResourceIdentifier, uint64_t useCount);

    enum class DidReceiveBackendCreationResult : bool {

        ReceivedAnyResponse,

Source/WebKit/WebProcess/GPU/graphics/RemoteResourceCacheProxy.cpp

@@void RemoteResourceCacheProxy::releaseNativeImage(RenderingResourceIdentifier re

    if (!m_nativeImages.remove(renderingResourceIdentifier))

        return;

124 // Tell the GPU process to remove this resource.

125 m_remoteRenderingBackendProxy.releaseRemoteResource(renderingResourceIdentifier);

124 m_remoteRenderingBackendProxy.releaseRemoteResource(renderingResourceIdentifier, 0); // FIXME: Pass the real use count here.

}

void RemoteResourceCacheProxy::prepareForNextRenderingUpdate()

@@void RemoteResourceCacheProxy::prepareForNextRenderingUpdate()

void RemoteResourceCacheProxy::clearFontMap()

{

    for (auto& item : m_fontIdentifierToLastRenderingUpdateVersionMap)

137 m_remoteRenderingBackendProxy.releaseRemoteResource(item.key);

136 m_remoteRenderingBackendProxy.releaseRemoteResource(item.key, 0); // FIXME: Pass the real use count here.

    m_fontIdentifierToLastRenderingUpdateVersionMap.clear();

    m_numberOfFontsUsedInCurrentRenderingUpdate = 0;

}

@@void RemoteResourceCacheProxy::finalizeRenderingUpdateForFonts()

    for (auto& item : m_fontIdentifierToLastRenderingUpdateVersionMap) {

        if (m_currentRenderingUpdateVersion - item.value >= minimumRenderingUpdateCountToKeepFontAlive) {

            toRemove.add(item.key);

155 m_remoteRenderingBackendProxy.releaseRemoteResource(item.key);

154 m_remoteRenderingBackendProxy.releaseRemoteResource(item.key, 0); // FIXME: Pass the real use count here.

        }

    }