Bug 227775 – [Live Text] Selection is misaligned on some images on twitter.com

• Status: RESOLVED
• Resolution: FIXED
• Component: Platform

[Wenson Hsieh]

rdar://77142364

[Wenson Hsieh]

Created attachment 433111 [details]
Patch

[Tim Horton]

Comment on attachment 433111 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=433111&action=review

> Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:43
>  RefPtr<ShareableBitmap> createShareableBitmap(RenderImage& renderImage, std::optional<FloatSize> screenSizeInPixels, AllowAnimatedImages allowAnimatedImages)

This seems like a somewhat bizarre behavior to impose on all clients of `createShareableBitmap for RenderImage`, suddenly giving content from a wide variety of different renderers...

> Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:53
> +        auto imageBuffer = snapshotFrameRect(frame.get(), snapshotRect, { snapshotFlags, PixelFormat::BGRA8, DestinationColorSpace::SRGB() });

... and this of course is quite limited (transforms, some kinds of video, etc. won't work); but probably in ways that don't affect your case.

[Wenson Hsieh]

Comment on attachment 433111 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=433111&action=review

>> Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:43
>>  RefPtr<ShareableBitmap> createShareableBitmap(RenderImage& renderImage, std::optional<FloatSize> screenSizeInPixels, AllowAnimatedImages allowAnimatedImages)
> 
> This seems like a somewhat bizarre behavior to impose on all clients of `createShareableBitmap for RenderImage`, suddenly giving content from a wide variety of different renderers...

From looking over the call sites of `createShareableBitmap`, I think this is compatible with all of the callers. The one in `WebHitTestResultData`, for instance, is only used for visual lookup, and the ones in WebPage are either about Live Text or node snapshotting, via the `includeSnapshot` flag on position information requests.

That said, perhaps this helper function deserves a name that clarifies this, like `createShareableBitmapFromVisibleImage` (I can't really think of a good name :/). Or maybe the snapshotting part of this can be moved out into a separate function that's specific to text recognition.

>> Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:53
>> +        auto imageBuffer = snapshotFrameRect(frame.get(), snapshotRect, { snapshotFlags, PixelFormat::BGRA8, DestinationColorSpace::SRGB() });
> 
> ... and this of course is quite limited (transforms, some kinds of video, etc. won't work); but probably in ways that don't affect your case.

Indeed.

[Tim Horton]

(In reply to Wenson Hsieh from comment #3)
> Comment on attachment 433111 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=433111&action=review
> 
> >> Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:43
> >>  RefPtr<ShareableBitmap> createShareableBitmap(RenderImage& renderImage, std::optional<FloatSize> screenSizeInPixels, AllowAnimatedImages allowAnimatedImages)
> > 
> > This seems like a somewhat bizarre behavior to impose on all clients of `createShareableBitmap for RenderImage`, suddenly giving content from a wide variety of different renderers...
> 
> From looking over the call sites of `createShareableBitmap`, I think this is
> compatible with all of the callers. The one in `WebHitTestResultData`, for
> instance, is only used for visual lookup, and the ones in WebPage are either
> about Live Text or node snapshotting, via the `includeSnapshot` flag on
> position information requests.

Does seem to be the case, but...

> That said, perhaps this helper function deserves a name that clarifies this,
> like `createShareableBitmapFromVisibleImage` (I can't really think of a good
> name :/). Or maybe the snapshotting part of this can be moved out into a
> separate function that's specific to text recognition.

... it does seem like it needs a better name. You could imagine e.g. the Web Inspector's node snapshotting functionality using something like this method, and it would be pretty weird if you got content from a different node!

> >> Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:53
> >> +        auto imageBuffer = snapshotFrameRect(frame.get(), snapshotRect, { snapshotFlags, PixelFormat::BGRA8, DestinationColorSpace::SRGB() });
> > 
> > ... and this of course is quite limited (transforms, some kinds of video, etc. won't work); but probably in ways that don't affect your case.
> 
> Indeed.

[Wenson Hsieh]

(In reply to Tim Horton from comment #4)
> (In reply to Wenson Hsieh from comment #3)
> > Comment on attachment 433111 [details]
> > Patch
> > 
> > View in context:
> > https://bugs.webkit.org/attachment.cgi?id=433111&action=review
> > 
> > >> Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:43
> > >>  RefPtr<ShareableBitmap> createShareableBitmap(RenderImage& renderImage, std::optional<FloatSize> screenSizeInPixels, AllowAnimatedImages allowAnimatedImages)
> > > 
> > > This seems like a somewhat bizarre behavior to impose on all clients of `createShareableBitmap for RenderImage`, suddenly giving content from a wide variety of different renderers...
> > 
> > From looking over the call sites of `createShareableBitmap`, I think this is
> > compatible with all of the callers. The one in `WebHitTestResultData`, for
> > instance, is only used for visual lookup, and the ones in WebPage are either
> > about Live Text or node snapshotting, via the `includeSnapshot` flag on
> > position information requests.
> 
> Does seem to be the case, but...
> 
> > That said, perhaps this helper function deserves a name that clarifies this,
> > like `createShareableBitmapFromVisibleImage` (I can't really think of a good
> > name :/). Or maybe the snapshotting part of this can be moved out into a
> > separate function that's specific to text recognition.
> 
> ... it does seem like it needs a better name. You could imagine e.g. the Web
> Inspector's node snapshotting functionality using something like this
> method, and it would be pretty weird if you got content from a different
> node!

After giving it some more thought, I decided to add an optional flag parameter (e.g. `enum class UseSnapshotForTransparentImages : bool`) that would be `No` by default. Only the Live Text codepaths would then explicitly pass in `Yes`. Hopefully that would address this scenario?

> 
> > >> Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:53
> > >> +        auto imageBuffer = snapshotFrameRect(frame.get(), snapshotRect, { snapshotFlags, PixelFormat::BGRA8, DestinationColorSpace::SRGB() });
> > > 
> > > ... and this of course is quite limited (transforms, some kinds of video, etc. won't work); but probably in ways that don't affect your case.
> > 
> > Indeed.

[Wenson Hsieh]

Created attachment 433137 [details]
Add an option struct

[EWS]

Committed r279751 (239527@main): <https://commits.webkit.org/239527@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 433137 [details].