Bug 227775 - [Live Text] Selection is misaligned on some images on twitter.com
Summary: [Live Text] Selection is misaligned on some images on twitter.com
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Platform (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Wenson Hsieh
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2021-07-07 14:59 PDT by Wenson Hsieh
Modified: 2021-07-08 13:59 PDT (History)
10 users (show)

See Also:


Attachments
Patch (11.51 KB, patch)
2021-07-07 20:33 PDT, Wenson Hsieh
no flags Details | Formatted Diff | Diff
Add an option struct (16.68 KB, patch)
2021-07-08 09:28 PDT, Wenson Hsieh
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Wenson Hsieh 2021-07-07 14:59:33 PDT
rdar://77142364
Comment 1 Wenson Hsieh 2021-07-07 20:33:06 PDT
Created attachment 433111 [details]
Patch
Comment 2 Tim Horton 2021-07-07 23:10:14 PDT
Comment on attachment 433111 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=433111&action=review

> Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:43
>  RefPtr<ShareableBitmap> createShareableBitmap(RenderImage& renderImage, std::optional<FloatSize> screenSizeInPixels, AllowAnimatedImages allowAnimatedImages)

This seems like a somewhat bizarre behavior to impose on all clients of `createShareableBitmap for RenderImage`, suddenly giving content from a wide variety of different renderers...

> Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:53
> +        auto imageBuffer = snapshotFrameRect(frame.get(), snapshotRect, { snapshotFlags, PixelFormat::BGRA8, DestinationColorSpace::SRGB() });

... and this of course is quite limited (transforms, some kinds of video, etc. won't work); but probably in ways that don't affect your case.
Comment 3 Wenson Hsieh 2021-07-07 23:20:32 PDT
Comment on attachment 433111 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=433111&action=review

>> Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:43
>>  RefPtr<ShareableBitmap> createShareableBitmap(RenderImage& renderImage, std::optional<FloatSize> screenSizeInPixels, AllowAnimatedImages allowAnimatedImages)
> 
> This seems like a somewhat bizarre behavior to impose on all clients of `createShareableBitmap for RenderImage`, suddenly giving content from a wide variety of different renderers...

From looking over the call sites of `createShareableBitmap`, I think this is compatible with all of the callers. The one in `WebHitTestResultData`, for instance, is only used for visual lookup, and the ones in WebPage are either about Live Text or node snapshotting, via the `includeSnapshot` flag on position information requests.

That said, perhaps this helper function deserves a name that clarifies this, like `createShareableBitmapFromVisibleImage` (I can't really think of a good name :/). Or maybe the snapshotting part of this can be moved out into a separate function that's specific to text recognition.

>> Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:53
>> +        auto imageBuffer = snapshotFrameRect(frame.get(), snapshotRect, { snapshotFlags, PixelFormat::BGRA8, DestinationColorSpace::SRGB() });
> 
> ... and this of course is quite limited (transforms, some kinds of video, etc. won't work); but probably in ways that don't affect your case.

Indeed.
Comment 4 Tim Horton 2021-07-07 23:36:21 PDT
(In reply to Wenson Hsieh from comment #3)
> Comment on attachment 433111 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=433111&action=review
> 
> >> Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:43
> >>  RefPtr<ShareableBitmap> createShareableBitmap(RenderImage& renderImage, std::optional<FloatSize> screenSizeInPixels, AllowAnimatedImages allowAnimatedImages)
> > 
> > This seems like a somewhat bizarre behavior to impose on all clients of `createShareableBitmap for RenderImage`, suddenly giving content from a wide variety of different renderers...
> 
> From looking over the call sites of `createShareableBitmap`, I think this is
> compatible with all of the callers. The one in `WebHitTestResultData`, for
> instance, is only used for visual lookup, and the ones in WebPage are either
> about Live Text or node snapshotting, via the `includeSnapshot` flag on
> position information requests.

Does seem to be the case, but...

> That said, perhaps this helper function deserves a name that clarifies this,
> like `createShareableBitmapFromVisibleImage` (I can't really think of a good
> name :/). Or maybe the snapshotting part of this can be moved out into a
> separate function that's specific to text recognition.

... it does seem like it needs a better name. You could imagine e.g. the Web Inspector's node snapshotting functionality using something like this method, and it would be pretty weird if you got content from a different node!

> >> Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:53
> >> +        auto imageBuffer = snapshotFrameRect(frame.get(), snapshotRect, { snapshotFlags, PixelFormat::BGRA8, DestinationColorSpace::SRGB() });
> > 
> > ... and this of course is quite limited (transforms, some kinds of video, etc. won't work); but probably in ways that don't affect your case.
> 
> Indeed.
Comment 5 Wenson Hsieh 2021-07-08 07:55:41 PDT
(In reply to Tim Horton from comment #4)
> (In reply to Wenson Hsieh from comment #3)
> > Comment on attachment 433111 [details]
> > Patch
> > 
> > View in context:
> > https://bugs.webkit.org/attachment.cgi?id=433111&action=review
> > 
> > >> Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:43
> > >>  RefPtr<ShareableBitmap> createShareableBitmap(RenderImage& renderImage, std::optional<FloatSize> screenSizeInPixels, AllowAnimatedImages allowAnimatedImages)
> > > 
> > > This seems like a somewhat bizarre behavior to impose on all clients of `createShareableBitmap for RenderImage`, suddenly giving content from a wide variety of different renderers...
> > 
> > From looking over the call sites of `createShareableBitmap`, I think this is
> > compatible with all of the callers. The one in `WebHitTestResultData`, for
> > instance, is only used for visual lookup, and the ones in WebPage are either
> > about Live Text or node snapshotting, via the `includeSnapshot` flag on
> > position information requests.
> 
> Does seem to be the case, but...
> 
> > That said, perhaps this helper function deserves a name that clarifies this,
> > like `createShareableBitmapFromVisibleImage` (I can't really think of a good
> > name :/). Or maybe the snapshotting part of this can be moved out into a
> > separate function that's specific to text recognition.
> 
> ... it does seem like it needs a better name. You could imagine e.g. the Web
> Inspector's node snapshotting functionality using something like this
> method, and it would be pretty weird if you got content from a different
> node!

After giving it some more thought, I decided to add an optional flag parameter (e.g. `enum class UseSnapshotForTransparentImages : bool`) that would be `No` by default. Only the Live Text codepaths would then explicitly pass in `Yes`. Hopefully that would address this scenario?

> 
> > >> Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:53
> > >> +        auto imageBuffer = snapshotFrameRect(frame.get(), snapshotRect, { snapshotFlags, PixelFormat::BGRA8, DestinationColorSpace::SRGB() });
> > > 
> > > ... and this of course is quite limited (transforms, some kinds of video, etc. won't work); but probably in ways that don't affect your case.
> > 
> > Indeed.
Comment 6 Wenson Hsieh 2021-07-08 09:28:26 PDT
Created attachment 433137 [details]
Add an option struct
Comment 7 EWS 2021-07-08 13:59:41 PDT
Committed r279751 (239527@main): <https://commits.webkit.org/239527@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 433137 [details].