RESOLVED FIXED 179129
Web Automation: inViewCenterPoint should not include topContentInset when computed in viewport coordinates
https://bugs.webkit.org/show_bug.cgi?id=179129
Summary Web Automation: inViewCenterPoint should not include topContentInset when com...
Blaze Burg
Reported 2017-11-01 11:02:23 PDT
This blocks adoption of inViewCenterPoint algorithm by safaridriver. Adopting it is a regression without fixing this.
Attachments
Patch (2.15 KB, patch)
2017-11-01 11:15 PDT, Blaze Burg
no flags
Patch (26.22 KB, patch)
2017-11-07 20:39 PST, Blaze Burg
no flags
Patch (26.84 KB, patch)
2017-11-13 11:29 PST, Blaze Burg
no flags
Blaze Burg
Comment 1 2017-11-01 11:15:43 PDT
Radar WebKit Bug Importer
Comment 2 2017-11-01 11:16:12 PDT
Simon Fraser (smfr)
Comment 3 2017-11-01 12:04:34 PDT
Comment on attachment 325595 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=325595&action=review > Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp:578 > if (containerElement) { The code above uses coreElement->clientRect(). Note the comment above it: // Note that this is not web-exposed, and does not use the same coordinate system as getBoundingClientRect() and friends. IntRect Element::clientRect() const > Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp:580 > inViewCenter = WebCore::IntPoint(coreFrameView->clientToDocumentPoint(clientCenterPoint.value())); So i think clientToDocumentPoint is the wrong way to convert this point. > Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp:583 > + inViewCenter.value().moveBy(WebCore::IntPoint(0, -coreFrameView->topContentInset())); I never want to see this kind of manual coordinate conversion. We should always go through FrameView helpers like "convert foo to bar".
Blaze Burg
Comment 4 2017-11-01 17:31:11 PDT
Comment on attachment 325595 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=325595&action=review >> Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp:578 >> if (containerElement) { > > The code above uses coreElement->clientRect(). Note the comment above it: > > // Note that this is not web-exposed, and does not use the same coordinate system as getBoundingClientRect() and friends. > IntRect Element::clientRect() const Okay, this should be easy to convert to snappedIntRect(boundingClientRect())... >> Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp:580 >> inViewCenter = WebCore::IntPoint(coreFrameView->clientToDocumentPoint(clientCenterPoint.value())); > > So i think clientToDocumentPoint is the wrong way to convert this point. ...and then clientToDocumentPoint will make sense. >> Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp:583 >> + inViewCenter.value().moveBy(WebCore::IntPoint(0, -coreFrameView->topContentInset())); > > I never want to see this kind of manual coordinate conversion. We should always go through FrameView helpers like "convert foo to bar". Let me preface by saying I'm super ignorant of layout code. So please humor me, I want to understand better :) I'm not aware of any existing conversions that go from content rects to what are essentially client rects. The most sensible name I could think up is contentsToClient
Blaze Burg
Comment 5 2017-11-01 17:33:56 PDT
(In reply to Brian Burg from comment #4) > Comment on attachment 325595 [details] > Patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=325595&action=review > > >> Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp:578 > >> if (containerElement) { > > > > The code above uses coreElement->clientRect(). Note the comment above it: > > > > // Note that this is not web-exposed, and does not use the same coordinate system as getBoundingClientRect() and friends. > > IntRect Element::clientRect() const > > Okay, this should be easy to convert to > snappedIntRect(boundingClientRect())... > > >> Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp:580 > >> inViewCenter = WebCore::IntPoint(coreFrameView->clientToDocumentPoint(clientCenterPoint.value())); > > > > So i think clientToDocumentPoint is the wrong way to convert this point. > > ...and then clientToDocumentPoint will make sense. > > >> Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp:583 > >> + inViewCenter.value().moveBy(WebCore::IntPoint(0, -coreFrameView->topContentInset())); > > > > I never want to see this kind of manual coordinate conversion. We should always go through FrameView helpers like "convert foo to bar". > > Let me preface by saying I'm super ignorant of layout code. So please humor > me, I want to understand better :) > > I'm not aware of any existing conversions that go from content rects to what > are essentially client rects. The most sensible name I could think up is > contentsToClient Sorry, comment got clipped off. I looked at this again (and again and again). My notes from today: For the clientRect (viewport case), I think we can just use the client rect as-is to send back to WebDriver land. In the non-viewport case, I just need to go from client to document/contents rect, I think? For the inViewCenterPoint (viewport case), we already return a client point so I don't think anything needs to be done, which is why the code is wrong right now. For the non-viewport case, we need to convert to document/contents rect as well.
Simon Fraser (smfr)
Comment 6 2017-11-01 17:54:17 PDT
My general thoughts: This code has an "useViewportCoordinates" branch, and the question there is what that means. Ideally it would explicitly refer to the visual or layout viewports (probably the second). For that you'd use something like FrameView::documentToClientPoint() because "client" coordinates are nominally layout viewport coordinates. Do this after you've converted the point to the root document/view. If not viewport coordinates, what are they?
Simon Fraser (smfr)
Comment 7 2017-11-01 17:55:53 PDT
Also, generally contentsToView() is the thing that deals with topContentInset (via documentScrollPositionRelativeToViewOrigin).
Blaze Burg
Comment 8 2017-11-02 12:59:39 PDT
(In reply to Simon Fraser (smfr) from comment #6) > My general thoughts: > > This code has an "useViewportCoordinates" branch, and the question there is > what that means. Ideally it would explicitly refer to the visual or layout > viewports (probably the second). For that you'd use something like > FrameView::documentToClientPoint() because "client" coordinates are > nominally layout viewport coordinates. Do this after you've converted the > point to the root document/view. > > If not viewport coordinates, what are they? Someday, if we support WebDriver for iOS, then this will need to support returning visual viewport coordinates so that simulated OS events hit on the right thing. My plan is to proceed to make useViewportCoordinates mean "layout viewport" for now, since that's the only thing I can reasonably test on Mac. When the time comes, and it's reasonable to test in a context where visual viewports are used, I can add a new VisualViewport mode for the command.
Blaze Burg
Comment 9 2017-11-07 20:39:12 PST
Simon Fraser (smfr)
Comment 10 2017-11-10 12:32:04 PST
Comment on attachment 326297 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=326297&action=review Is this covered by WPT? > Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.messages.in:33 > + ComputeElementLayout(uint64_t pageID, uint64_t frameID, String nodeHandle, bool scrollIntoViewIfNeeded, uint32_t coordinateSystem, uint64_t callbackID) Can't we use enums?
Blaze Burg
Comment 11 2017-11-10 13:47:42 PST
Comment on attachment 326297 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=326297&action=review >> Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.messages.in:33 >> + ComputeElementLayout(uint64_t pageID, uint64_t frameID, String nodeHandle, bool scrollIntoViewIfNeeded, uint32_t coordinateSystem, uint64_t callbackID) > > Can't we use enums? I had assumed not, but if it works then I'll switch it.
Blaze Burg
Comment 12 2017-11-10 13:51:18 PST
(In reply to Simon Fraser (smfr) from comment #10) > Comment on attachment 326297 [details] > Patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=326297&action=review > > Is this covered by WPT? Not right now. It progresses about 20 Selenium tests for safaridriver. If we shim WPT browser-internal methods to reuse this code, then it would be tested that way.
Blaze Burg
Comment 13 2017-11-13 11:29:01 PST
Created attachment 326774 [details] Patch Re-EWS after using enum instead of int32_t.
Blaze Burg
Comment 14 2017-11-13 15:21:37 PST
Note You need to log in before you can comment on or make changes to this bug.