Bug 218891 – element.rect doesn't yield correct co-ordinates on iOS

• Component: WebDriver
• Status: NEW

[Luke Hill]

I couldn't find the right location now to submit these.

After initially submitting on selenium, then on appium, I'm now here. `element.rect` does not yield correct results on iOS compared to other browsers

For full repro see: https://github.com/SeleniumHQ/selenium/issues/8869 / https://github.com/appium/ruby_lib_core/issues/285

```
Bad candidates

iPhone 8 - iOS 12 - #<struct Selenium::WebDriver::Rectangle x=17, **y=1**, width=293, height=48>
iPhone 8 - iOS 13 - #<struct Selenium::WebDriver::Rectangle x=17, **y=1**, width=293, height=48>
iPhone 11 - iOS 13 - #<struct Selenium::WebDriver::Rectangle x=17, **y=1**, width=332, height=48>
Good candidates

Windows 7 - Chrome v83 - #<struct Selenium::WebDriver::Rectangle x=17, y=**1032**, width=750, height=48>
Windows 10 - Chrome v86 - #<struct Selenium::WebDriver::Rectangle x=17, y=**1060**, width=750, height=48>
Windows 10 - Firefox v80 - #<struct Selenium::WebDriver::Rectangle x=17.0, y=**1010.0**, width=750.0, height=48.0>
OSX Mojave - Safari 12 - #<struct Selenium::WebDriver::Rectangle x=17, y=**820**, width=750, height=48>
```

[Luke Hill]

NB: During some parts of the testing. It's even possible to get the y ordinate to be negative in iOS. So something is definitely going wrong here!!

[Radar WebKit Bug Importer]

<rdar://problem/71468283>

[Blaze Burg]

Hello Luke, thanks for writing.

It's not quite clear to me how you are scrolling. Could you please attach a test case so that there is no ambiguity?

In general, iOS coordinates are computed and returned in visual viewport space, but the Get Element Rect command explicitly wants to add the scrollY to this coordinate system. In this case the scrollY of ~1000 is missing, so this indeed looks like a driver bug.

[Blaze Burg]

From reading the code, it looks like we need to change WebAutomationSessionProxy::computeElementLayout() for the CoordinateSystem::Page case, so that it shifts the element bounds by window.scrollX and window.scrollY.

Spec language is here: https://w3c.github.io/webdriver/#dfn-calculate-the-absolute-position

[Luke Hill]

I do use a JS "hack" to scroll. Which perhaps isn't computing well with the iOS Driver.

But it works (i.e. it does scroll). For ref I do a window.scrollTo(10000,0) call.

I "know" for iOS I could maybe do something different, but we don't care how the user scrolls - As for that webpage, we want to test it "stickies" the OISC Warning. To me this was the "best" way to test the sticky behaviour.

I.e. the rect should be moved from its starting position. So my test grabs the starting y value, scrolls down, asserts it has changed. But for iOS it goes all weird. It works fine on Mojave 12 Safari so it suggests it's not globally broken with the safaridriver. So I'm not sure.

[Blaze Burg]

Created attachment 426962 [details]
[WIP] Patch v1.0

[Darin Adler]

Comment on attachment 426962 [details]
[WIP] Patch v1.0

View in context: https://bugs.webkit.org/attachment.cgi?id=426962&action=review

> Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp:662
> +        // XXX

XXX?

[Simon Fraser (smfr)]

Comment on attachment 426962 [details]
[WIP] Patch v1.0

View in context: https://bugs.webkit.org/attachment.cgi?id=426962&action=review

> Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp:668
> +        scrollOffset.scale(2); // convertRectFromFrameClientToRootView subtracts scrollX/Y and we need to add it back and then again so it's positive

This is very weird.