Bug 133180 – Pointer events should allow for device-pixel accuracy

• Component: UI Events
• Status: RESOLVED FIXED

[Rick Byers]

Various APIs are attempting to move to floating-point co-ordinates (eg. see bug 132895).  Perhaps TouchEvent co-ordinates should also be float/double instead of long?  http://trac.webkit.org/browser/trunk/Source/WebCore/dom/Touch.idl

This may be valuable on high-dpi devices.  Eg. if scrolling is happening in physical pixel units, it should be possible to build JS applications that move things around just as smoothly (as is possible in native iOS apps).  Eg. imagine an app that does a custom pull effect when pulling down when the document is at scroll offset 0.  Today when moving slowly this is noticeably jumpier on retina devices than the buttery smooth browser scrolling you get moving in the other direction.

I'm in the process of switching blink to doubles:
https://code.google.com/p/chromium/issues/detail?id=323935

The TouchEvents community group is also discussing this:
http://lists.w3.org/Archives/Public/public-touchevents/2014May/0006.html

[Benjamin Poulain]

I agree, we should try this too.

Do you have a test page where it is jumpy for reference?

[Rick Byers]

Thanks!

It can be pretty subtle in practice (especially if the finger is moving quickly), but here's an easy to visualize example: http://www.rbyers.net/paint.html#points - a drawing program that draws on physical pixel point per touchmove event.  See https://code.google.com/p/chromium/issues/detail?id=323935#c34 for screenshots comparing integer to floating-point co-ordinates on a nexus 5 (which has DSF=3).

[Rick Byers]

The difference is also pretty noticeable when dragging something around slowly. eg: http://jsbin.com/idojig/2/quiet

[Rick Byers]

Assigning to Ben per #c1

[Bar Ziony]

Just wanted to ask if this was implemented already, and if not, if it in the works?

Thanks!

[Radar WebKit Bug Importer]

<rdar://problem/24597318>

[Dean Jackson]

http://www.rbyers.net/paint.html#points is a compelling demo

[Rick Byers]

Thanks Dean.  The drawing example is easiest to visualize but is fairly niche.  The really compelling scenarios IMHO are the dragging / custom scroll cases that are so common.  Most such existing pages got noticeably smoother (when dragging slowly) when we shipped this in chromium.

[Bar Ziony]

Hi Dean,

Is this planned for an upcoming release perhaps? Just wanted to get some updated info about this feature.

Thanks.

[Antoine Quint]

<rdar://problem/9564176>

[Antoine Quint]

Created attachment 281273 [details]
Patch

[Antoine Quint]

Created attachment 281277 [details]
Patch

[WebKit Commit Bot]

Attachment 281277 [details] did not pass style-queue:


ERROR: Source/WebKit2/Shared/ios/NativeWebTouchEventIOS.mm:99:  Comma should be at the beginning of the line in a member initialization list.  [whitespace/init] [4]
ERROR: Source/WebCore/ChangeLog:12:  You should remove the 'No new tests' and either add and list tests, or explain why no new tests were possible.  [changelog/nonewtests] [5]
Total errors found: 2 in 17 files


If any of these errors are false positives, please file a bug against check-webkit-style.

[Benjamin Poulain]

Comment on attachment 281277 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=281277&action=review

> Source/WebCore/platform/Widget.cpp:144
> +    if (!windowFraction.isEmpty()) {
> +        const int kFactor = 1000;
> +        IntPoint parentLineEnd = this->convertFromContainingWindow(flooredPoint + roundedIntSize(windowFraction.scaledBy(kFactor)));
> +        FloatSize parentFraction = (parentLineEnd - parentPoint).scaledBy(1.0f / kFactor);
> +        parentPoint.move(parentFraction);
> +    }

Any way to change convertFromContainingWindow() to handle floating point coordinates instead?

Linear interpolation of the decimal part is only gonna work for a small subset of transforms.

> Source/WebKit2/UIProcess/WebPageProxy.cpp:1944
> +        TrackingType touchPointTrackingType = m_scrollingCoordinatorProxy->eventTrackingTypeForPoint(roundedIntPoint(touchPoint.location()));

This is unfortunate.
I don't know how complicated it would be to add Region::contains() for FloatPoint.

[Antoine Quint]

(In reply to comment #14)
> Comment on attachment 281277 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=281277&action=review
> 
> > Source/WebCore/platform/Widget.cpp:144
> > +    if (!windowFraction.isEmpty()) {
> > +        const int kFactor = 1000;
> > +        IntPoint parentLineEnd = this->convertFromContainingWindow(flooredPoint + roundedIntSize(windowFraction.scaledBy(kFactor)));
> > +        FloatSize parentFraction = (parentLineEnd - parentPoint).scaledBy(1.0f / kFactor);
> > +        parentPoint.move(parentFraction);
> > +    }
> 
> Any way to change convertFromContainingWindow() to handle floating point
> coordinates instead?

We would have to change quite a few methods, but I don't see why we couldn't. I'm not certain this is warranted though.

> > Source/WebKit2/UIProcess/WebPageProxy.cpp:1944
> > +        TrackingType touchPointTrackingType = m_scrollingCoordinatorProxy->eventTrackingTypeForPoint(roundedIntPoint(touchPoint.location()));
> 
> This is unfortunate.
> I don't know how complicated it would be to add Region::contains() for
> FloatPoint.

I think we could change the signature of Region::contains(const IntPoint& point) to take in a FloatPoint and operate on float values instead. I'll look into this.

[Rick Byers]

(In reply to comment #15)
> (In reply to comment #14)
> > Comment on attachment 281277 [details]
> > Patch
> > 
> > View in context:
> > https://bugs.webkit.org/attachment.cgi?id=281277&action=review
> > 
> > > Source/WebCore/platform/Widget.cpp:144
> > > +    if (!windowFraction.isEmpty()) {
> > > +        const int kFactor = 1000;
> > > +        IntPoint parentLineEnd = this->convertFromContainingWindow(flooredPoint + roundedIntSize(windowFraction.scaledBy(kFactor)));
> > > +        FloatSize parentFraction = (parentLineEnd - parentPoint).scaledBy(1.0f / kFactor);
> > > +        parentPoint.move(parentFraction);
> > > +    }
> > 
> > Any way to change convertFromContainingWindow() to handle floating point
> > coordinates instead?
> 
> We would have to change quite a few methods, but I don't see why we
> couldn't. I'm not certain this is warranted though.

FWIW I felt really dirty doing this linear interpolation hack in blink (and left myself a FIXME to clean it up) but honestly after 2 years I haven't heard of a single example of anyone noticing it being less than perfect in the edge cases (rotated iframes mainly I guess).  I started to go down the path of fully supporting float but it exploded into a ton of work so I put it off.  I assumed we'd have other reasons to want fractional co-ordinate conversion between widgets but it's never come up since.  Maybe that was the wrong decision though - certainly feels wrong.

[Ge Yang]

Hey Guys, 

I'm trying to use the apple pencil on an iPad pro with canvas to replace native apps, this is when I realized that the Touch.clientX and clientY are quantized to integers. 

Floating point would be very important for this application! and I think using apple pencil to draw on iPad pros is going to be a common thing to do in the future. It would be a shame if we can't do this properly in the browser.

Thanks for all of the work!

Ge

[Brady Eidson]

Comment on attachment 281277 [details]
Patch

r-, as this has been pending review for over a year now. It is near-impossible that this patch still applies to trunk and unlikely to still be relevant in its current form.

[Jake Ingman]

I was wondering if there had been any progress addressing this bug. The lack of sub-pixel accuracy in touch events severely limits the capabilities of web-based drawing applications, and to my knowledge, there are no workarounds.

[Patrick H. Lauke]

the workaround, in future, would be to use Pointer Events (supported for the most part in Safari 13 onwards) rather than Touch Events

https://w3c.github.io/pointerevents/

[Roman A.]

Any progress on this issue ? Still touch events became with large numbers in 2020.

[Antoine Quint]

Someone needs to research if this is already addressed by Pointer Events. If it is, we'll likely mark this issue as WONTFIX because Pointer Events supersede Touch Events in every way. If it's not, then we need to reassess this in the context of Pointer Events.

[Antoine Quint]

I adjusted one of the demos to use Pointer Events at https://output.jsbin.com/hecifuweta/1. The issue is still relevant, we're still ending with integer values.

[Philip Jägenstedt]

This has not been addressed for PointerEvent. I've tested Safari 14.1 on a retina MacBook Pro and looked at pointermove events fired using `document.onpointermove = console.log` (on any page). The events have only integer coordinates.

This can also be seen directly in the IDL files. The coordinates in question are inherited from MouseEvent, and they're still of type long:
https://github.com/WebKit/WebKit/blob/9940f5c6cd901c3cce4efb372ebd94d808218d80/Source/WebCore/dom/MouseEvent.idl#L29-L32

[jcesarmobile]

on latest iOS 15 (GM?) both touch and pointer events are still integers on my iPad

[Oleguer Llopart]

Any update about this? Scroll events also have this problem.

[Alec Larson]

This is a major issue for webview-based iOS apps. Please, please fix it...

[Fabian]

Hey, any updates on this? As mentioned before, it's a bit of a headache for apps using WebView. The impact is pretty noticeable, especially on devices with DPI > 1. It's making animations and canvas work feel laggy, which isn't great for user experience. Any chance we could address this soon?

[Roman A.]

(In reply to Fabian from comment #28)
> Hey, any updates on this? As mentioned before, it's a bit of a headache for
> apps using WebView. The impact is pretty noticeable, especially on devices
> with DPI > 1. It's making animations and canvas work feel laggy, which isn't
> great for user experience. Any chance we could address this soon?

Have you tried this on latest Safari? Maybe create a Safari Fork and listen touch events directly from Native iOS listener? :D 
Because iOS Listener pass pointer information very accurate, the only need to grab it on Safari. 
For example, in Cordova can be created a special plugin that will be used only Device Pointer events value instead of Safari.

[Fabian]

(In reply to Roman A. from comment #29)
> (In reply to Fabian from comment #28)
> > Hey, any updates on this? As mentioned before, it's a bit of a headache for
> > apps using WebView. The impact is pretty noticeable, especially on devices
> > with DPI > 1. It's making animations and canvas work feel laggy, which isn't
> > great for user experience. Any chance we could address this soon?
> 
> Have you tried this on latest Safari? Maybe create a Safari Fork and listen
> touch events directly from Native iOS listener? :D 
> Because iOS Listener pass pointer information very accurate, the only need
> to grab it on Safari. 
> For example, in Cordova can be created a special plugin that will be used
> only Device Pointer events value instead of Safari.

This issue persists for both touch and pointer events. In my opinion, the solution shouldn't involve merely intercepting native gestures on a WebView, which, by the way, isn't a reliable approach, especially on iOS where UIWebView doesn't interact well with gesture recognizers. Resorting to such a method would be a hacky workaround. Instead, addressing the root cause directly would be more effective. Even if we were to implement a fix using Cordova or Capacitor Plugins, the underlying issue would remain unaddressed on the web platform.

The problem isn't with pointer events (or touch events) being inherently inaccurate. They function perfectly well on devices with a DPI of 1. However, devices with higher DPI, like iPads with Retina Displays boasting a DPI of 2, encounter issues. Despite having double the pixel density, touch or pointer events still report coordinates with the same pixel accuracy as devices with a DPI of 1.

Native gestures handle this disparity appropriately. For instance, on an iPad, values are doubled to accommodate the higher DPI (e.g., you'd receive values like 0.5). However, Safari fails to implement this behavior. Instead, it only provides whole integer values. Consequently, animations appear less fluid on high-DPI devices, as they miss out on finer pixel-level movements, resulting in perceptible micro-jumps.

What baffles me is that this isn't a technical limitation but rather an implementation choice. It's puzzling why Safari doesn't forward the accurate coordinate values. Restricting them to integers seems unnecessary and counterproductive.

[Hans Truelson]

All right guys... It's been way too long, we're still in the dark ages here. It's  kind of funny because Apple introduced the high res display but is behind the times on supporting it :)

Seriously though, I love Apple, but we gotta fix this.  Can't get precision touch move events on mobile and we need that!

[Rick Byers]

Created attachment 471720 [details]
Test of drawing points using WebKit on iPhone

[Rick Byers]

Created attachment 471721 [details]
Example of drawing points in Chromium

[Rick Byers]

To commemorate this bug being open for a decade, I've redone the test using https://rbyers.github.io/paint.html#points on an iPhone 12 (all browsers the same since it's WebKit), and Chrome on a Google Pixel 7 pro. Screenshots attached.

[Patrick H. Lauke]

> To commemorate this bug being open for a decade

Goodness... thanks for revisiting this, Rick

[Byungseon(Sun) Shin]

@Rick Byers Thanks for sharing the tests.

[Byungseon(Sun) Shin]

Created attachment 471931 [details]
simplified touch event accuracy test

[Byungseon(Sun) Shin]

Created attachment 471932 [details]
simplified touch event accuracy test results: iOS vs Android

[Byungseon(Sun) Shin]

Added simplified version of the touch accuracy tests and test results.
It's quite obvious that iOS's accuracy is inferior to Android.

[Byungseon(Sun) Shin]

To share a little more context, the resolution of touch events needs improvement. Users of the cloud gaming have reported that the current experience can appear buggy at times.

[Byungseon(Sun) Shin]

Pull request: https://github.com/WebKit/WebKit/pull/31867

[Byungseon(Sun) Shin]

This task will be duplicated by https://bugs.webkit.org/show_bug.cgi?id=279540.

*** This bug has been marked as a duplicate of bug 279540 ***

[Lily Spiniolas]

Correcting dupe order. Reopening.

[Lily Spiniolas]

*** Bug 279540 has been marked as a duplicate of this bug. ***

[Lily Spiniolas]

Pull request: https://github.com/WebKit/WebKit/pull/48412

[Lily Spiniolas]

Submitted web-platform-tests pull request: https://github.com/web-platform-tests/wpt/pull/54115

[EWS]

Committed 298383@main (8dc5033bb0ea): <https://commits.webkit.org/298383@main>

Reviewed commits have been landed. Closing PR #48412 and removing active labels.

[Byungseon(Sun) Shin]

Glad to hear that this issue got revolved. Thank you so much @Lily.