Bug 133180

Summary: Pointer events should allow for device-pixel accuracy
Product: WebKit Reporter: Rick Byers <rbyers>
Component: UI EventsAssignee: Abrar Rahman Protyasha <a_protyasha>
Status: RESOLVED DUPLICATE    
Severity: Normal CC: alec.stanford.larson, a_protyasha, bartzy, beidson, benjamin, commit-queue, dbates, dino, fabianmontagedu, graouts, hdtruelson, ik, jameshoward, jingman, juliosincesar, krevis, nekr.fabula, oleguer.llopart.mora, philip, redux, romwtb, simon.fraser, sun.shin, syoichi, tomac, webkit-bug-importer, yangge1987
Priority: P2 Keywords: InRadar
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Bug Depends on: 179167    
Bug Blocks:    
Attachments:
Description Flags
Patch
none
Patch
beidson: review-
Test of drawing points using WebKit on iPhone
none
Example of drawing points in Chromium
none
simplified touch event accuracy test
none
simplified touch event accuracy test results: iOS vs Android none

Description Rick Byers 2014-05-22 08:08:24 PDT
Various APIs are attempting to move to floating-point co-ordinates (eg. see bug 132895).  Perhaps TouchEvent co-ordinates should also be float/double instead of long?  http://trac.webkit.org/browser/trunk/Source/WebCore/dom/Touch.idl

This may be valuable on high-dpi devices.  Eg. if scrolling is happening in physical pixel units, it should be possible to build JS applications that move things around just as smoothly (as is possible in native iOS apps).  Eg. imagine an app that does a custom pull effect when pulling down when the document is at scroll offset 0.  Today when moving slowly this is noticeably jumpier on retina devices than the buttery smooth browser scrolling you get moving in the other direction.

I'm in the process of switching blink to doubles:
https://code.google.com/p/chromium/issues/detail?id=323935

The TouchEvents community group is also discussing this:
http://lists.w3.org/Archives/Public/public-touchevents/2014May/0006.html
Comment 1 Benjamin Poulain 2014-05-23 02:30:57 PDT
I agree, we should try this too.

Do you have a test page where it is jumpy for reference?
Comment 2 Rick Byers 2014-05-23 13:58:50 PDT
Thanks!

It can be pretty subtle in practice (especially if the finger is moving quickly), but here's an easy to visualize example: http://www.rbyers.net/paint.html#points - a drawing program that draws on physical pixel point per touchmove event.  See https://code.google.com/p/chromium/issues/detail?id=323935#c34 for screenshots comparing integer to floating-point co-ordinates on a nexus 5 (which has DSF=3).
Comment 3 Rick Byers 2014-05-23 14:14:59 PDT
The difference is also pretty noticeable when dragging something around slowly. eg: http://jsbin.com/idojig/2/quiet
Comment 4 Rick Byers 2014-12-09 08:30:28 PST
Assigning to Ben per #c1
Comment 5 Bar Ziony 2016-02-04 16:32:04 PST
Just wanted to ask if this was implemented already, and if not, if it in the works?

Thanks!
Comment 6 Radar WebKit Bug Importer 2016-02-10 15:17:28 PST
<rdar://problem/24597318>
Comment 7 Dean Jackson 2016-02-10 15:29:07 PST
http://www.rbyers.net/paint.html#points is a compelling demo
Comment 8 Rick Byers 2016-02-10 19:10:21 PST
Thanks Dean.  The drawing example is easiest to visualize but is fairly niche.  The really compelling scenarios IMHO are the dragging / custom scroll cases that are so common.  Most such existing pages got noticeably smoother (when dragging slowly) when we shipped this in chromium.
Comment 9 Bar Ziony 2016-04-20 03:37:29 PDT
Hi Dean,

Is this planned for an upcoming release perhaps? Just wanted to get some updated info about this feature.

Thanks.
Comment 10 Antoine Quint 2016-06-14 12:57:08 PDT
<rdar://problem/9564176>
Comment 11 Antoine Quint 2016-06-14 13:25:13 PDT
Created attachment 281273 [details]
Patch
Comment 12 Antoine Quint 2016-06-14 13:46:40 PDT
Created attachment 281277 [details]
Patch
Comment 13 WebKit Commit Bot 2016-06-14 13:48:49 PDT
Attachment 281277 [details] did not pass style-queue:


ERROR: Source/WebKit2/Shared/ios/NativeWebTouchEventIOS.mm:99:  Comma should be at the beginning of the line in a member initialization list.  [whitespace/init] [4]
ERROR: Source/WebCore/ChangeLog:12:  You should remove the 'No new tests' and either add and list tests, or explain why no new tests were possible.  [changelog/nonewtests] [5]
Total errors found: 2 in 17 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 14 Benjamin Poulain 2016-06-14 14:03:44 PDT
Comment on attachment 281277 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=281277&action=review

> Source/WebCore/platform/Widget.cpp:144
> +    if (!windowFraction.isEmpty()) {
> +        const int kFactor = 1000;
> +        IntPoint parentLineEnd = this->convertFromContainingWindow(flooredPoint + roundedIntSize(windowFraction.scaledBy(kFactor)));
> +        FloatSize parentFraction = (parentLineEnd - parentPoint).scaledBy(1.0f / kFactor);
> +        parentPoint.move(parentFraction);
> +    }

Any way to change convertFromContainingWindow() to handle floating point coordinates instead?

Linear interpolation of the decimal part is only gonna work for a small subset of transforms.

> Source/WebKit2/UIProcess/WebPageProxy.cpp:1944
> +        TrackingType touchPointTrackingType = m_scrollingCoordinatorProxy->eventTrackingTypeForPoint(roundedIntPoint(touchPoint.location()));

This is unfortunate.
I don't know how complicated it would be to add Region::contains() for FloatPoint.
Comment 15 Antoine Quint 2016-06-14 14:47:27 PDT
(In reply to comment #14)
> Comment on attachment 281277 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=281277&action=review
> 
> > Source/WebCore/platform/Widget.cpp:144
> > +    if (!windowFraction.isEmpty()) {
> > +        const int kFactor = 1000;
> > +        IntPoint parentLineEnd = this->convertFromContainingWindow(flooredPoint + roundedIntSize(windowFraction.scaledBy(kFactor)));
> > +        FloatSize parentFraction = (parentLineEnd - parentPoint).scaledBy(1.0f / kFactor);
> > +        parentPoint.move(parentFraction);
> > +    }
> 
> Any way to change convertFromContainingWindow() to handle floating point
> coordinates instead?

We would have to change quite a few methods, but I don't see why we couldn't. I'm not certain this is warranted though.

> > Source/WebKit2/UIProcess/WebPageProxy.cpp:1944
> > +        TrackingType touchPointTrackingType = m_scrollingCoordinatorProxy->eventTrackingTypeForPoint(roundedIntPoint(touchPoint.location()));
> 
> This is unfortunate.
> I don't know how complicated it would be to add Region::contains() for
> FloatPoint.

I think we could change the signature of Region::contains(const IntPoint& point) to take in a FloatPoint and operate on float values instead. I'll look into this.
Comment 16 Rick Byers 2016-06-14 15:11:42 PDT
(In reply to comment #15)
> (In reply to comment #14)
> > Comment on attachment 281277 [details]
> > Patch
> > 
> > View in context:
> > https://bugs.webkit.org/attachment.cgi?id=281277&action=review
> > 
> > > Source/WebCore/platform/Widget.cpp:144
> > > +    if (!windowFraction.isEmpty()) {
> > > +        const int kFactor = 1000;
> > > +        IntPoint parentLineEnd = this->convertFromContainingWindow(flooredPoint + roundedIntSize(windowFraction.scaledBy(kFactor)));
> > > +        FloatSize parentFraction = (parentLineEnd - parentPoint).scaledBy(1.0f / kFactor);
> > > +        parentPoint.move(parentFraction);
> > > +    }
> > 
> > Any way to change convertFromContainingWindow() to handle floating point
> > coordinates instead?
> 
> We would have to change quite a few methods, but I don't see why we
> couldn't. I'm not certain this is warranted though.

FWIW I felt really dirty doing this linear interpolation hack in blink (and left myself a FIXME to clean it up) but honestly after 2 years I haven't heard of a single example of anyone noticing it being less than perfect in the edge cases (rotated iframes mainly I guess).  I started to go down the path of fully supporting float but it exploded into a ton of work so I put it off.  I assumed we'd have other reasons to want fractional co-ordinate conversion between widgets but it's never come up since.  Maybe that was the wrong decision though - certainly feels wrong.
Comment 17 Ge Yang 2016-09-11 01:59:38 PDT
Hey Guys, 

I'm trying to use the apple pencil on an iPad pro with canvas to replace native apps, this is when I realized that the Touch.clientX and clientY are quantized to integers. 

Floating point would be very important for this application! and I think using apple pencil to draw on iPad pros is going to be a common thing to do in the future. It would be a shame if we can't do this properly in the browser.

Thanks for all of the work!

Ge
Comment 18 Brady Eidson 2017-08-19 16:01:42 PDT
Comment on attachment 281277 [details]
Patch

r-, as this has been pending review for over a year now. It is near-impossible that this patch still applies to trunk and unlikely to still be relevant in its current form.
Comment 19 Jake Ingman 2019-11-27 10:06:57 PST
I was wondering if there had been any progress addressing this bug. The lack of sub-pixel accuracy in touch events severely limits the capabilities of web-based drawing applications, and to my knowledge, there are no workarounds.
Comment 20 Patrick H. Lauke 2019-11-27 11:19:45 PST
the workaround, in future, would be to use Pointer Events (supported for the most part in Safari 13 onwards) rather than Touch Events

https://w3c.github.io/pointerevents/
Comment 21 Roman A. 2020-06-09 01:09:10 PDT
Any progress on this issue ? Still touch events became with large numbers in 2020.
Comment 22 Antoine Quint 2020-06-09 05:56:42 PDT
Someone needs to research if this is already addressed by Pointer Events. If it is, we'll likely mark this issue as WONTFIX because Pointer Events supersede Touch Events in every way. If it's not, then we need to reassess this in the context of Pointer Events.
Comment 23 Antoine Quint 2020-06-09 06:20:39 PDT
I adjusted one of the demos to use Pointer Events at https://output.jsbin.com/hecifuweta/1. The issue is still relevant, we're still ending with integer values.
Comment 24 Philip Jägenstedt 2021-06-30 04:27:25 PDT
This has not been addressed for PointerEvent. I've tested Safari 14.1 on a retina MacBook Pro and looked at pointermove events fired using `document.onpointermove = console.log` (on any page). The events have only integer coordinates.

This can also be seen directly in the IDL files. The coordinates in question are inherited from MouseEvent, and they're still of type long:
https://github.com/WebKit/WebKit/blob/9940f5c6cd901c3cce4efb372ebd94d808218d80/Source/WebCore/dom/MouseEvent.idl#L29-L32
Comment 25 jcesarmobile 2021-09-15 07:04:12 PDT
on latest iOS 15 (GM?) both touch and pointer events are still integers on my iPad
Comment 26 Oleguer Llopart 2021-12-20 06:48:43 PST
Any update about this? Scroll events also have this problem.
Comment 27 Alec Larson 2023-01-13 14:57:57 PST
This is a major issue for webview-based iOS apps. Please, please fix it...
Comment 28 Fabian 2024-03-22 16:12:26 PDT
Hey, any updates on this? As mentioned before, it's a bit of a headache for apps using WebView. The impact is pretty noticeable, especially on devices with DPI > 1. It's making animations and canvas work feel laggy, which isn't great for user experience. Any chance we could address this soon?
Comment 29 Roman A. 2024-03-22 21:57:05 PDT
(In reply to Fabian from comment #28)
> Hey, any updates on this? As mentioned before, it's a bit of a headache for
> apps using WebView. The impact is pretty noticeable, especially on devices
> with DPI > 1. It's making animations and canvas work feel laggy, which isn't
> great for user experience. Any chance we could address this soon?

Have you tried this on latest Safari? Maybe create a Safari Fork and listen touch events directly from Native iOS listener? :D 
Because iOS Listener pass pointer information very accurate, the only need to grab it on Safari. 
For example, in Cordova can be created a special plugin that will be used only Device Pointer events value instead of Safari.
Comment 30 Fabian 2024-03-23 02:19:14 PDT
(In reply to Roman A. from comment #29)
> (In reply to Fabian from comment #28)
> > Hey, any updates on this? As mentioned before, it's a bit of a headache for
> > apps using WebView. The impact is pretty noticeable, especially on devices
> > with DPI > 1. It's making animations and canvas work feel laggy, which isn't
> > great for user experience. Any chance we could address this soon?
> 
> Have you tried this on latest Safari? Maybe create a Safari Fork and listen
> touch events directly from Native iOS listener? :D 
> Because iOS Listener pass pointer information very accurate, the only need
> to grab it on Safari. 
> For example, in Cordova can be created a special plugin that will be used
> only Device Pointer events value instead of Safari.

This issue persists for both touch and pointer events. In my opinion, the solution shouldn't involve merely intercepting native gestures on a WebView, which, by the way, isn't a reliable approach, especially on iOS where UIWebView doesn't interact well with gesture recognizers. Resorting to such a method would be a hacky workaround. Instead, addressing the root cause directly would be more effective. Even if we were to implement a fix using Cordova or Capacitor Plugins, the underlying issue would remain unaddressed on the web platform.

The problem isn't with pointer events (or touch events) being inherently inaccurate. They function perfectly well on devices with a DPI of 1. However, devices with higher DPI, like iPads with Retina Displays boasting a DPI of 2, encounter issues. Despite having double the pixel density, touch or pointer events still report coordinates with the same pixel accuracy as devices with a DPI of 1.

Native gestures handle this disparity appropriately. For instance, on an iPad, values are doubled to accommodate the higher DPI (e.g., you'd receive values like 0.5). However, Safari fails to implement this behavior. Instead, it only provides whole integer values. Consequently, animations appear less fluid on high-DPI devices, as they miss out on finer pixel-level movements, resulting in perceptible micro-jumps.

What baffles me is that this isn't a technical limitation but rather an implementation choice. It's puzzling why Safari doesn't forward the accurate coordinate values. Restricting them to integers seems unnecessary and counterproductive.
Comment 31 Hans Truelson 2024-06-21 03:57:10 PDT
All right guys... It's been way too long, we're still in the dark ages here. It's  kind of funny because Apple introduced the high res display but is behind the times on supporting it :)

Seriously though, I love Apple, but we gotta fix this.  Can't get precision touch move events on mobile and we need that!
Comment 32 Rick Byers 2024-06-21 08:04:56 PDT
Created attachment 471720 [details]
Test of drawing points using WebKit on iPhone
Comment 33 Rick Byers 2024-06-21 08:05:32 PDT
Created attachment 471721 [details]
Example of drawing points in Chromium
Comment 34 Rick Byers 2024-06-21 08:09:01 PDT
To commemorate this bug being open for a decade, I've redone the test using https://rbyers.github.io/paint.html#points on an iPhone 12 (all browsers the same since it's WebKit), and Chrome on a Google Pixel 7 pro. Screenshots attached.
Comment 35 Patrick H. Lauke 2024-06-21 10:21:44 PDT
> To commemorate this bug being open for a decade

Goodness... thanks for revisiting this, Rick
Comment 36 Byungseon(Sun) Shin 2024-07-08 15:24:23 PDT
@Rick Byers Thanks for sharing the tests.
Comment 37 Byungseon(Sun) Shin 2024-07-22 14:29:48 PDT
Created attachment 471931 [details]
simplified touch event accuracy test
Comment 38 Byungseon(Sun) Shin 2024-07-22 14:30:36 PDT
Created attachment 471932 [details]
simplified touch event accuracy test results: iOS vs Android
Comment 39 Byungseon(Sun) Shin 2024-07-22 14:32:02 PDT
Added simplified version of the touch accuracy tests and test results.
It's quite obvious that iOS's accuracy is inferior to Android.
Comment 40 Byungseon(Sun) Shin 2024-07-23 09:35:47 PDT
To share a little more context, the resolution of touch events needs improvement. Users of the cloud gaming have reported that the current experience can appear buggy at times.
Comment 41 Byungseon(Sun) Shin 2024-08-07 16:36:50 PDT
Pull request: https://github.com/WebKit/WebKit/pull/31867
Comment 42 Byungseon(Sun) Shin 2024-09-11 15:07:14 PDT
This task will be duplicated by https://bugs.webkit.org/show_bug.cgi?id=279540.

*** This bug has been marked as a duplicate of bug 279540 ***