Bug 178616 - [Cocoa] Web Automation: add SPI to tell whether the automation session is currently simulating user interactions
Summary: [Cocoa] Web Automation: add SPI to tell whether the automation session is cur...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebKit API (show other bugs)
Version: WebKit Nightly Build
Hardware: All All
: P2 Normal
Assignee: BJ Burg
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2017-10-20 21:54 PDT by BJ Burg
Modified: 2017-11-15 12:58 PST (History)
5 users (show)

See Also:


Attachments
Patch (8.73 KB, patch)
2017-10-20 22:22 PDT, BJ Burg
no flags Details | Formatted Diff | Diff
For EWS (8.85 KB, patch)
2017-10-23 19:45 PDT, BJ Burg
no flags Details | Formatted Diff | Diff
For EWS (8.85 KB, patch)
2017-10-23 21:00 PDT, BJ Burg
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description BJ Burg 2017-10-20 21:54:24 PDT
We currently cannot distinguish actions that come from the user (i.e., clicking menu items) versus actions that come from AppKit default event handlers (i.e., Ctrl-a turns into selectAll:). This will allow clients to tell those cases apart.
Comment 1 BJ Burg 2017-10-20 22:22:12 PDT
Created attachment 324488 [details]
Patch
Comment 2 Joseph Pecoraro 2017-10-23 16:26:24 PDT
Comment on attachment 324488 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=324488&action=review

> Source/WebKit/UIProcess/API/Cocoa/_WKAutomationSession.h:46
> +@property (nonatomic, readonly, getter=isSimulatingUserInteraction) BOOL simulatingUserInteraction;

I think this needs:

    ... WK_API_AVAILABLE(macosx(WK_MAC_TBA), ios(WK_IOS_TBA));

> Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp:668
> +        m_simulatingUserInteraction = false;

It sounds like we should only be clearing the m_simulatingUserInteraction flag when the m_pendingKeyboardEventsFlushedCallbacksPerPage list is empty.

Right now it sounds like we could be simulating events in multiple pages at the same time and the first to finish will clear the global shared state.

> Source/WebKit/UIProcess/Automation/WebAutomationSession.h:264
> +    // True if a synthesized key or mouse event is still being processed.
> +    bool m_simulatingUserInteraction { false };

This says "or mouse event" but I don't see code below that sets this for mouse events. The comment seems inaccurate.
Comment 3 BJ Burg 2017-10-23 19:43:05 PDT
(In reply to Joseph Pecoraro from comment #2)
> Comment on attachment 324488 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=324488&action=review
> 
> > Source/WebKit/UIProcess/API/Cocoa/_WKAutomationSession.h:46
> > +@property (nonatomic, readonly, getter=isSimulatingUserInteraction) BOOL simulatingUserInteraction;
> 
> I think this needs:
> 
>     ... WK_API_AVAILABLE(macosx(WK_MAC_TBA), ios(WK_IOS_TBA));
> 
> > Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp:668
> > +        m_simulatingUserInteraction = false;
> 
> It sounds like we should only be clearing the m_simulatingUserInteraction
> flag when the m_pendingKeyboardEventsFlushedCallbacksPerPage list is empty.
> 
> Right now it sounds like we could be simulating events in multiple pages at
> the same time and the first to finish will clear the global shared state.
> 
> > Source/WebKit/UIProcess/Automation/WebAutomationSession.h:264
> > +    // True if a synthesized key or mouse event is still being processed.
> > +    bool m_simulatingUserInteraction { false };
> 
> This says "or mouse event" but I don't see code below that sets this for
> mouse events. The comment seems inaccurate.

I will probably fix this for mouse events in a later patch. I've seen some reports of unexpected glass pane popups underneath Element Click command. We should at least log an error and not show the dialog in this case.
Comment 4 BJ Burg 2017-10-23 19:45:24 PDT
Created attachment 324629 [details]
For EWS
Comment 5 BJ Burg 2017-10-23 20:33:45 PDT
EWS test failures on Mac-wk2 do not reproduce locally.
Comment 6 BJ Burg 2017-10-23 21:00:11 PDT
Created attachment 324633 [details]
For EWS
Comment 7 WebKit Commit Bot 2017-10-23 22:14:00 PDT
Comment on attachment 324633 [details]
For EWS

Clearing flags on attachment: 324633

Committed r223880: <https://trac.webkit.org/changeset/223880>
Comment 8 WebKit Commit Bot 2017-10-23 22:14:02 PDT
All reviewed patches have been landed.  Closing bug.
Comment 9 Radar WebKit Bug Importer 2017-11-15 12:58:20 PST
<rdar://problem/35568511>