Bug 178616 – [Cocoa] Web Automation: add SPI to tell whether the automation session is currently simulating user interactions

• Component: WebKit API
• Status: RESOLVED FIXED

[Blaze Burg]

We currently cannot distinguish actions that come from the user (i.e., clicking menu items) versus actions that come from AppKit default event handlers (i.e., Ctrl-a turns into selectAll:). This will allow clients to tell those cases apart.

[Blaze Burg]

Created attachment 324488 [details]
Patch

[Joseph Pecoraro]

Comment on attachment 324488 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=324488&action=review

> Source/WebKit/UIProcess/API/Cocoa/_WKAutomationSession.h:46
> +@property (nonatomic, readonly, getter=isSimulatingUserInteraction) BOOL simulatingUserInteraction;

I think this needs:

    ... WK_API_AVAILABLE(macosx(WK_MAC_TBA), ios(WK_IOS_TBA));

> Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp:668
> +        m_simulatingUserInteraction = false;

It sounds like we should only be clearing the m_simulatingUserInteraction flag when the m_pendingKeyboardEventsFlushedCallbacksPerPage list is empty.

Right now it sounds like we could be simulating events in multiple pages at the same time and the first to finish will clear the global shared state.

> Source/WebKit/UIProcess/Automation/WebAutomationSession.h:264
> +    // True if a synthesized key or mouse event is still being processed.
> +    bool m_simulatingUserInteraction { false };

This says "or mouse event" but I don't see code below that sets this for mouse events. The comment seems inaccurate.

[Blaze Burg]

(In reply to Joseph Pecoraro from comment #2)
> Comment on attachment 324488 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=324488&action=review
> 
> > Source/WebKit/UIProcess/API/Cocoa/_WKAutomationSession.h:46
> > +@property (nonatomic, readonly, getter=isSimulatingUserInteraction) BOOL simulatingUserInteraction;
> 
> I think this needs:
> 
>     ... WK_API_AVAILABLE(macosx(WK_MAC_TBA), ios(WK_IOS_TBA));
> 
> > Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp:668
> > +        m_simulatingUserInteraction = false;
> 
> It sounds like we should only be clearing the m_simulatingUserInteraction
> flag when the m_pendingKeyboardEventsFlushedCallbacksPerPage list is empty.
> 
> Right now it sounds like we could be simulating events in multiple pages at
> the same time and the first to finish will clear the global shared state.
> 
> > Source/WebKit/UIProcess/Automation/WebAutomationSession.h:264
> > +    // True if a synthesized key or mouse event is still being processed.
> > +    bool m_simulatingUserInteraction { false };
> 
> This says "or mouse event" but I don't see code below that sets this for
> mouse events. The comment seems inaccurate.

I will probably fix this for mouse events in a later patch. I've seen some reports of unexpected glass pane popups underneath Element Click command. We should at least log an error and not show the dialog in this case.

[Blaze Burg]

Created attachment 324629 [details]
For EWS

[Blaze Burg]

EWS test failures on Mac-wk2 do not reproduce locally.

[Blaze Burg]

Created attachment 324633 [details]
For EWS

[WebKit Commit Bot]

Comment on attachment 324633 [details]
For EWS

Clearing flags on attachment: 324633

Committed r223880: <https://trac.webkit.org/changeset/223880>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/35568511>