Bug 242401 - AX: Expose suggestion, insertion, deletion roles and attributes
Summary: AX: Expose suggestion, insertion, deletion roles and attributes
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Accessibility (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2022-07-06 11:26 PDT by Joshua Hoffman
Modified: 2022-07-15 14:33 PDT (History)
10 users (show)

See Also:


Attachments
Patch (34.38 KB, patch)
2022-07-06 13:09 PDT, Joshua Hoffman
no flags Details | Formatted Diff | Diff
Patch (36.98 KB, patch)
2022-07-06 14:15 PDT, Joshua Hoffman
no flags Details | Formatted Diff | Diff
Patch (36.73 KB, patch)
2022-07-06 14:42 PDT, Joshua Hoffman
no flags Details | Formatted Diff | Diff
Patch (37.25 KB, patch)
2022-07-11 08:59 PDT, Joshua Hoffman
no flags Details | Formatted Diff | Diff
Patch (36.78 KB, patch)
2022-07-11 16:19 PDT, Joshua Hoffman
no flags Details | Formatted Diff | Diff
Patch (36.96 KB, patch)
2022-07-11 17:17 PDT, Joshua Hoffman
no flags Details | Formatted Diff | Diff
Patch (37.38 KB, patch)
2022-07-12 14:27 PDT, Joshua Hoffman
ews-feeder: commit-queue-
Details | Formatted Diff | Diff
Patch (37.20 KB, patch)
2022-07-12 17:03 PDT, Joshua Hoffman
no flags Details | Formatted Diff | Diff
Patch (36.74 KB, patch)
2022-07-13 16:36 PDT, Joshua Hoffman
no flags Details | Formatted Diff | Diff
Patch (36.81 KB, patch)
2022-07-14 09:59 PDT, Joshua Hoffman
no flags Details | Formatted Diff | Diff
Patch (36.77 KB, patch)
2022-07-15 09:51 PDT, Joshua Hoffman
ews-feeder: commit-queue-
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Joshua Hoffman 2022-07-06 11:26:58 PDT
This change is necessary for AX clients to describe these elements.
Comment 1 Radar WebKit Bug Importer 2022-07-06 11:27:06 PDT
<rdar://problem/96534144>
Comment 2 Joshua Hoffman 2022-07-06 13:09:02 PDT
Created attachment 460721 [details]
Patch
Comment 3 Joshua Hoffman 2022-07-06 14:15:17 PDT
Created attachment 460722 [details]
Patch
Comment 4 Joshua Hoffman 2022-07-06 14:42:39 PDT
Created attachment 460723 [details]
Patch
Comment 5 Andres Gonzalez 2022-07-06 14:51:57 PDT
(In reply to Joshua Hoffman from comment #2)
> Created attachment 460721 [details]
> Patch

--- a/Source/WebCore/accessibility/AccessibilityObjectInterface.h
+++ a/Source/WebCore/accessibility/AccessibilityObjectInterface.h

@@ -503,6 +504,8 @@ ALWAYS_INLINE String accessibilityRoleToString(AccessibilityRole role)
         return "StaticText"_s;
     case AccessibilityRole::Subscript:
         return "Subscript"_s;
+    case AccessibilityRole::Suggestion:
+            return "Suggestion"_s;

Wrong indentation. Style checker should've gotten it.

--- a/Source/WebCore/accessibility/ios/WebAccessibilityObjectWrapperIOS.mm
+++ a/Source/WebCore/accessibility/ios/WebAccessibilityObjectWrapperIOS.mm

+- (BOOL)accessibilityIsInsertion
+{
+    if (![self _prepareAccessibilityCall])
+        return nil;

nil -> NO

+    for (AXCoreObject* obj = self.axBackingObject->parentObject(); obj; obj = obj->parentObject()) {

You can probably replace this for loop with a Accessibility::findAncestor. But if you keep the for loop:

Change AXCoreObject* obj -> auto* object

+        if (obj) {

It is unnecessary since it is the condition in the for statement.

+            if (obj->hasTagName(bodyTag))
+                return NO;

You can simply assume that if parentObject() is null you have reached the root of the tree, this is a rather expensive way of checking if you got to the root.

In short, I think you can re-write this as:

return Accessibility::findAncestor(*self.backingObject, true, [] (const auto& object) {
   return object.roleValue() == AccessibilityRole::Insertion;
});

+- (BOOL)accessibilityIsDeletion

Same comments as above apply here.

Further more, these methods may belong to AXCoreObject, in which case, in the platform wrapper you should only need to do 

return self.axBackingObject->isInsertion();

Some of the comments above apply also to the following methods:

+- (BOOL)accessibilityIsFirstItemInSuggestion

and

+- (BOOL)accessibilityIsLastItemInSuggestion

--- a/Source/WebCore/accessibility/mac/WebAccessibilityObjectWrapperMac.mm
+++ a/Source/WebCore/accessibility/mac/WebAccessibilityObjectWrapperMac.mm
@@ -860,6 +860,14 @@ static void AXAttributeStringSetStyle(NSMutableAttributedString* attrString, Ren

+        if (auto* axObject = node->document().axObjectCache()->getOrCreate(node)) {

axObjectCache() can be null.

+            if (axObject->roleValue() == AccessibilityRole::Insertion)
+                AXAttributeStringSetNumber(attrString, @"AXIsSuggestedInsertion", @YES, range);
+            else if (axObject->roleValue() == AccessibilityRole::Deletion)
+                AXAttributeStringSetNumber(attrString, @"AXIsSuggestedDeletion", @YES, range);
+            if (axObject->roleValue() == AccessibilityRole::Suggestion)
+                AXAttributeStringSetNumber(attrString, @"AXIsSuggestion", @YES, range);

You may want to use a switch here, or the last if should be an else if.

-    if (backingObject->isTextControl()) {
+    if (backingObject->isTextControl() || backingObject->isStaticText()) {

Why we need to change this. I think some of these properties apply to text controls but not to static text.

--- a/Tools/WebKitTestRunner/InjectedBundle/mac/AccessibilityUIElementMac.mm
+++ a/Tools/WebKitTestRunner/InjectedBundle/mac/AccessibilityUIElementMac.mm

The four new methods in the AccessibilityUIElement class return NO. Are we testing this on the Mac?
Comment 6 Joshua Hoffman 2022-07-11 08:59:49 PDT
Created attachment 460794 [details]
Patch
Comment 7 chris fleizach 2022-07-11 11:20:07 PDT
Comment on attachment 460794 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=460794&action=review

> Source/WebCore/accessibility/ios/WebAccessibilityObjectWrapperIOS.mm:2799
> +    for (AXCoreObject* parent = self.axBackingObject->parentObject(); parent; parent = parent->parentObject()) {

auto* parent

> Source/WebCore/accessibility/ios/WebAccessibilityObjectWrapperIOS.mm:2801
> +            auto children = parent->children();

I wonder if there's a cleaner way to write this. something like

auto *suggestion = Accessibility::findAncestor(*self.axBackingObject, false, [] (const auto& object) {
        return object.roleValue() == AccessibilityRole::Suggestion;
  });

auto children = suggestion.children();
return children.size() > 1 && children[0].isAncestorOrSelf(self);
Comment 8 chris fleizach 2022-07-11 11:22:59 PDT
Comment on attachment 460794 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=460794&action=review

> Source/WebCore/accessibility/mac/WebAccessibilityObjectWrapperMac.mm:863
> +        if (auto* axObjectCache = node->document().axObjectCache()) {

do we need to make AXObjects here or can we get away by checking attributes on the Node

> Source/WebCore/accessibility/mac/WebAccessibilityObjectWrapperMac.mm:1006
> +    [attrString appendAttributedString:[[NSAttributedString alloc] initWithString:text.createNSStringWithoutCopying().get()]];

I think this is going to leak the NSAttributedString. you would need to autorelease it

> Source/WebCore/accessibility/mac/WebAccessibilityObjectWrapperMac.mm:4039
> +        if ([attribute isEqualToString: (NSString*)kAXAttributedStringForRangeParameterizedAttribute])

extra space
ing: (NSStr

> Source/WebCore/accessibility/mac/WebAccessibilityObjectWrapperMac.mm:4040
> +            return rangeSet ? [self doAXAttributedStringForRange:range] : nil;

it doesn't look like you're using reangeSet here. do you mean to change range?
Comment 9 Joshua Hoffman 2022-07-11 16:19:54 PDT
Created attachment 460801 [details]
Patch
Comment 10 chris fleizach 2022-07-11 16:32:16 PDT
Comment on attachment 460801 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=460801&action=review

> Source/WebCore/accessibility/mac/WebAccessibilityObjectWrapperMac.mm:4040
> +        if ([attribute isEqualToString:(NSString*)kAXAttributedStringForRangeParameterizedAttribute])

(NSString*) -> (NSString *)

> LayoutTests/accessibility/ios-simulator/suggestion-insertion-deletion-attributes.html:18
> +		let output = "This test makes sure that suggestions, insertions, and deletions expose the right attributes.\n";

replace tabs with spaces (and only use 4 spaces)

> LayoutTests/accessibility/mac/attributed-string-includes-insertion-deletion.html:21
> +

bad indentations in this file

> LayoutTests/accessibility/mac/suggestion-subrole-and-roledescription.html:22
> +		var ins1 = accessibilityController.accessibleElementById("ins1");

bad indentations in this file
Comment 11 Joshua Hoffman 2022-07-11 17:17:59 PDT
Created attachment 460802 [details]
Patch
Comment 12 Andres Gonzalez 2022-07-12 06:06:47 PDT
(In reply to Joshua Hoffman from comment #11)
> Created attachment 460802 [details]
> Patch

--- a/Source/WebCore/accessibility/ios/WebAccessibilityObjectWrapperIOS.mm
+++ a/Source/WebCore/accessibility/ios/WebAccessibilityObjectWrapperIOS.mm

+- (BOOL)accessibilityIsFirstItemInSuggestion
+{
+    if (![self _prepareAccessibilityCall])
+        return NO;
+
+    auto *suggestion = Accessibility::findAncestor(*self.axBackingObject, false, [] (const auto& object) {

* in the wrong side, should be auto* suggestion since it is a C++ object.

+    auto children = suggestion->children();

suggestion may be nullptr, so you need to check.

+    return children.size() > 0 && (children[0]->isAncestorOfObject(self.axBackingObject) || children[0] == self.axBackingObject);

Don't need > 0, just return children.size() && ...

I would swap the operands of || since children[0] == self.axBackingObject is cheaper and more likely.

Same comments for 

+- (BOOL)accessibilityIsLastItemInSuggestion

--- a/Source/WebCore/accessibility/mac/WebAccessibilityObjectWrapperMac.mm
+++ a/Source/WebCore/accessibility/mac/WebAccessibilityObjectWrapperMac.mm
@@ -4025,6 +4036,11 @@ ALLOW_DEPRECATED_IMPLEMENTATIONS_END
         }
     }

+    if (backingObject->isTextControl() || backingObject->isStaticText()) {
+        if ([attribute isEqualToString:(NSString *)kAXAttributedStringForRangeParameterizedAttribute])

Do we need to cast kAXAttributedStringForRangeParameterizedAttribute?

+            return [self doAXAttributedStringForRange:range];

Why we don't need to check for rangeSet as before?

--- a/Tools/DumpRenderTree/win/AccessibilityUIElementWin.cpp
+++ a/Tools/DumpRenderTree/win/AccessibilityUIElementWin.cpp
@@ -1038,3 +1038,4 @@ void AccessibilityUIElement::rowHeaders(Vector<AccessibilityUIElement>&) const
 {
     // FIXME: implement
 }
+

Seems that the only change in this file is the addition of a blank line, could you undo that?

--- a/Tools/WebKitTestRunner/InjectedBundle/atspi/AccessibilityUIElementAtspi.cpp
+++ a/Tools/WebKitTestRunner/InjectedBundle/atspi/AccessibilityUIElementAtspi.cpp

+}
+
+
+bool AccessibilityUIElement::isFirstItemInSuggestion() const

Extra blank line before this function and the next one.

--- a/Tools/WebKitTestRunner/InjectedBundle/mac/AccessibilityUIElementMac.mm
+++ a/Tools/WebKitTestRunner/InjectedBundle/mac/AccessibilityUIElementMac.mm

Why we return NO in all the new methods on the Mac? If we don't need this on the Mac perhaps we should say notImplemented()?
Comment 13 Joshua Hoffman 2022-07-12 08:38:47 PDT
(In reply to Andres Gonzalez from comment #12)
> (In reply to Joshua Hoffman from comment #11)
> > Created attachment 460802 [details]
> > Patch

> --- a/Source/WebCore/accessibility/mac/WebAccessibilityObjectWrapperMac.mm
> +++ a/Source/WebCore/accessibility/mac/WebAccessibilityObjectWrapperMac.mm
> @@ -4025,6 +4036,11 @@ ALLOW_DEPRECATED_IMPLEMENTATIONS_END
>          }
>      }
> 
> +    if (backingObject->isTextControl() || backingObject->isStaticText()) {
> +        if ([attribute isEqualToString:(NSString
> *)kAXAttributedStringForRangeParameterizedAttribute])
> 
> Do we need to cast kAXAttributedStringForRangeParameterizedAttribute?
> 
> +            return [self doAXAttributedStringForRange:range];
> 
> Why we don't need to check for rangeSet as before?

On second thought, I will add the rangeSet check back in (especially in the context of the checks below it).

The cast prevents Xcode warnings, in addition to maintaining consistency (we cast CFStrings to NSStrings throughout WebKit).

> 
> --- a/Tools/WebKitTestRunner/InjectedBundle/mac/AccessibilityUIElementMac.mm
> +++ a/Tools/WebKitTestRunner/InjectedBundle/mac/AccessibilityUIElementMac.mm
> 
> Why we return NO in all the new methods on the Mac? If we don't need this on
> the Mac perhaps we should say notImplemented()?

Yeah, we don't need these implementations on the Mac. Good call-out about adding notImplemented—I will make that change.
Comment 14 Joshua Hoffman 2022-07-12 14:27:43 PDT
Created attachment 460827 [details]
Patch
Comment 15 Joshua Hoffman 2022-07-12 17:03:32 PDT
Created attachment 460835 [details]
Patch
Comment 16 Andres Gonzalez 2022-07-13 12:08:19 PDT
(In reply to Joshua Hoffman from comment #15)
> Created attachment 460835 [details]
> Patch

--- a/Source/WebCore/accessibility/ios/WebAccessibilityObjectWrapperIOS.mm
+++ a/Source/WebCore/accessibility/ios/WebAccessibilityObjectWrapperIOS.mm

+- (BOOL)accessibilityIsFirstItemInSuggestion
+{
...
+    auto children = suggestion->children();
+    auto parent = self.axBackingObject->parentObject();
+    BOOL isFirstChildOfParent = parent && parent->children().size() && parent->children()[0] == self.axBackingObject;
+    return children.size() && (children[0] == self.axBackingObject || (isFirstChildOfParent && children[0]->isAncestorOfObject(self.axBackingObject)));

I think this logic fails in the following case:

<suggestion>
    <grandparent>
        <uncle>
            <child1>
        <parent>
            <child2>

it will return true for child2.
Comment 17 Joshua Hoffman 2022-07-13 13:38:33 PDT
(In reply to Andres Gonzalez from comment #16)
> (In reply to Joshua Hoffman from comment #15)
> > Created attachment 460835 [details]
> > Patch
> 
> --- a/Source/WebCore/accessibility/ios/WebAccessibilityObjectWrapperIOS.mm
> +++ a/Source/WebCore/accessibility/ios/WebAccessibilityObjectWrapperIOS.mm
> 
> +- (BOOL)accessibilityIsFirstItemInSuggestion
> +{
> ...
> +    auto children = suggestion->children();
> +    auto parent = self.axBackingObject->parentObject();
> +    BOOL isFirstChildOfParent = parent && parent->children().size() &&
> parent->children()[0] == self.axBackingObject;
> +    return children.size() && (children[0] == self.axBackingObject ||
> (isFirstChildOfParent &&
> children[0]->isAncestorOfObject(self.axBackingObject)));
> 
> I think this logic fails in the following case:
> 
> <suggestion>
>     <grandparent>
>         <uncle>
>             <child1>
>         <parent>
>             <child2>
> 
> it will return true for child2.

I'd think a traversal up, where in each iteration we check both that a child is always the first element and check for a suggestion ancestor, could fix this? (not using FindAncestor to allow us to break early) Something like:

for (AXCoreObject* obj = self.axBackingObject->parentObject(); obj; obj = obj->parentObject())
{
    if (auto parent = obj->parentObject()) {
        if (parent->children().size() && parent->children()[0] != self.axBackingObject)
            return NO;
        if (parent->roleValue() == AccessibilityRole::Suggestion)
            return YES;
    }
}
Comment 18 Joshua Hoffman 2022-07-13 16:36:10 PDT
Created attachment 460871 [details]
Patch
Comment 19 Joshua Hoffman 2022-07-14 09:59:18 PDT
Created attachment 460896 [details]
Patch
Comment 20 Joshua Hoffman 2022-07-15 09:51:50 PDT
Created attachment 460927 [details]
Patch
Comment 21 EWS 2022-07-15 12:27:33 PDT
Committed 252516@main (3395d7913749): <https://commits.webkit.org/252516@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 460927 [details].