Bug 141303 – AX: WebKit exposes all Ruby Text as Unknown (Japanese EPUB accessibility blocker)

• Component: Accessibility
• Status: RESOLVED FIXED

[James Craig]

Created attachment 246124 [details]
ruby example

Load attached test case.
All <rt> text is exposed with no role.

WebKit should probably expose some new subroles for the APIs. This containing groups would allow VO and other AT to understand the element relationships.

I suggest the following subroles:

AXGroup:AXRuby
AXGroup:AXRubyInline
AXStaticText:AXRubyText (top)
AXStaticText:AXRubyBase (bottom)


Docs:
http://trac.webkit.org/wiki/Ruby
http://www.w3.org/TR/ruby/

[Radar WebKit Bug Importer]

<rdar://problem/19734366>

[James Craig]

AXGroup:AXRuby (<ruby> element)
AXGroup:AXRubyInline (WebKit implicit view block for each <rt>/<rb> pair, no directly associated DOM element)
AXStaticText:AXRubyText (<rt> ruby top)
AXStaticText:AXRubyBase (<rb> ruby base)

[James Craig]

Note that there is also a complex Ruby type (with <rbc> "ruby container") that is not yet supported in WebKit.

<ruby>
  <rbc>
    <rb>10</rb>
    <rb>31</rb>
    <rb>2002</rb>
  </rbc>
  <rtc>
    <rt>Month</rt>
    <rt>Day</rt>
    <rt>Year</rt>
  </rtc>
  <rtc>
    <rt rbspan="3">Expiration Date</rt>
  </rtc>
</ruby>

This would make it behave more like a table with rowspans, so that may affect the API decisions.

[Andres Gonzalez]

Have a fix for the problem of VO Right/Left arrows speaking and Braille showing "unknown" for ruby text elements.

[chris fleizach]

(In reply to comment #4)
> Have a fix for the problem of VO Right/Left arrows speaking and Braille
> showing "unknown" for ruby text elements.

can you upload your patch?

[Andres Gonzalez]

Created attachment 252429 [details]
Patch

[Build Bot]

Comment on attachment 252429 [details]
Patch

Attachment 252429 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.appspot.com/results/6306895727951872

New failing tests:
accessibility/ruby-elements-ignored.html

[Build Bot]

Created attachment 252433 [details]
Archive of layout-test-results from ews103 for mac-mavericks

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews103  Port: mac-mavericks  Platform: Mac OS X 10.9.5

[chris fleizach]

(In reply to comment #6)
> Created attachment 252429 [details]
> Patch

So it looks like this patch just hides all ruby elements from accessibility. I think we want to follow James original suggestion of exposing new roles for these objects so that VoiceOver can treat them as it needs

James suggestion was

-------------------------
WebKit should probably expose some new subroles for the APIs. This containing groups would allow VO and other AT to understand the element relationships.

I suggest the following subroles:

AXGroup:AXRuby
AXGroup:AXRubyInline
AXStaticText:AXRubyText (top)
AXStaticText:AXRubyBase (bottom)

------------------------
Thanks

[Ryosuke Niwa]

I agree just hiding ruby text isn't great. In fact, ruby text for Japanese kanji chanracters are most useful in dictations because they write out exactly how each character is read.

[Ryosuke Niwa]

(In reply to comment #10)
> I agree just hiding ruby text isn't great. In fact, ruby text for Japanese
> kanji chanracters are most useful in dictations because they write out
> exactly how each character is read.

That is, exaclty how each character is pronunced.

[Andres Gonzalez]

Created attachment 253298 [details]
Patch

[Andres Gonzalez]

Uploaded a new patch that exposes ruby containers with AXRole: AXGroup, and their corresponding subroles. This would expose to VoiceOver the proper relationship between the ruby elements to be conveyed to the user. This patch contains the fix only for OSX. For other platforms equivalent changes to WebAccessibilityObjectWrapperMac.mm still need to be made. This fixes the symptoms of VoiceOver reading "unknown" for <rt> elements.

[chris fleizach]

Comment on attachment 253298 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=253298&action=review

Patch looks good! Thanks. 
It looks like you need to do a ./Tools/Scripts/webkit-update to bring your code inline with ToT, then you should be able to upload again and have it apply cleanly

> Source/WebCore/ChangeLog:8
> +        Test: accessibility/ruby-hierarchy-roles.html

Please add a short description of what the problem was and how you fixed it here.

[Darin Adler]

Comment on attachment 253298 [details]
Patch

Since this patch doesn’t apply, please upload a re-based patch that does apply to trunk.

[chris fleizach]

Created attachment 254850 [details]
patch

Took Andres patch and re-applied to trunk

[chris fleizach]

Comment on attachment 254850 [details]
patch

Looks like it built and ran ok. let's get this in

[WebKit Commit Bot]

Comment on attachment 254850 [details]
patch

Clearing flags on attachment: 254850

Committed r185536: <http://trac.webkit.org/changeset/185536>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Alex Christensen]

accessibility/ruby-hierarchy-roles.html always fails on Windows.  Should it be skipped?

[chris fleizach]

Yes it should

We should probably move the test into platform Mac only for now

I can do that in a few hours (or feel free to do so)

[chris fleizach]

(In reply to comment #21)
> Yes it should
> 
> We should probably move the test into platform Mac only for now
> 
> I can do that in a few hours (or feel free to do so)

Moved tests to Mac
http://trac.webkit.org/changeset/185543

[Alexey Proskuryakov]

> Moved tests to Mac

This has not been communicated well, however this is not the right thing to do. Tests in platform/ subdirectories are hard to maintain, as it's impossible to figure out where to put expectations. E.g. would Mavericks WK2 expectations for this test be in platform/mac/platform/mac-mavericks/platform/wk2, or in platform/mac/platform/wk2/platform/mac-mavericks, or in platform/mac-mavericks/platform/wk2, or somewhere else?

Even barring that, a test that doesn't have any strong ties to a platform other than temporary implementation limitations should not be in a platform directory. Skipping the test in TestExpectations with an explanation of what needs to be implemented would provide a lot better visibility.

[chris fleizach]

(In reply to comment #23)
> > Moved tests to Mac
> 
> This has not been communicated well, however this is not the right thing to
> do. Tests in platform/ subdirectories are hard to maintain, as it's
> impossible to figure out where to put expectations. E.g. would Mavericks WK2
> expectations for this test be in
> platform/mac/platform/mac-mavericks/platform/wk2, or in
> platform/mac/platform/wk2/platform/mac-mavericks, or in
> platform/mac-mavericks/platform/wk2, or somewhere else?
> 
> Even barring that, a test that doesn't have any strong ties to a platform
> other than temporary implementation limitations should not be in a platform
> directory. Skipping the test in TestExpectations with an explanation of what
> needs to be implemented would provide a lot better visibility.

In this case I think a platform test is better. The fix mostly labels ruby groups with subrole affiliations which only exist on the Mac. If gtk were to do something similar they would have a different result. So this patch does not exercise a common core experience, but rather focuses on the Mac details. 

Since so much of the accessibility implementation is dependent on how the platform exposes attributes, there's many more platform tests that make sense than perhaps other areas of the code
Thanks