Bug 11200 – Image alt text not included in plain-text version when copying

• Status: RESOLVED
• Resolution: FIXED
• Component: Images

[Peter Hosey]

Steps to reproduce:
1. Select some range of text in a WebView that includes an image (the image having been specified with an img element that has a non-empty alt attribute).
2. Copy.
3. In TextEdit, create a new plain-text document.
4. Paste.

Expected behavior:
In plain-text flavors on the pasteboard (such as public.utf8-plain-text), the alt text is included in the place of the image, as demonstrated when pasted into the plain-text view.

Actual behavior:
The alt text is omitted.

Regression:
None known.

Notes:
In Adium, we use a WebView for message display (the WebKit Message View, or WKMV). When a user has an emoticon pack enabled, emoticons are inserted into the WKMV as images, with the emoticon's text equivalent (e.g. “:)”) in the alt attribute.

When the user tries to copy part of a message from the WKMV, he naturally expects that the emoticon will come with it. This works when pasting into a rich-text editor (such as TextEdit in rich-text mode), but not when pasting into a plain-text editor (such as TextEdit in plain-text mode); in the latter case, the image is missing.

The same expectation would apply on forum websites and some blogs, where emoticons in a defined set are transparently replaced with images.

[Peter Hosey]

Created attachment 10961 [details]
Test case: An image inline in a paragraph of text.

[Robert Blaut]

Created attachment 21686 [details]
standalone test case

[Robert Blaut]

The issue is side effect of a problem described in bug 5566 and, in my opinion bug 5566, is a blocker bug for this issue.

[Martin Robinson]

*** Bug 42810 has been marked as a duplicate of this bug. ***

[Martin Robinson]

*** Bug 34896 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

*** Bug 62772 has been marked as a duplicate of this bug. ***

[Vlad Alexander]

FYI, this bug is fixed in IE10 consumer preview. And FF has always supported this correct behavior.

[Tony Chang]

Created attachment 194106 [details]
Patch

[Tony Chang]

Ryosuke, can you review the Editor changes?  mitz, can you review the TextIterator changes?

[Darin Adler]

Comment on attachment 194106 [details]
Patch

So with this we’ll have image alt text in plain text, but not in styled text, on the clipboard.

[Darin Adler]

(In reply to comment #10)
> So with this we’ll have image alt text in plain text, but not in styled text, on the clipboard.

I guess that’s intentional because in styled text we have the actual images?

[Tony Chang]

(In reply to comment #11)
> (In reply to comment #10)
> > So with this we’ll have image alt text in plain text, but not in styled text, on the clipboard.
> 
> I guess that’s intentional because in styled text we have the actual images?

Yes, that's right.

[Alexey Proskuryakov]

I don't think that pasteboard methods always write to what one expects to be "clipboard". For example, OS X services use pasteboards.

[Peter Hosey]

(In reply to comment #13)
> I don't think that pasteboard methods always write to what one expects to be "clipboard". For example, OS X services use pasteboards.

Agreed. If I'm feeding the selection to a service that expects plain text, I expect the same behavior as if I'd copied the text and pasted into the app (or pasted into TextEdit and run the service from there).

[Tony Chang]

(In reply to comment #14)
> (In reply to comment #13)
> > I don't think that pasteboard methods always write to what one expects to be "clipboard". For example, OS X services use pasteboards.
> 
> Agreed. If I'm feeding the selection to a service that expects plain text, I expect the same behavior as if I'd copied the text and pasted into the app (or pasted into TextEdit and run the service from there).

Good point. I should rename the function to selectedTextForPasteboard (other suggestions welcome) to make it more clear that that is what the patch is doing.  E.g., it's intentional that the alt text ends up in drag&drop and in the find pasteboard.

This matches the behavior in Firefox on OS X.  E.g., in the standalone test case attached to this bug, if you select the Adium image and use the "Search with Google" service, it will search for text with "Adiumy" in it.

[Alexey Proskuryakov]

> E.g., in the standalone test case attached to this bug, if you select the Adium image and use the "Search with Google" service, it will search for text with "Adiumy" in it.

Indeed, that's the case I had in mind, and this behavior is clearly a bug in Firefox.

[Peter Hosey]

(In reply to comment #16)
> > E.g., in the standalone test case attached to this bug, if you select the Adium image and use the "Search with Google" service, it will search for text with "Adiumy" in it.
> 
> Indeed, that's the case I had in mind, and this behavior is clearly a bug in Firefox.

I disagree. If the service accepts images or rich text, then WebKit should send that service an image and/or rich text that includes the image (but that'd be a different bug). If it accepts plain text, then WebKit should send it all of the plain text in the selection—including the alternative text of the image, as described in my original bug report.

I borrow your own example: If I want to search Google for the Adium logo, I might select the logo and choose “Search with Google”. If possible, the service should actually reverse-image-search the image using Google Image Search, but I doubt that it does that; more probably, it can only accept and search for plain text, in which case a search for “Adiumy” will find the logo and/or relevant pages, whereas a search for “” will not.

[Ryosuke Niwa]

Comment on attachment 194106 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=194106&action=review

> Source/WebCore/editing/Editor.cpp:2614
>  String Editor::selectedText() const

I would have made this function take a boolean or enum instead.

> Source/WebCore/editing/Editor.cpp:2624
> +String Editor::selectedText(TextIteratorBehavior behavior) const

I don't think we want to expose TextIteratorBehavior like this. TextIteratorBehavior should be considered as implementation details of WebCore.

> Source/WebCore/editing/TextIterator.cpp:720
> +            m_positionOffsetBaseNode = m_node;
> +            m_positionStartOffset = 0;
> +            m_positionEndOffset = 1;

Why are we iterating into img element? Is this really necessary?

[Alexey Proskuryakov]

> If it accepts plain text, then WebKit should send it all of the plain text in the selection—including the alternative text of the image, as described in my original bug report.

I think that it's a huge stretch to claim that a user may expect some invisible text to pop up as a replacement when an image is selected. This is an amount of second-guessing the intent that's just creepy.

Also, this will clutter the services menu with unrelated items, which will not have the expected behavior when used on images. For example, "Find in Dictionary" clearly has nothing to do with images.

[Peter Hosey]

(In reply to comment #19)
> > If it accepts plain text, then WebKit should send it all of the plain text in the selection—including the alternative text of the image, as described in my original bug report.
> 
> I think that it's a huge stretch to claim that a user may expect some invisible text to pop up as a replacement when an image is selected.

I don't think it's a huge stretch to claim that a user expects everything they have selected to be present in what they send to a service or copy to the general or Find pasteboard.

Your example was of an image that is “obviously” (to use an inherently subjective word) not part of the text, but consider the example I originally presented, where the image—an emoticon—*is* part of the text.

The very purpose of the alt attribute is to provide text to substitute in when the image element is transferred to a context that cannot support images. Text-to-speech is one example (and, returning to the emoticon example, OS X's text-to-speech can pronounce emoticons as such rather than as punctuation), but services such as create-new-Mail-message are also good ones.

If I select, say, a forum message, and want to send it as a (plain text) Mail message or read it aloud using the speech service, any emoticons in it should be included. I hold that this is true generally of all images with alternate text specified. The images should not simply be excluded from representation.

> This is an amount of second-guessing the intent that's just creepy.

I think you've overusing the word “creepy”.

> Also, this will clutter the services menu with unrelated items, which will not have the expected behavior when used on images. For example, "Find in Dictionary" clearly has nothing to do with images.

That is a valid point.

That acknowledged, more services are concerned with larger runs of text—suitable for copying to an email message or new document, for example—than with single words. With such selections and such services, it makes more sense to include the image's alternate text than to exclude it.

One could also include some smarts for not promising text when the complete text is empty or all-whitespace. That way, if the user selects *only* an image and it doesn't have any alternate text, text-based services will not show up to dine at an empty banquet.

[Vlad Alexander]

(In reply to comment #19)

>some invisible text to pop up as a replacement when an image is selected
Are you saying that alternate text is invisible text? If so, it clearly is a matter of one’s perception. For blind users and those who turned off image rendering, images are invisible objects and alternate text is the norm.

700+ people including accessibility experts have signed off on these 3 test-cases:

http://a11ybugs.org/bug-1.php
http://a11ybugs.org/bug-2.php
http://a11ybugs.org/bug-3.php

PLEASE use these test-cases to implement correct behavior.

[Alexey Proskuryakov]

Comment on attachment 194106 [details]
Patch

I still think that the issue with services needs to be resolved to land this. Ryosuke's comments appear to require changes that need to be reviewed too.

[Tony Chang]

Created attachment 194355 [details]
Patch

[Tony Chang]

I reworked the patch to avoid putting alt text into Services.  This involved moving the enum and some logic into Pasteboard.

(In reply to comment #18)
> (From update of attachment 194106 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=194106&action=review
> 
> > Source/WebCore/editing/Editor.cpp:2614
> >  String Editor::selectedText() const
> 
> I would have made this function take a boolean or enum instead.

I tried to use the enum from Pasteboard, but it turns out that you can't include Pasteboard.h from Editor.h because Pasteboard.h and WebArchive.h declare WebArchivePboardType.

Also, by not changing the signature of selectedText(), I avoid having to update all the exports.

Let me know if this looks OK to you.

> > Source/WebCore/editing/Editor.cpp:2624
> > +String Editor::selectedText(TextIteratorBehavior behavior) const
> 
> I don't think we want to expose TextIteratorBehavior like this. TextIteratorBehavior should be considered as implementation details of WebCore.

We're not exposing TextIteratorBehavior; this method is private.

> > Source/WebCore/editing/TextIterator.cpp:720
> > +            m_positionOffsetBaseNode = m_node;
> > +            m_positionStartOffset = 0;
> > +            m_positionEndOffset = 1;
> 
> Why are we iterating into img element? Is this really necessary?

I moved the code to share more of the existing code. It should make it more clear that I'm not changing the iteration order, I'm just pointing it at the alt text.

[EFL EWS Bot]

Comment on attachment 194355 [details]
Patch

Attachment 194355 [details] did not pass efl-ews (efl):
Output: http://webkit-commit-queue.appspot.com/results/17216551

[Tony Chang]

Created attachment 194376 [details]
fix efl compile

[Ryosuke Niwa]

Comment on attachment 194376 [details]
fix efl compile

View in context: https://bugs.webkit.org/attachment.cgi?id=194376&action=review

> Source/WebCore/platform/Pasteboard.h:76
> +enum PasteboardSelectedTextType { DefaultSelectedTextType, ForClipboard };

I would have preferred to call the latter IncldueImageAltTextForClipboard to make the intent this enum clear
but others might prefer the current style. Also PasteboardSelectedTextType doesn't really communicate the semantics of this enum.
How about something like ShouldSerializeSelectedTextForClipboard?

[Ryosuke Niwa]

I've asked Enrica to take a look at this patch since she's most familiar with services.

[Tony Chang]

(In reply to comment #27)
> (From update of attachment 194376 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=194376&action=review
> 
> > Source/WebCore/platform/Pasteboard.h:76
> > +enum PasteboardSelectedTextType { DefaultSelectedTextType, ForClipboard };
> 
> I would have preferred to call the latter IncldueImageAltTextForClipboard to make the intent this enum clear
> but others might prefer the current style. Also PasteboardSelectedTextType doesn't really communicate the semantics of this enum.
> How about something like ShouldSerializeSelectedTextForClipboard?

I'm happy to make these renames.  I'll do it after Enrica takes a look.  Thanks!

[Enrica Casucci]

Comment on attachment 194376 [details]
fix efl compile

View in context: https://bugs.webkit.org/attachment.cgi?id=194376&action=review

> Source/WebCore/editing/mac/EditorMac.mm:271
> +    pasteboard.writeSelectionForTypes(pasteboardTypes, true, m_frame, DefaultSelectedTextType);

This call and the following are used by services. If I understand the code correctly, no change in behavior will occur with your patch.

[Enrica Casucci]

I share Alexey's concerns about this new approach.
This patch changes the behavior of copy and cut commands as well as the behavior of finding a string from the current selection. It doesn't seem to affect Mac OS X Services, since it doesn't change the behavior of stringSelectionForPasteboard. (Please confirm that this statement is true).

Overall I think this is a change with a lot of ramifications. It will change significantly what goes in the pasteboard when the user selects content that has both text and images.
Prior to this change the plain text representation contained only the text selected; after this change the pasteboard will contain text coming from different contexts, the text on the page and the text in the alt attribute of the image, which could produce text that makes no sense.

Maybe it would be better to limit this behavior to a different pasteboard type. Such approach has the disadvantage that existing applications would not inherit the new behavior automatically, but it would also prevent unwanted side effects.
Alternatively it could be limited only to ports that are interested in this feature.

[Tony Chang]

(In reply to comment #31)
> This patch changes the behavior of copy and cut commands as well as the behavior of finding a string from the current selection. It doesn't seem to affect Mac OS X Services, since it doesn't change the behavior of stringSelectionForPasteboard. (Please confirm that this statement is true).

Yes, that is correct.

[Ryosuke Niwa]

Perhaps we should add a new setting for this? I can see that different browsers want to do this differently.

[Peter Hosey]

(In reply to comment #31)
> Prior to this change the plain text representation contained only the text selected; after this change the pasteboard will contain text coming from different contexts, the text on the page and the text in the alt attribute of the image, …

That's the behavior I requested in the original bug. Please see the “Notes” in my original report for an example use case (in a word: emoticons). Also please see Vlad Alexander's comment above for the ways that this helps accessibility.

> … which could produce text that makes no sense.

Certainly true, but I think it is better to include the text, whereupon the user can spot it and remove it if it is unwanted, than to exclude it, where the user may not notice that it is missing and may have much greater difficulty in restoring it in a large selection or for distinctive alt texts.

> Maybe it would be better to limit this behavior to a different pasteboard type.

Pasteboard name, or data type? Either way, I suspect nobody would use it, and certainly nobody doing the standard copy-then-paste would benefit from it.

[Enrica Casucci]

(In reply to comment #34)
> (In reply to comment #31)
> > Prior to this change the plain text representation contained only the text selected; after this change the pasteboard will contain text coming from different contexts, the text on the page and the text in the alt attribute of the image, …
> 
> That's the behavior I requested in the original bug. Please see the “Notes” in my original report for an example use case (in a word: emoticons). Also please see Vlad Alexander's comment above for the ways that this helps accessibility.

I understand that this is the scenario you're trying to address, but even though I see a number of cases where this would be beneficial, I can come up with other examples where this would produce a not so great experience.

> Certainly true, but I think it is better to include the text, whereupon the user can spot it and remove it if it is unwanted, than to exclude it, where the user may not notice that it is missing and may have much greater difficulty in restoring it in a large selection or for distinctive alt texts.
> 
I agree that the text could be removed, but there could be cases, where the user is pasting a lot of text and doesn't notice the additional text that doesn't make sense together.

> Pasteboard name, or data type? Either way, I suspect nobody would use it, and certainly nobody doing the standard copy-then-paste would benefit from it.
Yes I was suggesting a different pasteboard type and I also said that it has the problem of requiring adoption.
I'm not trying to push back on the feature per se, but I would like to encourage everyone to vet the consequences of a similar change. That is why I proposed that maybe this is something that could be enabled only for the ports that feel comfortable with the new behavior.

[Enrica Casucci]

I'm also unsure what would be the impact of the new behavior on features like Voice over or Speak Selection. I'll talk to some of the accessibility experts at Apple to get their opinion.

[chris fleizach]

(In reply to comment #36)
> I'm also unsure what would be the impact of the new behavior on features like Voice over or Speak Selection. I'll talk to some of the accessibility experts at Apple to get their opinion.

Speak selection users would end up hearing the Alt text inline, which unless it's called out as the alt text of the image, would probably not be a great experience.

My two cents are that it seems like this is in need of an API for a web page to return what the selected range of text will be when put on the clipboard. That would solve the adium problem

[Vlad Alexander]

(In reply to comment #35)

>I can come up with other examples where this would produce a not so great experience.

Do you know what really is "not so great experience"? It's when a visually impaired person is deprived of equitable access to online information because their browser does not support basic HTML accessibility features. The quality of alternate text will not improve unless it is made usable to to everyone by including it in copy/paste, passing it to search/translation services and displaying it in place of images when an image cannot be loaded.

[chris fleizach]

(In reply to comment #38)
> (In reply to comment #35)
> 
> >I can come up with other examples where this would produce a not so great experience.
> 
> Do you know what really is "not so great experience"? It's when a visually impaired person is deprived of equitable access to online information because their browser does not support basic HTML accessibility features. The quality of alternate text will not improve unless it is made usable to to everyone by including it in copy/paste, passing it to search/translation services and displaying it in place of images when an image cannot be loaded.

I don't know if this bug is depriving anyone of equitable access. What you're getting here is plain text copied off a web page.

No matter whom views that plain text, visually impaired or not, they're going to get the same plain text. (right now without alt text). The sighted person is not going to see an image, and no one using a screen reader will hear the alt text.

However, who ever went to the web page would see the image, or hear the alt label. 

If instead, this bug is talking about when an image is pasted back into a rich text area, and it does not carry its alt text, than that would be an equal access issue.

[James Craig]

(In reply to comment #38)

> Do you know what really is "not so great experience"? It's when a visually impaired person is deprived of equitable access to online information because their browser does not support basic HTML accessibility features.

Try not to be combative, Vlad. Chris implemented the majority of accessibility in WebKit and has done more for the accessibility community in the past couple years than most accessibility professionals will accomplish in their lifetimes. Instead of questioning his motives, try to keep in mind that we're all in this together to come up with the best experience possible for all users, including those with disabilities and those without.

That said, I think it'd be reasonable to wrap image alternative text in delimiters clearly indicating it as such in the plain text PasteBoard. 

Something like:
  <p>Foo <img src="bar.png" alt="Bar"> Baz</p>

Could come out as:
  "Foo [Image: 'Bar'] Baz"

And could use the ARIA 1.1. "text" role to bypass the delimiters, such as:
  <p>Foo <img role="text" src="bar.png" alt="Bar"> Baz</p>

Could come out as:
  "Foo Bar Baz"

But then we get into the very subjective area of whether or not the plain text PasteBoard should include things like abbr[title] text, table[summary], etc.

[Peter Hosey]

(In reply to comment #40)
> That said, I think it'd be reasonable to wrap image alternative text in delimiters clearly indicating it as such in the plain text PasteBoard. 
> 
> Something like:
>   <p>Foo <img src="bar.png" alt="Bar"> Baz</p>
> 
> Could come out as:
>   "Foo [Image: 'Bar'] Baz"
> 
> And could use the ARIA 1.1. "text" role to bypass the delimiters, such as:
>   <p>Foo <img role="text" src="bar.png" alt="Bar"> Baz</p>
> 
> Could come out as:
>   "Foo Bar Baz"

That solution, all together, is very reasonable.

> But then we get into the very subjective area of whether or not the plain text PasteBoard should include things like abbr[title] text, table[summary], etc.

If anyone wants to propose including those other attributes in similar cases, I think they'd be worth debating as separate bugs.

The case for using the alt text here is backed up by its definition in the HTML spec:

http://www.whatwg.org/specs/web-apps/current-work/multipage/embedded-content-1.html#alt

> … the value [of the alt attribute] must be an appropriate replacement for the image. …
> 
> The most general rule to consider when writing alternative text is the following: the intent is that replacing every image with the text of its alt attribute not change the meaning of the page.
> 
> So, in general, alternative text can be written by considering what one would have written had one not been able to include the image.

[Ryosuke Niwa]

(In reply to comment #41)
>
> If anyone wants to propose including those other attributes in similar cases, I think they'd be worth debating as separate bugs.
> 
> The case for using the alt text here is backed up by its definition in the HTML spec:
> http://www.whatwg.org/specs/web-apps/current-work/multipage/embedded-content-1.html#alt

That's a good guideline for authors. Unfortunately, many existing websites do not follow this guidelines. Of course, this is somewhat of a chicken and egg problem in that providing better support for alt in browser will encourage more authors to use it correctly.

I think the best course of action for this bug is to wrap the feature in settings or build flags or that each embedder of WebKit can decide which behavior they want.

[Vlad Alexander]

> I don't know if this bug is depriving anyone of equitable access.
If the quality of alternate text on the web is poor, then people are deprived of equitable access. The only way to improve the quality of alternate text is to make is usable to everyone (for example, in cases of copy/paste and passing it to search/translation services). So if you work backwards, if this bug is not fixed properly, fewer people will use alternate text and content authors will have less incentive to write better alternate text. And then you are depriving people who rely on alternate text of equitable access.

> Try not to be combative, Vlad.
I don't mean to be. But this bug has been filled in 2006 with no results. I have been in communication with the Apple team since about the same time regarding other accessibility issues (check Radar) with the same results. I myself have also briefly worked at Apple so I know from experience that accessibility issues are misunderstood and are a low priority for Apple. So if my tone can be a bit short - I apologize, but please understand where it's coming from too.

>wrap image alternative text in delimiters
So long as those delimiters are not visible to end-users. Otherwise you are introducing content that is not there. And it does not translate into other languages (but that is besides the point).

Why can we not implement the same behavior as in Firefox?

[James Craig]

(In reply to comment #43)
> > I don't know if this bug is depriving anyone of equitable access.
> If the quality of alternate text on the web is poor, then people are deprived of equitable access. The only way to improve the quality of alternate text is to make is usable to everyone (for example, in cases of copy/paste and passing it to search/translation services). So if you work backwards, if this bug is not fixed properly, fewer people will use alternate text and content authors will have less incentive to write better alternate text. And then you are depriving people who rely on alternate text of equitable access.

I think there is a causation/correlation fallacy in that argument. A WebKit feature's primary purpose should be to do the right thing for the user, not to train web developers to write better alt text.

> I myself have also briefly worked at Apple so I know from experience that accessibility issues are misunderstood and are a low priority for Apple. 

Again, please try not to be combative or make assumptions of others motives. It does more harm than good.

> >wrap image alternative text in delimiters
> So long as those delimiters are not visible to end-users. Otherwise you are introducing content that is not there. 

The content is there. It's an image. 

> And it does not translate into other languages (but that is besides the point).

That's just a simple Loc string. WebKit is localized in many languages, and this string would be localized with the rest of the project.

[James Craig]

Tony, will you update the patch to include the delimiters and a few more tests?

// alt text
<p>Foo <img src="bar.png" alt="Bar"> Baz</p>
"Foo [Image: 'Bar'] Baz"

// empty alt text
<p>Foo <img src="bar.png" alt=""> Baz</p>
"Foo  Baz"

// missing alt attr
<p>Foo <img src="bar.png"> Baz</p>
"Foo  Baz"

// multiple images
<p>Foo <img src="bar.png" alt="Bar"> <img src="baz.png" alt="Baz"> Bop</p>
"Foo [Image: 'Bar'] [Image: 'Baz'] Bop"

// alt text with ARIA 1.1 text role
<p>Foo <img role="text" src="bar.png" alt="Bar"> Baz</p>
"Foo Bar Baz"

// make sure no extra spaces added to pasteboard string
<p>Foo<img src="bar.png" alt="Bar"><img src="baz.png" alt="Baz">Bop</p>
"Foo[Image: 'Bar'][Image: 'Baz']Bop"

// make sure no extra spaces added to pasteboard string (with text role)
<p>Foo<img src="bar.png" alt="Bar"><img src="baz.png" alt="Baz">Bop</p>
"FooBarBazBop"

[James Craig]

That last one should have read:

// make sure no extra spaces added to pasteboard string (with text role)
<p>Foo<img role="text" src="bar.png" alt="Bar"><img role="text" src="baz.png" alt="Baz">Bop</p>
"FooBarBazBop"

[Ryosuke Niwa]

I object to adding arbitrary delimiters like that.

[Ryosuke Niwa]

Given the discussion so far, it's probably worth announcing this feature on webkit-dev.  The current patch looks good but I strongly feel that we need a setting so that each port can opt-in to use this feature.

[Peter Hosey]

For the record, I'm not a fan of delimiters, either, but if going without them is a problem, having them by default with an attribute to turn them off seems like a reasonable solution.

An alternative would be to add the feature without delimiters, and then see whether it really is a problem to not have them. It might be that most extant non-empty alt texts are simply either (a) appropriate to be included without delimiters or (b) obviously not text anyway (e.g., filename templated in), so delimiters don't make it any more obvious.

[Ryosuke Niwa]

(In reply to comment #49)
> An alternative would be to add the feature without delimiters, and then see whether it really is a problem to not have them. It might be that most extant non-empty alt texts are simply either (a) appropriate to be included without delimiters or (b) obviously not text anyway (e.g., filename templated in), so delimiters don't make it any more obvious.

Right. Also, I'm concerned that the proposed delimiter won't be read well naturally by VoiceOver and auditory UAs.

[James Craig]

(In reply to comment #50)

> I'm concerned that the proposed delimiter won't be read well naturally by VoiceOver and auditory UAs.

I verified that it is spoken as expected with Speak Selection and VoiceOver on both iOS and OSX. Just as importantly, the “Image” delimiter gives both auditory renderings a context that is missing from the non-delimited version, one that is clearly conveyed to sighted users and VoiceOver users in the HTML version: the fact that this is an image, not rendered text. The delimiters also address the concern Chris mentioned regarding speak selection, and the concern I had about the plain text renderings.

[Alexey Proskuryakov]

FWIW, I share Ryosuke's concern about delimiters:

1. They, make it so that the original use case in this bug is served poorly, giving extra brackets around smileys. I really don't want Mac users to get nerdy brackets all over the place when copying from Adium. Hello dear! [Image: ':)']

2. They are quite arbitrary, not matching what other browsers do.

3. They still don't address the concern about getting unexpected stealthy content in pasted text. When there is a large body of text copied, it's easy to overlook the brackets.

[James Craig]

(In reply to comment #52)
> FWIW, I share Ryosuke's concern about delimiters:
> 
> 1. They, make it so that the original use case in this bug is served poorly, giving extra brackets around smileys. I really don't want Mac users to get nerdy brackets all over the place when copying from Adium. Hello dear! [Image: ':)']

Adium should use the 'text' role on the emoticon images then. That’s the point of that role.

<p>Hello dear! <img src="smiley.gif" role="text" alt=":)"></p>
"Hello dear! :)"

> 2. They are quite arbitrary, not matching what other browsers do.

I disagree that they are arbitrary, but would be open to discussing other delimiters if you have a suggestion. As to matching other browser’s behavior, that’s because this is behavior is not specified anywhere, and as such we should strive for the best solution, not the status quo.

I’m mostly concerned about the case where photos or other graphics with real, author-provided or generated alt text are inserted in the middle of a sentence with no context, and the current solution could break the sentence for auditory contexts (hence Chris’s concern with speak selection) without clarifying that this was alternative text from an inline image.

> 3. They still don't address the concern about getting unexpected stealthy content in pasted text. When there is a large body of text copied, it's easy to overlook the brackets.

It’s much easier to both notice, and search for, consistent delimiters than for arbitrary strings of alt text with no delineation.

[Vlad Alexander]

>Adium should use the 'text' role on the emoticon images then. That's the point of that role.

ARIA is complex, not supported by most authoring tools and puts an undue burden on non-technical content authors. As an authoring tool vendor, I can speak to content author behavior. It is highly unlikely that content authors will adopt this.

>As to matching other browser's behavior, that's because this is behavior is not specified anywhere, and as such we should strive for the best solution, not the status quo.

I don't mean to sound argumentative, but on what basis is your solution better than that of Firefox and IE10? I would argue that given accessibility expersts reviewed this at Mozilla and Microsoft, and 700+ petitioners asked for the same behavior as in Firefox, that it should have more weight than one developer's view of what is best.

[James Craig]

(In reply to comment #54)
> >Adium should use the 'text' role on the emoticon images then. That's the point of that role.
> 
> ARIA is complex, 

Adding a text role is not complex, and in the exception case like Adium's emoticons, adding a role is trivial. Your example case of "I ♥ you" is the exception, not the rule. Most alt text is used on regular images, not inline replacements like your example, so the default should be to indicate what was actually there. The word "love" is not displayed, so copying it to the pasteboard without any indication that it was alternative text for an image is not representing the true content. 

For example, the following text (and alt text) is from a NYTimes story promo picked at random. The insertion of the bracketed delimiters and "Image" role description gives this phrase context, where without it, it would seem like the author's first sentence (it's also not even a complete sentence), and a strangely phrased one if the intention is to get you to read the article.

Watch Out for the Horses on Your Way to the Train
By MELENA RYZIK
6:18 PM ET
[Image: "Students from the Ailey School at a dress rehearsal on Sunday at Grand Central Terminal for “Heard NY,” a performance that takes place there this week."]
An arts project is bringing life-size horses, each made of colorful raffia and fitting two dancers, to the main concourse of Grand Central Terminal.

> [ARIA is] not supported by most authoring tools and puts an undue burden on non-technical content authors. As an authoring tool vendor, I can speak to content author behavior. It is highly unlikely that content authors will adopt this.

The point is that authors won't do this most of the time, because most of the time it would be inappropriate to use the text role. See the above example, which is a more common use of @alt than for emoticons and inline text replacements.

> >As to matching other browser's behavior, that's because this is behavior is not specified anywhere, and as such we should strive for the best solution, not the status quo.
> 
> I don't mean to sound argumentative, but on what basis is your solution better than that of Firefox and IE10? I would argue that given accessibility expersts reviewed this at Mozilla and Microsoft, and 700+ petitioners asked for the same behavior as in Firefox, that it should have more weight than one developer's view of what is best.

First of all, the petition had nothing to do with matching Firefox behavior exactly. The petition was actually about exposing alt text in the plain text pasteboard, and we all seem to be in agreement of that. I'm sure I even have a duplicate bug filed somewhere.

Second, to your point about "experts" and signatures, even if the petition had specifically been about whether or not to use delimiters around the alt text, the number of votes would be irrelevant. Web browsers features should always be about technical merit, not popularity. If you want standardized behavior of this in pasteboard rules, that standardization should happen in the HTML Accessibility Task Force sub-team of the HTML Working Group (or perhaps in the PF group), not in a WebKit bug thread, and not on your personal petition website.

Finally, and most importantly, we have a lot more than 700 users using the previously mentioned "Speak Selection" feature on a daily basis on iOS and OS X. Chris's original point was to try to do this in a way that would not cause unexpected behavior for them, and I made a compromise suggestion that at least more people than just "one developer" consider reasonable. Speak Selection is an existing assistive technology that relies on the plain text pasteboard, and we want its users to be negatively affected by this change. I think it's reasonable to include alt text in that pasteboard, but we should not misrepresent it as plain text content, because it's not. It's an alternative for an image.

I've stated my case. If you want to convince us to have the pasteboard hide the fact that this is an image, then take the time to make a merited argument. Don't just say "experts" and "petitioners" want this, because you don't know that they do. They merely signed a petition stating that they want alt text exposed, which is something I think we all want.

[James Craig]

Ahem. Typo: …and we DO NOT want its users to be negatively affected by this change.

[Ryosuke Niwa]

> I've stated my case. If you want to convince us to have the pasteboard hide the fact that this is an image, then take the time to make a merited argument. Don't just say "experts" and "petitioners" want this, because you don't know that they do. They merely signed a petition stating that they want alt text exposed, which is something I think we all want.

Who are "us" here? I haven't seen anyone else supporting the proposed delimiters. Could you give me pointers to where this proposal has been supported?

[James Craig]

(In reply to comment #57)
> > I've stated my case. If you want to convince us to have the pasteboard hide the fact that this is an image, then take the time to make a merited argument. Don't just say "experts" and "petitioners" want this, because you don't know that they do. They merely signed a petition stating that they want alt text exposed, which is something I think we all want.
> 
> Who are "us" here? I haven't seen anyone else supporting the proposed delimiters. Could you give me pointers to where this proposal has been supported?

Sorry, I meant "us" in the general sense, not anyone in particular, and not an "us" versus "you" implication. Poor choice of words.

But since you asked, there's at least this comment above, from Peter Hosey ("That solution, all together, is very reasonable."), as well as others (Chris and Enrica) expressing concern for non-delimited alt text exposed as plain text when it is not. ("…unless it's called out as the alt text of the image, would probably not be a great experience.")

[Vlad Alexander]

I am advocating that in the absence of a clear consensus on alternate text delimiters (which is where we are now), the decision should be determined based on what other browser vendors have done. There is precedent for this. Browser vendors have copied each others behavior (bugs, etc.) for the sake of consistency. Consistent behavior across browsers has value in its own right by providing a predictable environment for developers, content authors and end-users. I therefore ask that WebKit fix this bug to match behavior in Firefox and IE10.

[200ok]

Adding a vote for including alt attribute contents in copy+pasted text.

I've worked on a product where users would include emoticons, tick/cross indications of yes/no, status lozenges and other graphics inline with text; and without the alt text the copy+pasted content either lacks important nuance or becomes unreadable.

Consider: 
* "You idiot! :D" vs. "You idiot!" ...the alt text of the smiley *significantly* alters the meaning/interpretation of the text.
* Issue1 (tick) Issue2 (cross) ...without the alt text, you cannot determine the status of the two issues. Accepting that tick/cross is vague in the first place, but it's a common pattern.

If this thread has reached a point where it is simply deciding about delimiters: delimiters seem unecessary, although arguably a character of white space at the start and end would be advisable to ensure the alt contents are evaluated on their own and not run into other text. 

Certainly if the choice is between no alt text at all; and delimited alt text - go with delimited alt text.

[Laura Carlson]

Please fix web kit so that it uses the alternate text when copying content that contains images. Make it a simple and non obtustive retrieval process for the user as in:
http://a11ybugs.org/bug-1.php
Over 700 people have asked that this be fixed. Being consistent with other browser solutions would improve usability and interoperability of the web.

> That said, I think it'd be reasonable to wrap image alternative text 
> in delimiters clearly indicating it as such in the plain text 
> PasteBoard. 

Using delimiters was for alt attribute values are not needed. In fact they were rejected in HTML5.
http://www.w3.org/html/wg/tracker/issues/58

>>> Adium should use the 'text' role on the emoticon images then.
>>> That's the point of that role.
>> 
>> ARIA is complex, 

WAI-ARIA is a bridging technology. Bridging technologies are not intended to replace native HTML semantic features. Detouring accessibility technical requirements into an "Accessible Rich Internet Applications" bridging specification is backward. The "Introduction to ARIA" document clearly states, "WAI-ARIA is intended to be used as a supplement for native language semantics, not a replacement. When the host language provides a feature that provides equivalent accessibility to the WAI-ARIA feature, use the host language feature."
http://www.w3.org/TR/wai-aria/introduction

WAI-ARIA should only be used as a last resort when content can't be made accessible using the host language.

The first rule of ARIA use is: "If you can use a native HTML element or attribute with the semantics and behaviour you require already built in, instead of re-purposing an element and adding an ARIA role, state or property to make it accessible, then do so."
http://dvcs.w3.org/hg/aria-unofficial/raw-file/tip/index.html#rule1

The Protocols and Formats Working Group is the group that authors WAI-ARIA. Their charter states: "Note that WAI-ARIA is intended to be a bridging technology. It is expected that, over time, host languages will evolve to provide semantics for objects that currently can only be declared with WAI-ARIA."
http://www.w3.org/WAI/PF/charter201006

A significant goal of WAI-ARIA is to: "help stimulate the emergence of more semantic and accessible markup. When native semantics for a given feature become available, it is appropriate for authors to use the native feature and stop using WAI-ARIA for that feature."
http://www.w3.org/TR/wai-aria/introduction#co-evolution

[James Craig]

(In reply to comment #60)

> If this thread has reached a point where it is simply deciding about delimiters: 

I think it has.

> delimiters seem unnecessary, 

They may seem unnecessary to because you are think about them for the use of smileys, which are only one use case, and one that will likely become irrelevant as more browsers support colored fonts and unicode smileys.

> although arguably a character of white space at the start and end would be advisable to ensure the alt contents are evaluated on their own and not run into other text. 

I don't think we should insert whitespace characters either. This would negatively affect Japanese Gaiji which, along with emoji or emoticons, is one of the few cases where the ARIA text role would be appropriate. Adding extra space around characters may break pronunciation.

[James Craig]

(In reply to comment #61)
> Using delimiters was for alt attribute values are not needed. In fact they were rejected in HTML5.
> http://www.w3.org/html/wg/tracker/issues/58

That's an entirely separate issue about what browsers should use for alt text when it's missing. It has nothing to do with this thread.

> WAI-ARIA is a bridging technology. Bridging technologies are not intended to replace native HTML semantic features. Detouring accessibility technical requirements into an "Accessible Rich Internet Applications" bridging specification is backward. 

No one is suggesting detouring accessibility requirements. When an image is used as text (as in the case of emoticons), the author is explicitly overriding the intended purpose of the img element.

> The "Introduction to ARIA" document clearly states, "WAI-ARIA is intended to be used as a supplement for native language semantics, not a replacement. When the host language provides a feature that provides equivalent accessibility to the WAI-ARIA feature, use the host language feature."
> http://www.w3.org/TR/wai-aria/introduction

I've read the ARIA spec a time or two. ☺ 

If you want to use the "ideal content" card, the proper host language feature, in this case, is to not use images for text. If you are using images for text (such as the emoji example), then clearing redefining the role is expected and proper. Better yet, emoji replacements for plain text emoticons should use plain text unicode representation of the emoji character, and sites (or apps like Adium) that need custom display of those characters should use @font-face with a colored emoji font.

The following paragraph should copy the right characters in any browser today.
<p>I <span class="emoji">♥</span> you, you idiot! <span class="emoji">☺</span></p>
"I ♥ you, you idiot! ☺"

> The first rule of ARIA use is: "If you can use a native HTML element or attribute with the semantics and behaviour you require already built in, instead of re-purposing an element and adding an ARIA role, state or property to make it accessible, then do so."

The built-in feature you are talking about is <img> which has the semantics of an image, not plain text. Since <img> is used as a text character here, adding the ARIA 1.1 "text" role is the right thing to do.

> The Protocols and Formats Working Group is the group that authors WAI-ARIA. Their charter states: "Note that WAI-ARIA is intended to be a bridging technology. It is expected that, over time, host languages will evolve to provide semantics for objects that currently can only be declared with WAI-ARIA."
> http://www.w3.org/WAI/PF/charter201006
> 
> A significant goal of WAI-ARIA is to: "help stimulate the emergence of more semantic and accessible markup. When native semantics for a given feature become available, it is appropriate for authors to use the native feature and stop using WAI-ARIA for that feature."
> http://www.w3.org/TR/wai-aria/introduction#co-evolution

I am very familiar with our group charter and the ARIA spec, probably more so than anyone, and I don't see the relevance of these comments. I don't see anyone is arguing that alt text should not be exposed, just that it should be clearly exposed as a text alternative, rather than misrepresenting it as text that was rendered by default.

[James Craig]

Giving some context to the text role, which was first publicly suggested by Steve Faulkner on the XTECH list in 2011.

http://lists.w3.org/Archives/Public/wai-xtech/2011Apr/0007.html

> a few suggestions for ARIA next
> 
> role="text"
> when a non text object has a role of text , its to be treated as text.
> 
> example use:
> My <img src="heart.png" alt="heart" role="text"> breaks.
> 
> provides a way convey an objects text alternative without exposing its role.

Please note the comment "…without exposing its role." This is my objection to the non-delimited suggestion. The author has chosen to use an image here. By default, every screen reader in existence clearly indicates through a spoken role description that an image is an image, not plain text. If the plain text pasteboard does not include this, it's misrepresenting the content for all users, and losing information. In addition, it's changing the way an existing assistive technology "Speak Selection" works today, without clearing indicating that the alt value is not actually rendered text.

[James Craig]

So in Steve's example, the plain text pasteboard of:

  <p>My <img src="heart.png" alt="heart" role="text"> breaks.</p>

should be:

  "My heart breaks."

…but the plain text pasteboard of:

  <p>My <img src="heart.png" alt="heart"> breaks.</p>

should be something like:

  "My [Image: 'heart'] breaks."

Copying Steve on this thread, too.

[Vlad Alexander]

>The built-in feature you are talking about is <img> which has the semantics of an image, not plain text.

James, do you believe that the web can provide equitable access to all? If so, then you should not give more value to the visual content of the img element than the textual content. The img element provides semantics of _content_ that can be rendered visually or textually. The names "img" and "alt" are just legacy names from a time that we did not know any better.

[Tony Chang]

Created attachment 194917 [details]
Patch

[Tony Chang]

- Added a setting that is off by default so ports can opt in to this behavior.
- I did the enum renames suggested in comment 27.
- I added test cases for no alt tag and 2 adjacent images.
- I updated the test case to test with the setting on and off.

[Ryosuke Niwa]

Comment on attachment 194917 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=194917&action=review

> LayoutTests/editing/pasteboard/copy-image-with-alt-text-expected.txt:1
> +This tests that image alt text makes it to the clipboard. This test requires DRT or WTR. The first two dumps should not include alt text and the third and forth dumps should include alt text.

We should be able to manually test it by copying the whole thing, right?

> LayoutTests/editing/pasteboard/copy-image-with-alt-text.html:35
> +    if (window.internals)
> +        internals.settings.setSelectionIncludesAltImageText(true);

Presumably, this setting is reset in the test runner automatically?

[Tony Chang]

Comment on attachment 194917 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=194917&action=review

>> LayoutTests/editing/pasteboard/copy-image-with-alt-text-expected.txt:1
>> +This tests that image alt text makes it to the clipboard. This test requires DRT or WTR. The first two dumps should not include alt text and the third and forth dumps should include alt text.
> 
> We should be able to manually test it by copying the whole thing, right?

Yes, but the setting is off by default, so it won't do anything unless you change the setting.

>> LayoutTests/editing/pasteboard/copy-image-with-alt-text.html:35
>> +        internals.settings.setSelectionIncludesAltImageText(true);
> 
> Presumably, this setting is reset in the test runner automatically?

Yes!

[Build Bot]

Comment on attachment 194917 [details]
Patch

Attachment 194917 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-commit-queue.appspot.com/results/17120875

[Tony Chang]

Created attachment 194927 [details]
Patch for landing

[Tony Chang]

Comment on attachment 194927 [details]
Patch for landing

Letting the mac ews bots test the patch.

[Ian 'Hixie' Hickson]

Someone using a screen reader doesn't need to be reminded that they can't see images if the writer of the page has actually provided equivalent text to replace the image with. That screen readers feel the need to keep reminding the user that they are missing images is a big (and IMHO rather unpleasant).

I think the same applies to copy-paste.

That there are pages where things work better if you do delimit than if you don't is true, but the opposite is true also. We should aim for where we want to be.

[James Craig]

(In reply to comment #74)
> Someone using a screen reader doesn't need to be reminded that they can't see images if the writer of the page has actually provided equivalent text to replace the image with. 

Most of the screen reader users I know would disagree with you. If they were operating without any contact with sighted co-workers or friends, your point might be accurate, but being able to collaborate with everyone is an important part of accessibility. If a sighted user can tell the difference between an image and text, then a blind user should be able to as well.

> That screen readers feel the need to keep reminding the user that they are missing images is a big (and IMHO rather unpleasant).

This is a user setting that can be changed in most screen readers. By default, it's usually set to accurately convey that contents of the screen, including role descriptions so the user knows if their sighted counterparts are seeing an image or text. 

In addition, as I stated above, this is about more than just screen reader users. Without the delimiter context, there is nothing to indicate than the first part of this NYTimes article (an incomplete sentence) is not the first sentence in the article, so conveying this in a plain text pasteboard without any delineation is a misrepresentation of the author's content.

Sample NYTimes Article (copied w/o delimiters):

Watch Out for the Horses on Your Way to the Train
By MELENA RYZIK 6:18 PM ET
Students from the Ailey School at a dress rehearsal on Sunday at Grand Central Terminal for “Heard NY,” a performance that takes place there this week. An arts project is bringing life-size horses, each made of colorful raffia and fitting two dancers, to the main concourse of Grand Central Terminal.

[Ryosuke Niwa]

I've already r+ed the current patch that doesn't use delimiter and match the behaviors of Firefox and Microsoft Internet Explorer.

James, I suggest we take the discussion of whether using delimiters is a good idea or not to HTML WG or one of many accessibility related working groups. Bugzilla is not an appropriate venue to continue this discussion at this point.

[WebKit Review Bot]

Comment on attachment 194927 [details]
Patch for landing

Clearing flags on attachment: 194927

Committed r146835: <http://trac.webkit.org/changeset/146835>

[WebKit Review Bot]

All reviewed patches have been landed.  Closing bug.

[Steve Faulkner]

(In reply to comment #40)
> (In reply to comment #38)
> 
> > Do you know what really is "not so great experience"? It's when a visually impaired person is deprived of equitable access to online information because their browser does not support basic HTML accessibility features.
> 
> Try not to be combative, Vlad. Chris implemented the majority of accessibility in WebKit and has done more for the accessibility community in the past couple years than most accessibility professionals will accomplish in their lifetimes. Instead of questioning his motives, try to keep in mind that we're all in this together to come up with the best experience possible for all users, including those with disabilities and those without.
> 
> That said, I think it'd be reasonable to wrap image alternative text in delimiters clearly indicating it as such in the plain text PasteBoard. 
> 
> Something like:
>   <p>Foo <img src="bar.png" alt="Bar"> Baz</p>
> 
> Could come out as:
>   "Foo [Image: 'Bar'] Baz"
> 
> And could use the ARIA 1.1. "text" role to bypass the delimiters, such as:
>   <p>Foo <img role="text" src="bar.png" alt="Bar"> Baz</p>
> 
> Could come out as:
>   "Foo Bar Baz"
> 
> But then we get into the very subjective area of whether or not the plain text PasteBoard should include things like abbr[title] text, table[summary], etc.

Hi james, 

I  agree with you that in the case where the alt text is included in a copy and paste, unless there is an indication of the context of the alt text important information is lost and the results in the majority of cases will be sub optimal.

[James Craig]

(In reply to comment #76)
> James, I suggest we take the discussion of whether using delimiters is a good idea or not to HTML WG or one of many accessibility related working groups. 

Fair enough. I’ll raise an HTML defect.

[James Craig]

HTML Defect: https://www.w3.org/Bugs/Public/show_bug.cgi?id=21403

[James Craig]

Following on at bug 115283 now that the HTML change landed.