Bug 266724 – AX: Feature request: Support WebVTT-based synthesized audio description in video

• Component: Accessibility
• Status: NEW

[Adrian Roselli]

WHATWG HTML specifies `kind="descriptions"` for `<track>` so authors can provide a script for audio descriptions for associated videos:
https://html.spec.whatwg.org/multipage/media.html#attr-track-kind-keyword-descriptions

WHATWG HTML also provides parsing rules (including for making them extended audio descriptions):
https://html.spec.whatwg.org/multipage/media.html#playing-the-media-resource

The current WCAG advisory technique guides authors to use `<track>` with `kind="descriptions"` to provide a source for the browser to provide synthesized audio description:
https://www.w3.org/WAI/WCAG22/Techniques/html/H96

A caveat is that the latest update from WebVTT CG Report suggests WHATWG HTML may have jumped the gun:
> The majority of the current version of this specification is dedicated to describing how to use WebVTT files for captioning or subtitling. There is minimal information about chapters and time-aligned metadata and nothing about video descriptions at this stage.
https://w3c.github.io/webvtt/#introduction

In other words, I would love to see this supported but if WHATWG HTML is punting to WebVTT and WebVTT has not fully specified it, then perhaps WebKitcan either:
1. take the lead and implement this, helping cement behaviors; or
2. push for clarity in WHATWG HTML using its outsized position of influence.

[Radar WebKit Bug Importer]

<rdar://problem/119949632>

[Eric Carlson]

WebKit added experimental support for Standard and Extended Audio Descriptions in 254266@main (332cd88dafc8) and 254502@main (c2f7594742ab).

[Adrian Roselli]

Thanks, Eric. I am copying over the instructions you added to a related WHATWG discussion (https://github.com/whatwg/html/issues/10866#issuecomment-2578996539), partly so I can track this in one place:

> WebKit has experimental support for both Standard and Extended Audio Descriptions. It uses the text to speech engine to render cues from a WebVTT track who's kind is descriptions.

> When Extended AD is enabled, playback is automatically paused if it takes longer to speak a cue's text than the cue's specified duration. If playback is paused, it is automatically resumed when the utterance finishes.

> When only Standard AD is enabled, playback is not automatically paused even if the text to speech engine has not finished speaking the cue text after the cue's end time.

> These features are not enabled by default yet, but you can try them by selecting Settings from the "Safari" menu, selecting the Feature Flags pane, typing "description" in the Search box, and checking one or both features.

Using Safari 18.2 on macOS 15.2 I did the following:

1. Safari > Settings...
2. chose "Featured Flags"
3. checked "Audio descriptions for video - Extended"
4. checked "Audio descriptions for video - Standard"
5. Visited the video at this anchor: https://adrianroselli.com/2023/12/ad-support-in-html-video.html#Synthesized
6. Played the video.
7. Heard no synthesized AD.

I closed and opened Safari. I rebooted my Mac. I reloaded the page. I tried with only one of the two options checked (and vice versa).

So either the feature does not work or my code is broken.

Do you have a sample page posted somewhere that works so I can try it?

[Eric Carlson]

My apologies, I forgot one important step - you need to enable Audio Descriptions in system preferences: 

 - Open System Settings
 - Open the Accessibility panel
 - Click on "Audio Descriptions"
 - Toggle the "Play audio descriptions when available" button

[Adrian Roselli]

I have enabled those settings and confirm it works. Thank you!

I see this feature has existed for over 2 years (a year before I filed this issues). When will this feature move out of experimental?

[Eric Carlson]

Pull request: https://github.com/WebKit/WebKit/pull/39076