Bug 229799 – Safari TP 131 audio canPlayType() reports false negatives

• Component: Media
• Status: RESOLVED FIXED

[Ashley Gullen]

In Safari TP 131, three audio formats can play with both <audio> and Web Audio, but report an empty string (indicating no support) via the canPlayType method.

Test URL: https://downloads.scirra.com/labs/audioCodecTests/

For the formats PCM WAV, WebM Vorbis and WebM Opus, observe the reported result for "HTML5 audio support".

Observed result: all three cases report an empty string, even though it is supported.

Expected result: "probably" or "maybe" result, indicating support.

[Sam Sneddon [:gsnedders]]

The lie was added in bug 226922 (to deal with the disparity between <audio> and decodeAudioData), but its fix said "Will be reverted once bug 227110 lands". however, bug 227110 has now landed and it's not been reverted.

[Radar WebKit Bug Importer]

<rdar://problem/82776747>

[Brooke Vibber]

This affects video codecs as well as audio; canPlayType() reports '' for 'video/webm; codecs="vp8"' and 'video/webm; codecs="vp9"' even if they play back just fine in <video>.

Same behavior on TP 131 and the current beta of Monterey.

This is a hard blocker for us being able to switch from WebAssembly decoding to native playback of WebM video on Wikipedia.

[Jean-Yves Avenard [:jya]]

(In reply to Brion Vibber from comment #3)
> This affects video codecs as well as audio; canPlayType() reports '' for
> 'video/webm; codecs="vp8"' and 'video/webm; codecs="vp9"' even if they play
> back just fine in <video>.
> 
> Same behavior on TP 131 and the current beta of Monterey.
> 
> This is a hard blocker for us being able to switch from WebAssembly decoding
> to native playback of WebM video on Wikipedia.

canPlayType will only return "" if either "opus" or "vorbis" was specified in the codec string.

It wouldn't affect type with just video.

Here are the results for the various queries
https://jyavenard.github.io/htmltests/tests/canPlayType.html

v.canPlayType("video/mp4; codecs="avc1.42E01E") = probably
v.canPlayType("video/mp4; codecs="vp9") = 
v.canPlayType("video/mp4; codecs="av1") = 
v.canPlayType("video/mp4; codecs="opus") = 
v.canPlayType("video/mp4; codecs="vorbis") = 
v.canPlayType("video/mp4; codecs="vp8") = 
v.canPlayType("video/webm; codecs="avc1.42E01E") = 
v.canPlayType("video/webm; codecs="vp9") = probably
v.canPlayType("video/webm; codecs="av1") = 
v.canPlayType("video/webm; codecs="opus") = 
v.canPlayType("video/webm; codecs="vorbis") = 
v.canPlayType("video/webm; codecs="vp8") = probably


as you can see, when you specify vp8 and vp9 codec, the answer is "probably"

[Jean-Yves Avenard [:jya]]

Note that when checking for MediaSource.isTypeSupported() the answer will be correct.

[Jean-Yves Avenard [:jya]]

Created attachment 437595 [details]
Patch

[youenn fablet]

Comment on attachment 437595 [details]
Patch

EWS WK1 errors seem unrelated. Relaunching bots to be sure.

[Jean-Yves Avenard [:jya]]

Comment on attachment 437595 [details]
Patch

Thanks for the review.

Yes, errors were unrelated.

[EWS]

Committed r282137 (241434@main): <https://commits.webkit.org/241434@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 437595 [details].

[Ashley Gullen]

This still reproduces in Safari TP 132. It will be a shame if Safari ships new audio codec support but they all feature detect as unavailable.

[Jean-Yves Avenard [:jya]]

(In reply to Ashley Gullen from comment #10)
> This still reproduces in Safari TP 132. It will be a shame if Safari ships
> new audio codec support but they all feature detect as unavailable.

Until the code makes its way to Safari TP; you will need to check in the experimental features "WebM Web Audio"

[Ashley Gullen]

Sorry, I'm confused - the actual support for playback of the codecs already works. The only thing not working is the feature detection, i.e. whether canPlayType() indicates support. Why would that still be marked an experimental feature or not have arrived in Safari TP yet?

[Brooke Vibber]

On Safari 15.0 shipped today; tested both Intel and ARM:

document.createElement('video').canPlayType('video/webm; codecs="vp8, vorbis"') == ''

However, Vorbis+VP8 WebM files seem to play in a <video> just fine.

[Jean-Yves Avenard [:jya]]

It didn't make it for STP 132 (https://webkit.org/blog/11971/release-notes-for-safari-technology-preview-132/) should be in STP 133

[Jean-Yves Avenard [:jya]]

(In reply to Ashley Gullen from comment #12)
> Sorry, I'm confused - the actual support for playback of the codecs already
> works. The only thing not working is the feature detection, i.e. whether
> canPlayType() indicates support. Why would that still be marked an
> experimental feature or not have arrived in Safari TP yet?

Because while the video and audio element do support webm; web audio however didn't and broke major sites as web audio doesn't have its own canPlayType() method.

So the decision was made to make canPlayType returns false if vorbis or opus codec was specified so that sites wouldn't attempt to use web audio with this content while allowing playback to still works.

It's an unfortunate set of circumstances, and as with all workarounds bring other set of issues. We've worked hard to fix as early as we could and with STP 133, web audio works with webm content and the workaround is removed.

[Brooke Vibber]

Will this ship in a point release to Safari in the near future, or should I plan to have a workaround that patches HTMLMediaElement.prototype.canPlayType so that regular video playback libraries start working again? Thanks for your time.

[Brooke Vibber]

(In the worst case, for now I can rewrite my input data to remove the codec string for VP8+Vorbis files, since canPlayType('video/webm') returns 'probably' and that's the assumed-default codec selection for WebM.)

[Ashley Gullen]

In Safari 15.2 the WebM tests now match canPlayType() (indicating it is not supported, and actually trying it also fails), but the PCM WAV support is still inconsistent: canPlayType() indicates it is not supported, but it does work when it is actually used. My original issue did mention this. Should I file a new issue?

[Ashley Gullen]

Actually it looks like it still reproduces in macOS. It's consistent on an iPad but in Safari 15.2 on macOS it can play the WebM formats but canPlayType() reports it cannot via an empty string. So this does not appear to be fixed.

[Jean-Yves Avenard [:jya]]

(In reply to Ashley Gullen from comment #18)
> In Safari 15.2 the WebM tests now match canPlayType() (indicating it is not
> supported, and actually trying it also fails), but the PCM WAV support is
> still inconsistent: canPlayType() indicates it is not supported, but it does
> work when it is actually used. My original issue did mention this. Should I
> file a new issue?

Please file a new bug for PCM support

For webm, which macOS version are you using?

[Ashley Gullen]

I used macOS 12.1 on a M1 Macbook.

I filed bug 234840 for PCM WAV.

[Derk-Jan Hartman]

Yes Safari 15.2 is STILL broken, but WTP is fixed. I guess knowing Apple this means that this fix won't arrive before spring now. Very much a shame that the feature test for something as important as VP9 is broken for over a YEAR after the feature got released.

In the mean time.. if you use feature detection to load a player, you can consider testing for: video/webm: codecs="vp9" AND audio/webm: codecs="opus". This works around the bug.

[Derk-Jan Hartman]

BTW, eventually I settled on MediaSource.isTypeSupported( 'video/webm; codecs="vp9, opus"' ) as the workaround and patched that into the canPlayType function of videojs.

So somewhere next week if all things go right, Wikipedia will finally be able to use native VP9 decoding on Desktop Safari for Macs from 2018 and later. (rest will still use wasm software decoder)

[Shane Sargent]

Yes Safari 15.2 is STILL broken, but WTP is fixed. I guess knowing Apple this means that this fix won't arrive before spring now. Very much a shame that the feature test for something as important as VP9 is broken for over a YEAR after the feature got released.

In the mean time.. if you use feature detection to load a player, you can consider testing for: video/webm: codecs="vp9" AND audio/webm: codecs="opus". This works around the bug. https://leveldevildash.com/

The issue you've mentioned regarding Safari TP 131's audio canPlayType() function reporting false negatives seems to be a bug or a limitation in the Technology Preview version of Safari. It's important to note that Technology Previews are beta versions of Safari meant for developers to test upcoming features and updates. https://jfif2.com/