Bug 191053 – Fix Safari macOS/iOS high speed audio/video algorithm

• Component: Media
• Status: RESOLVED FIXED

Created attachment 353354 [details]
Comparative examples of Chrome using Ilya Grigorik's video speed plugin vs. Safari's console solution

High-speed HTML5 video playback on Safari (both macOS and iOS) suffer from audio modulation that does not exist on Chrome. See attached videos as examples, or reproduce using YouTube's native speed acceleration controls to compare both browsers.

If there was a way to use AVAudioTimePitchAlgorithm (https://developer.apple.com/documentation/avfoundation/avaudiotimepitchalgorithm), that might solve the problem, at least on native apps.

I was asked by Ilya to file the bug; hopefully this gets to the right people to fix, thanks!

[Radar WebKit Bug Importer]

<rdar://problem/45685764>

[Jer Noble]

We already changed the time pitch algorithm from AVAudioTimePitchAlgorithmTimeDomain to AVAudioTimePitchAlgorithmSpectral in bug #142280 / http://trac.webkit.org/changeset/181006. What you're asking for is for either Spectral to be better, or for a third even higher-quality option.

@Jer: any option other than what is currently implemented would be ideal; the high-speed audio (specifically speech) quality on macOS/iOS is terrible compared to Chrome. And that includes native media player apps.

[Jer Noble]

And the TimeDomain is terrible for music playback. We can't switch back to TimeDomain for the same reason we switched away from it in the first place, and there is not a third option that's better than both.

I’d argue high-speed audio playback should primarily be optimized for spoken word, given its popularity with lecture/tutorials and also with the accessibility community, while music playback benefits from slow-speed playback for practicing purposes. Weigh each use case with data to make a determination on the best route.

[Yuval Greenfield]

Moving from Android to iPhone, I noticed the YouTube native app's 2x playback speed sounds tinny on the iPhone. Then I noticed the same issue with Safari sounding worse than Chrome on my Mac.

I mainly use playback speed up for spoken content. You can compare this MKBHD video https://www.youtube.com/watch?v=Q2aaCDNjWEg where right in his first few words, on Safari it sounds like he's in a metal bathroom, while in Chrome he's just talking fast. I can hear the problem at 1.25x and 2x speed.

I don't know what Chrome is doing, but it sounds much better.

[boettges]

Sadly, the inferior sound quality at higher playback speed is actually the sole remaining reason that keeps me from switching to WebKit based browsers.


Just some additional links that might be helpful:

The community around the open source media player mpv has looked into this as well: https://github.com/mpv-player/mpv/issues/4418 

Just recently someone seems to have successfully ported over the WSOLA algorithm that Chromium uses: https://github.com/mpv-player/mpv/pull/7865 

Maybe that is a point of reference to build upon to increase the quality in WebKit's playback engine.

Thanks to Ritam Sarmah, the developer of the Accelerate Safari Extension, for pointing me to this bug report. And thanks a lot for the continued work!

This bug is the only thing keeping me from using Safari!!! Please fix it!!!! Fatal flaw!!! Civilization is better off when people can absorb content in half the time

[Eric Carlson]

Apple's WebKit port uses CoreAudio for most audio rendering. We have made them aware of this issue.

Any updates on this issue? I've migrated to Mac, but God, I just can't use Safari for watching youtube. Firstly I thought it was a bug, but then after long investigation I found this issue. It's hard to believe, that in such big project issue like that can be opened for more than 3 years.

Reminder that not everyone agrees on what speedup algorithm sounds best

If this bug is "fixed", please leave a configuration option for the current algorithm

[Jer Noble]

(In reply to tellowkrinkle from comment #12)
> Reminder that not everyone agrees on what speedup algorithm sounds best
> 
> If this bug is "fixed", please leave a configuration option for the current
> algorithm

Hello tellowrinkle. When you say "configuration option", are you asking for a Safari preference, a WKWebView configuration option, or a web API?

[Sam Sneddon [:gsnedders]]

<rdar://22243364>

I think a Safari preference would make the most sense for this kind of thing.  If it were a web API, I'd probably end up having to use a userscript to enable it, as I have no faith that websites would add a control for it.

[Sam Sneddon [:gsnedders]]

(In reply to Jer Noble from comment #5)
> And the TimeDomain is terrible for music playback. We can't switch back to
> TimeDomain for the same reason we switched away from it in the first place,
> and there is not a third option that's better than both.

That said, we did switch the default back to it in Bug 220341 (232815@main), and shipped that in Safari 14.1. So what is the outstanding issue?

[boettges]

(In reply to Sam Sneddon [:gsnedders] from comment #16)
> That said, we did switch the default back to it in Bug 220341 (232815@main),
> and shipped that in Safari 14.1. So what is the outstanding issue?

The sound quality issue when using high-speed (anything higher than 1x, to be honest) playback as showcased initially in the comparison video attached to this bug report by efusion3 still has not improved. 

Searching for this on the web[1], countless users describe the high-speed playback's audio as "distorted", "tinny", "robotic", "high-pitched", "having an echo" or the audio quality generally is perceived as "poor" or "low" in comparison with other browsers. As a result, users state that videos become "unwatchable", spoken content "hard to follow" and "more tiring". 
Many state that this is the main reason to not use Safari on macOS. On iOS/iPadOS one sadly is left without choice. 
The question also is. Is this a WebKit or a CoreAudio problem that could only  Apple could fix? If so, wouldn't the feedback be better forwarded via the WebKit team instead of users filing feedback with Apple.

I mentioned in my prior comment that the community around the media player "mpv" also had issues with this. The user stevenexed posted audio filter settings for mpv that achieved a (for most) subjectively better sounding audio output at higher playback rates [2]. Maybe this could be a starting point to see how this can be improved in WebKit as well.

[1] https://www.google.com/search?client=firefox-b-d&q=safari+high-speed+playback+sounds+worse
[2] https://github.com/mpv-player/mpv/issues/4418#issuecomment-360597668

[Sam Sneddon [:gsnedders]]

*** Bug 243371 has been marked as a duplicate of this bug. ***

Should we change the Importance of this bug to P2 since it has the P2 level duplicate? I was sad to not see any improvements in it in the new version of Safari. Still have to open Edge every time I need to speed up the youtube video. Please fix the critical flaw in the basic stuff before implementing the new features.

[meteor-folktale]

I didn't know there were other people like me that avoided using Safari because of this audio quality issue! I just stumbled across this thread and knew I had to create an account and add my voice in. Is there any way to prioritize this issue? 

I want to use a webkit browser again because it gives me great battery life but this audio speed quality issue is a nonstarter. Just make it sound normal pls

(In reply to meteor-folktale from comment #20)

Same here. I almost gave up on this. Duckduckgo released their browser based on webkit, I was hoping they somehow fixed this bug, but it sounds same. Still using Microsoft Edge because of this.

@gsnedders @Jer can you please increase the priority of this?

[meteor-folktale]

Is there any way to donate to support this issue?

[Vladimir Prelovac]

(In reply to meteor-folktale from comment #22)
> Is there any way to donate to support this issue?


We (team at Orion browser) would like to give a shot at this. Support for development is welcome.

(In reply to Vladimir Prelovac from comment #23)
> (In reply to meteor-folktale from comment #22)
> > Is there any way to donate to support this issue?
> 
> 
> We (team at Orion browser) would like to give a shot at this. Support for
> development is welcome.

Great news! Will you create pull request in webkit so we can expect same feature in safari or it will be available only in Orion browser? In case of first, I will be glad to support the development of this.

[Vladimir Prelovac]

(In reply to saltukkos+webkit from comment #24)

> > We (team at Orion browser) would like to give a shot at this. Support for
> > development is welcome.
> 
> Great news! Will you create pull request in webkit so we can expect same
> feature in safari or it will be available only in Orion browser? In case of
> first, I will be glad to support the development of this.

Yes we would merge any improvements back to WebKit.

[Jer Noble]

*** Bug 251555 has been marked as a duplicate of this bug. ***

[Jer Noble]

Pull request: https://github.com/WebKit/WebKit/pull/9489

(In reply to Jer Noble from comment #27)
> Pull request: https://github.com/WebKit/WebKit/pull/9489

Great news! Would you mind to help us to understand, when to expect this fix in stable version of Safari?

Created attachment 464841 [details]
Comparison of Safari 16 to various AVAudioTimePitchAlgorithms

I'm a bit confused by that change

I went to go rerun the test in the original attachment, and compared it to a local test program playing back the video at 2.5x with both the Spectral and TimeDomain algorithms, and current Safari (16.1 18614.2.9.1.12) sounded identical to Spectral, even though that PR hasn't been merged.  (And neither sounds like the original attachment's Safari.)

Wait, is the PR to change from TimeDomain to Spectral? The proposal is to use TimeDomain as it's supposedly better for speech.

Please check with what's implemented in audiobooks in Books.app, Podcast.app and Developer.app, especially the latter when you compare videos played on the mobile app vs. on web (Safari vs. Chrome). There might be a case where the exact value mapped to the speed multiples might have to be tweaked to prevent the modulation or comb filtering effect. I say this last bit because other audio playing apps do that in code as well.

[Jer Noble]

(In reply to efusion3 from comment #30)
> Wait, is the PR to change from TimeDomain to Spectral? The proposal is to
> use TimeDomain as it's supposedly better for speech.

See the most recent version of the PR. TimeDomain is still the preferred algorithm, but various bugs and inconsistencies where Spectral is still being used have been cleared up.

[Jer Noble]

rdar://103940613

[EWS]

Committed 264402@main (f01494f3adc7): <https://commits.webkit.org/264402@main>

Reviewed commits have been landed. Closing PR #9489 and removing active labels.