Bug 191053 - Fix Safari macOS/iOS high speed audio/video algorithm
Summary: Fix Safari macOS/iOS high speed audio/video algorithm
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Media (show other bugs)
Version: Safari 12
Hardware: All All
: P1 Normal
Assignee: Jer Noble
URL:
Keywords: HTML5, InRadar
: 243371 251555 (view as bug list)
Depends on:
Blocks:
 
Reported: 2018-10-29 21:04 PDT by efusion3
Modified: 2023-05-22 21:17 PDT (History)
12 users (show)

See Also:


Attachments
Comparative examples of Chrome using Ilya Grigorik's video speed plugin vs. Safari's console solution (11.78 MB, application/zip)
2018-10-29 21:04 PDT, efusion3
no flags Details
Comparison of Safari 16 to various AVAudioTimePitchAlgorithms (1.43 MB, application/zip)
2023-02-04 16:41 PST, tellowkrinkle
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description efusion3 2018-10-29 21:04:43 PDT
Created attachment 353354 [details]
Comparative examples of Chrome using Ilya Grigorik's video speed plugin vs. Safari's console solution

High-speed HTML5 video playback on Safari (both macOS and iOS) suffer from audio modulation that does not exist on Chrome. See attached videos as examples, or reproduce using YouTube's native speed acceleration controls to compare both browsers.

If there was a way to use AVAudioTimePitchAlgorithm (https://developer.apple.com/documentation/avfoundation/avaudiotimepitchalgorithm), that might solve the problem, at least on native apps.
Comment 1 efusion3 2018-10-29 21:05:40 PDT
I was asked by Ilya to file the bug; hopefully this gets to the right people to fix, thanks!
Comment 2 Radar WebKit Bug Importer 2018-10-30 16:50:10 PDT
<rdar://problem/45685764>
Comment 3 Jer Noble 2018-10-31 08:40:11 PDT
We already changed the time pitch algorithm from AVAudioTimePitchAlgorithmTimeDomain to AVAudioTimePitchAlgorithmSpectral in bug #142280 / http://trac.webkit.org/changeset/181006. What you're asking for is for either Spectral to be better, or for a third even higher-quality option.
Comment 4 efusion3 2018-10-31 10:03:08 PDT
@Jer: any option other than what is currently implemented would be ideal; the high-speed audio (specifically speech) quality on macOS/iOS is terrible compared to Chrome. And that includes native media player apps.
Comment 5 Jer Noble 2018-10-31 10:16:01 PDT
And the TimeDomain is terrible for music playback. We can't switch back to TimeDomain for the same reason we switched away from it in the first place, and there is not a third option that's better than both.
Comment 6 efusion3 2018-10-31 10:26:50 PDT
I’d argue high-speed audio playback should primarily be optimized for spoken word, given its popularity with lecture/tutorials and also with the accessibility community, while music playback benefits from slow-speed playback for practicing purposes. Weigh each use case with data to make a determination on the best route.
Comment 7 Yuval Greenfield 2020-06-30 19:27:30 PDT
Moving from Android to iPhone, I noticed the YouTube native app's 2x playback speed sounds tinny on the iPhone. Then I noticed the same issue with Safari sounding worse than Chrome on my Mac.

I mainly use playback speed up for spoken content. You can compare this MKBHD video https://www.youtube.com/watch?v=Q2aaCDNjWEg where right in his first few words, on Safari it sounds like he's in a metal bathroom, while in Chrome he's just talking fast. I can hear the problem at 1.25x and 2x speed.

I don't know what Chrome is doing, but it sounds much better.
Comment 8 boettges 2020-07-22 12:51:04 PDT
Sadly, the inferior sound quality at higher playback speed is actually the sole remaining reason that keeps me from switching to WebKit based browsers.


Just some additional links that might be helpful:

The community around the open source media player mpv has looked into this as well: https://github.com/mpv-player/mpv/issues/4418 

Just recently someone seems to have successfully ported over the WSOLA algorithm that Chromium uses: https://github.com/mpv-player/mpv/pull/7865 

Maybe that is a point of reference to build upon to increase the quality in WebKit's playback engine.

Thanks to Ritam Sarmah, the developer of the Accelerate Safari Extension, for pointing me to this bug report. And thanks a lot for the continued work!
Comment 9 aherrera31 2021-08-27 09:35:22 PDT
This bug is the only thing keeping me from using Safari!!! Please fix it!!!! Fatal flaw!!! Civilization is better off when people can absorb content in half the time
Comment 10 Eric Carlson 2021-08-27 09:51:47 PDT
Apple's WebKit port uses CoreAudio for most audio rendering. We have made them aware of this issue.
Comment 11 saltukkos+webkit 2021-11-22 09:13:49 PST
Any updates on this issue? I've migrated to Mac, but God, I just can't use Safari for watching youtube. Firstly I thought it was a bug, but then after long investigation I found this issue. It's hard to believe, that in such big project issue like that can be opened for more than 3 years.
Comment 12 tellowkrinkle 2022-02-09 03:26:59 PST
Reminder that not everyone agrees on what speedup algorithm sounds best

If this bug is "fixed", please leave a configuration option for the current algorithm
Comment 13 Jer Noble 2022-03-16 08:40:42 PDT
(In reply to tellowkrinkle from comment #12)
> Reminder that not everyone agrees on what speedup algorithm sounds best
> 
> If this bug is "fixed", please leave a configuration option for the current
> algorithm

Hello tellowrinkle. When you say "configuration option", are you asking for a Safari preference, a WKWebView configuration option, or a web API?
Comment 14 Sam Sneddon [:gsnedders] 2022-03-16 09:03:01 PDT
<rdar://22243364>
Comment 15 tellowkrinkle 2022-03-16 13:38:09 PDT
I think a Safari preference would make the most sense for this kind of thing.  If it were a web API, I'd probably end up having to use a userscript to enable it, as I have no faith that websites would add a control for it.
Comment 16 Sam Sneddon [:gsnedders] 2022-07-21 07:41:38 PDT
(In reply to Jer Noble from comment #5)
> And the TimeDomain is terrible for music playback. We can't switch back to
> TimeDomain for the same reason we switched away from it in the first place,
> and there is not a third option that's better than both.

That said, we did switch the default back to it in Bug 220341 (232815@main), and shipped that in Safari 14.1. So what is the outstanding issue?
Comment 17 boettges 2022-07-21 08:51:33 PDT
(In reply to Sam Sneddon [:gsnedders] from comment #16)
> That said, we did switch the default back to it in Bug 220341 (232815@main),
> and shipped that in Safari 14.1. So what is the outstanding issue?

The sound quality issue when using high-speed (anything higher than 1x, to be honest) playback as showcased initially in the comparison video attached to this bug report by efusion3 still has not improved. 

Searching for this on the web[1], countless users describe the high-speed playback's audio as "distorted", "tinny", "robotic", "high-pitched", "having an echo" or the audio quality generally is perceived as "poor" or "low" in comparison with other browsers. As a result, users state that videos become "unwatchable", spoken content "hard to follow" and "more tiring". 
Many state that this is the main reason to not use Safari on macOS. On iOS/iPadOS one sadly is left without choice. 
The question also is. Is this a WebKit or a CoreAudio problem that could only  Apple could fix? If so, wouldn't the feedback be better forwarded via the WebKit team instead of users filing feedback with Apple.

I mentioned in my prior comment that the community around the media player "mpv" also had issues with this. The user stevenexed posted audio filter settings for mpv that achieved a (for most) subjectively better sounding audio output at higher playback rates [2]. Maybe this could be a starting point to see how this can be improved in WebKit as well.

[1] https://www.google.com/search?client=firefox-b-d&q=safari+high-speed+playback+sounds+worse
[2] https://github.com/mpv-player/mpv/issues/4418#issuecomment-360597668
Comment 18 Sam Sneddon [:gsnedders] 2022-07-31 18:18:30 PDT
*** Bug 243371 has been marked as a duplicate of this bug. ***
Comment 19 saltukkos+webkit 2022-09-12 21:56:03 PDT
Should we change the Importance of this bug to P2 since it has the P2 level duplicate? I was sad to not see any improvements in it in the new version of Safari. Still have to open Edge every time I need to speed up the youtube video. Please fix the critical flaw in the basic stuff before implementing the new features.
Comment 20 meteor-folktale 2022-10-22 08:05:33 PDT
I didn't know there were other people like me that avoided using Safari because of this audio quality issue! I just stumbled across this thread and knew I had to create an account and add my voice in. Is there any way to prioritize this issue? 

I want to use a webkit browser again because it gives me great battery life but this audio speed quality issue is a nonstarter. Just make it sound normal pls
Comment 21 saltukkos+webkit 2022-10-22 09:12:59 PDT
(In reply to meteor-folktale from comment #20)

Same here. I almost gave up on this. Duckduckgo released their browser based on webkit, I was hoping they somehow fixed this bug, but it sounds same. Still using Microsoft Edge because of this.

@gsnedders @Jer can you please increase the priority of this?
Comment 22 meteor-folktale 2022-10-25 10:46:37 PDT
Is there any way to donate to support this issue?
Comment 23 Vladimir Prelovac 2022-11-09 10:38:29 PST
(In reply to meteor-folktale from comment #22)
> Is there any way to donate to support this issue?


We (team at Orion browser) would like to give a shot at this. Support for development is welcome.
Comment 24 saltukkos+webkit 2022-11-09 21:04:49 PST
(In reply to Vladimir Prelovac from comment #23)
> (In reply to meteor-folktale from comment #22)
> > Is there any way to donate to support this issue?
> 
> 
> We (team at Orion browser) would like to give a shot at this. Support for
> development is welcome.

Great news! Will you create pull request in webkit so we can expect same feature in safari or it will be available only in Orion browser? In case of first, I will be glad to support the development of this.
Comment 25 Vladimir Prelovac 2022-11-12 12:19:57 PST
(In reply to saltukkos+webkit from comment #24)

> > We (team at Orion browser) would like to give a shot at this. Support for
> > development is welcome.
> 
> Great news! Will you create pull request in webkit so we can expect same
> feature in safari or it will be available only in Orion browser? In case of
> first, I will be glad to support the development of this.

Yes we would merge any improvements back to WebKit.
Comment 26 Jer Noble 2023-02-01 15:22:05 PST
*** Bug 251555 has been marked as a duplicate of this bug. ***
Comment 27 Jer Noble 2023-02-01 15:41:16 PST
Pull request: https://github.com/WebKit/WebKit/pull/9489
Comment 28 saltukkos+webkit 2023-02-04 01:30:05 PST
(In reply to Jer Noble from comment #27)
> Pull request: https://github.com/WebKit/WebKit/pull/9489

Great news! Would you mind to help us to understand, when to expect this fix in stable version of Safari?
Comment 29 tellowkrinkle 2023-02-04 16:41:14 PST
Created attachment 464841 [details]
Comparison of Safari 16 to various AVAudioTimePitchAlgorithms

I'm a bit confused by that change

I went to go rerun the test in the original attachment, and compared it to a local test program playing back the video at 2.5x with both the Spectral and TimeDomain algorithms, and current Safari (16.1 18614.2.9.1.12) sounded identical to Spectral, even though that PR hasn't been merged.  (And neither sounds like the original attachment's Safari.)
Comment 30 efusion3 2023-02-06 10:00:02 PST
Wait, is the PR to change from TimeDomain to Spectral? The proposal is to use TimeDomain as it's supposedly better for speech.

Please check with what's implemented in audiobooks in Books.app, Podcast.app and Developer.app, especially the latter when you compare videos played on the mobile app vs. on web (Safari vs. Chrome). There might be a case where the exact value mapped to the speed multiples might have to be tweaked to prevent the modulation or comb filtering effect. I say this last bit because other audio playing apps do that in code as well.
Comment 31 Jer Noble 2023-05-22 15:04:53 PDT
(In reply to efusion3 from comment #30)
> Wait, is the PR to change from TimeDomain to Spectral? The proposal is to
> use TimeDomain as it's supposedly better for speech.

See the most recent version of the PR. TimeDomain is still the preferred algorithm, but various bugs and inconsistencies where Spectral is still being used have been cleared up.
Comment 32 Jer Noble 2023-05-22 15:08:03 PDT
rdar://103940613
Comment 33 EWS 2023-05-22 21:17:02 PDT
Committed 264402@main (f01494f3adc7): <https://commits.webkit.org/264402@main>

Reviewed commits have been landed. Closing PR #9489 and removing active labels.