Bug 227199 - Progressively worsening crackling / distortion under high load (Resonance Audio + Augmented Reality)
Summary: Progressively worsening crackling / distortion under high load (Resonance Aud...
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Web Audio (show other bugs)
Version: Safari 14
Hardware: iPhone / iPad iOS 14
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2021-06-20 15:40 PDT by Lucas Levin
Modified: 2021-10-04 14:07 PDT (History)
10 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Lucas Levin 2021-06-20 15:40:28 PDT
Reproduction url: https://lucaslcode.github.io/ios-web-audio-bug/

I am creating a web app that combines AFrame for augmented reality and Resonance Audio for spatial audio. Resonance uses the Web Audio api.

On iOS, the audio becomes progressively more crackly and distorted like a feedback loop, until it stops completely. I have tested this on an iPhone 6s running iOS 14.6 within about 10 seconds, and also occurs on an iPhone 12 mini on iOS 14.6 after a few minutes. 

The issue does not occur on a Macbook Air (Early 2014, safari 14.0.1, macOS 10.14.6), windows 10 laptop or xiaomi phone running Chrome.

I appreciate any insight - this is a blocker for my app on iOS. Thanks for your time!
Comment 1 Radar WebKit Bug Importer 2021-06-27 15:41:14 PDT
<rdar://problem/79844025>
Comment 2 Brian Peiris 2021-07-18 09:22:30 PDT
I'm confirming this issue in Safari 14.1.1 on both iOS 14.6 (iPhone 11) and macOS 11.4 (MacBook Air M1 2020).
 
The issue can be reproduced with a simpler WebAudio test and does not require high load or WebGL content. All you need is a spatial PannerNode that updates its position and orientation continuously. Here is a simplified demo: https://brianpeiris.github.io/spatial-audio-test/

The demo shows that the issue occurs even if you use ramped transform changes with an AudioParam (linearRampToValueAtTime) or with instantaneous transform changes (setPosition and setOrientation).

The issue reproduces faster with instantaneous changes, and it reproduces at different rates in different Safari browsers. I've found that it takes ~50 minutes of playback to reproduce in Safari Technology Preview (15.0) using ramped transform changes, but only a few minutes with instantaneous transform changes. Safari 14.1.1 reproduces in 15 minutes with ramped transforms, and a few minutes with instantaneous transforms. Safari 14.1.1 on iOS reproduces within a minute in either method.

The issue does not reproduce in Firefox or Chrome on the same MacBook.
Comment 3 Brian Peiris 2021-07-19 12:41:58 PDT
In addition to the progressively distorted audio, this bug also seems to cause CPU usage to increase continuously. The simple test page posted above goes from 20% CPU usage to 99% CPU usage within a matter of minutes.
Comment 4 Anton 2021-08-04 11:12:36 PDT
On mobile Apple devices (iPhone, iPad) we got cracklings when using a simple oscillator from the P5.js library. It started recently about 2 weeks ago. I think it is related to the issue described here. Same issue using mobile Safari/Chrome/Firefox, 7 years old or new apple devices.
Comment 5 juretriglav 2021-09-20 09:13:16 PDT
We are seeing the same and receiving complaints from customers on desktop Safari or on iOS.

Since in our specific situation the PannerNodes positions aren't animated/constantly updating, we've implemented a band-aid where we compare old position vs new position and only call PannerNode.setPosition when necessary.

Still, this issue greatly diminishes the utility of spatial audio on WebKit browsers.
Comment 6 juretriglav 2021-09-20 13:31:06 PDT
Adding some more relevant links. 

Here is a very relevant bug that should be fixed already, regarding memory management of AudioBufferSourceNodes: https://bugs.webkit.org/show_bug.cgi?id=222098

Some more information about the bug, reproductions: https://github.com/mozilla/hubs/issues/4411
Comment 7 Chris Dumez 2021-09-20 13:41:18 PDT
(In reply to juretriglav from comment #6)
> Adding some more relevant links. 
> 
> Here is a very relevant bug that should be fixed already, regarding memory
> management of AudioBufferSourceNodes:
> https://bugs.webkit.org/show_bug.cgi?id=222098
> 
> Some more information about the bug, reproductions:
> https://github.com/mozilla/hubs/issues/4411

I have tried https://takahirox.github.io/ios-webaudio-test/ with WebAudio (with createBufferSource()) on iOS 15 (iPhone) and could not reproduce after playing for several minutes. No issue on macOS Monterey either.
Comment 8 juretriglav 2021-09-21 01:25:33 PDT
Thanks for trying to replicate it, Chris! Unfortunately takahirox's repro is not entirely related, though some audio degradation can occur there as well.

The most sure way to replicate this bug is using: https://brianpeiris.github.io/spatial-audio-test/?instant

This is minimal example that sets the position and orientation of the PannerNode on every frame.

I just installed iOS 15 and tested with the above URL and can reproduce severe audio degradation in under a minute. I've also tested with the latest Safari Technology Preview r132 and can reproduce the issue there as well, though it can take a few minutes. Can't test on Monterey yet, since it's only in preview.
Comment 9 juretriglav 2021-10-04 12:42:47 PDT
As a quick update, I've tried and am still able to reproduce this in 100% of cases in the latest Safari 15.1 beta, using https://brianpeiris.github.io/spatial-audio-test/?instant, letting it run for a few minutes.

Some observations:
1. Leaving it running for a few minutes results in severely degraded audio quality, with a fairly high CPU usage as well
2. Switching to a different tab results in clean audio (and lower CPU usage), presumably because the PannerNode gets taken out of the audio path?
3. Switching back to the offending tab results in severely degraded audio quality immediately, i.e. it does not reset to the initial quality
Comment 10 Chris Dumez 2021-10-04 12:44:18 PDT
If it is PannerNode-related it could be https://bugs.webkit.org/show_bug.cgi?id=230950 which was just fixed on trunk.
Comment 11 Takahiro Aoyagi 2021-10-04 13:48:29 PDT
> I have tried https://takahirox.github.io/ios-webaudio-test/ with WebAudio (with createBufferSource()) on iOS 15 (iPhone) and could not reproduce after playing for several minutes. No issue on macOS Monterey either.

To clarify, I noticed (at least) three audio problems.

1. WebAudio createMediaElementSource() audio noise problem. Audio which is made with WebAudio createMediaElementSource() can cause noise. https://takahirox.github.io/ios-webaudio-test/ is the test to check it. I faced this problem on my iPhoneX iOS 14.8 Safari, but the problem seems to have been resolved in iOS 15.

2. WebAudio PannerNode distorted audio problem. As Brian and others mentioned, PannerNode causes distorted audio after a few minutes. This problem doesn't seem to have been resolved yet even in iOS 15. You can reproduce the problem on a Three.js example https://threejs.org/examples/#webaudio_orientation Sounds like https://bugs.webkit.org/show_bug.cgi?id=230950 can resolve the problem. I will update my iOS and test again.

3. Audio of video media (and others?) is crackling if you get user media with echo cancellation option. It seems this problem is reproducible only on old or low-end models. I opened the issue https://bugs.webkit.org/show_bug.cgi?id=231191 please refer to it for the details.

And one more issue. Our member reported audio can cause noise on his iPhoneSE2 iOS 15 Safari. https://github.com/mozilla/hubs/issues/4411#issuecomment-933539778 We couldn't find the root issue yet but I have an impression that noise can happen especially if CPU usage is high (or other resources, eg. memory, is consumed a lot). I'm not really sure yet whether it is another problem or related to 2 or 3 above.
Comment 12 Chris Dumez 2021-10-04 14:07:26 PDT
(In reply to Brian Peiris from comment #2)
> I'm confirming this issue in Safari 14.1.1 on both iOS 14.6 (iPhone 11) and
> macOS 11.4 (MacBook Air M1 2020).
>  
> The issue can be reproduced with a simpler WebAudio test and does not
> require high load or WebGL content. All you need is a spatial PannerNode
> that updates its position and orientation continuously. Here is a simplified
> demo: https://brianpeiris.github.io/spatial-audio-test/
> 
> The demo shows that the issue occurs even if you use ramped transform
> changes with an AudioParam (linearRampToValueAtTime) or with instantaneous
> transform changes (setPosition and setOrientation).
> 
> The issue reproduces faster with instantaneous changes, and it reproduces at
> different rates in different Safari browsers. I've found that it takes ~50
> minutes of playback to reproduce in Safari Technology Preview (15.0) using
> ramped transform changes, but only a few minutes with instantaneous
> transform changes. Safari 14.1.1 reproduces in 15 minutes with ramped
> transforms, and a few minutes with instantaneous transforms. Safari 14.1.1
> on iOS reproduces within a minute in either method.
> 
> The issue does not reproduce in Firefox or Chrome on the same MacBook.

Thank you for the reduced test case. I am able to reproduce with this test case on shipping Safari 15. However, it no longer reproduces on WebKit trunk. It is almost certainly the same issue as Bug 230950, which I have fixed a few days ago.