Bug 215314 – Distorted audio when using createMediaElementSource

• Status: RESOLVED
• Resolution: FIXED
• Component: Web Audio

[Steve]

This bug report is a follow up to https://bugs.webkit.org/show_bug.cgi?id=211394 and radar id 48560346 (submitted march 2019 - no response yet) 

Summary:
When an audio element is connected to a web audio graph, this leads to audible noise (crackles) on iOS devices (independent of the browser being used - tested on safari and chrome). The issue does not show up on android devices or on macOS.

Steps to Reproduce:
visit https://jsfiddle.net/st___h/36w7vxn2/15/show and press "click to start". audio should play. In case audio plays without noise, reload page and try again. Issue usually shows up within less then 10 tries.

Expected Results:
Audio plays without issues.

Actual Results:
Audio sometimes plays with crackles/stutter. Issue shows up both in safari and chrome browser. Issue is independent of the sampleRate reported by the audioContext.
A recording from the iPads headphone output, illustrating the type of noise can be found here: https://st-h.github.io/test/ios_playback_noise.mp3

Version/Build:
tested with:
iPhone 6s iOS 12.1.4 up to 13.6
iPad Pro 2017 iOS 12.1.4 up to 13.6
using safari and chrome browser.

Configuration:
source code of test page:
https://jsfiddle.net/st___h/36w7vxn2/15

relevant code snippet:

    var context = new(window.AudioContext || window.webkitAudioContext)();
    var gainNode = context.createGain();

    var audio = new Audio();
    audio.loop = true;
    audio.crossOrigin = "anonymous";

    audio.addEventListener('error', function(e) {
      alert(e.toString());
    });
    var source = context.createMediaElementSource(audio);
    source.connect(gainNode);
    gainNode.connect(context.destination);

    audio.src = "https://st-h.github.io/test/latencies.mp3";
    audio.play();
    gainNode.gain.setValueAtTime(0.9, gainNode.context.currentTime);

[Radar WebKit Bug Importer]

<rdar://problem/66869052>

[Chris Dumez]

I think this may have been fixed very recently as part of our efforts to improve our WebAudio implementation. I can reproduce on shipping but not on the very latest WebKit trunk build.

[Luigi Pulcini]

As I mentioned in https://bugs.webkit.org/show_bug.cgi?id=211394, this behavior in connection to the use of the CreateMediaElementSource method seems to be related to a resampling problem when there is a mismatch between the sample rate of the audio file being played and the one of the user's device (which is in no way controllable by the developer).

The following video clearly demonstrates the different behavior between Chrome 86 (86.0.4240.198) and Safari 14.0 (15610.1.28.1.9, 15610) under MacOS 10.15.7.
https://youtu.be/KYtb-FnjbpQ

The test page shown in the video can be found at the following address: https://www.waveplayer.info/createmediaelementsource-test/

Regardless of the sample rate and the bitrate of the audio file being played back, my tests seem to conclude that the audio distortion is less likely to occur while downsampling rather than upsampling. So, if the audio context is created with the device set to 96 kHz, then switching the device sample rate to a smaller value doesn't affect the audio. By contrast, setting the sample rate of the device to a smaller value (e.g. 44.1 kHz) and then switching to a greater value is more likely to introduce the resampling distortion.

None of these behaviors occur in any other browser or device.
It is just Safari and iOS.

I hope this helps to clarify.

[Chris Dumez]

(In reply to Luigi Pulcini from comment #3)
> As I mentioned in https://bugs.webkit.org/show_bug.cgi?id=211394, this
> behavior in connection to the use of the CreateMediaElementSource method
> seems to be related to a resampling problem when there is a mismatch between
> the sample rate of the audio file being played and the one of the user's
> device (which is in no way controllable by the developer).
> 
> The following video clearly demonstrates the different behavior between
> Chrome 86 (86.0.4240.198) and Safari 14.0 (15610.1.28.1.9, 15610) under
> MacOS 10.15.7.
> https://youtu.be/KYtb-FnjbpQ

It is honestly hard for me to tell what's wrong when watching this video. The audio does not sound obviously different / wrong to me.

> 
> The test page shown in the video can be found at the following address:
> https://www.waveplayer.info/createmediaelementsource-test/
> 
> Regardless of the sample rate and the bitrate of the audio file being played
> back, my tests seem to conclude that the audio distortion is less likely to
> occur while downsampling rather than upsampling. So, if the audio context is
> created with the device set to 96 kHz, then switching the device sample rate
> to a smaller value doesn't affect the audio. By contrast, setting the sample
> rate of the device to a smaller value (e.g. 44.1 kHz) and then switching to
> a greater value is more likely to introduce the resampling distortion.
> 
> None of these behaviors occur in any other browser or device.
> It is just Safari and iOS.
> 
> I hope this helps to clarify.

We did fix major resampling issues super recently via Bug 219201 so hopefully this fixes this too. The next STP build will hopefully get the resampling fix.

[Luigi Pulcini]

Hi, Chris,

Thanks for your prompt reply.

> It is honestly hard for me to tell what's wrong when watching this video. The audio does not sound obviously different / wrong to me.

The video is 2-minute long and compares the good behavior in Chrome to the wrong behavior in Safari.

The first minute shows how everything is perfect in Chrome, even when switching the device sample rate during the playback (so, after the creation of the audio context).

But you probably have not reached 1:12 (https://youtu.be/KYtb-FnjbpQ?t=72) when, in Safari, I jump from 44.1 kHz to 48 kHz. Starting from there, any sort of very audible audio glitches are literally devastating the audio. At 48 kHz they are impossible to go unnoticed. At 88.2 and 96, they simply destroy the audio completely.

> We did fix major resampling issues super recently via Bug 219201 so hopefully this fixes this too. The next STP build will hopefully get the resampling fix.

I am very anxious to see those results. By the way, in the example I showed in the video, I am still using the prefixed webkitAudioContext in Safari.

[Chris Dumez]

(In reply to Luigi Pulcini from comment #5)
> Hi, Chris,
> 
> Thanks for your prompt reply.
> 
> > It is honestly hard for me to tell what's wrong when watching this video. The audio does not sound obviously different / wrong to me.
> 
> The video is 2-minute long and compares the good behavior in Chrome to the
> wrong behavior in Safari.
> 
> The first minute shows how everything is perfect in Chrome, even when
> switching the device sample rate during the playback (so, after the creation
> of the audio context).
> 
> But you probably have not reached 1:12 (https://youtu.be/KYtb-FnjbpQ?t=72)
> when, in Safari, I jump from 44.1 kHz to 48 kHz. Starting from there, any
> sort of very audible audio glitches are literally devastating the audio. At
> 48 kHz they are impossible to go unnoticed. At 88.2 and 96, they simply
> destroy the audio completely.

Ahah indeed, I only watched the Chrome part. I agree that the sound is indeed terrible in the Safari part.
Could you let me know which tool you were using to change the sample rate? I'd like to test it locally on a WebKit trunk build to see if it is the same resampling bug I fixed recently.

[Luigi Pulcini]

In MacOS, there is a core App called "Audio MIDI Setup". It is in the UTILITIES subfolder of the Applications (or you can Spotlight it).

Once you open it, it usually shows the Audio Devices window. If that doesn't pop up automatically, you can always opening it from the Window menu (Show Audio Devices) or using the CMD+1 shortcut.

The Audio Devices window allows you to select the system sound card and all its settings.

I hope this helps.

[Chris Dumez]

(In reply to Luigi Pulcini from comment #7)
> In MacOS, there is a core App called "Audio MIDI Setup". It is in the
> UTILITIES subfolder of the Applications (or you can Spotlight it).
> 
> Once you open it, it usually shows the Audio Devices window. If that doesn't
> pop up automatically, you can always opening it from the Window menu (Show
> Audio Devices) or using the CMD+1 shortcut.
> 
> The Audio Devices window allows you to select the system sound card and all
> its settings.
> 
> I hope this helps.

Have you tried the latest Safari Technology Preview? The issue does not reproduce for me with the latest Safari Technology Preview so hopefully this has been fixed already.

[Chris Dumez]

Can reproduce in shipping Safari but not in latest Safari Technology Preview. Seems to have been fixed. Thanks for the test cases.

[Chris Dumez]

(In reply to Chris Dumez from comment #8)
> (In reply to Luigi Pulcini from comment #7)
> > In MacOS, there is a core App called "Audio MIDI Setup". It is in the
> > UTILITIES subfolder of the Applications (or you can Spotlight it).
> > 
> > Once you open it, it usually shows the Audio Devices window. If that doesn't
> > pop up automatically, you can always opening it from the Window menu (Show
> > Audio Devices) or using the CMD+1 shortcut.
> > 
> > The Audio Devices window allows you to select the system sound card and all
> > its settings.
> > 
> > I hope this helps.
> 
> Have you tried the latest Safari Technology Preview? The issue does not
> reproduce for me with the latest Safari Technology Preview so hopefully this
> has been fixed already.

-> https://developer.apple.com/safari/download/

[Luigi Pulcini]

Hi, Chris,

The Safari Technology Preview is much better but I can still reproduce in some cases. The distortion is not as heavy as before, which confirms that the patch you applied definitely solves most of the resampling problems.

The scenario where I can reproduce this issue most often is the following one.

a. set the sample rate of the built-in device to 48 kHz, 32bit float
b. go to https://www.waveplayer.info/createmediaelementsource-test/
c. click on a track encoded at 44.1 kHz: the sound is glitchy (not as terrible as in shipping Safari, but still very noticeable)

I could not reproduce this issue 100% of the time, but I managed to get a glitchy sound with some combinations of device sample rate and audio file sample rate.

If I find a way to reproduce the issue in a consistent way, I will let you know.

In the meantime, thank you for reacting so promptly to my test cases.

[Chris Dumez]

(In reply to Luigi Pulcini from comment #11)
> Hi, Chris,
> 
> The Safari Technology Preview is much better but I can still reproduce in
> some cases. The distortion is not as heavy as before, which confirms that
> the patch you applied definitely solves most of the resampling problems.
> 
> The scenario where I can reproduce this issue most often is the following
> one.
> 
> a. set the sample rate of the built-in device to 48 kHz, 32bit float
> b. go to https://www.waveplayer.info/createmediaelementsource-test/
> c. click on a track encoded at 44.1 kHz: the sound is glitchy (not as
> terrible as in shipping Safari, but still very noticeable)
> 
> I could not reproduce this issue 100% of the time, but I managed to get a
> glitchy sound with some combinations of device sample rate and audio file
> sample rate.
> 
> If I find a way to reproduce the issue in a consistent way, I will let you
> know.
> 
> In the meantime, thank you for reacting so promptly to my test cases.

I bisected this and this is the change that greatly progressed this test case: http://trac.webkit.org/changeset/267014/webkit

However, I have also confirmed that the audio is still bad is *some* cases (I am able to reproduce when choosing the 44.1 Khz track and a very large sample rate in Audio MIDI Setup).

[Luigi Pulcini]

Hi, Chris,

Thanks for confirming this. Yes, the audio is still problematic in some cases, even though this is moving in the right direction. As you pointed out, this happens more frequently when the device sample rate is set at higher values (88.2 and 96 kHz). In addition to that, I sporadically get glitchy audio also when the device is initialized at the lower ones.

According to the Changeset you shared with us, there might be some issues with the MultiChannelResampler.

> When the context's sample rate differs from the hardware sample rate, instantiate a MultiChannelResampler and use it in render() to do the resampling.

[Luigi Pulcini]

Hello,

I would like to remind everyone involved in this thread that the new Safari Technology Preview, 119, released on January 27, 2021 DOES NOT include a fix to this problem.

The audio is still distorted depending on the sample rate of the device.

This issue has emerged more than a year ago and, every time someone reports it here, the report gets listed as RESOLVED FIXED very hastily and without verifying whether the fix is really effective or not.

As of today, this is STILL a problem that needs to be fixed.

Thanks,
Luigi.