Bug 221553 – Audio is sometimes distorted when using createMediaElementSource

• Status: RESOLVED
• Resolution: FIXED
• Component: Web Audio

[Luigi Pulcini]

Hello,

I have just confirmed this is still a problem in the new Safari Technology Preview 119, released on January 27, 2021.

This issue has emerged more than a year ago and, every time someone reports it (bug #211394, bug #215314), the report gets listed as RESOLVED FIXED very hastily and without verifying whether the fix is really effective or not. Whatever was the patch believed to have fixed this, it has not.

The problem appears to be affecting the audio more heavily with audio files sampled at 48kHz, while it seems to be less relevant when the audio file is sampled at 44.1kHz. In any case, this seems to prove there is still an issue with the audio resampling. In the previous bug report (bug #215314), we possibly detected the source of the issue being within the test case http://trac.webkit.org/changeset/267014/webkit

Particularly this:
> When the context's sample rate differs from the hardware sample rate, instantiate a MultiChannelResampler and use it in render() to do the resampling.


How to reproduce the issue (even in Safari Technology Preview 119) with your MacOS/OSX computer:
– go to https://www.waveplayer.info/createmediaelementsource-test/
– play any track (you will notice the waveform animating, which proves the WebAudio API is working)
– open the "Audio MIDI Setup" app (under /Applications/Utilities)
– while the track is still playing, change the sample rate of the audio device
– you should hear the audio distorting more or less heavily depending on which sample rate you selected for your device

You can watch a video demonstrating the issue here:
https://www.youtube.com/watch?v=KYtb-FnjbpQ&t=62
(what for it at 1:12!!!)

The first part of the same video shows that none of this is an issue in Chrome that can perfectly resample the audio, no matter what sample rate the audio device or the audio files are set to.

Thanks,
Luigi.

[Chris Dumez]

The steps in the video did not reproduce an issue on my machine. I used the source at 44.1khz and used to Audio MIDI Setup tool to switch from 44.1KHz to 48Khz and all the way up to 96Khz. The audio seems alright.

http://trac.webkit.org/changeset/267014/webkit is indeed a patch where we added support for resampling in WebKit. This patch is recent. It got committed after we started getting bug reports about this issue so I doubt this is the source of the problem. Also, in the test case of the video, WebKit does not do any resampling to my knowledge because the source is 44.1 Khz is the AudioContext also uses 44.1Khz. I understand that the hardware's sample rate gets changed after that via the Audio MIDI tool. However, this does not impact the sample rate of the AudioContext AFAIK. The rate of the AudioContext is decided upon construction (it is either the sample rate being provided upon construction or the current hardware sample rate at the time of construction). Certainly some resampling must take place when we change the hardware sample rate with the Audio MIDI tool but I suspect this is happening in the Audio framework under us (I will double check).

It is also interesting that it is working in Chrome because we're using the same implementation of MultiChannelResampler as Chrome nowadays AFAIK. This seems to indicate that the bug is likely outside the MultiChannelResampler.

I will do more testing (maybe with bluetooth headphones since it seems they were used in the test video).

[Luigi Pulcini]

Hi, Chris,

Thank you so much for checking this so promptly.

Interestingly enough, after receiving your comment, I tested that player again and now the problem seems to occur right when the player starts, whatever is the audio file and the initial sample rate. Then it is likely to disappear (but not always) when switching the sample rate of the device.

So, a new test you could do is this:

1) start the playback on any file
2) change the device sample rate
3) switch to any other file

If you repeat steps 2 and 3 repeatedly, you might notice that audio distortion starts when doing 3) and stops (most of the time) when doing 2).
At least, that is my latest result on STP 119.

By the way, the video shows a test with the built-in audio device of my MacBookPro (OSX 10.15 Catalina). No Bluetooth headphones were involved but I did try with both the internal speaker and wired headphones connected to the audio jack of my MBP.

I hope this helps.

Best,
Luigi.

[Chris Dumez]

Ok. it is not 100% reproducible but I can indeed fairly frequently reproduce a recurring crackling sounds when I play the source file at 44.1Khz and my hardware rate is 48Khz. This does seem to indicate a potential issue with the WebKit resampler. I am investigating.

[Chris Dumez]

The issue does not reproduce in Chrome. Interestingly, I have just tried copy/pasting Chrome's MultiChannelResampler & SincResampler and the issue still occurs in WebKit. The issue must be outside the resampler somehow.

[Luigi Pulcini]

Thanks for further investigating this issue.

I wish I could be more helpful but I hope the information I provided you with can help you track it down.

[Chris Dumez]

Looks like the issue may be inside AudioSourceProviderAVFObjC::provideInput().

[Chris Dumez]

Created attachment 419907 [details]
WIP Patch

[Chris Dumez]

Created attachment 419908 [details]
WIP Patch

[Chris Dumez]

Created attachment 419910 [details]
WIP Patch

[Chris Dumez]

Created attachment 419914 [details]
Patch

[EWS]

Committed r272730: <https://commits.webkit.org/r272730>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 419914 [details].

[Radar WebKit Bug Importer]

<rdar://problem/74235096>