Bug 216349 – onaudioprocess does not get called on Safari when external microphone is connected

• Component: Web Audio
• Status: NEW

[Toshish]

We're using new webkitAudioContext() in Safari 13.1.2 on MacBook or iOS Safari on iPhone and iPad to create AudioContext instance, and we're creating ScriptProcessorNode and attaching it to the source create using audioContext.createMediaStreamSource(). The onaudioprocess callback gets called with the audio data, but when you switch to external microphone, Bluetooth headset with microphone or AirPods, onaudioprocess does not get called at all. 

If you also connect Analyser node to the same audio source create using audioContext.createMediaStreamSource(), analyser.getByteTimeDomainData(dataArray) populates the data but onaudioprocess on the ScriptProcessorNode on the same source doesn't get called when connected to external microphone (anything other than internal MacBook Micorphone).

What has been tried:
* We confirmed that the stream being used is the only stream in the tab and getUserMedia() was only called once to get the stream. 
* We also tried using explicit deviceId of external microphone device as well as internal microphone while external microphone is connected. In both cases, onaudioprocess does not get called.

Only scenario where it works with external microphone is - if you start the stream while no external microphone is connected, and while the stream is active if you switch to external microphone by connecting physically - the audio from external microphone is used and onaudioprocess keeps continue to get called.


Code sample used to create the ScriptProcessorNode:

const AudioContext = window.AudioContext || window.webkitAudioContext;

audioContext = new AudioContext();
const audioSource = audioContext.createMediaStreamSource(stream);

  const processor = audioContext.createScriptProcessor(2048, 1, 1);
  processor.connect(audioContext.destination);
  processor.onaudioprocess = (e) => {
    // Does not get called when connected to external microphone
    // Gets called when using internal MacBook microphone
    console.log('print audio buffer', e);
  }

  audioSource.connect(processor);


The exact same behavior is also observed on iOS Safari on iPhone and iPad.

[Radar WebKit Bug Importer]

<rdar://problem/68647033>

[Zarco Nontol]

I can confirm that this isn't working as expected. Looking forward to hear from Webkit.
On the other hand we are trying to find a workaround for this. Will keep you posted!

[Chris Dumez]

Do you have a test case to help reproduce this?

[Zarco Nontol]

Hello Chris! Thanks your answering us!

We will prepare a test case today to provide it ASAP.

Thanks again!

[Chris Dumez]

(In reply to Zarco Nontol from comment #4)
> Hello Chris! Thanks your answering us!
> 
> We will prepare a test case today to provide it ASAP.
> 
> Thanks again!

Thank you, this will be very helpful. I also encourage you to test Safari Technology Preview on macOS if you have not yet. A lot of WebAudio fixes have not shipped yet but a lot of them are already in Safari Technology Preview.

[Zarco Nontol]

Created attachment 409910 [details]
onaudioprocess issue when using airpods or something below 16khz sample rate.

This quick script ask you to use your audio input device and if it notices audio data it changes to "onaudioprocess is being triggered OK!"

It also says the audio that it's being recognized.

On the other hand, one thing we notice is that this is not working correctly for Airpods with 16khz sample rate (16000).

In Safari Technology Preview is working as expected, we are using the following versions:

- Release 113 (Safari 14.0.1, WebKit 15610.2.3.1)

Also, for  Regular Safari: Version 14.0 (15610.1.28.1.9, 15610) -> here is not working.

[Zarco Nontol]

Hello Chris, I have left the test case attached to the issue,

Thanks!

[Chris Dumez]

(In reply to Zarco Nontol from comment #7)
> Hello Chris, I have left the test case attached to the issue,
> 
> Thanks!

I have just tried your test case on macOS Big Sur and see:

"""
onaudioprocess is being triggered OK!

Is using: AirPods
"""

I think this means I am unable to reproduce the issue?

[Joaquin Kenta Heianna]

(In reply to Chris Dumez from comment #8)
> (In reply to Zarco Nontol from comment #7)
> > Hello Chris, I have left the test case attached to the issue,
> > 
> > Thanks!
> 
> I have just tried your test case on macOS Big Sur and see:
> 
> """
> onaudioprocess is being triggered OK!
> 
> Is using: AirPods
> """
> 
> I think this means I am unable to reproduce the issue?

Hello Chris! Joaquin from Zarco's team, jumping in. 

Yes, it seems in Big Sur the issue cannot be reproduced. I suppose Safari in Big Sur already has the fixes you've mentioned from Safari Technology Preview, in which it is working properly.

In our test case that failed, we ran it using macOS Catalina (10.15.7) with Safari 14.0 (15610.1.28.1.9, 15610) - using wireless headphones such as AirPods and also Galaxy Buds.
We also tested with iOS 14.0.1 using AirPods and it also failed. Could I ask you to test also in iOS? (Note that on iPhone devices the "Is using" label from our test case will always display as "iPhone Microphone") 

Thanks!

[Chris Dumez]

(In reply to Joaquin Kenta Heianna from comment #9)
> (In reply to Chris Dumez from comment #8)
> > (In reply to Zarco Nontol from comment #7)
> > > Hello Chris, I have left the test case attached to the issue,
> > > 
> > > Thanks!
> > 
> > I have just tried your test case on macOS Big Sur and see:
> > 
> > """
> > onaudioprocess is being triggered OK!
> > 
> > Is using: AirPods
> > """
> > 
> > I think this means I am unable to reproduce the issue?
> 
> Hello Chris! Joaquin from Zarco's team, jumping in. 
> 
> Yes, it seems in Big Sur the issue cannot be reproduced. I suppose Safari in
> Big Sur already has the fixes you've mentioned from Safari Technology
> Preview, in which it is working properly.
> 
> In our test case that failed, we ran it using macOS Catalina (10.15.7) with
> Safari 14.0 (15610.1.28.1.9, 15610) - using wireless headphones such as
> AirPods and also Galaxy Buds.
> We also tested with iOS 14.0.1 using AirPods and it also failed. Could I ask
> you to test also in iOS? (Note that on iPhone devices the "Is using" label
> from our test case will always display as "iPhone Microphone") 
> 
> Thanks!

I am unsure how to get the demo page to use my AirPods on iOS. It keeps says "iPhone Microphone" even though my AirPods are connected.

Depending on when the fix landed, it is a small chance that the fix made it into macOS Big Sur but not iOS 14 since the 2 releases are not in sync this year.

[Joaquin Kenta Heianna]

Created attachment 410110 [details]
onaudioprocess audio recording test

Hi Chris!

I've attached another demo page where we record audio (using a library which uses ScriptProcessorNode behind the scenes) and after you stop recording, you'll hear what has been recorded. 
If there is any issue with it, the audio player embedded will display "Error". Hope this makes it more clear than the previous demo.

On iOS / iPadOS it always says "iPhone Microphone" due to some limitations in Safari or iOS itself (I believe), even if you connect AirPods or wired headphones with microphone correctly. 
So to test that, make sure to connect the AirPods and then hit refresh to the demo, so it will use that device for the test.

On the other hand, thanks for clarifying that this fix will be fixed on big sur. Is there a chance that this fix will land on iOS when Big Sur gets released?

Thanks!