Web audio without audio output should not require any user gesture on iOS

[Adam]

We use the following code to analyse the audio input level in a local MediaStream coming from getUserMedia. This works fine in Safari on a Mac but on iOS you get constant 0 (because timeDomainData[idx] always returns 128).

navigator.mediaDevices.getUserMedia({audio:true}).then(stream => {
  const context = new (window.AudioContext || window.webkitAudioContext)();
  const sourceNode = context.createMediaStreamSource(stream);
  const analyser = context.createAnalyser();
  sourceNode.connect(analyser);
  const timeDomainData = new Uint8Array(analyser.frequencyBinCount);

  setInterval(() => {
    analyser.getByteTimeDomainData(timeDomainData);
    
    let max = 0;
    for (let idx = 0; idx < timeDomainData.length; idx++) {
      max = Math.max(max, Math.abs(timeDomainData[idx] - 128));
    }
    audioLevel.innerHTML = (max / 128);
  }, 100);
}).catch(err => {
   alert(err.name + ' ' + err.message);
});


You can see it working at https://output.jsbin.com/rexolan

[Adam]

It also does not work for remote streams.

[youenn fablet]

(In reply to Adam from comment #1)
> It also does not work for remote streams.

On iOS, AudioContext needs a user gesture.
Can you retry by starting AudioContext as part of user gesture?
We should probably remove that restriction when getUserMedia is on.

[Andrew Morris]

> On iOS, AudioContext needs a user gesture.

Is it intended that AudioContext needs a user gesture even when there is no audio output? Being able to visualize the audio volume would be especially useful in exactly this use case when audio output is blocked.

[youenn fablet]

(In reply to Andrew Morris from comment #3)
> > On iOS, AudioContext needs a user gesture.
> 
> Is it intended that AudioContext needs a user gesture even when there is no
> audio output? Being able to visualize the audio volume would be especially
> useful in exactly this use case when audio output is blocked.

Good point, maybe the restriction should be targeted at the audio output.

[Radar WebKit Bug Importer]

<rdar://problem/35938085>

[Radar WebKit Bug Importer]

<rdar://problem/35938084>

[Adam]

Yes, it does work if we create the audio context when you click a button.
https://output.jsbin.com/juzufum

Like Andrew said though it would be great if that wasn't the case.

[youenn fablet]

bug 180680 is fixing the case of a page capturing data.
Let's keep this bug open for the wider question of allowing web audio analysis without user gesture

[Adam]

Fantastic, thanks

[Dag-Inge Aas]

We hit this bug as well, where we want to do an automated microphone check for users on their way into a conversation. Our code looks something like this:

function handleUserClick() {
    return mediaDevicesService
      .getUserMedia(constraints)
      .then(mediaStream => {
        this.setState({ mediaStream });
      })
      .then(() => enumerateDevices())
      .then(() => verifyMicrophoneWorks(this.state.mediaStream.stream))
      .then(isMicrophoneWorking =>
        this.props.updateMicStatus(isMicrophoneWorking)
      )
      .catch(error => <GetUserMediaErrorFeedback error={error} />);
}

This all happens in a single promise-chain, but because only the original button click is triggered as a user action, the audio context checking fails, even if all we do is check the getByteFrequencyData, and no audio is actually playing.

It would be great if we were allowed to play audio/handle AudioContext once getUserMedia permission is granted for the page for that session.

[youenn fablet]

daginge, I believe that would be addressed in bug 180680.
Can you try the latest iOS beta?

[Dag-Inge Aas]

Sorry for the late reply youenn. We decided to refactor as more browsers are adopting autoplay restrictions now, and we don't want to take the chance that this causes issues in the future. All of our AudioContext's are triggered by a user action now.

[youenn fablet]

(In reply to daginge from comment #12)
> Sorry for the late reply youenn. We decided to refactor as more browsers are
> adopting autoplay restrictions now, and we don't want to take the chance
> that this causes issues in the future. All of our AudioContext's are
> triggered by a user action now.

No problem daginge.
I believe we could tackle that issue by introducing some WebAudio specific constructs that would allow analyzing audio but not producing any audio.
In that case, we could bypass autoplay restrictions.

[youenn fablet]

Maybe OfflineAudioContext is what we want.

[youenn fablet]

(In reply to youenn fablet from comment #14)
> Maybe OfflineAudioContext is what we want.

Discussed with Jer and this is not designed for this use case.
Filed https://github.com/WebAudio/web-audio-api/issues/1551