Bug 208516 – Audio fails to capture stream in WebRTC if AudioSession gets interrupted

• Component: WebRTC
• Status: RESOLVED FIXED

[David Engelmaier]

Created attachment 392279 [details]
Multiple audio tracks listed

When an application that uses camera and/or microphone is used(ie called to) during the time a web app with webrtc functionality is open, the web app is never
able to capture audio from the stream.

Prerequisites:
1. iphone 8 with ios 13.3.1
2. skype installed on the device with an account logged in, also teams, call app or facetime have been reported to work, although i was using mostly skype

Steps to reproduce:
1. open https://webrtc.github.io/samples/src/content/peerconnection/pc1/ in safari
2. start, call, hangup, reload the tab
3. call the device on skype, reject the call
4. go back to the safari tab, start, call, video frozen, no sound, from now on you can not use webrtc audio in safari, not even newly open tabs
5. connect to the iphone from safari develoop, run:
navigator.mediaDevices.getUserMedia({audio: true, video: true}).then(stream => {stream.getTracks().forEach(track => {track.stop();console.log(track.getSettings().sampleRate);console.log(track)})}).catch(error => {console.log(error)});

you'll see the sampleRate being always 0, sometimes calling the above code along with document.location.reload() multiple times(see attached screenshot) can bring safari back to working
webrtc audio, however this is very sporadic and can not be used as a workaround

This very issue was first filed as a bug here on 2017-12-13 06:02:09 PST https://bugs.webkit.org/show_bug.cgi?id=180748 and marked as resolved, however still happening.

There is also an interesting listing of media tracks in the attachment as you can see at one point there are 2 audio and 2 video tracks for the same interfaces

[Radar WebKit Bug Importer]

<rdar://problem/60020467>

[Nándor Kostyó-Pajer]

this is exactly the same problem i'm facing with a webrtc implementation. i created a simple app, that does not even use webrtc at all, just invokes getUserMedia, and then looks for the audio sampleRate, which in this case is simply 0.

trouble is, that if i manage to break the functionality on a tab with the steps described by David, the audio track will also be broken on other tabs, requiring us to kill the browser to make it function again.

[youenn fablet]

I tried on iOS 13.4 and do not see that issue.
I did step 1 and 2.
Instead of 3, I did a FaceTime call.
I did step 4 and was able to capture and do regular WebRTC calls.

The listing of two video tracks/two audio tracks seems indeed weird.

[youenn fablet]

Created attachment 394246 [details]
Patch

[Nándor Kostyó-Pajer]

we are constantly able to reproduce, and found some other strange symptoms: in case user media is requested, the appropriate icons appear in the address bar. when user media tracks are stopped, the camera icon is crossed, and stays there. in case of page reload, the icon still persists, and only disappears on the second pageload.

one workaround we found, is to navigate away from the page to a different subdomain, which will effectively remove the crossed camera icon. this reduced the possibility of breaking the audio in the tab to a minimum, otherwise, after the user media is requested, but no longer used by webrtc will easily break if another app takes over the microphone.

[youenn fablet]

(In reply to Nándor Kostyó-Pajer from comment #5)
> we are constantly able to reproduce, and found some other strange symptoms:
> in case user media is requested, the appropriate icons appear in the address
> bar. when user media tracks are stopped, the camera icon is crossed, and
> stays there. in case of page reload, the icon still persists, and only
> disappears on the second pageload.
> 
> one workaround we found, is to navigate away from the page to a different
> subdomain, which will effectively remove the crossed camera icon. this
> reduced the possibility of breaking the audio in the tab to a minimum,
> otherwise, after the user media is requested, but no longer used by webrtc
> will easily break if another app takes over the microphone.

Can you provide a fiddle?
Can you try it on the latest iOS 13.4 version?

I tried the following on iOS 13.4:
- Open https://webrtc.github.io/samples/src/content/peerconnection/pc1/ in iOS device
- Click Start and Call
- Open Web Inspector and execute localStream.getVideoTracks()[0].stop()
- The icon should switch from camera to microphone
- Execute localStream.getAudioTracks()[0].stop()
- The icon should disappear.

[Nándor Kostyó-Pajer]

i was doing reproductions on 13.3.1, now that i upgraded to the latest 13.4 beta, the persisting media icons cannot be reproduced.

will check the rest of the issue.

[youenn fablet]

Test is failing due to change in Page::setMuted.

[youenn fablet]

Created attachment 394358 [details]
Patch

[youenn fablet]

Created attachment 394361 [details]
Patch

[Eric Carlson]

Comment on attachment 394361 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=394361&action=review

> Source/WebCore/Modules/mediastream/MediaStreamTrack.cpp:471
> +static MediaStreamTrack* selectDocumentCaptureTrack(Document& document, RealtimeMediaSource* activeSource, RealtimeMediaSource::Type type)

Nit: this method doesn't select a track, so maybe something like `findActiveCaptureTrackForDocument` would be better?

[youenn fablet]

Created attachment 394481 [details]
Patch for landing

[youenn fablet]

(In reply to Eric Carlson from comment #11)
> Comment on attachment 394361 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=394361&action=review
> 
> > Source/WebCore/Modules/mediastream/MediaStreamTrack.cpp:471
> > +static MediaStreamTrack* selectDocumentCaptureTrack(Document& document, RealtimeMediaSource* activeSource, RealtimeMediaSource::Type type)
> 
> Nit: this method doesn't select a track, so maybe something like
> `findActiveCaptureTrackForDocument` would be better?

Thanks, done.

[EWS]

Committed r258977: <https://trac.webkit.org/changeset/258977>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 394481 [details].

[Ryan Haddad]

Follow up build fix in https://trac.webkit.org/changeset/258991

[Nándor Kostyó-Pajer]

would you please help me find information regarding which ios version will contain this patch?

i was still able to break webrtc functionality with a simple MS Teams call in 13.4.5 beta using the described way in the original post.

[youenn fablet]

This probably broke unmuting of video capture in case of tab switching.
The issue is that video unmuting when unhidden or not, might either mute/unmute the AVVideoCaptureSource or the RealtimeVideoSource wrapping the source.

Follow-up patch is at https://bugs.webkit.org/show_bug.cgi?id=211509.

[youenn fablet]

A I understand it, the current workaround is the following:
- when page gets back to visible, validate that the audio capture track is not muted and not ended
- If it stays muted or is ended, call navigator.mediaDevices.getUserMedia({ audio: true}) to make a new capture audio track and use RTCRtpSender.replaceTrack to use the new audio track in lieu of the old one

[Buddhika Jayawardhana]

I've made the following observations regarding this issue on iOS 14

1. If the webRTC call is interrupted by a phone call, audio track will emit `mute` event and when the interruptions is over, `some times` the audio track emits `unmute` event.
2. If we try to recapture media once the interruption is detected, the webRTC call can use the microphone, but the phone call is unable to use the microphone or speaker.
3. If the user leaves the browser the videoTrack emits `mute` event and `unmute` event when the user comes back to the browser.