Bug 208516

Summary: Audio fails to capture stream in WebRTC if AudioSession gets interrupted
Product: WebKit Reporter: David Engelmaier <david.engelmaier>
Component: WebRTCAssignee: youenn fablet <youennf>
Status: RESOLVED FIXED    
Severity: Major CC: buddhika.anushka, cdumez, eric.carlson, esprehn+autocc, ews-watchlist, glenn, hta, jer.noble, kangil.han, kostyo.nandor, philipj, sergio, tommyw, webkit-bug-importer, youennf
Priority: P2 Keywords: InRadar
Version: Safari 13   
Hardware: iPhone / iPad   
OS: iOS 13   
Bug Depends on: 210555    
Bug Blocks: 204681    
Attachments:
Description Flags
Multiple audio tracks listed
none
Patch
none
Patch
none
Patch
none
Patch for landing none

Description David Engelmaier 2020-03-03 08:53:12 PST
Created attachment 392279 [details]
Multiple audio tracks listed

When an application that uses camera and/or microphone is used(ie called to) during the time a web app with webrtc functionality is open, the web app is never
able to capture audio from the stream.

Prerequisites:
1. iphone 8 with ios 13.3.1
2. skype installed on the device with an account logged in, also teams, call app or facetime have been reported to work, although i was using mostly skype

Steps to reproduce:
1. open https://webrtc.github.io/samples/src/content/peerconnection/pc1/ in safari
2. start, call, hangup, reload the tab
3. call the device on skype, reject the call
4. go back to the safari tab, start, call, video frozen, no sound, from now on you can not use webrtc audio in safari, not even newly open tabs
5. connect to the iphone from safari develoop, run:
navigator.mediaDevices.getUserMedia({audio: true, video: true}).then(stream => {stream.getTracks().forEach(track => {track.stop();console.log(track.getSettings().sampleRate);console.log(track)})}).catch(error => {console.log(error)});

you'll see the sampleRate being always 0, sometimes calling the above code along with document.location.reload() multiple times(see attached screenshot) can bring safari back to working
webrtc audio, however this is very sporadic and can not be used as a workaround

This very issue was first filed as a bug here on 2017-12-13 06:02:09 PST https://bugs.webkit.org/show_bug.cgi?id=180748 and marked as resolved, however still happening.

There is also an interesting listing of media tracks in the attachment as you can see at one point there are 2 audio and 2 video tracks for the same interfaces
Comment 1 Radar WebKit Bug Importer 2020-03-03 17:13:33 PST
<rdar://problem/60020467>
Comment 2 Nándor Kostyó-Pajer 2020-03-03 23:44:56 PST
this is exactly the same problem i'm facing with a webrtc implementation. i created a simple app, that does not even use webrtc at all, just invokes getUserMedia, and then looks for the audio sampleRate, which in this case is simply 0.

trouble is, that if i manage to break the functionality on a tab with the steps described by David, the audio track will also be broken on other tabs, requiring us to kill the browser to make it function again.
Comment 3 youenn fablet 2020-03-05 02:05:38 PST
I tried on iOS 13.4 and do not see that issue.
I did step 1 and 2.
Instead of 3, I did a FaceTime call.
I did step 4 and was able to capture and do regular WebRTC calls.

The listing of two video tracks/two audio tracks seems indeed weird.
Comment 4 youenn fablet 2020-03-23 01:47:33 PDT
Created attachment 394246 [details]
Patch
Comment 5 Nándor Kostyó-Pajer 2020-03-23 02:04:36 PDT
we are constantly able to reproduce, and found some other strange symptoms: in case user media is requested, the appropriate icons appear in the address bar. when user media tracks are stopped, the camera icon is crossed, and stays there. in case of page reload, the icon still persists, and only disappears on the second pageload.

one workaround we found, is to navigate away from the page to a different subdomain, which will effectively remove the crossed camera icon. this reduced the possibility of breaking the audio in the tab to a minimum, otherwise, after the user media is requested, but no longer used by webrtc will easily break if another app takes over the microphone.
Comment 6 youenn fablet 2020-03-23 02:22:00 PDT
(In reply to Nándor Kostyó-Pajer from comment #5)
> we are constantly able to reproduce, and found some other strange symptoms:
> in case user media is requested, the appropriate icons appear in the address
> bar. when user media tracks are stopped, the camera icon is crossed, and
> stays there. in case of page reload, the icon still persists, and only
> disappears on the second pageload.
> 
> one workaround we found, is to navigate away from the page to a different
> subdomain, which will effectively remove the crossed camera icon. this
> reduced the possibility of breaking the audio in the tab to a minimum,
> otherwise, after the user media is requested, but no longer used by webrtc
> will easily break if another app takes over the microphone.

Can you provide a fiddle?
Can you try it on the latest iOS 13.4 version?

I tried the following on iOS 13.4:
- Open https://webrtc.github.io/samples/src/content/peerconnection/pc1/ in iOS device
- Click Start and Call
- Open Web Inspector and execute localStream.getVideoTracks()[0].stop()
- The icon should switch from camera to microphone
- Execute localStream.getAudioTracks()[0].stop()
- The icon should disappear.
Comment 7 Nándor Kostyó-Pajer 2020-03-23 03:57:40 PDT
i was doing reproductions on 13.3.1, now that i upgraded to the latest 13.4 beta, the persisting media icons cannot be reproduced.

will check the rest of the issue.
Comment 8 youenn fablet 2020-03-23 11:09:33 PDT
Test is failing due to change in Page::setMuted.
Comment 9 youenn fablet 2020-03-24 04:12:00 PDT
Created attachment 394358 [details]
Patch
Comment 10 youenn fablet 2020-03-24 05:43:37 PDT
Created attachment 394361 [details]
Patch
Comment 11 Eric Carlson 2020-03-24 09:35:47 PDT
Comment on attachment 394361 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=394361&action=review

> Source/WebCore/Modules/mediastream/MediaStreamTrack.cpp:471
> +static MediaStreamTrack* selectDocumentCaptureTrack(Document& document, RealtimeMediaSource* activeSource, RealtimeMediaSource::Type type)

Nit: this method doesn't select a track, so maybe something like `findActiveCaptureTrackForDocument` would be better?
Comment 12 youenn fablet 2020-03-25 03:46:51 PDT
Created attachment 394481 [details]
Patch for landing
Comment 13 youenn fablet 2020-03-25 06:33:55 PDT
(In reply to Eric Carlson from comment #11)
> Comment on attachment 394361 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=394361&action=review
> 
> > Source/WebCore/Modules/mediastream/MediaStreamTrack.cpp:471
> > +static MediaStreamTrack* selectDocumentCaptureTrack(Document& document, RealtimeMediaSource* activeSource, RealtimeMediaSource::Type type)
> 
> Nit: this method doesn't select a track, so maybe something like
> `findActiveCaptureTrackForDocument` would be better?

Thanks, done.
Comment 14 EWS 2020-03-25 06:44:23 PDT
Committed r258977: <https://trac.webkit.org/changeset/258977>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 394481 [details].
Comment 15 Ryan Haddad 2020-03-25 11:37:42 PDT
Follow up build fix in https://trac.webkit.org/changeset/258991
Comment 16 Nándor Kostyó-Pajer 2020-04-03 06:44:22 PDT
would you please help me find information regarding which ios version will contain this patch?

i was still able to break webrtc functionality with a simple MS Teams call in 13.4.5 beta using the described way in the original post.
Comment 17 youenn fablet 2020-05-06 06:24:58 PDT
This probably broke unmuting of video capture in case of tab switching.
The issue is that video unmuting when unhidden or not, might either mute/unmute the AVVideoCaptureSource or the RealtimeVideoSource wrapping the source.

Follow-up patch is at https://bugs.webkit.org/show_bug.cgi?id=211509.
Comment 18 youenn fablet 2020-05-06 11:55:11 PDT
A I understand it, the current workaround is the following:
- when page gets back to visible, validate that the audio capture track is not muted and not ended
- If it stays muted or is ended, call navigator.mediaDevices.getUserMedia({ audio: true}) to make a new capture audio track and use RTCRtpSender.replaceTrack to use the new audio track in lieu of the old one
Comment 19 Buddhika Jayawardhana 2021-03-14 16:40:36 PDT
I've made the following observations regarding this issue on iOS 14

1. If the webRTC call is interrupted by a phone call, audio track will emit `mute` event and when the interruptions is over, `some times` the audio track emits `unmute` event.
2. If we try to recapture media once the interruption is detected, the webRTC call can use the microphone, but the phone call is unable to use the microphone or speaker.
3. If the user leaves the browser the videoTrack emits `mute` event and `unmute` event when the user comes back to the browser.