Bug 208516 - Audio fails to capture stream in WebRTC if AudioSession gets interrupted
Summary: Audio fails to capture stream in WebRTC if AudioSession gets interrupted
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebRTC (show other bugs)
Version: Safari 13
Hardware: iPhone / iPad iOS 13
: P2 Major
Assignee: youenn fablet
URL:
Keywords: InRadar
Depends on: 210555
Blocks: 204681
  Show dependency treegraph
 
Reported: 2020-03-03 08:53 PST by David Engelmaier
Modified: 2021-03-14 16:40 PDT (History)
15 users (show)

See Also:


Attachments
Multiple audio tracks listed (1.95 MB, image/png)
2020-03-03 08:53 PST, David Engelmaier
no flags Details
Patch (7.31 KB, patch)
2020-03-23 01:47 PDT, youenn fablet
no flags Details | Formatted Diff | Diff
Patch (11.05 KB, patch)
2020-03-24 04:12 PDT, youenn fablet
no flags Details | Formatted Diff | Diff
Patch (11.09 KB, patch)
2020-03-24 05:43 PDT, youenn fablet
no flags Details | Formatted Diff | Diff
Patch for landing (11.11 KB, patch)
2020-03-25 03:46 PDT, youenn fablet
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description David Engelmaier 2020-03-03 08:53:12 PST
Created attachment 392279 [details]
Multiple audio tracks listed

When an application that uses camera and/or microphone is used(ie called to) during the time a web app with webrtc functionality is open, the web app is never
able to capture audio from the stream.

Prerequisites:
1. iphone 8 with ios 13.3.1
2. skype installed on the device with an account logged in, also teams, call app or facetime have been reported to work, although i was using mostly skype

Steps to reproduce:
1. open https://webrtc.github.io/samples/src/content/peerconnection/pc1/ in safari
2. start, call, hangup, reload the tab
3. call the device on skype, reject the call
4. go back to the safari tab, start, call, video frozen, no sound, from now on you can not use webrtc audio in safari, not even newly open tabs
5. connect to the iphone from safari develoop, run:
navigator.mediaDevices.getUserMedia({audio: true, video: true}).then(stream => {stream.getTracks().forEach(track => {track.stop();console.log(track.getSettings().sampleRate);console.log(track)})}).catch(error => {console.log(error)});

you'll see the sampleRate being always 0, sometimes calling the above code along with document.location.reload() multiple times(see attached screenshot) can bring safari back to working
webrtc audio, however this is very sporadic and can not be used as a workaround

This very issue was first filed as a bug here on 2017-12-13 06:02:09 PST https://bugs.webkit.org/show_bug.cgi?id=180748 and marked as resolved, however still happening.

There is also an interesting listing of media tracks in the attachment as you can see at one point there are 2 audio and 2 video tracks for the same interfaces
Comment 1 Radar WebKit Bug Importer 2020-03-03 17:13:33 PST
<rdar://problem/60020467>
Comment 2 Nándor Kostyó-Pajer 2020-03-03 23:44:56 PST
this is exactly the same problem i'm facing with a webrtc implementation. i created a simple app, that does not even use webrtc at all, just invokes getUserMedia, and then looks for the audio sampleRate, which in this case is simply 0.

trouble is, that if i manage to break the functionality on a tab with the steps described by David, the audio track will also be broken on other tabs, requiring us to kill the browser to make it function again.
Comment 3 youenn fablet 2020-03-05 02:05:38 PST
I tried on iOS 13.4 and do not see that issue.
I did step 1 and 2.
Instead of 3, I did a FaceTime call.
I did step 4 and was able to capture and do regular WebRTC calls.

The listing of two video tracks/two audio tracks seems indeed weird.
Comment 4 youenn fablet 2020-03-23 01:47:33 PDT
Created attachment 394246 [details]
Patch
Comment 5 Nándor Kostyó-Pajer 2020-03-23 02:04:36 PDT
we are constantly able to reproduce, and found some other strange symptoms: in case user media is requested, the appropriate icons appear in the address bar. when user media tracks are stopped, the camera icon is crossed, and stays there. in case of page reload, the icon still persists, and only disappears on the second pageload.

one workaround we found, is to navigate away from the page to a different subdomain, which will effectively remove the crossed camera icon. this reduced the possibility of breaking the audio in the tab to a minimum, otherwise, after the user media is requested, but no longer used by webrtc will easily break if another app takes over the microphone.
Comment 6 youenn fablet 2020-03-23 02:22:00 PDT
(In reply to Nándor Kostyó-Pajer from comment #5)
> we are constantly able to reproduce, and found some other strange symptoms:
> in case user media is requested, the appropriate icons appear in the address
> bar. when user media tracks are stopped, the camera icon is crossed, and
> stays there. in case of page reload, the icon still persists, and only
> disappears on the second pageload.
> 
> one workaround we found, is to navigate away from the page to a different
> subdomain, which will effectively remove the crossed camera icon. this
> reduced the possibility of breaking the audio in the tab to a minimum,
> otherwise, after the user media is requested, but no longer used by webrtc
> will easily break if another app takes over the microphone.

Can you provide a fiddle?
Can you try it on the latest iOS 13.4 version?

I tried the following on iOS 13.4:
- Open https://webrtc.github.io/samples/src/content/peerconnection/pc1/ in iOS device
- Click Start and Call
- Open Web Inspector and execute localStream.getVideoTracks()[0].stop()
- The icon should switch from camera to microphone
- Execute localStream.getAudioTracks()[0].stop()
- The icon should disappear.
Comment 7 Nándor Kostyó-Pajer 2020-03-23 03:57:40 PDT
i was doing reproductions on 13.3.1, now that i upgraded to the latest 13.4 beta, the persisting media icons cannot be reproduced.

will check the rest of the issue.
Comment 8 youenn fablet 2020-03-23 11:09:33 PDT
Test is failing due to change in Page::setMuted.
Comment 9 youenn fablet 2020-03-24 04:12:00 PDT
Created attachment 394358 [details]
Patch
Comment 10 youenn fablet 2020-03-24 05:43:37 PDT
Created attachment 394361 [details]
Patch
Comment 11 Eric Carlson 2020-03-24 09:35:47 PDT
Comment on attachment 394361 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=394361&action=review

> Source/WebCore/Modules/mediastream/MediaStreamTrack.cpp:471
> +static MediaStreamTrack* selectDocumentCaptureTrack(Document& document, RealtimeMediaSource* activeSource, RealtimeMediaSource::Type type)

Nit: this method doesn't select a track, so maybe something like `findActiveCaptureTrackForDocument` would be better?
Comment 12 youenn fablet 2020-03-25 03:46:51 PDT
Created attachment 394481 [details]
Patch for landing
Comment 13 youenn fablet 2020-03-25 06:33:55 PDT
(In reply to Eric Carlson from comment #11)
> Comment on attachment 394361 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=394361&action=review
> 
> > Source/WebCore/Modules/mediastream/MediaStreamTrack.cpp:471
> > +static MediaStreamTrack* selectDocumentCaptureTrack(Document& document, RealtimeMediaSource* activeSource, RealtimeMediaSource::Type type)
> 
> Nit: this method doesn't select a track, so maybe something like
> `findActiveCaptureTrackForDocument` would be better?

Thanks, done.
Comment 14 EWS 2020-03-25 06:44:23 PDT
Committed r258977: <https://trac.webkit.org/changeset/258977>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 394481 [details].
Comment 15 Ryan Haddad 2020-03-25 11:37:42 PDT
Follow up build fix in https://trac.webkit.org/changeset/258991
Comment 16 Nándor Kostyó-Pajer 2020-04-03 06:44:22 PDT
would you please help me find information regarding which ios version will contain this patch?

i was still able to break webrtc functionality with a simple MS Teams call in 13.4.5 beta using the described way in the original post.
Comment 17 youenn fablet 2020-05-06 06:24:58 PDT
This probably broke unmuting of video capture in case of tab switching.
The issue is that video unmuting when unhidden or not, might either mute/unmute the AVVideoCaptureSource or the RealtimeVideoSource wrapping the source.

Follow-up patch is at https://bugs.webkit.org/show_bug.cgi?id=211509.
Comment 18 youenn fablet 2020-05-06 11:55:11 PDT
A I understand it, the current workaround is the following:
- when page gets back to visible, validate that the audio capture track is not muted and not ended
- If it stays muted or is ended, call navigator.mediaDevices.getUserMedia({ audio: true}) to make a new capture audio track and use RTCRtpSender.replaceTrack to use the new audio track in lieu of the old one
Comment 19 Buddhika Jayawardhana 2021-03-14 16:40:36 PDT
I've made the following observations regarding this issue on iOS 14

1. If the webRTC call is interrupted by a phone call, audio track will emit `mute` event and when the interruptions is over, `some times` the audio track emits `unmute` event.
2. If we try to recapture media once the interruption is detected, the webRTC call can use the microphone, but the phone call is unable to use the microphone or speaker.
3. If the user leaves the browser the videoTrack emits `mute` event and `unmute` event when the user comes back to the browser.