Bug 180748 - Audio sometimes fail to capture in WebRTC
Summary: Audio sometimes fail to capture in WebRTC
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebRTC (show other bugs)
Version: Safari 11
Hardware: iPhone / iPad iOS 11
: P2 Normal
Assignee: youenn fablet
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2017-12-13 06:02 PST by daginge
Modified: 2019-09-10 08:27 PDT (History)
15 users (show)

See Also:


Attachments
Log of failed audio capture in desktop Safari 12.1 (57.10 KB, text/plain)
2019-05-16 12:36 PDT, daginge
no flags Details
Patch (5.00 KB, patch)
2019-09-09 07:53 PDT, youenn fablet
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description daginge 2017-12-13 06:02:09 PST
I'm seeing intermittent fails to capture/playback local audio requested through getUserMedia, both locally and when sent over a peer connection. This bug also shows up on WebRTC Samples. Oddly, this seems most often happens when I navigate to a new page, with subsequent refreshes working.

Steps to reproduce:
1. On an iPad, go to https://webrtc.github.io/samples/
2. Select sample "Audio-only getUserMedia() output to local audio element"
3. Make a pop sound and see if you can hear yourself.
4. If yes, close tab and open it again. Repeat 2-4 until you cannot hear yourself.

The above was tested on an iPad Pro MQDW2KN/A using iOS 11.2 (15C114). User agent reports AppleWebKit/604.4.7 Safari/604.1.

In the Developer Console, I see the success callback is triggered, and the stream object itself looks normal (muted = false). This is also true for the AudioTrack. However, no audio is playing. Other tests also fail, including AppRTC and our own solution. The "select camera and microphone" sometimes fails to show video as well, although I'm not sure if those are related.
Comment 1 daginge 2017-12-13 06:21:12 PST
From my testing:

- If it works the first time (when you open the tab), it will continue to work every time for that tab, even if you reload, or call getUserMedia again and replace the stream being played.
- If it doesn't work when you open the tab, it will never work no matter how many times you call getUserMedia  and replace the played back stream.
- Clicking the controls manually does not change anything.
- If a tab didn't work, sometimes it can start working on reload, but most likely, it will never work.

// test code run in dev console from Safari Desktop Remote Debugging to iPad Safari
a = document.getElementById("gum-local");
navigator.mediaDevices.getUserMedia({ video: true, audio: true}).then(stream => a.srcObject = stream).catch(console.log)
Comment 2 Radar WebKit Bug Importer 2017-12-13 14:44:04 PST
<rdar://problem/36032346>
Comment 3 youenn fablet 2017-12-13 15:01:19 PST
I can also reproduce this on an iPhone 6s.
It seems that you need to quickly close the capturing tab and reopen a new one.
Racing issue somewhere in CoreAudioCaptureSource maybe?
Comment 4 daginge 2017-12-13 23:46:34 PST
Could be, I unfortunately don't know enough about debug logging on Safari for iOS to be able to confirm.

We started noticing this bug when we initiated a new getUserMedia call right while another stream was stopping. (Our calling UI has a mirror so you can see yourself. Once the other party accepts we stop that stream, but almost immediately start a new stream inside the conversation).

I'm thinking we disable the mirror on iOS in hopes that we might not trigger the bug. I'll try to make a minimal case for this to confirm.
Comment 5 daginge 2017-12-13 23:56:39 PST
It seems that doesn't cause issues. I made a simple test for this.

1. Go to https://smart-skate.surge.sh/ on iPad
2. In remote Web Debugger, run "window.start()" and accept permissions
3. Observe that you can hear yourself
4. Run "window.restart()", and observe that after a brief pause, you can still hear yourself

I tested this quite a bit now, and it never seems to fail in that condition.
Comment 6 daginge 2017-12-19 02:01:28 PST
Made some additional testing. To me, it looks like the issue happens on the initial getUserMedia call in a tab. If that works, all subsequent calls to getUserMedia for that tab will work. This sounds to me like capturing the audio for that tab silently fails somewhere in the pipeline on the initial call, and that particular code path does not get retriggered on subsequent calls to getUserMedia.

This issue is currently making WebRTC unusable for audio calls in Safari. Our data shows a close to 50% failure rate on iPad Pro, slightly less on iPhone 6s, and non-existent on iPhone 6. As it stands, we cannot roll out WebRTC in Safari for iOS because of this.

Are you aware of any workaround to make this work?
Comment 7 Zeeshan Ghalib 2017-12-19 09:17:04 PST
I believe this bug is related to Bug 179964.
Bug 179964 has the steps to reproduce the issue consistently.
Comment 8 valbeattie 2018-01-23 20:10:21 PST
I am also seeing this issue of the unreliability of getUserMedia requests for audio. I cannot submit a reproducible scenario because it is not reproducible.  

But similar to the comments above, the behavior is the following, and it happens for me (specs below) more than 50% of the time:

- the getUserMedia call appears to succeed, but no audio from the mic flows (in fact, zeroes are returned)

- if you retry the getUserMedia call after it has failed as above (while on the page), you can retry as many times as you like, and it will never succeed.    

I see this behavior on an iPad Air, on 11.2.5 b7.  I have observed it both in our own webapp which does recording, and on the following demo site:

https://www.webrtc-experiment.com/RecordRTC/simple-demos/audio-recording.html

My testing does not involve interruptions from other apps or switching between tabs, I see this behavior when just trying to record in a single page with nothing else competing for the microphone.  I guess that makes me unsure that it is indeed related to https://bugs.webkit.org/show_bug.cgi?id=179964 as in the previous comment, but maybe. There are of course multiple (at least 2 anyway) calls to getUserMedia involved - one to obtain the mic permissions, and another to record.
Comment 9 valbeattie 2018-02-01 09:45:34 PST
Still seeing this issue on iOS 11.3
Comment 10 youenn fablet 2018-02-01 09:57:47 PST
> There are of course multiple (at least 2 anyway) calls to
> getUserMedia involved - one to obtain the mic permissions, and another to
> record.

Can you clarify why you would need to make two getUserMedia calls?

If you do so on iOS, the latter stream will take precedence over the former and the first stream tracks will all get muted.
Can you check the muted state of the tracks you are trying to record?
Comment 11 valbeattie 2018-02-01 10:11:45 PST
The muted state is false on the stream (from the second request, which is the one we are trying to record with).  In general for our web application (across platforms and browsers), the two calls to getUserMedia are needed to first get the permissions, and then second to select the device we want from the available list.  We have specific requirements for audio devices and do not want to take/allow just the default device necessarily.  We can't access the list of available media until the permissions are given. 

It would be fine for the second call (latter stream) to take precedence, and mute the initial one, we do not try to record with the initial one.  

I am not sure how necessary the two calls are on iOS.  It is still the case that we want the recording to be using a headset, so would need to know reliably that a headset device would be chosen (over the built in mic), based on just the initial call.
Comment 12 valbeattie 2018-02-07 10:53:09 PST
So one update: for our application, we modified it so that we are now only making one getUserMedia call, but we continue to see the issue that the returned stream is just feeding zeroes a significant percentage of the time (maybe roughly 30%).
Comment 13 youenn fablet 2018-02-07 18:19:18 PST
(In reply to valbeattie from comment #12)
> So one update: for our application, we modified it so that we are now only
> making one getUserMedia call, but we continue to see the issue that the
> returned stream is just feeding zeroes a significant percentage of the time
> (maybe roughly 30%).

valbeattie, I have a potential fix for https://bugs.webkit.org/show_bug.cgi?id=179964
If you have a precise repro case, I can try to see whether this would also fix your issue.
Comment 14 youenn fablet 2018-02-07 21:21:14 PST
OK, problem is still reproducing apparently.
Comment 15 jshoop 2018-02-08 13:51:08 PST
Obviously not the best solution,  but I have found that restarting Safari(on iOs, and between getUserMedia calls) allows the microphone to operate consistently.  This was mentioned in (https://bugs.webkit.org/show_bug.cgi?id=179964), but perhaps it can aid in resolution.
Comment 16 youenn fablet 2018-02-09 09:44:02 PST
Two workarounds that seem to work when done from the inspector console on https://webrtc.github.io/samples/src/content/getusermedia/volume/ tab with a dead meter:

1.
location.reload()

2.
// Stop capturing
stream.getAudioTracks()[0].stop()
//At that point, tab should no longer show any capturing icon, let's redo getUserMedia then
navigator.mediaDevices.getUserMedia(constraints).then(handleSuccess).catch(handleError);


Calling stop+getUserMedia synchronously does not seem to work if called once.
Replaying it twice makes it on my setup.

We probably have a race condition issue.

Let me know the result of this workaround if some of you have time to try it.
Comment 17 youenn fablet 2018-02-09 11:18:01 PST
https://youennf.github.io/webrtc-tests/src/content/getusermedia/volume/ helps playing with the workaround, need to do the "Retrying capture" dance twice.
Comment 18 valbeattie 2018-02-26 13:57:14 PST
The bug seems to still be there in the latest 11.3 beta.

I am a little unclear on how to apply your workaround in our app.  We see this e.g. if we use the microphone, exit the web app, and go back to it without restarting safari.

Are you saying we should call getUserMedia, then call the audioTracks stop(), then call getUserMedia again to reset things? 

Thanks.
Comment 19 youenn fablet 2018-02-26 14:11:28 PST
You can do a getUserMedia call. If you do not get any audio, you can do the following:
1. stop the audio track
2. call getUserMedia to get audio
3. Redo step 1 and 2 and you should hopefully get some audio
Let me know if that works for you.
Comment 20 jshoop 2018-02-27 13:28:20 PST
Will there be a fix for this issue or is this the official workaround solution? It  would be preferred not have Safari-specific switch statements. The other 'workaround' is to restart Safari after each GUM call: could this mean that perhaps something is not being cleaned up after(or before?) a GUM call?
Comment 21 valbeattie 2018-03-01 12:07:58 PST
So I did try the suggested workaround, and it does seem to work, but is far from ideal.  The user has to try and fail to record first, and then retry.  Given that this is an educational product used by young kids whose tolerance for frustration is low, it is not a good thing ... and makes the product appear flaky to both adults and kids.  

Please let us know of any progress on a fix.  Thanks!
Comment 22 valbeattie 2018-03-27 10:58:05 PDT
Some feedback on this bug and any likelihood of a fix would be very much appreciated.  It is blocking us and may delay the release of our webapp on iOS.
Comment 23 youenn fablet 2018-03-27 11:48:54 PDT
(In reply to valbeattie from comment #22)
> Some feedback on this bug and any likelihood of a fix would be very much
> appreciated.  It is blocking us and may delay the release of our webapp on
> iOS.

Hi, valbeattie, this is only a temporary workaround but I haven't been able to look at the underlying issue yet...

That said, the workaround can probably work without affecting too much the user.
You can for instance always call twice getUserMedia.
Not great for sure but user will not notice it (except maybe some additional delay) since there will be just one prompt on the first call.
Comment 24 valbeattie 2018-03-27 11:57:39 PDT
Hi thanks for replying.

Have tried just calling getUserMedia again behind the scenes but it did not work.  Not sure if it is because of the gesture requirement for mic access?  So, hard to do this in a way that is not intrusive/does not expose this flakiness to the user.
Comment 25 smith.marta@gmail.com 2018-03-27 13:06:09 PDT
Where does this issue stand in terms of priority to resolve? Can we expect a resolution for the 11.3 release?
Comment 26 Ronald Gemao 2019-02-22 02:38:45 PST
The issue will fixed if you mute/unmute or replace the video track. I don't know why it fixes in that way. 

Just make sure you have created a video track along with audio.
Comment 27 Ronald Gemao 2019-02-22 02:45:40 PST
The issue will be fixed when you unmute/mute the video track. I encountered this issue when using libjitsi library. I don't know why it fixes in that way. 


When testing,  make sure you have created both audio and video track. I hope there's a clear picture why this issue occurs. Thanks
Comment 28 daginge 2019-05-16 12:36:41 PDT
Created attachment 370059 [details]
Log of failed audio capture in desktop Safari 12.1

We're seeing this bug again in Safari 12.1. Refresh solved the issue with no other changes being made.

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1 Safari/605.1.15

To observe the issue in the log file, do the following steps:
1. Open https://fippo.github.io/webrtc-dump-importer/rtcstats
2. Open the file attached to this comment
3. Observe microphone used is "MacBook Air microfoon"
4. Scroll down to RTCOutboundRTPAudioStream
5. Uncheck "ssrc"
6. Observe that "bytesSent" is 0 through the call.

Seconds after the user refreshes, accepts media, restarts the call and it works no problem (this is another, much bigger, log file and is not included. Let me know if you want it). Note that video worked and was transmitted successfully in both these cases.

We'll be creating some telemetry on this to find the scope of this problem, but we're seeing increased reports of audio issues since the release of Safari 12.1.
Comment 29 daginge 2019-06-02 07:08:02 PDT
We have now implemented telemetry of this bug, and can report that the bug is still active in Safari 12.1.1.

We track it by looking at RTCOutboundRTPAudioStream's bytesSent 5 seconds after the peer connection goes to "connected". This also closely matches real user feedback for calls that fail.

Are there any status updates on this bug? It's very serious for us at the moment, as around 50% of our users are on iOS currently on the web.
Comment 30 Alojz Milicevic 2019-06-25 13:23:09 PDT
Can confirm 12.1.1 issues as reported by previous commentators. We are also looking for a solution to this and are eager for any news and help we can get!

I'm going to spend the next week on this issues and related issues, if any of you have ideas on something to try i am all ears! That being said here is what we have noticed so far:

 - If a tab has sound and we disable it by audioStream.enabled = false, the same tab can not regain it unless 
A. requesting and enabling video by calling videoStream.enabled. 
B. closing safari and opening it again (doesn't have to be a hard shutdown just switch to home screen and back).

(audioStream and videoStream are the streams we get from getUserMedia)

- If a tab/session/window does not have audio, there is nothing we have been able to do to gain it. Without first force quiting safari.

- When muting your sound, and switching video tracks. There seems to be no way of regaining your sound unless you also first enable the video track. 

This is what we have found so far. It is not bulletproof but will continue testing through this week at least.
Comment 31 youenn fablet 2019-06-25 13:26:19 PDT
@Alojz, thanks for the report.
Is it Desktop Safari, iOS safari or both?
Comment 32 Alojz Milicevic 2019-06-25 13:34:34 PDT
(In reply to youenn fablet from comment #31)
> @Alojz, thanks for the report.
> Is it Desktop Safari, iOS safari or both?

It seems to be only iOS, i just ran the same series of tests on desktop safari and wasn't able to reproduce the issues.
Comment 33 daginge 2019-09-03 06:42:13 PDT
We now have some concrete data for this bug. We have tracked bytesSent in outbound-rtp of the RTCRtpSender.getStats 5 seconds after the call connects, and have found an error rate of 1.5%. Users report killing Safari and opening it again fixes the issue most of the time.

We can also report no incidents on Safari Desktop, but then again, this configuration is so uncommon on our platform that it's hard to tell if we just have too few sessions, or that the problem is not present on OS X.

Have you gotten any further in debugging this issue youenn? Or anyone else in this thread?
Comment 34 youenn fablet 2019-09-03 07:07:47 PDT
(In reply to daginge from comment #33)
> We now have some concrete data for this bug. We have tracked bytesSent in
> outbound-rtp of the RTCRtpSender.getStats 5 seconds after the call connects,
> and have found an error rate of 1.5%. Users report killing Safari and
> opening it again fixes the issue most of the time.

Given no RTP packet is sent, I would expect the audio capture to have failed.
Can you confirm that the track is ended at that 5 seconds point?
Comment 35 daginge 2019-09-04 07:30:10 PDT
Very interesting result from that, youenn. Here's the result from one test:

{
    "userAgent": {
        "browser": {
            "family": "Mobile Safari",
            "major": "12",
            "minor": "1",
            "patch": "2",
            "source": "Mozilla/5.0 (iPhone; CPU iPhone OS 12_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.2 Mobile/15E148 Safari/604.1"
        },
        "os": "iOS"
    },
    "id": "RTCOutboundRTPAudioStream_3100197038",
    "timestamp": 1567592898380,
    "type": "outbound-rtp",
    "codecId": "RTCCodec_audio_Outbound_111",
    "isRemote": false,
    "mediaType": "audio",
    "qpSum": 0,
    "ssrc": 3100197038,
    "trackId": "RTCMediaStreamTrack_sender_5",
    "transportId": "RTCTransport_audio_1",
    "bytesSent": 0,
    "packetsSent": 0,
    "tracks": [
        {
            "contentHint": "",
            "enabled": true,
            "id": "39d78be2-ab05-4e3b-a3f2-8ff1c830e50d",
            "kind": "audio",
            "label": "iPhone Mikrofon",
            "muted": true,
            "readyState": "live"
        }
    ]
}

Notice the muted: true on the audio track. Now, this isn't under the application's control, and it's something the OS does. This will explain why the audio isn't transmitted, but WHY is it muted?

So we did some more testing and created a minimal test case. What we wanted to do was hook onto onmute and onunmute and observe when the OS triggers mute and unmute of the audio track, and if this gets stuck somehow.

I'm happy to report we found several cases where this could happen, and this seems to be the root cause. To keep this from being too long I'll post detailed reproduction steps in a new post.
Comment 36 daginge 2019-09-04 07:39:13 PDT
To reproduce this error, do the following on iOS:

1. Kill Safari _on iOS_ to make sure you got a clean testing slate.
2. Open https://codepen.io/daginge/full/pozdrwj in a new tab
3. Accept media permissions
4. Observe that you can hear yourself with some feedback, and see that the logs say that audio is unmuted.
5. Go to home screen
6. Observe that the red pill/bar is there
7. Trigger Siri and say "Go away"
8. Observe that the red pill/bar is there
9. Trigger Siri and say "Go away"
10. Observe that the red pill is now gone
11. Go back to Safari and see that the logs show that audio was muted, but never unmuted. You can also no longer hear yourself
12. Refresh the tab, notice that getUserMedia says audio is unmuted, yet you cannot hear feedback. This tab is now broken forever. Killing Safari resets it.

I can't reliably reproduce this on all phones, some phones opening Spotify and playing audio works to trigger the audio, on some a single call to Siri will trigger it. There seems to be a race condition here.

I have also recorded this happening, you can read the logs towards the end of the video and see that audio is never unmuted. https://photos.app.goo.gl/hA9fEFSo7fF8PSny7

It took us two years, but we have finally narrowed down the bug to something reproducible!
Comment 37 daginge 2019-09-04 07:57:02 PDT
I tried replicating this on the Whereby/appear.in native app, and when in a call there the red pill is now green, and Siri cannot be invoked (using the home screen/power button). So here the issue isn't triggerable.

Why is the pill green and not red? And why does Safari get the red pill and triggering Siri while getUserMedia is active?
Comment 38 sventy 2019-09-04 08:48:59 PDT
I can reproduce this issue consistently by:

1. Opening daginge's fiddle in Safari
2. Background Safari and open another app that plays sound (eg Spotify)
3. Kill that other app

Stopping the sound in the other app and going back to Safari will not produce the issue - the sound will then work again in Safari.

So it appears there is some missing cleanup going on when the app that took over the sound is killed.
Comment 39 youenn fablet 2019-09-04 08:53:36 PDT
Thanks for all reports, this makes it a very actionable bug.

It seems there is an issue in the audio interruption handling.
We should in theory uninterrupt ourselves based on a callback.

I am surprised that refreshing the page or triggering another getUserMedia call does not fix the issue. We should fix this.
Comment 40 youenn fablet 2019-09-09 07:53:26 PDT
Created attachment 378367 [details]
Patch
Comment 41 Eric Carlson 2019-09-09 09:50:29 PDT
Comment on attachment 378367 [details]
Patch

r=me
Comment 42 WebKit Commit Bot 2019-09-10 08:27:48 PDT
Comment on attachment 378367 [details]
Patch

Clearing flags on attachment: 378367

Committed r249715: <https://trac.webkit.org/changeset/249715>
Comment 43 WebKit Commit Bot 2019-09-10 08:27:50 PDT
All reviewed patches have been landed.  Closing bug.