Bug 180748 – Audio sometimes fail to capture in WebRTC

• Status: RESOLVED
• Resolution: FIXED
• Component: WebRTC

[Dag-Inge Aas]

I'm seeing intermittent fails to capture/playback local audio requested through getUserMedia, both locally and when sent over a peer connection. This bug also shows up on WebRTC Samples. Oddly, this seems most often happens when I navigate to a new page, with subsequent refreshes working.

Steps to reproduce:
1. On an iPad, go to https://webrtc.github.io/samples/
2. Select sample "Audio-only getUserMedia() output to local audio element"
3. Make a pop sound and see if you can hear yourself.
4. If yes, close tab and open it again. Repeat 2-4 until you cannot hear yourself.

The above was tested on an iPad Pro MQDW2KN/A using iOS 11.2 (15C114). User agent reports AppleWebKit/604.4.7 Safari/604.1.

In the Developer Console, I see the success callback is triggered, and the stream object itself looks normal (muted = false). This is also true for the AudioTrack. However, no audio is playing. Other tests also fail, including AppRTC and our own solution. The "select camera and microphone" sometimes fails to show video as well, although I'm not sure if those are related.

[Dag-Inge Aas]

From my testing:

- If it works the first time (when you open the tab), it will continue to work every time for that tab, even if you reload, or call getUserMedia again and replace the stream being played.
- If it doesn't work when you open the tab, it will never work no matter how many times you call getUserMedia  and replace the played back stream.
- Clicking the controls manually does not change anything.
- If a tab didn't work, sometimes it can start working on reload, but most likely, it will never work.

// test code run in dev console from Safari Desktop Remote Debugging to iPad Safari
a = document.getElementById("gum-local");
navigator.mediaDevices.getUserMedia({ video: true, audio: true}).then(stream => a.srcObject = stream).catch(console.log)

[Radar WebKit Bug Importer]

<rdar://problem/36032346>

[youenn fablet]

I can also reproduce this on an iPhone 6s.
It seems that you need to quickly close the capturing tab and reopen a new one.
Racing issue somewhere in CoreAudioCaptureSource maybe?

[Dag-Inge Aas]

Could be, I unfortunately don't know enough about debug logging on Safari for iOS to be able to confirm.

We started noticing this bug when we initiated a new getUserMedia call right while another stream was stopping. (Our calling UI has a mirror so you can see yourself. Once the other party accepts we stop that stream, but almost immediately start a new stream inside the conversation).

I'm thinking we disable the mirror on iOS in hopes that we might not trigger the bug. I'll try to make a minimal case for this to confirm.

[Dag-Inge Aas]

It seems that doesn't cause issues. I made a simple test for this.

1. Go to https://smart-skate.surge.sh/ on iPad
2. In remote Web Debugger, run "window.start()" and accept permissions
3. Observe that you can hear yourself
4. Run "window.restart()", and observe that after a brief pause, you can still hear yourself

I tested this quite a bit now, and it never seems to fail in that condition.

[Dag-Inge Aas]

Made some additional testing. To me, it looks like the issue happens on the initial getUserMedia call in a tab. If that works, all subsequent calls to getUserMedia for that tab will work. This sounds to me like capturing the audio for that tab silently fails somewhere in the pipeline on the initial call, and that particular code path does not get retriggered on subsequent calls to getUserMedia.

This issue is currently making WebRTC unusable for audio calls in Safari. Our data shows a close to 50% failure rate on iPad Pro, slightly less on iPhone 6s, and non-existent on iPhone 6. As it stands, we cannot roll out WebRTC in Safari for iOS because of this.

Are you aware of any workaround to make this work?

[Zeeshan Ghalib]

I believe this bug is related to Bug 179964.
Bug 179964 has the steps to reproduce the issue consistently.

I am also seeing this issue of the unreliability of getUserMedia requests for audio. I cannot submit a reproducible scenario because it is not reproducible.  

But similar to the comments above, the behavior is the following, and it happens for me (specs below) more than 50% of the time:

- the getUserMedia call appears to succeed, but no audio from the mic flows (in fact, zeroes are returned)

- if you retry the getUserMedia call after it has failed as above (while on the page), you can retry as many times as you like, and it will never succeed.    

I see this behavior on an iPad Air, on 11.2.5 b7.  I have observed it both in our own webapp which does recording, and on the following demo site:

https://www.webrtc-experiment.com/RecordRTC/simple-demos/audio-recording.html

My testing does not involve interruptions from other apps or switching between tabs, I see this behavior when just trying to record in a single page with nothing else competing for the microphone.  I guess that makes me unsure that it is indeed related to https://bugs.webkit.org/show_bug.cgi?id=179964 as in the previous comment, but maybe. There are of course multiple (at least 2 anyway) calls to getUserMedia involved - one to obtain the mic permissions, and another to record.

Still seeing this issue on iOS 11.3

[youenn fablet]

> There are of course multiple (at least 2 anyway) calls to
> getUserMedia involved - one to obtain the mic permissions, and another to
> record.

Can you clarify why you would need to make two getUserMedia calls?

If you do so on iOS, the latter stream will take precedence over the former and the first stream tracks will all get muted.
Can you check the muted state of the tracks you are trying to record?

The muted state is false on the stream (from the second request, which is the one we are trying to record with).  In general for our web application (across platforms and browsers), the two calls to getUserMedia are needed to first get the permissions, and then second to select the device we want from the available list.  We have specific requirements for audio devices and do not want to take/allow just the default device necessarily.  We can't access the list of available media until the permissions are given. 

It would be fine for the second call (latter stream) to take precedence, and mute the initial one, we do not try to record with the initial one.  

I am not sure how necessary the two calls are on iOS.  It is still the case that we want the recording to be using a headset, so would need to know reliably that a headset device would be chosen (over the built in mic), based on just the initial call.

So one update: for our application, we modified it so that we are now only making one getUserMedia call, but we continue to see the issue that the returned stream is just feeding zeroes a significant percentage of the time (maybe roughly 30%).

[youenn fablet]

(In reply to valbeattie from comment #12)
> So one update: for our application, we modified it so that we are now only
> making one getUserMedia call, but we continue to see the issue that the
> returned stream is just feeding zeroes a significant percentage of the time
> (maybe roughly 30%).

valbeattie, I have a potential fix for https://bugs.webkit.org/show_bug.cgi?id=179964
If you have a precise repro case, I can try to see whether this would also fix your issue.

[youenn fablet]

OK, problem is still reproducing apparently.

[jshoop]

Obviously not the best solution,  but I have found that restarting Safari(on iOs, and between getUserMedia calls) allows the microphone to operate consistently.  This was mentioned in (https://bugs.webkit.org/show_bug.cgi?id=179964), but perhaps it can aid in resolution.

[youenn fablet]

Two workarounds that seem to work when done from the inspector console on https://webrtc.github.io/samples/src/content/getusermedia/volume/ tab with a dead meter:

1.
location.reload()

2.
// Stop capturing
stream.getAudioTracks()[0].stop()
//At that point, tab should no longer show any capturing icon, let's redo getUserMedia then
navigator.mediaDevices.getUserMedia(constraints).then(handleSuccess).catch(handleError);


Calling stop+getUserMedia synchronously does not seem to work if called once.
Replaying it twice makes it on my setup.

We probably have a race condition issue.

Let me know the result of this workaround if some of you have time to try it.

[youenn fablet]

https://youennf.github.io/webrtc-tests/src/content/getusermedia/volume/ helps playing with the workaround, need to do the "Retrying capture" dance twice.

The bug seems to still be there in the latest 11.3 beta.

I am a little unclear on how to apply your workaround in our app.  We see this e.g. if we use the microphone, exit the web app, and go back to it without restarting safari.

Are you saying we should call getUserMedia, then call the audioTracks stop(), then call getUserMedia again to reset things? 

Thanks.

[youenn fablet]

You can do a getUserMedia call. If you do not get any audio, you can do the following:
1. stop the audio track
2. call getUserMedia to get audio
3. Redo step 1 and 2 and you should hopefully get some audio
Let me know if that works for you.

[jshoop]

Will there be a fix for this issue or is this the official workaround solution? It  would be preferred not have Safari-specific switch statements. The other 'workaround' is to restart Safari after each GUM call: could this mean that perhaps something is not being cleaned up after(or before?) a GUM call?

So I did try the suggested workaround, and it does seem to work, but is far from ideal.  The user has to try and fail to record first, and then retry.  Given that this is an educational product used by young kids whose tolerance for frustration is low, it is not a good thing ... and makes the product appear flaky to both adults and kids.  

Please let us know of any progress on a fix.  Thanks!

Some feedback on this bug and any likelihood of a fix would be very much appreciated.  It is blocking us and may delay the release of our webapp on iOS.

[youenn fablet]

(In reply to valbeattie from comment #22)
> Some feedback on this bug and any likelihood of a fix would be very much
> appreciated.  It is blocking us and may delay the release of our webapp on
> iOS.

Hi, valbeattie, this is only a temporary workaround but I haven't been able to look at the underlying issue yet...

That said, the workaround can probably work without affecting too much the user.
You can for instance always call twice getUserMedia.
Not great for sure but user will not notice it (except maybe some additional delay) since there will be just one prompt on the first call.

Hi thanks for replying.

Have tried just calling getUserMedia again behind the scenes but it did not work.  Not sure if it is because of the gesture requirement for mic access?  So, hard to do this in a way that is not intrusive/does not expose this flakiness to the user.

[[e-mail]]

Where does this issue stand in terms of priority to resolve? Can we expect a resolution for the 11.3 release?

[Ronald Gemao]

The issue will fixed if you mute/unmute or replace the video track. I don't know why it fixes in that way. 

Just make sure you have created a video track along with audio.

[Ronald Gemao]

The issue will be fixed when you unmute/mute the video track. I encountered this issue when using libjitsi library. I don't know why it fixes in that way. 


When testing,  make sure you have created both audio and video track. I hope there's a clear picture why this issue occurs. Thanks

[Dag-Inge Aas]

Created attachment 370059 [details]
Log of failed audio capture in desktop Safari 12.1

We're seeing this bug again in Safari 12.1. Refresh solved the issue with no other changes being made.

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1 Safari/605.1.15

To observe the issue in the log file, do the following steps:
1. Open https://fippo.github.io/webrtc-dump-importer/rtcstats
2. Open the file attached to this comment
3. Observe microphone used is "MacBook Air microfoon"
4. Scroll down to RTCOutboundRTPAudioStream
5. Uncheck "ssrc"
6. Observe that "bytesSent" is 0 through the call.

Seconds after the user refreshes, accepts media, restarts the call and it works no problem (this is another, much bigger, log file and is not included. Let me know if you want it). Note that video worked and was transmitted successfully in both these cases.

We'll be creating some telemetry on this to find the scope of this problem, but we're seeing increased reports of audio issues since the release of Safari 12.1.

[Dag-Inge Aas]

We have now implemented telemetry of this bug, and can report that the bug is still active in Safari 12.1.1.

We track it by looking at RTCOutboundRTPAudioStream's bytesSent 5 seconds after the peer connection goes to "connected". This also closely matches real user feedback for calls that fail.

Are there any status updates on this bug? It's very serious for us at the moment, as around 50% of our users are on iOS currently on the web.

[Alojz Milicevic]

Can confirm 12.1.1 issues as reported by previous commentators. We are also looking for a solution to this and are eager for any news and help we can get!

I'm going to spend the next week on this issues and related issues, if any of you have ideas on something to try i am all ears! That being said here is what we have noticed so far:

 - If a tab has sound and we disable it by audioStream.enabled = false, the same tab can not regain it unless 
A. requesting and enabling video by calling videoStream.enabled. 
B. closing safari and opening it again (doesn't have to be a hard shutdown just switch to home screen and back).

(audioStream and videoStream are the streams we get from getUserMedia)

- If a tab/session/window does not have audio, there is nothing we have been able to do to gain it. Without first force quiting safari.

- When muting your sound, and switching video tracks. There seems to be no way of regaining your sound unless you also first enable the video track. 

This is what we have found so far. It is not bulletproof but will continue testing through this week at least.

[youenn fablet]

@Alojz, thanks for the report.
Is it Desktop Safari, iOS safari or both?

[Alojz Milicevic]

(In reply to youenn fablet from comment #31)
> @Alojz, thanks for the report.
> Is it Desktop Safari, iOS safari or both?

It seems to be only iOS, i just ran the same series of tests on desktop safari and wasn't able to reproduce the issues.

[Dag-Inge Aas]

We now have some concrete data for this bug. We have tracked bytesSent in outbound-rtp of the RTCRtpSender.getStats 5 seconds after the call connects, and have found an error rate of 1.5%. Users report killing Safari and opening it again fixes the issue most of the time.

We can also report no incidents on Safari Desktop, but then again, this configuration is so uncommon on our platform that it's hard to tell if we just have too few sessions, or that the problem is not present on OS X.

Have you gotten any further in debugging this issue youenn? Or anyone else in this thread?

[youenn fablet]

(In reply to daginge from comment #33)
> We now have some concrete data for this bug. We have tracked bytesSent in
> outbound-rtp of the RTCRtpSender.getStats 5 seconds after the call connects,
> and have found an error rate of 1.5%. Users report killing Safari and
> opening it again fixes the issue most of the time.

Given no RTP packet is sent, I would expect the audio capture to have failed.
Can you confirm that the track is ended at that 5 seconds point?

[Dag-Inge Aas]

Very interesting result from that, youenn. Here's the result from one test:

{
    "userAgent": {
        "browser": {
            "family": "Mobile Safari",
            "major": "12",
            "minor": "1",
            "patch": "2",
            "source": "Mozilla/5.0 (iPhone; CPU iPhone OS 12_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.2 Mobile/15E148 Safari/604.1"
        },
        "os": "iOS"
    },
    "id": "RTCOutboundRTPAudioStream_3100197038",
    "timestamp": 1567592898380,
    "type": "outbound-rtp",
    "codecId": "RTCCodec_audio_Outbound_111",
    "isRemote": false,
    "mediaType": "audio",
    "qpSum": 0,
    "ssrc": 3100197038,
    "trackId": "RTCMediaStreamTrack_sender_5",
    "transportId": "RTCTransport_audio_1",
    "bytesSent": 0,
    "packetsSent": 0,
    "tracks": [
        {
            "contentHint": "",
            "enabled": true,
            "id": "39d78be2-ab05-4e3b-a3f2-8ff1c830e50d",
            "kind": "audio",
            "label": "iPhone Mikrofon",
            "muted": true,
            "readyState": "live"
        }
    ]
}

Notice the muted: true on the audio track. Now, this isn't under the application's control, and it's something the OS does. This will explain why the audio isn't transmitted, but WHY is it muted?

So we did some more testing and created a minimal test case. What we wanted to do was hook onto onmute and onunmute and observe when the OS triggers mute and unmute of the audio track, and if this gets stuck somehow.

I'm happy to report we found several cases where this could happen, and this seems to be the root cause. To keep this from being too long I'll post detailed reproduction steps in a new post.

[Dag-Inge Aas]

To reproduce this error, do the following on iOS:

1. Kill Safari _on iOS_ to make sure you got a clean testing slate.
2. Open https://codepen.io/daginge/full/pozdrwj in a new tab
3. Accept media permissions
4. Observe that you can hear yourself with some feedback, and see that the logs say that audio is unmuted.
5. Go to home screen
6. Observe that the red pill/bar is there
7. Trigger Siri and say "Go away"
8. Observe that the red pill/bar is there
9. Trigger Siri and say "Go away"
10. Observe that the red pill is now gone
11. Go back to Safari and see that the logs show that audio was muted, but never unmuted. You can also no longer hear yourself
12. Refresh the tab, notice that getUserMedia says audio is unmuted, yet you cannot hear feedback. This tab is now broken forever. Killing Safari resets it.

I can't reliably reproduce this on all phones, some phones opening Spotify and playing audio works to trigger the audio, on some a single call to Siri will trigger it. There seems to be a race condition here.

I have also recorded this happening, you can read the logs towards the end of the video and see that audio is never unmuted. https://photos.app.goo.gl/hA9fEFSo7fF8PSny7

It took us two years, but we have finally narrowed down the bug to something reproducible!

[Dag-Inge Aas]

I tried replicating this on the Whereby/appear.in native app, and when in a call there the red pill is now green, and Siri cannot be invoked (using the home screen/power button). So here the issue isn't triggerable.

Why is the pill green and not red? And why does Safari get the red pill and triggering Siri while getUserMedia is active?

[sventy]

I can reproduce this issue consistently by:

1. Opening daginge's fiddle in Safari
2. Background Safari and open another app that plays sound (eg Spotify)
3. Kill that other app

Stopping the sound in the other app and going back to Safari will not produce the issue - the sound will then work again in Safari.

So it appears there is some missing cleanup going on when the app that took over the sound is killed.

[youenn fablet]

Thanks for all reports, this makes it a very actionable bug.

It seems there is an issue in the audio interruption handling.
We should in theory uninterrupt ourselves based on a callback.

I am surprised that refreshing the page or triggering another getUserMedia call does not fix the issue. We should fix this.

[youenn fablet]

Created attachment 378367 [details]
Patch

[Eric Carlson]

Comment on attachment 378367 [details]
Patch

r=me

[WebKit Commit Bot]

Comment on attachment 378367 [details]
Patch

Clearing flags on attachment: 378367

Committed r249715: <https://trac.webkit.org/changeset/249715>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.