Bug 220599 – WebRTC live Opus audio stream stutters

• Status: RESOLVED
• Resolution: FIXED
• Component: WebRTC

[Mikael Nousiainen]

Created attachment 417555 [details]
Stuttering audio from live WebRTC Opus stream

The symptoms of this bug are very similar to bug https://bugs.webkit.org/show_bug.cgi?id=218762
but this is clearly a separate issue, as fixing that bug did not fix this one.

WebRTC live Opus audio stream stutters all the time.
The audio sounds like a "machine gun" (there's an audio clip as an attachment), where the audio is sliced into small pieces.

Here's a small JavaScript application, which reproduces the issue consistently every time.
The application should start playing back a 750 Hz sine wave tone after clicking the Play button.

https://webrtctest.incompleteopus.net:9001/webrtctest.html

There is Janus Gateway running in the background and the audio is generated by GStreamer in real time.
There is also a Coturn STUN/TURN server running too.

This sounds like some buffering issue in Safari or something related -- at least based on bug mentioned above, which was fixed.

The sine wave stream plays back cleanly in Firefox and Chrome (also on macOS) and the exact same WebRTC streaming setup was working
fine in Safari too until June/July 2020, somewhere around iOS 13.5 release. After that it has been broken and all Safari 14
release I've tested are broken too.

The issue can be reproduced on both iOS and macOS Safari.

I've tested this on Safari Technology Preview Release 118 (Safari 14.1, WebKit 16611.1.9.3.1)
and verified that the bug still exists on this latest TP release at the moment.

[Mikael Nousiainen]

A note about the test application implementation:

The incoming WebRTC audio stream is played back through an AudioContext (to allow audio processing and visualization) and streaming is initiated by creating an Audio element where the stream is first assigned as srcObject and then wrapped via createMediaStreamSource() to be connected into the AudioContext.

The source code can be read from the example website.

[Mikael Nousiainen]

This is a real blocker for my use case and I have not found any workarounds for it. I wonder that if anyone else has the same issue with live Opus streams? Seems they simple don't work -- not with AudioContext at least...

Anyway, I've updated the test application to work properly on desktop Safari 14 (there were some ES6 features in the code only supported by Safari Technology Preview). Now the test app works on desktop Safari 14 (14.0.2) and reproduces the issue on macOS.

However, while I can play back the audio on iOS 14 in the application the test case originates from, this example seems to produce only silence on iOS 14 (iPhone 7) currently. I have checked logs on Web Inspector and get no errors or other indications. I'm using a user gesture to trigger the audio playback and specifically resuming AudioContext in the click handler too.

[Radar WebKit Bug Importer]

<rdar://problem/73190139>

[youenn fablet]

Thanks Mikeal, I can reproduce the issue.

[youenn fablet]

I wrote https://jsfiddle.net/b5sr8ey9/ which seems to be a reduced test case for this issue.
Creating two media stream source nodes on the same track is not working properly.
@Mikael, can you update your code to not do that and validate it fixes everything on your side?

A fix is needed on WebKit side.

[youenn fablet]

Reduced test case with just local capture audio track: https://jsfiddle.net/7kf8r3de/

[youenn fablet]

Created attachment 417689 [details]
Patch

[Mikael Nousiainen]

Thanks for finding out the root cause, providing a patch and even pointing out a mistake in my code! (example updated)

I somehow never realized the stream was being created twice (because of some internal logic on Janus Gateway client code).

I can confirm that the stream plays without glitches in Safari 14 when there's only one source node attached.

However, what still bothers me is why iOS Safari produces only silence with this example. I'm using an iPhone 7 and just upgraded to latest iOS 14.4 Beta 2 yesterday.

Can you find out if there's another bug only on iOS -- or what is preventing  playback on mobile devices?

[youenn fablet]

> Can you find out if there's another bug only on iOS -- or what is preventing
> playback on mobile devices?

It might be auto play policy restrictions.
You can try to do suspend/resume in inspector or enable Media/WebRTC logging to check that within the inspector.

[youenn fablet]

(In reply to youenn fablet from comment #9)
> > Can you find out if there's another bug only on iOS -- or what is preventing
> > playback on mobile devices?
> 
> It might be auto play policy restrictions.
> You can try to do suspend/resume in inspector or enable Media/WebRTC logging
> to check that within the inspector.

If you execute JS from inspector, make sure to enable the "Emulate User Gesture" check box.

[youenn fablet]

Created attachment 417695 [details]
Patch

[Mikael Nousiainen]

I'm testing the example app directly on an iPhone with no Web Inspector or anything attached and it doesn't play anything.

There is code to make sure that AudioContext is resumed in the button click handler, but it doesn't seem to help.

Any other ideas? Are you able to test this yourself? Shouldn't the macOS and iOS Safari behave the same way here?

[Mikael Nousiainen]

I enabled WebRTC and Media logging and got some log statements regarding audio playback not being permitted because it's not initiated by user gesture. However, I do get the exact same logs in desktop/macOS Safari and the playback works.

I've attached the full log file to this bug, but here are some interesting parts:

[Log] HTMLMediaElement::HTMLMediaElement(C5E20E3B8BE94F27) 
[Log] MediaSessionManageriOS::addSession(0) (B71CA082AB67FBF9)
[Log] MediaSessionManageriOS::updateSessionState(0) types: AudioCapture(0), Video(0), Audio(1), VideoAudio(0), WebAudio(5)
[Log] MediaSessionManageriOS::updateSessionState(0) setting category = AmbientSound, policy = Default
[Info] MediaElementSession::addBehaviorRestriction(C5E20E3B8BE94F27) adding RequireUserGestureForFullscreen
[Info] MediaElementSession::addBehaviorRestriction(C5E20E3B8BE94F27) adding RequirePageConsentToLoadMedia
[Info] MediaElementSession::addBehaviorRestriction(C5E20E3B8BE94F27) adding RequireUserGestureToAutoplayToExternalDevice
[Info] MediaElementSession::addBehaviorRestriction(C5E20E3B8BE94F27) adding RequireUserGestureToControlControlsManager
[Info] MediaElementSession::addBehaviorRestriction(C5E20E3B8BE94F27) adding RequirePlaybackToControlControlsManager
[Info] MediaElementSession::addBehaviorRestriction(C5E20E3B8BE94F27) adding InvisibleAutoplayNotPermitted
[Info] MediaElementSession::addBehaviorRestriction(C5E20E3B8BE94F27) adding RequireUserGestureForAudioRateChange
[Info] MediaElementSession::addBehaviorRestriction(C5E20E3B8BE94F27) adding RequireUserGestureToShowPlaybackTargetPicker
[Info] MediaElementSession::addBehaviorRestriction(C5E20E3B8BE94F27) adding AutoPreloadingNotPermitted
[Log] MediaElementSession::clientWillBeginAutoplaying(C5E20E3B8BE94F27) state = Idle
[Log] MediaElementSession::setState(C5E20E3B8BE94F27) Autoplaying
[Log] MediaSessionManageriOS::updateSessionState(0) types: AudioCapture(0), Video(0), Audio(1), VideoAudio(0), WebAudio(5)
[Log] MediaSessionManageriOS::updateSessionState(0) setting category = AmbientSound, policy = Default
[Info] HTMLMediaElement::setSrcObject(C5E20E3B8BE94F27) 
[Log] HTMLMediaElement::prepareForLoad(C5E20E3B8BE94F27) gesture = false
[Info] HTMLMediaElement::createMediaPlayer(C5E20E3B8BE94F27) 
[Info] HTMLMediaElement::cancelPendingEventsAndCallbacks(C5E20E3B8BE94F27) 
[Log] HTMLMediaElement::setPlaybackRate(C5E20E3B8BE94F27) 1
[Log] MediaElementSession::clientWillBeginAutoplaying(C5E20E3B8BE94F27) state = Autoplaying
[Log] HTMLMediaElement::setShouldDelayLoadEvent(C5E20E3B8BE94F27) true
[Info] MediaElementSession::removeBehaviorRestriction(C5E20E3B8BE94F27) removed RequirePageConsentToLoadMedia
[Info] RTCPeerConnection::dispatchEvent(912A778D) dispatching 'addstream'
[Log] RealtimeIncomingAudioSource::setMuted(BC77DDBD1D697ED7) false
[Log] RealtimeIncomingAudioSource::start(BC77DDBD1D697ED7) 
[Log] MediaStreamPrivate::trackStarted(6D8380859E058110) (BC77DDBD1D697ED7)
[Log] MediaStreamPrivate::trackMutedChanged(6D8380859E058110) (BC77DDBD1D697ED7) false


[Debug] HTMLMediaElement::dispatchEvent(C5E20E3B8BE94F27) loadstart
[Log] MediaElementSession::playbackPermitted(C5E20E3B8BE94F27) Returning FALSE because a user gesture is required for audio rate change restriction
[Info] MediaElementSession::canShowControlsManager(C5E20E3B8BE94F27) returning FALSE: playback not permitted
[Info] MediaElementSession::canShowControlsManager(C5E20E3B8BE94F27) returning FALSE: audio element is not suitable
[Log] MediaElementSession::playbackPermitted(C5E20E3B8BE94F27) Returning FALSE because a user gesture is required for audio rate change restriction
[Log] HTMLMediaElement::scheduleMediaEngineWasUpdated(C5E20E3B8BE94F27) lambda(), task fired
[Log] HTMLMediaElement::mediaEngineWasUpdated(C5E20E3B8BE94F27)
[Info] MediaElementSession::mediaEngineUpdated(C5E20E3B8BE94F27)
[Log] VideoLayerManagerObjC::setVideoFullscreenFrame(C5E20E3B8BE94F27) 0, 0, 0, 0
[Log] MediaElementSession::playbackPermitted(C5E20E3B8BE94F27) Returning FALSE because a user gesture is required for audio rate change restriction
[Info] MediaElementSession::canShowControlsManager(C5E20E3B8BE94F27) returning FALSE: playback not permitted
[Info] MediaElementSession::canShowControlsManager(C5E20E3B8BE94F27) returning FALSE: audio element is not suitable
[Log] MediaElementSession::playbackPermitted(C5E20E3B8BE94F27) Returning FALSE because a user gesture is required for audio rate change restriction
[Debug] HTMLMediaElement::dispatchEvent(C5E20E3B8BE94F27) progress
[Debug] HTMLMediaElement::dispatchEvent(C5E20E3B8BE94F27) suspend
[Debug] HTMLMediaElement::dispatchEvent(C5E20E3B8BE94F27) durationchange
[Debug] HTMLMediaElement::dispatchEvent(C5E20E3B8BE94F27) loadedmetadata
[Log] BaseAudioContext::createMediaStreamSource(9589982D114A0B5C)
[Log] AudioNode::AudioNode(9589982D114A0002)
[Log] AudioNode::setNodeType(9589982D114A0002) NodeTypeMediaStreamAudioSource
[Info] AudioNode::addOutput(9589982D114A0002) NodeTypeMediaStreamAudioSource
[Log] BaseAudioContext::createGain(9589982D114A0B5C)
[Log] AudioNode::AudioNode(9589982D114A0003)
[Log] AudioParam::AudioParam(9589982D114A0001) name = gain, value = 1, default = 1, min = -3.4028234663852886e+38, max = 3.4028234663852886e+38, units = 0
[Log] AudioNode::setNodeType(9589982D114A0003) NodeTypeGain
[Info] AudioNode::addInput(9589982D114A0003) NodeTypeGain
[Info] AudioNode::addOutput(9589982D114A0003) NodeTypeGain
[Log] AudioNode::setChannelCount(9589982D114A0003) 2
[Log] AudioNode::setChannelCountMode(9589982D114A0003) 0
[Log] AudioNode::setChannelInterpretation(9589982D114A0003) 0
[Debug] AudioParam::setValue(9589982D114A0001) 1
[Log] AudioNode::connect(9589982D114A0002) NodeTypeGain, output = 0, input = 0
[Log] AudioNode::connect(9589982D114A0003) NodeTypeDestination, output = 0, input = 0
[Log] HTMLMediaElement::scheduleUpdateMediaState(C5E20E3B8BE94F27) lambda(), task fired
[Log] MediaElementSession::playbackPermitted(C5E20E3B8BE94F27) Returning FALSE because a user gesture is required for audio rate change restriction
[Info] MediaElementSession::canShowControlsManager(C5E20E3B8BE94F27) returning FALSE: playback not permitted
[Info] MediaElementSession::canShowControlsManager(C5E20E3B8BE94F27) returning FALSE: audio element is not suitable
[Log] MediaElementSession::playbackPermitted(C5E20E3B8BE94F27) Returning FALSE because a user gesture is required for audio rate change restriction
[Log] MediaElementSession::playbackPermitted(C5E20E3B8BE94F27) Returning FALSE because a user gesture is required for audio rate change restriction
[Info] MediaElementSession::canShowControlsManager(C5E20E3B8BE94F27) returning FALSE: playback not permitted
[Info] MediaElementSession::canShowControlsManager(C5E20E3B8BE94F27) returning FALSE: audio element is not suitable
[Log] MediaElementSession::playbackPermitted(C5E20E3B8BE94F27) Returning FALSE because a user gesture is required for audio rate change restriction
[Info] MediaElementSession::canShowControlsManager(C5E20E3B8BE94F27) returning FALSE: audio element is not suitable

[Mikael Nousiainen]

Created attachment 417729 [details]
WebRTC/Media logs from iPhone where the audio is not playing

[Mikael Nousiainen]

The code initiates the WebRTC connection from user gesture (in button click handler).

However, the Audio element and the AudioContext graph are created in WebRTC remote stream callback when we have the remote MediaStream available. But I'm not sure how that could be different, as that's where we get the MediaStream...

[EWS]

Committed r271575: <https://trac.webkit.org/changeset/271575>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 417695 [details].

[youenn fablet]

(In reply to Mikael Nousiainen from comment #15)
> The code initiates the WebRTC connection from user gesture (in button click
> handler).
> 
> However, the Audio element and the AudioContext graph are created in WebRTC
> remote stream callback when we have the remote MediaStream available. But
> I'm not sure how that could be different, as that's where we get the
> MediaStream...

I tried on my iPhone and it seems to work now.