(Safari 15 - iOS15): Increased audio latency on streaming via webrtc

• Component: WebRTC
• Status: RESOLVED FIXED

We are observing noticeable increase in audio latency in Safari browser iOS 15.4 beta.

Our Client side implementation plays webrtc audio/video stream received from server on video element.


We have looked in the mediaserverd logs(presented below) and here is our finding



default	12:34:35.138370+0530	mediaserverd	-CMVAEndptMgr- vaemSetDeviceBufferDuration: duration = 0.085, sampleRate = 48000.000, numPCMSamples = 4095, NearestPowerOf2 = 4096, UseQuietBufferSizeProperty = ‘NO’


When we play the same after enabling the "AudioContext", we are getting following log
  
default	12:42:35.257580+0530	mediaserverd	-CMVAEndptMgr- vaemSetDeviceBufferDuration: duration = 0.003, sampleRate = 48000.000, numPCMSamples = 127, NearestPowerOf2 = 128, UseQuietBufferSizeProperty = ‘NO’

the numPCMSamples in case of AudioContext is reducing down to 127 and its having direct impact on our audio latency.

[Alexey Proskuryakov]

Could you please clarify if this is a new issue in Safari 15.4 beta, or if it affects shipping Safari too?

[youenn fablet]

Also, is the application capturing microphone?
It is not unexpected that web audio will get you the lowest latency.

(In reply to Alexey Proskuryakov from comment #1)
> Could you please clarify if this is a new issue in Safari 15.4 beta, or if
> it affects shipping Safari too?

It affects shipping safari too

(In reply to youenn fablet from comment #2)
> Also, is the application capturing microphone?
> It is not unexpected that web audio will get you the lowest latency.

The application is not capturing microphone.

What should be the difference in latency between playing with and without WebAudio.

[youenn fablet]

(In reply to asanand from comment #4)
> (In reply to youenn fablet from comment #2)
> > Also, is the application capturing microphone?
> > It is not unexpected that web audio will get you the lowest latency.
> 
> The application is not capturing microphone.
> 
> What should be the difference in latency between playing with and without
> WebAudio.

A small latency is better for reactiveness, less so for battery life and stability.
With WebAudio, spec mentions a small audio quantum so we tune the audio renderer accordingly.
For regular audio, we do a different set up but I can see some applications wanting to get low latency for incoming WebRTC audio.

We could look at implementing https://w3c.github.io/webrtc-extensions/#dom-rtcrtpreceiver-playoutdelay so that your application could provide a hint about this.

Can you provide more info on your usecase?

(In reply to youenn fablet from comment #5)
> (In reply to asanand from comment #4)
> > (In reply to youenn fablet from comment #2)
> > > Also, is the application capturing microphone?
> > > It is not unexpected that web audio will get you the lowest latency.
> > 
> > The application is not capturing microphone.
> > 
> > What should be the difference in latency between playing with and without
> > WebAudio.
> 
> A small latency is better for reactiveness, less so for battery life and
> stability.
> With WebAudio, spec mentions a small audio quantum so we tune the audio
> renderer accordingly.
> For regular audio, we do a different set up but I can see some applications
> wanting to get low latency for incoming WebRTC audio.
> 
> We could look at implementing
> https://w3c.github.io/webrtc-extensions/#dom-rtcrtpreceiver-playoutdelay so
> that your application could provide a hint about this.
> 
> Can you provide more info on your usecase?
We are having application for cloud gaming where we stream audio/video with optional upstream(mic streams)

[Radar WebKit Bug Importer]

<rdar://problem/88969850>

(In reply to youenn fablet from comment #5)
> (In reply to asanand from comment #4)
> > (In reply to youenn fablet from comment #2)
> > > Also, is the application capturing microphone?
> > > It is not unexpected that web audio will get you the lowest latency.
> > 
> > The application is not capturing microphone.
> > 
> > What should be the difference in latency between playing with and without
> > WebAudio.
> 
> A small latency is better for reactiveness, less so for battery life and
> stability.
> With WebAudio, spec mentions a small audio quantum so we tune the audio
> renderer accordingly.
> For regular audio, we do a different set up but I can see some applications
> wanting to get low latency for incoming WebRTC audio.
> 
> We could look at implementing
> https://w3c.github.io/webrtc-extensions/#dom-rtcrtpreceiver-playoutdelay so
> that your application could provide a hint about this.
> 
> Can you provide more info on your usecase?

When can we expect a fix for this issue where we could provide a hint as mentioned here
> We could look at implementing
> https://w3c.github.io/webrtc-extensions/#dom-rtcrtpreceiver-playoutdelay so
> that your application could provide a hint about this.

[Virginia Farley]

Youenn, could you please provide an update on the fix for this?

[youenn fablet]

Created attachment 456906 [details]
Patch

[EWS]

Committed r292563 (249401@main): <https://commits.webkit.org/249401@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 456906 [details].

(In reply to youenn fablet from comment #10)
> Created attachment 456906 [details]
> Patch

Which Safari version should have this fix ?

[Brent Fulgham]

This fix shipped with Safari 15.5 (all platforms).

[Virginia Farley]

Still observing an additional ~50ms of latency after this patch.  Could we please reevaluate whether the preferred buffer size could be further reduced to address the audio latency, or whether a different approach could be considered to resolve this?