236363 2022-02-09 07:03:40 -0800 (Safari 15 - iOS15): Increased audio latency on streaming via webrtc 2022-06-29 10:30:34 -0700 1 1 1 Unclassified WebKit WebRTC Safari 15 iPhone / iPad iOS 15 RESOLVED FIXED InRadar P2 Normal --- 1 asanand youennf ap bfulgham cdumez eric.carlson ews-watchlist glenn jer.noble philipj sergio sun.shin vfarley webkit-bug-importer youennf oldest_to_newest 1839125 0 asanand 2022-02-09 07:03:40 -0800 We are observing noticeable increase in audio latency in Safari browser iOS 15.4 beta. Our Client side implementation plays webrtc audio/video stream received from server on video element. We have looked in the mediaserverd logs(presented below) and here is our finding default 12:34:35.138370+0530 mediaserverd -CMVAEndptMgr- vaemSetDeviceBufferDuration: duration = 0.085, sampleRate = 48000.000, numPCMSamples = 4095, NearestPowerOf2 = 4096, UseQuietBufferSizeProperty = ‘NO’ When we play the same after enabling the "AudioContext", we are getting following log default 12:42:35.257580+0530 mediaserverd -CMVAEndptMgr- vaemSetDeviceBufferDuration: duration = 0.003, sampleRate = 48000.000, numPCMSamples = 127, NearestPowerOf2 = 128, UseQuietBufferSizeProperty = ‘NO’ the numPCMSamples in case of AudioContext is reducing down to 127 and its having direct impact on our audio latency. 1839785 1 ap 2022-02-10 10:32:30 -0800 Could you please clarify if this is a new issue in Safari 15.4 beta, or if it affects shipping Safari too? 1839811 2 youennf 2022-02-10 10:53:27 -0800 Also, is the application capturing microphone? It is not unexpected that web audio will get you the lowest latency. 1840156 3 asanand 2022-02-10 20:21:25 -0800 (In reply to Alexey Proskuryakov from comment #1) > Could you please clarify if this is a new issue in Safari 15.4 beta, or if > it affects shipping Safari too? It affects shipping safari too 1840158 4 asanand 2022-02-10 20:25:06 -0800 (In reply to youenn fablet from comment #2) > Also, is the application capturing microphone? > It is not unexpected that web audio will get you the lowest latency. The application is not capturing microphone. What should be the difference in latency between playing with and without WebAudio. 1840255 5 youennf 2022-02-11 01:03:17 -0800 (In reply to asanand from comment #4) > (In reply to youenn fablet from comment #2) > > Also, is the application capturing microphone? > > It is not unexpected that web audio will get you the lowest latency. > > The application is not capturing microphone. > > What should be the difference in latency between playing with and without > WebAudio. A small latency is better for reactiveness, less so for battery life and stability. With WebAudio, spec mentions a small audio quantum so we tune the audio renderer accordingly. For regular audio, we do a different set up but I can see some applications wanting to get low latency for incoming WebRTC audio. We could look at implementing https://w3c.github.io/webrtc-extensions/#dom-rtcrtpreceiver-playoutdelay so that your application could provide a hint about this. Can you provide more info on your usecase? 1841209 6 asanand 2022-02-14 05:04:34 -0800 (In reply to youenn fablet from comment #5) > (In reply to asanand from comment #4) > > (In reply to youenn fablet from comment #2) > > > Also, is the application capturing microphone? > > > It is not unexpected that web audio will get you the lowest latency. > > > > The application is not capturing microphone. > > > > What should be the difference in latency between playing with and without > > WebAudio. > > A small latency is better for reactiveness, less so for battery life and > stability. > With WebAudio, spec mentions a small audio quantum so we tune the audio > renderer accordingly. > For regular audio, we do a different set up but I can see some applications > wanting to get low latency for incoming WebRTC audio. > > We could look at implementing > https://w3c.github.io/webrtc-extensions/#dom-rtcrtpreceiver-playoutdelay so > that your application could provide a hint about this. > > Can you provide more info on your usecase? We are having application for cloud gaming where we stream audio/video with optional upstream(mic streams) 1841757 7 webkit-bug-importer 2022-02-15 09:32:08 -0800 <rdar://problem/88969850> 1851766 8 asanand 2022-03-16 06:18:30 -0700 (In reply to youenn fablet from comment #5) > (In reply to asanand from comment #4) > > (In reply to youenn fablet from comment #2) > > > Also, is the application capturing microphone? > > > It is not unexpected that web audio will get you the lowest latency. > > > > The application is not capturing microphone. > > > > What should be the difference in latency between playing with and without > > WebAudio. > > A small latency is better for reactiveness, less so for battery life and > stability. > With WebAudio, spec mentions a small audio quantum so we tune the audio > renderer accordingly. > For regular audio, we do a different set up but I can see some applications > wanting to get low latency for incoming WebRTC audio. > > We could look at implementing > https://w3c.github.io/webrtc-extensions/#dom-rtcrtpreceiver-playoutdelay so > that your application could provide a hint about this. > > Can you provide more info on your usecase? When can we expect a fix for this issue where we could provide a hint as mentioned here > We could look at implementing > https://w3c.github.io/webrtc-extensions/#dom-rtcrtpreceiver-playoutdelay so > that your application could provide a hint about this. 1856184 9 vfarley 2022-03-29 14:20:08 -0700 Youenn, could you please provide an update on the fix for this? 1859196 10 456906 youennf 2022-04-07 04:29:13 -0700 Created attachment 456906 Patch 1859427 11 ews-feeder 2022-04-07 13:59:46 -0700 Committed r292563 (249401@main): <https://commits.webkit.org/249401@main> All reviewed patches have been landed. Closing bug and clearing flags on attachment 456906. 1859579 12 asanand 2022-04-07 21:08:51 -0700 (In reply to youenn fablet from comment #10) > Created attachment 456906 [details] > Patch Which Safari version should have this fix ? 1872166 13 bfulgham 2022-05-26 15:05:23 -0700 This fix shipped with Safari 15.5 (all platforms). 1878259 14 vfarley 2022-06-24 14:02:11 -0700 Still observing an additional ~50ms of latency after this patch. Could we please reevaluate whether the preferred buffer size could be further reduced to address the audio latency, or whether a different approach could be considered to resolve this? 456906 2022-04-07 04:29:13 -0700 2022-04-07 13:59:49 -0700 Patch bug-236363-20220407132911.patch text/plain 5076 youennf U3VidmVyc2lvbiBSZXZpc2lvbjogMjkyNDUwCmRpZmYgLS1naXQgYS9Tb3VyY2UvV2ViQ29yZS9D aGFuZ2VMb2cgYi9Tb3VyY2UvV2ViQ29yZS9DaGFuZ2VMb2cKaW5kZXggYzIyNDViMmJhZDRkNTc3 ZjdkYzgzNDJkMWMzNTI3NGM0YzRhMWE3YS4uZDgxMzM5NWI4MzljNDk4YzA3OTNhZjRlMjUwOTAw OTQ5ZGE5NzcwMSAxMDA2NDQKLS0tIGEvU291cmNlL1dlYkNvcmUvQ2hhbmdlTG9nCisrKyBiL1Nv dXJjZS9XZWJDb3JlL0NoYW5nZUxvZwpAQCAtMSwzICsxLDIyIEBACisyMDIyLTA0LTA3ICBZb3Vl bm4gRmFibGV0ICA8eW91ZW5uQGFwcGxlLmNvbT4KKworICAgICAgICAoU2FmYXJpIDE1IC0gaU9T MTUpOiAgSW5jcmVhc2VkIGF1ZGlvIGxhdGVuY3kgb24gc3RyZWFtaW5nIHZpYSB3ZWJydGMKKyAg ICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTIzNjM2MworICAg ICAgICA8cmRhcjovL3Byb2JsZW0vODg5Njk4NTA+CisKKyAgICAgICAgUmV2aWV3ZWQgYnkgTk9C T0RZIChPT1BTISkuCisKKyAgICAgICAgT24gbWFjT1MgMTIuMywgdGhlIGRlZmF1bHQgcHJlZmVy cmVkIGJ1ZmZlciBzaXplIGlzIHJvdWdobHkgMTAwIG1zLgorICAgICAgICBUaGlzIGlzIG9rIGZv ciByZWd1bGFyIGF1ZGlvIHBsYXliYWNrIGJ1dCBpcyBub3QgZGVzaXJhYmxlIHdoZW4gcGxheWlu ZyByZWFsdGltZSBhdWRpby4KKyAgICAgICAgVG8gcmVkdWNlIHRoZSBwZXJjZWl2ZWQgbGF0ZW5j eSwgd2Ugbm93IHJlZHVjZSB0aGUgcHJlZmVycmVkIGJ1ZmZlciBzaXplIHRvIDIwbXMKKyAgICAg ICAgd2hlbmV2ZXIgcGxheWluZyBhbiBhdWRpbyBNZWRpYVN0cmVhbVRyYWNrLCBzaW1pbGFybHkg dG8gd2hlbiBjYXB0dXJpbmcgYXVkaW8uCisKKyAgICAgICAgTWFudWFsbHkgdGVzdGVkLgorCisg ICAgICAgICogcGxhdGZvcm0vYXVkaW8vUGxhdGZvcm1NZWRpYVNlc3Npb24uY3BwOgorICAgICAg ICAqIHBsYXRmb3JtL2F1ZGlvL1BsYXRmb3JtTWVkaWFTZXNzaW9uLmg6CisgICAgICAgICogcGxh dGZvcm0vYXVkaW8vY29jb2EvTWVkaWFTZXNzaW9uTWFuYWdlckNvY29hLm1tOgorCiAyMDIyLTA0 LTA2ICBZb3Vlbm4gRmFibGV0ICA8eW91ZW5uQGFwcGxlLmNvbT4KIAogICAgICAgICBJbXBsZW1l bnQgU2VydmljZVdvcmtlcldpbmRvd0NsaWVudC5uYXZpZ2F0ZQpkaWZmIC0tZ2l0IGEvU291cmNl L1dlYkNvcmUvcGxhdGZvcm0vYXVkaW8vUGxhdGZvcm1NZWRpYVNlc3Npb24uY3BwIGIvU291cmNl L1dlYkNvcmUvcGxhdGZvcm0vYXVkaW8vUGxhdGZvcm1NZWRpYVNlc3Npb24uY3BwCmluZGV4IGM2 ZjdjY2FkZjdkZWM0Y2JjODQxYmY5ZmVhODY0YzNhZGFhN2I4NTguLmFmY2VjOTFiZDFmMmIwYjBi ZTZmMjBhMDM5YWJiMTVhYWFkYjE0NGQgMTAwNjQ0Ci0tLSBhL1NvdXJjZS9XZWJDb3JlL3BsYXRm b3JtL2F1ZGlvL1BsYXRmb3JtTWVkaWFTZXNzaW9uLmNwcAorKysgYi9Tb3VyY2UvV2ViQ29yZS9w bGF0Zm9ybS9hdWRpby9QbGF0Zm9ybU1lZGlhU2Vzc2lvbi5jcHAKQEAgLTM2NSw2ICszNjUsMTEg QEAgYm9vbCBQbGF0Zm9ybU1lZGlhU2Vzc2lvbjo6Y2FuUHJvZHVjZUF1ZGlvKCkgY29uc3QKICAg ICByZXR1cm4gbV9jbGllbnQuY2FuUHJvZHVjZUF1ZGlvKCk7CiB9CiAKK2Jvb2wgUGxhdGZvcm1N ZWRpYVNlc3Npb246Omhhc01lZGlhU3RyZWFtU291cmNlKCkgY29uc3QKK3sKKyAgICByZXR1cm4g bV9jbGllbnQuaGFzTWVkaWFTdHJlYW1Tb3VyY2UoKTsKK30KKwogdm9pZCBQbGF0Zm9ybU1lZGlh U2Vzc2lvbjo6Y2FuUHJvZHVjZUF1ZGlvQ2hhbmdlZCgpCiB7CiAgICAgUGxhdGZvcm1NZWRpYVNl c3Npb25NYW5hZ2VyOjpzaGFyZWRNYW5hZ2VyKCkuc2Vzc2lvbkNhblByb2R1Y2VBdWRpb0NoYW5n ZWQoKTsKZGlmZiAtLWdpdCBhL1NvdXJjZS9XZWJDb3JlL3BsYXRmb3JtL2F1ZGlvL1BsYXRmb3Jt TWVkaWFTZXNzaW9uLmggYi9Tb3VyY2UvV2ViQ29yZS9wbGF0Zm9ybS9hdWRpby9QbGF0Zm9ybU1l ZGlhU2Vzc2lvbi5oCmluZGV4IDllYTY3NDUzMmQwMGQ2NjY5OGZmN2ExZjc3YjQ4NzBkNWQ2YzBm NDMuLjViNTI5ZTAxYWMxN2EwNjU4M2Q0YTk2NWQ4MTZlY2UyMTMwZjdjNTcgMTAwNjQ0Ci0tLSBh L1NvdXJjZS9XZWJDb3JlL3BsYXRmb3JtL2F1ZGlvL1BsYXRmb3JtTWVkaWFTZXNzaW9uLmgKKysr IGIvU291cmNlL1dlYkNvcmUvcGxhdGZvcm0vYXVkaW8vUGxhdGZvcm1NZWRpYVNlc3Npb24uaApA QCAtMTc5LDYgKzE3OSw3IEBAIHB1YmxpYzoKIAogICAgIGJvb2wgYWN0aXZlQXVkaW9TZXNzaW9u UmVxdWlyZWQoKSBjb25zdDsKICAgICBib29sIGNhblByb2R1Y2VBdWRpbygpIGNvbnN0OworICAg IGJvb2wgaGFzTWVkaWFTdHJlYW1Tb3VyY2UoKSBjb25zdDsKICAgICB2b2lkIGNhblByb2R1Y2VB dWRpb0NoYW5nZWQoKTsKIAogICAgIHZpcnR1YWwgdm9pZCByZXNldFBsYXliYWNrU2Vzc2lvblN0 YXRlKCkgeyB9CmRpZmYgLS1naXQgYS9Tb3VyY2UvV2ViQ29yZS9wbGF0Zm9ybS9hdWRpby9jb2Nv YS9NZWRpYVNlc3Npb25NYW5hZ2VyQ29jb2EubW0gYi9Tb3VyY2UvV2ViQ29yZS9wbGF0Zm9ybS9h dWRpby9jb2NvYS9NZWRpYVNlc3Npb25NYW5hZ2VyQ29jb2EubW0KaW5kZXggOTZjZGZhYTM2MjFj NWM5NjJkYTBlMDUzMDJkNGVhOWQ5Y2Q3MDg3Yy4uMWYyMjA5MGQyNDcyYzE0ZmYyMzJkNTQxNmQ5 MWNjZTNiOWVkYjM2MSAxMDA2NDQKLS0tIGEvU291cmNlL1dlYkNvcmUvcGxhdGZvcm0vYXVkaW8v Y29jb2EvTWVkaWFTZXNzaW9uTWFuYWdlckNvY29hLm1tCisrKyBiL1NvdXJjZS9XZWJDb3JlL3Bs YXRmb3JtL2F1ZGlvL2NvY29hL01lZGlhU2Vzc2lvbk1hbmFnZXJDb2NvYS5tbQpAQCAtMTEzLDYg KzExMyw3IEBAIHZvaWQgTWVkaWFTZXNzaW9uTWFuYWdlckNvY29hOjp1cGRhdGVTZXNzaW9uU3Rh dGUoKQogICAgIGludCB2aWRlb0F1ZGlvQ291bnQgPSAwOwogICAgIGludCBhdWRpb0NvdW50ID0g MDsKICAgICBpbnQgd2ViQXVkaW9Db3VudCA9IDA7CisgICAgaW50IGF1ZGlvTWVkaWFTdHJlYW1U cmFja0NvdW50ID0gMDsKICAgICBpbnQgY2FwdHVyZUNvdW50ID0gY291bnRBY3RpdmVBdWRpb0Nh cHR1cmVTb3VyY2VzKCk7CiAgICAgYm9vbCBoYXNBdWRpYmxlQXVkaW9PclZpZGVvTWVkaWFUeXBl ID0gZmFsc2U7CiAgICAgYm9vbCBpc1BsYXlpbmdBdWRpbyA9IGZhbHNlOwpAQCAtMTI2LDkgKzEy NywxMyBAQCB2b2lkIE1lZGlhU2Vzc2lvbk1hbmFnZXJDb2NvYTo6dXBkYXRlU2Vzc2lvblN0YXRl KCkKICAgICAgICAgICAgIGJyZWFrOwogICAgICAgICBjYXNlIFBsYXRmb3JtTWVkaWFTZXNzaW9u OjpNZWRpYVR5cGU6OlZpZGVvQXVkaW86CiAgICAgICAgICAgICArK3ZpZGVvQXVkaW9Db3VudDsK KyAgICAgICAgICAgIGlmIChzZXNzaW9uLmNhblByb2R1Y2VBdWRpbygpICYmIHNlc3Npb24uaGFz TWVkaWFTdHJlYW1Tb3VyY2UoKSkKKyAgICAgICAgICAgICAgICArK2F1ZGlvTWVkaWFTdHJlYW1U cmFja0NvdW50OwogICAgICAgICAgICAgYnJlYWs7CiAgICAgICAgIGNhc2UgUGxhdGZvcm1NZWRp YVNlc3Npb246Ok1lZGlhVHlwZTo6QXVkaW86CiAgICAgICAgICAgICArK2F1ZGlvQ291bnQ7Cisg ICAgICAgICAgICBpZiAoc2Vzc2lvbi5jYW5Qcm9kdWNlQXVkaW8oKSAmJiBzZXNzaW9uLmhhc01l ZGlhU3RyZWFtU291cmNlKCkpCisgICAgICAgICAgICAgICAgKythdWRpb01lZGlhU3RyZWFtVHJh Y2tDb3VudDsKICAgICAgICAgICAgIGJyZWFrOwogICAgICAgICBjYXNlIFBsYXRmb3JtTWVkaWFT ZXNzaW9uOjpNZWRpYVR5cGU6OldlYkF1ZGlvOgogICAgICAgICAgICAgaWYgKHNlc3Npb24uY2Fu UHJvZHVjZUF1ZGlvKCkpIHsKQEAgLTE1Miw2ICsxNTcsNyBAQCB2b2lkIE1lZGlhU2Vzc2lvbk1h bmFnZXJDb2NvYTo6dXBkYXRlU2Vzc2lvblN0YXRlKCkKIAogICAgIEFMV0FZU19MT0coTE9HSURF TlRJRklFUiwgInR5cGVzOiAiCiAgICAgICAgICJBdWRpb0NhcHR1cmUoIiwgY2FwdHVyZUNvdW50 LCAiKSwgIgorICAgICAgICAiQXVkaW9UcmFjaygiLCBhdWRpb01lZGlhU3RyZWFtVHJhY2tDb3Vu dCwgIiksICIKICAgICAgICAgIlZpZGVvKCIsIHZpZGVvQ291bnQsICIpLCAiCiAgICAgICAgICJB dWRpbygiLCBhdWRpb0NvdW50LCAiKSwgIgogICAgICAgICAiVmlkZW9BdWRpbygiLCB2aWRlb0F1 ZGlvQ291bnQsICIpLCAiCkBAIC0xNjAsOCArMTY2LDggQEAgdm9pZCBNZWRpYVNlc3Npb25NYW5h Z2VyQ29jb2E6OnVwZGF0ZVNlc3Npb25TdGF0ZSgpCiAgICAgc2l6ZV90IGJ1ZmZlclNpemUgPSBt X2RlZmF1bHRCdWZmZXJTaXplOwogICAgIGlmICh3ZWJBdWRpb0NvdW50KQogICAgICAgICBidWZm ZXJTaXplID0gQXVkaW9VdGlsaXRpZXM6OnJlbmRlclF1YW50dW1TaXplOwotICAgIGVsc2UgaWYg KGNhcHR1cmVDb3VudCkgewotICAgICAgICAvLyBJbiBjYXNlIG9mIGF1ZGlvIGNhcHR1cmUsIHdl IHdhbnQgdG8gZ3JhYiAyMCBtcyBjaHVua3MgdG8gbGltaXQgdGhlIGxhdGVuY3kgc28gdGhhdCBp dCBpcyBub3Qgbm90aWNlYWJsZSBieSB1c2VycworICAgIGVsc2UgaWYgKGNhcHR1cmVDb3VudCB8 fCBhdWRpb01lZGlhU3RyZWFtVHJhY2tDb3VudCkgeworICAgICAgICAvLyBJbiBjYXNlIG9mIGF1 ZGlvIGNhcHR1cmUgb3IgYXVkaW8gTWVkaWFTdHJlYW1UcmFjayBwbGF5aW5nLCB3ZSB3YW50IHRv IGdyYWIgMjAgbXMgY2h1bmtzIHRvIGxpbWl0IHRoZSBsYXRlbmN5IHNvIHRoYXQgaXQgaXMgbm90 IG5vdGljZWFibGUgYnkgdXNlcnMKICAgICAgICAgLy8gd2hpbGUgaGF2aW5nIGEgbGFyZ2UgZW5v dWdoIGJ1ZmZlciBzbyB0aGF0IHRoZSBhdWRpbyByZW5kZXJpbmcgcmVtYWlucyBzdGFibGUsIGhl bmNlIGEgY29tcHV0YXRpb24gYmFzZWQgb24gc2FtcGxlIHJhdGUuCiAgICAgICAgIGJ1ZmZlclNp emUgPSBBdWRpb1Nlc3Npb246OnNoYXJlZFNlc3Npb24oKS5zYW1wbGVSYXRlKCkgLyA1MDsKICAg ICB9IGVsc2UgaWYgKG1fc3VwcG9ydGVkQXVkaW9IYXJkd2FyZUJ1ZmZlclNpemVzICYmIERlcHJl Y2F0ZWRHbG9iYWxTZXR0aW5nczo6bG93UG93ZXJWaWRlb0F1ZGlvQnVmZmVyU2l6ZUVuYWJsZWQo KSkK