196214 2019-03-25 14:07:46 -0700 getUserMedia framerate unusable under low light in iOS 12.2 2019-06-02 09:31:35 -0700 1 1 1 Unclassified WebKit WebRTC Safari 12 iPhone / iPad Unspecified RESOLVED FIXED InRadar P2 Major --- 1 daginge youennf commit-queue eric.carlson ggaren ibc jhenderson2177 jonlee kevinday michael simon.fraser webkit-bug-importer youennf oldest_to_newest 1521003 0 daginge 2019-03-25 14:07:46 -0700 Since upgrading to iOS 12.2, I tested the frame rate of getUserMedia by going to https://webrtc.github.io/samples/src/content/getusermedia/gum/. The framerate is horrible to the point of unusable, as you can no longer see the mouth move when talking, and moving the camera makes an interesting long-tail effect of the previous image. I've attached videos showing the getUserMedia sample page with the problem, and the native camera app, which shows no sign of this issue. getUserMedia on WebRTC samples https://photos.app.goo.gl/1RnJQ7SFhG6rqSBQ6 Actual flipped camera framerate in Camera app https://photos.app.goo.gl/1CmBPr1ho3wyNP579 I have tried killing the Safari app, fully rebooting my phone, and anything else I can think of, and the problem remains. What's worse, this is not only a local rendering issue, it's also an issue when sending the video to the other side (using H264 too). Audio is not affected luckily. As it stands now, using getUserMedia for anything except taking perfectly stationary images will not work, and it is completely breaking for any video experience in the browser. My phone's about page shows: Software version: 12.2 (16E227) Model name: iPhone X Model version: MQAD2QN/A (A1901) Any help would be appreciated in working around this problem, as this will completely break Confrere for patients/physicians when they upgrade to iOS 12.2 1521019 1 daginge 2019-03-25 14:22:22 -0700 Able to replicate this on my iPad Pro as well. Software version: 16E227 Model name: iPad Pro 10.5 inch Model number: MQDW2KN/A (A1701) 1521023 2 daginge 2019-03-25 14:25:33 -0700 Interestingly, the problem fixes itself (as does the resolution (??)) if I go to the home screen and open Safari again. To repro: 1. Go to https://webrtc.github.io/samples/src/content/getusermedia/gum/ 2. Click "open camera" 3. Observe that frame rate is slow 4. Go to the home screen by swiping up 5.Wait at least 2 seconds 6. Open Safari again 7. Observe that the resolution has changed (black borders around the video frame), and the video is now fast again (as fast as native camera app) 1521032 3 daginge 2019-03-25 14:33:42 -0700 Problem also persists with all standard resolutions tried here: https://webrtc.github.io/samples/src/content/getusermedia/resolution/. Workaround by going to the home screen and back seems to fix the issue (and changes the aspect ration) for all resolutions. 1521061 4 webkit-bug-importer 2019-03-25 15:21:28 -0700 <rdar://problem/49232193> 1521073 5 ibc 2019-03-25 15:47:20 -0700 I've seen this issue in Macbook Pro OSX latest versions and Safari Tech latest versions. "Sometimes" it does not happen. No idea what it depends on, but most times it happens. 1521235 6 daginge 2019-03-26 01:44:24 -0700 As of this morning I'm unable to replicate the issue as seen in the videos in either of my devices. None of the devices have been shut off during the night. https://photos.app.goo.gl/4hmARduvZd9jrFND9 Aspect ratio still changes when going to the home screen and back though, but at least the horrible lag isn't there anymore. 1521691 7 daginge 2019-03-27 08:45:36 -0700 I FINALLY FIGURED IT OUT! iOS 12.2 behaves badly in _low light_ scenarios. That's why it didn't work in the evening with poor lighting conditions, and worked well in daylight when I tested it again. https://photos.app.goo.gl/akjHTfQ6oAZtfwCW8 See attached video showing extremely poor performance in low light, then going to home screen and back magically fixes the problem (and changes aspect ratio). Moving to a better lit scenario also fixes the problem. This was not a problem in previous versions. Do I get a cake now? :D 1522018 8 youennf 2019-03-28 01:25:02 -0700 > Do I get a cake now? :D Thanks! I was also able to repro on iOS and somehow on MacOS as well. 1522020 9 youennf 2019-03-28 02:08:06 -0700 The call to setActiveVideoMinFrameDuration/setActiveVideoMaxFrameDuration in AVVideoCaptureSource::setSizeAndFrameRateWithPreset might actually trigger this behavior. 1522183 10 jhenderson2177 2019-03-28 12:51:11 -0700 I'm able to reproduce this on my iPhone X (MQCN2LL/A A1865) running iOS 12.2 (16E227) if I don't have perfect lighting. Another workaround is to request "frameRate: 60" in the constraints and the problem goes away, regardless of the lighting quality. Lower frameRate values, or not specifying one, and the problem still happens. 1533987 11 369305 youennf 2019-05-07 11:21:50 -0700 Created attachment 369305 Patch 1534033 12 369305 ggaren 2019-05-07 12:19:05 -0700 Comment on attachment 369305 Patch r=me 1534104 13 369305 commit-queue 2019-05-07 14:04:15 -0700 Comment on attachment 369305 Patch Clearing flags on attachment: 369305 Committed r245032: <https://trac.webkit.org/changeset/245032> 1534105 14 commit-queue 2019-05-07 14:04:17 -0700 All reviewed patches have been landed. Closing bug. 1534107 15 369305 simon.fraser 2019-05-07 14:06:46 -0700 Comment on attachment 369305 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=369305&action=review > Source/WebCore/platform/mediastream/mac/AVVideoCaptureSource.mm:332 > + if (requestedFrameRate < frameRateRange.minFrameRate) > + requestedFrameRate = frameRateRange.minFrameRate; > + else if (requestedFrameRate > frameRateRange.maxFrameRate) > + requestedFrameRate = frameRateRange.maxFrameRate; Use clampTo()? 1534423 16 youennf 2019-05-08 12:24:41 -0700 *** Bug 197703 has been marked as a duplicate of this bug. *** 1534425 17 youennf 2019-05-08 12:26:25 -0700 (In reply to Simon Fraser (smfr) from comment #15) > Comment on attachment 369305 [details] > Patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=369305&action=review > > > Source/WebCore/platform/mediastream/mac/AVVideoCaptureSource.mm:332 > > + if (requestedFrameRate < frameRateRange.minFrameRate) > > + requestedFrameRate = frameRateRange.minFrameRate; > > + else if (requestedFrameRate > frameRateRange.maxFrameRate) > > + requestedFrameRate = frameRateRange.maxFrameRate; > > Use clampTo()? Makes sense, will fix it in bug 197704. 1534441 18 michael 2019-05-08 12:58:53 -0700 (In reply to JH from comment #10) > I'm able to reproduce this on my iPhone X (MQCN2LL/A A1865) running iOS 12.2 > (16E227) if I don't have perfect lighting. > > Another workaround is to request "frameRate: 60" in the constraints and the > problem goes away, regardless of the lighting quality. Lower frameRate > values, or not specifying one, and the problem still happens. Although the frame rate problem seems to go away with this workaround, the black bars seem to still be an issue. I wonder if the patches fix the black bars too? 1534453 19 youennf 2019-05-08 13:22:50 -0700 (In reply to micsun-al from comment #18) > (In reply to JH from comment #10) > > I'm able to reproduce this on my iPhone X (MQCN2LL/A A1865) running iOS 12.2 > > (16E227) if I don't have perfect lighting. > > > > Another workaround is to request "frameRate: 60" in the constraints and the > > problem goes away, regardless of the lighting quality. Lower frameRate > > values, or not specifying one, and the problem still happens. > > Although the frame rate problem seems to go away with this workaround, the > black bars seem to still be an issue. I wonder if the patches fix the black > bars too? That is a separate issue, I filed https://bugs.webkit.org/show_bug.cgi?id=197707. 1541084 20 daginge 2019-06-02 06:53:07 -0700 This issue is not fixed for me on iOS 12.3.1 using Safari 12.1.1. I can still reliably reproduce this in low light on https://webrtc.github.io/samples/src/content/getusermedia/gum/ on my iPhone X. Let me know if you need additional debugging information. 1541085 21 daginge 2019-06-02 06:58:49 -0700 See difference between native camera and webrtc: native: https://share.icloud.com/photos/0oKCLxUqdxlPfFSjxDg7eGwPQ webrtc: https://share.icloud.com/photos/0hYZ4piriFK45trA0hk_gPJ9g These were recorded right after one another in the exact same lighting conditions. 1541105 22 simon.fraser 2019-06-02 09:31:35 -0700 12.3.1 doesn't have the fix for this bug. Fixing a bug in bugzilla does not correspond with when it's available in an OS update. 369305 2019-05-07 11:21:50 -0700 2019-05-07 14:04:15 -0700 Patch bug-196214-20190507112149.patch text/plain 4073 youennf U3VidmVyc2lvbiBSZXZpc2lvbjogMjQ0OTYxCmRpZmYgLS1naXQgYS9Tb3VyY2UvV2ViQ29yZS9D aGFuZ2VMb2cgYi9Tb3VyY2UvV2ViQ29yZS9DaGFuZ2VMb2cKaW5kZXggMTYwZDU1ODFlYmIwNjky ZjE3Yjg1NjljZDZiNDhjZjg0NDg3NGEyMS4uM2RhZDVlZGZjYWE5YmRhNzdlZmU3NWQzZjE4MzQw MDFkY2ZmODM4NiAxMDA2NDQKLS0tIGEvU291cmNlL1dlYkNvcmUvQ2hhbmdlTG9nCisrKyBiL1Nv dXJjZS9XZWJDb3JlL0NoYW5nZUxvZwpAQCAtMSwzICsxLDIzIEBACisyMDE5LTA1LTA3ICBZb3Vl bm4gRmFibGV0ICA8eW91ZW5uQGFwcGxlLmNvbT4KKworICAgICAgICBnZXRVc2VyTWVkaWEgZnJh bWVyYXRlIHVudXNhYmxlIHVuZGVyIGxvdyBsaWdodCBpbiBpT1MgMTIuMgorICAgICAgICBodHRw czovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9MTk2MjE0CisgICAgICAgIDxyZGFy Oi8vcHJvYmxlbS80OTIzMjE5Mz4KKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMh KS4KKworICAgICAgICBXaGVuIHNldHRpbmcgdGhlIGZyYW1lIHJhdGUsIHNldCBpdCB0byB0aGUg ZXhhY3QgdmFsdWUgaW5zdGVhZCBvZiBhIHJhbmdlLgorICAgICAgICBPdGhlcndpc2UsIHRoZSBj YXB0dXJlIGRldmljZSBtaWdodCB1c2UgdGhlIGxvd2VzdCBmcmFtZSByYXRlIGFjY29yZGluZyB0 aGUgbGlnaHQgY29uZGl0aW9ucworICAgICAgICBmb3IgYmVzdCBwaWN0dXJlIHF1YWxpdHkgd2hp Y2ggaXMgbm90IHdoYXQgaXMgZXhwZWN0ZWQgYnkgbW9zdCB3ZWIgcGFnZXMuCisKKyAgICAgICAg TW92ZSBmcmFtZSByYXRlIHJhbmdlIGNvbXB1dGF0aW9uIHRvIGNsb3NlciB3aGVyZSBhY3R1YWxs eSB1c2VkLgorICAgICAgICBTaW5jZSBmcmFtZSByYXRlIG1hdGNoaW5nIGlzIGZ1enp5LCBhZGQg c29tZSBjaGVja3MgaW4gY2FzZSB0aGUgZXhwZWN0ZWQgZnJhbWUgcmF0ZSBpcyBzbGlnaHRseSBv dXQgb2YgbWluL21heCByYW5nZS4KKworICAgICAgICBNYW51YWxseSB0ZXN0ZWQgb24gYSByZWFs IGRldmljZS4KKworICAgICAgICAqIHBsYXRmb3JtL21lZGlhc3RyZWFtL21hYy9BVlZpZGVvQ2Fw dHVyZVNvdXJjZS5tbToKKyAgICAgICAgKFdlYkNvcmU6OkFWVmlkZW9DYXB0dXJlU291cmNlOjpz ZXRTaXplQW5kRnJhbWVSYXRlV2l0aFByZXNldCk6CisKIDIwMTktMDUtMDYgIFlvdWVubiBGYWJs ZXQgIDx5b3Vlbm5AYXBwbGUuY29tPgogCiAgICAgICAgIFZpZGVvIHN0cmVhbSBmcmVlemUgb24g ZnJvbnQgY2FtZXJhIG9yaWVudGF0aW9uIGNoYW5naW5nCmRpZmYgLS1naXQgYS9Tb3VyY2UvV2Vi Q29yZS9wbGF0Zm9ybS9tZWRpYXN0cmVhbS9tYWMvQVZWaWRlb0NhcHR1cmVTb3VyY2UubW0gYi9T b3VyY2UvV2ViQ29yZS9wbGF0Zm9ybS9tZWRpYXN0cmVhbS9tYWMvQVZWaWRlb0NhcHR1cmVTb3Vy Y2UubW0KaW5kZXggODUxNDRhZGY0NTkwYTZkOGM0MmZlMDViODEzZmM1MWZhOGIzY2IzYi4uMTA2 NzM0ZmZlOTA1YWIyOWFjODQ0YjgzNTMwN2UxNDUwMDhkMzlkMyAxMDA2NDQKLS0tIGEvU291cmNl L1dlYkNvcmUvcGxhdGZvcm0vbWVkaWFzdHJlYW0vbWFjL0FWVmlkZW9DYXB0dXJlU291cmNlLm1t CisrKyBiL1NvdXJjZS9XZWJDb3JlL3BsYXRmb3JtL21lZGlhc3RyZWFtL21hYy9BVlZpZGVvQ2Fw dHVyZVNvdXJjZS5tbQpAQCAtMjk5LDExICsyOTksNiBAQCB2b2lkIEFWVmlkZW9DYXB0dXJlU291 cmNlOjpzZXRTaXplQW5kRnJhbWVSYXRlV2l0aFByZXNldChJbnRTaXplIHJlcXVlc3RlZFNpemUs CiAgICAgbV9wZW5kaW5nUHJlc2V0ID0gbnVsbHB0cjsKICAgICBtX3BlbmRpbmdGcmFtZVJhdGUg PSAwOwogCi0gICAgYXV0byogZnJhbWVSYXRlUmFuZ2UgPSBmcmFtZUR1cmF0aW9uRm9yRnJhbWVS YXRlKHJlcXVlc3RlZEZyYW1lUmF0ZSk7Ci0gICAgQVNTRVJUKGZyYW1lUmF0ZVJhbmdlKTsKLSAg ICBpZiAoIWZyYW1lUmF0ZVJhbmdlKQotICAgICAgICByZXR1cm47Ci0KICAgICBpZiAoIWF2UHJl c2V0KQogICAgICAgICByZXR1cm47CiAKQEAgLTMxNiwxMSArMzExLDYgQEAgdm9pZCBBVlZpZGVv Q2FwdHVyZVNvdXJjZTo6c2V0U2l6ZUFuZEZyYW1lUmF0ZVdpdGhQcmVzZXQoSW50U2l6ZSByZXF1 ZXN0ZWRTaXplLAogICAgICAgICAgICAgaWYgKCFtX2N1cnJlbnRQcmVzZXQgfHwgIVttX2N1cnJl bnRQcmVzZXQtPmZvcm1hdC5nZXQoKSBpc0VxdWFsOmF2UHJlc2V0LT5mb3JtYXQuZ2V0KCldKSB7 CiAgICAgICAgICAgICAgICAgW2RldmljZSgpIHNldEFjdGl2ZUZvcm1hdDphdlByZXNldC0+Zm9y bWF0LmdldCgpXTsKIAotICAgICAgICAgICAgICAgIGZyYW1lUmF0ZVJhbmdlID0gZnJhbWVEdXJh dGlvbkZvckZyYW1lUmF0ZShyZXF1ZXN0ZWRGcmFtZVJhdGUpOwotICAgICAgICAgICAgICAgIEFT U0VSVChmcmFtZVJhdGVSYW5nZSk7Ci0gICAgICAgICAgICAgICAgaWYgKCFmcmFtZVJhdGVSYW5n ZSkKLSAgICAgICAgICAgICAgICAgICAgcmV0dXJuOwotCiAjaWYgUExBVEZPUk0oTUFDKQogICAg ICAgICAgICAgICAgIGF1dG8gc2V0dGluZ3NEaWN0aW9uYXJ5ID0gQHsKICAgICAgICAgICAgICAg ICAgICAgKF9fYnJpZGdlIE5TU3RyaW5nICopa0NWUGl4ZWxCdWZmZXJQaXhlbEZvcm1hdFR5cGVL ZXk6IEAoYXZWaWRlb0NhcHR1cmVQaXhlbEJ1ZmZlckZvcm1hdCgpKSwKQEAgLTMzMSwxMCArMzIx LDIwIEBAIHZvaWQgQVZWaWRlb0NhcHR1cmVTb3VyY2U6OnNldFNpemVBbmRGcmFtZVJhdGVXaXRo UHJlc2V0KEludFNpemUgcmVxdWVzdGVkU2l6ZSwKICAgICAgICAgICAgICAgICBbbV92aWRlb091 dHB1dCBzZXRWaWRlb1NldHRpbmdzOnNldHRpbmdzRGljdGlvbmFyeV07CiAjZW5kaWYKICAgICAg ICAgICAgIH0KKyAgICAgICAgICAgIGF1dG8qIGZyYW1lUmF0ZVJhbmdlID0gZnJhbWVEdXJhdGlv bkZvckZyYW1lUmF0ZShyZXF1ZXN0ZWRGcmFtZVJhdGUpOworICAgICAgICAgICAgQVNTRVJUKGZy YW1lUmF0ZVJhbmdlKTsKKyAgICAgICAgICAgIGlmICghZnJhbWVSYXRlUmFuZ2UpCisgICAgICAg ICAgICAgICAgcmV0dXJuOworCisgICAgICAgICAgICBpZiAocmVxdWVzdGVkRnJhbWVSYXRlIDwg ZnJhbWVSYXRlUmFuZ2UubWluRnJhbWVSYXRlKQorICAgICAgICAgICAgICAgIHJlcXVlc3RlZEZy YW1lUmF0ZSA9IGZyYW1lUmF0ZVJhbmdlLm1pbkZyYW1lUmF0ZTsKKyAgICAgICAgICAgIGVsc2Ug aWYgKHJlcXVlc3RlZEZyYW1lUmF0ZSA+IGZyYW1lUmF0ZVJhbmdlLm1heEZyYW1lUmF0ZSkKKyAg ICAgICAgICAgICAgICByZXF1ZXN0ZWRGcmFtZVJhdGUgPSBmcmFtZVJhdGVSYW5nZS5tYXhGcmFt ZVJhdGU7CisKKyAgICAgICAgICAgIEFMV0FZU19MT0dfSUYobG9nZ2VyUHRyKCksIExPR0lERU5U SUZJRVIsICJzZXR0aW5nIGZyYW1lIHJhdGUgdG8gIiwgcmVxdWVzdGVkRnJhbWVSYXRlKTsKKyAg ICAgICAgICAgIFtkZXZpY2UoKSBzZXRBY3RpdmVWaWRlb01pbkZyYW1lRHVyYXRpb246IENNVGlt ZU1ha2UoMSwgcmVxdWVzdGVkRnJhbWVSYXRlKV07CisgICAgICAgICAgICBbZGV2aWNlKCkgc2V0 QWN0aXZlVmlkZW9NYXhGcmFtZUR1cmF0aW9uOiBDTVRpbWVNYWtlKDEsIHJlcXVlc3RlZEZyYW1l UmF0ZSldOwogCi0gICAgICAgICAgICBBTFdBWVNfTE9HX0lGKGxvZ2dlclB0cigpLCBMT0dJREVO VElGSUVSLCAic2V0dGluZyBmcmFtZSByYXRlICIsIENNVGltZUdldFNlY29uZHMoW2ZyYW1lUmF0 ZVJhbmdlIG1pbkZyYW1lRHVyYXRpb25dKSwgIiAuLiAiLCBDTVRpbWVHZXRTZWNvbmRzKFtmcmFt ZVJhdGVSYW5nZSBtYXhGcmFtZUR1cmF0aW9uXSkpOwotICAgICAgICAgICAgW2RldmljZSgpIHNl dEFjdGl2ZVZpZGVvTWluRnJhbWVEdXJhdGlvbjpbZnJhbWVSYXRlUmFuZ2UgbWluRnJhbWVEdXJh dGlvbl1dOwotICAgICAgICAgICAgW2RldmljZSgpIHNldEFjdGl2ZVZpZGVvTWF4RnJhbWVEdXJh dGlvbjpbZnJhbWVSYXRlUmFuZ2UgbWF4RnJhbWVEdXJhdGlvbl1dOwogICAgICAgICAgICAgW2Rl dmljZSgpIHVubG9ja0ZvckNvbmZpZ3VyYXRpb25dOwogICAgICAgICB9CiAgICAgfSBAY2F0Y2go TlNFeGNlcHRpb24gKmV4Y2VwdGlvbikgewo=