Bug 196214

Summary: getUserMedia framerate unusable under low light in iOS 12.2
Product: WebKit Reporter: daginge
Component: WebRTCAssignee: youenn fablet <youennf>
Status: RESOLVED FIXED    
Severity: Major CC: commit-queue, eric.carlson, ggaren, ibc, jhenderson2177, jonlee, kevinday, michael, simon.fraser, webkit-bug-importer, youennf
Priority: P2 Keywords: InRadar
Version: Safari 12   
Hardware: iPhone / iPad   
OS: Unspecified   
Attachments:
Description Flags
Patch none

Description daginge 2019-03-25 14:07:46 PDT
Since upgrading to iOS 12.2, I tested the frame rate of getUserMedia by going to https://webrtc.github.io/samples/src/content/getusermedia/gum/. The framerate is horrible to the point of unusable, as you can no longer see the mouth move when talking, and moving the camera makes an interesting long-tail effect of the previous image. I've attached videos showing the getUserMedia sample page with the problem, and the native camera app, which shows no sign of this issue.

getUserMedia on WebRTC samples
https://photos.app.goo.gl/1RnJQ7SFhG6rqSBQ6

Actual flipped camera framerate in Camera app
https://photos.app.goo.gl/1CmBPr1ho3wyNP579

I have tried killing the Safari app, fully rebooting my phone, and anything else I can think of, and the problem remains.

What's worse, this is not only a local rendering issue, it's also an issue when sending the video to the other side (using H264 too). Audio is not affected luckily.

As it stands now, using getUserMedia for anything except taking perfectly stationary images will not work, and it is completely breaking for any video experience in the browser.

My phone's about page shows:
Software version: 12.2 (16E227)
Model name: iPhone X
Model version: MQAD2QN/A (A1901)

Any help would be appreciated in working around this problem, as this will completely break Confrere for patients/physicians when they upgrade to iOS 12.2
Comment 1 daginge 2019-03-25 14:22:22 PDT
Able to replicate this on my iPad Pro as well.

Software version: 16E227
Model name: iPad Pro 10.5 inch
Model number: MQDW2KN/A (A1701)
Comment 2 daginge 2019-03-25 14:25:33 PDT
Interestingly, the problem fixes itself (as does the resolution (??)) if I go to the home screen and open Safari again.

To repro:
1. Go to https://webrtc.github.io/samples/src/content/getusermedia/gum/
2. Click "open camera"
3. Observe that frame rate is slow
4. Go to the home screen by swiping up
5.Wait at least 2 seconds
6. Open Safari again
7. Observe that the resolution has changed (black borders around the video frame), and the video is now fast again (as fast as native camera app)
Comment 3 daginge 2019-03-25 14:33:42 PDT
Problem also persists with all standard resolutions tried here: https://webrtc.github.io/samples/src/content/getusermedia/resolution/. Workaround by going to the home screen and back seems to fix the issue (and changes the aspect ration) for all resolutions.
Comment 4 Radar WebKit Bug Importer 2019-03-25 15:21:28 PDT
<rdar://problem/49232193>
Comment 5 IƱaki Baz 2019-03-25 15:47:20 PDT
I've seen this issue in Macbook Pro OSX latest versions and Safari Tech latest versions. "Sometimes" it does not happen. No idea what it depends on, but most times it happens.
Comment 6 daginge 2019-03-26 01:44:24 PDT
As of this morning I'm unable to replicate the issue as seen in the videos in either of my devices. None of the devices have been shut off during the night.

https://photos.app.goo.gl/4hmARduvZd9jrFND9

Aspect ratio still changes when going to the home screen and back though, but at least the horrible lag isn't there anymore.
Comment 7 daginge 2019-03-27 08:45:36 PDT
I FINALLY FIGURED IT OUT!

iOS 12.2 behaves badly in _low light_ scenarios. That's why it didn't work in the evening with poor lighting conditions, and worked well in daylight when I tested it again.

https://photos.app.goo.gl/akjHTfQ6oAZtfwCW8

See attached video showing extremely poor performance in low light, then going to home screen and back magically fixes the problem (and changes aspect ratio). 

Moving to a better lit scenario also fixes the problem. This was not a problem in previous versions.

Do I get a cake now? :D
Comment 8 youenn fablet 2019-03-28 01:25:02 PDT
> Do I get a cake now? :D

Thanks!
I was also able to repro on iOS and somehow on MacOS as well.
Comment 9 youenn fablet 2019-03-28 02:08:06 PDT
The call to setActiveVideoMinFrameDuration/setActiveVideoMaxFrameDuration in AVVideoCaptureSource::setSizeAndFrameRateWithPreset might actually trigger this behavior.
Comment 10 JH 2019-03-28 12:51:11 PDT
I'm able to reproduce this on my iPhone X (MQCN2LL/A A1865) running iOS 12.2 (16E227) if I don't have perfect lighting.

Another workaround is to request "frameRate: 60" in the constraints and the problem goes away, regardless of the lighting quality. Lower frameRate values, or not specifying one, and the problem still happens.
Comment 11 youenn fablet 2019-05-07 11:21:50 PDT
Created attachment 369305 [details]
Patch
Comment 12 Geoffrey Garen 2019-05-07 12:19:05 PDT
Comment on attachment 369305 [details]
Patch

r=me
Comment 13 WebKit Commit Bot 2019-05-07 14:04:15 PDT
Comment on attachment 369305 [details]
Patch

Clearing flags on attachment: 369305

Committed r245032: <https://trac.webkit.org/changeset/245032>
Comment 14 WebKit Commit Bot 2019-05-07 14:04:17 PDT
All reviewed patches have been landed.  Closing bug.
Comment 15 Simon Fraser (smfr) 2019-05-07 14:06:46 PDT
Comment on attachment 369305 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=369305&action=review

> Source/WebCore/platform/mediastream/mac/AVVideoCaptureSource.mm:332
> +            if (requestedFrameRate < frameRateRange.minFrameRate)
> +                requestedFrameRate = frameRateRange.minFrameRate;
> +            else if (requestedFrameRate > frameRateRange.maxFrameRate)
> +                requestedFrameRate = frameRateRange.maxFrameRate;

Use clampTo()?
Comment 16 youenn fablet 2019-05-08 12:24:41 PDT
*** Bug 197703 has been marked as a duplicate of this bug. ***
Comment 17 youenn fablet 2019-05-08 12:26:25 PDT
(In reply to Simon Fraser (smfr) from comment #15)
> Comment on attachment 369305 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=369305&action=review
> 
> > Source/WebCore/platform/mediastream/mac/AVVideoCaptureSource.mm:332
> > +            if (requestedFrameRate < frameRateRange.minFrameRate)
> > +                requestedFrameRate = frameRateRange.minFrameRate;
> > +            else if (requestedFrameRate > frameRateRange.maxFrameRate)
> > +                requestedFrameRate = frameRateRange.maxFrameRate;
> 
> Use clampTo()?

Makes sense, will fix it in bug 197704.
Comment 18 micsun-al 2019-05-08 12:58:53 PDT
(In reply to JH from comment #10)
> I'm able to reproduce this on my iPhone X (MQCN2LL/A A1865) running iOS 12.2
> (16E227) if I don't have perfect lighting.
> 
> Another workaround is to request "frameRate: 60" in the constraints and the
> problem goes away, regardless of the lighting quality. Lower frameRate
> values, or not specifying one, and the problem still happens.

Although the frame rate problem seems to go away with this workaround, the black bars seem to still be an issue.  I wonder if the patches fix the black bars too?
Comment 19 youenn fablet 2019-05-08 13:22:50 PDT
(In reply to micsun-al from comment #18)
> (In reply to JH from comment #10)
> > I'm able to reproduce this on my iPhone X (MQCN2LL/A A1865) running iOS 12.2
> > (16E227) if I don't have perfect lighting.
> > 
> > Another workaround is to request "frameRate: 60" in the constraints and the
> > problem goes away, regardless of the lighting quality. Lower frameRate
> > values, or not specifying one, and the problem still happens.
> 
> Although the frame rate problem seems to go away with this workaround, the
> black bars seem to still be an issue.  I wonder if the patches fix the black
> bars too?

That is a separate issue, I filed https://bugs.webkit.org/show_bug.cgi?id=197707.
Comment 20 daginge 2019-06-02 06:53:07 PDT
This issue is not fixed for me on iOS 12.3.1 using Safari 12.1.1. I can still reliably reproduce this in low light on https://webrtc.github.io/samples/src/content/getusermedia/gum/ on my iPhone X.

Let me know if you need additional debugging information.
Comment 21 daginge 2019-06-02 06:58:49 PDT
See difference between native camera and webrtc:

native: https://share.icloud.com/photos/0oKCLxUqdxlPfFSjxDg7eGwPQ
webrtc: https://share.icloud.com/photos/0hYZ4piriFK45trA0hk_gPJ9g

These were recorded right after one another in the exact same lighting conditions.
Comment 22 Simon Fraser (smfr) 2019-06-02 09:31:35 PDT
12.3.1 doesn't have the fix for this bug. Fixing a bug in bugzilla does not correspond with when it's available in an OS update.