Bug 196214 – getUserMedia framerate unusable under low light in iOS 12.2

• Component: WebRTC
• Status: RESOLVED FIXED

[Dag-Inge Aas]

Since upgrading to iOS 12.2, I tested the frame rate of getUserMedia by going to https://webrtc.github.io/samples/src/content/getusermedia/gum/. The framerate is horrible to the point of unusable, as you can no longer see the mouth move when talking, and moving the camera makes an interesting long-tail effect of the previous image. I've attached videos showing the getUserMedia sample page with the problem, and the native camera app, which shows no sign of this issue.

getUserMedia on WebRTC samples
https://photos.app.goo.gl/1RnJQ7SFhG6rqSBQ6

Actual flipped camera framerate in Camera app
https://photos.app.goo.gl/1CmBPr1ho3wyNP579

I have tried killing the Safari app, fully rebooting my phone, and anything else I can think of, and the problem remains.

What's worse, this is not only a local rendering issue, it's also an issue when sending the video to the other side (using H264 too). Audio is not affected luckily.

As it stands now, using getUserMedia for anything except taking perfectly stationary images will not work, and it is completely breaking for any video experience in the browser.

My phone's about page shows:
Software version: 12.2 (16E227)
Model name: iPhone X
Model version: MQAD2QN/A (A1901)

Any help would be appreciated in working around this problem, as this will completely break Confrere for patients/physicians when they upgrade to iOS 12.2

[Dag-Inge Aas]

Able to replicate this on my iPad Pro as well.

Software version: 16E227
Model name: iPad Pro 10.5 inch
Model number: MQDW2KN/A (A1701)

[Dag-Inge Aas]

Interestingly, the problem fixes itself (as does the resolution (??)) if I go to the home screen and open Safari again.

To repro:
1. Go to https://webrtc.github.io/samples/src/content/getusermedia/gum/
2. Click "open camera"
3. Observe that frame rate is slow
4. Go to the home screen by swiping up
5.Wait at least 2 seconds
6. Open Safari again
7. Observe that the resolution has changed (black borders around the video frame), and the video is now fast again (as fast as native camera app)

[Dag-Inge Aas]

Problem also persists with all standard resolutions tried here: https://webrtc.github.io/samples/src/content/getusermedia/resolution/. Workaround by going to the home screen and back seems to fix the issue (and changes the aspect ration) for all resolutions.

[Radar WebKit Bug Importer]

<rdar://problem/49232193>

[Iñaki Baz]

I've seen this issue in Macbook Pro OSX latest versions and Safari Tech latest versions. "Sometimes" it does not happen. No idea what it depends on, but most times it happens.

[Dag-Inge Aas]

As of this morning I'm unable to replicate the issue as seen in the videos in either of my devices. None of the devices have been shut off during the night.

https://photos.app.goo.gl/4hmARduvZd9jrFND9

Aspect ratio still changes when going to the home screen and back though, but at least the horrible lag isn't there anymore.

[Dag-Inge Aas]

I FINALLY FIGURED IT OUT!

iOS 12.2 behaves badly in _low light_ scenarios. That's why it didn't work in the evening with poor lighting conditions, and worked well in daylight when I tested it again.

https://photos.app.goo.gl/akjHTfQ6oAZtfwCW8

See attached video showing extremely poor performance in low light, then going to home screen and back magically fixes the problem (and changes aspect ratio). 

Moving to a better lit scenario also fixes the problem. This was not a problem in previous versions.

Do I get a cake now? :D

[youenn fablet]

> Do I get a cake now? :D

Thanks!
I was also able to repro on iOS and somehow on MacOS as well.

[youenn fablet]

The call to setActiveVideoMinFrameDuration/setActiveVideoMaxFrameDuration in AVVideoCaptureSource::setSizeAndFrameRateWithPreset might actually trigger this behavior.

[JH]

I'm able to reproduce this on my iPhone X (MQCN2LL/A A1865) running iOS 12.2 (16E227) if I don't have perfect lighting.

Another workaround is to request "frameRate: 60" in the constraints and the problem goes away, regardless of the lighting quality. Lower frameRate values, or not specifying one, and the problem still happens.

[youenn fablet]

Created attachment 369305 [details]
Patch

[Geoffrey Garen]

Comment on attachment 369305 [details]
Patch

r=me

[WebKit Commit Bot]

Comment on attachment 369305 [details]
Patch

Clearing flags on attachment: 369305

Committed r245032: <https://trac.webkit.org/changeset/245032>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Simon Fraser (smfr)]

Comment on attachment 369305 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=369305&action=review

> Source/WebCore/platform/mediastream/mac/AVVideoCaptureSource.mm:332
> +            if (requestedFrameRate < frameRateRange.minFrameRate)
> +                requestedFrameRate = frameRateRange.minFrameRate;
> +            else if (requestedFrameRate > frameRateRange.maxFrameRate)
> +                requestedFrameRate = frameRateRange.maxFrameRate;

Use clampTo()?

[youenn fablet]

*** Bug 197703 has been marked as a duplicate of this bug. ***

[youenn fablet]

(In reply to Simon Fraser (smfr) from comment #15)
> Comment on attachment 369305 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=369305&action=review
> 
> > Source/WebCore/platform/mediastream/mac/AVVideoCaptureSource.mm:332
> > +            if (requestedFrameRate < frameRateRange.minFrameRate)
> > +                requestedFrameRate = frameRateRange.minFrameRate;
> > +            else if (requestedFrameRate > frameRateRange.maxFrameRate)
> > +                requestedFrameRate = frameRateRange.maxFrameRate;
> 
> Use clampTo()?

Makes sense, will fix it in bug 197704.

[micsun-al]

(In reply to JH from comment #10)
> I'm able to reproduce this on my iPhone X (MQCN2LL/A A1865) running iOS 12.2
> (16E227) if I don't have perfect lighting.
> 
> Another workaround is to request "frameRate: 60" in the constraints and the
> problem goes away, regardless of the lighting quality. Lower frameRate
> values, or not specifying one, and the problem still happens.

Although the frame rate problem seems to go away with this workaround, the black bars seem to still be an issue.  I wonder if the patches fix the black bars too?

[youenn fablet]

(In reply to micsun-al from comment #18)
> (In reply to JH from comment #10)
> > I'm able to reproduce this on my iPhone X (MQCN2LL/A A1865) running iOS 12.2
> > (16E227) if I don't have perfect lighting.
> > 
> > Another workaround is to request "frameRate: 60" in the constraints and the
> > problem goes away, regardless of the lighting quality. Lower frameRate
> > values, or not specifying one, and the problem still happens.
> 
> Although the frame rate problem seems to go away with this workaround, the
> black bars seem to still be an issue.  I wonder if the patches fix the black
> bars too?

That is a separate issue, I filed https://bugs.webkit.org/show_bug.cgi?id=197707.

[Dag-Inge Aas]

This issue is not fixed for me on iOS 12.3.1 using Safari 12.1.1. I can still reliably reproduce this in low light on https://webrtc.github.io/samples/src/content/getusermedia/gum/ on my iPhone X.

Let me know if you need additional debugging information.

[Dag-Inge Aas]

See difference between native camera and webrtc:

native: https://share.icloud.com/photos/0oKCLxUqdxlPfFSjxDg7eGwPQ
webrtc: https://share.icloud.com/photos/0hYZ4piriFK45trA0hk_gPJ9g

These were recorded right after one another in the exact same lighting conditions.

[Simon Fraser (smfr)]

12.3.1 doesn't have the fix for this bug. Fixing a bug in bugzilla does not correspond with when it's available in an OS update.