Bug 250142 – [Cocoa, MediaStream] support AVCapture virtual cameras

• Component: WebRTC
• Status: RESOLVED FIXED

[Eric Carlson]

Support AVCapture virtual cameras.

[Radar WebKit Bug Importer]

<rdar://problem/103925476>

[Eric Carlson]

Pull request: https://github.com/WebKit/WebKit/pull/8249

[EWS]

Committed 258585@main (ba4235d06159): <https://commits.webkit.org/258585@main>

Reviewed commits have been landed. Closing PR #8249 and removing active labels.

[Mo]

Wow, thanks a lot for getting this resolved in such a timely manner!

Will this still be merged into Safari 16.3 or is it only landed in 16.4?
Sorry, I am not very firm with the release process.

[Simon Taylor]

What's the rationale for adding these to WebKit?

As far as I understand, the main reason to use these cameras is to expose a larger continuous zoom range from the ultra wide to the telephoto end and have the device switch the primary device being used during the zoom.

However I can't see any usage of `videoZoomFactor` in WebKit currently, so I don't understand the benefit of selecting (and preferring) the virtual cameras here.

[Mo]

From my understanding, virtual cameras not only switch the device based on the zoom factor but also pick the optimal camera for the current focus distance.

So by getting access to the virtual camera, the focus range would be much larger without having to switch between devices manually.

Putting these considerations aside, adding support for `videoZoomFactor` in WebKit would amazing, but that's another story.

[Simon Taylor]

(In reply to Mo from comment #6)
> From my understanding, virtual cameras not only switch the device based on
> the zoom factor but also pick the optimal camera for the current focus
> distance.
> 
> So by getting access to the virtual camera, the focus range would be much
> larger without having to switch between devices manually.

Thanks, I guess there might be a small benefit to extreme macro performance by switching to the ultrawide lens.

Looking just in the iOS camera app on my 12 Pro it doesn't automatically switch at 1x zoom even if it's out-of-focus, but zooming out to 0.9 does show the ultrawide lens is sharper from the same distance. The distance from the surface is similar to the distance between the cameras so it's not exactly a seamless switch. I'll attach a video.

It would be good to confirm iOS does do this automatically based only on autofocus for extreme macro, otherwise there doesn't seem a great deal of value in preferring these more complex virtual cameras in WebKit.

[Simon Taylor]

Created attachment 465236 [details]
iOS camera app ultrawide / wide switch

Ultrawide can focus from closer to the subject but the camera switch is jarring at this distance. It's not clear iOS would do this purely based on autofocus.

Hey Simon, let me add something to this discussion.

When using the virtual cameras on the supported devices, Apples handles the switch between the physical camera based on different factors: zoom factor, light level, and focus position.

As I said, on supported devices: the iPhone 12 Pro you are testing on is not supported for the switch between the standard camera and the ultra wide (UW), as the UW there is without AF. If you where to try with an iPhone 13 Pro or 14 Pro, you will see how it automatically switches between the cameras (enabling the "Macro Mode").

Another point is that on these iPhone 13 Pro or 14 Pro, the minimum focus distance of the standard camera increased a lot - and for many situation you need to switch to the ultra-wide camera to actually focus on the subjects. 
This is the main reason why we would need the possibility to perform an automatic switch based on the focus distance.

Last point is about the virtualZoomFactor: as reported before, zoom is one of the factor for the switch. 
For example, when using the builtInDualWideCamera for example, you will be able to switch automatically between Wide and UltraWide camera only if the zoomFactor is set to 2x. The full list of zoomFactors is available at https://developer.apple.com/documentation/avfoundation/avcapturedevice/3153003-virtualdeviceswitchovervideozoom

So, if we want to let the users be able to focus at all ranges with iPhone 13.14 Pro/Pro Max, we would need to use the builtInDualWideCamera and set the virtualZoomFactor to 2x. At that point, the field-of-view will be the same of the standard camera, but Apple will handle the switch between cameras - letting the user focus at different distances.

[Simon Taylor]

Hi Raffaele,

Thanks very much for the extra information. I happen to have an iPhone 14 Pro here too and see that do the automatic switching as you get closer to the object in the camera viewfinder.

Thanks also for the note about videoZoomFactor, I've just verified in a native app that the AVCaptureDeviceTypeBuiltInTripleCamera starts up with videoZoomFactor of 1x, and showing the full ultrawide image. As you say videoZoomFactor has to be set to 2x to get the auto-switching for macro behaviour [The fact the iPhone camera app calls the ultrawide 0.5x made this surprising to me, I was expecting it to start up in the "1x" mode from the camera app].

I think the current WebKit code is going to prefer the virtual camera, but still with videoZoomFactor at 1x - ie the default will now appear ultra-wide for users which I think is a surprising default. I'm installing the 16.4 beta on the iPhone 14 Pro now to see what that does.

I can see for the potentially-small code scanning use case, you would need decent macro focus support and your preference would probably be the virtual camera with videoZoomFactor of 2x.

Our use case is primarily AR experiences. Some of these involving moving the camera towards and away from content that will take it through the transition distance, and there the transition between cameras is really jarring (never mind also creating problems for computer vision code).

I'll make another bug report after I've tried 16.4 beta on the 14 Pro, as I do think the default needs to be adjusted here.

[Simon Taylor]

I've added Bug 253186 for more discussion of the concerns around changing the default environment facing camera.