Bug 253186 – REGRESSION iOS 16.4 beta selects ultra-wide for facingMode: environment

• Component: WebRTC
• Status: RESOLVED FIXED

[Simon Taylor]

Related: Bug 250142 and Bug 252560

Recently WebKit has added support for exposing the virtual cameras on iOS and logic to prefer these virtual cameras when choosing a camera based on a facingMode constraint.

The virtual cameras start with videoZoomFactor set to 1, which means the field of view is set to the widest-FOV constituent device in the set. In most recent iPhone devices (all of them since iPhone 11) that means the default camera stream that comes from a {facingMode: 'environment'} constraint is now an ultra-wide one. I've verified this on an iPhone 12 Pro using the latest 16.4 Beta 2 build.

After https://github.com/WebKit/WebKit/pull/10784 was applied, the current priority list in AVVideoCaptureSource::facingModeFitnessScoreAdjustment() looks like this:
 1. AVCaptureDeviceTypeBuiltInTripleCamera
 2. AVCaptureDeviceTypeBuiltInDualWideCamera
 3. AVCaptureDeviceTypeBuiltInUltraWideCamera
 4. AVCaptureDeviceTypeBuiltInDualCamera
 5. AVCaptureDeviceTypeBuiltInWideAngleCamera
 6. AVCaptureDeviceTypeBuiltInTelephotoCamera
 7. AVCaptureDeviceTypeDeskViewCamera

The previous default was to use the standard FOV AVCaptureDeviceTypeBuiltInWideAngleCamera (now only the 5th priority).

Changing this default has far-ranging consequences and will likely break many of our sites (due to assumptions on camera model used in computer vision algorithms, as the Web API doesn't expose any specific characteristics on this).

Sites that want a simple camera UI such as ours just use a toggle button that swaps between 'user' and 'environment' with a facingMode constraint. I think most users would expect that to swap between the standard 1x rear camera and the front-facing one, hence why I have called this a regression.

Right now WebKit doesn't expose the zoom settings, so the virtual cameras always use the smallest possible zoom and hence are effectively just identical to the ultra-wide cameras (also exposed in the enumeration).

From the discussion on bug 250142 it's possible the intent was to match the default behaviour seen in the iOS camera app on an iPhone 13/14 Pro. The standard camera on those devices has quite a high minimum focus distance, so the iPhone camera app viewfinder in Photo mode switches to the ultrawide when getting close to objects as it provides better macro focus.

If the changes to add virtual camera support to WebKit were intended to match that behaviour then the videoZoomFactor needs to be set to 2x when using the virtual cameras to achieve that.

However the auto switching for macro is not the right default for every possible getUserMedia use-case. It would definitely help for scanning barcodes from closer distances on the iPhone 14 Pro, but for AR use cases (which is our primary focus) the switch of camera is visually jarring and we'd have a strong preference to lock to the standard camera and put up with a bit of blur in the background view if users move too close.

Note not all native camera use cases on iPhone 14 Pro use the auto-switching behaviour - when a video is being recorded the camera doesn't switch until you manually tap-to-focus. AR Quick Look (and presumably other ARKit experiences) don't switch to the ultrawide when getting close to a surface.

Therefore for the best backward compatibility, I'd suggest switching back to a preference for the standard cameras when selecting just based on the facingMode constraint. Note that's also what the native iOS API does for backwards compatibility - the original `[AVCaptureDevice devicesWithMediaType:] method only returns the standard one.

Of course I would also welcome a way for sites such as barcode scanners to opt-in to "auto macro" mode involving a lens switch - perhaps a new "Auto Macro" virtual camera that would be the Triple or DualWide with videoZoomFactor set to 2x. I still think the lens switch is pretty jarring for many getUserMedia use cases so I'd argue that should be opt-in rather than the default for the environment facing mode.

I'd like to add to this that there's no usuable workaround to maintain the existing (<= iOS 16.3) behaviour of selecting a device with a natural field of view.

To do so we'd have to iterate through the device list and find "Back Camera" or similar, but that list is localised into the user's chosen device language - so what's "Back Camera" in English would be "Rückkamera" in German.

In Beta 2 we have the situation that:

 - the default environment facing camera is ultra-wide; very different from iOS 16.3 and earlier

 - the device list is localised, so it's very difficult to pattern match to find the camera with the natural field of view

 - there's no visibility on the field of view or other parameters of a given camera in the list

In our opinion the best behavior would be (in descending order of importance :-))

 - avoid breaking existing sites that expect a natural FOV back camera by ensuring the environment facingMode constraint returns the same device it did in earlier iOS, 'Back Camera'

 - provide non-localised fields on MediaDeviceInfo that empower sites to select a camera that best fits their requirements in the case that the default camera does not. An example would be 'lensType': 'wide' | 'ultrawide' | 'telephoto' | 'macro' - something that doesn't add any fingerprinting entropy over the device name but does allow developers to unambiguously pick a camera that suits their needs

Finally I'd like to say that the current behavior would break many sites we have in production, plus those of our customers who use our SDK. We'd be very grateful of your attention on this bug :-)

[Radar WebKit Bug Importer]

<rdar://problem/106142457>

[Simon Taylor]

Created attachment 465280 [details]
Video demonstrating the "auto macro switch" behaviour in iPhone 14 Pro Camera app

I thought it was worth adding a video so anyone without an iPhone 13 / 14 Pro can see automatic macro switching behaviour.

You can see it's triggered in the camera viewfinder when there's an object closer than 10 cm or so and that the switch is quite jarring. I also show that when actually recording a video, the switch only happens when manually tapping to focus on something too close.

[Simon Taylor]

I know Apple as a rule don't comment on upcoming software but some "forward guidance" on what to expect in 16.4 would be really appreciated.

If this change to the default isn't going to be reverted we'll need to start planning how to roll-out explicit camera selection to all our live sites to maintain the previous default camera selection behaviour.

[youenn fablet]

This bug is being investigated.
It is still opened so there is no fix in WebKit trunk yet.

See related bugs https://bugs.webkit.org/show_bug.cgi?id=253645 and https://bugs.webkit.org/show_bug.cgi?id=253646

[Simon Taylor]

Thanks for the links to the other bugs.

There are two slightly different issues:
 1) Making it easier for pages to pick the best camera for their use case from the enumerated set
 2) Changing the default environment-facing camera

It's great to see the work on those other bugs which will really help with 1, but my concern here is 2.

It can definitely be argued that it's still valid to return ultrawide for `facing: environment` and even to prefer it for its larger supported focus range on iPhone 13/14 Pro. I argued the default shouldn't be changed as it would be unexpected for users and disruptive for existing sites (such as ours) that are built around the assumption of the standard "wide" camera being the default.

It's forward guidance on where you stand on that question really that I'm after. If the default is going to change to ultrawide it would be good to have a heads up before the final 16.4 drops so we can get ahead of all the disruption that would cause for us.

[youenn fablet]

@Simon, @connell, can you share publicly or privately ([e-mail]) some websites that are broken?

[Maciej Stachowiak]

Folks who have filed this or commented here, can you tell us about some specific sites impacted by this change and what the user impact is?

[Maciej Stachowiak]

(Asking because we’re trying to figure out the best changes to make and appropriate timing, and it’s easier to judge the impact with specific sites and user-level repro steps to see the problem.)

[Simon Taylor]

Thanks for asking.

https://web.zappar.com shows one of our "WebAR" runtimes built on getUserMedia, motion sensor data, and web assembly. Once you tap launch you'll see the `facingMode: environment` camera by default which is now equivalent to the ultra-wide on my iPhone 14 Pro on the current 16.4 beta.

There's a "switch camera" button in the toolbar to switch to the user-facing one and back. That's the "simple UI" I mentioned in the original report rather than offering the full enumerated drop down (which now lists no less than 7 devices with the latest beta on the iPhone 14 Pro).

I think that's enough to see this as a regression - the toggle doesn't do the same thing as in the native camera app, which starts up by default with the "1x / Wide" camera FOV. It means you've got to be twice as close to objects in the web now to view them at the same size, and there's a more extreme and unusual perspective effect on the images.

In terms of the actual impact on our sites, it causes a regression in tracking quality as our computer vision algorithms assume a standard field-of-view. You can open the site at https://docs.zap.works/app-embed/regression-tests/ on a laptop screen and scan the #3 code, then point towards the target image at the top of the page. When viewing from non-straight on angles you can see the green background plane should align with the image on the screen but the perspective is significantly different, and the content doesn't feel properly "anchored" in the camera view (as the virtual camera field-of-view is very different from the camera one).

To avoid this regression, we would need to manually enumerate and select the "correct" rear facing camera (Connell covered the challenges of doing this with localized names in a previous comment - we'd likely rely on enumeration order for now, but that's not guaranteed in any spec so we'd need to check webkit code to see if it is at least a short-term solution).

As well as needing to figure out the "right" workaround for for our camera selection code, we'd also need to deploy updated sites. We have some "shared" runtime sites like web.zappar.com, but also hundreds of domains with white-labelled versions. Alongside that we have SDKs for common WebGL engines like three.js distributed as npm packages, which customers can bundle whatever versions of into their independent builds. To get those updated we'd need to give an advisory to customers along with a reasonable time for them to update their dependencies (which they may be unwilling or unable to do for various reasons).

Hopefully that gives some insight into the level of disruption and difficulty this change would cause for us.

[youenn fablet]

Pull request: https://github.com/WebKit/WebKit/pull/11705

[EWS]

Committed 261877@main (1ed9893d3737): <https://commits.webkit.org/261877@main>

Reviewed commits have been landed. Closing PR #11705 and removing active labels.

We've been in comms with the Apple team on this by email over the last few weeks and they've let us know that we should have a contingency plan in place for this bug as it's still present in the latest public release candidate build.

In the meantime we've been preparing a workaround for our platform and we thought we'd outline how it works here for anyone else who encounters this Safari bug. If you're a Zappar user, we've released builds with these workarounds in place already.

Instead of relying on the constraints for selecting the environment facing camera, we do the following (for iOS 16.4+ only):
 - Call getUserMedia() requesting a video device with no audio, just to get permissions from the user
 - Call enumerateDevices() to get a list of the devices
 - Filter our any audio devices from the list
 - Call getUserMedia() again, passing as a constraint the deviceID of the first enumerated device in the list for the front camera, or the second in the list for the rear camera.

In our tests the first camera in the list is always the correct front camera, and the second is always the correct rear camera. Unfortunately it's not possible to rely on the names of the devices in the list as they are localised into the user's chosen device language. We've checked the current WebKit camera enumeration code and we believe this is a stable solution for the time being.

Note that the underlying spec does not guarantee the order of the list in general (other than the very first entry) so it's essential to only take this approach on iOS - it does not select the correct environment camera on Android Chrome and is unlikely to on other browsers.

I can confirm that this issue is present in the now-shipping 16.4 release.