Bug 252560 – REGRESSION (iOS 16.4 Public Beta) getUserMedia ignores facingMode constraint

• Component: WebRTC
• Status: RESOLVED FIXED

Created attachment 465085 [details]
A minimal test case showing the issue

In iOS 16.4 Beta (20E5212f), a call to `navigator.mediaDevices.getUserMedia` passing a `facingMode: "environment"` video constraint (as below) starts a stream using the user-facing camera (rather than the rear-facing camera).

```
navigator.mediaDevices.getUserMedia({
	audio: false,
	video: {
		facingMode: "environment"
	}
}).then(stream => {
	vid.srcObject = stream;
	vid.play();
})

```

This looks to be a significant regression - our testing with a device running iOS 16.3.1 shows that it starts the correct camera.

I've attached a minimal example of the issue - index.html - to this report. I'm also hosting the index.html file here:
https://bhqq.zappar.io/1962332228841157357/1.0.0/

Just tap the 'Start' button. On the affected iOS Safari version the user-facing camera appears. On 16.3.1 and earlier the rear-facing camera appears.

This is an important bug for us - we have large number of sites in production that make use of this constraint in order to select the correct device.

Let me know if I can help in any way :-)

[youenn fablet]

Thanks, we will investigate.

[Radar WebKit Bug Importer]

<rdar://problem/105677398>

[youenn fablet]

@connell, which device have you used to reproduce the bug?
On my iPhone 12 Pro, it does not repro.

Hi @youenn! It's an iPhone XR (MRY82B/A). Thanks for taking a look so quickly :-)

[Dat Chu]

I can confirm that this issue is repro on my iPhone 14 Plus running iOS 16.4.

It doesn't repro on my iPhone 13 Pro running iOS 16.4.

[youenn fablet]

Pull request: https://github.com/WebKit/WebKit/pull/10784

[Simon Taylor]

Thanks for the link to the pull request @youenn. Capping the fitness distance increase to a max of 0.5 sounds sensible.

However worth noting that Connell initially saw this on iPhone XR which only has a single environment-facing camera (no telephoto ones) so it seems unexpected that this camera would be penalised at all with a fitness distance increase?

[EWS]

Committed 260953@main (c9e35ddc3cab): <https://commits.webkit.org/260953@main>

Reviewed commits have been landed. Closing PR #10784 and removing active labels.

[youenn fablet]

(In reply to Simon Taylor from comment #7)
> Thanks for the link to the pull request @youenn. Capping the fitness
> distance increase to a max of 0.5 sounds sensible.
> 
> However worth noting that Connell initially saw this on iPhone XR which only
> has a single environment-facing camera (no telephoto ones) so it seems
> unexpected that this camera would be penalised at all with a fitness
> distance increase?

The issue is that the fitness score would be the same for the front camera and the back camera even if facing mode is set to environment. The PR fixes that to guarantee that the fitness score would be higher for back cameras if facing mode is set to environment

Great news @youenn! Thank you for moving so quickly on it. Do you know if this will make it into the next iOS 16.4 beta? Will make sure to give it a good test if so :-)

[Simon Taylor]

(In reply to youenn fablet from comment #9)
> (In reply to Simon Taylor from comment #7)
> > Thanks for the link to the pull request @youenn. Capping the fitness
> > distance increase to a max of 0.5 sounds sensible.
> > 
> > However worth noting that Connell initially saw this on iPhone XR which only
> > has a single environment-facing camera (no telephoto ones) so it seems
> > unexpected that this camera would be penalised at all with a fitness
> > distance increase?
> 
> The issue is that the fitness score would be the same for the front camera
> and the back camera even if facing mode is set to environment. The PR fixes
> that to guarantee that the fitness score would be higher for back cameras if
> facing mode is set to environment

Yup I was confused by why when there was only one camera on the device it would have been getting a penalty before. I've read through the landed PR now, so I see the priority is a hardcoded ordering regardless of which devices a camera actually has.

The change to fitness score looks reasonable though.

The current priority list in AVVideoCaptureSource::facingModeFitnessScoreAdjustment() looks like this:
 1. AVCaptureDeviceTypeBuiltInTripleCamera
 2. AVCaptureDeviceTypeBuiltInDualWideCamera
 3. AVCaptureDeviceTypeBuiltInUltraWideCamera
 4. AVCaptureDeviceTypeBuiltInDualCamera
 5. AVCaptureDeviceTypeBuiltInWideAngleCamera
 6. AVCaptureDeviceTypeBuiltInTelephotoCamera
 7. AVCaptureDeviceTypeDeskViewCamera

ie the virtual "compound" cameras are favoured if available, but then ultra wide is preferred to the standard "1x" camera.

We make some approximate camera model assumptions in our sites assuming that the 1x camera will be the default for `facingMode: environment` scenarios.

I assume the compound cameras start up by default in a 1x mode, so I wouldn't mind those being preferred but in my view AVCaptureDeviceTypeBuiltInWideAngleCamera should be the next highest priority to limit behaviour change on other sites that don't have a UI for an explicit camera selection.

[Simon Taylor]

(In reply to Simon Taylor from comment #11)
> I assume the compound cameras start up by default in a 1x mode, so I
> wouldn't mind those being preferred but in my view
> AVCaptureDeviceTypeBuiltInWideAngleCamera should be the next highest
> priority to limit behaviour change on other sites that don't have a UI for
> an explicit camera selection.

@youenn Actually I think the virtual triple camera looks like it will be equivalent to the ultra-wide mode by default. Natively you need to set videoZoomFactor to 2 in order to match the for of the standard camera and automatically switch between standard and ultrawide based on focus.

Can you confirm if the default back camera now looks like the ultrawide on an iPhone that has that camera? I'm trying to confirm myself on a (freshly reset) iPhone 14 Pro here but am getting errors downloading the new 16.4 Beta 2 build.

https://bugs.webkit.org/show_bug.cgi?id=250142 has a bit more discussion.

Personally I think Safari should follow the native iOS API here and use just the standard front / back cameras by default, and only expose the other ones via enumerateDevices.

At least for our use cases the rear camera being ultrawide by default would likely break some of our assumptions in our computer vision code on hundreds of live sites.

[Simon Taylor]

I've added Bug 253186 for more discussion of the concerns around changing the default environment facing camera.

Just thought I'd add a note that this bug still appears to be present in iOS 16.4 Beta 3. It's making me nervous that it might end up in the shipping version. I know this isn't always possible but an insight into when a beta would contain the fix would be greatly appreciated.