Bug 265556 – REGRESSION (iOS 17): Front and Back Facing Camera Stutter

• Status: RESOLVED
• Resolution: FIXED
• Component: WebGL

Created attachment 468813 [details]
Front-facing camera stutter

This is a regression between iOS 16 and 17. 

It appears that the front-facing and back-facing Cameras via getUserMedia results in frame stutters on certain page refreshes. 
This change in behavior is problematic as it impacts computer vision applications on the web, and there is no way to prevent the stuttering. 

Our testing on devices (iPhone 13 Pro & iPhone 11) with iOS 16 reveals that this behavior does not occur prior to iOS 17. 

I've attached a minimal example video of the issue to this report. 

Reproduction Steps:

1) Open: https://8thwall.8thwall.app/face-effects-aframe/
2) Allow camera and microphone permissions
3) Move the phone and/or your head around
4) Observe that the camera feed stutters from time to time, particularly right after page refresh.
5) If you don't observe stuttering, reload the page until it occurs.

Created attachment 468828 [details]
Back Camera Stutter Behavior

[Radar WebKit Bug Importer]

<rdar://problem/119076271>

Created attachment 469026 [details]
Updated stutter video with debug tool showing camera feed frames being glitched

You can try out the experience yourself at https://8w.8thwall.app/frame-bug-slam/

[youenn fablet]

I am not able to reproduce on a recent iOS build with an iPhone 12 Pro.

[Kimmo Kinnunen]

Thank you for the report.
Would you be able to confirm couple of things in case you have time to work on this further before we are able to repro this:

If you attach the video element to the screen and show it side-by-side the WebGL canvas, you should be able to prove whether the stutter happens due to WebGL or due to video input.


Do you have multiple WebGL textures where you upload the video texture and use them in round-robin fashion? If so, you may try to use only one texture to workaround. It might be that the WebGL video texture content id cache is buggy in the round-robin case.

Created attachment 469044 [details]
video element + webGL canvas example 1

Created attachment 469045 [details]
video element + webGL canvas example 2

Hi all, thank you for looking into this and testing.

We have been successful in reproing with iPhone 13 Pro and iPhone 14 Pro but unsuccessful with iPhone 12.

Thank you for the debugging suggestion, we have uploaded two videos showing our findings. The top left element is showing a video html element from which we capture the content into webgl. Normally, this element has a minuscule size and opacity and is not perceivable by the user. You can see that the video does not have frame stutter as seen in the content.

Is there a recommended way to capture the last N frames in N WebGL texture in a round-robin fashion without incurring this content cache issue?

[Kimmo Kinnunen]

Pull request: https://github.com/WebKit/WebKit/pull/21860

[Kimmo Kinnunen]

Thanks for the debug help,  it indeed is the texture contents video upload caching logic that is the problem.

> Is there a recommended way to capture the last N frames in N WebGL texture in a round-robin fashion without incurring this content cache issue?

You can avoid the issue by not reusing the texture: destroy the texture you're currently reusing, and create a new one.

If that does not perform well enough, you can try to update the reused texture: use texSubImage and upload one pixel before uploading the video texture.

[Kimmo Kinnunen]

>If that does not perform well enough, you can try to update the reused texture: use texSubImage and upload one pixel before uploading the video texture.

If this does not perform well enough, you can try to bind the texture to a fbo and unbind it straight away.

[EWS]

Committed 272263@main (98d2c991bb5d): <https://commits.webkit.org/272263@main>

Reviewed commits have been landed. Closing PR #21860 and removing active labels.