Bug 265556

Summary: REGRESSION (iOS 17): Front and Back Facing Camera Stutter
Product: WebKit Reporter: joshmao
Component: WebGLAssignee: Kimmo Kinnunen <kkinnunen>
Status: RESOLVED FIXED    
Severity: Normal CC: dino, kbr, kkinnunen, webkit-bug-importer, youennf
Priority: P2 Keywords: InRadar
Version: Safari 17   
Hardware: iPhone / iPad   
OS: iOS 17   
Attachments:
Description Flags
Front-facing camera stutter
none
Back Camera Stutter Behavior
none
Updated stutter video with debug tool showing camera feed frames being glitched
none
video element + webGL canvas example 1
none
video element + webGL canvas example 2 none

joshmao
Reported 2023-11-29 16:55:38 PST
Created attachment 468813 [details] Front-facing camera stutter This is a regression between iOS 16 and 17. It appears that the front-facing and back-facing Cameras via getUserMedia results in frame stutters on certain page refreshes. This change in behavior is problematic as it impacts computer vision applications on the web, and there is no way to prevent the stuttering. Our testing on devices (iPhone 13 Pro & iPhone 11) with iOS 16 reveals that this behavior does not occur prior to iOS 17. I've attached a minimal example video of the issue to this report. Reproduction Steps: 1) Open: https://8thwall.8thwall.app/face-effects-aframe/ 2) Allow camera and microphone permissions 3) Move the phone and/or your head around 4) Observe that the camera feed stutters from time to time, particularly right after page refresh. 5) If you don't observe stuttering, reload the page until it occurs.
Attachments
Front-facing camera stutter (3.97 MB, video/quicktime)
2023-11-29 16:55 PST, joshmao
no flags
Back Camera Stutter Behavior (6.49 MB, video/quicktime)
2023-11-30 14:31 PST, joshmao
no flags
Updated stutter video with debug tool showing camera feed frames being glitched (7.37 MB, video/quicktime)
2023-12-13 15:18 PST, joshmao
no flags
video element + webGL canvas example 1 (15.29 MB, video/quicktime)
2023-12-14 13:33 PST, joshmao
no flags
video element + webGL canvas example 2 (23.82 MB, video/quicktime)
2023-12-14 13:35 PST, joshmao
no flags
joshmao
Comment 1 2023-11-30 14:31:01 PST
Created attachment 468828 [details] Back Camera Stutter Behavior
Radar WebKit Bug Importer
Comment 2 2023-12-02 10:53:32 PST
joshmao
Comment 3 2023-12-13 15:18:15 PST
Created attachment 469026 [details] Updated stutter video with debug tool showing camera feed frames being glitched You can try out the experience yourself at https://8w.8thwall.app/frame-bug-slam/
youenn fablet
Comment 4 2023-12-14 02:19:02 PST
I am not able to reproduce on a recent iOS build with an iPhone 12 Pro.
Kimmo Kinnunen
Comment 5 2023-12-14 04:03:47 PST
Thank you for the report. Would you be able to confirm couple of things in case you have time to work on this further before we are able to repro this: If you attach the video element to the screen and show it side-by-side the WebGL canvas, you should be able to prove whether the stutter happens due to WebGL or due to video input. Do you have multiple WebGL textures where you upload the video texture and use them in round-robin fashion? If so, you may try to use only one texture to workaround. It might be that the WebGL video texture content id cache is buggy in the round-robin case.
joshmao
Comment 6 2023-12-14 13:33:58 PST
Created attachment 469044 [details] video element + webGL canvas example 1
joshmao
Comment 7 2023-12-14 13:35:40 PST
Created attachment 469045 [details] video element + webGL canvas example 2
joshmao
Comment 8 2023-12-14 13:37:52 PST
Hi all, thank you for looking into this and testing. We have been successful in reproing with iPhone 13 Pro and iPhone 14 Pro but unsuccessful with iPhone 12. Thank you for the debugging suggestion, we have uploaded two videos showing our findings. The top left element is showing a video html element from which we capture the content into webgl. Normally, this element has a minuscule size and opacity and is not perceivable by the user. You can see that the video does not have frame stutter as seen in the content. Is there a recommended way to capture the last N frames in N WebGL texture in a round-robin fashion without incurring this content cache issue?
Kimmo Kinnunen
Comment 9 2023-12-15 04:45:19 PST
Kimmo Kinnunen
Comment 10 2023-12-15 04:56:58 PST
Thanks for the debug help, it indeed is the texture contents video upload caching logic that is the problem. > Is there a recommended way to capture the last N frames in N WebGL texture in a round-robin fashion without incurring this content cache issue? You can avoid the issue by not reusing the texture: destroy the texture you're currently reusing, and create a new one. If that does not perform well enough, you can try to update the reused texture: use texSubImage and upload one pixel before uploading the video texture.
Kimmo Kinnunen
Comment 11 2023-12-15 05:04:24 PST
>If that does not perform well enough, you can try to update the reused texture: use texSubImage and upload one pixel before uploading the video texture. If this does not perform well enough, you can try to bind the texture to a fbo and unbind it straight away.
EWS
Comment 12 2023-12-19 05:50:59 PST
Committed 272263@main (98d2c991bb5d): <https://commits.webkit.org/272263@main> Reviewed commits have been landed. Closing PR #21860 and removing active labels.
Note You need to log in before you can comment on or make changes to this bug.