Bug 235845

Summary: REGRESSION: iOS 15.4b1 – gl.texture2D uploads not working
Product: WebKit Reporter: rigel
Component: WebGLAssignee: Nobody <webkit-unassigned>
Status: RESOLVED CONFIGURATION CHANGED    
Severity: Critical CC: ahmad.saleem792, connell, dino, kbr, kkinnunen, nb, simontaylor1, webkit-bug-importer, youennf
Priority: P2 Keywords: InRadar
Version: Safari 15   
Hardware: iPhone / iPad   
OS: iOS 15   
Attachments:
Description Flags
Minimal WebGL Camera Pipeline none

Description rigel 2022-01-28 17:57:53 PST
In iOS 15.4b1 (tested on iPhone 11), using the captureStream() method to set the srcObject of the <video> element is not working, resulting in a black screen.

Minimum repro test: https://webrtc.github.io/samples/src/content/capture/video-video/
Expectation: the video on the left is streamed to the <video> on the right
Result: a black screen instead of the video stream

link to image: https://imgur.com/a/TCRANaP

As a result of this bug, all 8th Wall and Zappar sites are broken impacting major brands including: Marvel, LA Dodgers, Universal Pictures, MGM, Xbox, & More

Example 1: https://aircards.8thwall.app/dodgers/
Example 2: https://idareyou.candymanmovie.com/
More examples: https://www.8thwall.com/8thwall
Comment 1 youenn fablet 2022-01-29 06:56:30 PST
I do not think https://webrtc.github.io/samples/src/content/capture/video-video/ is expected to work in iOS. captureStream works for with a canvas source but is not implemented for video sources. The workaround is for now to use a canvas.
https://webrtc.github.io/samples/src/content/capture/canvas-video/ should be working.
@rigel, can you clarify?
Comment 2 youenn fablet 2022-01-29 06:57:28 PST
Especially can you clarify whether it is a regression or a feature request?
Comment 3 rigel 2022-01-29 21:05:01 PST
youenn, I just tested https://webrtc.github.io/samples/src/content/capture/video-video/ on an iOS 15.3 device and found the same behavior. I then tested https://webrtc.github.io/samples/src/content/capture/canvas-video/ on the 15.4b1 device and that test is working.

It seems this may not be the root cause of the black camera feed issue. The behavior of the example sites I listed appear the same as this bug marked as resolved: https://bugs.webkit.org/show_bug.cgi?id=230617.
Comment 4 rigel 2022-01-29 21:17:52 PST
To be clear, I am reporting a regression in 15.4b1, this is breaking all commercial websites I am aware of that rely on camera access and WebGL.

This link https://8th.io/fps works on iOS 15.3 but does not on iOS 15.4b1
Comment 5 Nicholas Butko 2022-01-30 23:14:55 PST
Created attachment 450383 [details]
Minimal WebGL Camera Pipeline
Comment 6 Nicholas Butko 2022-01-30 23:16:06 PST
I have attached a standalone repro case for this regression. For convenience, here is a link to a published version of the attached code:

https://8w.8thwall.app/webgl-camera

Per rigel@8thwall's testing:
- iOS 15.3 - Working: this sample renders a sepia camera feed.
- iOS 15.4 beta 1 - Regression: this sample just shows a white screen.

Note that because this sample uses getUserMedia, the attached files must be served with https.
Comment 7 Radar WebKit Bug Importer 2022-01-31 09:43:06 PST
<rdar://problem/88276617>
Comment 8 Kimmo Kinnunen 2022-01-31 12:22:14 PST
Thanks for checking! Bug 230617 did not seem to make it to seed 1. Leaving this open, I'll comment more once future releases are shipping.
Comment 9 rigel 2022-02-09 11:29:44 PST
I just tested the supplied sample link: https://8w.8thwall.app/webgl-camera on iOS 15.4b2, and while I'm no longer seeing a missing camera feed, I am now seeing collections of old ghosted frames flashing between new frames. This creates a strobe-like effect seen in the video below:

https://youtu.be/iPbo-7dkESk
Comment 10 Kimmo Kinnunen 2022-02-10 05:16:04 PST
(In reply to rigel from comment #9)
> I just tested the supplied sample link: https://8w.8thwall.app/webgl-camera
> on iOS 15.4b2, and while I'm no longer seeing a missing camera feed, I am
> now seeing collections of old ghosted frames flashing between new frames.

Thanks for re-testing. I can repro the issue in the seed build and it is fixed in trunk.

Was it the case that the test app uses the "draw the video frame to alternative textures at alternative iterations"? Unfortunately that triggers a long-standing edge case bug 203148. This bug is now fixed in trunk but it is not shipping in 15.4b2. 

Note, in case the app used the double-buffering scheme:

The suggested alternative, if it is only remotely possible, is to use only one texture, at least until the bug fix ships. This would be also good for memory use.


In bug 203148 given rationale for the double buffering is this:

> rAF #1:
> upload frame 1 to tex 1
> start processing data in tex 1
> rAF #2:
> upload frame 2 to tex 2
> start processing data in tex 2
> render from tex 1 (frame 1) along with processed result
> rAF #3:
> upload frame 3 to tex 1
> start processing data in tex 1
> render from tex 2 (frame 2) along with processed result


Given the rationale that the texture is uploaded in rAF, this is actually inferior strategy to uploading it to just one texture. The browsers implement the optimization to skip uploads if nothing changes. So in this scenario, uploading to alternating textures causes extra uploads.

The above is specific to initiating the video texture uploads via rAF: If the rAF completes at 60FPS and the camera at 24, 30 fps, then there is bound to be redundant uploads.

Trunk (and maybe the seed you refer to?) has support for requestVideoFrameCallback. So theoretically with that, one could make the case for the double-buffering. However, I'm not entirely convinced that the double-buffering uploads should be used.
Comment 11 Kimmo Kinnunen 2022-02-10 05:28:21 PST
At least the attached minimal webgl camera pipeline test-case uses
`PIPELINE_SIZE = 10`. This would trigger bug 203148.

So on iOS, the frame rate would typically default to 30fps, and rAF is 60FPS, so from the get-go the pipeline is filled with 5 identical textures.

If you want to necessarily use multiple textures, one approach to test would be to upload 1 pixel sized sub image to each texture before uploading the video element. This would clear the buggy cache. I know it's a bit inconvenient and, probably it is quite a bit extra overhead.
Comment 12 Kimmo Kinnunen 2022-02-10 05:31:48 PST
> So on iOS, the frame rate would typically default to 30fps, and rAF is
> 60FPS, so from the get-go the pipeline is filled with 5 identical textures.

That went wrong.
The pipeline contains, on average, 5 unique frames.
Comment 13 Nicholas Butko 2022-02-10 11:16:11 PST
The rationale for a multi-texture pipeline (not double-buffering per se) is laid out above. Keeping a pipeline of textures is a requirement. In order to power the same pipeline with a single texture in getUserMedia would require a framebuffer pipeline instead, adding an extra unneeded draw-with-shader to copy the full resolution texture to a framebuffer in addition to the texture upload. Is there another pipelining paradigm that's expected to be more efficient?

video.currentTime increments by ~16ms per frame. If the framerate were lower than 60FPS, we would expect that this should not increment.

Setting `framerate: {min: 60}` improves the current sample somewhat, but still gives unacceptable visible artifacts which are also not present in iOS 15.3.

We had seen similar issues previously, which we had noted in https://bugs.webkit.org/show_bug.cgi?id=206636. It was suggested in Jan 2020 that this had been fixed, and in our testing this issue no longer reproduced in iOS 14 and above. We have since removed our workarounds for iOS Safari versions 14 and higher.

The current issue is a REGRESSION from 15.3 to 15.4 beta. The hypothesized cause for the issue only accounts for the observed regressions if 15.3 had a framerate that was always higher than 60FPS AND 15.4 beta sets a lower default AND doesn't reliably provide 60FPS at 60FPS capture.

Regardless, this is a business critical regression in iOS 15.4 beta. If the fix for https://bugs.webkit.org/show_bug.cgi?id=203148 is also expected to fix this issue, we look forward to testing in upcoming versions of iOS 15.4 beta.
Comment 14 Kimmo Kinnunen 2022-02-15 01:19:18 PST
(In reply to Nicholas Butko from comment #13)
> still gives unacceptable visible artifacts which are also not present in iOS
> 15.3.
> The current issue is a REGRESSION from 15.3 to 15.4 beta.

You are right -- while the same bug has existed for video, it would be unfortunate if this performance improvement would bring it to user media too. I'll see what I can do.
Comment 15 Simon Taylor 2022-02-15 03:32:33 PST
That makes sense - applying the logic to skip unnecessary uploads to user media without fixing Bug 203148 too may have made this visible on sites where it wasn't before.

Kimmo's suggested texSubImage2d workaround does seem to force a "full" upload of the video, at least in the mp4 case from 203148 (tested on 15.2 only for now). Checking with Systrace in Instruments it looks to add minimal overhead.

// A 1-pixel array buffer
let greenPixel = new Uint8Array([0, 128, 0, 255]);

// Inside rAF loop
  gl.bindTexture(gl.TEXTURE_2D, tex2);
+ gl.texSubImage2D(gl.TEXTURE_2D, 0, 0, 0, 1, 1, gl.RGBA, gl.UNSIGNED_BYTE, greenPixel);
  gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGB, gl.RGB, gl.UNSIGNED_BYTE, video);

I'm going to add support for user media in my test case and then update to 15.4b2 to see what's changed. Will also check out requestVideoFrameCallback, that would save unnecessary uploads at all, as well as helping to avoid 203148.
Comment 16 Simon Taylor 2022-02-15 11:55:00 PST
I've got 15.4b2 installed now (iPhone 12 Pro).

As suspected, Bug 203148 does now also apply to video elements with a getUserMedia MediaStream. On the plus side, the actual texImage2D is much faster now, and the workaround of setting a pixel with texSubImage2D does allow the same frame to be uploaded to multiple textures.

So not too bad overall for user media streams.

However there is a *catastrophic* performance regression with the texImage2d with the normal mp4 source. It's over 40ms per upload now! I haven't done a search to see if there's a specific bug for that. I also don't have beta Xcode installed yet so can't take a look with Systrace.
Comment 17 Nicholas Butko 2022-02-15 14:00:20 PST
(In reply to Simon Taylor from comment #16)
> the workaround of setting a pixel with texSubImage2D does
> allow the same frame to be uploaded to multiple textures.

Thanks for confirming this workaround.

> So not too bad overall for user media streams.

Since this is a regression, all websites that don't upgrade to the above workaround will see this bug. Many commercial customers choose to freeze the version of javascript libraries that their sites use for enhanced reliability and stability. Without a fix browser level fix for this regression, any such experience would suddenly break.
Comment 18 rigel 2022-02-15 15:22:39 PST
I just updated a test device to the latest beta and this regression is still present in 15.4b3.
Comment 19 Simon Taylor 2022-02-16 02:25:29 PST
(In reply to Nicholas Butko from comment #17)
> 
> > So not too bad overall for user media streams.
> 
> Since this is a regression, all websites that don't upgrade to the above
> workaround will see this bug. Many commercial customers choose to freeze the
> version of javascript libraries that their sites use for enhanced
> reliability and stability. Without a fix browser level fix for this
> regression, any such experience would suddenly break.

Yes of course, I'm with you that patching all of our live sites would be a massive pain, and many previously-working sites would instantly break.

I just wanted to add a slight note of positivity that I see and appreciate the work to make MediaStream uploads faster too - it's 5x or more faster in 15.4b2 than previously in iOS 15, and now there is a patch we can apply to get our sites to work - it's a strictly better situation than beta 1, though of course still not ideal yet.

Kimmo's acknowledged it would be unfortunate to ship with this regression, so I'm still hoping it will get resolved in time. I suspect the "texture pool" pattern is pretty common in sites applying JS/Wasm processing to MediaStreams.

I've added Bug 236699 to track the mp4 texImage2D regression, so we can keep this one focussed on the MediaStream case.
Comment 20 Simon Taylor 2022-02-28 02:20:37 PST
I'm pleased to report this is now fixed in the latest 15.4 beta 4.

Thanks Kimmo and team for getting this resolved before 15.4!
Comment 21 Ahmad Saleem 2022-10-08 12:19:50 PDT
Based on Comment 20, it seems that the issue is fixed. Hence, marking this as "RESOLVED CONFIGURATION CHANGED".

Please reopen, if it is still not working. Thanks!