Bug 235845 – REGRESSION: iOS 15.4b1 – gl.texture2D uploads not working

• Component: WebGL
• Status: RESOLVED CONFIGURATION CHANGED

In iOS 15.4b1 (tested on iPhone 11), using the captureStream() method to set the srcObject of the <video> element is not working, resulting in a black screen.

Minimum repro test: https://webrtc.github.io/samples/src/content/capture/video-video/
Expectation: the video on the left is streamed to the <video> on the right
Result: a black screen instead of the video stream

link to image: https://imgur.com/a/TCRANaP

As a result of this bug, all 8th Wall and Zappar sites are broken impacting major brands including: Marvel, LA Dodgers, Universal Pictures, MGM, Xbox, & More

Example 1: https://aircards.8thwall.app/dodgers/
Example 2: https://idareyou.candymanmovie.com/
More examples: https://www.8thwall.com/8thwall

[youenn fablet]

I do not think https://webrtc.github.io/samples/src/content/capture/video-video/ is expected to work in iOS. captureStream works for with a canvas source but is not implemented for video sources. The workaround is for now to use a canvas.
https://webrtc.github.io/samples/src/content/capture/canvas-video/ should be working.
@rigel, can you clarify?

[youenn fablet]

Especially can you clarify whether it is a regression or a feature request?

youenn, I just tested https://webrtc.github.io/samples/src/content/capture/video-video/ on an iOS 15.3 device and found the same behavior. I then tested https://webrtc.github.io/samples/src/content/capture/canvas-video/ on the 15.4b1 device and that test is working.

It seems this may not be the root cause of the black camera feed issue. The behavior of the example sites I listed appear the same as this bug marked as resolved: https://bugs.webkit.org/show_bug.cgi?id=230617.

To be clear, I am reporting a regression in 15.4b1, this is breaking all commercial websites I am aware of that rely on camera access and WebGL.

This link https://8th.io/fps works on iOS 15.3 but does not on iOS 15.4b1

[Nicholas Butko]

Created attachment 450383 [details]
Minimal WebGL Camera Pipeline

[Nicholas Butko]

I have attached a standalone repro case for this regression. For convenience, here is a link to a published version of the attached code:

https://8w.8thwall.app/webgl-camera

Per rigel@8thwall's testing:
- iOS 15.3 - Working: this sample renders a sepia camera feed.
- iOS 15.4 beta 1 - Regression: this sample just shows a white screen.

Note that because this sample uses getUserMedia, the attached files must be served with https.

[Radar WebKit Bug Importer]

<rdar://problem/88276617>

[Kimmo Kinnunen]

Thanks for checking! Bug 230617 did not seem to make it to seed 1. Leaving this open, I'll comment more once future releases are shipping.

I just tested the supplied sample link: https://8w.8thwall.app/webgl-camera on iOS 15.4b2, and while I'm no longer seeing a missing camera feed, I am now seeing collections of old ghosted frames flashing between new frames. This creates a strobe-like effect seen in the video below:

https://youtu.be/iPbo-7dkESk

[Kimmo Kinnunen]

(In reply to rigel from comment #9)
> I just tested the supplied sample link: https://8w.8thwall.app/webgl-camera
> on iOS 15.4b2, and while I'm no longer seeing a missing camera feed, I am
> now seeing collections of old ghosted frames flashing between new frames.

Thanks for re-testing. I can repro the issue in the seed build and it is fixed in trunk.

Was it the case that the test app uses the "draw the video frame to alternative textures at alternative iterations"? Unfortunately that triggers a long-standing edge case bug 203148. This bug is now fixed in trunk but it is not shipping in 15.4b2. 

Note, in case the app used the double-buffering scheme:

The suggested alternative, if it is only remotely possible, is to use only one texture, at least until the bug fix ships. This would be also good for memory use.


In bug 203148 given rationale for the double buffering is this:

> rAF #1:
> upload frame 1 to tex 1
> start processing data in tex 1
> rAF #2:
> upload frame 2 to tex 2
> start processing data in tex 2
> render from tex 1 (frame 1) along with processed result
> rAF #3:
> upload frame 3 to tex 1
> start processing data in tex 1
> render from tex 2 (frame 2) along with processed result


Given the rationale that the texture is uploaded in rAF, this is actually inferior strategy to uploading it to just one texture. The browsers implement the optimization to skip uploads if nothing changes. So in this scenario, uploading to alternating textures causes extra uploads.

The above is specific to initiating the video texture uploads via rAF: If the rAF completes at 60FPS and the camera at 24, 30 fps, then there is bound to be redundant uploads.

Trunk (and maybe the seed you refer to?) has support for requestVideoFrameCallback. So theoretically with that, one could make the case for the double-buffering. However, I'm not entirely convinced that the double-buffering uploads should be used.

[Kimmo Kinnunen]

At least the attached minimal webgl camera pipeline test-case uses
`PIPELINE_SIZE = 10`. This would trigger bug 203148.

So on iOS, the frame rate would typically default to 30fps, and rAF is 60FPS, so from the get-go the pipeline is filled with 5 identical textures.

If you want to necessarily use multiple textures, one approach to test would be to upload 1 pixel sized sub image to each texture before uploading the video element. This would clear the buggy cache. I know it's a bit inconvenient and, probably it is quite a bit extra overhead.

[Kimmo Kinnunen]

> So on iOS, the frame rate would typically default to 30fps, and rAF is
> 60FPS, so from the get-go the pipeline is filled with 5 identical textures.

That went wrong.
The pipeline contains, on average, 5 unique frames.

[Nicholas Butko]

The rationale for a multi-texture pipeline (not double-buffering per se) is laid out above. Keeping a pipeline of textures is a requirement. In order to power the same pipeline with a single texture in getUserMedia would require a framebuffer pipeline instead, adding an extra unneeded draw-with-shader to copy the full resolution texture to a framebuffer in addition to the texture upload. Is there another pipelining paradigm that's expected to be more efficient?

video.currentTime increments by ~16ms per frame. If the framerate were lower than 60FPS, we would expect that this should not increment.

Setting `framerate: {min: 60}` improves the current sample somewhat, but still gives unacceptable visible artifacts which are also not present in iOS 15.3.

We had seen similar issues previously, which we had noted in https://bugs.webkit.org/show_bug.cgi?id=206636. It was suggested in Jan 2020 that this had been fixed, and in our testing this issue no longer reproduced in iOS 14 and above. We have since removed our workarounds for iOS Safari versions 14 and higher.

The current issue is a REGRESSION from 15.3 to 15.4 beta. The hypothesized cause for the issue only accounts for the observed regressions if 15.3 had a framerate that was always higher than 60FPS AND 15.4 beta sets a lower default AND doesn't reliably provide 60FPS at 60FPS capture.

Regardless, this is a business critical regression in iOS 15.4 beta. If the fix for https://bugs.webkit.org/show_bug.cgi?id=203148 is also expected to fix this issue, we look forward to testing in upcoming versions of iOS 15.4 beta.

[Kimmo Kinnunen]

(In reply to Nicholas Butko from comment #13)
> still gives unacceptable visible artifacts which are also not present in iOS
> 15.3.
> The current issue is a REGRESSION from 15.3 to 15.4 beta.

You are right -- while the same bug has existed for video, it would be unfortunate if this performance improvement would bring it to user media too. I'll see what I can do.

[Simon Taylor]

That makes sense - applying the logic to skip unnecessary uploads to user media without fixing Bug 203148 too may have made this visible on sites where it wasn't before.

Kimmo's suggested texSubImage2d workaround does seem to force a "full" upload of the video, at least in the mp4 case from 203148 (tested on 15.2 only for now). Checking with Systrace in Instruments it looks to add minimal overhead.

// A 1-pixel array buffer
let greenPixel = new Uint8Array([0, 128, 0, 255]);

// Inside rAF loop
  gl.bindTexture(gl.TEXTURE_2D, tex2);
+ gl.texSubImage2D(gl.TEXTURE_2D, 0, 0, 0, 1, 1, gl.RGBA, gl.UNSIGNED_BYTE, greenPixel);
  gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGB, gl.RGB, gl.UNSIGNED_BYTE, video);

I'm going to add support for user media in my test case and then update to 15.4b2 to see what's changed. Will also check out requestVideoFrameCallback, that would save unnecessary uploads at all, as well as helping to avoid 203148.

[Simon Taylor]

I've got 15.4b2 installed now (iPhone 12 Pro).

As suspected, Bug 203148 does now also apply to video elements with a getUserMedia MediaStream. On the plus side, the actual texImage2D is much faster now, and the workaround of setting a pixel with texSubImage2D does allow the same frame to be uploaded to multiple textures.

So not too bad overall for user media streams.

However there is a *catastrophic* performance regression with the texImage2d with the normal mp4 source. It's over 40ms per upload now! I haven't done a search to see if there's a specific bug for that. I also don't have beta Xcode installed yet so can't take a look with Systrace.

[Nicholas Butko]

(In reply to Simon Taylor from comment #16)
> the workaround of setting a pixel with texSubImage2D does
> allow the same frame to be uploaded to multiple textures.

Thanks for confirming this workaround.

> So not too bad overall for user media streams.

Since this is a regression, all websites that don't upgrade to the above workaround will see this bug. Many commercial customers choose to freeze the version of javascript libraries that their sites use for enhanced reliability and stability. Without a fix browser level fix for this regression, any such experience would suddenly break.

I just updated a test device to the latest beta and this regression is still present in 15.4b3.

[Simon Taylor]

(In reply to Nicholas Butko from comment #17)
> 
> > So not too bad overall for user media streams.
> 
> Since this is a regression, all websites that don't upgrade to the above
> workaround will see this bug. Many commercial customers choose to freeze the
> version of javascript libraries that their sites use for enhanced
> reliability and stability. Without a fix browser level fix for this
> regression, any such experience would suddenly break.

Yes of course, I'm with you that patching all of our live sites would be a massive pain, and many previously-working sites would instantly break.

I just wanted to add a slight note of positivity that I see and appreciate the work to make MediaStream uploads faster too - it's 5x or more faster in 15.4b2 than previously in iOS 15, and now there is a patch we can apply to get our sites to work - it's a strictly better situation than beta 1, though of course still not ideal yet.

Kimmo's acknowledged it would be unfortunate to ship with this regression, so I'm still hoping it will get resolved in time. I suspect the "texture pool" pattern is pretty common in sites applying JS/Wasm processing to MediaStreams.

I've added Bug 236699 to track the mp4 texImage2D regression, so we can keep this one focussed on the MediaStream case.

[Simon Taylor]

I'm pleased to report this is now fixed in the latest 15.4 beta 4.

Thanks Kimmo and team for getting this resolved before 15.4!

[Ahmad Saleem]

Based on Comment 20, it seems that the issue is fixed. Hence, marking this as "RESOLVED CONFIGURATION CHANGED".

Please reopen, if it is still not working. Thanks!