Bug 226830 – texImage2D is slow with HTMLVideoObjects from getUserMedia

• Status: NEW
• Component: WebGL

[Paris Morgan]

Created attachment 430980 [details]
Results of the benchmark script on three different devices.

Getting HTMLVideoObject pixels from getUserMedia into a WebGL texture with texImage2D is slow. 

Repro app: https://8w.8thwall.app/performance-test
Repro code: https://codepen.io/jparismorgan/pen/vYxrapM

This sample code assigns the stream from getUserMedia() to a HTMLMediaElement.srcObject, then renders it to a texture with texImage2D. texImage2D is quite slow - it takes 6.904 ms on an iPhone 12 Pro running iOS 14.4.2 with WebGL 1. On an iPhone 12 Pro Max with iOS 15.0 with WebGL 1 and "WebGL via Metal"=true it takes 2.503 ms. 
On a Galaxy S10 it takes 0.422 ms. In the wild (on pages that do more GPU processing and rendering), we typically see this taking even longer. For example typical realistic in the wild values for an iPhone 12 Pro are 7.5 ms.

I also tested two other cases:
  1) Setting gl.pixelStorei(gl.UNPACK_PREMULTIPLY_ALPHA_WEBGL, true) before texImage2D, which speeds up the call slightly, i.e. from 6.904 ms down to 4.454 ms.
  2) Instead of calling texImage2D with a HTMLVideoElement, call it with an ArrayBufferView. This makes texImage2D faster - from 6.904 ms down to 0.401 ms. But this is not possible in practice because going from getUserMedia to an array buffer is very slow.

The repro above shows off the main path as well as these two modifications. Also attached are screenshots, and here is the data:

                                              iPhone 12 Pro iOS 14.4.2 WebGL 1 | iPhone 12 Pro Max iOS 15.0 WebGL 1 & "WebGL via Metal"=true | Galaxy S10 WebGL 1
texImage2D                                    6.904 ms                         | 4.297 ms                                                    | 0.407 ms
texImage2D + UNPACK_PREMULTIPLY_ALPHA_WEBGL   4.454 ms                         | 2.963 ms                                                    | 0.546 ms
texImage2D (from ArrayBufferView)             0.401 ms                         | 0.840 ms                                                    | 0.518 ms

Please let me know if I can provide more info, thank you.

[Paris Morgan]

Minor corrections:
- It was an iPhone 12 Pro running iOS 14.6, not 14.4.2.
- It was a Galaxy S10+, not a Galaxy S10.

[Kimmo Kinnunen]

Thanks for the report, I'll try to debug what's the holdup.

[Radar WebKit Bug Importer]

<rdar://problem/79406380>