Bug 216250 – Major WebGL video texture performance regression in iOS 14 beta

• Component: WebGL
• Status: RESOLVED DUPLICATE

[Simon Taylor]

Based on the current iOS 14 beta (beta 7) the performance of texImage2d from a video element has suffered a significant regression since iOS 13.

Some examples pages that have the very simplest shader and render loop for getting a 720p video into a WebGL canvas:
https://tango-bravo.net/WebGLVideoPerformance/texImage2d_video.html (uses gl.RGB format and internal format)
https://tango-bravo.net/WebGLVideoPerformance/texImage2d_video_rgba.html (uses gl.RGBA format and internal format)

On an iPod 7th gen running iOS 13.6, I've observed the reported total average draw time under 2ms for both of these and it has no difficulty maintaining a 60FPS render loop (aside: iOS seems to downclock when hitting 60FPS; opening the app switcher and interacting with the page (scrolling, zooming, etc) looks to bring the timings down).

With an iPhone XR on the iOS 14 beta 7, things are significantly worse. With the first example the render loop drops under 60 FPS (and the bulk of the time seems to be spent in drawElements). With the RGBA format (second link) 60 FPS is maintained but the timings remain high (above 7ms).

[Radar WebKit Bug Importer]

<rdar://problem/68513491>

[Alexey Proskuryakov]

Could you please clarify if this affects user facing websites or apps?

[Simon Taylor]

As demonstrated by the test case, any page making use of video textures for WebGL will suffer a performance regression.

Here's a user-facing site that is affected. It's a regression test (with a 1080p video) for our web-based AR platform that shows very poor performance in the beta on an iPhone XR, but is perfectly smooth on the iOS 13 iPod 7th gen.
https://web.zappar.com/?zid=z/wwMc1c

Both camera and video textures go via texImage2d from video elements. The black flashing background appears to be another WebGL related regression in iOS 14 that I'm currently attempting to make a minimal test case for so I can submit another bug report.

Most higher-res video content would use HLS, so clearly Bug 215908 is a bigger priority and breaks many live WebGL sites. However it seems whatever changes have been made to the WebGL video texture pipeline even outside of HLS have a significant performance impact too.

[Dustin Kerstein]

I can also replicate this regression on https://my.panomoments.com/u/dustinkerstein/m/grand-central using a first generation iPad Pro on the iPadOS 14 GM. With the UHD and HD resolutions the render framerate is about 25-50% of what I see on iPadOS 13.

[Dustin Kerstein]

Here is a pure WebGL-based JSFiddle that should replicate on iOS/iPadOS (though you should make sure to click into the preview window to avoid an iFrame iOS power saving feature) - https://jsfiddle.net/1obyw68k

I believe I'm also able to replicate this regression in YouTube's 360 viewer, but I no longer have a test device on iOS 13 so I can't directly compare performance.

[Dustin Kerstein]

Here's a more generic WebGL texture upload test using a benchmark video that contains solid colors - https://jsfiddle.net/qz9ka6xm

The test uses gl.readPixels to count how many unique colors are rendered and the average upload time. I'm only able to get the 1080p test running on my iOS devices (I believe due to the aspect ratio of the 2160p file going beyond the mobile WebGL texture size limits - I can fix that if so desired), but I'm able to replicate the regression using the 1080p file:

iPadOS 14 GM on iPad Pro ~21fps 
iOS 12.4.8 on iPhone 6 Plus ~24fps

[Dustin Kerstein]

Quick update, I can also replicate this regression in the current iPadOS 14.2 Beta (18B5052h)

We are seeing this behavior as well and can be reproduced here: https://8w.8thwall.app/ios14-video-perf

I've included videos of this test site running without issue on the older iPhone XS Max running iOS. 13.7 (https://www.youtube.com/watch?v=MLNgYmj-_5s) but drops to half the frame rate on the newer
iPhone 11 running iOS 14.0.1 (https://www.youtube.com/watch?v=L-v0q7iiY7E).

[Kimmo Kinnunen]

Thanks for the reports and testcases. This performance regression was due to video -> texture teximage2d failing the HW code path and reverting to SW conversion. It is the same root cause as bug 215908.

*** This bug has been marked as a duplicate of bug 215908 ***

[Simon Taylor]

I think this may have regressed in iOS 15.0.

I initially reported and iPod Touch 7 in iOS 13.6 having total average draw time under 2ms on the test pages below. On that same device on 15.0 it is now typically over 7ms, and I've never seen it drop under 5ms.

https://tango-bravo.net/WebGLVideoPerformance/texImage2d_video.html (uses gl.RGB format and internal format)
https://tango-bravo.net/WebGLVideoPerformance/texImage2d_video_rgba.html (uses gl.RGBA format and internal format)

I've interacted with the app switcher and page zooming / scrolling etc to try to keep clocks high but it's possible this change could be scheduler related. Worth investigating again though.

Another potentially relevant bit of info - the test case for https://bugs.webkit.org/show_bug.cgi?id=203148 is working on iOS 15.0 now. However before celebrating that it had been fixed I noticed the upload times looked high, so perhaps just another symptom of texUpload having dropping off the GPU path.

Some tests to verify the GPU path is taken when expected would be most welcome as it seems pretty easy to cause regressions here.

[Dustin Kerstein]

Simon, I believe this may also be related to a native code iOS / iPadOS 15 issue. I also originally thought it was just a WebKit / Safari issue, but I was able to replicate a playback regression with the native video player. Here was my original post on WebKit - https://bugs.webkit.org/show_bug.cgi?id=229413

I have since added two feedback entries for both iOS / iPadOS 15 Beta 7 and 9, and on iOS / iPadOS 15.1 Beta 1, but I have not gotten any update from Apple, and there's no way to share those links publicly unfortunately. You should be able to easily replicate the regression with this file - https://files.panomoments.com/bbb_sunflower_2160p_60fps_normal.mp4 - either played in Safari or locally once downloaded using the Files app. On iOS 15 and 15.1 Beta 1 I notice major frame drops at the beginning of the file when played back locally in the Files app, and when seeking to various points in the timeline. Anecdotally, I have seen several similar reports across the web, but nothing official in terms of a response / acknowledgment from Apple.

[Alexey Proskuryakov]

> I think this may have regressed in iOS 15.0.

Please file a new bug. While it is certainly useful to have the context about this same test having regressed in the past, this would be a new issue that needs to be investigated separately.

> there's no way to share those links publicly unfortunately

Do you have the FBxxxxxxx feedback numbers? I can check to make sure that the reports were prioritized correctly.

[Dustin Kerstein]

I'll let Simon file a new bug specifically about the webgl behavior. 

Thanks Alexey. Here are the two issues I reported via the Feedback Assistant tool: FB9666426 and FB9554184 - FYI, this behavior resulted in broken playback for PanoMoments.com and required a poor performing workaround across all iOS devices (as I can't get OS version reliably from the user agent). I'm sure there are other native and web video-based apps that are affected as well. Let me know if you need any further info. I'd be happy to setup a test page that has the original code without the workaround.

[Simon Taylor]

(In reply to Alexey Proskuryakov from comment #12)
> > I think this may have regressed in iOS 15.0.
> 
> Please file a new bug. While it is certainly useful to have the context
> about this same test having regressed in the past, this would be a new issue
> that needs to be investigated separately.

OK, done as Bug 231031 - looks related to the mixed GPU Process experimental settings for Media and WebGL (which are "On" and "Off" respectively by default).

Thanks for the additional comments Dustin, definitely sounds plausible there's an underlying native regression too, but it does seem like the main Safari part is being off the GPU path (yet again) by default in iOS 15.

The "RGB slower than RGBA even in HW path" I think is new with the Metal backend, but I'm not very sure about that and frankly don't really have the time or inclination to dig any further into it - I'd of course welcome any Apple folks doing that.

[Dustin Kerstein]

(In reply to Alexey Proskuryakov from comment #12)
> Do you have the FBxxxxxxx feedback numbers? I can check to make sure that
> the reports were prioritized correctly.

Alexey, were you able to look into those feedback numbers? I actually submitted another one, as I think the core issue is a bit more widespread (maybe in VideoToolbox) - FB9688897 - Let me know if there's anything else I can get you.

[Alexey Proskuryakov]

Just took another look to make sure that they are tracked properly.

> Alexey, were you able to look into those feedback numbers? I actually submitted another one, as I think the core issue is a bit more widespread (maybe in VideoToolbox) - FB9688897

I feel like we are talking about multiple separate issues here, and all of them different from what this bug was about, as it's been fixed. I do not have sufficient WebGL or video expertise to triage or explain them.

> Let me know if there's anything else I can get you.

Adding confirmed details about when an issue started to occur is generally very useful, and so is explaining the impact at the highest level (e.g. this issue affects N customers, there is or isn't a workaround that can be made).

[Dustin Kerstein]

(In reply to Alexey Proskuryakov from comment #16)
> I feel like we are talking about multiple separate issues here, and all of
> them different from what this bug was about, as it's been fixed. I do not
> have sufficient WebGL or video expertise to triage or explain them.

Ok, is there anyone at Apple with VideoToolbox expertise that we could loop in?

> Adding confirmed details about when an issue started to occur is generally
> very useful, and so is explaining the impact at the highest level (e.g. this
> issue affects N customers, there is or isn't a workaround that can be made).

I will add that to my Feedback Assistant issues. In my opinion, I think this issue is extremely widespread (VideoToolbox decoding performance is affected across the board, particularly at high resolutions + high fps). There is no workaround that I am aware of. Let me know if there's anything else you think I can do to get this in front of the right team / person. Thanks.