Bug 215908 – REGRESSION: Textures Fail to Render in WebGL from HLS Stream [iOS 14]

• Status: RESOLVED
• Resolution: FIXED
• Component: WebGL

Created attachment 407441 [details]
Image comparing iOS 14 Beta 6 to iOS 13.6.1

Overview: As recently as iOS 14 Developer Beta 6, Safari no longer renders textures from HLS streams in WebGL. This is critical for streaming video within popular WebGL frameworks such as three.js, A-Frame, babylon.js etc. 

*Examples to Reproduce Issue*

three.js example: Visit https://8w.8thwall.app/hls-threejs on an iOS/iPadOS device running iOS 14 Developer Beta 6 or later. The rotating cube has a video texture applied from an HLS stream. The video in the top left corner is using the same HLS stream but rendering on top of the page outside WebGL.

babylon.js example: Visit https://www.babylonjs-playground.com/#9FSDC7#49 on iOS/iPadOS device running iOS 14 Developer Beta 6 or later. The plane has a video texture applied from an HLS stream.

Both examples work correctly on: 
iOS 13.6.1 Safari (iPhone 11 Pro Max)
iPadOS 13.6.1 Safari (iPad Pro 11", 2018)
Safari Technology Preview Release 112 [Safari 14.0, WebKit 15610.1.25.5.1] (MacBook Pro 15-inch, 2017; 10.15.6) 

However, both examples fail to produce a texture on iOS 14 Developer Beta 6 Safari (iPhone XR).

Here is an unlisted video comparing iOS 14 Beta 6 to iOS 13.6.1: https://youtu.be/jzjoHtADnmU

[Taeheon Kim]

I can confirm that I also see this issue on iOS 14 Beta 6. Interestingly, this issue can also be seen on iOS 13.6 simulator but not on device.

Here's a fairly simple WebGL sample I've put together that has the issue: https://bit.ly/2EF2a6b

[Radar WebKit Bug Importer]

<rdar://problem/68000962>

[Alexey Proskuryakov]

Thank you for the report! Could you please clarify if this worked in previous betas?

I believe this was working prior to iOS 14 Developer Beta 4.

[Kimmo Kinnunen]

Created attachment 408839 [details]
Patch

[EWS Watchlist]

Note that there are important steps to take when updating ANGLE. See https://trac.webkit.org/wiki/UpdatingANGLE

[rpr_ag]

I also was able to reproduce this issue on iOS 14 Beta 6 with multiple WebGL applications that use HLS streaming

I tested the same projects on iOS 13.6 and they functioned properly.

We have many live projects which utilize HLS video playback within a WebGL context. These projects will break when iOS 14 goes fully live, so it would be very helpful if this issue were patched before that point.

[Dean Jackson]

I'm just compiling this for iOS hardware to confirm, but I think this is good to go.

[Dean Jackson]

Huh. Weirdly this works for me without the patch on ToT.

[Dean Jackson]

It's a shame we don't have a test.

[Dean Jackson]

But does not work with the patch applied! I might have screwed up but I'll need to investigate this.

[Dean Jackson]

This was using https://8w.8thwall.app/hls-threejs.

[Kimmo Kinnunen]

Created attachment 409291 [details]
Patch

[Dean Jackson]

Comment on attachment 409291 [details]
Patch

We should be able to make a test case for this. There are some HLS tests in http/tests/media/hls.

[EWS]

[e-mail] does not have committer permissions according to https://svn.webkit.org/repository/webkit/trunk/Tools/Scripts/webkitpy/common/config/contributors.json.

Rejecting attachment 409291 [details] from commit queue.

[Kimmo Kinnunen]

> We should be able to make a test case for this. There are some HLS tests in http/tests/media/hls.

We don't know if we need add a test that would trigger any of the bugs in here.

Primarily the problem isn't whether there is a test case.
* _All_ gpu video image conversions failed with simulator, yet no test failed. I did file a bug about this.
* SW video conversion codepath failed on simulator and on device. I did file a bug about that. When this gets worked on, we need to come back here and understand why the sw codepath failed and use the content here to add a test case if existing video/hls tests wouldn't trigger the problem.

Further more, even if simulator tests would've worked, all video tests would have failed on device on gpu codepath. I think we have a bug about improving this situation too.

[EWS]

Committed r267520: <https://trac.webkit.org/changeset/267520>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 409291 [details].

[Kenneth Russell]

Great diagnosis and fix Kimmo. Filed https://bugs.chromium.org/p/angleproject/issues/detail?id=5104 to track upstreaming of the ANGLE changes.

[Kimmo Kinnunen]

*** Bug 216250 has been marked as a duplicate of this bug. ***

[Dustin Kerstein]

Good to hear this fix also addresses bug 216250. Do you happen to know if it's possible this fix will make it into iOS 14.2? Thanks.

[Chris Dumez]

(In reply to Dustin Kerstein from comment #20)
> Good to hear this fix also addresses bug 216250. Do you happen to know if
> it's possible this fix will make it into iOS 14.2? Thanks.

Apple does not comment on when a particular fix will be released to the public.

[Dustin Kerstein]

(In reply to Chris Dumez from comment #21)
> Apple does not comment on when a particular fix will be released to the
> public.

Ok, thanks for the info. Do you think it'd be possible to increase the importance of this bug to P1? Based on the rubric described on https://webkit.org/bug-prioritization it seems the behavior described in 215908 and 216250 would qualify as P1.

[Andrii Barvynko]

Hello. Faced with same issue on pixi.js. Could somebody clarify please, fix is still not available in the latest iOS 14.0.1, right? From what I can see examples from babylon and tree.js are not working as well, so, I'm wondering why is it marked as resolved and fixed?

[Dustin Kerstein]

FYI, I believe this fix has been pulled into iOS 14.2 beta 3 (18B5072f). Just tested in a 1st generation iPad Pro and the https://8w.8thwall.app/hls-threejs test works correctly, and the performance noted in bug 216250 has significantly improved.

[Daniel Rossi]

I put in some fixes to capture resize HLS events and update renderer sizes. However I have a client who observed this is not working in Iphone IOS 14.4 but ok in Ipad IOS 14.4. 

Anyone tested with Iphone 14.4 ?

[Simon Taylor]

(In reply to Daniel Rossi from comment #25)
> I put in some fixes to capture resize HLS events and update renderer sizes.
> However I have a client who observed this is not working in Iphone IOS 14.4
> but ok in Ipad IOS 14.4. 
> 
> Anyone tested with Iphone 14.4 ?

My guess is your client has an iPhone 12, in which case this is tracked in bug 218637 - there are still known issues with HLS videos and the accelerated video texture pipeline on the iPhone 12 series with iOS 14.4 (and the 14.5 beta 2).

That bug is apparently fixed on the webkit side, so I've got my fingers crossed the fix might make it into the final 14.5 release.

[Daniel Rossi]

Yes an Iphone 12 IOS 14.4. Would the fix be in a Chrome based Webkit ? So testing on an Ipad has never showed the problem apart from a work around to resize things on a HLS resize event to start rendering. So they have to wait for an IOS 14.5 release with the possible webkit fix ?

[Kimmo Kinnunen]

Reopening to attach new patch.

[Kimmo Kinnunen]

Created attachment 421396 [details]
Patch

[Daniel Rossi]

Just an observation. As I was the first to report multiple HLS Webgl texture regressions with various work arounds I eventually found a few years ago or more. This was during the transition from IOS 10-11.1. But problems keep appearing since then. 

The Webkit webGL tests do not even have HLS textures configured. Originally I had to  modify some of the tests with HLS when reporting the black rendering issue. 

For these regressions to stop, convert one of the video textures to HLS and include that ! 

Hopefully that might stop these. They are extremely disruptive to deal with after and hard to debug and time consuming to make tests for bug reporting. 

And obviously take a long time to officially make their way into IOS. Sometimes until a major revision is released !

[Daniel Rossi]

I believe Ipad was an issue the whole time since IOS 14, I found it would start rendering black. But a resize event work around fixed it, I would resize with the HLS resize events to capture the updated video width and height as on metadata the video dimensions is zero. 

I could not test Iphone so perhaps it was still a problem on Iphone the whole time for some reason.

[Mateusz Śpiewak]

I'm unable to confirm this issue is resolved. I can reproduce the issue on iPhone 12 iOS 14.5 Beta.

@Kimmo Kinnunen I'm not really sure why this issue has status Resolved Fixed. It is not resolved.

[Daniel Rossi]

Are you saying it's still broken in 14.5 beta ? Has 14.5 been released yet to check ? Why is it only broken in Iphone ?

[abot]

I also have the same problem. After testing, Iphone7 ios version 14 and above can render to webgl display normally using hls stream, but iphone12 ios version 14 and above can not render to webgl normally, how can I deal with this problem? thanks.

[Simon Taylor]

The fix described in this bug was part of iOS 14.2 and fixed the issue on all iPhones apart from iPhone 12 series.

iPhone 12 series still can't play HLS video in WebGL as of iOS 14.5 - there's a separate bug about that here: https://bugs.webkit.org/show_bug.cgi?id=218637

That bug is now apparently fixed too but hasn't yet made it into an iOS release. Someone on the comments on that bug reports that it works in 14.6 beta 2, so hopefully iOS 14.6 will finally see this fixed for good on all devices.

[Daniel Rossi]

Will this actually be fixed in 14.6 ? As it was supposed to be returned back to normal in 14.5. How long to wait for 14.6 ? This has been opened since August 2020. It was working in previous versions on Iphone pretty sure of it.

[Alexey Proskuryakov]

iOS 14.6 shipped this week.

It looks like there are multiple connected issues here. Some were fixed in 14.5, and some in 14.6.

If there is still something broken with WebGL and HLS as of iOS 14.6, I suggest filing new bugs to avoid any confusion, and to make sure that someone takes another look.