Bug 215908 - REGRESSION: Textures Fail to Render in WebGL from HLS Stream [iOS 14]
Summary: REGRESSION: Textures Fail to Render in WebGL from HLS Stream [iOS 14]
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebGL (show other bugs)
Version: Other
Hardware: iPhone / iPad Other
: P1 Critical
Assignee: Kimmo Kinnunen
URL:
Keywords: InRadar
: 216250 (view as bug list)
Depends on:
Blocks: 216915 218971 216573 216574 218637
  Show dependency treegraph
 
Reported: 2020-08-27 18:46 PDT by rigel
Modified: 2021-05-26 10:25 PDT (History)
25 users (show)

See Also:


Attachments
Image comparing iOS 14 Beta 6 to iOS 13.6.1 (400.85 KB, image/png)
2020-08-27 18:46 PDT, rigel
no flags Details
Patch (3.13 KB, patch)
2020-09-15 11:33 PDT, Kimmo Kinnunen
no flags Details | Formatted Diff | Diff
Patch (4.81 KB, patch)
2020-09-21 12:01 PDT, Kimmo Kinnunen
no flags Details | Formatted Diff | Diff
Patch (3.73 KB, patch)
2021-02-24 04:53 PST, Kimmo Kinnunen
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description rigel 2020-08-27 18:46:56 PDT
Created attachment 407441 [details]
Image comparing iOS 14 Beta 6 to iOS 13.6.1

Overview: As recently as iOS 14 Developer Beta 6, Safari no longer renders textures from HLS streams in WebGL. This is critical for streaming video within popular WebGL frameworks such as three.js, A-Frame, babylon.js etc. 

*Examples to Reproduce Issue*

three.js example: Visit https://8w.8thwall.app/hls-threejs on an iOS/iPadOS device running iOS 14 Developer Beta 6 or later. The rotating cube has a video texture applied from an HLS stream. The video in the top left corner is using the same HLS stream but rendering on top of the page outside WebGL.

babylon.js example: Visit https://www.babylonjs-playground.com/#9FSDC7#49 on iOS/iPadOS device running iOS 14 Developer Beta 6 or later. The plane has a video texture applied from an HLS stream.

Both examples work correctly on: 
iOS 13.6.1 Safari (iPhone 11 Pro Max)
iPadOS 13.6.1 Safari (iPad Pro 11", 2018)
Safari Technology Preview Release 112 [Safari 14.0, WebKit 15610.1.25.5.1] (MacBook Pro 15-inch, 2017; 10.15.6) 

However, both examples fail to produce a texture on iOS 14 Developer Beta 6 Safari (iPhone XR).

Here is an unlisted video comparing iOS 14 Beta 6 to iOS 13.6.1: https://youtu.be/jzjoHtADnmU
Comment 1 Taeheon Kim 2020-08-27 22:18:59 PDT
I can confirm that I also see this issue on iOS 14 Beta 6. Interestingly, this issue can also be seen on iOS 13.6 simulator but not on device.

Here's a fairly simple WebGL sample I've put together that has the issue: https://bit.ly/2EF2a6b
Comment 2 Radar WebKit Bug Importer 2020-08-29 11:32:31 PDT
<rdar://problem/68000962>
Comment 3 Alexey Proskuryakov 2020-09-01 08:54:31 PDT
Thank you for the report! Could you please clarify if this worked in previous betas?
Comment 4 rigel 2020-09-01 09:00:30 PDT
I believe this was working prior to iOS 14 Developer Beta 4.
Comment 5 Kimmo Kinnunen 2020-09-15 11:33:35 PDT
Created attachment 408839 [details]
Patch
Comment 6 EWS Watchlist 2020-09-15 11:34:23 PDT
Note that there are important steps to take when updating ANGLE. See https://trac.webkit.org/wiki/UpdatingANGLE
Comment 7 rpr_ag 2020-09-15 19:19:37 PDT
I also was able to reproduce this issue on iOS 14 Beta 6 with multiple WebGL applications that use HLS streaming

I tested the same projects on iOS 13.6 and they functioned properly.

We have many live projects which utilize HLS video playback within a WebGL context. These projects will break when iOS 14 goes fully live, so it would be very helpful if this issue were patched before that point.
Comment 8 Dean Jackson 2020-09-16 12:48:05 PDT
I'm just compiling this for iOS hardware to confirm, but I think this is good to go.
Comment 9 Dean Jackson 2020-09-16 15:37:23 PDT
Huh. Weirdly this works for me without the patch on ToT.
Comment 10 Dean Jackson 2020-09-16 15:45:20 PDT
It's a shame we don't have a test.
Comment 11 Dean Jackson 2020-09-16 15:48:10 PDT
But does not work with the patch applied! I might have screwed up but I'll need to investigate this.
Comment 12 Dean Jackson 2020-09-16 15:58:40 PDT
This was using https://8w.8thwall.app/hls-threejs.
Comment 13 Kimmo Kinnunen 2020-09-21 12:01:33 PDT
Created attachment 409291 [details]
Patch
Comment 14 Dean Jackson 2020-09-23 11:21:00 PDT
Comment on attachment 409291 [details]
Patch

We should be able to make a test case for this. There are some HLS tests in http/tests/media/hls.
Comment 15 EWS 2020-09-23 23:43:40 PDT
kkinnunen@apple.com does not have committer permissions according to https://svn.webkit.org/repository/webkit/trunk/Tools/Scripts/webkitpy/common/config/contributors.json.

Rejecting attachment 409291 [details] from commit queue.
Comment 16 Kimmo Kinnunen 2020-09-23 23:57:12 PDT
> We should be able to make a test case for this. There are some HLS tests in http/tests/media/hls.

We don't know if we need add a test that would trigger any of the bugs in here.

Primarily the problem isn't whether there is a test case.
* _All_ gpu video image conversions failed with simulator, yet no test failed. I did file a bug about this.
* SW video conversion codepath failed on simulator and on device. I did file a bug about that. When this gets worked on, we need to come back here and understand why the sw codepath failed and use the content here to add a test case if existing video/hls tests wouldn't trigger the problem.

Further more, even if simulator tests would've worked, all video tests would have failed on device on gpu codepath. I think we have a bug about improving this situation too.
Comment 17 EWS 2020-09-24 00:01:41 PDT
Committed r267520: <https://trac.webkit.org/changeset/267520>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 409291 [details].
Comment 18 Kenneth Russell 2020-09-25 16:32:50 PDT
Great diagnosis and fix Kimmo. Filed https://bugs.chromium.org/p/angleproject/issues/detail?id=5104 to track upstreaming of the ANGLE changes.
Comment 19 Kimmo Kinnunen 2020-09-29 23:32:32 PDT
*** Bug 216250 has been marked as a duplicate of this bug. ***
Comment 20 Dustin Kerstein 2020-09-30 05:15:48 PDT
Good to hear this fix also addresses bug 216250. Do you happen to know if it's possible this fix will make it into iOS 14.2? Thanks.
Comment 21 Chris Dumez 2020-09-30 07:09:08 PDT
(In reply to Dustin Kerstein from comment #20)
> Good to hear this fix also addresses bug 216250. Do you happen to know if
> it's possible this fix will make it into iOS 14.2? Thanks.

Apple does not comment on when a particular fix will be released to the public.
Comment 22 Dustin Kerstein 2020-09-30 07:17:41 PDT
(In reply to Chris Dumez from comment #21)
> Apple does not comment on when a particular fix will be released to the
> public.

Ok, thanks for the info. Do you think it'd be possible to increase the importance of this bug to P1? Based on the rubric described on https://webkit.org/bug-prioritization it seems the behavior described in 215908 and 216250 would qualify as P1.
Comment 23 Andrii Barvynko 2020-10-02 06:07:31 PDT
Hello. Faced with same issue on pixi.js. Could somebody clarify please, fix is still not available in the latest iOS 14.0.1, right? From what I can see examples from babylon and tree.js are not working as well, so, I'm wondering why is it marked as resolved and fixed?
Comment 24 Dustin Kerstein 2020-10-19 12:17:13 PDT
FYI, I believe this fix has been pulled into iOS 14.2 beta 3 (18B5072f). Just tested in a 1st generation iPad Pro and the https://8w.8thwall.app/hls-threejs test works correctly, and the performance noted in bug 216250 has significantly improved.
Comment 25 Daniel Rossi 2021-02-23 05:58:29 PST
I put in some fixes to capture resize HLS events and update renderer sizes. However I have a client who observed this is not working in Iphone IOS 14.4 but ok in Ipad IOS 14.4. 

Anyone tested with Iphone 14.4 ?
Comment 26 Simon Taylor 2021-02-23 21:05:49 PST
(In reply to Daniel Rossi from comment #25)
> I put in some fixes to capture resize HLS events and update renderer sizes.
> However I have a client who observed this is not working in Iphone IOS 14.4
> but ok in Ipad IOS 14.4. 
> 
> Anyone tested with Iphone 14.4 ?

My guess is your client has an iPhone 12, in which case this is tracked in bug 218637 - there are still known issues with HLS videos and the accelerated video texture pipeline on the iPhone 12 series with iOS 14.4 (and the 14.5 beta 2).

That bug is apparently fixed on the webkit side, so I've got my fingers crossed the fix might make it into the final 14.5 release.
Comment 27 Daniel Rossi 2021-02-23 23:04:04 PST
Yes an Iphone 12 IOS 14.4. Would the fix be in a Chrome based Webkit ? So testing on an Ipad has never showed the problem apart from a work around to resize things on a HLS resize event to start rendering. So they have to wait for an IOS 14.5 release with the possible webkit fix ?
Comment 28 Kimmo Kinnunen 2021-02-24 04:52:58 PST
Reopening to attach new patch.
Comment 29 Kimmo Kinnunen 2021-02-24 04:53:03 PST
Created attachment 421396 [details]
Patch
Comment 30 Daniel Rossi 2021-02-24 05:39:44 PST
Just an observation. As I was the first to report multiple HLS Webgl texture regressions with various work arounds I eventually found a few years ago or more. This was during the transition from IOS 10-11.1. But problems keep appearing since then. 

The Webkit webGL tests do not even have HLS textures configured. Originally I had to  modify some of the tests with HLS when reporting the black rendering issue. 

For these regressions to stop, convert one of the video textures to HLS and include that ! 

Hopefully that might stop these. They are extremely disruptive to deal with after and hard to debug and time consuming to make tests for bug reporting. 

And obviously take a long time to officially make their way into IOS. Sometimes until a major revision is released !
Comment 31 Daniel Rossi 2021-02-24 05:42:42 PST
I believe Ipad was an issue the whole time since IOS 14, I found it would start rendering black. But a resize event work around fixed it, I would resize with the HLS resize events to capture the updated video width and height as on metadata the video dimensions is zero. 

I could not test Iphone so perhaps it was still a problem on Iphone the whole time for some reason.
Comment 32 Mateusz Śpiewak 2021-03-02 13:56:40 PST
I'm unable to confirm this issue is resolved. I can reproduce the issue on iPhone 12 iOS 14.5 Beta.

@Kimmo Kinnunen I'm not really sure why this issue has status Resolved Fixed. It is not resolved.
Comment 33 Daniel Rossi 2021-05-20 14:48:16 PDT
Are you saying it's still broken in 14.5 beta ? Has 14.5 been released yet to check ? Why is it only broken in Iphone ?
Comment 34 abot 2021-05-20 20:47:48 PDT
I also have the same problem. After testing, Iphone7 ios version 14 and above can render to webgl display normally using hls stream, but iphone12 ios version 14 and above can not render to webgl normally, how can I deal with this problem? thanks.
Comment 35 Simon Taylor 2021-05-20 21:27:34 PDT
The fix described in this bug was part of iOS 14.2 and fixed the issue on all iPhones apart from iPhone 12 series.

iPhone 12 series still can't play HLS video in WebGL as of iOS 14.5 - there's a separate bug about that here: https://bugs.webkit.org/show_bug.cgi?id=218637

That bug is now apparently fixed too but hasn't yet made it into an iOS release. Someone on the comments on that bug reports that it works in 14.6 beta 2, so hopefully iOS 14.6 will finally see this fixed for good on all devices.
Comment 36 Daniel Rossi 2021-05-26 09:30:01 PDT
Will this actually be fixed in 14.6 ? As it was supposed to be returned back to normal in 14.5. How long to wait for 14.6 ? This has been opened since August 2020. It was working in previous versions on Iphone pretty sure of it.
Comment 37 Alexey Proskuryakov 2021-05-26 10:25:36 PDT
iOS 14.6 shipped this week.

It looks like there are multiple connected issues here. Some were fixed in 14.5, and some in 14.6.

If there is still something broken with WebGL and HLS as of iOS 14.6, I suggest filing new bugs to avoid any confusion, and to make sure that someone takes another look.