Bug 179417 - HLS WebGL video textures black rendering since IOS 11
Summary: HLS WebGL video textures black rendering since IOS 11
Status: RESOLVED WORKSFORME
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebGL (show other bugs)
Version: Safari 11
Hardware: iPhone / iPad iOS 11
: P2 Blocker
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2017-11-08 00:39 PST by Daniel Rossi
Modified: 2019-05-30 08:21 PDT (History)
10 users (show)

See Also:


Attachments
Remote Debug Log of IOS HLS texture test failures. (223.13 KB, image/png)
2017-11-08 00:39 PST, Daniel Rossi
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Rossi 2017-11-08 00:39:58 PST
Created attachment 326313 [details]
Remote Debug Log of IOS HLS texture test failures.

Hi there, referencing this earlier ticket #163866 where HLS video textures was failing in IOS10 until FlipY was disabled with work arounds. It has become now apparent HLS video textures is failing completely since IOS 11 even with the CORS fix in place. This issue is still unresolved. 

This same problem of black frame rendering was apparant in IOS9 and so only IOS10 was working with the work around. 

I have produced some various tests including three.js , raw webgl with both mp4 and HLS streams to display the problem. No webgl flag addition / removal seems to fix it. 

I have investigated the webkit conformance test video texture files. It seems only mp4 was ever tested against Safari and it seems the tests were never designed for IOS as there is no user interaction code to begin the video. Therefore the IOS tests out of the box don't even begin. 

I don't believe these tests were ever done for IOS and most definitely HLS was never tested with webgl on IOS. 

I have had to modify the test files to include a HLS version of the mp4 video test files. All webgl tests fail and it will show the tiny canvas as a black frame at the top. I have made a heading where the gl canvas appears as a black square "Gl Canvas".  It will not render the texture. 



 IOS Mp4 Texture Tests:

threejs -http://dev.electroteque.org/webgl/three-mp4.html
raw webgl - http://dev.electroteque.org/webgl/webgl-mp4.html

IOS HLS Texture Tests:

threejs - http://dev.electroteque.org/webgl/three-hls.html
raw webgl - http://dev.electroteque.org/webgl/webgl-hls.html

Webkit Mp4 Texture Tests (works in IOS11):

http://dev.electroteque.org/webgl/webkit/tex-image-and-sub-image-2d-with-video.html
http://dev.electroteque.org/webgl/webkit/tex-image-and-sub-image-2d-with-video-rgb565.html
http://dev.electroteque.org/webgl/webkit/tex-image-and-sub-image-2d-with-video-rgba4444.html
http://dev.electroteque.org/webgl/webkit/tex-image-and-sub-image-2d-with-video-rgba5551.html

Webkit HLS Text Tests (black frame in IOS11 all tests fail because there is no picture):

http://dev.electroteque.org/webgl/webkit/tex-image-and-sub-image-2d-with-hls-video.html
http://dev.electroteque.org/webgl/webkit/tex-image-and-sub-image-2d-with-hls-video-rgb565.html
http://dev.electroteque.org/webgl/webkit/tex-image-and-sub-image-2d-with-hls-video-rgba4444.html
http://dev.electroteque.org/webgl/webkit/tex-image-and-sub-image-2d-with-video-rgba5551.html
Comment 1 Daniel Rossi 2017-11-08 00:42:36 PST
It didn't hyperlink the ticket number. Here is the original bug ticket that was unresolved. 

IOS10 can be worked around with the FlipY work around code posted. 

https://bugs.webkit.org/show_bug.cgi?id=163866
Comment 2 Radar WebKit Bug Importer 2017-11-08 11:44:46 PST
<rdar://problem/35420452>
Comment 3 Daniel Rossi 2017-11-14 00:16:22 PST
Anything ? IOS 11 has broken HLS live 360 streaming for many people. But they use it for VOD also not mp4.
Comment 4 Stephan Gensch 2017-11-14 02:08:14 PST
Thanks Daniel for keeping this issue alive and being persistent about this.

We have been using HLS for our 360 video player and have run into the same dead end for at least a third of our users that before could enjoy their stories on their iPhones. It is extremely frustrating, since our users of course don't dive into the technicalities of cross-browser/ platform issues and just expect things to work and of course the blame is on our end with this.

If this is not looked in to, we'd have to do a considerable technology switch, or even worse develop on multiple paths even adding the app option.
Comment 5 Arnaud Leyder 2017-11-14 02:22:27 PST
We also have the same issue which prevents us from supporting 360° video on iOS 11+ with HLS. We could work our way around in iOS 10 but that is not possible with iOS 11+. 

Given that HLS is the only mean to reach iOS Safari with ABR streaming this is a deal breaker for many trying to bring the latest features to iOS.
Comment 6 Daniel Rossi 2017-11-14 07:34:15 PST
My fix in the previous ticket works around HLS black rendering issues in IOS10. take a look. So it's good enough for IOS10.  

It has been posted on the three.js ticket also.
Comment 7 Jer Noble 2017-11-14 08:46:44 PST
These tests were originally designed to run within our testing harness, and not necessarily all browsers. So it would be helpful if they were modified to include <video playsinline muted> to each of the elements. Then, they'd be testable on all iOS devices, including iPhones.
Comment 8 Jer Noble 2017-11-14 08:59:07 PST
Dino implemented flipY in bug# 176491, but not in time for iOS 11.0 or macOS 10.13.0. If you have access to Apple's Developer program, you can test out the fix in-place in the macOS 10.13.2 beta 3 seed or the iOS 11.2 beta 3 seed: <https://developer.apple.com/download/>

Here on my 11.2 beta device, I see the HLS tests which Daniel posted are working correctly (though annoyingly trying to enter full screen mode due to a lack of the 'playsinline' attribute on the elements in the test).
Comment 9 Daniel Rossi 2017-11-14 09:31:54 PST
The HLS tests don't work with FlipY disabled. In IOS10 it does, and the shader fix will flip it around.  

Is that what you mean. It now requires it enabled but is pushed to 11.2 ?

I'm sorry but I had to buy a new device to test (don't specifically work with IOS or OSX) and all I could get was an Ipad no Iphone. Which inline isn't a problem with Ipad.  

I added the inline attributes to the original webkit test files scripts. 

video.setAttribute("playsinline", "");
video.setAttribute("webkit-playsinline", ""); 

Hopefully that helps. 

So you are saying the webkit test files with HLS bring up a frame in 11.2 or showing a black square ? 

I'm installing the beta now and let you know what I discover.
Comment 10 Daniel Rossi 2017-11-14 10:18:56 PST
The webkit tests I made now work in 11.2. So we got past that part. 

However my threejs and raw webgl tests are still not rendering correctly. I tried to replicate the webgl code directly. however if I use RGBA I get a transparent canvas. 

Any ideas then ? 

threejs - http://dev.electroteque.org/webgl/three-hls.html
raw webgl - http://dev.electroteque.org/webgl/webgl-hls.html
Comment 11 Daniel Rossi 2017-11-14 10:31:56 PST
I can't confirm because I can't figure out how to hack the test files from repeating. There is a black frame showing up with my demo HLS video.
Comment 12 Daniel Rossi 2017-11-14 10:37:42 PST
The test HLS file I made from the mp4 in the webkit tests doesn't display in the raw webgl example. It's a black frame or transparent if RBA is chosen. 

So it seems the code has to match and be specific to the test code. 

Any ideas how to replicate all that over I tried adding some.
Comment 13 Jer Noble 2017-11-14 11:01:47 PST
(In reply to Daniel Rossi from comment #10)
> The webkit tests I made now work in 11.2. So we got past that part. 
> 
> However my threejs and raw webgl tests are still not rendering correctly. I
> tried to replicate the webgl code directly. however if I use RGBA I get a
> transparent canvas. 
> 
> Any ideas then ? 
> 
> threejs - http://dev.electroteque.org/webgl/three-hls.html
> raw webgl - http://dev.electroteque.org/webgl/webgl-hls.html

Looks like the 'loadeddata' event isn't firing with HLS streams, so your init() function is never being called. I believe this is a known issue, but you should be able to work around it by using a different event to do your initialization.
Comment 14 Daniel Rossi 2017-11-14 19:21:29 PST
That is not correct. If that is the case it would have been broken in 7.2 from 7.1. The animation frame is starting, loadeddata event is working correctly. 

I have debugged that to confirm, although the log won't keep showing it. 

I've updated this , to show in the console the animationframe is starting. 

Black frame. I will have to spend time picking out the code from the test files to replicate it. 

http://dev.electroteque.org/webgl/webgl-hls.html
Comment 15 Daniel Rossi 2017-11-14 19:22:26 PST
I meant 11.2 from 11.1 can't edit the comment.
Comment 16 Daniel Rossi 2017-11-14 20:14:17 PST
I've produced another test with my external video. the color needs to be set to RGB or else it's a transparent frame. I had to comment out some code regarding to color mask also.  

I have heavily modified this to run an animation frame and remove the color tests.

There is a log to show the animation is running.  

It's just a black frame here. I suspect it might be CORS related which I will test next. 

Only the local color test HLS video is working so far. 

http://dev.electroteque.org/webgl/webkit/hlstest.html
Comment 17 Daniel Rossi 2017-11-14 20:24:00 PST
Thankfully it is not CORS related. Same problem locally. Don't know what else to report there yet. 

Is there a special webgl flag required for this particular video ? I've encoded and packaged them via Elastic Transcoder. 

It's fine on IOS 10 with the flipy work arounds is all.
Comment 18 Daniel Rossi 2017-11-14 20:27:24 PST
I'm so sorry for the noise. 

The original HLS webkit tests are still not working and black for me on IOS 11.2 beta. 

I was clicking the mp4 examples farout. I just wasted some time to AB my external source.
Comment 19 Daniel Rossi 2017-11-15 23:25:25 PST
Do you mind uploading your own test for me to try on IOS 11.2 beta ? Is there a different IOS version to install now ? 

I am unable to replicate the fix for HLS. I was confusing myself clicking the wrong mp4 tests. Hence why my own external tests are still a black frame too.
Comment 20 Jer Noble 2017-11-16 13:46:32 PST
Sorry, I haven't had time to look at this today. I'll keep you updated when I get a chance to dig into this again.
Comment 21 Daniel Rossi 2017-11-16 19:59:04 PST
I don't have an environment to run builds myself and get something working I'm sorry I am no help there.  I'm on Windows. 

I was able to play around with webgl flags before and get it working so I'm wondering if it's another one of those issues.
Comment 22 Jer Noble 2017-11-16 23:55:01 PST
I think I have a handle on what might be causing this. The OpenGL code which copies textures in from the video element may be stomping on some of the WebGL state established by the page. Moving the VideoTextureCopierCV to use a shared context (with shared textures but separate state) may fix this. Stay tuned.
Comment 23 Daniel Rossi 2017-11-17 20:16:47 PST
Oh righ. yes according to the gl upload code, I see what you mean about the texture context. Could the viewport size have anything to do with it also ? 

https://github.com/graouts/webkit/blob/0b791d19078bac6b864285cee874218f1374a7ad/Source/WebCore/platform/graphics/mac/WebGLLayer.mm#L94
Comment 24 Daniel Rossi 2017-11-17 20:19:22 PST
What about this also ? 

The previous IOS10 fix required switching the colourspace on the shader to bgra for IOS and HLS. 

After disabling FlipY , the colours were inverted. So that was the full partial work around for IOS 10 and HLS. 

According to this it's not using colourspace at all for IOS, as far as I see. 

https://github.com/graouts/webkit/blob/0b791d19078bac6b864285cee874218f1374a7ad/Source/WebCore/platform/graphics/mac/WebGLLayer.mm#L142
Comment 25 Daniel Rossi 2017-11-23 03:29:24 PST
How is it going ? Any luck ?
Comment 26 siddhant loya 2017-12-12 04:25:26 PST
Any update ?
Comment 27 Daniel Rossi 2017-12-12 04:38:45 PST
Hi it looks like its going to be a very long wait. 

I waited a little before deciding to have to do a software rendering workaround which will chew battery power and cause performance and frame drop problems. 

you have to draw to a temp canvas and use the canvas as a webgl texture. Im not at my IDevice yet to properly test if this works see

https://github.com/mrdoob/three.js/issues/9754#issuecomment-350685693

I have this same method now working again for IE 11. 

So if using three.js you set the canvas as the texture. 

 new THREE.Texture(canvas);

Then draw the video to the context

context.drawImage( video, 0, 0, canvas.width, canvas.height );

I'm working on a demo to update that other ticket once I can test on IDevice. 

Until there is more users migrating to IOS 11 from 10. 

You will have to keep the original HLS flipy workarounds documented in the previous ticket especially the colour channel inversion on the fragment shader. 

You will have to do IOS 11 version detection to treat everything different so it doesn't use the Flipy and colour inversion work arounds also as its also needed to check for CORS support.
Comment 28 siddhant loya 2017-12-12 05:07:41 PST
Hey, Thanks for suggesting the workaround. 

Can you share some stats regarding performance and frame drop ?
Comment 29 Daniel Rossi 2017-12-12 06:33:15 PST
I won't be back to an IDevice in a few days. But will do a test with a fps stats. I dont even know yet if it will even render. 

Stats or not stats I have people waiting for a fix on my end for Live HLS so has to be for now.
Comment 30 Dean Jackson 2017-12-12 11:43:23 PST
We've fixed some bugs in iOS video -> GL textures recently. I can't confirm that this particular issue is resolved yet though.
Comment 31 Daniel Rossi 2017-12-12 19:59:37 PST
I am still running Beta IOS 11 to keep on the same page. And have some updates to do and will confirm in a few days. 

Thanks for the update.
Comment 32 Daniel Rossi 2017-12-14 08:14:11 PST
Hi I am on 11.2.5 beta. There is still no change with HLS. It is a black frame even with FlipY disabled. 

I have just discovered with Safari 11 on macOS. The FlipY work around is still required for HLS. Or else it wont render. I may have to make a separate report for that ?
Comment 33 Daniel Rossi 2017-12-14 08:14:56 PST
I have made a canvas texture test. It is working but a larger resolution video might make it fall over. 

This HLS is low quality so is keeping at 40 fps. A larger resolution mp4 but also low quality stayed at 30fps. 

A production resolution file might get 10fps out of it. 

http://dev.electroteque.org/webgl/three-canvas.html
Comment 34 siddhant loya 2017-12-14 08:30:43 PST
Hey daniel, great work.

It is running at 60 fps for my device (mac book pro - 2015).
Comment 35 Daniel Rossi 2017-12-14 08:41:13 PST
This is an issue for IOS. I am testing with a current basic Ipad. 


Although Safari 11 suffers a FlipY issue on macOS still when using HLS. I just tested it. 

I have documented a fix for that in a previous ticket. I thought it was fixed up but isn't ? 

You need to disable FlipY and use the shader to reverse it's Y position. 

You don't need canvas texture rendering for macOS ! 

You need to do crazy version checks for macOS to first detect it can support CORS. Then you need to do a further check for IOS 11 to support CORS. Or else have to provide a CORS proxy source for previous Safari.  

Then you need to do the FlipY shader fix for Safari macOS and IOS10 . A colour inversion fix for IOS 10 on the fragment shader. 

And now do canvas rendering for IOS 11 HLS only. 

Pretty complicated I know. That whole bit above is still causing me problems getting it all to work because of bugs with the version detection requirements.
Comment 36 Daniel Rossi 2017-12-14 08:45:11 PST
Here is the fix for what you might need for macOS here. 

https://bugs.webkit.org/show_bug.cgi?id=163866#c16
Comment 37 Daniel Rossi 2017-12-15 04:30:54 PST
I have created a seperate ticket for HLS textures on Safari 11 macOS . The problem is still there with FlipY. 

This current ticket is related to IOS 11. 

https://bugs.webkit.org/show_bug.cgi?id=180863
Comment 38 Daniel Rossi 2017-12-24 17:58:00 PST
So in regards to this comment

https://bugs.webkit.org/show_bug.cgi?id=179417#c8

11.2 beta of IOS was still a problem. But I missed the bit about macOS which I created a seperate ticket for. 

I don't believe this issue is related to FlipY as disabling it does nothing. But FlipY does affect macOS with HLS. 

If you think 10.13 is fixed with FlipY for HLS textures I might not be able to confirm for a while as I would need to get access to that. 

It's another configuration I cant get access to as I have no hardware anymore, and in the past doing major OS updates affected and broke all my other audio software I was running. I don't have a dedicated test machine just for these problems.   

Mp4 textures was never a problem with FlipY enabled on macOS or IOS only HLS. 

see

https://bugs.webkit.org/show_bug.cgi?id=180863#c3
Comment 39 Daniel Rossi 2017-12-26 04:01:49 PST
More information has come to light and I have updated the title. 

Both IOS 11.2 beta and macOS 10.13 suffer the same problem now. Both are affected and the documented flipy fix does nothing for HLS video textures. Both require temporary canvas texture rendering work arounds which gets about 30fps on the Ipad. 

It seems possible this said FlipY ticket created a regression of soughts ? 

https://bugs.webkit.org/show_bug.cgi?id=176491

https://bugs.webkit.org/show_bug.cgi?id=180863#c4
Comment 40 Daniel Rossi 2017-12-26 08:34:54 PST
I cant delete my comment. disabling FlipY for 10.13 is ok now I believe. This problem is only relevant to IOS 11.
Comment 41 Daniel Rossi 2018-01-10 03:38:13 PST
Any update here ? I have been keeping the IOS beta up to date and no change yet.
Comment 42 Daniel Rossi 2018-02-14 02:11:42 PST
Any update please ? I reported this in November now.
Comment 43 Daniel Rossi 2018-02-14 19:02:26 PST
The test is now working in 11.3 beta. 

No idea what is going on but it seems to be working. Without any response or update !
Comment 44 Daniel Rossi 2018-02-14 19:16:00 PST
It's also rendering without the flipY workaround but I think I mentioned this before. FlipY is still a problem for macOS Safari.
Comment 45 Stephan Gensch 2018-04-07 01:35:03 PDT
Thanks Daniel,

I tried this now with iOS 11.3 rolled out for our A-frame based application and can confirm videos are working again. Had to remove the color channel switch that was necessary pre iOS 10.3.

Thanks again for keeping this bug report alive over time.
Comment 46 Daniel Rossi 2018-04-07 05:37:34 PDT
That's ok. I'm marking as resolved.
Comment 47 Gideon Keyson 2019-05-29 12:57:38 PDT
Hi Daniel, thanks for the detailed bug report. I am still getting the same bug on iOS and safari 12. Could I contact you privately in order to ask some questions? Thank!
Comment 48 Daniel Rossi 2019-05-30 08:21:03 PDT
I have no macOS hardware atm to test it. I can get access to an Ipad to see what is going on. 

It may require the flipY work around. The colourspace workaround, or using software render for that version like with 11.0. 

I have 3 sets of versions checks in place to provide all those workarounds. I haven't checked back in on my VR feature for a while and haven't had much interest apart from one recently.