Bug 179417 – HLS WebGL video textures black rendering since IOS 11

• Status: RESOLVED
• Resolution: WORKSFORME
• Component: WebGL

[Daniel Rossi]

Created attachment 326313 [details]
Remote Debug Log of IOS HLS texture test failures.

Hi there, referencing this earlier ticket #163866 where HLS video textures was failing in IOS10 until FlipY was disabled with work arounds. It has become now apparent HLS video textures is failing completely since IOS 11 even with the CORS fix in place. This issue is still unresolved. 

This same problem of black frame rendering was apparant in IOS9 and so only IOS10 was working with the work around. 

I have produced some various tests including three.js , raw webgl with both mp4 and HLS streams to display the problem. No webgl flag addition / removal seems to fix it. 

I have investigated the webkit conformance test video texture files. It seems only mp4 was ever tested against Safari and it seems the tests were never designed for IOS as there is no user interaction code to begin the video. Therefore the IOS tests out of the box don't even begin. 

I don't believe these tests were ever done for IOS and most definitely HLS was never tested with webgl on IOS. 

I have had to modify the test files to include a HLS version of the mp4 video test files. All webgl tests fail and it will show the tiny canvas as a black frame at the top. I have made a heading where the gl canvas appears as a black square "Gl Canvas".  It will not render the texture. 



 IOS Mp4 Texture Tests:

threejs -http://dev.electroteque.org/webgl/three-mp4.html
raw webgl - http://dev.electroteque.org/webgl/webgl-mp4.html

IOS HLS Texture Tests:

threejs - http://dev.electroteque.org/webgl/three-hls.html
raw webgl - http://dev.electroteque.org/webgl/webgl-hls.html

Webkit Mp4 Texture Tests (works in IOS11):

http://dev.electroteque.org/webgl/webkit/tex-image-and-sub-image-2d-with-video.html
http://dev.electroteque.org/webgl/webkit/tex-image-and-sub-image-2d-with-video-rgb565.html
http://dev.electroteque.org/webgl/webkit/tex-image-and-sub-image-2d-with-video-rgba4444.html
http://dev.electroteque.org/webgl/webkit/tex-image-and-sub-image-2d-with-video-rgba5551.html

Webkit HLS Text Tests (black frame in IOS11 all tests fail because there is no picture):

http://dev.electroteque.org/webgl/webkit/tex-image-and-sub-image-2d-with-hls-video.html
http://dev.electroteque.org/webgl/webkit/tex-image-and-sub-image-2d-with-hls-video-rgb565.html
http://dev.electroteque.org/webgl/webkit/tex-image-and-sub-image-2d-with-hls-video-rgba4444.html
http://dev.electroteque.org/webgl/webkit/tex-image-and-sub-image-2d-with-video-rgba5551.html

[Daniel Rossi]

It didn't hyperlink the ticket number. Here is the original bug ticket that was unresolved. 

IOS10 can be worked around with the FlipY work around code posted. 

https://bugs.webkit.org/show_bug.cgi?id=163866

[Radar WebKit Bug Importer]

<rdar://problem/35420452>

[Daniel Rossi]

Anything ? IOS 11 has broken HLS live 360 streaming for many people. But they use it for VOD also not mp4.

[Stephan Gensch]

Thanks Daniel for keeping this issue alive and being persistent about this.

We have been using HLS for our 360 video player and have run into the same dead end for at least a third of our users that before could enjoy their stories on their iPhones. It is extremely frustrating, since our users of course don't dive into the technicalities of cross-browser/ platform issues and just expect things to work and of course the blame is on our end with this.

If this is not looked in to, we'd have to do a considerable technology switch, or even worse develop on multiple paths even adding the app option.

[Arnaud Leyder]

We also have the same issue which prevents us from supporting 360° video on iOS 11+ with HLS. We could work our way around in iOS 10 but that is not possible with iOS 11+. 

Given that HLS is the only mean to reach iOS Safari with ABR streaming this is a deal breaker for many trying to bring the latest features to iOS.

[Daniel Rossi]

My fix in the previous ticket works around HLS black rendering issues in IOS10. take a look. So it's good enough for IOS10.  

It has been posted on the three.js ticket also.

[Jer Noble]

These tests were originally designed to run within our testing harness, and not necessarily all browsers. So it would be helpful if they were modified to include <video playsinline muted> to each of the elements. Then, they'd be testable on all iOS devices, including iPhones.

[Jer Noble]

Dino implemented flipY in bug# 176491, but not in time for iOS 11.0 or macOS 10.13.0. If you have access to Apple's Developer program, you can test out the fix in-place in the macOS 10.13.2 beta 3 seed or the iOS 11.2 beta 3 seed: <https://developer.apple.com/download/>

Here on my 11.2 beta device, I see the HLS tests which Daniel posted are working correctly (though annoyingly trying to enter full screen mode due to a lack of the 'playsinline' attribute on the elements in the test).

[Daniel Rossi]

The HLS tests don't work with FlipY disabled. In IOS10 it does, and the shader fix will flip it around.  

Is that what you mean. It now requires it enabled but is pushed to 11.2 ?

I'm sorry but I had to buy a new device to test (don't specifically work with IOS or OSX) and all I could get was an Ipad no Iphone. Which inline isn't a problem with Ipad.  

I added the inline attributes to the original webkit test files scripts. 

video.setAttribute("playsinline", "");
video.setAttribute("webkit-playsinline", ""); 

Hopefully that helps. 

So you are saying the webkit test files with HLS bring up a frame in 11.2 or showing a black square ? 

I'm installing the beta now and let you know what I discover.

[Daniel Rossi]

The webkit tests I made now work in 11.2. So we got past that part. 

However my threejs and raw webgl tests are still not rendering correctly. I tried to replicate the webgl code directly. however if I use RGBA I get a transparent canvas. 

Any ideas then ? 

threejs - http://dev.electroteque.org/webgl/three-hls.html
raw webgl - http://dev.electroteque.org/webgl/webgl-hls.html

[Daniel Rossi]

I can't confirm because I can't figure out how to hack the test files from repeating. There is a black frame showing up with my demo HLS video.

[Daniel Rossi]

The test HLS file I made from the mp4 in the webkit tests doesn't display in the raw webgl example. It's a black frame or transparent if RBA is chosen. 

So it seems the code has to match and be specific to the test code. 

Any ideas how to replicate all that over I tried adding some.

[Jer Noble]

(In reply to Daniel Rossi from comment #10)
> The webkit tests I made now work in 11.2. So we got past that part. 
> 
> However my threejs and raw webgl tests are still not rendering correctly. I
> tried to replicate the webgl code directly. however if I use RGBA I get a
> transparent canvas. 
> 
> Any ideas then ? 
> 
> threejs - http://dev.electroteque.org/webgl/three-hls.html
> raw webgl - http://dev.electroteque.org/webgl/webgl-hls.html

Looks like the 'loadeddata' event isn't firing with HLS streams, so your init() function is never being called. I believe this is a known issue, but you should be able to work around it by using a different event to do your initialization.

[Daniel Rossi]

That is not correct. If that is the case it would have been broken in 7.2 from 7.1. The animation frame is starting, loadeddata event is working correctly. 

I have debugged that to confirm, although the log won't keep showing it. 

I've updated this , to show in the console the animationframe is starting. 

Black frame. I will have to spend time picking out the code from the test files to replicate it. 

http://dev.electroteque.org/webgl/webgl-hls.html

[Daniel Rossi]

I meant 11.2 from 11.1 can't edit the comment.

[Daniel Rossi]

I've produced another test with my external video. the color needs to be set to RGB or else it's a transparent frame. I had to comment out some code regarding to color mask also.  

I have heavily modified this to run an animation frame and remove the color tests.

There is a log to show the animation is running.  

It's just a black frame here. I suspect it might be CORS related which I will test next. 

Only the local color test HLS video is working so far. 

http://dev.electroteque.org/webgl/webkit/hlstest.html

[Daniel Rossi]

Thankfully it is not CORS related. Same problem locally. Don't know what else to report there yet. 

Is there a special webgl flag required for this particular video ? I've encoded and packaged them via Elastic Transcoder. 

It's fine on IOS 10 with the flipy work arounds is all.

[Daniel Rossi]

I'm so sorry for the noise. 

The original HLS webkit tests are still not working and black for me on IOS 11.2 beta. 

I was clicking the mp4 examples farout. I just wasted some time to AB my external source.

[Daniel Rossi]

Do you mind uploading your own test for me to try on IOS 11.2 beta ? Is there a different IOS version to install now ? 

I am unable to replicate the fix for HLS. I was confusing myself clicking the wrong mp4 tests. Hence why my own external tests are still a black frame too.

[Jer Noble]

Sorry, I haven't had time to look at this today. I'll keep you updated when I get a chance to dig into this again.

[Daniel Rossi]

I don't have an environment to run builds myself and get something working I'm sorry I am no help there.  I'm on Windows. 

I was able to play around with webgl flags before and get it working so I'm wondering if it's another one of those issues.

[Jer Noble]

I think I have a handle on what might be causing this. The OpenGL code which copies textures in from the video element may be stomping on some of the WebGL state established by the page. Moving the VideoTextureCopierCV to use a shared context (with shared textures but separate state) may fix this. Stay tuned.

[Daniel Rossi]

Oh righ. yes according to the gl upload code, I see what you mean about the texture context. Could the viewport size have anything to do with it also ? 

https://github.com/graouts/webkit/blob/0b791d19078bac6b864285cee874218f1374a7ad/Source/WebCore/platform/graphics/mac/WebGLLayer.mm#L94

[Daniel Rossi]

What about this also ? 

The previous IOS10 fix required switching the colourspace on the shader to bgra for IOS and HLS. 

After disabling FlipY , the colours were inverted. So that was the full partial work around for IOS 10 and HLS. 

According to this it's not using colourspace at all for IOS, as far as I see. 

https://github.com/graouts/webkit/blob/0b791d19078bac6b864285cee874218f1374a7ad/Source/WebCore/platform/graphics/mac/WebGLLayer.mm#L142

[Daniel Rossi]

How is it going ? Any luck ?

[siddhant loya]

Any update ?

[Daniel Rossi]

Hi it looks like its going to be a very long wait. 

I waited a little before deciding to have to do a software rendering workaround which will chew battery power and cause performance and frame drop problems. 

you have to draw to a temp canvas and use the canvas as a webgl texture. Im not at my IDevice yet to properly test if this works see

https://github.com/mrdoob/three.js/issues/9754#issuecomment-350685693

I have this same method now working again for IE 11. 

So if using three.js you set the canvas as the texture. 

 new THREE.Texture(canvas);

Then draw the video to the context

context.drawImage( video, 0, 0, canvas.width, canvas.height );

I'm working on a demo to update that other ticket once I can test on IDevice. 

Until there is more users migrating to IOS 11 from 10. 

You will have to keep the original HLS flipy workarounds documented in the previous ticket especially the colour channel inversion on the fragment shader. 

You will have to do IOS 11 version detection to treat everything different so it doesn't use the Flipy and colour inversion work arounds also as its also needed to check for CORS support.

[siddhant loya]

Hey, Thanks for suggesting the workaround. 

Can you share some stats regarding performance and frame drop ?

[Daniel Rossi]

I won't be back to an IDevice in a few days. But will do a test with a fps stats. I dont even know yet if it will even render. 

Stats or not stats I have people waiting for a fix on my end for Live HLS so has to be for now.

[Dean Jackson]

We've fixed some bugs in iOS video -> GL textures recently. I can't confirm that this particular issue is resolved yet though.

[Daniel Rossi]

I am still running Beta IOS 11 to keep on the same page. And have some updates to do and will confirm in a few days. 

Thanks for the update.

[Daniel Rossi]

Hi I am on 11.2.5 beta. There is still no change with HLS. It is a black frame even with FlipY disabled. 

I have just discovered with Safari 11 on macOS. The FlipY work around is still required for HLS. Or else it wont render. I may have to make a separate report for that ?

[Daniel Rossi]

I have made a canvas texture test. It is working but a larger resolution video might make it fall over. 

This HLS is low quality so is keeping at 40 fps. A larger resolution mp4 but also low quality stayed at 30fps. 

A production resolution file might get 10fps out of it. 

http://dev.electroteque.org/webgl/three-canvas.html

[siddhant loya]

Hey daniel, great work.

It is running at 60 fps for my device (mac book pro - 2015).

[Daniel Rossi]

This is an issue for IOS. I am testing with a current basic Ipad. 


Although Safari 11 suffers a FlipY issue on macOS still when using HLS. I just tested it. 

I have documented a fix for that in a previous ticket. I thought it was fixed up but isn't ? 

You need to disable FlipY and use the shader to reverse it's Y position. 

You don't need canvas texture rendering for macOS ! 

You need to do crazy version checks for macOS to first detect it can support CORS. Then you need to do a further check for IOS 11 to support CORS. Or else have to provide a CORS proxy source for previous Safari.  

Then you need to do the FlipY shader fix for Safari macOS and IOS10 . A colour inversion fix for IOS 10 on the fragment shader. 

And now do canvas rendering for IOS 11 HLS only. 

Pretty complicated I know. That whole bit above is still causing me problems getting it all to work because of bugs with the version detection requirements.

[Daniel Rossi]

Here is the fix for what you might need for macOS here. 

https://bugs.webkit.org/show_bug.cgi?id=163866#c16

[Daniel Rossi]

I have created a seperate ticket for HLS textures on Safari 11 macOS . The problem is still there with FlipY. 

This current ticket is related to IOS 11. 

https://bugs.webkit.org/show_bug.cgi?id=180863

[Daniel Rossi]

So in regards to this comment

https://bugs.webkit.org/show_bug.cgi?id=179417#c8

11.2 beta of IOS was still a problem. But I missed the bit about macOS which I created a seperate ticket for. 

I don't believe this issue is related to FlipY as disabling it does nothing. But FlipY does affect macOS with HLS. 

If you think 10.13 is fixed with FlipY for HLS textures I might not be able to confirm for a while as I would need to get access to that. 

It's another configuration I cant get access to as I have no hardware anymore, and in the past doing major OS updates affected and broke all my other audio software I was running. I don't have a dedicated test machine just for these problems.   

Mp4 textures was never a problem with FlipY enabled on macOS or IOS only HLS. 

see

https://bugs.webkit.org/show_bug.cgi?id=180863#c3

[Daniel Rossi]

More information has come to light and I have updated the title. 

Both IOS 11.2 beta and macOS 10.13 suffer the same problem now. Both are affected and the documented flipy fix does nothing for HLS video textures. Both require temporary canvas texture rendering work arounds which gets about 30fps on the Ipad. 

It seems possible this said FlipY ticket created a regression of soughts ? 

https://bugs.webkit.org/show_bug.cgi?id=176491

https://bugs.webkit.org/show_bug.cgi?id=180863#c4

[Daniel Rossi]

I cant delete my comment. disabling FlipY for 10.13 is ok now I believe. This problem is only relevant to IOS 11.

[Daniel Rossi]

Any update here ? I have been keeping the IOS beta up to date and no change yet.

[Daniel Rossi]

Any update please ? I reported this in November now.

[Daniel Rossi]

The test is now working in 11.3 beta. 

No idea what is going on but it seems to be working. Without any response or update !

[Daniel Rossi]

It's also rendering without the flipY workaround but I think I mentioned this before. FlipY is still a problem for macOS Safari.

[Stephan Gensch]

Thanks Daniel,

I tried this now with iOS 11.3 rolled out for our A-frame based application and can confirm videos are working again. Had to remove the color channel switch that was necessary pre iOS 10.3.

Thanks again for keeping this bug report alive over time.

[Daniel Rossi]

That's ok. I'm marking as resolved.

[Gideon Keyson]

Hi Daniel, thanks for the detailed bug report. I am still getting the same bug on iOS and safari 12. Could I contact you privately in order to ask some questions? Thank!

[Daniel Rossi]

I have no macOS hardware atm to test it. I can get access to an Ipad to see what is going on. 

It may require the flipY work around. The colourspace workaround, or using software render for that version like with 11.0. 

I have 3 sets of versions checks in place to provide all those workarounds. I haven't checked back in on my VR feature for a while and haven't had much interest apart from one recently.

[Jeremy Gordon]

I am seeing what seems to be this bug (or related) again on iOS 14.3 on an iPhone 12 Pro. It does not happen on non iPhone 12 iOS devices. Also is an issue with iOS 14.2.1; I'm not sure at what iOS version the iPhone 12 started rendering HLS video all black when used as a text source in WebGL.

Our WebGL app uses .m3u8 HLS streams and renders them using a from scratch WebGL renderer with a <video> element as the texture source.

This has been working for the last year or more, and still works fine on iOS devices other than iPhone 12s.

If I change to "fast start" MP4 files, things work fine on all iOS devices for us.

We have hundreds of thousands of transcoded .m3u8 HLS streams so it's not a simple matter to switch to MP4 files.

[Daniel Rossi]

You need to update things on rendition resize events. here is mine. I've let others know also. It will stop the black rendering on startup for me. 

 video.addEventListener("resize", () => {
                vrVideo.update();
                vrVideo.fullScreenResize(player.isFullscreen);
});

within update I have

this.video.width = MathUtils.ceilPowerOfTwo(this.video.videoWidth);
        this.video.height = MathUtils.ceilPowerOfTwo(this.video.videoHeight);

[Jeremy Gordon]

Thank you for your help!

Unfortunately, it's not working and/or maybe I'm not following.

Here's my situation:

I have a custom WebGL based renderer. I create an offscreen <video> element pointing to an .m3u8 URL that has CORS setup appropriately. When the video has reached a readyState of "4" I create an RGB format WebGL texture of size videoWidth and videoHeight passing the HTMLVideoElement instance for the pixel data. The texture renders as all black pixels.

If I changed the <video> element to point to an .mp4 URL on the same server, wait for the offscreen <video> element to reach a readyState of "4" the texture renders with the video pixels.

Taking your advice, I added an event handler for "resize" to my offscreen <video> HTMLVideoElement element. That event handler does get called back, and my plan was to recreate the WebGL texture and make sure I'm using updated dimensions for videoWidth and videoHeight.

Unfortunately, the "resize" callbacks happen well before the video reaches a readyState of "4" and the videoWidth and videoHeight dimensions are unchanged (and correct).

In summary, the WebGL texture is only created after the readyState is "4" and all the "resize" events have been fired on HTMLVideoElement and I still see all black texels in the video. This is only the case for HLS/.m3u8 videos. Everything works fine in the case of .mp4 videos. The .m3u8 videos that do not work are encoded with identical resolution, bitrates, profile settings, etc as the .mp4 videos that do work.

Additionally, everything works fine on all other iOS devices with either HLS/.m3u8 or .mp4 except on iPhone 12 devices, where only .mp4 videos are working as a TextureSource for WebGL textures.

I have confirmed the problem behavior on multiple iPhone 12 devices with iOS 14.2.1+ and confirmed the successful behavior on multiple other iOS devices (i.e. iPad, iPhone 11, etc).

Am I misunderstanding your suggestion?

Thank you so much for taking the time to reply!

[Daniel Rossi]

I just set it up on metadata. I've moved to an AMD system so macOS vmware for IOS simulator is tricky right now. But I double checked people's complaints on IOS 14 Ipad. It was not a problem. I'll double check if I can get access to Ipad again. This was my work around to fix the black rendering issue. You should be seeing resize events for HLS on IOS.