| Summary: | HLS WebGL video textures black rendering since IOS 11 | ||||||
|---|---|---|---|---|---|---|---|
| Product: | WebKit | Reporter: | Daniel Rossi <electroteque> | ||||
| Component: | WebGL | Assignee: | Nobody <webkit-unassigned> | ||||
| Status: | RESOLVED WORKSFORME | ||||||
| Severity: | Blocker | CC: | anthony.violo, arnaud, ashley, dino, dustin.kerstein, florent.thiery, formatkaka, gideon.keyson, jeremy, jer.noble, stephan, webkit-bug-importer | ||||
| Priority: | P2 | Keywords: | InRadar | ||||
| Version: | Safari 11 | ||||||
| Hardware: | iPhone / iPad | ||||||
| OS: | iOS 11 | ||||||
| See Also: |
https://bugs.webkit.org/show_bug.cgi?id=163866 https://bugs.webkit.org/show_bug.cgi?id=181445 |
||||||
| Attachments: |
|
||||||
|
Description
Daniel Rossi
2017-11-08 00:39:58 PST
It didn't hyperlink the ticket number. Here is the original bug ticket that was unresolved. IOS10 can be worked around with the FlipY work around code posted. https://bugs.webkit.org/show_bug.cgi?id=163866 Anything ? IOS 11 has broken HLS live 360 streaming for many people. But they use it for VOD also not mp4. Thanks Daniel for keeping this issue alive and being persistent about this. We have been using HLS for our 360 video player and have run into the same dead end for at least a third of our users that before could enjoy their stories on their iPhones. It is extremely frustrating, since our users of course don't dive into the technicalities of cross-browser/ platform issues and just expect things to work and of course the blame is on our end with this. If this is not looked in to, we'd have to do a considerable technology switch, or even worse develop on multiple paths even adding the app option. We also have the same issue which prevents us from supporting 360° video on iOS 11+ with HLS. We could work our way around in iOS 10 but that is not possible with iOS 11+. Given that HLS is the only mean to reach iOS Safari with ABR streaming this is a deal breaker for many trying to bring the latest features to iOS. My fix in the previous ticket works around HLS black rendering issues in IOS10. take a look. So it's good enough for IOS10. It has been posted on the three.js ticket also. These tests were originally designed to run within our testing harness, and not necessarily all browsers. So it would be helpful if they were modified to include <video playsinline muted> to each of the elements. Then, they'd be testable on all iOS devices, including iPhones. Dino implemented flipY in bug# 176491, but not in time for iOS 11.0 or macOS 10.13.0. If you have access to Apple's Developer program, you can test out the fix in-place in the macOS 10.13.2 beta 3 seed or the iOS 11.2 beta 3 seed: <https://developer.apple.com/download/> Here on my 11.2 beta device, I see the HLS tests which Daniel posted are working correctly (though annoyingly trying to enter full screen mode due to a lack of the 'playsinline' attribute on the elements in the test). The HLS tests don't work with FlipY disabled. In IOS10 it does, and the shader fix will flip it around.
Is that what you mean. It now requires it enabled but is pushed to 11.2 ?
I'm sorry but I had to buy a new device to test (don't specifically work with IOS or OSX) and all I could get was an Ipad no Iphone. Which inline isn't a problem with Ipad.
I added the inline attributes to the original webkit test files scripts.
video.setAttribute("playsinline", "");
video.setAttribute("webkit-playsinline", "");
Hopefully that helps.
So you are saying the webkit test files with HLS bring up a frame in 11.2 or showing a black square ?
I'm installing the beta now and let you know what I discover.
The webkit tests I made now work in 11.2. So we got past that part. However my threejs and raw webgl tests are still not rendering correctly. I tried to replicate the webgl code directly. however if I use RGBA I get a transparent canvas. Any ideas then ? threejs - http://dev.electroteque.org/webgl/three-hls.html raw webgl - http://dev.electroteque.org/webgl/webgl-hls.html I can't confirm because I can't figure out how to hack the test files from repeating. There is a black frame showing up with my demo HLS video. The test HLS file I made from the mp4 in the webkit tests doesn't display in the raw webgl example. It's a black frame or transparent if RBA is chosen. So it seems the code has to match and be specific to the test code. Any ideas how to replicate all that over I tried adding some. (In reply to Daniel Rossi from comment #10) > The webkit tests I made now work in 11.2. So we got past that part. > > However my threejs and raw webgl tests are still not rendering correctly. I > tried to replicate the webgl code directly. however if I use RGBA I get a > transparent canvas. > > Any ideas then ? > > threejs - http://dev.electroteque.org/webgl/three-hls.html > raw webgl - http://dev.electroteque.org/webgl/webgl-hls.html Looks like the 'loadeddata' event isn't firing with HLS streams, so your init() function is never being called. I believe this is a known issue, but you should be able to work around it by using a different event to do your initialization. That is not correct. If that is the case it would have been broken in 7.2 from 7.1. The animation frame is starting, loadeddata event is working correctly. I have debugged that to confirm, although the log won't keep showing it. I've updated this , to show in the console the animationframe is starting. Black frame. I will have to spend time picking out the code from the test files to replicate it. http://dev.electroteque.org/webgl/webgl-hls.html I meant 11.2 from 11.1 can't edit the comment. I've produced another test with my external video. the color needs to be set to RGB or else it's a transparent frame. I had to comment out some code regarding to color mask also. I have heavily modified this to run an animation frame and remove the color tests. There is a log to show the animation is running. It's just a black frame here. I suspect it might be CORS related which I will test next. Only the local color test HLS video is working so far. http://dev.electroteque.org/webgl/webkit/hlstest.html Thankfully it is not CORS related. Same problem locally. Don't know what else to report there yet. Is there a special webgl flag required for this particular video ? I've encoded and packaged them via Elastic Transcoder. It's fine on IOS 10 with the flipy work arounds is all. I'm so sorry for the noise. The original HLS webkit tests are still not working and black for me on IOS 11.2 beta. I was clicking the mp4 examples farout. I just wasted some time to AB my external source. Do you mind uploading your own test for me to try on IOS 11.2 beta ? Is there a different IOS version to install now ? I am unable to replicate the fix for HLS. I was confusing myself clicking the wrong mp4 tests. Hence why my own external tests are still a black frame too. Sorry, I haven't had time to look at this today. I'll keep you updated when I get a chance to dig into this again. I don't have an environment to run builds myself and get something working I'm sorry I am no help there. I'm on Windows. I was able to play around with webgl flags before and get it working so I'm wondering if it's another one of those issues. I think I have a handle on what might be causing this. The OpenGL code which copies textures in from the video element may be stomping on some of the WebGL state established by the page. Moving the VideoTextureCopierCV to use a shared context (with shared textures but separate state) may fix this. Stay tuned. Oh righ. yes according to the gl upload code, I see what you mean about the texture context. Could the viewport size have anything to do with it also ? https://github.com/graouts/webkit/blob/0b791d19078bac6b864285cee874218f1374a7ad/Source/WebCore/platform/graphics/mac/WebGLLayer.mm#L94 What about this also ? The previous IOS10 fix required switching the colourspace on the shader to bgra for IOS and HLS. After disabling FlipY , the colours were inverted. So that was the full partial work around for IOS 10 and HLS. According to this it's not using colourspace at all for IOS, as far as I see. https://github.com/graouts/webkit/blob/0b791d19078bac6b864285cee874218f1374a7ad/Source/WebCore/platform/graphics/mac/WebGLLayer.mm#L142 How is it going ? Any luck ? Any update ? Hi it looks like its going to be a very long wait. I waited a little before deciding to have to do a software rendering workaround which will chew battery power and cause performance and frame drop problems. you have to draw to a temp canvas and use the canvas as a webgl texture. Im not at my IDevice yet to properly test if this works see https://github.com/mrdoob/three.js/issues/9754#issuecomment-350685693 I have this same method now working again for IE 11. So if using three.js you set the canvas as the texture. new THREE.Texture(canvas); Then draw the video to the context context.drawImage( video, 0, 0, canvas.width, canvas.height ); I'm working on a demo to update that other ticket once I can test on IDevice. Until there is more users migrating to IOS 11 from 10. You will have to keep the original HLS flipy workarounds documented in the previous ticket especially the colour channel inversion on the fragment shader. You will have to do IOS 11 version detection to treat everything different so it doesn't use the Flipy and colour inversion work arounds also as its also needed to check for CORS support. Hey, Thanks for suggesting the workaround. Can you share some stats regarding performance and frame drop ? I won't be back to an IDevice in a few days. But will do a test with a fps stats. I dont even know yet if it will even render. Stats or not stats I have people waiting for a fix on my end for Live HLS so has to be for now. We've fixed some bugs in iOS video -> GL textures recently. I can't confirm that this particular issue is resolved yet though. I am still running Beta IOS 11 to keep on the same page. And have some updates to do and will confirm in a few days. Thanks for the update. Hi I am on 11.2.5 beta. There is still no change with HLS. It is a black frame even with FlipY disabled. I have just discovered with Safari 11 on macOS. The FlipY work around is still required for HLS. Or else it wont render. I may have to make a separate report for that ? I have made a canvas texture test. It is working but a larger resolution video might make it fall over. This HLS is low quality so is keeping at 40 fps. A larger resolution mp4 but also low quality stayed at 30fps. A production resolution file might get 10fps out of it. http://dev.electroteque.org/webgl/three-canvas.html Hey daniel, great work. It is running at 60 fps for my device (mac book pro - 2015). This is an issue for IOS. I am testing with a current basic Ipad. Although Safari 11 suffers a FlipY issue on macOS still when using HLS. I just tested it. I have documented a fix for that in a previous ticket. I thought it was fixed up but isn't ? You need to disable FlipY and use the shader to reverse it's Y position. You don't need canvas texture rendering for macOS ! You need to do crazy version checks for macOS to first detect it can support CORS. Then you need to do a further check for IOS 11 to support CORS. Or else have to provide a CORS proxy source for previous Safari. Then you need to do the FlipY shader fix for Safari macOS and IOS10 . A colour inversion fix for IOS 10 on the fragment shader. And now do canvas rendering for IOS 11 HLS only. Pretty complicated I know. That whole bit above is still causing me problems getting it all to work because of bugs with the version detection requirements. Here is the fix for what you might need for macOS here. https://bugs.webkit.org/show_bug.cgi?id=163866#c16 I have created a seperate ticket for HLS textures on Safari 11 macOS . The problem is still there with FlipY. This current ticket is related to IOS 11. https://bugs.webkit.org/show_bug.cgi?id=180863 So in regards to this comment https://bugs.webkit.org/show_bug.cgi?id=179417#c8 11.2 beta of IOS was still a problem. But I missed the bit about macOS which I created a seperate ticket for. I don't believe this issue is related to FlipY as disabling it does nothing. But FlipY does affect macOS with HLS. If you think 10.13 is fixed with FlipY for HLS textures I might not be able to confirm for a while as I would need to get access to that. It's another configuration I cant get access to as I have no hardware anymore, and in the past doing major OS updates affected and broke all my other audio software I was running. I don't have a dedicated test machine just for these problems. Mp4 textures was never a problem with FlipY enabled on macOS or IOS only HLS. see https://bugs.webkit.org/show_bug.cgi?id=180863#c3 More information has come to light and I have updated the title. Both IOS 11.2 beta and macOS 10.13 suffer the same problem now. Both are affected and the documented flipy fix does nothing for HLS video textures. Both require temporary canvas texture rendering work arounds which gets about 30fps on the Ipad. It seems possible this said FlipY ticket created a regression of soughts ? https://bugs.webkit.org/show_bug.cgi?id=176491 https://bugs.webkit.org/show_bug.cgi?id=180863#c4 I cant delete my comment. disabling FlipY for 10.13 is ok now I believe. This problem is only relevant to IOS 11. Any update here ? I have been keeping the IOS beta up to date and no change yet. Any update please ? I reported this in November now. The test is now working in 11.3 beta. No idea what is going on but it seems to be working. Without any response or update ! It's also rendering without the flipY workaround but I think I mentioned this before. FlipY is still a problem for macOS Safari. Thanks Daniel, I tried this now with iOS 11.3 rolled out for our A-frame based application and can confirm videos are working again. Had to remove the color channel switch that was necessary pre iOS 10.3. Thanks again for keeping this bug report alive over time. That's ok. I'm marking as resolved. Hi Daniel, thanks for the detailed bug report. I am still getting the same bug on iOS and safari 12. Could I contact you privately in order to ask some questions? Thank! I have no macOS hardware atm to test it. I can get access to an Ipad to see what is going on. It may require the flipY work around. The colourspace workaround, or using software render for that version like with 11.0. I have 3 sets of versions checks in place to provide all those workarounds. I haven't checked back in on my VR feature for a while and haven't had much interest apart from one recently. I am seeing what seems to be this bug (or related) again on iOS 14.3 on an iPhone 12 Pro. It does not happen on non iPhone 12 iOS devices. Also is an issue with iOS 14.2.1; I'm not sure at what iOS version the iPhone 12 started rendering HLS video all black when used as a text source in WebGL. Our WebGL app uses .m3u8 HLS streams and renders them using a from scratch WebGL renderer with a <video> element as the texture source. This has been working for the last year or more, and still works fine on iOS devices other than iPhone 12s. If I change to "fast start" MP4 files, things work fine on all iOS devices for us. We have hundreds of thousands of transcoded .m3u8 HLS streams so it's not a simple matter to switch to MP4 files. You need to update things on rendition resize events. here is mine. I've let others know also. It will stop the black rendering on startup for me.
video.addEventListener("resize", () => {
vrVideo.update();
vrVideo.fullScreenResize(player.isFullscreen);
});
within update I have
this.video.width = MathUtils.ceilPowerOfTwo(this.video.videoWidth);
this.video.height = MathUtils.ceilPowerOfTwo(this.video.videoHeight);
Thank you for your help! Unfortunately, it's not working and/or maybe I'm not following. Here's my situation: I have a custom WebGL based renderer. I create an offscreen <video> element pointing to an .m3u8 URL that has CORS setup appropriately. When the video has reached a readyState of "4" I create an RGB format WebGL texture of size videoWidth and videoHeight passing the HTMLVideoElement instance for the pixel data. The texture renders as all black pixels. If I changed the <video> element to point to an .mp4 URL on the same server, wait for the offscreen <video> element to reach a readyState of "4" the texture renders with the video pixels. Taking your advice, I added an event handler for "resize" to my offscreen <video> HTMLVideoElement element. That event handler does get called back, and my plan was to recreate the WebGL texture and make sure I'm using updated dimensions for videoWidth and videoHeight. Unfortunately, the "resize" callbacks happen well before the video reaches a readyState of "4" and the videoWidth and videoHeight dimensions are unchanged (and correct). In summary, the WebGL texture is only created after the readyState is "4" and all the "resize" events have been fired on HTMLVideoElement and I still see all black texels in the video. This is only the case for HLS/.m3u8 videos. Everything works fine in the case of .mp4 videos. The .m3u8 videos that do not work are encoded with identical resolution, bitrates, profile settings, etc as the .mp4 videos that do work. Additionally, everything works fine on all other iOS devices with either HLS/.m3u8 or .mp4 except on iPhone 12 devices, where only .mp4 videos are working as a TextureSource for WebGL textures. I have confirmed the problem behavior on multiple iPhone 12 devices with iOS 14.2.1+ and confirmed the successful behavior on multiple other iOS devices (i.e. iPad, iPhone 11, etc). Am I misunderstanding your suggestion? Thank you so much for taking the time to reply! I just set it up on metadata. I've moved to an AMD system so macOS vmware for IOS simulator is tricky right now. But I double checked people's complaints on IOS 14 Ipad. It was not a problem. I'll double check if I can get access to Ipad again. This was my work around to fix the black rendering issue. You should be seeing resize events for HLS on IOS. |