Bug 163866 - UNPACK_FLIP_Y_WEBGL breaks HLS video textures
Summary: UNPACK_FLIP_Y_WEBGL breaks HLS video textures
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebGL (show other bugs)
Version: Safari 10
Hardware: Macintosh OS X 10.10
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2016-10-23 02:53 PDT by Daniel Rossi
Modified: 2017-04-06 19:05 PDT (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Rossi 2016-10-23 02:53:57 PDT
Hi there after some extensive testing I have confirmed that in Safari 10 it has broken WebGL video textures for HLS streams. It is an issue on IOS 9 Safari as well as Webkit nightly. 


It is regarding this FlipY flag

_gl.pixelStorei( _gl.UNPACK_FLIP_Y_WEBGL, true );

If this is disabled the video texture is rendering but it is upside down.Although I believe this flag is needed for hardware acceleration correct ? 


Reference to three.js ticket. 

https://github.com/mrdoob/three.js/issues/9754

Reference to possibly related webkit ticket #162491

I have found some raw webgl examples that I have modified for the test. The HLS streams use a CORS proxy because of the known severe limitation there. 

With FlipY disabled and "working"

http://dev.electroteque.org/threejs/webglworking.html


With FlipY enabled

http://dev.electroteque.org/threejs/webglflip.html
Comment 1 Daniel Rossi 2016-10-23 03:05:03 PDT
Just to do heads in further. This FlipY "fix" doesn't help on IOS 9. Something else is a problem there now. It took days just to track this down.
Comment 2 Daniel Rossi 2016-10-23 04:02:37 PDT
I don't know how to replicate this with raw webgl but flipping the "uv" Y around works around this possible bug. So it shows the right way up. 

It still won't render on IOS 9 although I can see it trying to. The frame rate with webgl is appalling on the Ipad 3 so usually sits around 5 frames. It is showing 10 frames per second. 

http://dev.electroteque.org/threejs/hls.html
Comment 3 Radar WebKit Bug Importer 2016-10-24 18:01:46 PDT
<rdar://problem/28927576>
Comment 4 Daniel Rossi 2016-11-05 04:41:43 PDT
According to this comment.  the flipy work around seems to work on IOS 10 but not IOS 9. 

I need to confirm this. 

Still it would be of use to fix this FlipY flag bug also than constant work arounds just because of Webkit. 

https://bugs.webkit.org/show_bug.cgi?id=135379#c75
Comment 5 Daniel Rossi 2016-11-05 04:42:33 PDT
I am hoping the WebView in IOS10 is the same as Safari then and doesn't have the bug still.
Comment 6 Edwin 2016-11-05 05:06:41 PDT
(In reply to comment #5)
> I am hoping the WebView in IOS10 is the same as Safari then and doesn't have
> the bug still.

Daniel, I tried to run this example on iOS 10.1.1 Safari, but seems not working, as long as webgl start to render texture from video, safari crashed. So maybe still not be possible to run on iOS
Comment 7 Daniel Rossi 2016-11-05 05:23:15 PDT
That was the same for me. 

And you say this is working on IOS 10 , just ignoring the texture upside down ? 

http://dev.electroteque.org/threejs/webglworking.html
Comment 8 Daniel Rossi 2016-11-05 05:24:52 PDT
This is working on OSX Safari so there is clear differences for no reason with IOS

http://dev.electroteque.org/threejs/hls.html
Comment 9 Daniel Rossi 2016-11-05 21:36:19 PDT
According to Edwin the raw webgl is working in IOS 10 where the fix for the FLIPY flag in three.js is not.  The comment about the colourspace is of interest. Perhaps the RGB flag is causing the issue when rendering the texture ? 

"http://dev.electroteque.org/threejs/webglworking.html this one.

I tried on my iPhone 7 iOS 10.1.1 Safari, and it can successfuly render the image from video through webgl, however the color is incorrect and the image is upside down.

Also the image seems pretty laggey, Iā€™m now trying to make this example better and try to play the whole m3u8 video with correct color."
Comment 10 Daniel Rossi 2016-11-08 12:14:04 PST
After some very heavy and painful testing I have fined tuned the demos and the issue. 

The FlipY work around is required for HLS across both OSX 10.11 and macOS , IOS 9 and 10. The raw webgl example doesn't use FlipY because its doing in in the shader program. With three.js the geometry has to be flipped around. 

HLS rendering on IOS 10 is displaying but has severe colour artifact issues. The frames stop working but I believe its an issue with the emulator and frames dropping. I now have no device that can be updated to IOS 10.  It doesn't show up at all on IOS 9. Both require the CORS proxy for mp4 and HLS. 

I may have to now provide another ticket in regards to the colour rendering problem with HLS on IOS 10 once I confirm it's not the emulator. 

This is absolutely painful no wonder nobody wants to support Safari but people need it working. 

I've had to provide here different rgb format settings. The default one produces vertical coiourbars , the rgba one the colours are incorrect. But in the emulator. I am now unable to properly test IOS 10 because my Ipad device has become unsupported. I'll try and get a hold of an Iphone with IOS 10 to confirm. 

http://dev.electroteque.org/webgl/threejs-hls.html
http://dev.electroteque.org/webgl/threejsrgba-hls.html

http://dev.electroteque.org/webgl/webgl.html
http://dev.electroteque.org/webgl/webglrgba.html
Comment 11 Daniel Rossi 2016-11-09 03:45:22 PST
Please concentrate on these two examples. the RGB flags make no difference on an actual device, the simulator produces faulty output but similar output when using the RGBA format. 

The FlipY work around is required for IOS 9 / 10/ OSX Safari with HLS. 

I have been able to replicate a colour issue with HLS webGL rendering which now needs a second ticket. 

So many massive flaws and bugs causing this to be a show stopper. 

http://dev.electroteque.org/webgl/threejs-hls.html
http://dev.electroteque.org/webgl/webgl.html
Comment 12 Daniel Rossi 2016-11-09 03:47:17 PST
So such HLS webgl output is only partially working on IOS 10 not IOS 9. Many devices are stuck on 9 now. I need to find a work around for 9.
Comment 13 Daniel Rossi 2016-12-20 07:13:02 PST
please see fix here. Close this if you wish. This works around this particular bug. 

https://github.com/mrdoob/three.js/issues/9754#issuecomment-268197088
Comment 14 Dean Jackson 2017-04-06 19:05:01 PDT
https://github.com/mrdoob/three.js/issues/9754#issuecomment-268197088

I'm taking a look at fixing this in iOS now. But I'm a bit confused by the comments.

For HLS videos:

- iOS is providing color channels in the wrong order (bgra instead of rgba). The values are otherwise correct though.
- macOS is producing a texture that requires flipping in the X axis. This can be worked around by using setPixelStorei(UNPACK_FLIP_Y_WEBGL, true)
- iOS is producing a texture that requires flipping in the X axis, but the above workaround does not work.

Is this correct?