Bug 164540 - HLS WebGL video textures produces faulty colour rendering in IOS 10
Summary: HLS WebGL video textures produces faulty colour rendering in IOS 10
Status: RESOLVED CONFIGURATION CHANGED
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebGL (show other bugs)
Version: Safari 10
Hardware: iPhone / iPad iOS 10
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2016-11-09 03:55 PST by Daniel Rossi
Modified: 2018-04-17 16:19 PDT (History)
6 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Rossi 2016-11-09 03:55:59 PST
Regarding #163866

I am separating this into a different ticket but they might be related to the FlipY bug. 

I was able to successfully test the HLS webGL rendering on an Iphone after the FlipY work arounds. The simulator was causing similar colour faults, but when using the RGBA format flag, the output I see on the Iphone is the same regardless which RGB format is used. 

The colours are rendering incorrect. Using this code

gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGB, gl.RGB, gl.UNSIGNED_BYTE, video);

If there is a quick fix work around rather waiting months for fixes that could be a start. 

So such HLS webgl output is only partially working on IOS 10 not IOS 9. Many devices are stuck on 9 now like my Ipad 3. I need to find a work around for 9.

http://dev.electroteque.org/webgl/threejs-hls.html
http://dev.electroteque.org/webgl/webgl.html

Thanks.
Comment 1 Daniel Rossi 2016-11-09 03:56:40 PST
Regarding this FlipY bug  ticket even 

https://bugs.webkit.org/show_bug.cgi?id=163866
Comment 2 Radar WebKit Bug Importer 2016-11-09 03:58:00 PST
<rdar://problem/29177493>
Comment 3 Radar WebKit Bug Importer 2016-11-09 03:58:46 PST
<rdar://problem/29177494>
Comment 4 Daniel Rossi 2016-11-14 04:18:11 PST
According to this comment. 

On their IDevice only the RGBA flag would get a render for them but was producing the same colourspace issue with HLS. On the IDevice I tested RGB flag was ok and also producing the colourspace problem. 

Within the simulator the RGB flag would render with green vertical bars through it and only the RGBA flag would work. 

I'm hoping there is an actual combination of flags between the colour format and the byte type that will make it work ? 

https://github.com/mrdoob/three.js/issues/9754#issuecomment-260238925
Comment 5 Daniel Rossi 2016-11-20 18:07:12 PST
Is there going to be any way to fix it ? 

IOS is pretty much locked out of Live WebVR with HLS right now. 

IOS9 seems buggered up and all older devices stuck on IOS 9 unable to get the IOS10 update are stuffed. 

I tried all possible colour format flag, and any byte flag available and no good. Only unsigned byte type and RGB RGBA colour format is possible with this. 

If it can be fixed with how the HLS is encoded and packaged that would be good to know. I can't see what the difference is between an mp4 file and a HLS packaged version of it, same encoding.
Comment 6 pedrohfranceschi 2016-11-30 10:17:42 PST
This bug is very annoying and is putting iOS completely out of VR live streaming.

Is there any timeline regarding this bug? Please fix it.
Comment 7 Daniel Rossi 2016-12-19 09:31:29 PST
Someone was lucky to figure out what it was. 

Webkit is delivering textures from HLS video with rgb colours switched around. The red and blue is switched. 

I believe they ran a test by switching the colours in the source file. This solution isn't good enough because it will be impossible to work around with live streams. 

please see commend

https://github.com/mrdoob/three.js/issues/9754#issuecomment-267840695

I believe you can do something like this to swap the red and blue colours but it doesn't work in that raw webgl demo. It needs other stuff changed

gl.shaderSource(ps, "precision mediump float;uniform sampler2D sm;varying vec2 tx;void main(){gl_FragColor=texture2D(sm,txt).bgra;}");


So 

texture2D(sm,txt)

is collecting the texture colours

and

bgra swaps the red and blue colour channel order.
Comment 8 Daniel Rossi 2016-12-19 09:57:34 PST
There was a program error simply copying and pasting. 

Using this code above. I can confirm what is being rendered is what is visible in IOS 10 so it should hopefully invert the red and blur colours that are swapped in the shader ? 

http://dev.electroteque.org/webgl/webglbgra.html

Is this easily fixable ? Is anyone out there ?
Comment 9 Daniel Rossi 2016-12-19 10:42:38 PST
Confirming rotating the colour channel order , it gets inverted back on IOS 10. I had to add this colour format to display properly in the simulator for now. 

gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, video);
Comment 10 Daniel Rossi 2016-12-20 07:13:52 PST
please see fix here. Close this if you wish as I have no doubt it will ever be bothered to be fixed. 

This works around this particular bug. IOS is inverting the rgba colours on video textures for HLS. They need to be pre-inverted and it will switch them around. 

https://github.com/mrdoob/three.js/issues/9754#issuecomment-268197088
Comment 11 Daniel Rossi 2016-12-20 07:14:34 PST
IOS9 still refuses to render HLS video textures.
Comment 12 Jon Lee 2016-12-20 14:34:41 PST
(In reply to comment #10)
> please see fix here. Close this if you wish as I have no doubt it will ever
> be bothered to be fixed. 

Thanks for the pointer and for doing a lot of the investigation. But statements like this last one aren't productive to the project.
Comment 13 Daniel Rossi 2016-12-20 21:04:51 PST
A response would be appreciated. I spent months with others figuring that out. As I mentioned when it can be bothered to be fixed. I said nothing wrong. 

You may close if you wish and people can see the work around. Thanks.
Comment 14 Daniel Rossi 2017-11-06 02:08:40 PST
This is now invalid for IOS 11. Work around fix is for IOS 10 only. HLS completely broken for IOS 11 now. No possible work around yet.
Comment 15 Justin Fan 2018-04-17 16:19:53 PDT
Configuration change in iOS 11.3 fixed this issue. Tested this and is all working now.