Bug 163866 – UNPACK_FLIP_Y_WEBGL breaks HLS video textures

• Status: RESOLVED
• Resolution: CONFIGURATION CHANGED
• Component: WebGL

[Daniel Rossi]

Hi there after some extensive testing I have confirmed that in Safari 10 it has broken WebGL video textures for HLS streams. It is an issue on IOS 9 Safari as well as Webkit nightly. 


It is regarding this FlipY flag

_gl.pixelStorei( _gl.UNPACK_FLIP_Y_WEBGL, true );

If this is disabled the video texture is rendering but it is upside down.Although I believe this flag is needed for hardware acceleration correct ? 


Reference to three.js ticket. 

https://github.com/mrdoob/three.js/issues/9754

Reference to possibly related webkit ticket #162491

I have found some raw webgl examples that I have modified for the test. The HLS streams use a CORS proxy because of the known severe limitation there. 

With FlipY disabled and "working"

http://dev.electroteque.org/threejs/webglworking.html


With FlipY enabled

http://dev.electroteque.org/threejs/webglflip.html

[Daniel Rossi]

Just to do heads in further. This FlipY "fix" doesn't help on IOS 9. Something else is a problem there now. It took days just to track this down.

[Daniel Rossi]

I don't know how to replicate this with raw webgl but flipping the "uv" Y around works around this possible bug. So it shows the right way up. 

It still won't render on IOS 9 although I can see it trying to. The frame rate with webgl is appalling on the Ipad 3 so usually sits around 5 frames. It is showing 10 frames per second. 

http://dev.electroteque.org/threejs/hls.html

[Radar WebKit Bug Importer]

<rdar://problem/28927576>

[Daniel Rossi]

According to this comment.  the flipy work around seems to work on IOS 10 but not IOS 9. 

I need to confirm this. 

Still it would be of use to fix this FlipY flag bug also than constant work arounds just because of Webkit. 

https://bugs.webkit.org/show_bug.cgi?id=135379#c75

[Daniel Rossi]

I am hoping the WebView in IOS10 is the same as Safari then and doesn't have the bug still.

[Edwin]

(In reply to comment #5)
> I am hoping the WebView in IOS10 is the same as Safari then and doesn't have
> the bug still.

Daniel, I tried to run this example on iOS 10.1.1 Safari, but seems not working, as long as webgl start to render texture from video, safari crashed. So maybe still not be possible to run on iOS

[Daniel Rossi]

That was the same for me. 

And you say this is working on IOS 10 , just ignoring the texture upside down ? 

http://dev.electroteque.org/threejs/webglworking.html

[Daniel Rossi]

This is working on OSX Safari so there is clear differences for no reason with IOS

http://dev.electroteque.org/threejs/hls.html

[Daniel Rossi]

According to Edwin the raw webgl is working in IOS 10 where the fix for the FLIPY flag in three.js is not.  The comment about the colourspace is of interest. Perhaps the RGB flag is causing the issue when rendering the texture ? 

"http://dev.electroteque.org/threejs/webglworking.html this one.

I tried on my iPhone 7 iOS 10.1.1 Safari, and it can successfuly render the image from video through webgl, however the color is incorrect and the image is upside down.

Also the image seems pretty laggey, I’m now trying to make this example better and try to play the whole m3u8 video with correct color."

[Daniel Rossi]

After some very heavy and painful testing I have fined tuned the demos and the issue. 

The FlipY work around is required for HLS across both OSX 10.11 and macOS , IOS 9 and 10. The raw webgl example doesn't use FlipY because its doing in in the shader program. With three.js the geometry has to be flipped around. 

HLS rendering on IOS 10 is displaying but has severe colour artifact issues. The frames stop working but I believe its an issue with the emulator and frames dropping. I now have no device that can be updated to IOS 10.  It doesn't show up at all on IOS 9. Both require the CORS proxy for mp4 and HLS. 

I may have to now provide another ticket in regards to the colour rendering problem with HLS on IOS 10 once I confirm it's not the emulator. 

This is absolutely painful no wonder nobody wants to support Safari but people need it working. 

I've had to provide here different rgb format settings. The default one produces vertical coiourbars , the rgba one the colours are incorrect. But in the emulator. I am now unable to properly test IOS 10 because my Ipad device has become unsupported. I'll try and get a hold of an Iphone with IOS 10 to confirm. 

http://dev.electroteque.org/webgl/threejs-hls.html
http://dev.electroteque.org/webgl/threejsrgba-hls.html

http://dev.electroteque.org/webgl/webgl.html
http://dev.electroteque.org/webgl/webglrgba.html

[Daniel Rossi]

Please concentrate on these two examples. the RGB flags make no difference on an actual device, the simulator produces faulty output but similar output when using the RGBA format. 

The FlipY work around is required for IOS 9 / 10/ OSX Safari with HLS. 

I have been able to replicate a colour issue with HLS webGL rendering which now needs a second ticket. 

So many massive flaws and bugs causing this to be a show stopper. 

http://dev.electroteque.org/webgl/threejs-hls.html
http://dev.electroteque.org/webgl/webgl.html

[Daniel Rossi]

So such HLS webgl output is only partially working on IOS 10 not IOS 9. Many devices are stuck on 9 now. I need to find a work around for 9.

[Daniel Rossi]

please see fix here. Close this if you wish. This works around this particular bug. 

https://github.com/mrdoob/three.js/issues/9754#issuecomment-268197088

[Dean Jackson]

https://github.com/mrdoob/three.js/issues/9754#issuecomment-268197088

I'm taking a look at fixing this in iOS now. But I'm a bit confused by the comments.

For HLS videos:

- iOS is providing color channels in the wrong order (bgra instead of rgba). The values are otherwise correct though.
- macOS is producing a texture that requires flipping in the X axis. This can be worked around by using setPixelStorei(UNPACK_FLIP_Y_WEBGL, true)
- iOS is producing a texture that requires flipping in the X axis, but the above workaround does not work.

Is this correct?

[cjl]

This bug still exist on IOS 11, anyone has solved this bug,please tell me ,thanks!

[Daniel Rossi]

@Dean Jackson

I am the original reporter. 

_gl.pixelStorei( _gl.UNPACK_FLIP_Y_WEBGL, true );

This has to be disabled. This inverts the picture natively because it renders upside down by default. 

Using this causes a black picture with HLS. 

I am now seeing reports by clients my flipy work arounds are now broken in IOS11. It's now a black picture again with HLS ! 

I don't have a new device to get IOS11. Iphone 5 has now been deprecated so I have absolutely nothing left to test with. 

The simulator shows up different I think. 


It has to be done within the shader. 

For a flipy in the vertex shader for Safari with HLS it's this

varying vec2 vUV;
            void main() {
            	vUV = vec2( uv.x, 1.0 - uv.y); 
            	gl_Position = projectionMatrix * modelViewMatrix * vec4( position, 1.0 );
            }

This is the current work around and

_gl.pixelStorei( _gl.UNPACK_FLIP_Y_WEBGL, false );

I am very much struggling to test faults bought in with IOS11 because , more devices are knocked off the list getting it.

The HLS color channel inversion for IOS is another bug work around which I mentioned in Three.js where I reported the fix. 

It's this for IOS10 and I guess 11 with HLS. In the fragment shader. 

uniform sampler2D texture;
            varying vec2 vUV;
            void main() {
            	gl_FragColor = texture2D( texture, vUV ).bgra;
            }

[cjl]

Do you have resolvent of this problem?I have ios11 device and I need to solve this problem now，could you tell me something about this？

[Daniel Rossi]

It sounds like both Flipy and inverting the flip y fix. 

Does not resolve HLS black rendering for IOS11. There has been a "regression" of a bug. 

Everyone can replicate it. It's not a problem in the simulator. 

I am helpless to find another work around as I have no new device yet. I've been caught up in planned obsoleteness. I don't do IOS developement and don't need new devices normally. My focus has been on GearVR and S8 for mobile. 

Once I manage to at least get an old Iphone 6 although and Ipad is more useful. I'll track a fix down and report it yet again.

[Daniel Rossi]

Confirming my flipY fixes have been sabotaged. For absolutely everyone. Well done ! 

Black rendering frame for HLS. 

Disabling flipy should show a picture but isn't. 

I finally got a new device to test it. The IOS11 simulator shows it working somehow. 

I will try my best to get a work around working again heres hoping. This issue needs to be fixed ASAP. It has rendered HLS / webgl useless. 

This could possibly still be a CORS issue but just for HLS. Will need to spend time working it out.

[Daniel Rossi]

This exact same problem was for IOS9 and was unfixable. I wouldn't call it a regression as such because it was broken to begin with.

[Daniel Rossi]

I'm going to create a new bug ticket. With yet a brand new problem with HLS video textures and WebgL in IOS 11.1. 

I highly doubt this was ever tested and HLS is supposed to be an Apple streaming format. 

I will create multiple tests from these original ones to prove the FlipY issue doesn't resolve it anymore. 

Then I will close this stale ticket as move I suppose. It has now moved beyond the original problem. 

Please hold.

[Daniel Rossi]

Closing this and starting a new ticket. This problem with HLS video textures has moved on now from last year since this ticket was created. 

The work around will work for IOS10 only. IOS9 won't work and IOS11 is now suffering the same problem as IOS9. 

According to the WebKit git. I see absolutely no reference to HLS video texture tests with WebGL. Only mp4 has been used in the tests. 

Meaning Apple's own streaming format has never been tested against webgl and video textures for an Apple supplied browser. 

This would explain why it's broken. 

I had to purchased a new device myself, to first replicate the problem and be able to test and compile examples and evidence of the problem. 

I am hoping to use the webkit tests exactly and replace with a HLS stream as a variation also.

Please leave this open as I can still reproduce the issue on ios 11, steps:

1. Visit https://www.learnthreejs.com/beginners-tutorial/
2. Use anything that is labeled three.js
3. Issue still occurs.

Thoughts?

[Daniel Rossi]

Please follow this new ticket. FlipY has no effect now in IOS11. 

It has moved onto much bigger problems rather than the new OS fixing problems.  

https://bugs.webkit.org/show_bug.cgi?id=179417

[Daniel Rossi]

I have created a seperate ticket for HLS textures on Safari 11 macOS . The problem is still there with FlipY. 

https://bugs.webkit.org/show_bug.cgi?id=180863