Bug 133511 – iOS 8 WebGL cannot pass video to texImage2d

• Component: WebGL
• Status: RESOLVED FIXED

[Ashley Gullen]

Visit the URL http://www.scirra.com/labs/bugs/webglvideo/. It creates a simple spinning cube and plays a clip of "big buck bunny". The video is used as the cube's texture. The video is 

On desktop Chrome and Firefox it works fine. On IE11 it does not work since it also does not support video being passed to texImage2d, but it's possible to work around since it can render video to a canvas 2D context and then pass that 2D canvas to texImage2d.

On Chrome for Android it works. Note the video does not start until the first touch event (the demo is coded to start the video in the first touchstart to work around the gesture requirement).

On Safari iOS 8 beta on an iPhone 4S, after touching the display remains blank - the texture is not updated. However the audio from the video starts playing and the fullscreen video playback controls appear. These are unwanted for this demo and likely unwanted for most use cases of playing video within a WebGL context. I could not get the workaround of rendering via a canvas2d to work either, and playback controls still appeared. (FWIW it appears iOS 7 does not support passing video to canvas2d drawImage() at all, and that iOS 8 adds support for this, but in my experimenting I still did not get a canvas2d with video drawn on to it to upload to a texture either.)

The most convenient solution is if video can be passed directly to texImage2d, and not have playback controls appear.

[Ashley Gullen]

Sorry, editing fail. Last line of first paragraph should read "the video is h.264 encoded in .mp4 container and sent by the server with the MIME type video/mp4", so it should be a supported video format on iOS.

[Dean Jackson]

Why isn't Radar picking this up?

[Dean Jackson]

dfarler, do you know why webkit-bug-importer is not importing?

[Ashley Gullen]

In addition to this, and somewhat related: currently playback controls still appear when the video element is not added to the DOM but instead passed to canvas2d's drawImage(). For the same reasons as this report it should work without forcing fullscreen controls to appear.

[Dean Jackson]

(In reply to comment #4)
> In addition to this, and somewhat related: currently playback controls still appear when the video element is not added to the DOM but instead passed to canvas2d's drawImage(). For the same reasons as this report it should work without forcing fullscreen controls to appear.

Can you explain this a bit more? You are seeing the controls render even if the video is not in the page? Where are the controls?

[Dean Jackson]

<rdar://problem/14754378>

[Alex Christensen]

It opens the video fullscreen as if it were playing an html5 video (making the page no longer visible), but the video is black.  The video controls work and the audio plays, but I can't see if the video is being drawn to texture because the canvas is covered.

[Radar WebKit Bug Importer]

<rdar://problem/17256389>

[Klaus Reinfeld]

On iOS a 'SecurityError: DOM Exception 18' will be thrown when trying to upload a video via texImage2d or when using it in a canvas drawImage().

In both cases the problem seems to be here in the 'wouldTaintOrigin' function:
https://trac.webkit.org/browser/trunk/Source/WebCore/html/canvas/CanvasRenderingContext.cpp#L70

It seems that either the 'video->hasSingleSecurityOrigin()' or the 'video->player()->didPassCORSAccessCheck()' check fails.

Please fix this, iOS is the only missing platform here!

[Klaus Reinfeld]

That limitation is unfortunately still there in iOS 8 beta 2 (Webkit/538.39.2, Safari/9537.53).

[Vincent Ogloblinsky]

Dean,

Any news about for this bug ?

Could you assign it ?

Klaus give us good informations for investing it.

Regards

[Alex Christensen]

The hasSingleSecurityOrigin  check fails.  The source of this problem is in MediaPlayerPrivateAVFoundationObjC.mm:

#if PLATFORM(IOS)
// FIXME: Implement for iOS in WebKit System Interface.
static inline NSURL *wkAVAssetResolvedURL(AVAsset*)
{
    return nil;
}
#endif

[Alex Christensen]

Created attachment 233445 [details]
Patch

[Jer Noble]

Comment on attachment 233445 [details]
Patch

r=me

[Klaus Reinfeld]

Thanks for the patches!
Great to see progress here!

In the meantime I have made an additional simple and very reduced example for testing this case here:

http://krpano.com/ios/bugs/ios8-webgl-video/

This example also shows the iPhone-only problem with the 'video fullscreen'.

[WebKit Commit Bot]

Comment on attachment 233445 [details]
Patch

Rejecting attachment 233445 [details] from commit-queue.

Failed to run "['/Volumes/Data/EWS/WebKit/Tools/Scripts/webkit-patch', '--status-host=webkit-queues.appspot.com', '--bot-id=webkit-cq-01', 'build', '--no-clean', '--no-update', '--build-style=release', '--port=mac']" exit_code: 2 cwd: /Volumes/Data/EWS/WebKit

Last 500 characters of output:
rce/WebKit/mac/WebView/WebTextCompletionController.mm -o /Volumes/Data/EWS/WebKit/WebKitBuild/WebKit.build/Release/WebKitLegacy.build/Objects-normal/x86_64/WebTextCompletionController.o

** BUILD FAILED **


The following build commands failed:
	CompileC /Volumes/Data/EWS/WebKit/WebKitBuild/WebKit.build/Release/WebKitLegacy.build/Objects-normal/x86_64/WebSystemInterface.o mac/WebCoreSupport/WebSystemInterface.mm normal x86_64 objective-c++ com.apple.compilers.llvm.clang.1_0.compiler
(1 failure)

Full output: http://webkit-queues.appspot.com/results/5835844803887104

[Eric Carlson]

(In reply to comment #15)
> Thanks for the patches!
> Great to see progress here!
> 
> In the meantime I have made an additional simple and very reduced example for testing this case here:
> 
> http://krpano.com/ios/bugs/ios8-webgl-video/
> 
> This example also shows the iPhone-only problem with the 'video fullscreen'.

Please file a separate bug for any additional problems you find.

[Alex Christensen]

This compiles fine.
http://trac.webkit.org/changeset/170295

[WebKit Commit Bot]

Re-opened since this is blocked by bug 134199

[Simon Fraser (smfr)]

Followup build fixes in r170299 and r170300. This did not "compile fine".

[Ashley Gullen]

Possibly related: video playback in canvas2d drawImage() is also buggy. See https://bugs.webkit.org/show_bug.cgi?id=134251

[Ashley Gullen]

This is marked "resolved fixed", but is only partially fixed in iOS 8 beta 3: the unwanted fullscreen playback controls still appear. They can however be exited and the cube displays correctly, albeit with poor performance on an iPhone 4S (looks like 3-4 FPS). As the original comment noted, the fullscreen playback controls are likely unsuitable for most use cases of playing video in WebGL. Also, surely the performance can be improved?

[Klaus Reinfeld]

That bug is only partially fixed, when using cross-domain (CORS) videos, this still fails.

Here the new bug report:
https://bugs.webkit.org/show_bug.cgi?id=135379

and here a test case:
http://krpano.com/ios/bugs/ios8-webgl-video-cors/

[MooLee]

(In reply to comment #23)
> That bug is only partially fixed, when using cross-domain (CORS) videos,
> this still fails.
> 
> Here the new bug report:
> https://bugs.webkit.org/show_bug.cgi?id=135379
> 
> and here a test case:
> http://krpano.com/ios/bugs/ios8-webgl-video-cors/



Hi~~How do you solve this problem.

[Richard C]

In iOS 9.0.2 (13A452) video textures works but video does not play using this demo.  I get the first frame but repeated attempts to play() after touch event will not trigger video playback.  Am I missing something new?

[Richard C]

(In reply to comment #25)
> In iOS 9.0.2 (13A452) video textures works but video does not play using
> this demo.  I get the first frame but repeated attempts to play() after
> touch event will not trigger video playback.  Am I missing something new?

OK. Caught again by the iOS click-to-play "feature".  Problem solved.