Bug 205734 – REGRESSION: [ Mac ] fast/canvas/webgl/texImage2D-video-flipY-false.html is Timing out

• Status: RESOLVED
• Resolution: FIXED
• Component: WebGL

[Truitt Savell]

fast/canvas/webgl/texImage2D-video-flipY-false.html

Description:
This test began timing out flakily after https://trac.webkit.org/changeset/253926/webkit

History:
https://results.webkit.org/?suite=layout-tests&test=fast%2Fcanvas%2Fwebgl%2FtexImage2D-video-flipY-false.html

[Radar WebKit Bug Importer]

<rdar://problem/58302063>

[Alexey Proskuryakov]

rdar://problem/58228020

[Kenneth Russell]

I'll investigate.

[Kenneth Russell]

Presumably this started timing out again after the recent switch to ANGLE in bug 205483.

[Kenneth Russell]

Something's going wrong with the handling of LayoutTests/fast/canvas/webgl/resources/orientation-flipped.mp4 . Playback of that video when ANGLE is the WebGL backend is broken; the frame counter in the video never advances past 0, and there are strange green glitches on the right-hand side of the video. timeupdate events aren't fired past currentTime=3, and the browser isn't responsive - the only way to switch URLs is to close the current window.

This doesn't happen with resources/orientation-normal.mp4.

Continuing to investigate what code paths are taken when this video is displayed.

Note that neither Chrome nor Firefox will load resources/orientation-flipped.mp4 - both browsers' media stacks report that the media resource was not suitable.

[Kenneth Russell]

Here's ffmpeg's information for the two videos:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'orientation-normal.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    creation_time   : 2017-08-28T20:46:43.000000Z
    encoder         : HandBrake 1.0.7 2017040900
  Duration: 00:00:10.01, start: 0.000000, bitrate: 110 kb/s
    Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, smpte170m/smpte170m/bt709), 300x300 [SAR 1:1 DAR 1:1], 106 kb/s, 29.97 fps, 29.97 tbr, 90k tbn, 180k tbc (default)
    Metadata:
      creation_time   : 2017-08-28T20:46:43.000000Z
      handler_name    : VideoHandler


Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'orientation-flipped.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: mp42mp41
    creation_time   : 2017-09-05T23:25:39.000000Z
  Duration: 00:00:10.01, start: 0.000000, bitrate: 152 kb/s
    Stream #0:0(eng): Video: mpeg4 (Simple Profile) (mp4v / 0x7634706D), yuv420p(pc, smpte170m/unknown/smpte170m), 300x300 [SAR 1:1 DAR 1:1], 149 kb/s, 29.97 fps, 29.97 tbr, 1k tbn, 1k tbc (default)
    Metadata:
      creation_time   : 2017-09-05T23:25:39.000000Z
      handler_name    : Apple Video Media Handler

[Kenneth Russell]

It looks like the video's currentTime never advances past 3 when the test fails. Still debugging what is going on.

One interesting thing to note is that this test takes the software uploading path:

    frame #0: 0x000000010ff55697 WebCore`WebCore::VideoTextureCopierCV::copyImageToPlatformTexture(this=0x000000013ad9d4c8, image=0x000000012ffe1650, width=300, height=300, outputTexture=2, outputTarget=3553, level=0, internalFormat=6408, format=6408, type=5121, premultiplyAlpha=false, flipY=false) at VideoTextureCopierCV.cpp:810:9
   807 	    OSType pixelFormat = CVPixelBufferGetPixelFormatType(image);
   808 	    if (pixelFormat != kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange && pixelFormat != kCVPixelFormatType_420YpCbCr8BiPlanarFullRange) {
   809 	        LOG(WebGL, "VideoTextureCopierCV::copyVideoTextureToPlatformTexture(%p) - Asked to copy an unsupported pixel format ('%s').", this, FourCC(pixelFormat).toString().utf8().data());
-> 810 	        return false;
   811 	    }
   812 	
   813 	    IOSurfaceRef surface = CVPixelBufferGetIOSurface(image);
Target 0: (MiniBrowser) stopped.
(lldb) p pixelFormat
(OSType) $3 = '2vuy' '2vuy'

[Kenneth Russell]

When the test times out, the video is staying in the "seeking" state; HTMLMediaElement::m_seeking is repeatedly true. The test case seems to be able to set the video's currentTime multiple times, but changing it to avoid that doesn't address the problem. Couple of things to investigate:

1) Whether the use of ANGLE is somehow causing the video to be decoded in a different format.

2) Why the video and/or MediaPlayer doesn't successfully seek when ANGLE is enabled.

[Kenneth Russell]

When ANGLE is disabled for this test case, WebKit goes down the TextureCacheCV code path in VideoTextureCopierCV::copyImageToPlatformTexture, so it retains hardware-accelerated video-to-texture copies. (I thought that CVOpenGLTextureCache was deprecated - in my earlier experiments it looked like CVOpenGLTextureCacheCreateTextureFromImage was failing in WebKit's ANGLE build.)

Will see whether this is the root cause of the test timeout or whether there are more issues.

[Kenneth Russell]

The bugs are in the software fallback path for uploading video elements to WebGL textures. This code path is taken because when ANGLE is enabled, TextureCacheCV (which uses CVOpenGLTextureCache / CVOpenGLESTextureCache) is not used; only the IOSurface code path (for GPU-accelerated video-to-texture uploads, as implemented in Bug 200904) and the MediaPlayer::paintCurrentFrameInContext (the software fallback) are implemented.

The basic bug is that when MediaPlayerPrivateAVFoundationObjC::seekToTime calls [AVPlayerItem seekToTime:toleranceBefore:toleranceAfter:completionHandler:], the completion handler is never called.

The reason for this is an apparent bug in AVFoundation. Fetching a CoreGraphics image from the video is done in MediaPlayerPrivateAVFoundationObjC::createImageForTimeInRect using the AVAssetImageGenerator. Calling [AVAssetImageGenerator copyCGImageAtTime:actualTime:error:] causes the completion handler from [AVPlayerItem seekToTime:] to get squelched somehow, and the seek never completes. This is why the test times out with the software video upload path - the seek to 3 seconds never finishes.

There's another bug where the software path always displays only the first frame of the video. This happens because MediaPlayerPrivateAVFoundationObjC must override and implement:
   float currentTime() const;

Still trying to figure out whether there's any workaround for the above AVFoundation issue.

[Kenneth Russell]

Tried changing MediaPlayerPrivateAVFoundationObjC::createImageGenerator to stop setting requestedTimeToleranceBefore and requestedTimeToleranceAfter, on the AVAssetImageGenerator, instead using the default values. Thought this might prevent it from interfering with the AVPlayerItem's seeking (though I see that all AVAssetImageGenerator receives is the AVURLAsset) - unfortunately the problem still happens.

jer.noble@ - do you have any ideas or suggestions about this AVFoundation problem? Thanks in advance.

[Jer Noble]

We should avoid using AVAssetImageGenerator whenever possible, preferring to use AVPlayerItemVideoOutput. The reason is twofold: 1) AVAssetImageGenerator will fetch the video again, separately, effectively doubling network cost and 2) it will redecode the resource, effectively doubling CPU cost.

If the asset generator is running, it's probably because the AVPlayerItemVideoOutput doesn't never receive a decoded frame, which indicates a deeper problem with the media stream.

I don't remember how I generated the flipped video; it could just be that the stream is corrupt or malformed in a way that used to work in AVFoundation and stopped. One solution to this test may be to generate a "correct" flipped video that decodes in all the major browsers. IIRC, it was a matter of setting the video transform inside the 'mvhd' box.

[Kenneth Russell]

Created attachment 392532 [details]
Patch

[Kenneth Russell]

dino, jer.noble: could you please review? It makes this layout test run reliably on my machine. Ran all four of the texImage2D-video-flipY-* and texImage2D-mse-flipY-* tests repeatedly; all pass reliably.

Thanks.

[Kenneth Russell]

FYI, the win layout test failures are unrelated to this patch. Do I need to do anything in response to them?

[Dean Jackson]

Comment on attachment 392532 [details]
Patch

Thanks Ken!

[WebKit Commit Bot]

The commit-queue encountered the following flaky tests while processing attachment 392532 [details]:

editing/spelling/spellcheck-async-remove-frame.html bug 158401 (authors: [e-mail], [e-mail], and [e-mail])
The commit-queue is continuing to process your patch.

[WebKit Commit Bot]

Comment on attachment 392532 [details]
Patch

Clearing flags on attachment: 392532

Committed r257932: <https://trac.webkit.org/changeset/257932>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Kenneth Russell]

*** Bug 207857 has been marked as a duplicate of this bug. ***