Bug 221820 – [GPUP] Some media tests related to canvas fail when media in GPU Process is enabled

• Component: Media
• Status: RESOLVED FIXED

[Peng Liu]

EVENT(canplaythrough)
 EVENT(seeked)
-Saw close enough to (194, 193, 14) at (40, 165) OK
-Saw close enough to (17, 197, 195) at (56, 165) OK
-Saw close enough to (20, 200, 18) at (73, 165) OK
+Expected (194, 193, 14) at (40, 165) but saw (0, 0, 0) FAIL
+Expected (17, 197, 195) at (56, 165) but saw (0, 0, 0) FAIL
+Expected (20, 200, 18) at (73, 165) but saw (0, 0, 0) FAIL
 EVENT(seeked)
-Saw close enough to (194, 193, 14) at (40, 165) OK
-Saw close enough to (17, 197, 195) at (56, 165) OK
-Saw close enough to (20, 200, 18) at (73, 165) OK
+Expected (194, 193, 14) at (40, 165) but saw (0, 0, 0) FAIL
+Expected (17, 197, 195) at (56, 165) but saw (0, 0, 0) FAIL
+Expected (20, 200, 18) at (73, 165) but saw (0, 0, 0) FAIL
 EVENT(seeked)
-Saw close enough to (194, 193, 14) at (40, 165) OK
-Saw close enough to (17, 197, 195) at (56, 165) OK
-Saw close enough to (20, 200, 18) at (73, 165) OK
+Expected (194, 193, 14) at (40, 165) but saw (0, 0, 0) FAIL
+Expected (17, 197, 195) at (56, 165) but saw (0, 0, 0) FAIL
+Expected (20, 200, 18) at (73, 165) but saw (0, 0, 0) FAIL
 EVENT(seeked)
-Saw close enough to (194, 193, 14) at (40, 165) OK
-Saw close enough to (17, 197, 195) at (56, 165) OK
-Saw close enough to (20, 200, 18) at (73, 165) OK
+Expected (194, 193, 14) at (40, 165) but saw (0, 0, 0) FAIL
+Expected (17, 197, 195) at (56, 165) but saw (0, 0, 0) FAIL
+Expected (20, 200, 18) at (73, 165) but saw (0, 0, 0) FAIL
 END OF TEST

[Radar WebKit Bug Importer]

<rdar://problem/74525909>

[Peng Liu]

media/video-canvas-drawing-output.html [ Failure ]
media/video-canvas-createPattern.html [ Crash ]
media/video-orientation-canvas.html [ Failure ]

[Peng Liu]

The patch for bug 222461 fixes following two failures:

media/video-canvas-drawing-output.html [ Failure ]
media/video-canvas-createPattern.html [ Crash ]

[Peng Liu]

Created attachment 422330 [details]
Patch

[Eric Carlson]

Comment on attachment 422330 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=422330&action=review

> Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:2053
> +    m_imageRotationSession = makeUnique<ImageRotationSessionVT>(WTFMove(finalTransform), naturalSize, kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, ImageRotationSessionVT::IsCGImageCompatible::Yes);

You should check to make sure this is the preferred pixel format on both iOS and macOS. 

We found that at least for capture and compression, kCVPixelFormatType_420YpCbCr8BiPlanarFullRange is more efficient on iOS so we use the function `preferedPixelBufferFormat()` from RealtimeVideoUtilities.h

[Peng Liu]

Comment on attachment 422330 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=422330&action=review

>> Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:2053
>> +    m_imageRotationSession = makeUnique<ImageRotationSessionVT>(WTFMove(finalTransform), naturalSize, kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, ImageRotationSessionVT::IsCGImageCompatible::Yes);
> 
> You should check to make sure this is the preferred pixel format on both iOS and macOS. 
> 
> We found that at least for capture and compression, kCVPixelFormatType_420YpCbCr8BiPlanarFullRange is more efficient on iOS so we use the function `preferedPixelBufferFormat()` from RealtimeVideoUtilities.h

Good point!

However, in this case, the ImageRotationSessionVT's pixel format (essentially its PixelBufferPool's pixel format) needs to match the pixel buffers need to be rotated. Otherwise, the rotation will fail.

Based on some discussions with Jer, I am going to implement a mechanism to configure ImageRotationSessionVT's pixel format based on the pixel buffers from AVPlayerItemVideoOutput to ensure the rotation will always succeed.

[Peng Liu]

Created attachment 422511 [details]
Revise the patch based on Jer's suggestions

[Peng Liu]

(In reply to Peng Liu from comment #7)
> Created attachment 422511 [details]
> Revise the patch based on Jer's suggestions

Thanks for Darin's review.

This patch needs another review because of some nontrivial changes.

[EWS]

Committed r274076: <https://commits.webkit.org/r274076>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 422511 [details].