Bug 221820 - [GPUP] Some media tests related to canvas fail when media in GPU Process is enabled
Summary: [GPUP] Some media tests related to canvas fail when media in GPU Process is e...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Media (show other bugs)
Version: Other
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Peng Liu
URL:
Keywords: InRadar
Depends on:
Blocks: 221830
  Show dependency treegraph
 
Reported: 2021-02-12 09:43 PST by Peng Liu
Modified: 2021-04-01 12:09 PDT (History)
10 users (show)

See Also:


Attachments
Patch (5.13 KB, patch)
2021-03-04 20:35 PST, Peng Liu
darin: review+
ews-feeder: commit-queue-
Details | Formatted Diff | Diff
Revise the patch based on Jer's suggestions (16.66 KB, patch)
2021-03-06 16:03 PST, Peng Liu
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Peng Liu 2021-02-12 09:43:49 PST
EVENT(canplaythrough)
 EVENT(seeked)
-Saw close enough to (194, 193, 14) at (40, 165) OK
-Saw close enough to (17, 197, 195) at (56, 165) OK
-Saw close enough to (20, 200, 18) at (73, 165) OK
+Expected (194, 193, 14) at (40, 165) but saw (0, 0, 0) FAIL
+Expected (17, 197, 195) at (56, 165) but saw (0, 0, 0) FAIL
+Expected (20, 200, 18) at (73, 165) but saw (0, 0, 0) FAIL
 EVENT(seeked)
-Saw close enough to (194, 193, 14) at (40, 165) OK
-Saw close enough to (17, 197, 195) at (56, 165) OK
-Saw close enough to (20, 200, 18) at (73, 165) OK
+Expected (194, 193, 14) at (40, 165) but saw (0, 0, 0) FAIL
+Expected (17, 197, 195) at (56, 165) but saw (0, 0, 0) FAIL
+Expected (20, 200, 18) at (73, 165) but saw (0, 0, 0) FAIL
 EVENT(seeked)
-Saw close enough to (194, 193, 14) at (40, 165) OK
-Saw close enough to (17, 197, 195) at (56, 165) OK
-Saw close enough to (20, 200, 18) at (73, 165) OK
+Expected (194, 193, 14) at (40, 165) but saw (0, 0, 0) FAIL
+Expected (17, 197, 195) at (56, 165) but saw (0, 0, 0) FAIL
+Expected (20, 200, 18) at (73, 165) but saw (0, 0, 0) FAIL
 EVENT(seeked)
-Saw close enough to (194, 193, 14) at (40, 165) OK
-Saw close enough to (17, 197, 195) at (56, 165) OK
-Saw close enough to (20, 200, 18) at (73, 165) OK
+Expected (194, 193, 14) at (40, 165) but saw (0, 0, 0) FAIL
+Expected (17, 197, 195) at (56, 165) but saw (0, 0, 0) FAIL
+Expected (20, 200, 18) at (73, 165) but saw (0, 0, 0) FAIL
 END OF TEST
Comment 1 Radar WebKit Bug Importer 2021-02-19 09:44:13 PST
<rdar://problem/74525909>
Comment 2 Peng Liu 2021-02-25 16:49:45 PST
media/video-canvas-drawing-output.html [ Failure ]
media/video-canvas-createPattern.html [ Crash ]
media/video-orientation-canvas.html [ Failure ]
Comment 3 Peng Liu 2021-02-26 15:18:38 PST
The patch for bug 222461 fixes following two failures:

media/video-canvas-drawing-output.html [ Failure ]
media/video-canvas-createPattern.html [ Crash ]
Comment 4 Peng Liu 2021-03-04 20:35:10 PST
Created attachment 422330 [details]
Patch
Comment 5 Eric Carlson 2021-03-05 16:45:09 PST
Comment on attachment 422330 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=422330&action=review

> Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:2053
> +    m_imageRotationSession = makeUnique<ImageRotationSessionVT>(WTFMove(finalTransform), naturalSize, kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, ImageRotationSessionVT::IsCGImageCompatible::Yes);

You should check to make sure this is the preferred pixel format on both iOS and macOS. 

We found that at least for capture and compression, kCVPixelFormatType_420YpCbCr8BiPlanarFullRange is more efficient on iOS so we use the function `preferedPixelBufferFormat()` from RealtimeVideoUtilities.h
Comment 6 Peng Liu 2021-03-05 17:43:14 PST
Comment on attachment 422330 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=422330&action=review

>> Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:2053
>> +    m_imageRotationSession = makeUnique<ImageRotationSessionVT>(WTFMove(finalTransform), naturalSize, kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, ImageRotationSessionVT::IsCGImageCompatible::Yes);
> 
> You should check to make sure this is the preferred pixel format on both iOS and macOS. 
> 
> We found that at least for capture and compression, kCVPixelFormatType_420YpCbCr8BiPlanarFullRange is more efficient on iOS so we use the function `preferedPixelBufferFormat()` from RealtimeVideoUtilities.h

Good point!

However, in this case, the ImageRotationSessionVT's pixel format (essentially its PixelBufferPool's pixel format) needs to match the pixel buffers need to be rotated. Otherwise, the rotation will fail.

Based on some discussions with Jer, I am going to implement a mechanism to configure ImageRotationSessionVT's pixel format based on the pixel buffers from AVPlayerItemVideoOutput to ensure the rotation will always succeed.
Comment 7 Peng Liu 2021-03-06 16:03:12 PST
Created attachment 422511 [details]
Revise the patch based on Jer's suggestions
Comment 8 Peng Liu 2021-03-06 16:09:13 PST
(In reply to Peng Liu from comment #7)
> Created attachment 422511 [details]
> Revise the patch based on Jer's suggestions

Thanks for Darin's review.

This patch needs another review because of some nontrivial changes.
Comment 9 EWS 2021-03-08 09:51:36 PST
Committed r274076: <https://commits.webkit.org/r274076>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 422511 [details].