Attachment #422511 for bug #221820

View Details Raw Unified Return to Bug 221820

| Differences between



2021-03-06  Peng Liu  <peng.liu6@apple.com>

        [GPUP] Some media tests related to canvas fail when media in GPU Process is enabled
        https://bugs.webkit.org/show_bug.cgi?id=221820

        Reviewed by NOBODY (OOPS!).

        When "GPU Process: Media" is enabled and "GPU Process: Canvas Rendering" is disabled,
        `MediaPlayerPrivateAVFoundationObjC::nativeImageForCurrentTime()` will be called
        to get native images (video frames) from a video player. This function will copy
        pixel buffers from CoreVideo, and we may need to rotate the pixel buffers before
        converting them to native images. The process to copy pixel buffers from CoreVideo
        and rotate them is implemented in `MediaPlayerPrivateAVFoundationObjC::updateLastPixelBuffer()`.
        It rotates pixel buffers with an `ImageRotationSessionVT` object (`m_imageRotationSession`).

        With the current implementation of `ImageRotationSessionVT`, we can rotate pixel buffers
        with following two approaches:
        - Construct an uninitialized instance, and call `ImageRotationSessionVT::rotate(
        MediaSample&, const RotationProperties&, IsCGImageCompatible)`. The `rotate` function
        will initialize the object if needed.
        - Construct an instance with given rotation properties, size, pixel buffer format,
        and CGImage compatibility, then call `ImageRotationSessionVT::rotate(CVPixelBufferRef)`.
        When we call the `rotate` function, we need to make sure the pixel buffer's format
        is consistent with the `ImageRotationSessionVT` instance. Otherwise, the rotation will
        fail and the caller will get an empty pixel buffer (this bug).

        In order to use the second approach in `MediaPlayerPrivateAVFoundationObjC`, we need to
        figure out the pixel buffer format to construct an ImageRotationSessionVT instance.
        But that can only be done after we get a pixel buffer from CoreVideo.

        This patch refactors `ImageRotationSessionVT` to better support the aforementioned
        second approach. `MediaPlayerPrivateAVFoundationObjC` constructs an `ImageRotationSessionVT`
        instance without pixel buffer format, and the `ImageRotationSessionVT` instance
        will figure out the pixel buffer format by itself and configures its `CVPixelBufferPool`
        properly when we call `rotate` function.

        No new tests. Fix a test failure:
        - media/video-orientation-canvas.html

        * platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.mm:
        (WebCore::ImageDecoderAVFObjC::readTrackMetadata):
        * platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:
        (WebCore::MediaPlayerPrivateAVFoundationObjC::updateRotationSession):
        (WebCore::MediaPlayerPrivateAVFoundationObjC::nativeImageForCurrentTime):
        We have to update the image synchronously to ensure that the caller gets the correct image.

        * platform/graphics/cv/ImageRotationSessionVT.h:
        Remove isMatching() because it is not defined/used.
        * platform/graphics/cv/ImageRotationSessionVT.mm:
        (WebCore::ImageRotationSessionVT::ImageRotationSessionVT):
        (WebCore::ImageRotationSessionVT::initialize):
        (WebCore::ImageRotationSessionVT::rotate):

        * platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.mm:
        (WebCore::RealtimeOutgoingVideoSourceCocoa::rotatePixelBuffer):

2021-03-05  Ryosuke Niwa  <rniwa@webkit.org>

        Inline isDisabledFormControl() in isDisabledOrReadOnly() by making it final rather than duplicating code

Lines 390-396 void ImageDecoderAVFObjC::readTrackMetadata() a/Source/WebCore/platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.mm_sec1
- a/Source/WebCore/platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.mm -1 / +1 lines
390	\|\| !m_imageRotationSession->transform()	390	\|\| !m_imageRotationSession->transform()
391	\|\| m_imageRotationSession->transform().value() != finalTransform	391	\|\| m_imageRotationSession->transform().value() != finalTransform
392	\|\| m_imageRotationSession->size() != size)	392	\|\| m_imageRotationSession->size() != size)
393	m_imageRotationSession = makeUnique<ImageRotationSessionVT>(WTFMove(finalTransform), size, kCVPixelFormatType_32BGRA, ImageRotationSessionVT::IsCGImageCompatible::Yes);	393	m_imageRotationSession = makeUnique<ImageRotationSessionVT>(WTFMove(finalTransform), size, ImageRotationSessionVT::IsCGImageCompatible::Yes);
394		394
395	m_size = expandedIntSize(m_imageRotationSession->rotatedSize());	395	m_size = expandedIntSize(m_imageRotationSession->rotatedSize());
396	}	396	}



        && m_imageRotationSession->size() == naturalSize)
        return;

    m_imageRotationSession = makeUnique<ImageRotationSessionVT>(WTFMove(finalTransform), naturalSize, ImageRotationSessionVT::IsCGImageCompatible::Yes);
}

template <typename RefT, typename PassRefT>


RefPtr<NativeImage> MediaPlayerPrivateAVFoundationObjC::nativeImageForCurrentTime()
{
    updateLastImage(UpdateType::UpdateSynchronously);
    return m_lastImage;
}




        Yes,
    };

    ImageRotationSessionVT(AffineTransform&&, FloatSize, IsCGImageCompatible);
    ImageRotationSessionVT(const RotationProperties&, FloatSize, IsCGImageCompatible);
    ImageRotationSessionVT() = default;

    const Optional<AffineTransform>& transform() const { return m_transform; }

    RetainPtr<CVPixelBufferRef> rotate(CVPixelBufferRef);
    WEBCORE_EXPORT RetainPtr<CVPixelBufferRef> rotate(MediaSample&, const RotationProperties&, IsCGImageCompatible);

    bool isMatching(MediaSample&, const RotationProperties&);

private:
    void initialize(const RotationProperties&, FloatSize, IsCGImageCompatible);

    Optional<AffineTransform> m_transform;
    RotationProperties m_rotationProperties;
    FloatSize m_size;
    Optional<AffineTransform> m_transform;
    OSType m_pixelFormat;
    IsCGImageCompatible m_isCGImageCompatible;
    FloatSize m_rotatedSize;
    RetainPtr<VTImageRotationSessionRef> m_rotationSession;
    RetainPtr<CVPixelBufferPoolRef> m_rotationPool;



    return rotation;
}

ImageRotationSessionVT::ImageRotationSessionVT(AffineTransform&& transform, FloatSize size, IsCGImageCompatible isCGImageCompatible)
    : ImageRotationSessionVT(transformToRotationProperties(transform), size, isCGImageCompatible)
{
    m_transform = WTFMove(transform);
}

ImageRotationSessionVT::ImageRotationSessionVT(const RotationProperties& rotation, FloatSize size, IsCGImageCompatible isCGImageCompatible)
{
    initialize(rotation, size, isCGImageCompatible);
}

void ImageRotationSessionVT::initialize(const RotationProperties& rotation, FloatSize size, IsCGImageCompatible isCGImageCompatible)
{
    m_rotationProperties = rotation;
    m_size = size;
    m_isCGImageCompatible = isCGImageCompatible;

    if (m_rotationProperties.angle == 90 || m_rotationProperties.angle == 270)
        size = size.transposedSize();

        VTImageRotationSessionSetProperty(m_rotationSession.get(), kVTImageRotationPropertyKey_FlipVerticalOrientation, kCFBooleanTrue);
    if (m_rotationProperties.flipX)
        VTImageRotationSessionSetProperty(m_rotationSession.get(), kVTImageRotationPropertyKey_FlipHorizontalOrientation, kCFBooleanTrue);

    auto pixelAttributes = @{
        (__bridge NSString *)kCVPixelBufferWidthKey: @(m_rotatedSize.width()),
        (__bridge NSString *)kCVPixelBufferHeightKey: @(m_rotatedSize.height()),
        (__bridge NSString *)kCVPixelBufferPixelFormatTypeKey: @(pixelFormat),
        (__bridge NSString *)kCVPixelBufferCGImageCompatibilityKey: (cvImageCompatibility == IsCGImageCompatible::Yes ? @YES : @NO),
#if PLATFORM(IOS_SIMULATOR) || PLATFORM(MAC)
        (__bridge NSString *)kCVPixelBufferIOSurfacePropertiesKey : @{ }
#endif
    };
    CVPixelBufferPoolRef rawPool = nullptr;
    if (auto err = CVPixelBufferPoolCreate(kCFAllocatorDefault, nullptr, (__bridge CFDictionaryRef)pixelAttributes, &rawPool); err != noErr)
        RELEASE_LOG_ERROR(WebRTC, "ImageRotationSessionVT failed creating buffer pool with error %d", err);
    m_rotationPool = adoptCF(rawPool);
}

RetainPtr<CVPixelBufferRef> ImageRotationSessionVT::rotate(CVPixelBufferRef pixelBuffer)
{
    auto pixelFormat = CVPixelBufferGetPixelFormatType(pixelBuffer);
    if (pixelFormat != m_pixelFormat || !m_rotationPool) {
        m_pixelFormat = pixelFormat;
        auto pixelAttributes = @{
            (__bridge NSString *)kCVPixelBufferWidthKey: @(m_rotatedSize.width()),
            (__bridge NSString *)kCVPixelBufferHeightKey: @(m_rotatedSize.height()),
            (__bridge NSString *)kCVPixelBufferPixelFormatTypeKey: @(m_pixelFormat),
            (__bridge NSString *)kCVPixelBufferCGImageCompatibilityKey: (m_isCGImageCompatible == IsCGImageCompatible::Yes ? @YES : @NO),
#if PLATFORM(IOS_SIMULATOR) || PLATFORM(MAC)
            (__bridge NSString *)kCVPixelBufferIOSurfacePropertiesKey : @{ }
#endif
        };

        CVPixelBufferPoolRef rawPool = nullptr;
        if (auto err = CVPixelBufferPoolCreate(kCFAllocatorDefault, nullptr, (__bridge CFDictionaryRef)pixelAttributes, &rawPool); err != noErr) {
            RELEASE_LOG_ERROR(WebRTC, "ImageRotationSessionVT failed creating buffer pool with error %d", err);
            return nullptr;
        }

        m_rotationPool = adoptCF(rawPool);
    }

    RetainPtr<CVPixelBufferRef> result;
    CVPixelBufferRef rawRotatedBuffer = nullptr;
    auto status = CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, m_rotationPool.get(), &rawRotatedBuffer);
    if (status != kCVReturnSuccess) {

    return result;
}

RetainPtr<CVPixelBufferRef> ImageRotationSessionVT::rotate(MediaSample& sample, const RotationProperties& rotation, IsCGImageCompatible cgImageCompatible)
{
    auto pixelBuffer = static_cast<CVPixelBufferRef>(PAL::CMSampleBufferGetImageBuffer(sample.platformSample().sample.cmSampleBuffer));
    ASSERT(pixelBuffer);
    if (!pixelBuffer)
        return nullptr;

    m_pixelFormat = CVPixelBufferGetPixelFormatType(pixelBuffer);
    IntSize size { (int)CVPixelBufferGetWidth(pixelBuffer), (int)CVPixelBufferGetHeight(pixelBuffer) };
    if (rotation != m_rotationProperties || m_size != size)
        initialize(rotation, size, cgImageCompatible);

    return rotate(pixelBuffer);
}

Lines 75-81 RetainPtr<CVPixelBufferRef> RealtimeOutgoingVideoSourceCocoa::rotatePixelBuffer( a/Source/WebCore/platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.mm_sec1
- a/Source/WebCore/platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.mm -1 / +1 lines
75	IntSize size = { (int)CVPixelBufferGetWidth(pixelBuffer) , (int)CVPixelBufferGetHeight(pixelBuffer) };	75	IntSize size = { (int)CVPixelBufferGetWidth(pixelBuffer) , (int)CVPixelBufferGetHeight(pixelBuffer) };
76	AffineTransform transform;	76	AffineTransform transform;
77	transform.rotate(rotationToAngle(rotation));	77	transform.rotate(rotationToAngle(rotation));
78	m_rotationSession = makeUnique<ImageRotationSessionVT>(WTFMove(transform), size, CVPixelBufferGetPixelFormatType(pixelBuffer), ImageRotationSessionVT::IsCGImageCompatible::No);	78	m_rotationSession = makeUnique<ImageRotationSessionVT>(WTFMove(transform), size, ImageRotationSessionVT::IsCGImageCompatible::No);
79	}	79	}
80		80
81	return m_rotationSession->rotate(pixelBuffer);	81	return m_rotationSession->rotate(pixelBuffer);



2021-03-06  Peng Liu  <peng.liu6@apple.com>

        [GPUP] Some media tests related to canvas fail when media in GPU Process is enabled
        https://bugs.webkit.org/show_bug.cgi?id=221820

        Reviewed by NOBODY (OOPS!).

        * media/video-orientation-canvas.html:
        * platform/wk2/TestExpectations:

2021-03-05  Ryosuke Niwa  <rniwa@webkit.org>

        Add leak tests for Range and StaticRange

Lines 65-71 a/LayoutTests/media/video-orientation-canvas.html_sec1
- a/LayoutTests/media/video-orientation-canvas.html -1 lines
65	context.drawImage(video3, 0, 0);	65	context.drawImage(video3, 0, 0);
66	arraysAreApproximatelyEqual('context.getImageData(200, 237, 1, 1).data', redColor, 5);	66	arraysAreApproximatelyEqual('context.getImageData(200, 237, 1, 1).data', redColor, 5);
67		67
68
69	endTest();	68	endTest();
70	});	69	});
71	</script>	70	</script>




webkit.org/b/221819 imported/w3c/web-platform-tests/webaudio/the-audio-api/the-mediaelementaudiosourcenode-interface/mediaElementAudioSourceToScriptProcessorTest.html [ Failure ]

# webkit.org/b/221820
media/video-orientation-canvas.html [ Failure ]

webkit.org/b/221783 [ Debug ] loader/change-src-during-iframe-load-crash.html [ Skip ]

# webkit.org/b/220375

Return to Bug 221820

Lines 2050-2056 void MediaPlayerPrivateAVFoundationObjC::updateRotationSession() a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm_sec1
- a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm -2 / +2 lines
2050	&& m_imageRotationSession->size() == naturalSize)	2050	&& m_imageRotationSession->size() == naturalSize)
2051	return;	2051	return;
2052		2052
2053	m_imageRotationSession = makeUnique<ImageRotationSessionVT>(WTFMove(finalTransform), naturalSize, kCVPixelFormatType_32BGRA, ImageRotationSessionVT::IsCGImageCompatible::Yes);	2053	m_imageRotationSession = makeUnique<ImageRotationSessionVT>(WTFMove(finalTransform), naturalSize, ImageRotationSessionVT::IsCGImageCompatible::Yes);
2054	}	2054	}
2055		2055
2056	template <typename RefT, typename PassRefT>	2056	template <typename RefT, typename PassRefT>
Lines 2402-2408 RetainPtr<CVPixelBufferRef> MediaPlayerPrivateAVFoundationObjC::pixelBufferForCu a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm_sec2
2402		2402
2403	RefPtr<NativeImage> MediaPlayerPrivateAVFoundationObjC::nativeImageForCurrentTime()	2403	RefPtr<NativeImage> MediaPlayerPrivateAVFoundationObjC::nativeImageForCurrentTime()
2404	{	2404	{
2405	updateLastImage();	2405	updateLastImage(UpdateType::UpdateSynchronously);
2406	return m_lastImage;	2406	return m_lastImage;
2407	}	2407	}
2408		2408

Lines 257-265 webkit.org/b/221817 imported/w3c/web-platform-tests/fetch/range/general.window.h a/LayoutTests/platform/wk2/TestExpectations_sec1
- a/LayoutTests/platform/wk2/TestExpectations -3 lines
257		257
258	webkit.org/b/221819 imported/w3c/web-platform-tests/webaudio/the-audio-api/the-mediaelementaudiosourcenode-interface/mediaElementAudioSourceToScriptProcessorTest.html [ Failure ]	258	webkit.org/b/221819 imported/w3c/web-platform-tests/webaudio/the-audio-api/the-mediaelementaudiosourcenode-interface/mediaElementAudioSourceToScriptProcessorTest.html [ Failure ]
259		259
260	# webkit.org/b/221820
261	media/video-orientation-canvas.html [ Failure ]
262
263	webkit.org/b/221783 [ Debug ] loader/change-src-during-iframe-load-crash.html [ Skip ]	260	webkit.org/b/221783 [ Debug ] loader/change-src-during-iframe-load-crash.html [ Skip ]
264		261
265	# webkit.org/b/220375	262	# webkit.org/b/220375

Lines 1-3 a/Source/WebCore/ChangeLog_sec1
- a/Source/WebCore/ChangeLog +56 lines
		1	2021-03-06 Peng Liu <peng.liu6@apple.com>
		2
		3	[GPUP] Some media tests related to canvas fail when media in GPU Process is enabled
		4	https://bugs.webkit.org/show_bug.cgi?id=221820
		5
		6	Reviewed by NOBODY (OOPS!).
		7
		8	When "GPU Process: Media" is enabled and "GPU Process: Canvas Rendering" is disabled,
		9	`MediaPlayerPrivateAVFoundationObjC::nativeImageForCurrentTime()` will be called
		10	to get native images (video frames) from a video player. This function will copy
		11	pixel buffers from CoreVideo, and we may need to rotate the pixel buffers before
		12	converting them to native images. The process to copy pixel buffers from CoreVideo
		13	and rotate them is implemented in `MediaPlayerPrivateAVFoundationObjC::updateLastPixelBuffer()`.
		14	It rotates pixel buffers with an `ImageRotationSessionVT` object (`m_imageRotationSession`).
		15
		16	With the current implementation of `ImageRotationSessionVT`, we can rotate pixel buffers
		17	with following two approaches:
		18	- Construct an uninitialized instance, and call `ImageRotationSessionVT::rotate(
		19	MediaSample&, const RotationProperties&, IsCGImageCompatible)`. The `rotate` function
		20	will initialize the object if needed.
		21	- Construct an instance with given rotation properties, size, pixel buffer format,
		22	and CGImage compatibility, then call `ImageRotationSessionVT::rotate(CVPixelBufferRef)`.
		23	When we call the `rotate` function, we need to make sure the pixel buffer's format
		24	is consistent with the `ImageRotationSessionVT` instance. Otherwise, the rotation will
		25	fail and the caller will get an empty pixel buffer (this bug).
		26
		27	In order to use the second approach in `MediaPlayerPrivateAVFoundationObjC`, we need to
		28	figure out the pixel buffer format to construct an ImageRotationSessionVT instance.
		29	But that can only be done after we get a pixel buffer from CoreVideo.
		30
		31	This patch refactors `ImageRotationSessionVT` to better support the aforementioned
		32	second approach. `MediaPlayerPrivateAVFoundationObjC` constructs an `ImageRotationSessionVT`
		33	instance without pixel buffer format, and the `ImageRotationSessionVT` instance
		34	will figure out the pixel buffer format by itself and configures its `CVPixelBufferPool`
		35	properly when we call `rotate` function.
		36
		37	No new tests. Fix a test failure:
		38	- media/video-orientation-canvas.html
		39
		40	* platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.mm:
		41	(WebCore::ImageDecoderAVFObjC::readTrackMetadata):
		42	* platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:
		43	(WebCore::MediaPlayerPrivateAVFoundationObjC::updateRotationSession):
		44	(WebCore::MediaPlayerPrivateAVFoundationObjC::nativeImageForCurrentTime):
		45	We have to update the image synchronously to ensure that the caller gets the correct image.
		46
		47	* platform/graphics/cv/ImageRotationSessionVT.h:
		48	Remove isMatching() because it is not defined/used.
		49	* platform/graphics/cv/ImageRotationSessionVT.mm:
		50	(WebCore::ImageRotationSessionVT::ImageRotationSessionVT):
		51	(WebCore::ImageRotationSessionVT::initialize):
		52	(WebCore::ImageRotationSessionVT::rotate):
		53
		54	* platform/mediastream/mac/RealtimeOutgoingVideoSourceCocoa.mm:
		55	(WebCore::RealtimeOutgoingVideoSourceCocoa::rotatePixelBuffer):
		56
1	2021-03-05 Ryosuke Niwa <rniwa@webkit.org>	57	2021-03-05 Ryosuke Niwa <rniwa@webkit.org>
2		58
3	Inline isDisabledFormControl() in isDisabledOrReadOnly() by making it final rather than duplicating code	59	Inline isDisabledFormControl() in isDisabledOrReadOnly() by making it final rather than duplicating code

Lines 52-59 public: a/Source/WebCore/platform/graphics/cv/ImageRotationSessionVT.h_sec1
- a/Source/WebCore/platform/graphics/cv/ImageRotationSessionVT.h -6 / +6 lines
52	Yes,	52	Yes,
53	};	53	};
54		54
55	ImageRotationSessionVT(AffineTransform&&, FloatSize, OSType, IsCGImageCompatible);	55	ImageRotationSessionVT(AffineTransform&&, FloatSize, IsCGImageCompatible);
56	ImageRotationSessionVT(const RotationProperties&, FloatSize, OSType, IsCGImageCompatible);	56	ImageRotationSessionVT(const RotationProperties&, FloatSize, IsCGImageCompatible);
57	ImageRotationSessionVT() = default;	57	ImageRotationSessionVT() = default;
58		58
59	const Optional<AffineTransform>& transform() const { return m_transform; }	59	const Optional<AffineTransform>& transform() const { return m_transform; }
Lines 64-77 public: a/Source/WebCore/platform/graphics/cv/ImageRotationSessionVT.h_sec2
64	RetainPtr<CVPixelBufferRef> rotate(CVPixelBufferRef);	64	RetainPtr<CVPixelBufferRef> rotate(CVPixelBufferRef);
65	WEBCORE_EXPORT RetainPtr<CVPixelBufferRef> rotate(MediaSample&, const RotationProperties&, IsCGImageCompatible);	65	WEBCORE_EXPORT RetainPtr<CVPixelBufferRef> rotate(MediaSample&, const RotationProperties&, IsCGImageCompatible);
66		66
67	bool isMatching(MediaSample&, const RotationProperties&);
68
69	private:	67	private:
70	void initialize(const RotationProperties&, FloatSize, OSType pixelFormat, IsCGImageCompatible);	68	void initialize(const RotationProperties&, FloatSize, IsCGImageCompatible);
71		69
72	Optional<AffineTransform> m_transform;
73	RotationProperties m_rotationProperties;	70	RotationProperties m_rotationProperties;
74	FloatSize m_size;	71	FloatSize m_size;
		72	Optional<AffineTransform> m_transform;
		73	OSType m_pixelFormat;
		74	IsCGImageCompatible m_isCGImageCompatible;
75	FloatSize m_rotatedSize;	75	FloatSize m_rotatedSize;
76	RetainPtr<VTImageRotationSessionRef> m_rotationSession;	76	RetainPtr<VTImageRotationSessionRef> m_rotationSession;
77	RetainPtr<CVPixelBufferPoolRef> m_rotationPool;	77	RetainPtr<CVPixelBufferPoolRef> m_rotationPool;

Lines 59-79 static ImageRotationSessionVT::RotationProperties transformToRotationProperties( a/Source/WebCore/platform/graphics/cv/ImageRotationSessionVT.mm_sec1
- a/Source/WebCore/platform/graphics/cv/ImageRotationSessionVT.mm -22 / +31 lines
59	return rotation;	59	return rotation;
60	}	60	}
61		61
62	ImageRotationSessionVT::ImageRotationSessionVT(AffineTransform&& transform, FloatSize size, OSType pixelFormat, IsCGImageCompatible cvImageCompatibility)	62	ImageRotationSessionVT::ImageRotationSessionVT(AffineTransform&& transform, FloatSize size, IsCGImageCompatible isCGImageCompatible)
63	: ImageRotationSessionVT(transformToRotationProperties(transform), size, pixelFormat, cvImageCompatibility)	63	: ImageRotationSessionVT(transformToRotationProperties(transform), size, isCGImageCompatible)
64	{	64	{
65	m_transform = WTFMove(transform);	65	m_transform = WTFMove(transform);
66	}	66	}
67		67
68	ImageRotationSessionVT::ImageRotationSessionVT(const RotationProperties& rotation, FloatSize size, OSType pixelFormat, IsCGImageCompatible cvImageCompatibility)	68	ImageRotationSessionVT::ImageRotationSessionVT(const RotationProperties& rotation, FloatSize size, IsCGImageCompatible isCGImageCompatible)
69	{	69	{
70	initialize(rotation, size, pixelFormat, cvImageCompatibility);	70	initialize(rotation, size, isCGImageCompatible);
71	}	71	}
72		72
73	void ImageRotationSessionVT::initialize(const RotationProperties& rotation, FloatSize size, OSType pixelFormat, IsCGImageCompatible cvImageCompatibility)	73	void ImageRotationSessionVT::initialize(const RotationProperties& rotation, FloatSize size, IsCGImageCompatible isCGImageCompatible)
74	{	74	{
75	m_rotationProperties = rotation;	75	m_rotationProperties = rotation;
76	m_size = size;	76	m_size = size;
		77	m_isCGImageCompatible = isCGImageCompatible;
77		78
78	if (m_rotationProperties.angle == 90 \|\| m_rotationProperties.angle == 270)	79	if (m_rotationProperties.angle == 90 \|\| m_rotationProperties.angle == 270)
79	size = size.transposedSize();	80	size = size.transposedSize();
Lines 89-114 void ImageRotationSessionVT::initialize(const RotationProperties& rotation, Floa a/Source/WebCore/platform/graphics/cv/ImageRotationSessionVT.mm_sec2
89	VTImageRotationSessionSetProperty(m_rotationSession.get(), kVTImageRotationPropertyKey_FlipVerticalOrientation, kCFBooleanTrue);	90	VTImageRotationSessionSetProperty(m_rotationSession.get(), kVTImageRotationPropertyKey_FlipVerticalOrientation, kCFBooleanTrue);
90	if (m_rotationProperties.flipX)	91	if (m_rotationProperties.flipX)
91	VTImageRotationSessionSetProperty(m_rotationSession.get(), kVTImageRotationPropertyKey_FlipHorizontalOrientation, kCFBooleanTrue);	92	VTImageRotationSessionSetProperty(m_rotationSession.get(), kVTImageRotationPropertyKey_FlipHorizontalOrientation, kCFBooleanTrue);
92
93	auto pixelAttributes = @{
94	(__bridge NSString *)kCVPixelBufferWidthKey: @(m_rotatedSize.width()),
95	(__bridge NSString *)kCVPixelBufferHeightKey: @(m_rotatedSize.height()),
96	(__bridge NSString *)kCVPixelBufferPixelFormatTypeKey: @(pixelFormat),
97	(__bridge NSString *)kCVPixelBufferCGImageCompatibilityKey: (cvImageCompatibility == IsCGImageCompatible::Yes ? @YES : @NO),
98	#if PLATFORM(IOS_SIMULATOR) \|\| PLATFORM(MAC)
99	(__bridge NSString *)kCVPixelBufferIOSurfacePropertiesKey : @{ }
100	#endif
101	};
102	CVPixelBufferPoolRef rawPool = nullptr;
103	if (auto err = CVPixelBufferPoolCreate(kCFAllocatorDefault, nullptr, (__bridge CFDictionaryRef)pixelAttributes, &rawPool); err != noErr)
104	RELEASE_LOG_ERROR(WebRTC, "ImageRotationSessionVT failed creating buffer pool with error %d", err);
105	m_rotationPool = adoptCF(rawPool);
106	}	93	}
107		94
108	RetainPtr<CVPixelBufferRef> ImageRotationSessionVT::rotate(CVPixelBufferRef pixelBuffer)	95	RetainPtr<CVPixelBufferRef> ImageRotationSessionVT::rotate(CVPixelBufferRef pixelBuffer)
109	{	96	{
110	RetainPtr<CVPixelBufferRef> result;	97	auto pixelFormat = CVPixelBufferGetPixelFormatType(pixelBuffer);
		98	if (pixelFormat != m_pixelFormat \|\| !m_rotationPool) {
		99	m_pixelFormat = pixelFormat;
		100	auto pixelAttributes = @{
		101	(__bridge NSString *)kCVPixelBufferWidthKey: @(m_rotatedSize.width()),
		102	(__bridge NSString *)kCVPixelBufferHeightKey: @(m_rotatedSize.height()),
		103	(__bridge NSString *)kCVPixelBufferPixelFormatTypeKey: @(m_pixelFormat),
		104	(__bridge NSString *)kCVPixelBufferCGImageCompatibilityKey: (m_isCGImageCompatible == IsCGImageCompatible::Yes ? @YES : @NO),
		105	#if PLATFORM(IOS_SIMULATOR) \|\| PLATFORM(MAC)
		106	(__bridge NSString *)kCVPixelBufferIOSurfacePropertiesKey : @{ }
		107	#endif
		108	};
		109
		110	CVPixelBufferPoolRef rawPool = nullptr;
		111	if (auto err = CVPixelBufferPoolCreate(kCFAllocatorDefault, nullptr, (__bridge CFDictionaryRef)pixelAttributes, &rawPool); err != noErr) {
		112	RELEASE_LOG_ERROR(WebRTC, "ImageRotationSessionVT failed creating buffer pool with error %d", err);
		113	return nullptr;
		114	}
111		115
		116	m_rotationPool = adoptCF(rawPool);
		117	}
		118
		119	RetainPtr<CVPixelBufferRef> result;
112	CVPixelBufferRef rawRotatedBuffer = nullptr;	120	CVPixelBufferRef rawRotatedBuffer = nullptr;
113	auto status = CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, m_rotationPool.get(), &rawRotatedBuffer);	121	auto status = CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, m_rotationPool.get(), &rawRotatedBuffer);
114	if (status != kCVReturnSuccess) {	122	if (status != kCVReturnSuccess) {
Lines 126-141 RetainPtr<CVPixelBufferRef> ImageRotationSessionVT::rotate(CVPixelBufferRef pixe a/Source/WebCore/platform/graphics/cv/ImageRotationSessionVT.mm_sec3
126	return result;	134	return result;
127	}	135	}
128		136
129	RetainPtr<CVPixelBufferRef> ImageRotationSessionVT::rotate(MediaSample& sample, const RotationProperties& rotation, IsCGImageCompatible cvImageCompatibility)	137	RetainPtr<CVPixelBufferRef> ImageRotationSessionVT::rotate(MediaSample& sample, const RotationProperties& rotation, IsCGImageCompatible cgImageCompatible)
130	{	138	{
131	auto pixelBuffer = static_cast<CVPixelBufferRef>(PAL::CMSampleBufferGetImageBuffer(sample.platformSample().sample.cmSampleBuffer));	139	auto pixelBuffer = static_cast<CVPixelBufferRef>(PAL::CMSampleBufferGetImageBuffer(sample.platformSample().sample.cmSampleBuffer));
132	ASSERT(pixelBuffer);	140	ASSERT(pixelBuffer);
133	if (!pixelBuffer)	141	if (!pixelBuffer)
134	return nullptr;	142	return nullptr;
135		143
		144	m_pixelFormat = CVPixelBufferGetPixelFormatType(pixelBuffer);
136	IntSize size { (int)CVPixelBufferGetWidth(pixelBuffer), (int)CVPixelBufferGetHeight(pixelBuffer) };	145	IntSize size { (int)CVPixelBufferGetWidth(pixelBuffer), (int)CVPixelBufferGetHeight(pixelBuffer) };
137	if (rotation != m_rotationProperties \|\| m_size != size)	146	if (rotation != m_rotationProperties \|\| m_size != size)
138	initialize(rotation, size, CVPixelBufferGetPixelFormatType(pixelBuffer), cvImageCompatibility);	147	initialize(rotation, size, cgImageCompatible);
139		148
140	return rotate(pixelBuffer);	149	return rotate(pixelBuffer);
141	}	150	}

Lines 1-3 a/LayoutTests/ChangeLog_sec1
- a/LayoutTests/ChangeLog +10 lines
		1	2021-03-06 Peng Liu <peng.liu6@apple.com>
		2
		3	[GPUP] Some media tests related to canvas fail when media in GPU Process is enabled
		4	https://bugs.webkit.org/show_bug.cgi?id=221820
		5
		6	Reviewed by NOBODY (OOPS!).
		7
		8	* media/video-orientation-canvas.html:
		9	* platform/wk2/TestExpectations:
		10
1	2021-03-05 Ryosuke Niwa <rniwa@webkit.org>	11	2021-03-05 Ryosuke Niwa <rniwa@webkit.org>
2		12
3	Add leak tests for Range and StaticRange	13	Add leak tests for Range and StaticRange