Attachment #439150 for bug #230617

View Details Raw Unified Return to Bug 230617

| Differences between



2021-09-24  Kimmo Kinnunen  <kkinnunen@apple.com>

        REGRESSION (r280963): gl.texImage2D upload of getUserMedia streams via <video> element fails on iOS 15.1 beta 1
        https://bugs.webkit.org/show_bug.cgi?id=230617
        <rdar://problem/83407577>

        Reviewed by NOBODY (OOPS!).

        r280963 implemented the SW readback with the assumption that all video element sources
        would implement MediaPlayerPrivateInterface::nativeImageForCurrentTime().
        The MediaPlayerPrivateMediaStreamAVFObjC would not do so, so the 
        texImage2D calls from videos with media streams attached would not produce
        results.
        
        Fix this by implementing nativeImageForCurrentTime() and the related 
        pixelBufferForCurrentTime(). This means that this implements the GPU codepath
        for media streams.

        The underlying pixel buffers provided by the  MediaPlayerPrivateMediaStreamAVFObjC
        have orientation, so add the transform to the return value of the 
        nativeImageForCurrentTime() and pixelBufferForCurrentTime().

        Tests to be added..

        * html/HTMLVideoElement.cpp:
        * html/HTMLVideoElement.h:
        * html/canvas/CanvasRenderingContext2DBase.cpp:
        (WebCore::CanvasRenderingContext2DBase::drawImage):
        (WebCore::CanvasRenderingContext2DBase::createPattern):
        * html/canvas/WebGLRenderingContextBase.cpp:
        (WebCore::WebGLRenderingContextBase::videoFrameToImage):
        * platform/graphics/MediaPlayer.cpp:
        (WebCore::MediaPlayer::pixelBufferForCurrentTime):
        (WebCore::MediaPlayer::nativeImageForCurrentTime):
        * platform/graphics/MediaPlayer.h:
        * platform/graphics/MediaPlayerPrivate.h:
        (WebCore::MediaPlayerPrivateInterface::pixelBufferForCurrentTime):
        (WebCore::MediaPlayerPrivateInterface::nativeImageForCurrentTime):
        * platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.h:
        * platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:
        (WebCore::MediaPlayerPrivateAVFoundationObjC::pixelBufferForCurrentTime):
        (WebCore::MediaPlayerPrivateAVFoundationObjC::nativeImageForCurrentTime):
        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.h:
        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm:
        (WebCore::MediaPlayerPrivateMediaSourceAVFObjC::nativeImageForCurrentTime):
        (WebCore::MediaPlayerPrivateMediaSourceAVFObjC::paintCurrentFrameInContext):
        (WebCore::MediaPlayerPrivateMediaSourceAVFObjC::pixelBufferForCurrentTime):
        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:
        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::paintCurrentFrameInContext):
        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::nativeImageForCurrentTime):
        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::pixelBufferForCurrentTime):
        (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::transformForCurrentSample):
        * platform/graphics/cv/GraphicsContextGLCV.h:
        * platform/graphics/cv/GraphicsContextGLCVANGLE.cpp:
        (WebCore::GraphicsContextGLCVANGLE::initializeUVContextObjects):
        (WebCore::GraphicsContextGLCVANGLE::copyPixelBufferToTexture):
        * platform/graphics/cv/GraphicsContextGLCVANGLE.h:
        * platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:
        (WebCore::MediaPlayerPrivateGStreamer::nativeImageForCurrentTime):
        * platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.h:
        * platform/graphics/opengl/GraphicsContextGLOpenGL.cpp:
        (WebCore::GraphicsContextGLOpenGL::copyTextureFromMedia):

2021-09-22  Kimmo Kinnunen  <kkinnunen@apple.com>

        MediaStream canvas.captureStream() fails for WebGL



2021-09-24  Kimmo Kinnunen  <kkinnunen@apple.com>

        REGRESSION (r280963): gl.texImage2D upload of getUserMedia streams via <video> element fails on iOS 15.1 beta 1
        https://bugs.webkit.org/show_bug.cgi?id=230617
        <rdar://problem/83407577>

        Reviewed by NOBODY (OOPS!).

        Add the ability to pass transform of the pixels along with the
        pixel buffers from nativeImageForCurrentTime and pixelBufferForCurrentTime
        in GPU process objects.

        * GPUProcess/graphics/RemoteGraphicsContextGL.cpp:
        (WebKit::RemoteGraphicsContextGL::copyTextureFromMedia):
        * GPUProcess/media/RemoteMediaPlayerProxy.h:
        * GPUProcess/media/RemoteMediaPlayerProxy.messages.in:
        * GPUProcess/media/cocoa/RemoteMediaPlayerProxyCocoa.mm:
        (WebKit::RemoteMediaPlayerProxy::nativeImageForCurrentTime):
        (WebKit::RemoteMediaPlayerProxy::pixelBufferForCurrentTime):
        * WebProcess/GPU/media/MediaPlayerPrivateRemote.cpp:
        (WebKit::MediaPlayerPrivateRemote::paintCurrentFrameInContext):
        (WebKit::MediaPlayerPrivateRemote::pixelBufferForCurrentTime):
        (WebKit::MediaPlayerPrivateRemote::nativeImageForCurrentTime):
        * WebProcess/GPU/media/MediaPlayerPrivateRemote.h:
        * WebProcess/GPU/media/cocoa/MediaPlayerPrivateRemoteCocoa.mm:
        (WebKit::MediaPlayerPrivateRemote::nativeImageForCurrentTime):
        (WebKit::MediaPlayerPrivateRemote::pixelBufferForCurrentTime):
        * WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:
        (WebKit::createShareableBitmap):

2021-09-16  Kimmo Kinnunen  <kkinnunen@apple.com>

        Add utility to create CVPixelBuffers from IOSurfaces



    return player()->hasVideo() && player()->hasAvailableVideoFrame();
}

RefPtr<NativeImage> HTMLVideoElement::nativeImageForCurrentTime()
{
    if (!player())
        return nullptr;

    return player()->nativeImageForCurrentTime();
}

ExceptionOr<void> HTMLVideoElement::webkitEnterFullscreen()
{
    ALWAYS_LOG(LOGIDENTIFIER);



    // Used by canvas to gain raw pixel access
    void paintCurrentFrameInContext(GraphicsContext&, const FloatRect&);

    WEBCORE_EXPORT RefPtr<NativeImage> nativeImageForCurrentTime();

    WEBCORE_EXPORT bool shouldDisplayPosterImage() const;

    URL posterImageURL() const;



    checkOrigin(&video);

#if USE(CG)
    if (auto player = video.player()) {
        if (auto frame = player->nativeImageForCurrentTime()) {
            if (frame->transform.isIdentity()) {
                c->drawNativeImage(frame->nativeImage, FloatSize(video.videoWidth(), video.videoHeight()), dstRect, srcRect);

                if (isEntireBackingStoreDirty())
                    didDraw(std::nullopt);
                else if (rectContainsCanvas(dstRect))
                    didDrawEntireCanvas();
                else
                    didDraw(dstRect);

                return { };
            }
        }
    }
#endif


    bool originClean = canvasBase().originClean();

#if USE(CG)
    if (auto player = videoElement.player()) {
        if (auto frame = player->nativeImageForCurrentTime()) {
            if (frame->transform.isIdentity())
                return RefPtr<CanvasPattern> { CanvasPattern::create(WTFMove(frame->nativeImage), repeatX, repeatY, originClean) };
        }
    }
#endif

    auto renderingMode = drawingContext() ? drawingContext()->renderingMode() : RenderingMode::Unaccelerated;

Lines 5900-5922 RefPtr<Image> WebGLRenderingContextBase::drawImageIntoBuffer(Image& image, int w a/Source/WebCore/html/canvas/WebGLRenderingContextBase.cpp_sec1
- a/Source/WebCore/html/canvas/WebGLRenderingContextBase.cpp -6 / +14 lines
5900	RefPtr<Image> WebGLRenderingContextBase::videoFrameToImage(HTMLVideoElement* video, BackingStoreCopy backingStoreCopy, const char* functionName)	5900	RefPtr<Image> WebGLRenderingContextBase::videoFrameToImage(HTMLVideoElement* video, BackingStoreCopy backingStoreCopy, const char* functionName)
5901	{	5901	{
5902	#if USE(AVFOUNDATION)	5902	#if USE(AVFOUNDATION)
5903	auto nativeImage = video->nativeImageForCurrentTime();	5903	auto player = video->player();
		5904	if (!player)
		5905	return nullptr;
		5906	auto frame = player->nativeImageForCurrentTime();
5904	// Currently we might be missing an image due to MSE not being able to provide the first requested frame.	5907	// Currently we might be missing an image due to MSE not being able to provide the first requested frame.
5905	// https://bugs.webkit.org/show_bug.cgi?id=228997	5908	// https://bugs.webkit.org/show_bug.cgi?id=228997
5906	if (!nativeImage)	5909	if (!frame)
5907	return nullptr;	5910	return nullptr;
5908	IntSize imageSize = nativeImage->size();	5911	IntSize imageSize = frame->nativeImage->size();
5909	if (imageSize.isEmpty()) {	5912	if (imageSize.isEmpty()) {
5910	synthesizeGLError(GraphicsContextGL::INVALID_VALUE, functionName, "video visible size is empty");	5913	synthesizeGLError(GraphicsContextGL::INVALID_VALUE, functionName, "video visible size is empty");
5911	return nullptr;	5914	return nullptr;
5912	}	5915	}
5913	FloatRect imageRect { { }, imageSize };	5916	IntRect imageRect = { { }, imageSize };
5914	ImageBuffer* imageBuffer = m_generatedImageCache.imageBuffer(imageSize, CompositeOperator::Copy);	5917	// FIXME: Why is there a inverse() here.
		5918	IntRect bufferRect = frame->transform.inverse().value_or(AffineTransform { }).mapRect(imageRect);
		5919	ImageBuffer* imageBuffer = m_generatedImageCache.imageBuffer(bufferRect.size(), CompositeOperator::Copy);
5915	if (!imageBuffer) {	5920	if (!imageBuffer) {
5916	synthesizeGLError(GraphicsContextGL::OUT_OF_MEMORY, functionName, "out of memory");	5921	synthesizeGLError(GraphicsContextGL::OUT_OF_MEMORY, functionName, "out of memory");
5917	return nullptr;	5922	return nullptr;
5918	}	5923	}
5919	imageBuffer->context().drawNativeImage(*nativeImage, imageRect.size(), imageRect, imageRect, CompositeOperator::Copy);	5924	auto& context = imageBuffer->context();
		5925	GraphicsContextStateSaver stateSaver(context);
		5926	context.concatCTM(frame->transform);
		5927	context.drawNativeImage(frame->nativeImage, imageRect.size(), bufferRect, bufferRect, CompositeOperator::Copy);
5920	#else	5928	#else
5921	// This is a legacy code path that produces incompatible texture size when the	5929	// This is a legacy code path that produces incompatible texture size when the
5922	// video visible size is different to the natural size. This should be removed	5930	// video visible size is different to the natural size. This should be removed




#else

std::optional<MediaPlayer::PixelBufferFrame> MediaPlayer::pixelBufferForCurrentTime()
{
    return m_private->pixelBufferForCurrentTime();
}

#endif

std::optional<MediaPlayer::NativeImageFrame> MediaPlayer::nativeImageForCurrentTime()
{
    return m_private->nativeImageForCurrentTime();
}




#if ENABLE(VIDEO)

#include "AffineTransform.h"
#include "AudioTrackPrivate.h"
#include "ContentType.h"
#include "Cookie.h"

#include <wtf/URL.h>
#include "VideoTrackPrivate.h"
#include <JavaScriptCore/Uint8Array.h>
#include <optional>
#include <wtf/CompletionHandler.h>
#include <wtf/Function.h>
#include <wtf/HashSet.h>

#if !USE(AVFOUNDATION)
    bool copyVideoTextureToPlatformTexture(GraphicsContextGL*, PlatformGLObject texture, GCGLenum target, GCGLint level, GCGLenum internalFormat, GCGLenum format, GCGLenum type, bool premultiplyAlpha, bool flipY);
#else
    struct PixelBufferFrame {
        RetainPtr<CVPixelBufferRef> pixelBuffer;
        AffineTransform transform;
    };
    std::optional<PixelBufferFrame> pixelBufferForCurrentTime();
#endif
    struct NativeImageFrame {
        Ref<NativeImage> nativeImage;
        AffineTransform transform;
    };
    std::optional<NativeImageFrame> nativeImageForCurrentTime();

    using MediaPlayerEnums::NetworkState;
    NetworkState networkState();



#if !USE(AVFOUNDATION)
    virtual bool copyVideoTextureToPlatformTexture(GraphicsContextGL*, PlatformGLObject, GCGLenum, GCGLint, GCGLenum, GCGLenum, GCGLenum, bool, bool) { return false; }
#else
    virtual std::optional<MediaPlayer::PixelBufferFrame> pixelBufferForCurrentTime() { return std::nullopt; }
#endif
    virtual std::optional<MediaPlayer::NativeImageFrame> nativeImageForCurrentTime() { return std::nullopt; }

    virtual void setPreload(MediaPlayer::Preload) { }




    bool updateLastPixelBuffer();
    bool videoOutputHasAvailableFrame();
    void paintWithVideoOutput(GraphicsContext&, const FloatRect&);
    std::optional<MediaPlayer::PixelBufferFrame> pixelBufferForCurrentTime() final;
    std::optional<MediaPlayer::NativeImageFrame> nativeImageForCurrentTime() final;
    void waitForVideoOutputMediaDataWillChange();

    RetainPtr<CVPixelBufferRef> pixelBufferForCurrentTime() final;

#if ENABLE(LEGACY_ENCRYPTED_MEDIA)
    void keyAdded() final;
    std::unique_ptr<LegacyCDMSession> createSession(const String& keySystem, LegacyCDMSessionClient*) final;




}

std::optional<MediaPlayer::PixelBufferFrame> MediaPlayerPrivateAVFoundationObjC::pixelBufferForCurrentTime()
{
    updateLastPixelBuffer();
    if (!m_lastPixelBuffer)
        return std::nullopt;
    return MediaPlayer::PixelBufferFrame { m_lastPixelBuffer, { } };
    return m_lastPixelBuffer;
}

std::optional<MediaPlayer::NativeImageFrame> MediaPlayerPrivateAVFoundationObjC::nativeImageForCurrentTime()
{
    updateLastImage(UpdateType::UpdateSynchronously);
    if (m_lastImage)
        return std::nullopt;
    return MediaPlayer::NativeImageFrame { *m_lastImage, { } };
}

void MediaPlayerPrivateAVFoundationObjC::waitForVideoOutputMediaDataWillChange()




    bool didLoadingProgress() const override;

    std::optional<MediaPlayer::NativeImageFrame> nativeImageForCurrentTime() override;
    bool updateLastPixelBuffer();
    bool updateLastImage();
    void paint(GraphicsContext&, const FloatRect&) override;
    void paintCurrentFrameInContext(GraphicsContext&, const FloatRect&) override;
    std::optional<MediaPlayer::PixelBufferFrame>  pixelBufferForCurrentTime() final;

    bool supportsAcceleratedRendering() const override;
    // called when the rendering system flips the into or out of accelerated rendering mode.



    return loadingProgressed;
}

std::optional<MediaPlayer::NativeImageFrame> MediaPlayerPrivateMediaSourceAVFObjC::nativeImageForCurrentTime()
{
    updateLastImage();
    if (!m_lastImage)
        return std::nullopt;
    return MediaPlayer::NativeImageFrame { *m_lastImage, { } };
}

bool MediaPlayerPrivateMediaSourceAVFObjC::updateLastPixelBuffer()

        return;

    GraphicsContextStateSaver stateSaver(context);
    FloatRect imageRect { FloatPoint::zero(), image->nativeImage->size() };
    context.drawNativeImage(image->nativeImage, imageRect.size(), outputRect, imageRect);
}

std::optional<MediaPlayer::PixelBufferFrame> MediaPlayerPrivateMediaSourceAVFObjC::pixelBufferForCurrentTime()
{
    // We have been asked to paint into a WebGL canvas, so take that as a signal to create
    // a decompression session, even if that means the native video can't also be displayed

        acceleratedRenderingStateChanged();
    }

    updateLastPixelBuffer();
    if (!m_lastPixelBuffer)
        return std::nullopt;
    return MediaPlayer::PixelBufferFrame { m_lastPixelBuffer, { } };

    return m_lastPixelBuffer;
}

bool MediaPlayerPrivateMediaSourceAVFObjC::hasAvailableVideoFrame() const

Lines 80-85 public: a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h_sec1
- a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h +4 lines
80	const void* logIdentifier() const final { return reinterpret_cast<const void*>(m_logIdentifier); }	80	const void* logIdentifier() const final { return reinterpret_cast<const void*>(m_logIdentifier); }
81	WTFLogChannel& logChannel() const final;	81	WTFLogChannel& logChannel() const final;
82		82
		83	std::optional<MediaPlayer::NativeImageFrame> nativeImageForCurrentTime() final;
		84	std::optional<MediaPlayer::PixelBufferFrame> pixelBufferForCurrentTime() final;
		85
83	using MediaStreamTrackPrivate::Observer::weakPtrFactory;	86	using MediaStreamTrackPrivate::Observer::weakPtrFactory;
84	using WeakValueType = MediaStreamTrackPrivate::Observer::WeakValueType;	87	using WeakValueType = MediaStreamTrackPrivate::Observer::WeakValueType;
85		88
Lines 219-224 private: a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h_sec2
219	bool hideRootLayer() const { return (!activeVideoTrack() \|\| m_waitingForFirstImage) && m_displayMode != PaintItBlack; }	222	bool hideRootLayer() const { return (!activeVideoTrack() \|\| m_waitingForFirstImage) && m_displayMode != PaintItBlack; }
220		223
221	MediaStreamTrackPrivate* activeVideoTrack() const;	224	MediaStreamTrackPrivate* activeVideoTrack() const;
		225	AffineTransform transformForCurrentSample();
222		226
223	MediaPlayer* m_player { nullptr };	227	MediaPlayer* m_player { nullptr };
224	RefPtr<MediaStreamPrivate> m_mediaStreamPrivate;	228	RefPtr<MediaStreamPrivate> m_mediaStreamPrivate;



#pragma mark -
#pragma mark AVSampleBuffer Methods

// Returns transform matrix for video when 0.5, 0.5 anchor point is used.
static inline CGAffineTransform videoTransformationMatrix(MediaSample& sample)
{
    CMSampleBufferRef sampleBuffer = sample.platformSample().sample.cmSampleBuffer;

    if (m_displayMode == None || !metaDataAvailable() || context.paintingDisabled())
        return;

    if (m_displayMode != PaintItBlack && m_imagePainter.mediaSample)
        updateCurrentFrameImage();

    GraphicsContextStateSaver stateSaver(context);
    if (m_displayMode == PaintItBlack) {
        context.fillRect(IntRect(IntPoint(), IntSize(destRect.width(), destRect.height())), Color::black);
        return;
    }

    updateCurrentFrameImage();

    if (!m_imagePainter.cgImage || !m_imagePainter.mediaSample)
        return;
    GraphicsContextStateSaver stateSaver(context);
    auto& image = m_imagePainter.cgImage;
    FloatRect imageRect { FloatPoint::zero(), image->size() };
    if (!m_videoTransform)
        m_videoTransform = videoTransformationMatrix(*m_imagePainter.mediaSample);
    // FIXME: Why is there a inverse() here?
    AffineTransform videoTransform = *m_videoTransform;
    FloatRect transformedDestRect = videoTransform.inverse().value_or(AffineTransform()).mapRect(destRect);
    context.concatCTM(videoTransform);

    return LogMedia;
}

std::optional<MediaPlayer::NativeImageFrame> MediaPlayerPrivateMediaStreamAVFObjC::nativeImageForCurrentTime()
{
    if (m_displayMode == None || !metaDataAvailable())
        return std::nullopt;
    // FIXME: For accelerated operation, we could return a black pixel here.
    if (m_displayMode == PaintItBlack)
        return std::nullopt;
    updateCurrentFrameImage();
    if (!m_imagePainter.cgImage || !m_imagePainter.mediaSample)
        return std::nullopt;
    return MediaPlayer::NativeImageFrame { *m_imagePainter.cgImage, transformForCurrentSample() };
}

std::optional<MediaPlayer::PixelBufferFrame> MediaPlayerPrivateMediaStreamAVFObjC::pixelBufferForCurrentTime()
{
    if (m_displayMode == None || !metaDataAvailable())
        return std::nullopt;
    // FIXME: For accelerated operation, we could return a black pixel here.
    if (m_displayMode == PaintItBlack)
        return std::nullopt;
    if (!m_imagePainter.mediaSample)
        return std::nullopt;
    RetainPtr pixelBuffer = static_cast<CVPixelBufferRef>(PAL::CMSampleBufferGetImageBuffer(m_imagePainter.mediaSample->platformSample().sample.cmSampleBuffer));
    if (!pixelBuffer)
        return std::nullopt;
    return MediaPlayer::PixelBufferFrame { WTFMove(pixelBuffer), transformForCurrentSample() };
}

AffineTransform MediaPlayerPrivateMediaStreamAVFObjC::transformForCurrentSample()
{
    ASSERT(m_imagePainter.mediaSample);
    // FIXME: why is this reset this way during enqueue, and not just obtained per
    // sample?
    if (!m_videoTransform)
        m_videoTransform = videoTransformationMatrix(*m_imagePainter.mediaSample);
    if (auto transform = AffineTransform(*m_videoTransform).inverse()) {
        AffineTransform result;
        result.translate(0.5, 0.5);
        result.multiply(*m_videoTransform);
        result.translate(-0.5, -0.5);
        return result;
    }
    return { };
}

}

#endif




#if ENABLE(WEBGL) && USE(AVFOUNDATION)

#include "AffineTransform.h"
#include "GraphicsContextGL.h"

typedef struct __CVBuffer* CVPixelBufferRef;

    virtual ~GraphicsContextGLCV() = default;

    using FlipY = GraphicsContextGL::FlipY;
    // Copies CV pixel buffer to GraphicsContextGL TEXTURE_2D texture of the same size as the pixel buffer with the transform applied.
    // Returns true on success.
    virtual bool copyPixelBufferToTexture(CVPixelBufferRef, AffineTransform, PlatformGLObject outputTexture, GCGLint level, GCGLenum internalFormat, GCGLenum format, GCGLenum type, FlipY) = 0;
};

}



    "attribute vec2 a_position;"
    "uniform vec2 u_yTextureSize;"
    "uniform vec2 u_uvTextureSize;"
    "uniform mat3 u_transform;"
    "uniform int u_flipY;"
    "varying vec2 v_yTextureCoordinate;"
    "varying vec2 v_uvTextureCoordinate;"
    "void main()"
    "{"
    "    gl_Position = vec4(a_position, 0, 1.0);"
    "    vec2 normalizedPosition = (u_transform * vec3(a_position * .5 + .5, 1.0)).xy;"
    "    if (u_flipY == 1)"
    "        normalizedPosition.y = 1.0 - normalizedPosition.y;"
    "    v_yTextureCoordinate = normalizedPosition;"

    "attribute vec2 a_position;"
    "uniform vec2 u_yTextureSize;"
    "uniform vec2 u_uvTextureSize;"
    "uniform mat3 u_transform;"
    "uniform int u_flipY;"
    "varying vec2 v_yTextureCoordinate;"
    "varying vec2 v_uvTextureCoordinate;"
    "void main()"
    "{"
    "    gl_Position = vec4(a_position, 0, 1.0);"
    "    vec2 normalizedPosition = (u_transform * vec3(a_position * .5 + .5, 1.0)).xy;"
    "    if (u_flipY == 1)"
    "        normalizedPosition.y = 1.0 - normalizedPosition.y;"
    "    v_yTextureCoordinate = normalizedPosition * u_yTextureSize;"

    m_yTextureUniformLocation = m_context->getUniformLocation(m_yuvProgram, "u_yTexture"_s);
    m_uvTextureUniformLocation = m_context->getUniformLocation(m_yuvProgram, "u_uvTexture"_s);
    m_colorMatrixUniformLocation = m_context->getUniformLocation(m_yuvProgram, "u_colorMatrix"_s);
    m_transformUniformLocation = m_context->getUniformLocation(m_yuvProgram, "u_transform"_s);
    m_yuvFlipYUniformLocation = m_context->getUniformLocation(m_yuvProgram, "u_flipY"_s);
    m_yTextureSizeUniformLocation = m_context->getUniformLocation(m_yuvProgram, "u_yTextureSize"_s);
    m_uvTextureSizeUniformLocation = m_context->getUniformLocation(m_yuvProgram, "u_uvTextureSize"_s);

    EGL_DestroySurface(display, handle);
}

bool GraphicsContextGLCVANGLE::copyPixelBufferToTexture(CVPixelBufferRef image, AffineTransform transform, PlatformGLObject outputTexture, GCGLint level, GCGLenum internalFormat, GCGLenum format, GCGLenum type, FlipY flipY)
{
    // FIXME: This currently only supports '420v' and '420f' pixel formats. Investigate supporting more pixel formats.
    OSType pixelFormat = CVPixelBufferGetPixelFormatType(image);

    IOSurfaceRef surface = CVPixelBufferGetIOSurface(image);
    if (!surface)
        return false;
    transform = *transform.inverse();
    auto newSurfaceSeed = IOSurfaceGetSeed(surface);
    if (flipY == m_lastFlipY
        && surface == m_lastSurface 

        // image hasn't been modified since the last time it was copied, this is a no-op.
        return true;
    }

    if (!m_yuvProgram) {
        if (!initializeUVContextObjects()) {
            LOG(WebGL, "GraphicsContextGLCVANGLE::copyVideoTextureToPlatformTexture(%p) - Unable to initialize OpenGL context objects.", this);
            return false;
        }
    }
    IntRect imageRect { { }, { static_cast<int>(CVPixelBufferGetWidth(image)), static_cast<int>(CVPixelBufferGetHeight(image)) } };
    auto outputRect = transform.mapRect(imageRect);

    size_t width = outputRect.width();
    size_t height = outputRect.height();

    m_context->bindFramebuffer(GraphicsContextGL::FRAMEBUFFER, m_framebuffer);


    auto transferFunction = transferFunctionFromString(dynamic_cf_cast<CFStringRef>(CVBufferGetAttachment(image, kCVImageBufferYCbCrMatrixKey, nil)));
    auto colorMatrix = YCbCrToRGBMatrixForRangeAndTransferFunction(range, transferFunction);
    m_context->uniformMatrix4fv(m_colorMatrixUniformLocation, GL_FALSE, colorMatrix);
    GLfloat transformMatrix[] = {
        static_cast<GLfloat>(transform.a()),
        static_cast<GLfloat>(transform.b()),
        0.f,
        static_cast<GLfloat>(transform.c()),
        static_cast<GLfloat>(transform.d()),
        0.f,
        static_cast<GLfloat>(transform.e()),
        static_cast<GLfloat>(transform.f()),
        1.f
    };
    m_context->uniformMatrix3fv(m_transformUniformLocation, GL_FALSE, transformMatrix);

    // Do the actual drawing.
    m_context->drawArrays(GraphicsContextGL::TRIANGLES, 0, 6);



    GraphicsContextGLCVANGLE(GraphicsContextGLOpenGL&);
    ~GraphicsContextGLCVANGLE() final;

    bool copyPixelBufferToTexture(CVPixelBufferRef, AffineTransform, PlatformGLObject outputTexture, GCGLint level, GCGLenum internalFormat, GCGLenum format, GCGLenum type, FlipY) final;

private:
    bool initializeUVContextObjects();

    GCGLint m_uvTextureUniformLocation { -1 };
    GCGLint m_yuvFlipYUniformLocation { -1 };
    GCGLint m_colorMatrixUniformLocation { -1 };
    GCGLint m_transformUniformLocation { -1 };
    GCGLint m_yuvPositionAttributeLocation { -1 };
    GCGLint m_yTextureSizeUniformLocation { -1 };
    GCGLint m_uvTextureSizeUniformLocation { -1 };



    return m_videoTextureCopier->copyVideoTextureToPlatformTexture(*layerBuffer.get(), size, outputTexture, outputTarget, level, internalFormat, format, type, flipY, m_videoSourceOrientation, premultiplyAlpha);
}

std::optional<MediaPlayer::NativeImageFrame> MediaPlayerPrivateGStreamer::nativeImageForCurrentTime()
{
    return std::nullopt;
}
#endif // USE(GSTREAMER_GL)


Lines 209-215 public: a/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.h_sec1
- a/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.h -1 / +1 lines
209		209
210	#if USE(GSTREAMER_GL)	210	#if USE(GSTREAMER_GL)
211	bool copyVideoTextureToPlatformTexture(GraphicsContextGL*, PlatformGLObject, GCGLenum, GCGLint, GCGLenum, GCGLenum, GCGLenum, bool, bool) override;	211	bool copyVideoTextureToPlatformTexture(GraphicsContextGL*, PlatformGLObject, GCGLenum, GCGLint, GCGLenum, GCGLenum, GCGLenum, bool, bool) override;
212	RefPtr<NativeImage> nativeImageForCurrentTime() override;	212	std::optional<MediaPlayer::NativeImageFrame> nativeImageForCurrentTime() override;
213	#endif	213	#endif
214		214
215	void updateEnabledVideoTrack();	215	void updateEnabledVideoTrack();



bool GraphicsContextGLOpenGL::copyTextureFromMedia(MediaPlayer& player, PlatformGLObject outputTexture, GCGLenum outputTarget, GCGLint level, GCGLenum internalFormat, GCGLenum format, GCGLenum type, bool premultiplyAlpha, bool flipY)
{
#if USE(AVFOUNDATION)
    auto frame = player.pixelBufferForCurrentTime();

    if (!frame)
        return false;

    auto contextCV = asCV();


    UNUSED_VARIABLE(premultiplyAlpha);
    ASSERT_UNUSED(outputTarget, outputTarget == GraphicsContextGL::TEXTURE_2D);
    return contextCV->copyPixelBufferToTexture(frame->pixelBuffer.get(), frame->transform, outputTexture, level, internalFormat, format, type, GraphicsContextGL::FlipY(flipY));
#else
    return player.copyVideoTextureToPlatformTexture(this, outputTexture, outputTarget, level, internalFormat, format, type, premultiplyAlpha, flipY);
#endif



    UNUSED_VARIABLE(premultiplyAlpha);
    ASSERT_UNUSED(target, target == GraphicsContextGL::TEXTURE_2D);

    std::optional<MediaPlayer::PixelBufferFrame> frame;
    auto getPixelBuffer = [&] {
        if (!m_gpuConnectionToWebProcess)
            return;

        if (auto mediaPlayer = m_gpuConnectionToWebProcess->remoteMediaPlayerManagerProxy().mediaPlayer(mediaPlayerIdentifier))
            frame = mediaPlayer->pixelBufferForCurrentTime();
    };

    callOnMainRunLoopAndWait(WTFMove(getPixelBuffer));

    if (!frame) {
        completionHandler(false);
        return;
    }

        return;
    }

    completionHandler(contextCV->copyPixelBufferToTexture(frame->pixelBuffer.get(), frame->transform, texture, level, internalFormat, format, type, GraphicsContextGL::FlipY(flipY)));
#else
    UNUSED_VARIABLE(mediaPlayerIdentifier);
    UNUSED_VARIABLE(texture);



    void currentTimeChanged(const MediaTime&);

#if PLATFORM(COCOA)
    void nativeImageForCurrentTime(CompletionHandler<void(std::optional<WTF::MachSendRight>&&, WebCore::AffineTransform&&)>&&);
#endif
#if USE(AVFOUNDATION)
    void pixelBufferForCurrentTime(CompletionHandler<void(RetainPtr<CVPixelBufferRef>&&, WebCore::AffineTransform&&)>&&);
#endif

#if !RELEASE_LOG_DISABLED



#endif

#if PLATFORM(COCOA)
    NativeImageForCurrentTime() -> (std::optional<MachSendRight> sendRight, WebCore::AffineTransform transform) Synchronous
#endif
#if USE(AVFOUNDATION)
    PixelBufferForCurrentTime() -> (RetainPtr<CVPixelBufferRef> pixelBuffer, WebCore::AffineTransform transform) Synchronous
#endif

    PlayAtHostTime(MonotonicTime time)



    setVideoInlineSizeIfPossible(*m_inlineLayerHostingContext, size);
}

void RemoteMediaPlayerProxy::nativeImageForCurrentTime(CompletionHandler<void(std::optional<WTF::MachSendRight>&&, WebCore::AffineTransform&&)>&& completionHandler)
{
    if (!m_player) {
        completionHandler(std::nullopt, { });
        return;
    }

    auto frame = m_player->nativeImageForCurrentTime();
    if (!frame) {
        completionHandler(std::nullopt, { });
        return;
    }

    auto platformImage = frame->nativeImage->platformImage();
    if (!platformImage) {
        completionHandler(std::nullopt, { });
        return;
    }

    auto surface = WebCore::IOSurface::createFromImage(platformImage.get());
    if (!surface) {
        completionHandler(std::nullopt, { });
        return;
    }

    completionHandler(surface->createSendRight(), WTFMove(frame->transform));
}

void RemoteMediaPlayerProxy::pixelBufferForCurrentTime(CompletionHandler<void(RetainPtr<CVPixelBufferRef>&&, WebCore::AffineTransform&&)>&& completionHandler)
{
    std::optional<MediaPlayer::PixelBufferFrame> result;
    if (m_player) {
        result = m_player->pixelBufferForCurrentTime();
    if (result)
        completionHandler(WTFMove(result->pixelBuffer), WTFMove(result->transform));
    else
        completionHandler(nullptr, { });
}

}

} // namespace WebKit



    if (context.paintingDisabled())
        return;

    auto frame = nativeImageForCurrentTime();
    if (!frame)
        return;

    FloatRect imageRect { FloatPoint::zero(), frame->nativeImage->size() };
    // FIXME: Why do we map with the inverse but use the straight transform?
    // See MediaPlayerPrivateMediaStreamSoruceAVFObjC.mm.
    imageRect = frame->transform.inverse().value_or(AffineTransform { }).mapRect(imageRect);
    GraphicsContextStateSaver stateSaver(context);
    context.concatCTM(frame->transform);
    context.drawNativeImage(frame->nativeImage, imageRect.size(), rect, imageRect);
}

#if !USE(AVFOUNDATION)

    return false;
}
#elif !PLATFORM(COCOA)
std::optional<WebCore::MediaPlayer::PixelBufferFrame> MediaPlayerPrivateRemote::pixelBufferForCurrentTime()
{
    notImplemented();
    return std::nullopt;
}
#endif

#if !PLATFORM(COCOA)
std::optional<WebCore::MediaPlayer::NativeImageFrame> MediaPlayerPrivateRemote::nativeImageForCurrentTime()
{
    notImplemented();
    return std::nullopt;
}
#endif




#if !USE(AVFOUNDATION)
    bool copyVideoTextureToPlatformTexture(WebCore::GraphicsContextGL*, PlatformGLObject, GCGLenum, GCGLint, GCGLenum, GCGLenum, GCGLenum, bool, bool) final;
#else
    std::optional<WebCore::MediaPlayer::PixelBufferFrame> pixelBufferForCurrentTime() final;
#endif
    std::optional<WebCore::MediaPlayer::NativeImageFrame> nativeImageForCurrentTime() final;

    WebCore::MediaPlayerIdentifier identifier() const final;




}
#endif

std::optional<MediaPlayer::NativeImageFrame> MediaPlayerPrivateRemote::nativeImageForCurrentTime()
{
    std::optional<MachSendRight> sendRight;
    WebCore::AffineTransform transform;
    if (!connection().sendSync(Messages::RemoteMediaPlayerProxy::NativeImageForCurrentTime(), Messages::RemoteMediaPlayerProxy::NativeImageForCurrentTime::Reply(sendRight, transform), m_id))
        return std::nullopt;

    if (!sendRight)
        return std::nullopt;

    auto surface = WebCore::IOSurface::createFromSendRight(WTFMove(*sendRight), WebCore::DestinationColorSpace::SRGB());
    if (!surface)
        return std::nullopt;

    auto platformImage = WebCore::IOSurface::sinkIntoImage(WTFMove(surface));
    if (!platformImage)
        return std::nullopt;

    auto image = NativeImage::create(WTFMove(platformImage));
    if (!image)
        return std::nullopt;
    return MediaPlayer::NativeImageFrame { image.releaseNonNull(), WTFMove(transform) };
}

std::optional<MediaPlayer::PixelBufferFrame> MediaPlayerPrivateRemote::pixelBufferForCurrentTime()
{

    RetainPtr<CVPixelBufferRef> result;
    WebCore::AffineTransform transform;
    if (!connection().sendSync(Messages::RemoteMediaPlayerProxy::PixelBufferForCurrentTime(), Messages::RemoteMediaPlayerProxy::PixelBufferForCurrentTime::Reply(result, transform), m_id))
        return std::nullopt;
    if (!result)
        return std::nullopt;
    return MediaPlayer::PixelBufferFrame { WTFMove(result), WTFMove(transform) };
}

} // namespace WebKit

Lines 73-84 RefPtr<ShareableBitmap> createShareableBitmap(RenderImage& renderImage, CreateSh a/Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp_sec1
- a/Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp -5 / +13 lines
73		73
74	if (is<RenderVideo>(renderImage)) {	74	if (is<RenderVideo>(renderImage)) {
75	auto& renderVideo = downcast<RenderVideo>(renderImage);	75	auto& renderVideo = downcast<RenderVideo>(renderImage);
76	Ref video = renderVideo.videoElement();	76	auto player = renderVideo.videoElement().player();
77	auto image = video->nativeImageForCurrentTime();	77	if (!player)
78	if (!image)	78	return { };
		79	auto frame = player->nativeImageForCurrentTime();
		80	if (!frame)
79	return { };	81	return { };
80		82
81	auto imageSize = image->size();	83	auto imageSize = frame->nativeImage->size();
82	if (imageSize.isEmpty() \|\| imageSize.width() <= 1 \|\| imageSize.height() <= 1)	84	if (imageSize.isEmpty() \|\| imageSize.width() <= 1 \|\| imageSize.height() <= 1)
83	return { };	85	return { };
84		86
Lines 90-96 RefPtr<ShareableBitmap> createShareableBitmap(RenderImage& renderImage, CreateSh a/Source/WebKit/WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp_sec2
90	if (!context)	92	if (!context)
91	return { };	93	return { };
92		94
93	context->drawNativeImage(*image, imageSize, FloatRect { { }, imageSize }, FloatRect { { }, imageSize });	95	GraphicsContextStateSaver stateSaver(*context);
		96	auto srcRect = FloatRect { { }, imageSize };
		97	// FIXME: Why do we map with the inverse but use the straight transform?
		98	// See MediaPlayerPrivateMediaStreamSoruceAVFObjC.mm.
		99	srcRect = frame->transform.inverse().value_or(AffineTransform { }).mapRect(srcRect);
		100	context->concatCTM(frame->transform);
		101	context->drawNativeImage(frame->nativeImage, imageSize, FloatRect { { }, imageSize }, srcRect);
94	return bitmap;	102	return bitmap;
95	}	103	}
96		104

Return to Bug 230617

Lines 308-321 bool HTMLVideoElement::hasAvailableVideoFrame() const a/Source/WebCore/html/HTMLVideoElement.cpp_sec1
- a/Source/WebCore/html/HTMLVideoElement.cpp -8 lines
308	return player()->hasVideo() && player()->hasAvailableVideoFrame();	308	return player()->hasVideo() && player()->hasAvailableVideoFrame();
309	}	309	}
310		310
311	RefPtr<NativeImage> HTMLVideoElement::nativeImageForCurrentTime()
312	{
313	if (!player())
314	return nullptr;
315
316	return player()->nativeImageForCurrentTime();
317	}
318
319	ExceptionOr<void> HTMLVideoElement::webkitEnterFullscreen()	311	ExceptionOr<void> HTMLVideoElement::webkitEnterFullscreen()
320	{	312	{
321	ALWAYS_LOG(LOGIDENTIFIER);	313	ALWAYS_LOG(LOGIDENTIFIER);

Lines 1677-1693 ExceptionOr<void> CanvasRenderingContext2DBase::drawImage(HTMLVideoElement& vide a/Source/WebCore/html/canvas/CanvasRenderingContext2DBase.cpp_sec1
- a/Source/WebCore/html/canvas/CanvasRenderingContext2DBase.cpp -13 / +21 lines
1677	checkOrigin(&video);	1677	checkOrigin(&video);
1678		1678
1679	#if USE(CG)	1679	#if USE(CG)
1680	if (auto image = video.nativeImageForCurrentTime()) {	1680	if (auto player = video.player()) {
1681	c->drawNativeImage(*image, FloatSize(video.videoWidth(), video.videoHeight()), dstRect, srcRect);	1681	if (auto frame = player->nativeImageForCurrentTime()) {
1682		1682	if (frame->transform.isIdentity()) {
1683	if (isEntireBackingStoreDirty())	1683	c->drawNativeImage(frame->nativeImage, FloatSize(video.videoWidth(), video.videoHeight()), dstRect, srcRect);
1684	didDraw(std::nullopt);	1684
1685	else if (rectContainsCanvas(dstRect))	1685	if (isEntireBackingStoreDirty())
1686	didDrawEntireCanvas();	1686	didDraw(std::nullopt);
1687	else	1687	else if (rectContainsCanvas(dstRect))
1688	didDraw(dstRect);	1688	didDrawEntireCanvas();
1689		1689	else
1690	return { };	1690	didDraw(dstRect);
		1691
		1692	return { };
		1693	}
		1694	}
1691	}	1695	}
1692	#endif	1696	#endif
1693		1697
Lines 2018-2025 ExceptionOr<RefPtr<CanvasPattern>> CanvasRenderingContext2DBase::createPattern(H a/Source/WebCore/html/canvas/CanvasRenderingContext2DBase.cpp_sec2
2018	bool originClean = canvasBase().originClean();	2022	bool originClean = canvasBase().originClean();
2019		2023
2020	#if USE(CG)	2024	#if USE(CG)
2021	if (auto nativeImage = videoElement.nativeImageForCurrentTime())	2025	if (auto player = videoElement.player()) {
2022	return RefPtr<CanvasPattern> { CanvasPattern::create(nativeImage.releaseNonNull(), repeatX, repeatY, originClean) };	2026	if (auto frame = player->nativeImageForCurrentTime()) {
		2027	if (frame->transform.isIdentity())
		2028	return RefPtr<CanvasPattern> { CanvasPattern::create(WTFMove(frame->nativeImage), repeatX, repeatY, originClean) };
		2029	}
		2030	}
2023	#endif	2031	#endif
2024		2032
2025	auto renderingMode = drawingContext() ? drawingContext()->renderingMode() : RenderingMode::Unaccelerated;	2033	auto renderingMode = drawingContext() ? drawingContext()->renderingMode() : RenderingMode::Unaccelerated;

Lines 1049-1062 bool MediaPlayer::copyVideoTextureToPlatformTexture(GraphicsContextGL* context, a/Source/WebCore/platform/graphics/MediaPlayer.cpp_sec1
- a/Source/WebCore/platform/graphics/MediaPlayer.cpp -2 / +2 lines
1049		1049
1050	#else	1050	#else
1051		1051
1052	RetainPtr<CVPixelBufferRef> MediaPlayer::pixelBufferForCurrentTime()	1052	std::optional<MediaPlayer::PixelBufferFrame> MediaPlayer::pixelBufferForCurrentTime()
1053	{	1053	{
1054	return m_private->pixelBufferForCurrentTime();	1054	return m_private->pixelBufferForCurrentTime();
1055	}	1055	}
1056		1056
1057	#endif	1057	#endif
1058		1058
1059	RefPtr<NativeImage> MediaPlayer::nativeImageForCurrentTime()	1059	std::optional<MediaPlayer::NativeImageFrame> MediaPlayer::nativeImageForCurrentTime()
1060	{	1060	{
1061	return m_private->nativeImageForCurrentTime();	1061	return m_private->nativeImageForCurrentTime();
1062	}	1062	}

Lines 174-182 public: a/Source/WebCore/platform/graphics/MediaPlayerPrivate.h_sec1
- a/Source/WebCore/platform/graphics/MediaPlayerPrivate.h -2 / +2 lines
174	#if !USE(AVFOUNDATION)	174	#if !USE(AVFOUNDATION)
175	virtual bool copyVideoTextureToPlatformTexture(GraphicsContextGL*, PlatformGLObject, GCGLenum, GCGLint, GCGLenum, GCGLenum, GCGLenum, bool, bool) { return false; }	175	virtual bool copyVideoTextureToPlatformTexture(GraphicsContextGL*, PlatformGLObject, GCGLenum, GCGLint, GCGLenum, GCGLenum, GCGLenum, bool, bool) { return false; }
176	#else	176	#else
177	virtual RetainPtr<CVPixelBufferRef> pixelBufferForCurrentTime() { return nullptr; }	177	virtual std::optional<MediaPlayer::PixelBufferFrame> pixelBufferForCurrentTime() { return std::nullopt; }
178	#endif	178	#endif
179	virtual RefPtr<NativeImage> nativeImageForCurrentTime() { return nullptr; }	179	virtual std::optional<MediaPlayer::NativeImageFrame> nativeImageForCurrentTime() { return std::nullopt; }
180		180
181	virtual void setPreload(MediaPlayer::Preload) { }	181	virtual void setPreload(MediaPlayer::Preload) { }
182		182

Lines 253-263 private: a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.h_sec1
- a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.h -3 / +2 lines
253	bool updateLastPixelBuffer();	253	bool updateLastPixelBuffer();
254	bool videoOutputHasAvailableFrame();	254	bool videoOutputHasAvailableFrame();
255	void paintWithVideoOutput(GraphicsContext&, const FloatRect&);	255	void paintWithVideoOutput(GraphicsContext&, const FloatRect&);
256	RefPtr<NativeImage> nativeImageForCurrentTime() final;	256	std::optional<MediaPlayer::PixelBufferFrame> pixelBufferForCurrentTime() final;
		257	std::optional<MediaPlayer::NativeImageFrame> nativeImageForCurrentTime() final;
257	void waitForVideoOutputMediaDataWillChange();	258	void waitForVideoOutputMediaDataWillChange();
258		259
259	RetainPtr<CVPixelBufferRef> pixelBufferForCurrentTime() final;
260
261	#if ENABLE(LEGACY_ENCRYPTED_MEDIA)	260	#if ENABLE(LEGACY_ENCRYPTED_MEDIA)
262	void keyAdded() final;	261	void keyAdded() final;
263	std::unique_ptr<LegacyCDMSession> createSession(const String& keySystem, LegacyCDMSessionClient*) final;	262	std::unique_ptr<LegacyCDMSession> createSession(const String& keySystem, LegacyCDMSessionClient*) final;

Lines 1-3 a/Source/WebCore/ChangeLog_sec1
- a/Source/WebCore/ChangeLog +64 lines
		1	2021-09-24 Kimmo Kinnunen <kkinnunen@apple.com>
		2
		3	REGRESSION (r280963): gl.texImage2D upload of getUserMedia streams via <video> element fails on iOS 15.1 beta 1
		4	https://bugs.webkit.org/show_bug.cgi?id=230617
		5	<rdar://problem/83407577>
		6
		7	Reviewed by NOBODY (OOPS!).
		8
		9	r280963 implemented the SW readback with the assumption that all video element sources
		10	would implement MediaPlayerPrivateInterface::nativeImageForCurrentTime().
		11	The MediaPlayerPrivateMediaStreamAVFObjC would not do so, so the
		12	texImage2D calls from videos with media streams attached would not produce
		13	results.
		14
		15	Fix this by implementing nativeImageForCurrentTime() and the related
		16	pixelBufferForCurrentTime(). This means that this implements the GPU codepath
		17	for media streams.
		18
		19	The underlying pixel buffers provided by the MediaPlayerPrivateMediaStreamAVFObjC
		20	have orientation, so add the transform to the return value of the
		21	nativeImageForCurrentTime() and pixelBufferForCurrentTime().
		22
		23	Tests to be added..
		24
		25	* html/HTMLVideoElement.cpp:
		26	* html/HTMLVideoElement.h:
		27	* html/canvas/CanvasRenderingContext2DBase.cpp:
		28	(WebCore::CanvasRenderingContext2DBase::drawImage):
		29	(WebCore::CanvasRenderingContext2DBase::createPattern):
		30	* html/canvas/WebGLRenderingContextBase.cpp:
		31	(WebCore::WebGLRenderingContextBase::videoFrameToImage):
		32	* platform/graphics/MediaPlayer.cpp:
		33	(WebCore::MediaPlayer::pixelBufferForCurrentTime):
		34	(WebCore::MediaPlayer::nativeImageForCurrentTime):
		35	* platform/graphics/MediaPlayer.h:
		36	* platform/graphics/MediaPlayerPrivate.h:
		37	(WebCore::MediaPlayerPrivateInterface::pixelBufferForCurrentTime):
		38	(WebCore::MediaPlayerPrivateInterface::nativeImageForCurrentTime):
		39	* platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.h:
		40	* platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:
		41	(WebCore::MediaPlayerPrivateAVFoundationObjC::pixelBufferForCurrentTime):
		42	(WebCore::MediaPlayerPrivateAVFoundationObjC::nativeImageForCurrentTime):
		43	* platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.h:
		44	* platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm:
		45	(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::nativeImageForCurrentTime):
		46	(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::paintCurrentFrameInContext):
		47	(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::pixelBufferForCurrentTime):
		48	* platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.h:
		49	* platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm:
		50	(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::paintCurrentFrameInContext):
		51	(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::nativeImageForCurrentTime):
		52	(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::pixelBufferForCurrentTime):
		53	(WebCore::MediaPlayerPrivateMediaStreamAVFObjC::transformForCurrentSample):
		54	* platform/graphics/cv/GraphicsContextGLCV.h:
		55	* platform/graphics/cv/GraphicsContextGLCVANGLE.cpp:
		56	(WebCore::GraphicsContextGLCVANGLE::initializeUVContextObjects):
		57	(WebCore::GraphicsContextGLCVANGLE::copyPixelBufferToTexture):
		58	* platform/graphics/cv/GraphicsContextGLCVANGLE.h:
		59	* platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:
		60	(WebCore::MediaPlayerPrivateGStreamer::nativeImageForCurrentTime):
		61	* platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.h:
		62	* platform/graphics/opengl/GraphicsContextGLOpenGL.cpp:
		63	(WebCore::GraphicsContextGLOpenGL::copyTextureFromMedia):
		64
1	2021-09-22 Kimmo Kinnunen <kkinnunen@apple.com>	65	2021-09-22 Kimmo Kinnunen <kkinnunen@apple.com>
2		66
3	MediaStream canvas.captureStream() fails for WebGL	67	MediaStream canvas.captureStream() fails for WebGL

Lines 1-3 a/Source/WebKit/ChangeLog_sec1
- a/Source/WebKit/ChangeLog +30 lines
		1	2021-09-24 Kimmo Kinnunen <kkinnunen@apple.com>
		2
		3	REGRESSION (r280963): gl.texImage2D upload of getUserMedia streams via <video> element fails on iOS 15.1 beta 1
		4	https://bugs.webkit.org/show_bug.cgi?id=230617
		5	<rdar://problem/83407577>
		6
		7	Reviewed by NOBODY (OOPS!).
		8
		9	Add the ability to pass transform of the pixels along with the
		10	pixel buffers from nativeImageForCurrentTime and pixelBufferForCurrentTime
		11	in GPU process objects.
		12
		13	* GPUProcess/graphics/RemoteGraphicsContextGL.cpp:
		14	(WebKit::RemoteGraphicsContextGL::copyTextureFromMedia):
		15	* GPUProcess/media/RemoteMediaPlayerProxy.h:
		16	* GPUProcess/media/RemoteMediaPlayerProxy.messages.in:
		17	* GPUProcess/media/cocoa/RemoteMediaPlayerProxyCocoa.mm:
		18	(WebKit::RemoteMediaPlayerProxy::nativeImageForCurrentTime):
		19	(WebKit::RemoteMediaPlayerProxy::pixelBufferForCurrentTime):
		20	* WebProcess/GPU/media/MediaPlayerPrivateRemote.cpp:
		21	(WebKit::MediaPlayerPrivateRemote::paintCurrentFrameInContext):
		22	(WebKit::MediaPlayerPrivateRemote::pixelBufferForCurrentTime):
		23	(WebKit::MediaPlayerPrivateRemote::nativeImageForCurrentTime):
		24	* WebProcess/GPU/media/MediaPlayerPrivateRemote.h:
		25	* WebProcess/GPU/media/cocoa/MediaPlayerPrivateRemoteCocoa.mm:
		26	(WebKit::MediaPlayerPrivateRemote::nativeImageForCurrentTime):
		27	(WebKit::MediaPlayerPrivateRemote::pixelBufferForCurrentTime):
		28	* WebProcess/WebCoreSupport/ShareableBitmapUtilities.cpp:
		29	(WebKit::createShareableBitmap):
		30
1	2021-09-16 Kimmo Kinnunen <kkinnunen@apple.com>	31	2021-09-16 Kimmo Kinnunen <kkinnunen@apple.com>
2		32
3	Add utility to create CVPixelBuffers from IOSurfaces	33	Add utility to create CVPixelBuffers from IOSurfaces

Lines 77-84 public: a/Source/WebCore/html/HTMLVideoElement.h_sec1
- a/Source/WebCore/html/HTMLVideoElement.h -2 lines
77	// Used by canvas to gain raw pixel access	77	// Used by canvas to gain raw pixel access
78	void paintCurrentFrameInContext(GraphicsContext&, const FloatRect&);	78	void paintCurrentFrameInContext(GraphicsContext&, const FloatRect&);
79		79
80	WEBCORE_EXPORT RefPtr<NativeImage> nativeImageForCurrentTime();
81
82	WEBCORE_EXPORT bool shouldDisplayPosterImage() const;	80	WEBCORE_EXPORT bool shouldDisplayPosterImage() const;
83		81
84	URL posterImageURL() const;	82	URL posterImageURL() const;

Lines 27-32 a/Source/WebCore/platform/graphics/MediaPlayer.h_sec1
- a/Source/WebCore/platform/graphics/MediaPlayer.h -4 / +13 lines
27		27
28	#if ENABLE(VIDEO)	28	#if ENABLE(VIDEO)
29		29
		30	#include "AffineTransform.h"
30	#include "AudioTrackPrivate.h"	31	#include "AudioTrackPrivate.h"
31	#include "ContentType.h"	32	#include "ContentType.h"
32	#include "Cookie.h"	33	#include "Cookie.h"
Lines 46-51 a/Source/WebCore/platform/graphics/MediaPlayer.h_sec2
46	#include <wtf/URL.h>	47	#include <wtf/URL.h>
47	#include "VideoTrackPrivate.h"	48	#include "VideoTrackPrivate.h"
48	#include <JavaScriptCore/Uint8Array.h>	49	#include <JavaScriptCore/Uint8Array.h>
		50	#include <optional>
49	#include <wtf/CompletionHandler.h>	51	#include <wtf/CompletionHandler.h>
50	#include <wtf/Function.h>	52	#include <wtf/Function.h>
51	#include <wtf/HashSet.h>	53	#include <wtf/HashSet.h>
Lines 466-475 public: a/Source/WebCore/platform/graphics/MediaPlayer.h_sec3
466	#if !USE(AVFOUNDATION)	468	#if !USE(AVFOUNDATION)
467	bool copyVideoTextureToPlatformTexture(GraphicsContextGL*, PlatformGLObject texture, GCGLenum target, GCGLint level, GCGLenum internalFormat, GCGLenum format, GCGLenum type, bool premultiplyAlpha, bool flipY);	469	bool copyVideoTextureToPlatformTexture(GraphicsContextGL*, PlatformGLObject texture, GCGLenum target, GCGLint level, GCGLenum internalFormat, GCGLenum format, GCGLenum type, bool premultiplyAlpha, bool flipY);
468	#else	470	#else
469	RetainPtr<CVPixelBufferRef> pixelBufferForCurrentTime();	471	struct PixelBufferFrame {
470	#endif	472	RetainPtr<CVPixelBufferRef> pixelBuffer;
471		473	AffineTransform transform;
472	RefPtr<NativeImage> nativeImageForCurrentTime();	474	};
		475	std::optional<PixelBufferFrame> pixelBufferForCurrentTime();
		476	#endif
		477	struct NativeImageFrame {
		478	Ref<NativeImage> nativeImage;
		479	AffineTransform transform;
		480	};
		481	std::optional<NativeImageFrame> nativeImageForCurrentTime();
473		482
474	using MediaPlayerEnums::NetworkState;	483	using MediaPlayerEnums::NetworkState;
475	NetworkState networkState();	484	NetworkState networkState();

Lines 2544-2562 void MediaPlayerPrivateAVFoundationObjC::paintWithVideoOutput(GraphicsContext& c a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm_sec1
- a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm -6 / +7 lines
2544		2544
2545	}	2545	}
2546		2546
2547	RetainPtr<CVPixelBufferRef> MediaPlayerPrivateAVFoundationObjC::pixelBufferForCurrentTime()	2547	std::optional<MediaPlayer::PixelBufferFrame> MediaPlayerPrivateAVFoundationObjC::pixelBufferForCurrentTime()
2548	{	2548	{
2549	updateLastPixelBuffer();	2549	updateLastPixelBuffer();
2550	if (!m_lastPixelBuffer)	2550	if (!m_lastPixelBuffer)
2551	return nullptr;	2551	return std::nullopt;
2552		2552	return MediaPlayer::PixelBufferFrame { m_lastPixelBuffer, { } };
2553	return m_lastPixelBuffer;
2554	}	2553	}
2555		2554
2556	RefPtr<NativeImage> MediaPlayerPrivateAVFoundationObjC::nativeImageForCurrentTime()	2555	std::optional<MediaPlayer::NativeImageFrame> MediaPlayerPrivateAVFoundationObjC::nativeImageForCurrentTime()
2557	{	2556	{
2558	updateLastImage(UpdateType::UpdateSynchronously);	2557	updateLastImage(UpdateType::UpdateSynchronously);
2559	return m_lastImage;	2558	if (m_lastImage)
		2559	return std::nullopt;
		2560	return MediaPlayer::NativeImageFrame { *m_lastImage, { } };
2560	}	2561	}
2561		2562
2562	void MediaPlayerPrivateAVFoundationObjC::waitForVideoOutputMediaDataWillChange()	2563	void MediaPlayerPrivateAVFoundationObjC::waitForVideoOutputMediaDataWillChange()

Lines 216-227 private: a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.h_sec1
- a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.h -2 / +2 lines
216		216
217	bool didLoadingProgress() const override;	217	bool didLoadingProgress() const override;
218		218
219	RefPtr<NativeImage> nativeImageForCurrentTime() override;	219	std::optional<MediaPlayer::NativeImageFrame> nativeImageForCurrentTime() override;
220	bool updateLastPixelBuffer();	220	bool updateLastPixelBuffer();
221	bool updateLastImage();	221	bool updateLastImage();
222	void paint(GraphicsContext&, const FloatRect&) override;	222	void paint(GraphicsContext&, const FloatRect&) override;
223	void paintCurrentFrameInContext(GraphicsContext&, const FloatRect&) override;	223	void paintCurrentFrameInContext(GraphicsContext&, const FloatRect&) override;
224	RetainPtr<CVPixelBufferRef> pixelBufferForCurrentTime() final;	224	std::optional<MediaPlayer::PixelBufferFrame> pixelBufferForCurrentTime() final;
225		225
226	bool supportsAcceleratedRendering() const override;	226	bool supportsAcceleratedRendering() const override;
227	// called when the rendering system flips the into or out of accelerated rendering mode.	227	// called when the rendering system flips the into or out of accelerated rendering mode.

Lines 647-656 bool MediaPlayerPrivateMediaSourceAVFObjC::didLoadingProgress() const a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm_sec1
- a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm -11 / +11 lines
647	return loadingProgressed;	647	return loadingProgressed;
648	}	648	}
649		649
650	RefPtr<NativeImage> MediaPlayerPrivateMediaSourceAVFObjC::nativeImageForCurrentTime()	650	std::optional<MediaPlayer::NativeImageFrame> MediaPlayerPrivateMediaSourceAVFObjC::nativeImageForCurrentTime()
651	{	651	{
652	updateLastImage();	652	updateLastImage();
653	return m_lastImage;	653	if (!m_lastImage)
		654	return std::nullopt;
		655	return MediaPlayer::NativeImageFrame { *m_lastImage, { } };
654	}	656	}
655		657
656	bool MediaPlayerPrivateMediaSourceAVFObjC::updateLastPixelBuffer()	658	bool MediaPlayerPrivateMediaSourceAVFObjC::updateLastPixelBuffer()
Lines 709-719 void MediaPlayerPrivateMediaSourceAVFObjC::paintCurrentFrameInContext(GraphicsCo a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm_sec2
709	return;	711	return;
710		712
711	GraphicsContextStateSaver stateSaver(context);	713	GraphicsContextStateSaver stateSaver(context);
712	FloatRect imageRect { FloatPoint::zero(), image->size() };	714	FloatRect imageRect { FloatPoint::zero(), image->nativeImage->size() };
713	context.drawNativeImage(*image, imageRect.size(), outputRect, imageRect);	715	context.drawNativeImage(image->nativeImage, imageRect.size(), outputRect, imageRect);
714	}	716	}
715		717
716	RetainPtr<CVPixelBufferRef> MediaPlayerPrivateMediaSourceAVFObjC::pixelBufferForCurrentTime()	718	std::optional<MediaPlayer::PixelBufferFrame> MediaPlayerPrivateMediaSourceAVFObjC::pixelBufferForCurrentTime()
717	{	719	{
718	// We have been asked to paint into a WebGL canvas, so take that as a signal to create	720	// We have been asked to paint into a WebGL canvas, so take that as a signal to create
719	// a decompression session, even if that means the native video can't also be displayed	721	// a decompression session, even if that means the native video can't also be displayed
Lines 723-734 RetainPtr<CVPixelBufferRef> MediaPlayerPrivateMediaSourceAVFObjC::pixelBufferFor a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm_sec3
723	acceleratedRenderingStateChanged();	725	acceleratedRenderingStateChanged();
724	}	726	}
725		727
726	if (updateLastPixelBuffer()) {	728	updateLastPixelBuffer();
727	if (!m_lastPixelBuffer)	729	if (!m_lastPixelBuffer)
728	return nullptr;	730	return std::nullopt;
729	}	731	return MediaPlayer::PixelBufferFrame { m_lastPixelBuffer, { } };
730
731	return m_lastPixelBuffer;
732	}	732	}
733		733
734	bool MediaPlayerPrivateMediaSourceAVFObjC::hasAvailableVideoFrame() const	734	bool MediaPlayerPrivateMediaSourceAVFObjC::hasAvailableVideoFrame() const

Lines 228-233 MediaPlayer::SupportsType MediaPlayerPrivateMediaStreamAVFObjC::supportsType(con a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm_sec1
- a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm -5 / +50 lines
228	#pragma mark -	228	#pragma mark -
229	#pragma mark AVSampleBuffer Methods	229	#pragma mark AVSampleBuffer Methods
230		230
		231	// Returns transform matrix for video when 0.5, 0.5 anchor point is used.
231	static inline CGAffineTransform videoTransformationMatrix(MediaSample& sample)	232	static inline CGAffineTransform videoTransformationMatrix(MediaSample& sample)
232	{	233	{
233	CMSampleBufferRef sampleBuffer = sample.platformSample().sample.cmSampleBuffer;	234	CMSampleBufferRef sampleBuffer = sample.platformSample().sample.cmSampleBuffer;
Lines 985-1006 void MediaPlayerPrivateMediaStreamAVFObjC::paintCurrentFrameInContext(GraphicsCo a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm_sec2
985	if (m_displayMode == None \|\| !metaDataAvailable() \|\| context.paintingDisabled())	986	if (m_displayMode == None \|\| !metaDataAvailable() \|\| context.paintingDisabled())
986	return;	987	return;
987		988
988	if (m_displayMode != PaintItBlack && m_imagePainter.mediaSample)
989	updateCurrentFrameImage();
990
991	GraphicsContextStateSaver stateSaver(context);
992	if (m_displayMode == PaintItBlack) {	989	if (m_displayMode == PaintItBlack) {
993	context.fillRect(IntRect(IntPoint(), IntSize(destRect.width(), destRect.height())), Color::black);	990	context.fillRect(IntRect(IntPoint(), IntSize(destRect.width(), destRect.height())), Color::black);
994	return;	991	return;
995	}	992	}
996		993
		994	updateCurrentFrameImage();
		995
997	if (!m_imagePainter.cgImage \|\| !m_imagePainter.mediaSample)	996	if (!m_imagePainter.cgImage \|\| !m_imagePainter.mediaSample)
998	return;	997	return;
999		998	GraphicsContextStateSaver stateSaver(context);
1000	auto& image = m_imagePainter.cgImage;	999	auto& image = m_imagePainter.cgImage;
1001	FloatRect imageRect { FloatPoint::zero(), image->size() };	1000	FloatRect imageRect { FloatPoint::zero(), image->size() };
1002	if (!m_videoTransform)	1001	if (!m_videoTransform)
1003	m_videoTransform = videoTransformationMatrix(*m_imagePainter.mediaSample);	1002	m_videoTransform = videoTransformationMatrix(*m_imagePainter.mediaSample);
		1003	// FIXME: Why is there a inverse() here?
1004	AffineTransform videoTransform = *m_videoTransform;	1004	AffineTransform videoTransform = *m_videoTransform;
1005	FloatRect transformedDestRect = videoTransform.inverse().value_or(AffineTransform()).mapRect(destRect);	1005	FloatRect transformedDestRect = videoTransform.inverse().value_or(AffineTransform()).mapRect(destRect);
1006	context.concatCTM(videoTransform);	1006	context.concatCTM(videoTransform);
Lines 1088-1093 WTFLogChannel& MediaPlayerPrivateMediaStreamAVFObjC::logChannel() const a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm_sec3
1088	return LogMedia;	1088	return LogMedia;
1089	}	1089	}
1090		1090
		1091	std::optional<MediaPlayer::NativeImageFrame> MediaPlayerPrivateMediaStreamAVFObjC::nativeImageForCurrentTime()
		1092	{
		1093	if (m_displayMode == None \|\| !metaDataAvailable())
		1094	return std::nullopt;
		1095	// FIXME: For accelerated operation, we could return a black pixel here.
		1096	if (m_displayMode == PaintItBlack)
		1097	return std::nullopt;
		1098	updateCurrentFrameImage();
		1099	if (!m_imagePainter.cgImage \|\| !m_imagePainter.mediaSample)
		1100	return std::nullopt;
		1101	return MediaPlayer::NativeImageFrame { *m_imagePainter.cgImage, transformForCurrentSample() };
		1102	}
		1103
		1104	std::optional<MediaPlayer::PixelBufferFrame> MediaPlayerPrivateMediaStreamAVFObjC::pixelBufferForCurrentTime()
		1105	{
		1106	if (m_displayMode == None \|\| !metaDataAvailable())
		1107	return std::nullopt;
		1108	// FIXME: For accelerated operation, we could return a black pixel here.
		1109	if (m_displayMode == PaintItBlack)
		1110	return std::nullopt;
		1111	if (!m_imagePainter.mediaSample)
		1112	return std::nullopt;
		1113	RetainPtr pixelBuffer = static_cast<CVPixelBufferRef>(PAL::CMSampleBufferGetImageBuffer(m_imagePainter.mediaSample->platformSample().sample.cmSampleBuffer));
		1114	if (!pixelBuffer)
		1115	return std::nullopt;
		1116	return MediaPlayer::PixelBufferFrame { WTFMove(pixelBuffer), transformForCurrentSample() };
		1117	}
		1118
		1119	AffineTransform MediaPlayerPrivateMediaStreamAVFObjC::transformForCurrentSample()
		1120	{
		1121	ASSERT(m_imagePainter.mediaSample);
		1122	// FIXME: why is this reset this way during enqueue, and not just obtained per
		1123	// sample?
		1124	if (!m_videoTransform)
		1125	m_videoTransform = videoTransformationMatrix(*m_imagePainter.mediaSample);
		1126	if (auto transform = AffineTransform(*m_videoTransform).inverse()) {
		1127	AffineTransform result;
		1128	result.translate(0.5, 0.5);
		1129	result.multiply(*m_videoTransform);
		1130	result.translate(-0.5, -0.5);
		1131	return result;
		1132	}
		1133	return { };
		1134	}
		1135
1091	}	1136	}
1092		1137
1093	#endif	1138	#endif

Lines 27-32 a/Source/WebCore/platform/graphics/cv/GraphicsContextGLCV.h_sec1
- a/Source/WebCore/platform/graphics/cv/GraphicsContextGLCV.h -2 / +3 lines
27		27
28	#if ENABLE(WEBGL) && USE(AVFOUNDATION)	28	#if ENABLE(WEBGL) && USE(AVFOUNDATION)
29		29
		30	#include "AffineTransform.h"
30	#include "GraphicsContextGL.h"	31	#include "GraphicsContextGL.h"
31		32
32	typedef struct __CVBuffer* CVPixelBufferRef;	33	typedef struct __CVBuffer* CVPixelBufferRef;
Lines 39-47 public: a/Source/WebCore/platform/graphics/cv/GraphicsContextGLCV.h_sec2
39	virtual ~GraphicsContextGLCV() = default;	40	virtual ~GraphicsContextGLCV() = default;
40		41
41	using FlipY = GraphicsContextGL::FlipY;	42	using FlipY = GraphicsContextGL::FlipY;
42	// Copies CV pixel buffer to GraphicsContextGL TEXTURE_2D texture of the same size as the pixel buffer.	43	// Copies CV pixel buffer to GraphicsContextGL TEXTURE_2D texture of the same size as the pixel buffer with the transform applied.
43	// Returns true on success.	44	// Returns true on success.
44	virtual bool copyPixelBufferToTexture(CVPixelBufferRef, PlatformGLObject outputTexture, GCGLint level, GCGLenum internalFormat, GCGLenum format, GCGLenum type, FlipY) = 0;	45	virtual bool copyPixelBufferToTexture(CVPixelBufferRef, AffineTransform, PlatformGLObject outputTexture, GCGLint level, GCGLenum internalFormat, GCGLenum format, GCGLenum type, FlipY) = 0;
45	};	46	};
46		47
47	}	48	}

Lines 46-58 static constexpr auto s_yuvVertexShaderTexture2D { a/Source/WebCore/platform/graphics/cv/GraphicsContextGLCVANGLE.cpp_sec1
- a/Source/WebCore/platform/graphics/cv/GraphicsContextGLCVANGLE.cpp -7 / +24 lines
46	"attribute vec2 a_position;"	46	"attribute vec2 a_position;"
47	"uniform vec2 u_yTextureSize;"	47	"uniform vec2 u_yTextureSize;"
48	"uniform vec2 u_uvTextureSize;"	48	"uniform vec2 u_uvTextureSize;"
		49	"uniform mat3 u_transform;"
49	"uniform int u_flipY;"	50	"uniform int u_flipY;"
50	"varying vec2 v_yTextureCoordinate;"	51	"varying vec2 v_yTextureCoordinate;"
51	"varying vec2 v_uvTextureCoordinate;"	52	"varying vec2 v_uvTextureCoordinate;"
52	"void main()"	53	"void main()"
53	"{"	54	"{"
54	" gl_Position = vec4(a_position, 0, 1.0);"	55	" gl_Position = vec4(a_position, 0, 1.0);"
55	" vec2 normalizedPosition = a_position * .5 + .5;"	56	" vec2 normalizedPosition = (u_transform * vec3(a_position * .5 + .5, 1.0)).xy;"
56	" if (u_flipY == 1)"	57	" if (u_flipY == 1)"
57	" normalizedPosition.y = 1.0 - normalizedPosition.y;"	58	" normalizedPosition.y = 1.0 - normalizedPosition.y;"
58	" v_yTextureCoordinate = normalizedPosition;"	59	" v_yTextureCoordinate = normalizedPosition;"
Lines 64-76 static constexpr auto s_yuvVertexShaderTextureRectangle { a/Source/WebCore/platform/graphics/cv/GraphicsContextGLCVANGLE.cpp_sec2
64	"attribute vec2 a_position;"	65	"attribute vec2 a_position;"
65	"uniform vec2 u_yTextureSize;"	66	"uniform vec2 u_yTextureSize;"
66	"uniform vec2 u_uvTextureSize;"	67	"uniform vec2 u_uvTextureSize;"
		68	"uniform mat3 u_transform;"
67	"uniform int u_flipY;"	69	"uniform int u_flipY;"
68	"varying vec2 v_yTextureCoordinate;"	70	"varying vec2 v_yTextureCoordinate;"
69	"varying vec2 v_uvTextureCoordinate;"	71	"varying vec2 v_uvTextureCoordinate;"
70	"void main()"	72	"void main()"
71	"{"	73	"{"
72	" gl_Position = vec4(a_position, 0, 1.0);"	74	" gl_Position = vec4(a_position, 0, 1.0);"
73	" vec2 normalizedPosition = a_position * .5 + .5;"	75	" vec2 normalizedPosition = (u_transform * vec3(a_position * .5 + .5, 1.0)).xy;"
74	" if (u_flipY == 1)"	76	" if (u_flipY == 1)"
75	" normalizedPosition.y = 1.0 - normalizedPosition.y;"	77	" normalizedPosition.y = 1.0 - normalizedPosition.y;"
76	" v_yTextureCoordinate = normalizedPosition * u_yTextureSize;"	78	" v_yTextureCoordinate = normalizedPosition * u_yTextureSize;"
Lines 523-528 bool GraphicsContextGLCVANGLE::initializeUVContextObjects() a/Source/WebCore/platform/graphics/cv/GraphicsContextGLCVANGLE.cpp_sec3
523	m_yTextureUniformLocation = m_context->getUniformLocation(m_yuvProgram, "u_yTexture"_s);	525	m_yTextureUniformLocation = m_context->getUniformLocation(m_yuvProgram, "u_yTexture"_s);
524	m_uvTextureUniformLocation = m_context->getUniformLocation(m_yuvProgram, "u_uvTexture"_s);	526	m_uvTextureUniformLocation = m_context->getUniformLocation(m_yuvProgram, "u_uvTexture"_s);
525	m_colorMatrixUniformLocation = m_context->getUniformLocation(m_yuvProgram, "u_colorMatrix"_s);	527	m_colorMatrixUniformLocation = m_context->getUniformLocation(m_yuvProgram, "u_colorMatrix"_s);
		528	m_transformUniformLocation = m_context->getUniformLocation(m_yuvProgram, "u_transform"_s);
526	m_yuvFlipYUniformLocation = m_context->getUniformLocation(m_yuvProgram, "u_flipY"_s);	529	m_yuvFlipYUniformLocation = m_context->getUniformLocation(m_yuvProgram, "u_flipY"_s);
527	m_yTextureSizeUniformLocation = m_context->getUniformLocation(m_yuvProgram, "u_yTextureSize"_s);	530	m_yTextureSizeUniformLocation = m_context->getUniformLocation(m_yuvProgram, "u_yTextureSize"_s);
528	m_uvTextureSizeUniformLocation = m_context->getUniformLocation(m_yuvProgram, "u_uvTextureSize"_s);	531	m_uvTextureSizeUniformLocation = m_context->getUniformLocation(m_yuvProgram, "u_uvTextureSize"_s);
Lines 591-597 void GraphicsContextGLCVANGLE::detachIOSurfaceFromTexture(void* handle) a/Source/WebCore/platform/graphics/cv/GraphicsContextGLCVANGLE.cpp_sec4
591	EGL_DestroySurface(display, handle);	594	EGL_DestroySurface(display, handle);
592	}	595	}
593		596
594	bool GraphicsContextGLCVANGLE::copyPixelBufferToTexture(CVPixelBufferRef image, PlatformGLObject outputTexture, GCGLint level, GCGLenum internalFormat, GCGLenum format, GCGLenum type, FlipY flipY)	597	bool GraphicsContextGLCVANGLE::copyPixelBufferToTexture(CVPixelBufferRef image, AffineTransform transform, PlatformGLObject outputTexture, GCGLint level, GCGLenum internalFormat, GCGLenum format, GCGLenum type, FlipY flipY)
595	{	598	{
596	// FIXME: This currently only supports '420v' and '420f' pixel formats. Investigate supporting more pixel formats.	599	// FIXME: This currently only supports '420v' and '420f' pixel formats. Investigate supporting more pixel formats.
597	OSType pixelFormat = CVPixelBufferGetPixelFormatType(image);	600	OSType pixelFormat = CVPixelBufferGetPixelFormatType(image);
Lines 608-614 bool GraphicsContextGLCVANGLE::copyPixelBufferToTexture(CVPixelBufferRef image, a/Source/WebCore/platform/graphics/cv/GraphicsContextGLCVANGLE.cpp_sec5
608	IOSurfaceRef surface = CVPixelBufferGetIOSurface(image);	611	IOSurfaceRef surface = CVPixelBufferGetIOSurface(image);
609	if (!surface)	612	if (!surface)
610	return false;	613	return false;
611		614	transform = *transform.inverse();
612	auto newSurfaceSeed = IOSurfaceGetSeed(surface);	615	auto newSurfaceSeed = IOSurfaceGetSeed(surface);
613	if (flipY == m_lastFlipY	616	if (flipY == m_lastFlipY
614	&& surface == m_lastSurface	617	&& surface == m_lastSurface
Lines 618-632 bool GraphicsContextGLCVANGLE::copyPixelBufferToTexture(CVPixelBufferRef image, a/Source/WebCore/platform/graphics/cv/GraphicsContextGLCVANGLE.cpp_sec6
618	// image hasn't been modified since the last time it was copied, this is a no-op.	621	// image hasn't been modified since the last time it was copied, this is a no-op.
619	return true;	622	return true;
620	}	623	}
621
622	if (!m_yuvProgram) {	624	if (!m_yuvProgram) {
623	if (!initializeUVContextObjects()) {	625	if (!initializeUVContextObjects()) {
624	LOG(WebGL, "GraphicsContextGLCVANGLE::copyVideoTextureToPlatformTexture(%p) - Unable to initialize OpenGL context objects.", this);	626	LOG(WebGL, "GraphicsContextGLCVANGLE::copyVideoTextureToPlatformTexture(%p) - Unable to initialize OpenGL context objects.", this);
625	return false;	627	return false;
626	}	628	}
627	}	629	}
628	size_t width = CVPixelBufferGetWidth(image);	630	IntRect imageRect { { }, { static_cast<int>(CVPixelBufferGetWidth(image)), static_cast<int>(CVPixelBufferGetHeight(image)) } };
629	size_t height = CVPixelBufferGetHeight(image);	631	auto outputRect = transform.mapRect(imageRect);
		632
		633	size_t width = outputRect.width();
		634	size_t height = outputRect.height();
630		635
631	m_context->bindFramebuffer(GraphicsContextGL::FRAMEBUFFER, m_framebuffer);	636	m_context->bindFramebuffer(GraphicsContextGL::FRAMEBUFFER, m_framebuffer);
632		637
Lines 704-709 bool GraphicsContextGLCVANGLE::copyPixelBufferToTexture(CVPixelBufferRef image, a/Source/WebCore/platform/graphics/cv/GraphicsContextGLCVANGLE.cpp_sec7
704	auto transferFunction = transferFunctionFromString(dynamic_cf_cast<CFStringRef>(CVBufferGetAttachment(image, kCVImageBufferYCbCrMatrixKey, nil)));	709	auto transferFunction = transferFunctionFromString(dynamic_cf_cast<CFStringRef>(CVBufferGetAttachment(image, kCVImageBufferYCbCrMatrixKey, nil)));
705	auto colorMatrix = YCbCrToRGBMatrixForRangeAndTransferFunction(range, transferFunction);	710	auto colorMatrix = YCbCrToRGBMatrixForRangeAndTransferFunction(range, transferFunction);
706	m_context->uniformMatrix4fv(m_colorMatrixUniformLocation, GL_FALSE, colorMatrix);	711	m_context->uniformMatrix4fv(m_colorMatrixUniformLocation, GL_FALSE, colorMatrix);
		712	GLfloat transformMatrix[] = {
		713	static_cast<GLfloat>(transform.a()),
		714	static_cast<GLfloat>(transform.b()),
		715	0.f,
		716	static_cast<GLfloat>(transform.c()),
		717	static_cast<GLfloat>(transform.d()),
		718	0.f,
		719	static_cast<GLfloat>(transform.e()),
		720	static_cast<GLfloat>(transform.f()),
		721	1.f
		722	};
		723	m_context->uniformMatrix3fv(m_transformUniformLocation, GL_FALSE, transformMatrix);
707		724
708	// Do the actual drawing.	725	// Do the actual drawing.
709	m_context->drawArrays(GraphicsContextGL::TRIANGLES, 0, 6);	726	m_context->drawArrays(GraphicsContextGL::TRIANGLES, 0, 6);

Lines 41-47 public: a/Source/WebCore/platform/graphics/cv/GraphicsContextGLCVANGLE.h_sec1
- a/Source/WebCore/platform/graphics/cv/GraphicsContextGLCVANGLE.h -1 / +2 lines
41	GraphicsContextGLCVANGLE(GraphicsContextGLOpenGL&);	41	GraphicsContextGLCVANGLE(GraphicsContextGLOpenGL&);
42	~GraphicsContextGLCVANGLE() final;	42	~GraphicsContextGLCVANGLE() final;
43		43
44	bool copyPixelBufferToTexture(CVPixelBufferRef, PlatformGLObject outputTexture, GCGLint level, GCGLenum internalFormat, GCGLenum format, GCGLenum type, FlipY) final;	44	bool copyPixelBufferToTexture(CVPixelBufferRef, AffineTransform, PlatformGLObject outputTexture, GCGLint level, GCGLenum internalFormat, GCGLenum format, GCGLenum type, FlipY) final;
45		45
46	private:	46	private:
47	bool initializeUVContextObjects();	47	bool initializeUVContextObjects();
Lines 65-70 private: a/Source/WebCore/platform/graphics/cv/GraphicsContextGLCVANGLE.h_sec2
65	GCGLint m_uvTextureUniformLocation { -1 };	65	GCGLint m_uvTextureUniformLocation { -1 };
66	GCGLint m_yuvFlipYUniformLocation { -1 };	66	GCGLint m_yuvFlipYUniformLocation { -1 };
67	GCGLint m_colorMatrixUniformLocation { -1 };	67	GCGLint m_colorMatrixUniformLocation { -1 };
		68	GCGLint m_transformUniformLocation { -1 };
68	GCGLint m_yuvPositionAttributeLocation { -1 };	69	GCGLint m_yuvPositionAttributeLocation { -1 };
69	GCGLint m_yTextureSizeUniformLocation { -1 };	70	GCGLint m_yTextureSizeUniformLocation { -1 };
70	GCGLint m_uvTextureSizeUniformLocation { -1 };	71	GCGLint m_uvTextureSizeUniformLocation { -1 };

Lines 3293-3301 bool MediaPlayerPrivateGStreamer::copyVideoTextureToPlatformTexture(GraphicsCont a/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp_sec1
- a/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp -2 / +2 lines
3293	return m_videoTextureCopier->copyVideoTextureToPlatformTexture(*layerBuffer.get(), size, outputTexture, outputTarget, level, internalFormat, format, type, flipY, m_videoSourceOrientation, premultiplyAlpha);	3293	return m_videoTextureCopier->copyVideoTextureToPlatformTexture(*layerBuffer.get(), size, outputTexture, outputTarget, level, internalFormat, format, type, flipY, m_videoSourceOrientation, premultiplyAlpha);
3294	}	3294	}
3295		3295
3296	RefPtr<NativeImage> MediaPlayerPrivateGStreamer::nativeImageForCurrentTime()	3296	std::optional<MediaPlayer::NativeImageFrame> MediaPlayerPrivateGStreamer::nativeImageForCurrentTime()
3297	{	3297	{
3298	return nullptr;	3298	return std::nullopt;
3299	}	3299	}
3300	#endif // USE(GSTREAMER_GL)	3300	#endif // USE(GSTREAMER_GL)
3301		3301

Lines 198-205 std::optional<PixelBuffer> GraphicsContextGLOpenGL::readCompositedResults() a/Source/WebCore/platform/graphics/opengl/GraphicsContextGLOpenGL.cpp_sec1
- a/Source/WebCore/platform/graphics/opengl/GraphicsContextGLOpenGL.cpp -3 / +4 lines
198	bool GraphicsContextGLOpenGL::copyTextureFromMedia(MediaPlayer& player, PlatformGLObject outputTexture, GCGLenum outputTarget, GCGLint level, GCGLenum internalFormat, GCGLenum format, GCGLenum type, bool premultiplyAlpha, bool flipY)	198	bool GraphicsContextGLOpenGL::copyTextureFromMedia(MediaPlayer& player, PlatformGLObject outputTexture, GCGLenum outputTarget, GCGLint level, GCGLenum internalFormat, GCGLenum format, GCGLenum type, bool premultiplyAlpha, bool flipY)
199	{	199	{
200	#if USE(AVFOUNDATION)	200	#if USE(AVFOUNDATION)
201	auto pixelBuffer = player.pixelBufferForCurrentTime();	201	auto frame = player.pixelBufferForCurrentTime();
202	if (!pixelBuffer)	202
		203	if (!frame)
203	return false;	204	return false;
204		205
205	auto contextCV = asCV();	206	auto contextCV = asCV();
Lines 208-214 bool GraphicsContextGLOpenGL::copyTextureFromMedia(MediaPlayer& player, Platform a/Source/WebCore/platform/graphics/opengl/GraphicsContextGLOpenGL.cpp_sec2
208		209
209	UNUSED_VARIABLE(premultiplyAlpha);	210	UNUSED_VARIABLE(premultiplyAlpha);
210	ASSERT_UNUSED(outputTarget, outputTarget == GraphicsContextGL::TEXTURE_2D);	211	ASSERT_UNUSED(outputTarget, outputTarget == GraphicsContextGL::TEXTURE_2D);
211	return contextCV->copyPixelBufferToTexture(pixelBuffer.get(), outputTexture, level, internalFormat, format, type, GraphicsContextGL::FlipY(flipY));	212	return contextCV->copyPixelBufferToTexture(frame->pixelBuffer.get(), frame->transform, outputTexture, level, internalFormat, format, type, GraphicsContextGL::FlipY(flipY));
212	#else	213	#else
213	return player.copyVideoTextureToPlatformTexture(this, outputTexture, outputTarget, level, internalFormat, format, type, premultiplyAlpha, flipY);	214	return player.copyVideoTextureToPlatformTexture(this, outputTexture, outputTarget, level, internalFormat, format, type, premultiplyAlpha, flipY);
214	#endif	215	#endif

Lines 244-261 void RemoteGraphicsContextGL::copyTextureFromMedia(WebCore::MediaPlayerIdentifie a/Source/WebKit/GPUProcess/graphics/RemoteGraphicsContextGL.cpp_sec1
- a/Source/WebKit/GPUProcess/graphics/RemoteGraphicsContextGL.cpp -4 / +4 lines
244	UNUSED_VARIABLE(premultiplyAlpha);	244	UNUSED_VARIABLE(premultiplyAlpha);
245	ASSERT_UNUSED(target, target == GraphicsContextGL::TEXTURE_2D);	245	ASSERT_UNUSED(target, target == GraphicsContextGL::TEXTURE_2D);
246		246
247	RetainPtr<CVPixelBufferRef> pixelBuffer;	247	std::optional<MediaPlayer::PixelBufferFrame> frame;
248	auto getPixelBuffer = [&] {	248	auto getPixelBuffer = [&] {
249	if (!m_gpuConnectionToWebProcess)	249	if (!m_gpuConnectionToWebProcess)
250	return;	250	return;
251		251
252	if (auto mediaPlayer = m_gpuConnectionToWebProcess->remoteMediaPlayerManagerProxy().mediaPlayer(mediaPlayerIdentifier))	252	if (auto mediaPlayer = m_gpuConnectionToWebProcess->remoteMediaPlayerManagerProxy().mediaPlayer(mediaPlayerIdentifier))
253	pixelBuffer = mediaPlayer->pixelBufferForCurrentTime();	253	frame = mediaPlayer->pixelBufferForCurrentTime();
254	};	254	};
255		255
256	callOnMainRunLoopAndWait(WTFMove(getPixelBuffer));	256	callOnMainRunLoopAndWait(WTFMove(getPixelBuffer));
257		257
258	if (!pixelBuffer) {	258	if (!frame) {
259	completionHandler(false);	259	completionHandler(false);
260	return;	260	return;
261	}	261	}
Lines 266-272 void RemoteGraphicsContextGL::copyTextureFromMedia(WebCore::MediaPlayerIdentifie a/Source/WebKit/GPUProcess/graphics/RemoteGraphicsContextGL.cpp_sec2
266	return;	266	return;
267	}	267	}
268		268
269	completionHandler(contextCV->copyPixelBufferToTexture(pixelBuffer.get(), texture, level, internalFormat, format, type, GraphicsContextGL::FlipY(flipY)));	269	completionHandler(contextCV->copyPixelBufferToTexture(frame->pixelBuffer.get(), frame->transform, texture, level, internalFormat, format, type, GraphicsContextGL::FlipY(flipY)));
270	#else	270	#else
271	UNUSED_VARIABLE(mediaPlayerIdentifier);	271	UNUSED_VARIABLE(mediaPlayerIdentifier);
272	UNUSED_VARIABLE(texture);	272	UNUSED_VARIABLE(texture);

Lines 308-317 private: a/Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.h_sec1
- a/Source/WebKit/GPUProcess/media/RemoteMediaPlayerProxy.h -2 / +2 lines
308	void currentTimeChanged(const MediaTime&);	308	void currentTimeChanged(const MediaTime&);
309		309
310	#if PLATFORM(COCOA)	310	#if PLATFORM(COCOA)
311	void nativeImageForCurrentTime(CompletionHandler<void(std::optional<WTF::MachSendRight>&&)>&&);	311	void nativeImageForCurrentTime(CompletionHandler<void(std::optional<WTF::MachSendRight>&&, WebCore::AffineTransform&&)>&&);
312	#endif	312	#endif
313	#if USE(AVFOUNDATION)	313	#if USE(AVFOUNDATION)
314	void pixelBufferForCurrentTime(CompletionHandler<void(RetainPtr<CVPixelBufferRef>&&)>&&);	314	void pixelBufferForCurrentTime(CompletionHandler<void(RetainPtr<CVPixelBufferRef>&&, WebCore::AffineTransform&&)>&&);
315	#endif	315	#endif
316		316
317	#if !RELEASE_LOG_DISABLED	317	#if !RELEASE_LOG_DISABLED

Lines 83-122 void RemoteMediaPlayerProxy::setVideoInlineSizeFenced(const WebCore::FloatSize& a/Source/WebKit/GPUProcess/media/cocoa/RemoteMediaPlayerProxyCocoa.mm_sec1
- a/Source/WebKit/GPUProcess/media/cocoa/RemoteMediaPlayerProxyCocoa.mm -13 / +18 lines
83	setVideoInlineSizeIfPossible(*m_inlineLayerHostingContext, size);	83	setVideoInlineSizeIfPossible(*m_inlineLayerHostingContext, size);
84	}	84	}
85		85
86	void RemoteMediaPlayerProxy::nativeImageForCurrentTime(CompletionHandler<void(std::optional<WTF::MachSendRight>&&)>&& completionHandler)	86	void RemoteMediaPlayerProxy::nativeImageForCurrentTime(CompletionHandler<void(std::optional<WTF::MachSendRight>&&, WebCore::AffineTransform&&)>&& completionHandler)
87	{	87	{
88	if (!m_player) {	88	if (!m_player) {
89	completionHandler(std::nullopt);	89	completionHandler(std::nullopt, { });
90	return;	90	return;
91	}	91	}
92		92
93	auto nativeImage = m_player->nativeImageForCurrentTime();	93	auto frame = m_player->nativeImageForCurrentTime();
94	if (!nativeImage) {	94	if (!frame) {
95	completionHandler(std::nullopt);	95	completionHandler(std::nullopt, { });
96	return;	96	return;
97	}	97	}
98		98
99	auto platformImage = nativeImage->platformImage();	99	auto platformImage = frame->nativeImage->platformImage();
100	if (!platformImage) {	100	if (!platformImage) {
101	completionHandler(std::nullopt);	101	completionHandler(std::nullopt, { });
102	return;	102	return;
103	}	103	}
104		104
105	auto surface = WebCore::IOSurface::createFromImage(platformImage.get());	105	auto surface = WebCore::IOSurface::createFromImage(platformImage.get());
106	if (!surface) {	106	if (!surface) {
107	completionHandler(std::nullopt);	107	completionHandler(std::nullopt, { });
108	return;	108	return;
109	}	109	}
110		110
111	completionHandler(surface->createSendRight());	111	completionHandler(surface->createSendRight(), WTFMove(frame->transform));
112	}	112	}
113		113
114	void RemoteMediaPlayerProxy::pixelBufferForCurrentTime(CompletionHandler<void(RetainPtr<CVPixelBufferRef>&&)>&& completionHandler)	114	void RemoteMediaPlayerProxy::pixelBufferForCurrentTime(CompletionHandler<void(RetainPtr<CVPixelBufferRef>&&, WebCore::AffineTransform&&)>&& completionHandler)
115	{	115	{
116	RetainPtr<CVPixelBufferRef> result;	116	std::optional<MediaPlayer::PixelBufferFrame> result;
117	if (m_player)	117	if (m_player) {
118	result = m_player->pixelBufferForCurrentTime();	118	result = m_player->pixelBufferForCurrentTime();
119	completionHandler(WTFMove(result));	119	if (result)
		120	completionHandler(WTFMove(result->pixelBuffer), WTFMove(result->transform));
		121	else
		122	completionHandler(nullptr, { });
		123	}
		124
120	}	125	}
121		126
122	} // namespace WebKit	127	} // namespace WebKit

Lines 963-974 void MediaPlayerPrivateRemote::paintCurrentFrameInContext(GraphicsContext& conte a/Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.cpp_sec1
- a/Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.cpp -8 / +13 lines
963	if (context.paintingDisabled())	963	if (context.paintingDisabled())
964	return;	964	return;
965		965
966	auto nativeImage = nativeImageForCurrentTime();	966	auto frame = nativeImageForCurrentTime();
967	if (!nativeImage)	967	if (!frame)
968	return;	968	return;
969		969
970	FloatRect imageRect { FloatPoint::zero(), nativeImage->size() };	970	FloatRect imageRect { FloatPoint::zero(), frame->nativeImage->size() };
971	context.drawNativeImage(*nativeImage, imageRect.size(), rect, imageRect);	971	// FIXME: Why do we map with the inverse but use the straight transform?
		972	// See MediaPlayerPrivateMediaStreamSoruceAVFObjC.mm.
		973	imageRect = frame->transform.inverse().value_or(AffineTransform { }).mapRect(imageRect);
		974	GraphicsContextStateSaver stateSaver(context);
		975	context.concatCTM(frame->transform);
		976	context.drawNativeImage(frame->nativeImage, imageRect.size(), rect, imageRect);
972	}	977	}
973		978
974	#if !USE(AVFOUNDATION)	979	#if !USE(AVFOUNDATION)
Lines 978-995 bool MediaPlayerPrivateRemote::copyVideoTextureToPlatformTexture(WebCore::Graphi a/Source/WebKit/WebProcess/GPU/media/MediaPlayerPrivateRemote.cpp_sec2
978	return false;	983	return false;
979	}	984	}
980	#elif !PLATFORM(COCOA)	985	#elif !PLATFORM(COCOA)
981	RetainPtr<CVPixelBufferRef> MediaPlayerPrivateRemote::pixelBufferForCurrentTime()	986	std::optional<WebCore::MediaPlayer::PixelBufferFrame> MediaPlayerPrivateRemote::pixelBufferForCurrentTime()
982	{	987	{
983	notImplemented();	988	notImplemented();
984	return false;	989	return std::nullopt;
985	}	990	}
986	#endif	991	#endif
987		992
988	#if !PLATFORM(COCOA)	993	#if !PLATFORM(COCOA)
989	RefPtr<NativeImage> MediaPlayerPrivateRemote::nativeImageForCurrentTime()	994	std::optional<WebCore::MediaPlayer::NativeImageFrame> MediaPlayerPrivateRemote::nativeImageForCurrentTime()
990	{	995	{
991	notImplemented();	996	notImplemented();
992	return nullptr;	997	return std::nullopt;
993	}	998	}
994	#endif	999	#endif
995		1000

Lines 46-78 PlatformLayerContainer MediaPlayerPrivateRemote::createVideoFullscreenLayer() a/Source/WebKit/WebProcess/GPU/media/cocoa/MediaPlayerPrivateRemoteCocoa.mm_sec1
- a/Source/WebKit/WebProcess/GPU/media/cocoa/MediaPlayerPrivateRemoteCocoa.mm -12 / +18 lines
46	}	46	}
47	#endif	47	#endif
48		48
49	RefPtr<NativeImage> MediaPlayerPrivateRemote::nativeImageForCurrentTime()	49	std::optional<MediaPlayer::NativeImageFrame> MediaPlayerPrivateRemote::nativeImageForCurrentTime()
50	{	50	{
51	std::optional<MachSendRight> sendRight;	51	std::optional<MachSendRight> sendRight;
52	if (!connection().sendSync(Messages::RemoteMediaPlayerProxy::NativeImageForCurrentTime(), Messages::RemoteMediaPlayerProxy::NativeImageForCurrentTime::Reply(sendRight), m_id))	52	WebCore::AffineTransform transform;
53	return nullptr;	53	if (!connection().sendSync(Messages::RemoteMediaPlayerProxy::NativeImageForCurrentTime(), Messages::RemoteMediaPlayerProxy::NativeImageForCurrentTime::Reply(sendRight, transform), m_id))
		54	return std::nullopt;
54		55
55	if (!sendRight)	56	if (!sendRight)
56	return nullptr;	57	return std::nullopt;
57		58
58	auto surface = WebCore::IOSurface::createFromSendRight(WTFMove(*sendRight), WebCore::DestinationColorSpace::SRGB());	59	auto surface = WebCore::IOSurface::createFromSendRight(WTFMove(*sendRight), WebCore::DestinationColorSpace::SRGB());
59	if (!surface)	60	if (!surface)
60	return nullptr;	61	return std::nullopt;
61		62
62	auto platformImage = WebCore::IOSurface::sinkIntoImage(WTFMove(surface));	63	auto platformImage = WebCore::IOSurface::sinkIntoImage(WTFMove(surface));
63	if (!platformImage)	64	if (!platformImage)
64	return nullptr;	65	return std::nullopt;
65		66
66	return NativeImage::create(WTFMove(platformImage));	67	auto image = NativeImage::create(WTFMove(platformImage));
		68	if (!image)
		69	return std::nullopt;
		70	return MediaPlayer::NativeImageFrame { image.releaseNonNull(), WTFMove(transform) };
67	}	71	}
68		72
69	RetainPtr<CVPixelBufferRef> MediaPlayerPrivateRemote::pixelBufferForCurrentTime()	73	std::optional<MediaPlayer::PixelBufferFrame> MediaPlayerPrivateRemote::pixelBufferForCurrentTime()
70	{	74	{
71
72	RetainPtr<CVPixelBufferRef> result;	75	RetainPtr<CVPixelBufferRef> result;
73	if (!connection().sendSync(Messages::RemoteMediaPlayerProxy::PixelBufferForCurrentTime(), Messages::RemoteMediaPlayerProxy::PixelBufferForCurrentTime::Reply(result), m_id))	76	WebCore::AffineTransform transform;
74	return nullptr;	77	if (!connection().sendSync(Messages::RemoteMediaPlayerProxy::PixelBufferForCurrentTime(), Messages::RemoteMediaPlayerProxy::PixelBufferForCurrentTime::Reply(result, transform), m_id))
75	return result;	78	return std::nullopt;
		79	if (!result)
		80	return std::nullopt;
		81	return MediaPlayer::PixelBufferFrame { WTFMove(result), WTFMove(transform) };
76	}	82	}
77		83
78	} // namespace WebKit	84	} // namespace WebKit