Bug 230613 – MediaStream canvas.captureStream() fails for WebGL

• Component: WebGL
• Status: RESOLVED FIXED

[Kimmo Kinnunen]

MediaStream canvas.captureStream() fails for WebGL
Produces black or red images

[Kimmo Kinnunen]

<rdar://82307293>

[Kimmo Kinnunen]

Created attachment 438942 [details]
Patch

[Kenneth Russell]

Comment on attachment 438942 [details]
Patch

Nice work Kimmo figuring out how to make this work without the preserveDrawingBuffer hack. Let's see how it looks when the tests are green.

[Kimmo Kinnunen]

Created attachment 439134 [details]
Patch

[Kimmo Kinnunen]

Created attachment 439141 [details]
Patch

[Kimmo Kinnunen]

Created attachment 439142 [details]
Patch

[youenn fablet]

Comment on attachment 439142 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=439142&action=review

> Source/WebCore/ChangeLog:33
> +        flakyness lets rendering happen)

Is there not a way to make a basic test that would:
- check that frames are generated.
- check that frames are not uninitialized (either through video painting in canvas or through internals API, see observeMediaStreamTrack/grabNextMediaStreamTrackFrame).

> Source/WebCore/ChangeLog:37
> +        captureStream() and playback implementation is so flaky.

Some tests need rebasing of their expected.txt files

> Source/WebCore/platform/graphics/cocoa/GraphicsContextGLOpenGLCocoa.mm:898
> +    return MediaSampleAVFObjC::createImageSample(WTFMove(*pixelBuffer), MediaSampleAVFObjC::VideoRotation::UpsideDown, true);

Why do we get it upside down and mirrored?
Upside down should be correctly handled everywhere, not so sure about mirrored since I am not sure we have sources with mirrored frames.

[Kimmo Kinnunen]

Created attachment 439597 [details]
Patch

[EWS]

Committed r283238 (242280@main): <https://commits.webkit.org/242280@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 439597 [details].

[Jon Lee]

*** Bug 229611 has been marked as a duplicate of this bug. ***

[Yuwei Zhang]

(In reply to Jon Lee from comment #10)
> *** Bug 229611 has been marked as a duplicate of this bug. ***

After testing at Monterey, I found that the bug was not fixed and I was wondering if anyone was following up？

[Alexander Khovansky]

From my checks the bug was fixed but only for a moment, before this got broken again.
You can see the canvas to video demo is producing blank square https://webrtc.github.io/samples/src/content/capture/canvas-video/.

Caveat: the demo could work once, I think after a device restart. However if you reload the page, it would no longer work.

[Kimmo Kinnunen]

(In reply to Alexander Khovansky from comment #12)
> From my checks the bug was fixed but only for a moment, before this got
> broken again.
> You can see the canvas to video demo is producing blank square
> https://webrtc.github.io/samples/src/content/capture/canvas-video/.
> 
> Caveat: the demo could work once, I think after a device restart. However if
> you reload the page, it would no longer work.

The flaky behaviour is due to bug 230623. It will be fixed once that fix ships, unfortunately it is not yet in public releases.
To workaround the flaky behaviour you could try to assign to the srcObject after the video element is created, for example asynchronously in a setTimeout callback.

[Trung Le]

I still have this issue in Safari 15.3 as well as Technology Preview versions

This issue still not fixed.

1. generate a canvas and a 2d context.
2. fill random number to canvas by context.fillText();
3. canvas.captureStream() and use video tag to play canvas mediaStream

can be reproduced on Mac Safari 15.4

https://sdk-web-1252463788.cos.ap-hongkong.myqcloud.com/trtc/webrtc/test/rychou-test/canvas-safari/index.html

[Kimmo Kinnunen]

(In reply to rychouwei from comment #15)
> This issue still not fixed.
> 
> 1. generate a canvas and a 2d context.
> 2. fill random number to canvas by context.fillText();
> 3. canvas.captureStream() and use video tag to play canvas mediaStream
> 
> can be reproduced on Mac Safari 15.4
> 
> https://sdk-web-1252463788.cos.ap-hongkong.myqcloud.com/trtc/webrtc/test/
> rychou-test/canvas-safari/index.html

Thanks for the report. This context2d issue is being tracked in bug 230621.
I'll add your bug there.