Bug 230613 - MediaStream canvas.captureStream() fails for WebGL
Summary: MediaStream canvas.captureStream() fails for WebGL
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebGL (show other bugs)
Version: WebKit Local Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Kimmo Kinnunen
URL:
Keywords: InRadar
: 229611 (view as bug list)
Depends on: 230338
Blocks: 230620 230623
  Show dependency treegraph
 
Reported: 2021-09-22 05:32 PDT by Kimmo Kinnunen
Modified: 2022-03-29 11:17 PDT (History)
26 users (show)

See Also:


Attachments
Patch (31.82 KB, patch)
2021-09-22 06:07 PDT, Kimmo Kinnunen
no flags Details | Formatted Diff | Diff
Patch (32.01 KB, patch)
2021-09-24 00:53 PDT, Kimmo Kinnunen
ews-feeder: commit-queue-
Details | Formatted Diff | Diff
Patch (32.24 KB, patch)
2021-09-24 06:53 PDT, Kimmo Kinnunen
no flags Details | Formatted Diff | Diff
Patch (32.58 KB, patch)
2021-09-24 07:04 PDT, Kimmo Kinnunen
no flags Details | Formatted Diff | Diff
Patch (34.62 KB, patch)
2021-09-29 05:19 PDT, Kimmo Kinnunen
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Kimmo Kinnunen 2021-09-22 05:32:51 PDT
MediaStream canvas.captureStream() fails for WebGL
Produces black or red images
Comment 1 Kimmo Kinnunen 2021-09-22 05:34:46 PDT
<rdar://82307293>
Comment 2 Kimmo Kinnunen 2021-09-22 06:07:00 PDT
Created attachment 438942 [details]
Patch
Comment 3 Kenneth Russell 2021-09-22 16:34:55 PDT
Comment on attachment 438942 [details]
Patch

Nice work Kimmo figuring out how to make this work without the preserveDrawingBuffer hack. Let's see how it looks when the tests are green.
Comment 4 Kimmo Kinnunen 2021-09-24 00:53:32 PDT
Created attachment 439134 [details]
Patch
Comment 5 Kimmo Kinnunen 2021-09-24 06:53:32 PDT
Created attachment 439141 [details]
Patch
Comment 6 Kimmo Kinnunen 2021-09-24 07:04:01 PDT
Created attachment 439142 [details]
Patch
Comment 7 youenn fablet 2021-09-29 02:42:49 PDT
Comment on attachment 439142 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=439142&action=review

> Source/WebCore/ChangeLog:33
> +        flakyness lets rendering happen)

Is there not a way to make a basic test that would:
- check that frames are generated.
- check that frames are not uninitialized (either through video painting in canvas or through internals API, see observeMediaStreamTrack/grabNextMediaStreamTrackFrame).

> Source/WebCore/ChangeLog:37
> +        captureStream() and playback implementation is so flaky.

Some tests need rebasing of their expected.txt files

> Source/WebCore/platform/graphics/cocoa/GraphicsContextGLOpenGLCocoa.mm:898
> +    return MediaSampleAVFObjC::createImageSample(WTFMove(*pixelBuffer), MediaSampleAVFObjC::VideoRotation::UpsideDown, true);

Why do we get it upside down and mirrored?
Upside down should be correctly handled everywhere, not so sure about mirrored since I am not sure we have sources with mirrored frames.
Comment 8 Kimmo Kinnunen 2021-09-29 05:19:00 PDT
Created attachment 439597 [details]
Patch
Comment 9 EWS 2021-09-29 11:22:06 PDT
Committed r283238 (242280@main): <https://commits.webkit.org/242280@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 439597 [details].
Comment 10 Jon Lee 2022-01-03 14:49:49 PST
*** Bug 229611 has been marked as a duplicate of this bug. ***
Comment 11 Yuwei Zhang 2022-01-04 00:29:10 PST
(In reply to Jon Lee from comment #10)
> *** Bug 229611 has been marked as a duplicate of this bug. ***

After testing at Monterey, I found that the bug was not fixed and I was wondering if anyone was following up?
Comment 12 Alexander Khovansky 2022-01-26 20:50:39 PST
From my checks the bug was fixed but only for a moment, before this got broken again.
You can see the canvas to video demo is producing blank square https://webrtc.github.io/samples/src/content/capture/canvas-video/.

Caveat: the demo could work once, I think after a device restart. However if you reload the page, it would no longer work.
Comment 13 Kimmo Kinnunen 2022-01-27 00:29:07 PST
(In reply to Alexander Khovansky from comment #12)
> From my checks the bug was fixed but only for a moment, before this got
> broken again.
> You can see the canvas to video demo is producing blank square
> https://webrtc.github.io/samples/src/content/capture/canvas-video/.
> 
> Caveat: the demo could work once, I think after a device restart. However if
> you reload the page, it would no longer work.

The flaky behaviour is due to bug 230623. It will be fixed once that fix ships, unfortunately it is not yet in public releases.
To workaround the flaky behaviour you could try to assign to the srcObject after the video element is created, for example asynchronously in a setTimeout callback.
Comment 14 Trung Le 2022-03-24 19:25:02 PDT
I still have this issue in Safari 15.3 as well as Technology Preview versions
Comment 15 rychouwei 2022-03-28 22:29:58 PDT
This issue still not fixed.

1. generate a canvas and a 2d context.
2. fill random number to canvas by context.fillText();
3. canvas.captureStream() and use video tag to play canvas mediaStream

can be reproduced on Mac Safari 15.4

https://sdk-web-1252463788.cos.ap-hongkong.myqcloud.com/trtc/webrtc/test/rychou-test/canvas-safari/index.html
Comment 16 Kimmo Kinnunen 2022-03-29 11:17:09 PDT
(In reply to rychouwei from comment #15)
> This issue still not fixed.
> 
> 1. generate a canvas and a 2d context.
> 2. fill random number to canvas by context.fillText();
> 3. canvas.captureStream() and use video tag to play canvas mediaStream
> 
> can be reproduced on Mac Safari 15.4
> 
> https://sdk-web-1252463788.cos.ap-hongkong.myqcloud.com/trtc/webrtc/test/
> rychou-test/canvas-safari/index.html

Thanks for the report. This context2d issue is being tracked in bug 230621.
I'll add your bug there.