Bug 230621 - Canvas generated transparent pixels are not well handled by LocalSampleBufferDisplayLayer
Summary: Canvas generated transparent pixels are not well handled by LocalSampleBuffer...
Alias: None
Product: WebKit
Classification: Unclassified
Component: Canvas (show other bugs)
Version: WebKit Local Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: youenn fablet
Keywords: InRadar
: 231598 (view as bug list)
Depends on:
Reported: 2021-09-22 06:18 PDT by Kimmo Kinnunen
Modified: 2022-05-17 06:36 PDT (History)
9 users (show)

See Also:

Frozen video stream repro video (14.41 MB, video/quicktime)
2022-01-14 08:21 PST, Anastasiya Sarmant
no flags Details
working test case (946 bytes, text/html)
2022-05-12 15:57 PDT, Said Abou-Hallawa
no flags Details
Patch (2.33 KB, patch)
2022-05-13 06:15 PDT, youenn fablet
ews-feeder: commit-queue-
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Kimmo Kinnunen 2021-09-22 06:18:06 PDT
MediaStream canvas.captureStream() fails for Canvas Context2D

see the bug 230617 test case mediastream-canvas-to-video.html last few elements
Comment 1 Kimmo Kinnunen 2021-09-22 06:18:49 PDT
Sorry, bug 230613.
Comment 2 Radar WebKit Bug Importer 2021-09-29 06:19:16 PDT
Comment 3 Kimmo Kinnunen 2021-10-28 00:04:59 PDT
*** Bug 231598 has been marked as a duplicate of this bug. ***
Comment 4 Kimmo Kinnunen 2021-10-28 00:05:50 PDT
>This only occurs on Safari 15 and Safari Technology Preview. It does not occur on Safari 14.

> Here is a JSFiddle that reproduces the issue: https://jsfiddle.net/x8pz4u2y/1/. Click on the "Start" button to start capturing the canvas, which flashes between blue and green every second. Note that you may need to resize the window to get the video element to show in Safari 15 for some reason (which may be a separate bug but my main concern is that the resulting video is red).
Comment 5 btham 2021-12-01 10:23:55 PST
Hi Kimmo, I was wondering if there were any updates on this? (I am the original filer for https://bugs.webkit.org/show_bug.cgi?id=231598.)
Comment 6 Anastasiya Sarmant 2022-01-14 08:21:31 PST
Created attachment 449174 [details]
Frozen video stream repro video
Comment 7 Anastasiya Sarmant 2022-01-14 08:26:16 PST
Any updates on that bug?
We have developed webgl2 based background replacement feature for MS Teams and this bug prevents us from shipping it on Safari.
We have Macbook Air M1 and Macbook Pro Intel based. The result of the below code snippet is a bit different on those machines. On M1 the stream is frozen but I'm able to get frames flowing by randomly clicking the page or scrolling, it looks the canvas streams expects repaint or layout update. On Macbook Pro it's frozen regardless. Attached a video example.

// simple repro
public async processStream(mediaStream: MediaStream): Promise<MediaStream> { 
        const settings = mediaStream.getVideoTracks()[0].getSettings(); 
        const canvas = document.createElement('canvas'); 
        const ctx = canvas.getContext('2d'); 
        canvas.width = settings.width; 
        canvas.height = settings.height; 
        const video = document.createElement('video'); 
        video.srcObject = mediaStream; 
        video.autoplay = true; 
        video.onplay = (): void => { 
            const loop = (): void => { 
                if (!video.paused && !video.ended) { 
                    ctx.drawImage(video, 0, 0); 
                    setTimeout(loop,1000 /66);  
        return canvas.captureStream(); 
Comment 8 Mukesh 2022-01-14 10:35:30 PST
canvas.captureStream()is broken on Safari 15 and Safari Technology Preview. Any update on this or workaround to get a stream from canvas?
Comment 9 Tom Barnsbury 2022-02-04 12:11:08 PST
I have tested on a MacBook Pro (14-inch, 2021) on Monterey 12.0.1 (21A559).

I have observed the behaviour reported by Anastasiya Sarmant while testing on Safari Technology Preview Release 139 (Safari 15.4, WebKit 17613.

I do not see this behaviour on Safari 15.1, it works as expected for me there.
Comment 10 rychouwei 2022-03-28 23:34:37 PDT
This issue still not fixed.

1. generate a canvas and a 2d context.
2. fill random number to canvas by context.fillText();
3. canvas.captureStream() and use video tag to play canvas mediaStream

can be reproduced on Mac Safari 15.4

Comment 11 Kimmo Kinnunen 2022-03-29 11:17:51 PDT
From test-case from bug 230613 

> https://sdk-web-1252463788.cos.ap-hongkong.myqcloud.com/trtc/webrtc/test/
> rychou-test/canvas-safari/index.html
Comment 12 Said Abou-Hallawa 2022-05-12 15:57:15 PDT
Replacing this line in the test case: 

    ctx.clearRect(0, 0, 320, 240)


    ctx.fillStyle = 'green';
    ctx.fillRect(0, 0, 320, 240);
    ctx.fillStyle = 'white';

fixed the bug. So it seems video capturing has an issue with transparent pixels. They are displayed always black.
Comment 13 Said Abou-Hallawa 2022-05-12 15:57:46 PDT
Created attachment 459254 [details]
working test case
Comment 14 Said Abou-Hallawa 2022-05-12 16:11:07 PDT
An easy workaround for this bug is to fillRect() the canvas rect with 'white' instead of calling clearRect().
Comment 15 youenn fablet 2022-05-13 05:06:08 PDT
(In reply to Said Abou-Hallawa from comment #14)
> An easy workaround for this bug is to fillRect() the canvas rect with
> 'white' instead of calling clearRect().

There are two remaining bugs:
- Handling of transparent pixels in
- Generation of frames from canvas that are not in the DOM tree can be erratic or freeze (https://jsfiddle.net/0b3dqxeg/).

Let's focus this particular bug on transparent pixel handling.
I filed https://bugs.webkit.org/show_bug.cgi?id=240380 to jeep track of not in the DOM canvas.
Comment 16 youenn fablet 2022-05-13 05:08:11 PDT
Renaming the bug accordingly.
We should also look at how transparent pixels are handled in peer connections.
Comment 17 youenn fablet 2022-05-13 05:22:04 PDT
(In reply to youenn fablet from comment #16)
> Renaming the bug accordingly.
> We should also look at how transparent pixels are handled in peer
> connections.

https://jsfiddle.net/nfu7oL60/ seems to show transparency has not good WebRTC interop in Chrome/Firefox/Safari.
Comment 18 youenn fablet 2022-05-13 05:25:27 PDT
https://jsfiddle.net/wj1br5xg/ seems to indicate we should use a transparent background color for LocalSampleBufferDisplayLayer, if we look at what Chrome and Firefox are doing.
Comment 19 youenn fablet 2022-05-13 06:15:39 PDT
Created attachment 459296 [details]
Comment 20 youenn fablet 2022-05-16 02:59:00 PDT
Handled in https://github.com/WebKit/WebKit/pull/633
Comment 21 EWS 2022-05-16 09:13:11 PDT
Committed r294235 (250593@main): <https://commits.webkit.org/250593@main>

Reviewed commits have been landed. Closing PR #633 and removing active labels.
Comment 22 btham 2022-05-17 06:36:43 PDT
Which Safari version can we expect the fix in?