Canvas generated transparent pixels are not well handled by LocalSampleBufferDisplayLayer

• Component: Canvas
• Status: RESOLVED FIXED

[Kimmo Kinnunen]

MediaStream canvas.captureStream() fails for Canvas Context2D

see the bug 230617 test case mediastream-canvas-to-video.html last few elements

[Kimmo Kinnunen]

Sorry, bug 230613.

[Radar WebKit Bug Importer]

<rdar://problem/83668394>

[Kimmo Kinnunen]

*** Bug 231598 has been marked as a duplicate of this bug. ***

[Kimmo Kinnunen]

>This only occurs on Safari 15 and Safari Technology Preview. It does not occur on Safari 14.

> Here is a JSFiddle that reproduces the issue: https://jsfiddle.net/x8pz4u2y/1/. Click on the "Start" button to start capturing the canvas, which flashes between blue and green every second. Note that you may need to resize the window to get the video element to show in Safari 15 for some reason (which may be a separate bug but my main concern is that the resulting video is red).

Hi Kimmo, I was wondering if there were any updates on this? (I am the original filer for https://bugs.webkit.org/show_bug.cgi?id=231598.)

[Anastasiya Sarmant]

Created attachment 449174 [details]
Frozen video stream repro video

[Anastasiya Sarmant]

Any updates on that bug?
We have developed webgl2 based background replacement feature for MS Teams and this bug prevents us from shipping it on Safari.
We have Macbook Air M1 and Macbook Pro Intel based. The result of the below code snippet is a bit different on those machines. On M1 the stream is frozen but I'm able to get frames flowing by randomly clicking the page or scrolling, it looks the canvas streams expects repaint or layout update. On Macbook Pro it's frozen regardless. Attached a video example.

// simple repro
public async processStream(mediaStream: MediaStream): Promise<MediaStream> { 
        const settings = mediaStream.getVideoTracks()[0].getSettings(); 
        const canvas = document.createElement('canvas'); 
        const ctx = canvas.getContext('2d'); 
        canvas.width = settings.width; 
        canvas.height = settings.height; 
        const video = document.createElement('video'); 
        video.srcObject = mediaStream; 
        video.autoplay = true; 
        video.onplay = (): void => { 
            const loop = (): void => { 
                if (!video.paused && !video.ended) { 
                    ctx.drawImage(video, 0, 0); 
                    setTimeout(loop,1000 /66);  
                } 
            }; 
            loop(); 
        }; 
        video.play(); 
        return canvas.captureStream(); 
    }

[Mukesh]

canvas.captureStream()is broken on Safari 15 and Safari Technology Preview. Any update on this or workaround to get a stream from canvas?

[Tom Barnsbury]

I have tested on a MacBook Pro (14-inch, 2021) on Monterey 12.0.1 (21A559).

I have observed the behaviour reported by Anastasiya Sarmant while testing on Safari Technology Preview Release 139 (Safari 15.4, WebKit 17613.1.14.41.3)

I do not see this behaviour on Safari 15.1, it works as expected for me there.

This issue still not fixed.

1. generate a canvas and a 2d context.
2. fill random number to canvas by context.fillText();
3. canvas.captureStream() and use video tag to play canvas mediaStream

can be reproduced on Mac Safari 15.4

https://sdk-web-1252463788.cos.ap-hongkong.myqcloud.com/trtc/webrtc/test/rychou-test/canvas-safari/index.html

[Kimmo Kinnunen]

From test-case from bug 230613 

> https://sdk-web-1252463788.cos.ap-hongkong.myqcloud.com/trtc/webrtc/test/
> rychou-test/canvas-safari/index.html

[Said Abou-Hallawa]

Replacing this line in the test case: 

    ctx.clearRect(0, 0, 320, 240)


With

    ctx.fillStyle = 'green';
    ctx.fillRect(0, 0, 320, 240);
    ctx.fillStyle = 'white';

fixed the bug. So it seems video capturing has an issue with transparent pixels. They are displayed always black.

[Said Abou-Hallawa]

Created attachment 459254 [details]
working test case

[Said Abou-Hallawa]

An easy workaround for this bug is to fillRect() the canvas rect with 'white' instead of calling clearRect().

[youenn fablet]

(In reply to Said Abou-Hallawa from comment #14)
> An easy workaround for this bug is to fillRect() the canvas rect with
> 'white' instead of calling clearRect().

There are two remaining bugs:
- Handling of transparent pixels in
- Generation of frames from canvas that are not in the DOM tree can be erratic or freeze (https://jsfiddle.net/0b3dqxeg/).

Let's focus this particular bug on transparent pixel handling.
I filed https://bugs.webkit.org/show_bug.cgi?id=240380 to jeep track of not in the DOM canvas.

[youenn fablet]

Renaming the bug accordingly.
We should also look at how transparent pixels are handled in peer connections.

[youenn fablet]

(In reply to youenn fablet from comment #16)
> Renaming the bug accordingly.
> We should also look at how transparent pixels are handled in peer
> connections.

https://jsfiddle.net/nfu7oL60/ seems to show transparency has not good WebRTC interop in Chrome/Firefox/Safari.

[youenn fablet]

https://jsfiddle.net/wj1br5xg/ seems to indicate we should use a transparent background color for LocalSampleBufferDisplayLayer, if we look at what Chrome and Firefox are doing.

[youenn fablet]

Created attachment 459296 [details]
Patch

[youenn fablet]

Handled in https://github.com/WebKit/WebKit/pull/633

[EWS]

Committed r294235 (250593@main): <https://commits.webkit.org/250593@main>

Reviewed commits have been landed. Closing PR #633 and removing active labels.

Which Safari version can we expect the fix in?