Bug 181663 – Video Element cannot playback local Canvas.captureStream on iOS

• Status: RESOLVED
• Resolution: CONFIGURATION CHANGED
• Component: WebRTC

[Adam]

If you use Canvas.captureStream and pass the stream to a video element srcObject you just get a blank video. It works in Safari on Mac OS, just not on iOS.

Example code: 

  const videoEl = document.createElement('video');

  const canvas = document.createElement('canvas');
  const ctx = canvas.getContext('2d');
  canvas.width = 640;
  canvas.height = 480;

  const randomColour = () => {
    return Math.round(Math.random() * 255);
  };

  setInterval(function() {
    ctx.clearRect(0, 0, canvas.width, canvas.height);
    ctx.fillStyle = `rgb(${randomColour()}, ${randomColour()}, ${randomColour()})`;
    ctx.fillRect(0, 0, canvas.width, canvas.height);
  }, 1000);

  document.body.appendChild(canvas);
  document.body.appendChild(videoEl);

  videoEl.srcObject = canvas.captureStream(1);
  videoEl.setAttribute('playsinline', '');
  videoEl.play();

You can see the example running at: http://jsbin.com/nopusa

You are expected to see a Canvas element and a Video Element both with random colours being updated every 1 second. But on iOS it updates on the Canvas but not on the Video Element.

[Brooke Vibber]

I have the same problem under iOS 12.3; tested iPad Pro 9.7" and iPhone X.

It sees the stream and sets the <video> dimensions correctly, but never shows anything but solid black.

[Brooke Vibber]

On a whim I tested in Safari on the iPhone Simulator and Adam's sample works there. I suspect the sample buffer pixel layout used by the capture isn't playable by native iOS's media server, whereas the simulator piggybacks on the macOS one for actual video decoding.

I've seen problems like that when manually decoding video to uncompressed video sample buffers on iOS, where not all formats that work on the simulator work on the native devices...

In this case, it may be fixable by doing a pixel format conversion on the frame (or capturing it directly to a format that works).

[Radar WebKit Bug Importer]

<rdar://problem/51150406>

[Joe]

I can confirm that this is still an issue in iOS 13.3.1.

Here is another example that works with Safari on Mac but not iOS:

https://webrtc.github.io/samples/src/content/capture/canvas-video/

[Kahiro Okina]

I can confirm that this is still an issue in iOS 14 beta8

Here is another example that works with Safari on Mac but not iOS

https://codepen.io/k-okina/pen/QWNrWwM

[cabada]

I can confirm that this is still an issue in iOS 14 public release.

[Davy De Durpel]

I can confirm that this is still an issue in iOS 14.1 and iOS 14.2 Public beta

[Sam Wray]

Confirmed still an issue in iOS 14.3.

Anything blocking this from being worked on?
We've just surpassed three years of this API being available and not working in Safari on iOS.

[Francesco Durighetto (kekkokk)]

still an issue on 14.4.2

Any plan to take this in consideration?
For us it's necessary for the "virtual background" feature that is much appreciated during this pandemic period.

[David Halls]

Bug is still present. It'd be better if captureStream() just threw an exception (or didn't exist) if iOS isn't going to support it.

[Brooke Vibber]

(In reply to David Halls from comment #10)
> Bug is still present. It'd be better if captureStream() just threw an
> exception (or didn't exist) if iOS isn't going to support it.

Works in Safari in iPadOS 15 beta on 11” iPad Pro now. Haven’t tested on any iPhones yet.

[Brooke Vibber]

Seems to work on iPhone X as well on iOS 15 beta now. Yay!

For me it doesn't work in:
Safari Technology Preview: Release 132 (Safari 15.4, WebKit 16613.1.1.5)

Works fine in:
Safari: Version 14.1.2 (16611.3.10.1.6)

In both cases Mac OS Big Sur 11.6 was used to test.

[David Halls]

Works for me on iOS 15 now.

pmikolajczak: The MacOS bug is here: https://bugs.webkit.org/show_bug.cgi?id=229611

[Frank Robert]

It seems that this is still reproducible in Safari on Mac (as well as on iOS).

> Safari Version 15.2 (16612.3.6.1.8, 16612)
> Mac Version 11.6.2 (20G314)

Here are a few links to repro:

• https://www.demo2s.com/javascript/javascript-canvas-change-play-out-delay-in-webrtc-stream.html
• https://webkit.org/blog-files/webrtc/pc-with-effects/index.html
• https://webrtc.github.io/samples/src/content/capture/canvas-video/

Bonus, here's a reproducible related bug for MSE canvas captures to video:
• https://s3.amazonaws.com/storage2.interlude.fm/dev_temp/tomer/safari_mse_canvas_bug/index.html

Interestingly, in the `webrtc.github.io` example, they use WebGL and I've noticed that occasionally (1 in every ~10 refreshes _) you will get what appears to be the first frame drawn correctly from `captureStream`. This is supported by the Limitations & Additional Functionality section of the CaptureStream implementation spec sheet https://docs.google.com/document/d/1JmWfOtUP6ZqsYJ--U8y0OtHkBt-VyjX4N-JqIjb1t78/edit#heading=h.hkgqgoi0quq2 wherein they describe the background/alpha channel as being drawn with black pixels. 

When that happens a `drag` (possibly something else/similar) event on the canvas or video element seem to trigger a proper redraw of the video element and the video "plays". The caveat to this approach is that I've yet to be able to programmatically trigger the drag event to redraw.

[Joe]

Created attachment 449599 [details]
iOS Safari

I can confirm that this behaviour is inconsistent and glitchy. I just tested this on iOS 15.2.1 and occasionally it works fine, mostly it just shows 1 frame, sometimes it doesn't show any frames and sometimes it crashes the page. Here is a screen recording showing all of these different things happening.

[Joe]

I can confirm that this behaviour is inconsistent and glitchy. I just tested this on iOS 15.2.1 and occasionally it works fine, mostly it just shows 1 frame, sometimes it doesn't show any frames and sometimes it crashes the page. Here is a screen recording showing all of these different things happening:

https://bug-181663-attachments.webkit.org/attachment.cgi?id=449599

[Prabhat]

This issue still persists. Has anyone found a solution for this yet? I am using HTML Canvas element as the video source to publish the video. All I have been getting is a black screen and sometimes the first frame loads but ends up with a black screen.

[Mitch Talmadge]

This bug is preventing background blur from working in our meeting software on iOS devices, which is detrimental to students who need to meet with counselors online when their environment is embarrassing, like a messy room or a bathroom (some students can only get privacy in a place like that!)

Hopefully this can be solved soon.

test https://mantou132.github.io/javascript-learn/stream.html on macOS 12.2

Version 15.3 (17612.4.9.1.5): fail
Release 140 (Safari 15.4, WebKit 17614.1.1.5): fail

[youenn fablet]

https://webrtc.github.io/samples/src/content/capture/canvas-video/ is fully working in Safari Tech Preview now.
It is working on iOS once the teapot is moved in the latest iOS 15.4. I would expect webgl canvas to be working there.


2d canvas in https://mantou132.github.io/javascript-learn/stream.html seems to be broken still.

[Prabhat]

(In reply to Prabhat from comment #18)
> This issue still persists. Has anyone found a solution for this yet? I am
> using HTML Canvas element as the video source to publish the video. All I
> have been getting is a black screen and sometimes the first frame loads but
> ends up with a black screen.

Since the latest OS update, the issue seems to be fixed. I tested my video call application in the below scenarios:
Chrome → Chrome → Working as expected.
Chrome → Safari→ Working as expected.
Safari → Safari → Working as expected.

Version details:
Chrome - 100.0.4896.75
Safari - 15.4
Mac OS - 12.3.1

[youenn fablet]

*** Bug 206829 has been marked as a duplicate of this bug. ***

[Claudio Destro]

Hope this helps.

The issue still persists in Safari 15.4+ desktop but using "webgl2" context seems to work:

Modified example code:

            const videoEl = document.createElement('video');

            const canvas = document.createElement('canvas');
            const ctx = canvas.getContext('webgl2');
            canvas.width = 640;
            canvas.height = 480;

            const randomColour = () => {
                return Math.round(Math.random() * 255);
            };

            setInterval(function () {
                ctx.viewport(0, 0, canvas.width, canvas.height);
                ctx.clearColor(Math.random(), Math.random(), Math.random(), 1.0);
                ctx.clear(ctx.COLOR_BUFFER_BIT);

                // ctx.clearRect(0, 0, canvas.width, canvas.height);
                // ctx.fillStyle = `rgb(${randomColour()}, ${randomColour()}, ${randomColour()})`;
                // ctx.fillRect(0, 0, canvas.width, canvas.height);
            }, 1000);

            document.body.appendChild(canvas);
            document.body.appendChild(videoEl);

            videoEl.srcObject = canvas.captureStream(1);
            videoEl.setAttribute('playsinline', '');
            videoEl.play();

PS: In fact WebRTC Sample "Canvas to Video" works
https://webrtc.github.io/samples/src/content/capture/canvas-video/

[youenn fablet]

This works to me with a ToT WebKit build.
I believe this is fixed, though we identified issues with transparent pixels (https://bugs.webkit.org/show_bug.cgi?id=230621) and frame rate (https://bugs.webkit.org/show_bug.cgi?id=240380).

Marking as Configuration Changed.