Bug 230922 – REGRESSION (iOS 15): Autoplayed video element with mediaStream srcObject freezes

• Status: RESOLVED
• Resolution: CONFIGURATION CHANGED
• Component: Media

[Alec Bargas]

A video element with the 'autoplay + playsinline + muted' property set with a mediaStream srcObject can freeze. This freeze can either occur before the video starts playing (so the background is seen) or a single frame is frozen. The expectation is that the video should play continuously without freezing.

This behavior can be seen on the https://webrtc.github.io/samples/src/content/capture/canvas-video/ sample page.
The steps to reproduce the bug are:
1. Navigate to page.
2. Drag on the canvas (first teapot if two exist) and observe.
3a. If after dragging, a second teapot (appears if one didn't exist before and) moves, then the issue did not reproduce (move on to 4a).
3b. If after dragging a second teapot did not move, then the issue was reproduced and no need to follow the next steps.
4a. Close the tab, kill Safari, and relaunch Safari.
5a. From history, select the sample page.
6a. Drag on the canvas and observe, the issue should be seen.

When this happens, the media stream and tracks are still active and the video element is not paused. If the video is unmuted and the stream includes audio, the audio will still play. If using an iPad, putting Safari into split screen updates the video to what looks like the most recent frame, but the video still does not update continuously. On either iPad or iPhone, backgrounding Safari then coming back will allow the video to update continuously.

Reproduces with:
Version/15.0 Safari/605.1.15 (iOS 15.0)
Version/15.0 Safari/605.1.15 (iPadOS 15.1)

Does not reproduce with:
Prior to iOS 15 beta 8
Version/14.1.2 Safari/605.1.15 (macOS Big Sur 11.6)

[Radar WebKit Bug Importer]

<rdar://problem/83684099>

[Will Morgan]

This regression is affecting a number of our users and is blocking them from advancing through several important user journeys - authenticating to banks and healthcare etc. Could this be looked at urgently?

[Mark B]

This issue is affecting users of our web application in production. We would appreciate feedback on the planned resolution and timeline.

[Nils R]

I can confirm and reproduce the bug.
It is also affecting our customers.

Feedback would be appreciate.

[youenn fablet]

There is a known regression in canvas capture streams that is being worked on.
I am unclear though whether that bug is specific to canvas capture stream or also applies to camera and/or peer connection media streams.
Can anybody of you clarify?

[Will Morgan]

Hi Youenn, our application makes use of getUserMedia and the video element. We don't use CanvasCaptureMediaStreamTrack et al, at all.

Hopefully that clarifies?

[Nils R]

In our case the we use peer connection media streams and add those to a srcObject to a videoElement. It displays nothing as in the webrtc sample.

[youenn fablet]

Thanks.
Would you have repro cases? 
https://webrtc.github.io/samples/src/content/peerconnection/pc1/ should hopefully work fine.

Or you can send privately to me sysdiagnoses ([e-mail]).

[Chris Mo]

Our application have same issue start from Safari Technology Preview 126 or above.

[youenn fablet]

https://webrtc.github.io/samples/src/content/capture/canvas-video/ has an issue with canvas capture so I do not think it is good to consider it for .
Does any of you have a different repro case that I can use, not based on canvas capture?

[Kongpheng]

Youenn, using the sample repo you mentioned, it's possible to see video not auto-resume as expected.  Granted this isn't the same as freezing, but just wanted to share, in case it was helpful:
* Open https://webrtc.github.io/samples/src/content/peerconnection/pc1/
* Click Start button
* Grant access to microphone and camera
* Notice video is captured
* Make a call to the iPhone
* From Call UI dialog, decline it
* Notice that video doesn't get auto-resumed, instead you see a black rectangle where video used to be

This might be more similar to https://bugs.webkit.org/show_bug.cgi?id=231365 though.

[Charlie]

I'm able to reproduce using Youenn's sample repo. I added 'muted' property on the remote video. See it here https://charliesantos.github.io/samples/src/content/peerconnection/pc1/

Steps:
1. Load https://charliesantos.github.io/samples/src/content/peerconnection/pc1/
2. Press start then call button, grant access to the mic/camera
3. Make sure the video is captured
4. Make a phone call to the iPhone. Have it ring for about 10-15 seconds
5. Swipe up the call notification to go back to the page
6. Press Hang Up button, or interact to the page. Notice that the whole page is frozen.

[youenn fablet]

@Charlie, the freeze should be fixed in latest iOS 15.2 beta.
@Kongpheng, I think this is also the case in latest iOS 15.2 beta, can you validate and if not file another bug?
@Alec, the issue with the tea pot is related to canvas capture and is being worked on. Latest Safari Tech Preview should be working. Please reopen if that is not the case.

[Yevhen Lv]

Hi. iOS 15.2 has same problem. Using Webrtc without canvas, only simple video.
Chrome, Safari, Firefox with started video call freezes after incoming call.

Black screen instead video and page doesn't react on touches. Sometimes page reload helps, but usually I should close chrome and open it again.

[PhungDuong]

For this black camera error, we can sometimes reproduce it on iOS 15.3 and 15.4 on my WebRTC product.
Feedback would be appreciated.

[PhungDuong]

Hi @youenn,

We can confirm and reproduce this bug on iOS/iPadOS 15.4.1
Here are steps to reproduce:
1. Open the link using webRTC (https://it.dev-akameet.link/video/room/videocall)
2. Allow micro and camera.
3. Click the button to join the meeting.
Expected: The local video is normal.
Actual: The local video is black.

Note: From step 2 to step 3 do with speed as fast as possible. When you can't repo it, close the safari, open it again, and repeat the steps above, because the frequency isn't 100%.

[PhungDuong]

(In reply to PhungDuong from comment #16)
> Hi @youenn,
> 
> We can confirm and reproduce this bug on iOS/iPadOS 15.4.1
> Here are steps to reproduce:
> 1. Open the link using webRTC
> (https://it.dev-akameet.link/video/room/videocall)
> 2. Allow micro and camera.
> 3. Click the button to join the meeting.
> Expected: The local video is normal.
> Actual: The local video is black.
> 
> Note: From step 2 to step 3 do with speed as fast as possible. When you
> can't repo it, close the safari, open it again, and repeat the steps above,
> because the frequency isn't 100%.

Also can repo this on iOS/iPadOS 15.5

[Daniel Rossi]

Confirming this an issue, I've implemented canvas fullscreen support for iPhone using MediaStream and getting autoplay working was difficult. Muted stops autoplaying or it freezes. I think even playsinline causes it to fail. I don't need it inline I need a second video going fullscreen anyway. 

https://github.com/danrossi/canvas-pip-fullscreen/blob/main/src/CanvasFullscreen.js#L26

[jeeinn]

In iOS17.1.1 Safari will appear to autoplay but randomly go to full screen. 
In iOS17.2Beta3 it will automatically go to full screen black screen but won't play, when clicking on the picture-in-picture mode it can play normally

https://github.com/jeeinn/video_srcObject/blob/834922287206273c2e9bcad85819de775d4b1088/index.html#L65-L67

```
stream = canvas.captureStream();
video = document.getElementById('theVideo');
video.srcObject = stream;
```