Bug 217578

Summary:

Webcam video from navigator.mediaDevices.getUserMedia() to 2D canvas fails on Safari on iPhone

Product:

WebKit

Reporter:

jujjyl

Component:

New Bugs

Assignee:

youenn fablet <youennf>

Status:

RESOLVED FIXED

Severity:

Normal

CC:

cdumez, changseok, eric.carlson, esprehn+autocc, ews-watchlist, ggaren, glenn, gyuyoung.kim, jer.noble, philipj, sergio, simon.fraser, webkit-bug-importer, youennf

Priority:

Keywords:

InRadar

Version:

Safari 14

Hardware:

iPhone / iPad

OS:

iOS 13

Attachments:

Description	Flags
Patch	none
Patch for landing	none

jujjyl

Reported 2020-10-11 04:47:28 PDT

STR: <html><head></head><body><canvas></canvas><script> navigator.mediaDevices.getUserMedia({ audio: false, video: true }).then((stream) => { var video = document.createElement('video'); video.srcObject = stream; video.play(); var canvas = document.querySelector('canvas'); var c2d = canvas.getContext('2d'); function draw() { canvas.width = video.videoWidth; canvas.height = video.videoHeight; c2d.drawImage(video, 0, 0, video.videoWidth, video.videoHeight); requestAnimationFrame(draw); } requestAnimationFrame(draw); }); </script></body></html> Works on desktop Firefox, Chrome and Edge, and on mobile Android Firefox and Chrome, but produces a black canvas on Safari.

Attachments
Patch (6.46 KB, patch) 2020-10-13 03:56 PDT, youenn fablet	no flags	Details Formatted Diff Diff
Patch for landing (6.49 KB, patch) 2020-10-13 09:12 PDT, youenn fablet	no flags	Details Formatted Diff Diff
Show Obsolete (1) View All Add attachment proposed patch, testcase, etc.

Radar WebKit Bug Importer

Comment 1 2020-10-11 10:50:40 PDT

<rdar://problem/70183875>

youenn fablet

Comment 2 2020-10-12 03:51:15 PDT

I reproed the issue. This does not repro if the video element is rendering.

youenn fablet

Comment 3 2020-10-13 00:40:10 PDT

Here is a fiddle that reproduces the issue: https://jsfiddle.net/kw8fbcma/4/ Capture probably fails as the samples are queued in the local sample buffer display layer which is not actually rendering.

youenn fablet

Comment 4 2020-10-13 03:56:29 PDT

Created attachment 411209 [details] Patch

Eric Carlson

Comment 5 2020-10-13 08:11:52 PDT

Comment on attachment 411209 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=411209&action=review > Source/WebCore/platform/graphics/MediaPlayer.cpp:968 > +void MediaPlayer::setVisibleForCanvas(bool b) > +{ > + m_visible = b; > + m_private->setVisibleForCanvas(b); s/b/visible/ > Source/WebCore/platform/graphics/MediaPlayerPrivate.h:99 > + virtual void setVisibleForCanvas(bool b) { setVisible(b); } Ditto

youenn fablet

Comment 6 2020-10-13 09:12:52 PDT

Created attachment 411223 [details] Patch for landing

EWS

Comment 7 2020-10-13 10:06:21 PDT

Committed r268398: <https://trac.webkit.org/changeset/268398> All reviewed patches have been landed. Closing bug and clearing flags on attachment 411223 [details].

Simon Fraser (smfr)

Comment 8 2020-10-13 11:30:28 PDT

Why no layout test?

youenn fablet

Comment 9 2020-10-13 11:47:57 PDT

(In reply to Simon Fraser (smfr) from comment #8) > Why no layout test? It would be nice but I found it hard to test. Ideas welcome. We would basically need to verify that AVSampleBufferDisplayLayer is not keeping references too much video samples in its queue but I do not see APIs to get that info. Ideally, we would want to validate that the mock device CVPixelBufferPool is able to recycle its pixel buffers at some reasonable point in time. That would make the mock very close to the real camera. Not sure how we can do that though. Maybe we can observe the CVPixelBuffer pointers.

Geoffrey Garen

Comment 10 2020-10-15 12:23:10 PDT

What happens if you do this: setVisibleForCanvas(true); setVisible(true); ? I think you do not become visible, because of the early return in MediaPlayerPrivateMediaStreamAVFObjC::setVisible. And maybe there are other early returns like this?

youenn fablet

Comment 11 2020-10-15 12:39:50 PDT

(In reply to Geoffrey Garen from comment #10) > What happens if you do this: > setVisibleForCanvas(true); > setVisible(true); > ? > > I think you do not become visible, because of the early return in > MediaPlayerPrivateMediaStreamAVFObjC::setVisible. And maybe there are other > early returns like this? For MediaPlayerPrivateMediaStreamAVFObjC, the MediaPlayer::setVisibleForCanvas(true) call is a no-op: - MediaPlayerPrivateMediaStreamAVFObjC::setVisibleForCanvas is a no-op. - MediaPlayer::m_visible is not used by MediaPlayerPrivateMediaStreamAVFObjC. The MediaPlayer::setVisible(true) call would end up setting MediaPlayerPrivateMediaStreamAVFObjC::m_visible to true. For other MediaPlayerPrivateXX, this patch is hopefully a no-op as MediaPlayer::setVisibleForCanvas is the same as MediaPlayer::setVisible. I'll double check this tomorrow but I think we can remove MediaPlayer::m_visible.

youenn fablet

Comment 12 2020-10-16 01:38:16 PDT

> I'll double check this tomorrow but I think we can remove > MediaPlayer::m_visible. Filed https://bugs.webkit.org/show_bug.cgi?id=217810

Note You need to log in before you can comment on or make changes to this bug.