Bug 206843 – Can't draw stream->video->texture->canvas without also drawing video.

• Component: WebRTC
• Status: NEW

[Nicholas Butko]

Created attachment 388902 [details]
source code for repro case

WebKit appears to stop streaming the camera feed to a <video> tag when the element is not visible, either because it is display none, or it is fully occluded by other DOM elements. Streaming to an off-screen <video> is common when drawing the camera feed to a texture for canvas display, for example to apply visual effects on top of the camera feed, or to synchronize display of the camera feed with the result of some visual processing (QR code detection, image target tracking, SLAM, etc.).

Our workaround solution is to make the camera feed visible but all-but-hidden, e.g. by displaying as a block element behind the drawing canvas, offset by one pixel:

  const setVideoForSafari = ({video, canvas}) => {
    if (!canvas.style['z-index']) {
      canvas.style['z-index'] = 1
    }
    canvas.style.top = '1px'
    canvas.style.left = '0px'
    canvas.style.position = 'absolute'
    video.setAttribute('playsinline', true)
    video.style.display = 'block'
    video.style['z-index'] = canvas.style['z-index'] - 1
    video.style.top = '0px'
    video.style.left = '0px'
    video.style.position = 'absolute'
    video.style.opacity = '0.01'
    video.parentNode.style.position = 'absolute'
  }

This approach adds inefficiency because camera pixels are drawn twice, and adds a subtle visual defect because the camera feed is one-pixel visible. This workaround was required for WebAR activations by Pink Floyd, BBC America's Dr Who, Bailey's, Jumanji, Johnnie Walker, Game of Thrones, Monopoly, Ally Bank, Adidas, Porsche, Eva Air, London's Waterloo Station, Spider-Man: Far From Home, The Phillies, Lego, General Mills, Red Bull, Toyota, and many more.

A basic repro case can be viewed on an iPhone here: 8th.io/k4c5d

The repro app does the following:
* getUserMedia -> stream -> video
* iOS only: set the video style to be display: block, z-index: canvas.style.zIndex - 1, 1 pixel offset from canvas
* video -> texture (there is a rotating pool of 10 textures)
* texture -> sepia shader -> canvas -> window
* on a button press, toggle video.style.display to block / none

We tested this with an iPhone11 Pro with iOS 13.3.

Source code for the repro is attached.

[Radar WebKit Bug Importer]

<rdar://problem/58946401>