| Summary: | getUserMedia video stream to texture sometimes gives old frames. | ||||||
|---|---|---|---|---|---|---|---|
| Product: | WebKit | Reporter: | Nicholas Butko <nb> | ||||
| Component: | WebRTC | Assignee: | Nobody <webkit-unassigned> | ||||
| Status: | NEW --- | ||||||
| Severity: | Normal | CC: | eric.carlson, graouts, jer.noble, simon.fraser, simontaylor1, webkit-bug-importer, youennf | ||||
| Priority: | P2 | Keywords: | InRadar | ||||
| Version: | Safari 13 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | iOS 13 | ||||||
| Attachments: |
|
||||||
I am wondering whether this still repro with WebKit ToT. We got rid off re-enqueuing samples when the display layer could not go fast enough, which could explain why old samples would come back. As you mention using a texture pool, this sounds like a duplicate of Bug 203148. Effectively there's an optimization in WebKit that prevents re-uploading frames if the source video hasn't changed since the last texImage2d call, but it fails to check whether a different texture is now bound. In that case the previous content of the newly-bound texture is left unchanged, which with a texture pool will probably appear to be an older frame. |
Created attachment 388497 [details] source code for repro case Sometimes calling gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, video) populates the texture with frame data that is old, usually 2-3 frames old but sometimes 15 or more. When drawing these textures to a canvas, this gives an impression that the video feed is jumping, because a scene from a while ago will flash momentarily. The effect is rare in normal conditions but can be reliably triggered by extending the length of an animation frame by a long random interval, e.g. 20ms-200ms. The effect can be mitigated by skipping every other animation frame, i.e. const nativeRaf = window.requestAnimationFrame window.requestAnimationFrame = (fn) => nativeRaf(() => nativeRaf((timestamp) => fn(timestamp))) A repro case can be viewed on an iphone here: 8th.io/ft4h9 The repro app does the following: * getUserMedia -> stream -> video * video -> texture (there is a rotating pool of 10 textures) * texture -> luminance shader -> framebuffer -> readpixels -> uint8array * texture -> canvas -> window * uint8array -> frame hash code based on pixel content -> frame hash history * search frame hash history to find duplicate hashes * if there is a duplicate hash, print the number of frames ago for the duplicate frame pixels (do this for the 5 most recent duplicates), and increment a counter for the number of duplicate frames. We tested this with an iPhone11 Pro with iOS 13.3. Source code for the repro is attached.