Bug 206636 - getUserMedia video stream to texture sometimes gives old frames.
Summary: getUserMedia video stream to texture sometimes gives old frames.
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebRTC (show other bugs)
Version: Safari 13
Hardware: Unspecified iOS 13
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2020-01-22 17:40 PST by Nicholas Butko
Modified: 2022-01-07 23:41 PST (History)
7 users (show)

See Also:


Attachments
source code for repro case (7.34 KB, text/javascript)
2020-01-22 17:40 PST, Nicholas Butko
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Nicholas Butko 2020-01-22 17:40:47 PST
Created attachment 388497 [details]
source code for repro case

Sometimes calling 

gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, video)

populates the texture with frame data that is old, usually 2-3 frames old but sometimes 15 or more.  When drawing these textures to a canvas, this gives an impression that the video feed is jumping, because a scene from a while ago will flash momentarily.

The effect is rare in normal conditions but can be reliably triggered by extending the length of an animation frame by a long random interval, e.g. 20ms-200ms.

The effect can be mitigated by skipping every other animation frame, i.e. 

const nativeRaf = window.requestAnimationFrame
window.requestAnimationFrame = (fn) => nativeRaf(() => nativeRaf((timestamp) => fn(timestamp)))

A repro case can be viewed on an iphone here: 8th.io/ft4h9

The repro app does the following:
* getUserMedia -> stream -> video
* video -> texture (there is a rotating pool of 10 textures)
* texture -> luminance shader -> framebuffer -> readpixels -> uint8array
* texture -> canvas -> window
* uint8array -> frame hash code based on pixel content -> frame hash history
* search frame hash history to find duplicate hashes
* if there is a duplicate hash, print the number of frames ago for the duplicate frame pixels (do this for the 5 most recent duplicates), and increment a counter for the number of duplicate frames.

We tested this with an iPhone11 Pro with iOS 13.3.

Source code for the repro is attached.
Comment 1 Radar WebKit Bug Importer 2020-01-22 22:52:34 PST
<rdar://problem/58825426>
Comment 2 youenn fablet 2020-01-23 06:00:55 PST
I am wondering whether this still repro with WebKit ToT.

We got rid off re-enqueuing samples when the display layer could not go fast enough, which could explain why old samples would come back.
Comment 3 Simon Taylor 2022-01-07 23:41:03 PST
As you mention using a texture pool, this sounds like a duplicate of Bug 203148.

Effectively there's an optimization in WebKit that prevents re-uploading frames if the source video hasn't changed since the last texImage2d call, but it fails to check whether a different texture is now bound.

In that case the previous content of the newly-bound texture is left unchanged, which with a texture pool will probably appear to be an older frame.